Deploying to Production Every Week (Owen Rogers), part 1 of 5 from Agile Vancouver on Vimeo.

There is no way that we could have maintained this pace if we were taking down the test system for 10 minutes with each deploy. We needed to be able to continuously test the system throughout. This to me is one of the unsung values of zero-downtime deployment. Everyone normally focuses on the benefit of your users not seeing a fail whale page if they hit your site during a deploy. But let’s face it: unless you’re practicing continuous deployment, you’re deploying to test much more often than you are deploying to production. And the frequency with which we can cycle through and get feedback on changes in the test environment is one of the primary limiting factors determining how frequently we can release the software.

While many people that I’ve talked with like the idea of zero-downtime deployment, they have little access, control or influence over the production environment. So my advice is to start with the environments that you own, like test. Get it set up and running solidly there first. This greatly lowers the barrier to then rolling it out to production. And, as I describe above, the benefits of having zero-downtime in test are substantial.

In my next post, I’ll try to provide some more technical details about our zero-downtime set up.

Here are the slides from the presentation. They did evolve a bit over the course of the tech talks (and hopefully improve).

The presentation also stimulated some good side chatter on twitter. In general, zero-downtime database deployment, continuous monitoring and WAGMI seemed to be popular topics. Thanks to everyone who made it out and contributed. Your feedback has been very helpful in refining the presentation.

Also, a big thanks to Naresh and Amit for organizing the event and ensuring that we were well taken care of.

Hence, when releasing software to production every week, the cost of making style changes can be prohibitive. It is difficult to regression test all impacted pages in all browser combinations within a reasonable amount of time. Another challenge is that in CSS there are many ways to achieve the same thing, though each approach could render differently in different browsers. And in a site that is changing rapidly, existing designs need to evolve to incorporate new features, usability improvements, user feedback and better ways of doing things.

Stylesheets can’t be seen as being too risky to change. Otherwise they will be subverted through local inline styles and other workarounds (the same goes for common javascript libraries or pretty much any shared component for that matter). While local styles may seem expedient for that particular page or feature, they only introduce inconsistencies and make the site more difficult to maintain in the long term. So what’s a web dev to do?

We’ve developed an approach to help deal with this problem. I accept that there are probably smarter ways to achieve this – if you have a better idea, please let me know. Here goes:

1. All styles should be relative to some top-level class. For example:
   form.standard { margin: 0px 150px; }
   The class describes the type of component that is being styled – the type of table, form, or component we are designing here. In the example, we are defining a “standard” form, which is a particular type of form. Using a top-level class allows for other types of forms to be styled as required.
2. All sub-elements are defined relative to the top-level class:
   form.standard label { margin-left: -150px; }
   All labels for a “standard” form have a negative left margin. It is acceptable (even preferable) to style bare tags as long as the style is relative to a top-level class. This keeps the markup simple and consistent. We only add CSS classes to child elements as required.
3. The corollary to the above is that it is not acceptable to style bare tags (unless we are doing a style reset). The problem with styling bare tags directly is that it locks us in to one specific style and that makes it difficult to evolve the design of the site. For example:
   label { font-weight: bold }
   means that all form labels will be in bold. If we introduce a form later that shouldn’t have bold labels, we will have to explicitly override this style (which tends to be brittle and limiting).
4. Now, if we need to evolve an existing style, we have a few options. Say we want to replace the “standard” form with a new design. Rather than change the styles for the “standard” form class directly, which would immediately impact every page where this style is used, we can incrementally and selectively rollout the new design on a page-by-page basis. If the new style is significantly different from the initial style, we can fork the original style by defining a new top-level class:
   form.danger { background-color: red; }
   Simply by changing the class attribute for each form element from “standard” to “danger”, we can then roll out the new style to select pages testing the design in all supported browsers as we go. Think of this being like continuous integration for site design.
5. If the style changes are relatively minimal, we can override the style for specific elements. One way to achieve this is to use multiple top-level CSS classes. For example, we could incrementally apply both the “standard” and the “danger” classes to the form elements on each page testing as we go. The “danger” class could override styles as required – though dealing with precedence can be tricky. Alternately, the new class could be defined relative to a top-level identifier. This solves the precedence problem as styles defined relative to an identifier take precedence over styles that are relative to a class. Another option is to define specific classes for the child elements to be restyled – but this means changing a lot more markup during the rollout.

That’s it. It seems pretty simple – intuitive even. But I haven’t found many references to how others tackle this problem. Again, if you have a better idea, please let me know.

My intention with the presentation was to lay out the basis for an argument that would be subsequently debated. This proved to be more challenging than my last presentation on frequent releases which more of an experience report. It’s difficult to clearly convey the layers of a logical argument through a presentation. I’m impressed by how lawyers manage to do it.

One of the things that I asked attendees to do was to list the top three things that are preventing them from releasing software every week. I’ve compiled their responses in the chart below:

The results are hardly scientific, but I was heartened to see substantial overlap between the concerns that I addressed in the presentation and those identified by the audience. It turned out quite a large contingent of the audience are rich client developers, which brings its own shared of deployment headaches. Many are also working in regulated industries that require their software to be submitted to third party certification agencies for review. To those that attended the session, thanks for participating and for sharing your concerns with me.

• Arlo Belshee
• Eric Evans
• Michael Feathers
• Martin Fowler
• Michael Hugos
• Michael Nygard
• Jeff Patton
• Mary Poppendieck
• Linda Rising
• Johanna Rothmann

We are looking for speakers and we welcome you to send in your session proposals or experience reports. If you are interested in presenting at this year’s conference, please send your session proposal to proposals2009@agilevancouver.net. Proposals should include:

• session title and short description (300 words or less)
• speaker bio
• speaker photo
• learning objectives: what do you intend for participants to learn from attending your session?
• logistics considerations: what special equipment (whiteboards, print-outs, easels, etc) do you require?
• email address that you would like us to use for all conference-related correspondence

The deadline for submitted proposals is September 2nd and we will announce selected proposals by September 17th.

• Creating a weekly release process
• Zero down-time deployment
• Continuous Monitoring
• 5 whys
• Automated deployment

As I haven’t been doing .NET development in over a year, I feel like a bit of an impostor at the conference. However, I think that the ideas and experiences of short release cycles transcend technology. I also think that there’s a lot that the Java and .NET communities can learn from each other.

Here are the slides:

The downside to this level of monitoring is that it can amount to a lot of messages. Our challenge is to manage the signal-to-noise ratio so that:

• we are only notified about things that require immediate action,
• we don’t suffer from information overload; and
• emails that matter aren’t buried under a bunch of emails that don’t.

As part of our 5 Whys activity for production issues, we have found that most production issues actually occurred first in test, but just went unnoticed. This provides a compelling reason to keep the signal ratio high in all environments. Any time that we find ourselves automatically archiving or filtering an alert indicates an opportunity for improvement.

We have found that refining and tuning these alert messages is an ongoing maintenance activity. As part of our weekly meeting, we try to select one message to clarify or dispatch each week. We have a script that trawls the support emails received in the past week and builds a pareto distribution of the number of messages by logger. This helps us decide where to focus our efforts and to quantify the impact of our actions on the volume of messages we receive.

Determining what kinds of things we need to be alerted about is difficult to assess in advance. Often things that we are concerned about when building a feature turn out to less important in production, and conversely, we miss things in development that turn out to be very important once real customers start using them. Fortunately, deploying every week gives plenty of opportunity for improvement. Also if a message is logged more frequently than intended, we only have to put up with it for a week before it can be rectified.

I should mention that we have a circuit breaker in place in the log monitor. We do not allow duplicate messages to be sent any more frequently than once per hour. (Relatively early on we managed to get temporarily blacklisted by a mail provider when an errant message was generated much too frequently).

In terms of managing the signal-to-noise ratio, I’ve found that there are a few broad categories of messages to deal with:

• Message source: did the message originate in our code or in one of the libraries that we depend on? Clearly, warnings coming from our code are easier deal with than those from outside. I’ve been frustrated by the laissez-faire attitude that various open source Java frameworks take to logging errors and warnings. We use Apache CXF, and it generates over 10 severe messages with lengthy stacktraces every time the application starts up to inform us that JMS integration through JNDI is not enabled. WTF?!? Sometimes these messages can be controlled by setting custom log levels for specific loggers, but not always. And it typically feels a bit disconcerting to shut down logging just in case something important is missed.
• System conditions: was the message generated during normal operations, during a shut down or a crash? I’ve found that systems tend to be very noisy during shutdown, but (perversely) pretty quiet during a crash. In the world of Java app servers where memory leaks across deployments are common, trying to quietly quiesce a server is a real challenge.

In the (enterprise) environments that I’ve worked in the past, there was very little interaction between development and operations. Logs were used only for analyzing severe production problems – generally after a severe system problem (a crash) or a user had reported a problem. The log files were poorly tuned for diagnosing problems and they tended to be full of junk – problems that no one had noticed or reported that may have been going on for months (or longer).

In contrast, the approach that we follow at my current company means we are able to use logs to proactively find and remedy problems. It requires effort to maintain a high signal-to-noise ratio, but it is very worthwhile.

One thing that I was surprised to learn was that IMVU started out with continuous deployment. They were deploying to production with every commit before they had an automated build server or extensive automated test coverage in place. Intuitively this seemed completely backwards to me – surely it would be better to start with CI, build up the test coverage until it reached an acceptable level and then work on deploying continuously. In retrospect and with a better understanding of their context, their approach makes perfect sense. Moreover, approaching the problem from the direction that I had intuitively is a recipe for never reaching a point where continuous deployment is feasible.

Initially, IMVU sought to quickly build a product that would prove out the soundness of their ideas and test the validity of their business model. Their initial users were super early adopters who were willing to trade quality for access to new features. Getting features and fixes into hands of users was the greatest priority – a test environment would just get in the way and slow down the validation coming from having code running in production. As the product matured, they were able to ratchet up the quality to prevent regression on features that had been truly embraced by their customers.

Second, leveraging a dynamic scripting language (like PHP) for building web applications made it easy to quickly set up a simple, non-disruptive deployment process. There’s no compilation or packaging steps which would generally be performed by an automated build server – just copy and change the symlink.

Third, they evolved ways to selectively expose functionality to sets of users. As Eric said, “at IMVU, ‘release’ is a marketing term”. New functionality could be living in production for days or weeks before being released to the majority of users. They could test, get feedback and refine a new feature with a subset of users until it was ready for wider consumption. Users were not just an extension of the testing team – they were an extension of the product design team.

Understanding these three factors makes it clear as to why continuous deployment was a starting point for IMVU. In contrast, at most organizations – especially those with mature products – high quality is the starting point. It is assumed that users will not tolerate any decrease in quality. Users should only see new functionality once it is ready, fully implemented and thoroughly tested, lest they get a bad impression of the product that could adversely affect the company’s brand. They would rather build the wrong product well than risk this kind of exposure. In this context, the automated test coverage would need to be so good as to render continuous deployment infeasible for most systems. Starting instead from a position where feedback cycle time is the priority and allowing quality to ratchet up as the product matures provides a more natural lead in to continuous deployment.

For my company, even though we do weekly deployments, we’re still a fair way off from being able to deploy continuously. As we are operating in a new and rapidly evolving market, we focus on building and releasing a simple initial version of new features that demonstrate the potential of the software. We can then receive feedback and invest more effort in expanding those features that resonate with our clients. While we do routinely selectively expose new functionality to a subset of users (generally internal users) to solicit feedback, we still need to create more sophisticated ways to do user segmentation. Aside from the obvious bugbear of automated test coverage (we use JUnit and Selenium, but our coverage isn’t nearly good enough), our main blocking issue from a technology perspective is the deployment process itself.

To deploy continuously, the deployment has to be quick and it has to be transparent to end users (ie. there should be no visible downtime). Performing a rollback should have the same characteristics. Our deployment process is automated, but in the world of Java application servers (even lightweight ones like Glassfish) deployment is anything but fast. Deployment entails all kinds work that the app server needs to do (parsing configuration files, generating WSDLs, starting thread pools, etc) during which the application is unresponsive. Also, because of memory leak issues in the container, we always restart the application server with each deployment anyway. All in all, the only way to avoid downtime is to pull the application server out of the load balancer pool until the deployment completes. Rollback is the same process in reverse.

A bit of an aside, but I know of some teams that package Glassfish with their app, inverting the container metaphor and simply treating it as another library/dependency. This makes it easier to just flip the symlink on deployment and rollback. It’s an interesting idea, as long as you don’t mind copying a massive WAR to production with each deploy (which for us would just shift the deployment bottleneck to the network).

We have made a fair bit of head way on streamlining our deployment process, and while we’re not ready to do continuous deployments into production, I am trying to get us into a position where we can do continuous deployment to test. I used to be of the opinion that deployment to test was something that should be controlled by testers (via a deploy button on the automated build server). Most testers want to work against a stable baseline, limiting the number of variables that they are dealing with when testing the app. But this is a fallacy because a batch of changes is simply piling up behind whatever version is deployed into test. It’s classic batch-and-queue thinking.

What if deployments happened without downtime in a way that was invisible to the tester or the end user? What if test coverage was sufficient to ensure that there would be no regression on major areas of functionality? I think that the fears of continuous deployment into test and the need for a stable baseline would evaporate. Moreover, this is something that we would want to test because it would mirror the experience of users using the site when a new version goes into production. In our office, every time we do a deployment to test, someone needs to call out “deploying to test”. This too would go away.

That’s the plan anyway. Over the next couple of weeks, I’ll see if we can move closer to achieving it. I’ll let you know how it goes.

Far future expires means setting the expires HTTP header for static content to some date far in the future. Effectively, this means that static content within a web site will always be loaded from the browser cache after it is first requested. This has the impact of greatly improving the load time for your site as well as reducing the number of requests sent to your web servers. The flip-side, however, is that because the cached content never expires, if you do need to change an image or a stylesheet then the user will need to clear their browser cache before they see it.

Hence, taking advantage of far future expires means taking responsibility for versioning static content on the server. Any time static content changes, it needs to be served up from a different URL. In his book, Steve Souders alludes to the approach that they follow at Yahoo! to achieve this, but he doesn’t give enough detail to just go ahead and implement it. So here is how we’re solving the problem.

We’re currently using two approaches to versioning static content: one for images and one for SWFs, CSS, and Javascript:

• Every time an image is added to our site, we place it in a folder named after the current release (ie. /images/1.21/header.png). If we need to update an image then we move it from the folder it’s in to the folder for the current release and then update all links to the image accordingly. While this approach does require some manual effort, it has the advantage of being incredibly simple and easy to get going immediately. Because images change relatively infrequently within the site, this approach creates minimal overhead. It also only means that images that have changed from release to release get reloaded. The majority of the images will stay cached because they haven’t changed.
• Other static content like stylesheets, scripts and Flex applications change more frequently. They are versioned automatically with every build by getting copied to a folder named after the current build number and then bundled into the deployment package. We then dynamically build the path/URL to these resources using the current build number loaded from a bundled text file resource. This approach has the benefit of being completely automated. The only (small) disadvantage is that the version of the content changes with every release (as we’re releasing every week, this is quite often) regardless of whether the content has changed or not. However, given that this content generally changes weekly, it isn’t a problem.

As far as setting the expires and cache control headers, we’re using nginx as a reverse proxy server which makes it trivial to set HTTP headers by the file extension for each requested URI.

If you have suggestions for a better way to version static content or if I can provide more clarity on the approach that we’re using, please let me know.