After spending the better part of a day banging our heads against the problem, we had isolated it to a few statements. The code had changed in this area in the last release (one of the advantages of weekly releases is that it is easier to pinpoint the source of a problem as each release contains only one week’s worth of changes) but not in a way that should have caused the problem we were seeing. And everything ran correctly in the test environment. It seemed like the problem could be environment-related.

We didn’t have the necessary hooks and logging to properly exercise the problem area in isolation, but after a quick patch release, we did (another benefit of zero-downtime deployment is that it allows greater flexibility with the time and frequency of deployment). After the patch deploy, we were able to see that the production system returned one fewer result than the corresponding functionality run against the test environment. If the system was supposed to return only one result, the production system was returning none – which explained the gaps on the chart. The problem was clearly at a SQL driver or database level.

Just prior to the release, the MySQL driver had been upgraded to the latest version (5.1.10). We had been running this version in the test environment for several weeks without issue, so it seemed odd that it could be the source of the problem. The version of the database, however, was inconsistent between the two environments (MySQL 5.1.39 in production vs 5.1.36 in test). The newer database version had been running fine in the production environment for several days and hadn’t been touched with the release, so it seemed odd that this could be the source of the problem. This was enough to go from, however, and we reverted the version of the MySQL driver on the web servers which ended up fixing the problem.

Later, while trolling the MySQL release logs, I came across the nasty bug that bit us. Evidently, the 5.1.10 driver had changed the format of dates in a way that triggered this bug in 5.1.39. So it was the combination of these two version of the software that caused the problem – each in isolation worked fine. The issue has been fixed in MySQL 5.1.41, but it’s a pretty serious bug in core SQL functionality to come out of a sanctioned release.

Coming out of our 5 Whys analysis, the experience points to several ways that we need to tighten up our process as well.

1. We need to separate environment changes from software releases. Even seemingly innocuous changes can have repercussions when combined with other systems. Keeping environment changes separate will make it easier and faster to pinpoint the source of problems (ie. is it an environment problem or a software problem?).
2. We need to narrow the discrepancies between the test and production environments. Environment discrepancies are a common source of unexpected risk. While it’s not feasible to keep the environments perfectly in sync, this could be better. Ironically, it was an attempt to make the environments more consistent that caused this problem.
3. We need a more comprehensive set of automated validation tests that we can run against the system subsequent to deployment that verifies the integrity of the release. This is one of the tasks for this week.

While these actions may seem fairly obvious, it often (unfortunately) takes getting bitten by these types of bugs to illuminate areas where we need to improve. Have you faced something similar? What additional actions do you take to prevent these types of environment problems?

Overall, I was hugely impressed – the ad was able to drive fairly significant traffic to the site at a very reasonable cost. We were able to get great ad placement (often ours was the only ad showing) with a very low cost-per-click. Certainly in comparison with more conventional forms of advertisement, AdWords – especially with Google Analytics integration – offers more control and much better feedback and information.

One thing that struck me going through the process of getting set up with AdWords is that there is significant depth to the application and much to be learned about how to build an optimal campaign using it. One thing that I struggled a bit with was how to determine the estimated cost-per-click for newly added keywords. It seemed to be available with after digging around, but it strikes me that this is something that should be directly visible on the ad group page. Another thing that I found was that while cost for a phrase keyword match was generally cheaper than broad keyword match, this was not always the case. Some justification of the cost for a keyword would be useful (ie. “20 clients are already using these keywords” or something).

If you have any useful tips, advice or articles to read on getting the best results out of AdWords, please let me know.

If you’re interested in seeing in real-time the energy savings realized by the actions taken by some local sites for Earth Hour, check out the Pulse Dashboards.

The session was divided into 3 sections: I began with a presentation introducing the topic; next, participants were encouraged to work in small groups to design an andon dashboard for their project teams; the remainder of the session was spent discussing the implementation details involved in building a dashboard. My plan for the latter half of the session was to get participants to integrate metric data from different sources via RESTful XML web services into a simple Rails-based dashboard that I have thrown together, but given the size and interest of the group, it seemed easiest to just discuss the implementation rather than go through with the exercise. I had also intended to demo using a digital photo frame as a digital dashboard, but my photo frame couldn’t get onto the hotel’s wireless.

If you are interested in a copy of the presentation, I’ve uploaded it in Keynote or PowerPoint 2003. Please feel free to use the contents of the slides. The presentation is done in the Lessig style, so it might not be the easiest to follow. If you end up presenting on the topic, let me know — I’m interested to track the thinking and ideas as they evolve. Here’s the embedded slideshow from Slideshare:

As for the code that I used in the demo, I’ll get it uploaded to github soon.

Corrections:
There are a few statistics that I cited incorrectly off the top of my head during the podcast:

• The CruiseControl.NET project has consumed over 46 person years of effort – at least based on what oloh can divine from our subversion repository.
• The CruiseControl.NET project has had over 800,000 downloads – not 80,000 as I said during the interview. I was off by an order of magnitude. Oh and this doesn’t include all of the direct downloads from CCNetLive

• Database Migration in .NET (Sample Code)
• Recommended CI Practices
• Operations Database: Bridging the Gap between Development and Production

Unfortunately I ended up attending relatively few of the sessions as I was pretty busy preparing the materials for my presentations. But what I did see was quite good. I particularly liked Derek Hatcher’s Leveraging the Amazon Platform (EC2 and S3) and Greg Young’s DDDD, Unshackle Your Domain.

What I enjoyed most about the conference was getting to know and learn from some of the experts in a new technology circle. I missed last year’s DevTeach in Vancouver as I was in China at the time but I was glad to have made it out this one.

I’m in Toronto this week speaking at DevTeach. One of the sessions that I will be doing is on Recommended Practices for Continuous Integration and I’ll be referring this paper.

This year’s Earth Hour was a very busy event for me and my new company Small Energy Group. In partnership with the University of British Columbia, Vancouver City Hall, and the Village of Hartley Bay, we monitored the energy use over the day for each site to quantify and make visible the energy savings impact of their actions. UBC had committed to powering down non-essential services over the entire day; City Hall shut off the floodlights lighting the art deco facade for the building; and in Hartley Bay, they turned off the diesel generator that supplies power to the whole community plunging the village into darkness.

You can see the results of the savings achieved by each site by checking out the web site: http://earthhour.smallenergygroup.com/.

I still do quite a bit of work in Windows (both development and work with MS Office). I purchased a copy of VMWare Fusion to virtualize my Windows Bootcamp partition. Fusion makes it easy to operate concurrently on both Windows and OSX without any noticeable performance hit. I can drag files or copy and paste text back and forth from one environment to the other. Unlike Parallels, it provides a performant virtualized environment that I would actually do development in. That said, I have noticed some strangeness when trying to access a DVD drive or USB devices while running Fusion. I haven’t investigated enough to figure out what the issue is, but these problems seemed to disappear after shutting down the virtualized environment.

Another thing about the MacBook that is quite nice is the little remote that it comes with. It works well for driving presentations, pausing movies and shuffling songs. I’ve previously owned a USB-enabled remote for presentations, which I’ve now happily passed on to my wife.

Speaking of movies, I’ve been quite impressed with the quality of the MacBook Superdrive. I have a few DVD movies that I picked up in China that I haven’t been able to play on a PC. But the Superdrive can read them.

My one big annoyance is the lack of support for a direct VGA connection. I hate having to carry around the DVI-VGA dongle. It’s easy to forget or leave behind attached to the projector. I understand why Apple would provide support for the higher-quality, more future-proof DVI, but the majority of projectors and external monitors out there still require a VGA connection. Would providing VGA support really been such a big effort?

• Automated Database Evolution and Deployment
• Recommended practices for Continuous Integration
• Home-grown Production System Monitoring and Report

Let me know if you’re planning to attend the event and if there’s anything in particular that you would like to see covered in this event.