The downside to this level of monitoring is that it can amount to a lot of messages. Our challenge is to manage the signal-to-noise ratio so that:

• we are only notified about things that require immediate action,
• we don’t suffer from information overload; and
• emails that matter aren’t buried under a bunch of emails that don’t.

As part of our 5 Whys activity for production issues, we have found that most production issues actually occurred first in test, but just went unnoticed. This provides a compelling reason to keep the signal ratio high in all environments. Any time that we find ourselves automatically archiving or filtering an alert indicates an opportunity for improvement.

We have found that refining and tuning these alert messages is an ongoing maintenance activity. As part of our weekly meeting, we try to select one message to clarify or dispatch each week. We have a script that trawls the support emails received in the past week and builds a pareto distribution of the number of messages by logger. This helps us decide where to focus our efforts and to quantify the impact of our actions on the volume of messages we receive.

Determining what kinds of things we need to be alerted about is difficult to assess in advance. Often things that we are concerned about when building a feature turn out to less important in production, and conversely, we miss things in development that turn out to be very important once real customers start using them. Fortunately, deploying every week gives plenty of opportunity for improvement. Also if a message is logged more frequently than intended, we only have to put up with it for a week before it can be rectified.

I should mention that we have a circuit breaker in place in the log monitor. We do not allow duplicate messages to be sent any more frequently than once per hour. (Relatively early on we managed to get temporarily blacklisted by a mail provider when an errant message was generated much too frequently).

In terms of managing the signal-to-noise ratio, I’ve found that there are a few broad categories of messages to deal with:

• Message source: did the message originate in our code or in one of the libraries that we depend on? Clearly, warnings coming from our code are easier deal with than those from outside. I’ve been frustrated by the laissez-faire attitude that various open source Java frameworks take to logging errors and warnings. We use Apache CXF, and it generates over 10 severe messages with lengthy stacktraces every time the application starts up to inform us that JMS integration through JNDI is not enabled. WTF?!? Sometimes these messages can be controlled by setting custom log levels for specific loggers, but not always. And it typically feels a bit disconcerting to shut down logging just in case something important is missed.
• System conditions: was the message generated during normal operations, during a shut down or a crash? I’ve found that systems tend to be very noisy during shutdown, but (perversely) pretty quiet during a crash. In the world of Java app servers where memory leaks across deployments are common, trying to quietly quiesce a server is a real challenge.

In the (enterprise) environments that I’ve worked in the past, there was very little interaction between development and operations. Logs were used only for analyzing severe production problems – generally after a severe system problem (a crash) or a user had reported a problem. The log files were poorly tuned for diagnosing problems and they tended to be full of junk – problems that no one had noticed or reported that may have been going on for months (or longer).

In contrast, the approach that we follow at my current company means we are able to use logs to proactively find and remedy problems. It requires effort to maintain a high signal-to-noise ratio, but it is very worthwhile.

The session was divided into 3 sections: I began with a presentation introducing the topic; next, participants were encouraged to work in small groups to design an andon dashboard for their project teams; the remainder of the session was spent discussing the implementation details involved in building a dashboard. My plan for the latter half of the session was to get participants to integrate metric data from different sources via RESTful XML web services into a simple Rails-based dashboard that I have thrown together, but given the size and interest of the group, it seemed easiest to just discuss the implementation rather than go through with the exercise. I had also intended to demo using a digital photo frame as a digital dashboard, but my photo frame couldn’t get onto the hotel’s wireless.

If you are interested in a copy of the presentation, I’ve uploaded it in Keynote or PowerPoint 2003. Please feel free to use the contents of the slides. The presentation is done in the Lessig style, so it might not be the easiest to follow. If you end up presenting on the topic, let me know — I’m interested to track the thinking and ideas as they evolve. Here’s the embedded slideshow from Slideshare:

As for the code that I used in the demo, I’ll get it uploaded to github soon.

Corrections:
There are a few statistics that I cited incorrectly off the top of my head during the podcast:

• The CruiseControl.NET project has consumed over 46 person years of effort – at least based on what oloh can divine from our subversion repository.
• The CruiseControl.NET project has had over 800,000 downloads – not 80,000 as I said during the interview. I was off by an order of magnitude. Oh and this doesn’t include all of the direct downloads from CCNetLive