Last week, I set up something that I have tried and failed to convince many operations managers to implement: getting all production configuration files and scripts under source control. I’m not sure if it is out of security concerns or a fear of contaminating the production environment with version control software or because revision control is considered to be strictly in the domain of developers, but I’ve found a major resistance to the whole notion
of using SCM from operations folk (unless it is introduced under the banner of Configuration Management, and then it’s some antiquated SCM like PVCS). And, no, backup is no substitute for a good version control system.

Having configuration files under version control provides a comprehensive summary of changes to the production environment. If configuration is being altered, it is easy to see what’s changed and to revert any undesired modifications. It also makes it easy to analyze differences between one environment and another, which greatly simplifies the task of keeping test in sync with production.

We use Subversion in development for managing source code, so it’s a natural fit for us to use in production configuration as well. We simply tunnel svn over ssh to access the repository from each environment. We use the same repository as development, but for the security conscious, it’s easiest enough to set up a separate repository.

One challenge about bringing configuration files under version control is that the files tend to be spread out all over the file system. One option is to try to configure each application to load its configuration files from a common location. But this tends to be hassle and may not be supported by some applications. Moreover, it generally means taking that system offline to change its config file location. A better alternative, at least in *nix systems, is to use symbolic links to reference a common folder under version control.

In terms of organizing the files, we simply have a folder structure that looks like:

/trunk
  /configuration
    /<server or server class>
      /<application>
        /<configuration files and scripts>

I’ve always found the term Configuration Management to be pretty confusing. It can mean anything from “our code’s in version control” to having a separate team with their own SCM system serving as guardians tracking and vetting everything going from development into production. So in this context, I think that the term Configuration File Management is maybe a more accurate description of what I’m going for here. Regardless, it’s a practice that I would like to see more teams follow.