Provisioning a new host in five minutes rather than an hour. Licensing for variable loads slashed by a large fraction. Another ‘9’ in availability (from 99%, say, to 99.9%).

Many of the potentials of a “DevOps” orientation are familiar by now among systems administrators. Not so well known are several of the time-tested techniques programmers have long used to make the most of their software-oriented development efforts, and that also apply to administrative work.

It’s not that devops haven’t heard of parametrization, logging, version control, and peer review, or that programmers are smarter for making more use of them; mostly it’s just that these techniques haven’t yet become part of the routine culture of system administration. It’s time not just to enjoy the advantages of software-defined devops, but also to manage the technical debt it too-often incurs.

DRY for better configuration

To configure a new virtual machine or deploy a new service across a server farm with just a few lines of VMX, Puppet, or so on is exciting, and even liberating, when compared to the tedious chore of fumbling with a physical machine’s limitations and point-and-click clumsiness. Even the most mission-critical applications are increasingly living in a virtualized, software-defined world. Is there “boilerplate” in your configuration definitions, though? Do you have to be careful to make sure that the ‘MY_TEST_SERVER_37‘ hostname in line 4 matches the IP address of line 17?

Automate your way out of such error-prone situations. Programmers often intone “DRY”: don’t repeat yourself. The principle applies equally to any other textual realm, including configuration definition. Many modern configuration languages include variables with which to parametrize definitions. Those that don’t can be generated with pre-processors or similar facilities. The particular technology you use–anything from XSLT in a browser to a Cobol program running on a legacy CPM/86 antique–is secondary to the idea of leveraging the computing power available to you to wring the repetitive work out of your tasks.

Logs of all shapes and sizes

Sysads are experts in logs. Many of us think of Apache or ssh logs as recreational reading. Too often, though, we concentrate on their consumption, and forget the opportunity we have to produce them.

When you’re working through a configuration puzzle, don’t dissipate your brainpower with assumptions about state and flow; instead, have your processes automatically record these variables in systematic logs, so you can examine them at your leisure. Also recognize your freedom to treat different values with different responses: I like to

• log everything,
• e-mail myself notices about events that might bear on current projects, and
• SMS (telephone-text) myself when certain alarms fire,

for instance. A good logging framework makes all this easy.

Version control is a simple necessity

All your configurations belong in version control (VC). Period. That’s non-negotiable, as far as I’m concerned–and what’s checked in better correspond to what is “live”! Read a good book like Eric Sink’s on what VC does for you, and how to make the most of it. Sink’s focus is on distributed programming teams who use a tool like Git, while I generally recommend SVN or another more “conservative” VC system for devops. Those details don’t matter nearly as much, though, as the decision to use some VC, and use it well.

More eyeballs

A final lesson devops can learn from programmers is to make good use of your teammates. Sysads’ pride in solving problems on their own is a good instinct; it’s easy to take it too far, though. Programmers have led the way in practices from “pair programming” to “code review“; sysads need to learn about these forms of co-operation, especially for assignments realized as delivery of textual artifacts such as configuration definitions.