Guest Post: Test First in Operations

•

Today's post is by Matthias Marschall of the fantastic Agile Web Operations blog. Matthias is the CTO of Autoplenum, when he's not blogging.

Monitoring a server is very similar to continuously building and testing code.

In development, you write tested code, commit it to version control and the CI server tries to build and deploy the code to a test system and executes all tests.

In operations, you write a script (e.g. a puppet manifest or a capistrano recipe) describing the desired setup of your server and commit it to version control. A daemon like puppetd deploys the configuration changes to a test system. But who takes care of testing your new setup?

Testing Configuration Changes with Monitoring Tools

Just like development, it's a good idea to make sure that there's a way to verify every critical aspect of your server's configuration. In operations, monitoring services like Nagios act as tests. They are the last step of your operations build chain: Verifying that your deployed configuration changes work as expected and have no undesired side effects.

Test First In Operations

In operations, the fundamental guiding principle is often never touch a running system.

In development, you tend to think the same only about untested legacy code. Having tests, you can touch your code anytime as you know that your CI system will catch any mistakes.

A test driven approach is possible in operations as well, but I've hardly seen it anywhere:

  1. Write a monitoring service for the config change you want to make and ensure that monitoring goes red.
  2. Implement the config change on the system and watch your monitoring system go green.
  3. Rinse and repeat until you've implemented all desired configuration changes.

    4. Like in development, you should have a test server for deploying and testing your configuration changes before deploying them to production. No shortcuts here.

Having tests makes you fast and safe, no matter whether you're developing your web application oroperating it.

Image by Docklandsboy


DevOps New Zealand