Are you diving into automation without a safety net? Be careful automating what you can't test

1. Configuration Automation – a
Maturity Model
It’s been really interesting to watch the dramatic uptick in activity around the automation
space the last year or two. I don’t need to go into too much detail on the benefits that
automation offers here, consistency and scalability are two of the more prominent that
come to mind. What has struck me though is that it feels like the way that companies are
going about it is missing a key step.
In the Enterprise we are used to hearing about maturity models. The Capability Maturity
Model (CMM) for processes (originally software processes) being one of the most
common. One of the key things you learn with maturity models though is that when
progressing along them it is critical that you don’t skip levels. That is the point of it being
a “maturity” model. You grow through each stage, building on what you already have.
You can move quickly through a stage but if you try to skip one you are destined for
failure.
When looking at an IT automation implementation at a high level you can argue that it is
simply a case of moving from manual deployment of systems to automated
deployments. For a fuller picture though I believe that you need to take validation, or
testing, into account. If I were to illustrate the maturity model, or roadmap if you will, for
IT automation I would do it like this:

2. So why automate testing before automating your deployment? Think about it this way.
What do you need to do before you start automating your systems? You need to gather
your requirements for automation. Now you can gather them the classic way in a
document of some sort but how much utility do you get from collecting them this way?
Take a leaf from the book of software developers and test driven development and
gather your requirements as tests instead. Why? There are multiple benefits:
1. If your requirements are executable you have test driven automation and you can
track your progress with your automation implementation. How are we doing? Well how
many tests are passing?
2. Your tests live on and become the safety net for your automations ongoing.
Remember, automations are code too and they will have bugs. If you support them with
separate tests then you can guarantee their quality.
3. You reduce vendor lock in for your automations. It is a much less risky task to switch
from Puppet to Chef, for example, if you have an independent set of configuration tests
to validate your migration.
So, not only is automated configuration testing a natural step on the road to full
automation, it has multiple additional benefits.
How you automate your configuration testing is up to you. If you are comfortable with a
particular scripting solution or testing framework then that is fine although portability and
collaboration can suffer. Remember, configuration is something that multiple teams have

3. a stake in, from development, operations, security and even testers themselves. If you
are serious about bulletproofing your automations through testing you may want to
consider a more fit for purpose configuration testing solution like..um… oh yeah,
ScriptRock
However you implement though the mere exercise of doing so prepares you for
automation. Whilst tempting to ignore it is always time well spent.