Continuous Delivery: Should I start with test or Deployment Automation?

Someone recently asked whether teams who wish to start the Continuos Delivery journey should focus on deployment automation or test automation first.  This is a tough question and I’ve struggled with it over the last few weeks but I have finally decided on what I believe is the best approach.  BOTH!

To better understand my suggestion, lets first consider each approach individually.

Deployment Automation Only

If you fully automated deployment of your software, deployments could be quick and painless. Our deployments started as a high stress, mostly manual procedure with many steps/actors and spanned multiple hours/days.  Today its a single button click that takes about 15 minutes or less.  Implementing this automation was no small task.  It took us around 12 months to fully automate our application deployments. 

Once you achieve automated deployments, how often would you use it if your regression tests aren’t automated?   I would venture to guess manual regression testing times for the average software development team is between 2 and 4 weeks.  Regardless of the actual average we know excution of manual regression tests is 1 or 2 orders of magnitude slower than automated execution of regression tests.  So the point is you won’t need to deploy very often when you are still performing regression testing manually.  Therefore you won’t get as much value out of automating your deployments if your regression tests aren’t also automated.

Test Automation Only

On the other hand if you focused on fully automating your regression tests, you could run your tests in minutes/hours instead of days/weeks/months.  Again this is a big and complex effort.

• Developers may need to be taught how to test. 
• Testers may need to be taught how to develop automated tests.
• UI automation is notoriously tricky and typically requires lots of maintenance/refactoring over time.

Today we have around 95% of our regression tests automated which takes around 2-12 hours to complete depending on the team/product.  Each week we look for and attempt to resolve false failures and improve our test results.  Its a lot of work but now we can run our full regression test suite multiple times per day if needed which brings a ton of value.

Once you achieve this level of automated testing, how often would you want to deploy?  We deploy multiple times per week.  Would you want to do the same if your deployment was still a manual, high-stress, painful and costly process.  Probably not.

Implement Both Incrementally

The original question implies that teams may try to fully implement one before the other.  In our journey, we started implementing both in parallel.  I don’t remember any specific reason why we took that approach that’s just what we did.  Looking back I’m convienced working on both was the key to our success. 

The other key aspect is we took an incremental approach to these improvements in our process.  Each time looking for the next most painful aspect and defining small experiments/changes.  The goal was to implement and test each change quickly with hopes that each improved the process.  If unsuccessful, we would roll back and try a new experiment.

As the diagram above highlights, trying to fully implement either deployment or test automation first is not an ideal approach.  The question is what are the small changes we can make to either test or deployment automation that will improve the process over time.

Obviously, each team’s situation is different and these thoughts may not apply.  In my experience working with many teams (both inside EBSCO and out), these ideas seem to hold true.  I’m curious what you think.  If you have feedback, please leave a comment below, on Twitter or start a conversation on Reddit.