Continuous Delivery… Incrementally

There is a lot of buzz amonst the development community at large about Continuous Delivery.  Everyone seems to think its a good idea but I’ve seen few execute implementation well.  In this post, we will briefly review what CD is and then I’ll give my opinions on the best way to implement it (HINT: its in the title). 

What is Continuous Delivery?

At the highest level, CD is about speeding up the feedback loops across the software development life cycle. For example,

• Between development and quality assurance
• Between requirements and development
• From technial analysis to user acceptance testing
• From idea conception to delivery to the end user

Between each stage or across multiple stages, the sooner you know there is a problem the more easy and efficient it will be to fix it.  I think this diagram from Wikipedia is a wonderful way to show this visually.  Each change has the potential to be released to end users but if any validation fails along the pipeline it is rejected. A new change must be made to fix that failed validation in hopes that it will make it farther down the pipeline than the last change.

We won’t dig into each of these but there are 9 principles of CD that you should be aware of. 

• Everyone is responsible
• Bring pain forward
• Version control everything
• Automate almost everything
• Repeatable and reliable process
• Build quality in
• Done means released
• Continuous improvement

Each principle supports speeding up feedback loops in one way or another.

How should you approach Continuous Delivery?

Ask yourself the following questions:

What would prevent you from delivering a code change safely to production right now? 

How quickly could you deliver safely? 

What processes do you have that would slow down that delivery? 

These are the questions we ask on the road to Continuous Delivery.  Obviously you wouldn’t want to deploy to production without some level of testing/safety but how long would that take?  Within my development group, depending on the product, we routinely deploy changes within 10 hours or less.  That may sound like a long time (we still think it is too) but that includes a full automated regression test suite run based on all of our test cases.  This means we could deploy new changes to production 5-6 per week if necessary. 

Four years ago we were deploying at most once a month more likely once every three months.  So how did we get to this speed of delivery?

The Incremental Process

After being on the Continous Delivery journey for 4 years, my best advice is to make incremental improvements to your process.  I’ve seen too many organzations start the the journey with tool selection or grandeose process plans where they try to solve all their problems before making any improvements.  This is a bad idea.  Its VERY difficult to think through all the use cases/requirements up front and get it all right.  You see no value along the way as you wait for the “solution” to work or be available.  It turns into one huge project for the entire company/division.

Don’t be waterfall be agile.

It didn’t happen overnight for us but we took small steps and saw value from most of them as we went.  To get started we simply asked the following questions:

 What is the most painful part deploying your system/applications?

What is the most error-prone step in your deployment? 

What is the slowest step in your deployment? 

We thought through how we might improve these issues through small experiments and prioritized them by which would solve the biggest pain or bring the most value quickly.  Then we started implementing them one at a time to improve our process. Some worked exceedingly well, others made no change but these smaller steps were moving us ahead bit by bit.  Here is a list of some of the improvements we made (in roughly the order we made them) along the way that ended up improving the process.

• Automated the deployment of web applications using WebDeploy
• Created automated feature tests for the longest manual tests
• Changed warnings to errors in our build process
• Unit tested our  most complex code
• Created PowerShell scripts to automate server setup
• Automated Windows Service deployments
• Implemented Kanban for work item tracking
• Setup gated check-ins that ran all our unit tests
• Weekly review of deployments to find manual steps that can be automated
• Automated schedule task deployments
• Automated SQL database deployments
• Changed DEV definition of done to require unit tests for every change
• Created  a private Nuget repo to manage internal dependencies
• Changed DEV definition of done to require automated feature tests for every change
• Moved to a single branch and implemented Release Readiness
• Implemented a standard service versioning strategy
• Changed to building once and deploying many times
• Implemented delivery pipelines via Thoughtworks GO!
• Integrated unit test coverage guard (can’t decrease) into build process
• Created automated feature test reports to better track issues during development
• Created performance test suites to identify performance concerns proactively
• Created PowerShell scripts to automate local developer machine setup
• Made the first story of any new application a full deployment pipeline through all enviroments
• Setup regular lean operations reviews to discuss improvement opportunities

Each of these changes were driven by one or more of the CD principles.  Implementation time varied for each but the average was likely around 2 weeks and the longest was no more than 8.  Taking an incremental approach yielded a steady stream of improvements over the last 4 years.  There has been some wasted effort but overall this agile approach has been far more beneficial and efficient than trying to plan all of this work upfront. I have zero confidence that we would have made it as far as we have with a waterfall-like approach.

How do I get started?

Ask yourself the questions above and find one experiment/change you can make TODAY, make the change and evaluate is affect.  Rinse and repeat! 

If you aren’t sure Continuous Delivery makes sense or will work for you, I challenge you to dig deeper and learn more about it..  Change is uncomfortable for most people.  The changes listed above were all scary  for us but looking back I can honestly say our whole team is benefiting from this journey and is healthier/happier.

What do you think?  Leave a comment or tweet your thoughts to continue this discussion.