Continuous delivery shouldn’t be a controversial subject in the world of Agile software development – the third principle behind the Agile manifesto all but requires it: “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.” Yet as with true iteration, enterprises embracing Agile software development often balk at the idea.
Disclaimer: I only address the technical, delivery automation aspect of continuous delivery below. Your enterprise’s tech funding model, delivery and operations structure, and overall culture are also hugely important in realising the benefits of continuous delivery, and I plan to come back to them in future blogs.
One common roadblock is people conflating continuous delivery with continuous deployment, and imagining a Facebook style scenario where software developers can push changes into production with little or no supervision, QA, etc. Naturally, enterprises (particularly those in highly regulated industries) aren’t having a bar of that, and that resistance translates into resistance to continuous delivery.
Beyond that though, there’s a deeper uneasiness about automation, perhaps partly from a job-security perspective, but also because making deployments quicker, easier and more frequent with less hands-on involvement just feels like it must make production incidents more likely, when in-fact the reverse is true.
Where continuous deployment says “this code passed the automated test suite, time to push it to prod”, continuous delivery says “this code passed the automated test suite; if you like, and if it passes whatever manual tests and reviews you want to throw at it, it’s available for deployment to prod”. Pushing a deployment from one environment to the next should be entirely automated; the decision to push from one environment to the next can be entirely manual.
On the flip-side, the benefits of frequent, automated deployments are enormous. Smaller changesets mean fewer places for bugs to hide. Plus there are no more several-hundred-line deployment checklists, full of potential for human error and miscommunication: the deployment script is the checklist, and it either works, or it fails the first time you try it in the test environment. This also reduces your key-person risk/delay – deployments can’t be derailed because so-and-so from networks isn’t at their desk ready to switch over the router config.
When deployments are happening every couple of weeks, any manual actions that are required are always fresh in the minds of the people responsible for them. When deployments are months or years apart, even if you’re lucky enough that the same people are still around in the same roles, it’s been so long between performing those particular actions that they might-as well be doing them for the first time. And with continuous delivery, if there is a problem with the build that’s slipped through QA, your rollback is scripted too – no more getting to 4AM on deployment night, realising that there’s a problem, and having to launch into several hours of rollback tasks to ensure the system is back up and running for the start of business.
As I have discussed Agile needs iteration, and that means getting releases out in front of users, or at-least user-like SMEs, as often as possible. In order to achieve that you need to automate your pipeline all the way from dev into your QA environments at the very least. And if you care to ensure that the thing you’re putting into production is the same thing that you tested, you need to extend that automation to your production environments. Otherwise a single typo on install-night can introduce anything from complete failure to subtle bugs that take months to uncover – rendering all your testing in non-production environments completely pointless.
The last point above bears repeating and expanding: if you use a different deployment method (manual vs. automated) or a different deployable (e.g. with compiled-in debug/configuration options) in production compared to what you use in non-production, then what you’re putting into production hasn’t been tested. If you want to split hairs then you might say that you can never truly replicate your production environment in non-production – but it’s worth trying to get as close as possible.
So, how do you encourage your organisation to adopt continuous delivery? Obviously the messaging around your advice is critical – from the outset you need to allay fears that you’re proposing continuous deployment. You’re not trying to somehow automate your way around QA processes, but to automate all the tedious, error-prone manual labour around deployments that has the potential to render your QA processes ineffective. The real value that automation delivers is:
- Reduction of deployment failures/misconfigurations
- Reduction of key person risk
- Reduction of deployment time
- Reduction of rollback time
- Increased auditability and debugability of deployments
- Enabling more effective Agile iteration through increased deployment frequency
It’s this value that will sell the idea of continuous delivery to the holdouts in your enterprise, whereas visions of Facebook-like developer-driven releases will mostly scare them off. You may still run into job-security fears, but at this point I think most people understand: the machines are coming for everyone’s jobs, like it or not, so you’d better learn to be automators, or face having your job automated out from under you. That’s not the most diplomatic phrasing, but it’s true nonetheless.