Type “devops” into any job search site today and the overwhelming majority of results will be for some variation of “DevOps Engineer”. The skills required will centre on tools like Puppet/Chef/Ansible, AWS/Azure, scripting in Python/Perl/Bash/PowerShell etc. Essentially, they’ve taken a deployment automation engineer role, crossed out “deployment automation” and written “DevOps” in its place.
There’s nothing wrong with hiring deployment automation (or, if you must, DevOps) engineers if you don’t have enough people with the right skills to deliver the deployment automation part of your DevOps strategy. The real problem is when hiring DevOps engineers is your DevOps strategy.
Deployment automation is an ancient art compared to DevOps. How ancient? Here’s an abbreviated history (feel free to skip to the tl;dr if you’re not a history buff):
- 1969: Telnet defined in RFC 15
- 1973: UNIX unveiled to the public – rewritten from the original assembly into C, becoming the first portable OS
- 1990: CVS, the first major client-server revision control system released
- 1991: Continuous integration mentioned in “Object Oriented Design: With Applications”
- 1991: First Linux kernel released
- 1993: “Cfengine V2.0 : A network configuration tool” published
- 1995: SSH released
- 1999: “Extreme Programming Explained” advocates continuous integration running several times a day
- 2001: CruiseControl released
- 2005: Puppet released
- 2009: Chef released
- 2009: First Devopsdays conference
- 2010: “Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation” published
- 2011: SaltStack released
- 2012: Ansible released
tl;dr: Before 1973 pretty much every model of computer ran it’s own bespoke OS and centralised, automated administration was inconceivable. Just 20 years later (24 years ago), UNIX was everywhere and we had off-the-shelf tools to automate deployment to remote machines. “DevOps” wouldn’t be coined for another 16 years.
So what is DevOps, beyond the (at-least) 24 year old ops practice of deployment automation. Wikipedia has a decent summary:
“a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes”
Aside from broadening the scope beyond deployment to end-to-end delivery automation, this definition emphasizes that DevOps is about practices that improve collaboration between your dev and ops teams – not so much about dropping another set of engineers in-between them (Anti-Type B or E in the excellent DevOps Topologies article).
Further down, the Wikipedia article gets to what I regard as the real meat of the subject:
“DevOps is more than just a tool or a process change; it inherently requires an organizational culture shift. This cultural change is especially difficult, because of the conflicting nature of departmental roles:
Operations — seeks organizational stability;
Developers — seek change;
Testers — seek risk reduction.
Getting these groups to work cohesively is a critical challenge in enterprise DevOps adoption.”
I take it a step further, and regard this as the critical challenge in enterprise DevOps adoption. The crux of the problem is that in many (I daresay most) enterprises conflict is written into the KPIs of more-or-less siloed dev and ops organisations (I may come back to testers in a future article, but for now I’m limiting my focus to dev and ops). Here’s the basic scenario:
The executive in charge of the dev organisation has a hefty bonus tied to delivering changes, and takes little or no responsibility for production downtime (“I can’t help it if those idiots in ops mismanage our production environments”). Meanwhile the executive in charge of ops has a bonus tied to stability and takes little or no responsibility for enabling change (“If I could trust those morons in dev to deliver a stable product and not break things with every change, we wouldn’t need so many barriers to entry”). This is not the kind of environment where trust and collaboration thrive.
Far from being a straw-man, the above scenario was related to me by a friend last month, based on his current, firsthand experience at a large Australian enterprise where the first common manager of the dev and ops teams is the CEO. While I’m not prescribing that you restructure your dev and ops teams into a unified organisational structure, this scenario illustrates the kind of buy-in you need to succeed with DevOps in enterprise, and why there might instead be significant resistance.
There’s a notion that this scenario will drive a kind of competitive tension that will ensure each silo holds the other to a high standard and drive them to do their best. Even if that were true for waterfall-style methodologies where a dev team might only release every 18 months, it is absolutely toxic to an Agile environment where releases need to be frequent and frictionless. If all this talk of trust and collaboration sounds a bit too warm and fuzzy to you, consider that two, perhaps three, of the four values in the Agile manifesto are about working closely with other people. If you’re going to have any hope of making Agile work in your organisation, you need to nail these “warm and fuzzy” aspects of your process, and the same is true for DevOps.
It’s difficult to point to specific areas where the dev team will assist the ops team and vice-versa – examples that make perfect sense in one organisation will be absurd in another. In general, each team can help the other to solve problems by bringing a different perspective and set of skills coupled with a strong, shared technical background and problem-solving ability, and this close collaboration also allows each to modify their approaches to better accommodate the other.
In short, rather than inserting another team of “DevOps Engineers”, with its own exclusive set of KPIs, DevOps requires that dev and ops teams work together and share the responsibility for frequent delivery of stable software. In order to achieve this it is critical that your organisational culture (and funding model, resource management, etc.) allow for this collaboration to occur in an organic manner rather than be planned and mediated by senior management.
I guess you have read this book: https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592
Culture change is never easy, so be patient. ;)
Yep, I’m probably due to re-read it at this point.
Good read, thanks. Keep up the good fight!
In case you missed it, Steve Smith expounded on this topic recently in this great article: http://www.alwaysagileconsulting.com/articles/aim-for-operability-not-devops-as-a-cult/
Actually, I would strongly advocate for putting Operations team and Development teams under the same organizational structure. Same goes for QA and Application Security teams. And to take it a step further, co-locating long-lived cross-functional teams with a spectrum of skills and abilities is a proven method to bring positive cultural changes to organizations. It’s a move that signals how important the change is to everyone involved. I know it’s not always possible to do this, but organizations should even look at symbolic efforts to replicate having these teams in the same structure.
Thanks, I hadn’t seen that article.
I absolutely agree, I just don’t want people to come away from reading this article thinking “step 1 is restructure your enterprise at the highest levels and if you can’t do that then don’t bother with DevOps”. It’s difficult to imagine any business starting today and choosing to have such deep silos separating dev and ops (though I’m sure it still happens). Nonetheless I can appreciate that where those silos have existed for decades, it will probably take a while to break them down from the top, so a bottom-up approach of increased collaboration (enabled by support and appropriate KPIs at the top) is probably going to be the starting point.
Absolutely. That’s why something like a symbolic gesture of new seating arrangements, positioning teams closer together, combined status meetings, etc. are the types of actions that can be first steps in the right direction.