DevOps and traditional ITSM - why DevOps won't change the world any time soon

In which we discuss DevOps: the threat it presents to successful IT service delivery, and why I think DevOps will be a niche novelty for a while yet.

I've presented a couple of conference keynotes lately on the topic of how IT is "losing control of the centre" to IT cowboys backlashing against ITIL; to Agilistas and their cohort DevOps; and to Cloud hypesters. Blog post will follow one of these days. For now I'd like to look at DevOps.

[Update 20/1/2015: I did soften my position on DevOps, to the point of being a bit of a fan now. But actually, most of these challenges are still valid, in a slightly less confrontational form :) ]

What is DevOps?

That's a good question. Like any idealistic movement, that depends who you ask.

It seems sometimes to be about multi-skilled teams doing all the development, deployment and operation of their product.

Other times it is about the "sysops" staying separate but being nice to the developers.

To others it is about cool tools automating everyone but the developers out of a job.

Or DevOps is an attitude, a feeling, man.

What I like about DevOps is that it is a cultural movement, an attempt to change IT culture. To the extent that it attacks the misuse of ITSM principles by inept managers and petty empire-builders, I'm all for it. I like the way it tries to improve communications/relations between development/solutions/projects and delivery/operations/production.

I also like the way it tries to speed up delivery of new services and changes to services. IT systems and organisations can become sclerotic, like any other. A flush out of bureaucracy is a good thing. Challenging assumptions is good. All processes can be improved.

ITSM is predicated on improvement. So ITSM and DevOps are in agreement on these things. In other areas, not so much.

Why is DevOps a worry?

One of my concerns is that DevOps - and Agile - get seized on by IT cowboys as a blunt instrument to break down the controls around service delivery/production for political reasons that don't truly improve the situation. Instead of "turn us and them into just one us" it becomes "eliminate them so it's just us". I feel like the subtext of much discussion is "If developer-techs ruled the IT world, everything would be OK".

Another concern is the apparent lack of understanding of what IT Operations does by most (but not all) proponents of DevOps. Their conception seems to be primitive and simplistic. The way they keep calling us "sysops" or "sysadmins" is a symptom of this. The call for NoOps is a classic example. All Ops does is run up a server and database instance when needed, apparently. Oh, and some bureaucratic crap around that.

My main concern is that even the best-intentioned DevOps supporters have this peace-and-love concept that we can unite development and operations into one happy team. If this was really true I'd be right behind the DevOps movement. But it isn't. This flower-power will be no more successful than the hippies were in uniting mankind.

That's because DevOps is based on certain assumptions which don't scale. Agile and DevOps may one day succeed and become the generally accepted mode of running IT, but there is an awful lot of learning and failing has to go on before either one has the maturity to be handed the keys. They work with small impassioned teams working on small standalone projects, or pilots, or SMBs, or startups, or funky open source tools... anything standalone. Great. That doesn't mean they are ready for prime-time.

This post by Stephen Nelson-Smith (guest-posting on Patrick Debois's blog) is considered seminal in DevOps. It holds up IT's poor track record in project delivery as a problem, claims that silo-isation is the cause with no evidence for that claim, then promotes DevOps as the solution without noticing the following un-scalable assumptions:

Assumption: We can all be one happy team instead of in silos.

This works up to Dunbar's Number, so it works just great in the smaller teams of WebDev, and pilot trials of DevOps. It ain't gonna work in anything larger.

People will split into tribes and there isn't a damn thing you can do about it other than manage it. One good way to manage it is to build formal processes so everyone knows and agrees who does what.

Not only are silos inevitable in any organisation with more than say 100 IT staff, but silos are not a bad thing when managed well. Two of the key benefits of large organisations (and society) are specialisation and economies of scale. We centralise our BAs, operators, support, DBAs, coders, change control, account managers, librarians, testers, sysprogs, PMO, webmasters, architects and so on for very good reasons:

• we use the most expert people on the organisation (who are invariably in short supply).
• we set and follow standards to improve compatibility, consistency and efficiency; to reduce rework, duplication and redundancy of technology, skills, and effort.
• we make the organisation as a whole more efficient and effective. What so many developers seem unable to grasp is that it isn't just about their baby. One project may need to wait at times for access to centralised resource, but the overall effect is that they'll get the best result with minimum effort; they'll make maximum re-use of existing resources; and they'll minimise the introduction of additional overheads for new things to operate and support.

Instead DevOps wants to take us back to an era of self-sufficiency where developers live in communes, providing for all their needs locally. It didn't work when the hippies needed a tractor and it won't work when DevOps needs someone to support their live database.

Besides, if you organise your systems as self-sufficient teams doing everything from business requirements definition to infrastructure problem resolution, you've just moved to vertical silos instead of horizontal silos. You'll still get us-and-them, you'll still get animosity, 'cept it won't be between Development and Operations, it'll be between services. Ooh that'll sure improve the organisation's view of IT, not.

And if DevOps is not about building vertical silos, if it is just about improving communications between Dev and Ops so we can get on better, then we all agree this is a Good Thing, so can we all have a group hug and get back to work without this hype about revolutionary new approaches please?

Assumption: Good people are easy to find.

We sent my son to a Montessori pre-school. I quite like the Montessori approach, but more than that I like the fact that such an idealistic philosophy attracts the most passionate and professional teachers. You can't replicate Montessori success across the educational system because you can't get that many teachers that good.

In the same way, Agile and DevOps philosophies are predicated on an unrealistically rosy view of the standard of people available. Because they are exciting revolutionary new approaches, they attract passionate and professional people who make them work at exceptional levels with a minimum of oversight or controls. This simply will not scale. For every Dilbert there is a Wally. You cannot use the "empowered professionals" model on a large scale across the industry. You need accountability, formal controls, and performance management.

What's more, Agile and DevOps (and Lean) prefer small teams of multi-skilled individuals. I've already blogged about how that is also going to fall on its face the first time anyone tries to scale it across a large organisation. Stephen Nelson-Smith talks about

...experienced, talented 30-something sysadmin coders with a clear understanding that writing software is about making money and shipping product...
There’s been a certain amount of popularity recently around the concept of ‘polyglot programmers’. These are developers who know multiple languages, and multiple approaches to programming, such that when appropriate they can move swiftly between object-oriented Ruby and a functional language such as Erlang or OCaml. This is simply an example of a larger observation - there is no one IT skill that is more useful or more powerful than another. To solve problems well you need all the skills. When you build teams around people who can be developers, testers, and sysadmins, you build remarkable teams.

Remarkable indeed. Let's have a hundred million of these people then, to run the world's IT systems. More selfishly where can I recruit 200 of them locally to run just one of my client's shops?

Assumption: Tools can fix any problem.

Face it, DevOps people are tools people. They love technology. Most have an inherent assumption that any problem can be automated away, including cultural ones.

Even when automation prevents a problem, the secondary question that gets overlooked is whether that automation
(a) is worth the prize - whether the value exceeds the cost of building and maintaining it.
(b) is simply eliminating many small errors to replace them with the likelihood of one massive error one day when the automation inevitably fails.

And don't get me started with this concept that seems to be prevalent in DevOps of mixing and matching to produce a soup of languages and technologies as a solution. No considerations of risk, TCO, skills resourcing, supplier management, obsolescence...

Assumption: Process?

A lot of talk in DevOps is about freedom, giving people room, loosening up.

You'll look a long time to find any mention of

• capturing best methods and encapsulating them in the organisation in case the star performer gets another job offer
• measurement and accountability
• quality control and improvement
• managing interactions outside the team with other groups e.g suppliers

Process seems to be a dirty word in DevOps. You can get away with that in teams who go to the pub together. You can't manage large groups without it.

Assumption: We build it not buy it or outsource it.

I'm puzzled by the huge emphasis in DevOps on "coders" and "sysops". This seems to be going against the trend worldwide to off-the-shelf solutions, or - increasingly - outsourcing.

You hear DevOps talk about Cloud a lot as a driver for Devops, but their conception of Cloud seems to be lots more server instances to automate (read: play with), not SaaS. Once again, I think this is a reflection of the tiny environments in which DevOps is currently operating. Nobody builds new huge monolithic company-wide systems any more.

The more we outsource, (and buying solutions is just a form of outsourcing), the more process, controls and disciplines we need, and the tighter the risk management required.

Assumption: We can control risk in Agile software development.

This is the biggie. There is a reason why IT evolved all the controls it did. DevOps seem to think it is because Operations people are assholes. We like to think it is from bitter experience.

Agile and DevOps are risky.

Software only has a small number of correct states. Nearly all states are wrong, many of them catastrophically so. And with more interfaced systems, the number of states goes up exponentially. I will take a lot of convincing that changing these systems faster and more often with less controls does not increase the risk of introducing an incorrect state.

It is worth bearing in mind that it is not our money, nor our organisation. If the owners of your enterprise are telling you to take more risks in IT, then you should consider Devops. I bet for 90% of you they are not. Developers may think they are risk averse but I'd love to see some research: I bet the majority would have a higher risk appetite than Operations staff, and a higher risk appetite than their employers.

Much of the weeping and wailing of DevOps is because of the "unfair" constraints imposed by those who are more risk averse. People have different ideas of "good". And a lot of IT people have a different idea of good than say the Board of Directors. The governors - the Board - define good. Others must comply. They may not want to.

I don't disagree that "High performing people work better when they have freedom". In IT generally, dev and ops, I'd bet there are hundreds of millions of practitioners worldwide. They range from "high performing people" to those you wouldn't trust to organise lunch. So I think Agilistas are naive and optimistic. Neither is a good trait when you are tasked with protecting the organisation and managing risk, which I contend are some of the primary functions of IT Operations. And as Cary King said on this blog , that only becomes more important as more is externalised to third parties. I don't think these Agile ideas of looser control are enlightened, I think they are dangerous.

The future of DevOps (and Agile)

More and more large companies are going to perform the great experiment: base all their IT on Agile and/or DevOps principles. I'm betting it will end in tears: more projects will get delivered more quickly ... but sooner or later availability, continuity and data integrity will go down the toilet.

As I said above, there is an awful lot of learning and failing has to go on before either has the maturity to be handed the keys. Right now, Agilistas look like teenage hippies, propounding simplistic solutions to complex problems, based on naive and idealistic assumptions. Similarly it will end in tears. They'll crash the car, get pregnant, and come home for money. And long-suffering IT Operations will have to clean up the mess.

As I also said above, it is possible that these ideas represent the future. Every generation thinks it is smarter than its parents and every few centuries it is true. If this is one of those times, we still need to get these ideas into clean clothes, a bath, a job and a haircut. They aren't ready to run the world.

How to deal with DevOps (and Agile)

In the meantime they aren't going away. The hippy-dippy Agilistas will continue to sit on their Swiss-balls and espouse the wondrous benefits of agility. My view: ring-fence them. Build a safe fenced-off portion of Production. Let them have control. Let the SLA state that the DevOps team is responsible for operations and support, and accountable for availability, continuity and integrity. This includes running daily tasks (reorgs etc) and running a service desk, because I can't see how operational support staff can be expected to work with a system that changes daily or weekly, unless it is the only system they work with. Make sure the customer understands (and signs off) that the increased agility comes at a risk that is outside Operations' bounds of responsibility and accountability, and that taking it on would place the rest of the organisation's IT assets at risk.

These sorts of New Age ideas tend to spring from Marketing and HR and similar business units whose systems are thankfully often fairly self-contained.

What if they want to integrate with existing production systems and modify the data? Tell them to fuck off. They can pilot in their agile sandpit, but anything that impacts existing Production comes through existing Production change control. We want to see:

• Documentation that will still be useful after all the passionate and professional team have moved on to new and exciting opportunities to demonstrate their agile genius for fast results
• Sociability testing with other systems
• Training and tools for the operations staff and support staff who will be left to care for the system after the passionate and professional etc...
• An operational readiness checklist as evidence that this system is past playtime and ready for grown-up industrial strength IT.

What if management decide that all IT is to be Agile, and modeled on DevOps? Leave. Or hunker down and wait for the inevitable.

I leave you with one great anonymous comment I saw:

[Naive] people in the industry constantly think they are inventing a solution to the problem when they are only the continuation of the problem. Software is expensive and risky and needs to be managed properly. A good software developer will understand the bigger picture, the stake holders and be able to communicate with the stake holders. A good software developer will be able to say they aren't an expert and that one will need to be consulted.

Next up: DevOps and ITIL