The Emperor has no clothes. Where is the evidence for ITIL?

This article has been podcast

Since this is a skeptical blog, it is high time we examined the evidence. Where is the evidence for the benefits of ITIL? There isn’t any. Not the kind of hard empirical evidence that would stand up in, say, clinical trials. There is more evidence for quack alternative medicines than there is for ITIL. There is certainly more solid evidence for the application of CMMI (or CMM in solutions development see this presentation by SEI), an analogous methodology in a closely related area.

Granted there is some research around the benefits of aligning IT with the business but not around quantification of ROI and nothing (that I can find) specific to ITIL.

The itSMF themselves make a few outrageously unsubstantiated claims in An Introductory Overview of ITIL [version 2]

• Over 70% reduction in service downtime
• ROI up by over 1000%
• Savings of £100 million per annum
• New product cycles reduced by 50%.

Send no money now!!

The Best Practice Research Unit is associated with the ITIL 3 refresh [Updated: It was. The site has disappeared]. After 20 years it is about time there was such a unit. It is just a shame this is not an initiative of either OGC or itSMF (at least itSMF USA is doing something, in fact several things focused on research).
The BPRU website explicitly recognised this problem.

Much of the material published on IT management, including IT service management, has been normative or prescriptive in flavour. Few rigorous, academic studies have been undertaken to evaluate how tools, techniques, methods and management approaches have been selected, adapted, implemented and measurable benefits achieved.
There is a danger that new approaches arise out of the practitioner community with little empirical validation.

“Few rigorous, academic studies”? Don’t be nice, Tony. The solitary piece of academic research I can find carries a bold and, I think, unproven title “Evidence that use of the ITIL framework is effective” It is from Dr B.C. Potgieter, Waikato Institute of Technology (New Zealand), J.H. Botha, Oxford Brookes University, Dr C. Lew, Damelin International College.
It opens by saying “Very little academic material exists on ICT Service Management Best Practice…” and concludes its own research with:

We found that both customer satisfaction and operational performance improve as the activities in the ITIL framework increases. Increased use of the ITIL framework is therefore likely to result in improvements to customer satisfaction and operational performance. Although the study was limited to a single research site, claims made by executive management of the research site and OCG as to the contribution the ITIL framework seems to be justified. More definitive research delineating the nature of these “relationships” is however needed, especially regarding each process in the ITIL framework.

The data base is poor: “research site was a large service unit of ICT in a provincial government in South Africa during 2002/3.” One site. More importantly, the two things measured to support this brave conclusion were (1) customer satisfaction (the three surveys they conducted only included management in the final survey so all we can say is that non-managerial staff were happier) and (2) “objective service improvement” by measuring “the number of calls logged at the Help Desk” because “we can rather safely conclude that the number of problems logged would be a good reflection of objective service levels”. I expect that last statement leaves this research with zero credibility with anyone who understands ITIL and ITSM. No cost/benefit analysis. Not a single valid objective metric. Sure if you throw enough government money at anything and launch an aggressive enough PR campaign you can make the users happier. That proves nothing. And the fact that calls to the Service Desk went down over an initial nine month period would to me be a cause for concern not celebration. But you can bet this paper will be quoted all over the place as evidence of the effectiveness of ITIL.

Pink Elephant have finally extended the number of anecdotal stories beyond the tired old Proctor and Gamble, Caterpillar and the internationally famous Ontario Justice Enterprise. They now have a few more arbitrary statistics . I’ve been in vendorland and I have generated this kind of case study. These amount to no more than selective quotes from managers justifying their decision after the fact.

HP is one vendor putting numbers where their hype is, though this pertains to Service Desk product not ITIL:

IDC found that, for the companies surveyed, IT productivity increased by an average of 14% … contributing an average cost savings over three years of almost $4.2 million annually. When normalized for company size, these savings amounted to $17,235 per 100 users … Based on these
savings, the three-year hard ROI … averaged 130%, giving an average payback period of 13.5 months.

University researchers! are you listening? Those are results. Unfortunately, paid analysts doing surveys for a vendor are about the least useful sort of research. It is not subject to the same transparency, peer review or statistical rigour as academic research, and their results tend to be selectively quoted by the vendor.

But since we’ve started, here is another "sponsored survey":
“Did you make a business case before decision? (Base: 62 European firms): No 68%” TWO THIRDS had no business case!!!
And “Don’t know 11%”. Who on earth was answering the survey that didn’t know if there was a business case?
But wait!! There’s more! “Did you observe the expected ROI? (Base: 20 European firms). No 50% Don’t know 30% Yes 20%.” Good lord! If less than a third built a business case, one would guess the ones that did represented a sample biased towards those who had a good case, and yet only ONE FIFTH of them achieved the expected ROI. This is best business practice? I think I need to go lie down.

Before I do though, here is one more. If you are willing to make major decisions based on amateur research by vendors (as everyone adopting ITIL does), here is an interesting one to ponder:

In a survey carried out by Bruton of 400 sites, about half of the 125 organizations which were found to have adopted ITIL made no measured improvement in terms of their service performance or the rate at which they were able to close helpdesk calls. “Some helpdesks can way outperform a site that has adopted the best practices of ITIL," said Bruton. "Best practice does not mean superior performance. It is beginning to sound that ITIL is the only way to go. It isn’t. It is only one way to go.”

A man after my own heart.

I would like to see some solid scientific research on:

• Quantified cost/benefit analyses across a statistically significant number and diversity of organisations of adoption of ITL vs. other BPR methodologies, or vs. a simple process review and reorganisation, or vs. implementation of a service desk product.
• Quantified cost benefit analyses of organisations that have only done ITIL without concurrent Six Sigma or CMMI or other quality improvement programs.
• Analysis of the proportion of organisations that would actually benefit through adoption of ITIL.

The really delicious irony in all this is ITIL’s own emphasis on the importance of a business case and ROI. But the facts are that few organisations even bother to examine the business case before embarking on ITIL; even fewer measure results; and the few that do are building their business case in the absence of any solid research to justify their estimates. The Emperor has no clothes.

Browse this site for more skeptical views on ITIL.