a back-of-an-envelope re-analysis of Forrester and BMC's own CMDB research
This post has been podcast.
[Updated June 2009: the paper, entitled "The Total Economic Impact of the BMC Atrium CMDB Solution" is here.]
Several readers argue well that ROI is not the right measure for a CMDB, but nevertheless management want to know what they get for their money, and BMC's own research suggests "not much".
BMC got Forrester to do an analysis of 26 BMC clients. Their research waved around these numbers:
Best case Present Value of "savings [real money] and benefits [B.S. money]" over three years $1.48M
"Risk adjusted" case $1.26M
Looks impressive huh? But wait, there's less.
The IT Skeptic had a go at them because their numbers are obtained by asking those who spent the money (duh!), are those people's future projected estimates not a measured figure (naughty), and didn't factor in TCO (very naughty), or even the cost of the software (very VERY naughty).
So let's do a rough re-analysis of the data for ourselves shall we?
Forrester's PV of costs $263k included ZERO allocation for software purchase.
If you bought Atrium... say $100k conservatively [anyone know? BMC aren't saying. "If you have to ask..."] then costs would be $363k
If your environment was not pure BMC [imagine that! but you know what? I'm betting they only sellgive Atrium to wall-to-wall BMC shops and they clearly say they only interviewed pure ones] so that you had to pay for integration/federation/reconciliation... say VERY conservatively another $100k, total costs of $463k
If you accounted for the TCO instead of just the setup (only fair if you are going to measure the payback over time)... say another $100k per annum (it is that very same Forrester that says we need a CMDB manager over and above change and config managers, not to mention hardware, monitoring, maintenance of software and database....) so TCO costs of $300k over three years, Present Value at 12% = $265k (?? check this somebody, I'm a hopeless bean-counter), gives us total costs of $728k
If you apply a discount for asking those with their career at stake how well they did ... say 30% discount (your own views on how conservative that is) then the risk adjusted PV of the "savings and benefits" is about $900k or best case $1M , against costs of $700k++.
I'd say using BMC's own numbers more reasonably, you end up with a line ball of BMC Atrium CMDB just about maybe paying for itself after two or three years in a favourable case, if you can get them to give it to you for $100k.
It also assumes you are willing to accept Forrester's "benefits" in your business case.
This is not an idle question. Forrester created a hypothetical composite organisation based on interviews with the 26 then assumed various numbers about that hyppothetical composite. Fully half the total "savings and benefits" identified by Forrester is based on the assumption that "the BMC Atrium CMDB helps the service desk get information about recent changes to CIs that will help improve first-time fix rates, reducing failed changes from 20% to 10%" leading to "saving $330,000 annually ($990,000 over three years) in reduced labor costs".
Likewise they guesstimated "reducing time and effort involved by an average of 25% per [Service Desk] call" thanks to the wondrous Atrium, "saving the organization $187,500 annually or $562,500 over the three years of this analysis".
These are postulated numbers for a made-up "sample composite" organisation that might hypothetically exist based on the optimistic forecasts of 26 hand-picked BMC clients.
I can smell the bullshit from New Zealand.
Made in New Zealand 
Comments
This isn't research
There is a fundamental problem with work initiated by a vendor to determine the effectiveness of their product. They have a big incentive not to release the findings unless they are positive and they are unlikely to spend their cash on work if the results might not be useful. I wouldn't define this as research. It's PR and marketing. And this isn't a problem only found in the IT market. Research on new drugs is riven with difficulties. Evidence is fairly convincing that big pharmaceutical companies are subject to the same influences with similar effects. For example:-
"a paper in the New England Journal of Medicine dug out a list of all trials on SSRIs [Selective serotonin reuptake inhibitors - a class of antidepressants] which had ever been registered with the Food and Drug Administration, and then went to look for those same trials in the academic literature. There were 37 studies which were assessed by the FDA as positive and, with a single exception, every one of those positive trials was written up, proudly, and published in full. But there were also 33 studies which had negative or iffy results and, of those, 22 were simply not published at all — they were buried — while 11 were written up and published in a way that portrayed them as having a positive outcome." This from Ben Goldacre's excellent blog http://www.badscience.net/?p=619.
I think that you and Mr Goldacre are tackling very similar phenomena but from very different perspectives.
IT analysts don't do research
you are right, it's not "research" as understood by the academic, scientific, or in fact general communities. IT analysts don't do research. But then IT analysts make words mean whatever they want them to mean. It's high time there were some sort of code of practice.
IT's not science
I always try to remember my customers and colleages that IT is not science (at least not in the street... it is a science in the research labs and in the universities, but not outside, in the wild) so IT analysts dont' do science, they do business.
Even, it is interesting to read papers that are 10 or 20 years old where the same analysts defend a completely opposed point of view... it is a market and the needs must be created.
This is why nobody uses (for example) standard deviation in the SLA definitions :-)
Regards
Antonio
Customer Retention
Did they factor in customer retention to their numbers? While I agree that reducing TCO is a very significant part of ITSM (with the CMDB playing its role) I think customer retention needs to be included into the analysis. The cost to acquire a client can be high, but to cost of losing one let alone recovering one (if they want to be recovered) is far greater.
drawing a long bow
You'd be drawing a long bow to attribute customer retention directly and quantifiably to the introduction of a CMDB, but I wouldn't put it past this lot