The software analyst industry needs a code of practice.

This article has been podcast

In my country, the broadcast industry and the advertising industry both adhere to a voluntary code of practice to police the more extreme behaviours of their members. I wish the software vendor industry and their parasitic analysts would do the same.

There are a thousand examples of where it was needed. The one that has me wound up today is this one: The Forrester Wave: Application Mapping For The CMDB Q1 2006.

Nineteen pages of the benefits of application-to-infrastructure-mapping-tools (“better understanding of how applications are deployed in production … better control of infrastructure and application changes … possibility of controlling spiraling [sic] application costs … better way to consolidate infrastructure … better planning of backup sites” … heal warts … make her love you … reconcile East and West ….) and comparison of eight tools, and NOT ONE WORD about whether the tools are actually useful or what the limitations of the concept are.

Forrester’s research into the effectiveness of this mapping in general or the tools in particular consisted of “customer success”, which appears to have been measured via references.

Warning: Tangential Rant

Let’s talk about references, shall we. Any vendor who cannot rustle up one positive reference is so inept they are out of business anyway. References can be obtained by four main mechanisms:

1. Love them. This requires far more vendor resource than could ever be sustained in more than 2 or 3 clients, but no-one seems to notice this, least of all analysts.
2. Appeal to ego. Make them look a hero. Put their face on full page ads in magazines where their peers will see. Give them a poster-size framed version of the ad for their office wall.
3. Bribe them. I am sure research on the number of reference sites whose CIO went to the world conference at the vendor’s expense as a speaker or regularly appears on speaking tours to warm sunny countries would yield interesting results. This works particularly well with a CIO about two years from retirement: apply #1 and #2 above so they look like a hero, then once they retire employ them on contract to be an overseas superstar keynote speaker at conferences in exotic places.
4. In the face of defeat, declare victory. This was an old British military tactic when faced with unshakeable guerilla insurgence: walk away and hold a victory parade. What CIO will admit the half-million-dollar project is a failure when they can bluff their way out of it. Tell everyone how successful it was for long enough and even your own staff might start to believe it, especially if they start getting invited to conferences in exotic….

For the part of the actual research that looked at functionality “Forrester looked at the product architecture for its real-time capabilities in building maps and detecting changes. We considered key issues such as time to collect data, the need for manual intervention, the depth of data collected, and the security and maintenance of the resulting CMDB.” How about the usefulness of the data for managing a business service?

And they did this by (a) asking the vendors and (b) asking “three companies that had conducted independent evaluations of the vendors’ products”. There is no info on whether these three companies had actually installed the products and to what level they tested the practicality of the results, but if I were a betting man… Anyway there is no evidence that Forrester actually saw any of these products in action, let alone installed and tested them themselves.

For the uncritical reader - and as you will all know most IT people are far too busy to be critical readers - a 19-page analyst white paper with pretty graphs and lists and tables looks very authorative. Most will thumb through to the chart that shows nLayers at the front and CA at the back (to save you all downloading it), and just accept the premise that this must be a good idea else these people would not be selling tools and analyses to help select those tools.

Now I’ve never seen one of these tools successfully deployed (who has?) but I’ve worked at selling one and talked to the geeks who made it work. I suspect these tools are DUMB. They all detect some concept of “same” to link CIs together and they are easily misled. So if you are ever evaluating one of these things, test these:
Have two different (apps that provide) business services

• use the same email server to send automated emails
• share a common code routine
• share a common database table
• share a common app server and/or web server

…and see if it can tell them apart.

Then see how it handles:

• A load-balanced web-server or Citrix-server farm
• any EAI middleware
• Citrix, VMWare or any other virtualisation technology
• Web Services dynamically accessed from a UDDI

The IT Swami predicts: while some automation of CMDB service-to-infrastructure-mapping may result, it will be at an expense in money and resources wildly beyond the benefit derived, especially once you factor in the cost of constantly checking and fixing its dumb misunderstandings.

But you will look in vain in the Forrester paper for any discussion of how far the technology has progressed, how far it has to go (the gap), what industry pundits (e.g. itSMF) have to say about the current usefulness, or - heaven forbid - whether any of these tools actually WORK.

Where do these analysts get off, creating markets and then feeding off them? Analysts should call themselves what they are, marketing outsourcers, instead of dressing themselves up in a cloak of respectability by pretending to make impartial assessments in the best interests of their readers. At least the vendors are overt about it.

The monthly newsletter of the IT Skeptic: all the news on ITIL, CMDB and other IT topics that is fit to print... and then some!
Subscribe here. Privacy respected. No registration required.