The IT Skeptic - Comments for "Should every major incident produce a problem record?"

Mature environment

aroos — Mon, 04 Jan 2010 16:22:12 +0000

Dear Visitor

It looks like you are working in a mature and well organized environment. But I would not be surprised if you did not have many end user calling you direct, usually there are desktop support and application support in between them and mainframes. What works well for you may not work for other type of support organizations. My argument is that it is difficult to set fast rules on this area that fit all cases.

Aale

Incident/Problem records

Visitor — Mon, 04 Jan 2010 14:00:52 +0000

Having worked in a data center for more than 10 years, I find it interesting that folks don't have separate findings for the incident and the problem. Incident focus is to restore service and documenting how that was done. Problem looks to why the service break occurred, associated solutions and their risks. Our best practice is to have a problem record/investigation following every major incident, documenting the potential causes, identifying the risks if no action (or cost out weighs the risk) and making decisions to move forward. As the data center is working at a 99.997% availability, that practice works well and should be encouraged. We also have problem tickets when there isn't a major incident in an effort to avoid one from occurring (proactive/reactive management). Documentation sometimes hurts but it is a necessary evil and can help moving forward and down the road.

Our processes were worked out well before we were involved in ITIL and ISO 20000 (data center has been 20K certified since 2006). It made sense to us to investigate and document for ourselves and our clients.

Both camps had it wrong?

skeptic — Mon, 28 Dec 2009 09:54:00 +0000

It would seem that in this extended and heated LinkedIn argument maybe everyone had it wrong. Crudely there were two camps: (1) When a Major Incident is resolved, the root cause and its resolution should be documented in a problem record (2) it is OK to document it all in the incident record (by implication because that is what we use all along). ITIL describes how Major Incident response should involve both Incident and Problem teams from the start. So there should be two records from the start too.

See We should create the problem record right up front in an incident

I rest my case

pjotrg — Mon, 21 Dec 2009 21:04:00 +0000

I've posted a comment on Juans Blog. He answered. Reading is an art. See for yourself.

And Let's Not Forget ...

David Ratcliffe — Fri, 18 Dec 2009 15:22:30 +0000

... her majesty the Queen - Gawd Bless 'Er!

It was her personal publishing company that brought us ITIL after all!

best of British

skeptic — Wed, 16 Dec 2009 01:03:00 +0000

...the BBC, Winston Churchill, the Goon Show, rugby soccer and cricket, Rolling Stones, miniskirts, my ancestors, the Westminster system, nuclear fission, Shakespeare, fox terriers, Top Gear, Led Zepplin, Bertrand Russell, Oxford English Dictionary, the Lord of the Rings, the industrial revolution...

Priesthood and biblish behavior causes wars

pjotrg — Tue, 15 Dec 2009 21:26:09 +0000

Religions have besides comforting many many people caused the death of very many people. I do not think we should approach ITIL like a religion.

I always compare ITIL with a good cook book, build on lots of experience, but...if you want to bake your bread otherwise, no one will deny you the right to do so. It may just end up to have a different taste. ITIL is no norm, ISO is anor and so if an auditor checks against the norm which states: you shall have so an so and he discovers you do not have it, then you end up with a non-conformity. It's as binary as that.

The ISO 20.000 norm (ISO/IEC 20.000-1) states in the paragraph about Incident management: "Major incidents shall be classified and managed according to a process". In the paragraph about Service reporting it is mentioned that: "Service reporting shall include: ........d) performance reporting following major events, e.g. major incidents and changes;..." There is not other occurence of the term "Major Incident" in the norm.
This part one of the ISO norm is the actual norm and includes the mention: "shall". The auditor shall check compliance to these mentions.

Part 2 of ISO 20.000 (ISO/IEC 20.000-2) is the code of practice. This part is the best practice guidance along with the norm and uses the term "should".
Here it states (as part if the incident proces description):
8.2.2 Major incidents
There should be a clear definition of what constitutes a major incident and who is empowered to invoke
changes to the normal operation of the incident/problem process.
All major incidents should have a clearly defined responsible manager at all times.
Nomination as manager of a major incident should give the individual authority levels that are adequate to the
role of coordinating and controlling all aspects for the resolution. This should include the responsibility for
effective escalation and communication across all areas involved in resolution, and to the customers affected
by the major incident.
NOTE This level of authority can be temporary, and apply only during that major incident.
The process for a major incident should include a review which will inform a plan for improving the service.

Herafter the term "Major Incident"is not mentioned anymore.

So......it is not part of the norm and thus the auditor can and may never raise this as a non-conformity. End of story.

By the way: it strikes me that many good things come from the Brittish empire, to name a few: Monty Python, Tommy Cooper, Ashton Martin, The Beatles, Prince2, real marmalade, ISO 20.000, Scotch Whiskey and ITIL.

specialist priesthood

skeptic — Tue, 15 Dec 2009 20:35:00 +0000

let's not bring the individuals into it. there are two issues I see here:

- consultants, especially auditors, who see fit to impose interpretations (however sensible) above and beyond what is "standard". Advice is good, telling someone they are wrong when there is zero authority behind it is not. The absence of a problem record is not "wrong" anywhere but in the consultant's mind. ITIL does not require a specialist priesthood to interpret or infer it. Consultants add value when they advise not lay down laws

- the standards and frameworks SHOULD define some of this stuff in more detail. the ambiguities and oversights in ITIL are excused on the grounds of "adopt and adapt" - that every site is different. And yet the experts seem able to get utterly dogmatic about points that are not in ITIL or ISO20000. If they are that clearcut they should be in there. As it happens i think this particular point is NOT that clearcut, but there are plenty that are.

Suspect anybody who calls himself an ITIL evangelist or guru

aroos — Tue, 15 Dec 2009 20:20:43 +0000

Juan calls himself evangelist and he seems to have a clear model in his head but it is not based on ITIL or ISO 20000. The people who call themselves gurus or evangelists seem to be unable to discuss anything that would weaken their belief in their own infallibility. I would say poor Juan Jimenez ran into a wall of knowledge and could not take it. Diarmid Gibson had a good point and Juan terminated discussion after that. This is the simple fact that Diarmid stated:

"There is nothing in ISO20000 that stipulates the need for a problem record related to each Major Incident. In fact, the standard does not define Major Incident beyond that it is something that requires extraordinary management arrangements."

So ISO 20000 does not support Juan, either does common sense. I don't want to repeat the several good arguments of the long discussion but I did not notice this point mentioned: In many cases organizations take risks. The cost of a major incident can be less than the expected value of the risk (probablity X impact). Then the risk is realized and the incident happens. The root cause is known immediately but it is still a major incident because it has a lot of impact BUT it was a calculated risk. No need for problem analysis.

Aale

the battle of the blogs

skeptic — Tue, 15 Dec 2009 18:47:10 +0000

Well this is fun: the battle of the blogs. Being compared to Monty Python is a good thing right? Astute commentators who pointed out absurdities in orthodoxy.

Readers who are members of LinkedIn group "ITIL v2 / v3 Service Management (ITSM) and ISO 20000" may like to follow the whole thread for yourselves and draw your own conclusions. Then maybe we'll have a poll eh?

inappropriate

skeptic — Wed, 09 Dec 2009 16:55:56 +0000

I find them inappropriate but not as inappropriate as mine

Actually there are mentions

Philippe — Wed, 09 Dec 2009 12:11:23 +0000

Actually there are mentions of data schemes in the books. See for example Service Operation, page 50, 4.2.5.3. Even in some detail : see page 66, 4.4.7.2, Note in the second column.

But it's really risky to give a more detailed list of tables, and fields. You may contradict yourself or even start endless discussions with techies on subjects you don't really control. No one wants to start discussing with techies, they are sometimes right.

When you're an ITIL consultant, it's so much easier (and cost efficient) to spend time chatting with upper management than trying to build something that really works. Let that to the lower class.

Philippe, member of RSPCSV and Che Guevara of ITIL

PS : this could probably be called a "troll" based on the fact that most people who comment on this blog are ITIL consultants. Don't hesitate to cut if you find my comments inappropriate.

Does this stand up to scrutiny?

JamesFinister — Wed, 09 Dec 2009 11:20:26 +0000

Surely it would be in the interests of some of the vendors to promote their own data model, and it would be easier for them to "prove compliance" if such a model existed? I thought HP were actually undertaking this as part of v3, and I presume Ashley's role as a mentor in the update, ensuring the diagrams are consistent, reflects that on going interest on the part of HP.

Given a choice between conspiracy and incompetency I tend to go for the latter. Isn't it just the case that bringing some rigor to ITIL would reveal just how much of it is still very soft? That isn't to say there is anything wrong with it being soft, but it would be nice to know which bits are soft and which are capable of being subjected to detailed specification.

James Finister

What do the auditors know?

JamesFinister — Wed, 09 Dec 2009 11:12:02 +0000

In my experience auditors vary greatly in their capability. It worries me that many ITIL assessments are carried out by people who in practice have limited experience of the full range of ITIL processes, capabilities, functions or whatever we are calling things today. It is certainly clear to me that in depth knowledge of those disciplines that ITIL borrows from is often lacking. So expecting to find an ITIL auditor with the ability to make proper judgments about problem management is being very optimistic. So they lack the internal reference point anyway, and as Ian says, that means they are relying on what the books say. The catch with this is that if you get audited by someone who knows there stuff you are likely to get a lower score than if audited by someone reliant on book knowledge ( see http://en.wikipedia.org/wiki/Dunning–Kruger_effect yet again)

Of course you might argue that the auditors could adopt a black box approach, ignoring the mechanisms of problem management and just evaluating whether there was evidence of effective problem management in place - for instance the number of successful changes that have arisen out of problem management activity. The problem with this is that problem management capability and the the processes around it have to be relatively mature before it becomes effective, so often there would be nothing visible to audit.

As for ISO 20000 audits - I'm obviously a massive fan of ISO 20000, I wouldn't dedicate so much pro bono activity to it if I wasn't - but the audit is primarily about the paperwork, not the effectivness

James Finister
Wolston Limited
www.wolston.net
www.coreITSM.com
http://coreitsm.blogspot.com/

Audits, Root Causes and Driving Tests

ianclayton — Tue, 08 Dec 2009 23:53:32 +0000

Skep
Its my experience that auditors check for existence of compliance or similar - with some black and white 'shall'. Now how that compliance is demonstrated varies wildly and is wholly dependent upon the individual auditor, and what they use as a reference. Cause analysis is the activity here - and it includes root cause analysis... as well as change, task and control barrier analysis. In fact the latter three are typically performed PRIOR to RCA. Using whatever methods work for you - from fishbone diagrams, through fault trees, to crystal balls, a list of causes is developed, ranked, perhaps favorites tagged (all legitimate problem manager work), and then countermeasures proposed... Not new. Not invented here.

As for major incident - as I may have said earlier (sorry that was 9 hrs into a flight!) - what represents a major incident must be defined - typically based upon impact to a named stakeholder. The auditor will/should start there. All major incidents should have a complete record - indicating what action if any was taken, by whom and why. Its quite 'legal' from an auditors perspective to take no action as long as someone is tagged with that decision.

IMHO auditors have no personal stake in the result. They are inspecting for compliance - as I started. This requires some level of detailed reference as a comparison point. So far ISO20K and ITIL V2 and ITIL V3 lack that detail.... deferring it to the auditor. In any audit - PLEASE - ask the auditor what they will use as their detailed reference. Its quite healthy to ready the organization based upon that - its not cheating - rather like knowing the 'highway code' before taking a driving test....

A problem record means the alligator is still out there somewher

skeptic — Tue, 08 Dec 2009 21:29:00 +0000

My definitive (non-facetious) answer to this:

I think the auditor was making the point that it is good practice to always check for root cause of a major incident. An alligator mauled you. Did you kill it or did you just drive it off?

But I think the auditor made the point the wrong way.

How you record evidence of having done root cause analysis is up to you: ITIL and ISO20000 have nothing to say on it.

But you SHOULD do root cause analysis as part of the response to any Major Incident or as part of the later wash-up and review. That analysis is "officially" labelled as problem management activity and is what is meant when ITIL and ISO20000 say you need to "do problem" with a major Incident. If you don't methodically do RCA then I suspect that is the point the auditor was really making.

A problem record is a good way to record your RCA but not the only way, not the "official" way (in fact ITIL clearly implies you only create a problem record for recurring or unsolved incidents: SO 4.2.5 "ongoing or recurring problem"), and not in my personal observation the generally accepted way: I've seen it recorded more in the incident record and especially in a formal post-incident review. The Incident Review template I use has analysis of direct, contributing and root cause. Problem staff are involved in that review and they would create a problem record if the root cause was undetermined or unresolved, i.e. the alligator is still out there somewhere

Audit

JamesFinister — Tue, 08 Dec 2009 12:08:05 +0000

Where to begin? In this reply I will try and stick to the audit issues

Generic ITIL audits are of variable value. Most people designing and undertaking them have little understanding of or training in professional audit techniques, which lessens their value. In recent years there has been a trend to link audits to maturity models, something Rob has commented on elsewhere recently.

In any case there is no definitive guidance on how to audit ITIL, never mind what questions to ask and evidence to consider, which means the auditor must make up the short fall.

From an audit persepctive it would be legitimate for an auditor to point out that based on their experience and best/good practice it is at least advisable to raise a problem for every distinct major incident, even if a management decison is then made not to undertake any further problem management activity.

ISO 20000 is slightly different.

I believe I'm correct in saying that strictly speaking they only audit based on part 1 of the standard.

There was a formal BSI 15000 audit workbook which I wasn't that impressed with from an audit perspective. I vaguely recall this was re-issued as an ISO/IEC 20000 self assessment workbook, rather than as an offical auditors' checklist, but I don't know what the offical status of it is - IIRC it isn't an ISO document.

ISO 20000 auditors do have have leeway in assessing organisations. They will, correctly, take into account the size of an organisation, and obviously can only audit based on the agreed scope statement.

Part 1 does not say that all major incidents shoyuld lead to a problem record being raised. I think it would be reasonable again for an ISO 20000 auditor to ask about those cases where a probelm record was not raised, and assure themselves that controls were in place to ensure this was the result of a documented management decision, not an oversight.

James Finister
Wolston Limited
www.wolston.net
www.coreITSM.com
http://coreitsm.blogspot.com/

Do not have two processes doing the same thing

aroos — Tue, 08 Dec 2009 07:51:53 +0000

There is no sense in having two processes doing the same thing. IM must solve incidents, whatever it takes. PM is a different process with different goals. All PM is then proactive and the goal is minimize or eliminate the risk of the thing reocurring. This concept is from Jan van Bon and I think it makes a lot of sense. Unfortunaly you would lose points in ITIL Exam with this answer.

Aale

I may have missed that point but...

ianclayton — Tue, 08 Dec 2009 04:19:18 +0000

Skep
Accepted I may have missed the functional aspect. Thats for governance to determine on a case by case (customer/service/service model) basis - go to your copy of USMBOK for more on some rules. Anyone could open a problem record - why not - as long as they are trained or guided in how to start populating the problem definition and impact statements - two areas ITIL V3 offers no help on at all. (Strike 1). Regard problem as a suggestion box - as for major incident - specific governance should apply (Strike 2).... and be established as we do not want unnecessary thrashing of resources whilst someones barn is burning down!

As for root cause - complete fallacy. Its very rare indeed to find one. Experience and guidance in non-IT (ITIL) world explains that causes come in gaggles (or whatever the word is for more than one cause!). They have types, of which root is one. Each cause needs to be weighed after careful documentation and a countermeasure formed.... USMBOK terms this a 'solution set'. ITIL misses this as well (Strike 3).

So forget ITIL v3 - its useless for problem management - lets talk about problem management as it is commonly used outside of IT and ITIL... in healthcare for example.. and this will all be so much easier... As I have blogged before - ITIL V3 problem management sucks. Its poorly defined, misleading, incomplete, and without this part continual service improvement is in trouble......

whether a problem record is required

skeptic — Tue, 08 Dec 2009 00:33:00 +0000

I'll admit the point of the post is obscure but you are missing it I think :) The issue is not the difference between incident and problem. It is whether the Incident Resolution procedure is "allowed" to include resolving the underlying root causes, i.e. is allowed to wander into Problem Resolution teritory, and in particular whether a problem record is required (not whether it is a nice idea - whether it is REQUIRED) in EVERY Major Incident in order to document what was done to resolve the underlying root causes, as compared to recording that in the incident record.

At a practical level, there are advantages to creating a problem record, so as to have all underlying faults documented in one place, and so as to have the problem team look further to make sure the alligator was killed. On the other hand it is an overhead. So it is probably a good idea but that's not the issue here.

At a religious level, the holy ITIL book certainly implies to me that a problem is something that cannot or was not dealt with by the Incident Resolution procedure, or put another way if something was dealt with by Incident Resolution then the implication is that it never got to be a distinct problem entity, even if we might involve the problem people in dealing with it.

So i don't think it is at all clear that it is "wrong" to record everything that happened in the Major Incident record - ITIL is hopelessly vague on this. The way i have done it in the past is to produce a Major Incident Review template that includes root cause. I see that as equally valid. The problem team should be part of every Major Incident Review - if there is still a (suspicion of) problem there they can create the problem record. I'm not entirely convinced about creating problem records for their own sake when a Major Incident has been cleanly resolved, which was the advice of the auditor.