The IT Skeptic - Comments for "We should create the problem record right up front in an incident"

Multiple applications

JamesFinister — Wed, 06 Jan 2010 09:01:47 +0000

That probably isn't surprising since my systems experience pre-dates my ITIL experience, and systems thinking obviously influenced ITIL, and it really was pretty much the first thing we taught people on early v1 courses. Some readers here might remember the cookie factory story.

As an aside one of my concerns about the ITIL world is that it latches on to good ideas from external sources and then tries to develop them itself, rather than syncing with what the professionals in the domain are doing.

That Shifting the Burden model is applicable in the ITIL world in lots of areas, not just problem management. We fall into its trap in trying to change the behaviour of suppliers, in trying to keep customers happy and, from very painful experience, in capacity management.

"Shifting the Burden"

Visitor — Wed, 06 Jan 2010 00:36:00 +0000

What you've described is very common archetype found in organizations (complex systems).

http://www.systems-thinking.org/theWay/ssb/sb.htm

Simple is not enough

mbuzina — Tue, 05 Jan 2010 15:06:55 +0000

Why is preventing a recurrance "proactive"?? Proactive mean preventing occurance. In your point of view you could still open the problem early on, as not to lose vital information for your "pro (re) active" problem management. After everything is smooth again you can start investigating how to prevent this. If you documented your incident well, you may even have half a fault tree. But using that is dangerous, you may have missed the other half, which was not needed to restore the service.

Have look at the sample FTA in my recent article http://buzina.wordpress.com/2010/01/05/incident-and-problem-management-r....

Incident >- Problem -< Work-Arounds

bunkermentality — Tue, 05 Jan 2010 14:45:46 +0000

I agree that the schema doesn’t stand alone, nor does it come first, as it complements the processes and role descriptions. I focussed on it in an attempt to bring a new angle of approach to an old challenge.

The schema structure I had in mind differs to the one suggested by James as mine has Incident Record >- Problem Record (one or more Incidents to one Problem relationship - excuse clumsy crows-foot symbol) plus Problem Record -< Work-Around (one Problem to one or more Work-Arounds relationship). This implies the following process.

1. Match a new Incident Record to an existing Problem Record. If a Problem Record does not exist then create a new one.
2. Apply one, or more, appropriate Work-Arounds linked to the Problem Record in an attempt to clear the Incident. If necessary, create further Work-Arounds and link these to the Problem Record.
3. In the Incident Record describe which Work-Arounds were attempted, and which one(s) succeeded. (Use schema links for efficiency here.)

Thus there is an indirect link between the Incident Record and the successful Work-Around(s).

In addition a proactive Problem Management process is needed to manage the Problem Records and Work-Arounds.

1. For each Problem Record review the volume and impact of related Incident Records and assess this against the costs of implementing a permanent fix. (An ROI case in effect.)
2. Management (the Service Manager?) must then decide on the merits of the return on investment case and approve or disapprove the cost of the fix.
3. Thereafter the support teams, via Change and Release Management, implement and apply the fix, thus clearing the root cause of the Problem and ensuring no further Incidents of the type occur again.
4. Whilst reviewing each Problem Record also perform duplication and efficiency tasks by removing duplicate Problem Records and duplicate and/or inefficient Work-Arounds.

In this way the schema structure supports both Incident Management and proactive Problem Management.

work around link to problem

JamesFinister — Tue, 05 Jan 2010 12:16:59 +0000

An overall general purpose schema is just one of the things that I believe should have been delivered as part of ITIL a long time ago. But it is just one thing, it doesn't stand alone, it doesn't answer every question, and it has to be right.

I agree with Rob that a schema doesn't really help us here, in fact there is a danger of putting a horse behind a cart.

As to the schema you suggest:

The workaround (which is different from a fix) has an indirect link to a problem. We can apply a workaround without having an inking of what the underlying problem/known error is and there is no single right place in time to create a new problem record. I can successfully apply a workaround without having an understanding of the problem. Actually we can and do even fix something without going through formal problem management.

In theory all incidents have a cause but in reality most incidents are linked to a predefined catch all problem record such as "printer issue" Many incidents can have a workaround. The relationship between incidents and problems is a many to many one - via a join table obviously because I would hate to encourage poor database design. If it was a on.e to one relationship you might question why we need separate record

Everyone understands the schema (probably)

bunkermentality — Tue, 05 Jan 2010 11:00:57 +0000

Your view is that everyone understands the schema. I hope you are correct. I don't recall seeing any schema-related discussions and so I'm not as optimistic as you.

Using the simple schema to address your questions.

Q: When should a problem record be created?
A; The Work-Around is linked to the Problem Record so the Problem record MUST be created prior to the Work-Around being linked with the Incident Record, i.e. before the Incident is closed.

Q: Is there a problem record for every incident?
A: Yes there must be because the Work-Around is linked to the Problem Record.

Q; What if the underlying problem was quickly and permanently resolved as part of restoring the service?
A: If schema integrity is to be maintained then the Work-Around is always linked to a Problem Record. Integrity must be maintained to ensure that management information analyses are accurate.

The first lesson of a master...

JamesFinister — Tue, 05 Jan 2010 09:15:30 +0000

...is to listen to what people have been saying and the context in which they are saying it ;-)

The majoriy of posters here, if not on the linkedin group, are NOT talking about creating a problem record for every single incident, or at least certainly not on a one to one basis . We are talking about problem management only taking an active interest when the chronic or acute impact of the incidents is hurting the business. So we already have a by exception filter. We are mostly talking in this context about major incidents, if they are a regular occurrence then problem management isn't doing its job properly, so we already have a filter by exception.

That focus on impact on the business should make it clear that we haven't been discussing it as an abstract concept.

I suspect we might be differing substantially in our views of what tpoe of person the problem manager should be. From very early on in my ITIL career I've put the emphasis on the management of problems, that is to say the problem manager is making the call over whether the incident should be fixed is making decisions as a business centric manager. My view of problem management is emphatically NOT that of the L3 techie trying to collect every last scrap of information, but of a manager weighing up the risks and acting as an important balance to the incident manager's desire to close the incident ASAP.

So you appear to be finding incredible a view none of us have. What we are saying is that ITIL, and this is really a very very basic ITIL 101 point, wants to break out of the constant break fix cycle that frustrates the business and give them an assurance that the service is dependable. And we aren't talking abstract theory, I've spent the last two years in just that situation and time after time the C level message was the supplier was "Tell us what you've done to stop this happening again"

Incidentally I would say the greatest need for proactive PM is in stopping P1 incidents . You can afford to be reactive to the more minor ones, but then we are drifting into availability management.

Proactive PM is the point

aroos — Tue, 05 Jan 2010 07:57:28 +0000

Simplicity is good.

I have been trying to argue that all the things you do to get business back running are actually Incident Management, even if you need to find and fix the cause of the incident. Reactive Problem Management would be just another name for the same activity and that seems to be the cause of this confusion.

It becomes much simpler if you decide that problem management is always proactive, the goal is to prevent the recurrance of the incident. It is very likely that some people need to do both activities so it is important that they understand the different goals.

Aale

Still more I cannot believe

ITIL Master — Tue, 05 Jan 2010 02:46:49 +0000

It should be in only the smallest number of cases (by exception) that IM asks PM for permission to restore a Service and some of those few exceptions are discussed above. We seem to be having this discussion as if it were an abstract concept and totally missing the point of being business focused. The mere concept of IT going back on a regular basis to the business and saying we cannot restore a business service because we need to capture diagnostic information to determine if there is a root cause that we may or may not determine we need to fix at a later date is so far away from busines and customer focus that I find it incredible.

The question on PM OLA's is an interesting one when you relate it back to 'who cares'. Has the Service been restored? If yes, take a breath, smoke them if you have them and off to the next crisis. If you were to put OLA's on PM what would they be; time to diagnose RC, time to remove RC. I work with a large multinational finance organisation that have an OLA for RC on every Problem, but nothing for removing the RC which I have told them this is crazy and explained why (I dont need to do that here do I?). One of the driving factors in determining whether we even need to proceed to RC analysis is risk, what is the risk of this happening again and then in conjunction with Impact Prioritise which Problems need to be addressed first.

It really is not that hard with reactive PM - Mandatory for P1 Incidents, desirable for P2! For everything else it should be dealt with Proactive PM (with some exceptions of course). How come nobody is talking about Proactive PM as insn this where we will start driving down the largest number of reoccuring Incidents? We are all talking about reactive PM as we know that in most organisations the IM and PM people are exactly the same people so there is a tight connection

The last lesson of a Master is simplicity

Three environments

JamesFinister — Mon, 04 Jan 2010 23:38:14 +0000

It is also a good idea to have another alternative environment where people can play around without impacting the live environment, but it has to reflect the live environment so that what works as a fix there will work in live. So we end up with a live environment, an operational test environment where changes can be made without impacting the live environment or losing forensic data., and a forensic environment to preserve evidence.

I know to some that sounds like overkill, but in safety critical, highly secure, or highly regulated systems you sometimes need to do such things.

The bucket is the key

JamesFinister — Mon, 04 Jan 2010 23:25:32 +0000

All measures have to be taken as part of an overall framework of measures. In reality there are multiple possible outcomes, some of which the customer will find acceptable, and some they will not. SLAs, OLAs and contracts rarely capture this. And of course those possible outcomes have a range of probabilities. In an esoteric frame of mind I might suggest that what problem management does is to alter the balance of those probabilities. The classic question "How many incidents did problem management prevent this year?" can perhaps only be answered in that way.

It helps improve both IM and PM if standard procedures are followed. Somewhere I have the pilot notes for a Cessna 150 and the laconic page for fire says pretty much "Land the aircraft and get out" but in commercial aircraft we know a more detailed standard check list is followed, part of the intent of which is to avoid wrongly identifying the cause of the alarm. Misidentification of the alarm has been in a factor in turning several aviation incidents into major incidents.

Someone who has not yet been suggested as the decision maker about restoration is the Service Manager, a role early versions of ITIL tended to ignore because it didn't pigeon hole neatly into a process.

Firemen's SLAs

skeptic — Mon, 04 Jan 2010 23:23:00 +0000

"contracts / SLAs that bind the provider to specific restoration times"

Priority 1: If the entire factory catches fire we will extinguish the fire and rebuild the factory within 1 hour
Priority 2: If one room of the factory catches fire we will extinguish the fire and refurbish the room within 3 hours.
Priority 3: If a piece of equipment catches fire we will respond within 4 hours and extinguish within 1 working day
Priority 4: For waste-bin fires, trapped cats etc: we will resolve within 7 working days

Quote from real life: "Dammit the entire systems has been down for hours. how much longer will it take you people to find the problem?"

And the working people still need to cover their aXX

mbuzina — Mon, 04 Jan 2010 22:47:57 +0000

I agree. But still the "working class" will get the blame if important evidence is destroyed. Think about a security incident by a hacker attack. You bring back the service because management told you so and afterwords you have to tell them "no, sorry we could have identified the bastard, but we erased the logs by doing a restore....".

So my proposal: Use modern technology to ease the evidence recording and still stick to a fixed resoration routine. Backup your virtual machines and keep copies of the SAN disks of your system before starting the restore. Ideally keep some seperate HW available to restore the failed systems / machines / data to for inspection. Then PM can start poking at the smoking remains trying to find the black box while IM can put your passengers into a new plane heading towards their destination.

OLA/SLA for PM not on Resolving Problems

mbuzina — Mon, 04 Jan 2010 22:42:02 +0000

Of course fixing a problem is better resource usage than restoring X incidents. No one is there to argue, but working mostly on IT service providers where the client purchases a platform service (often including higher level services as well) I see contracts / SLAs that bind the provider to specific restoration times. This usually includes restoring to some previous point in time, which in turn destroys the logs (it can even be worse on security related issues, there it may be real evidence required to "fix the problem" in a court) for problem resolution process.

My usual aproach is to define a fixed procedure for restoration of the service and have that run. This way I get back up & running within the time my client required. If evidence is important, I would add a backup of the current data / system. In virtualization this is usually an easy job. I would prefer to add an evidence recording step in the service restoration process skript rather than involving problem management, who (in my experiance) ar not used to deal with fixed timelines.

If you recommend to ask PM for permission to restore (even a short "yes, go ahead" requires resource availability) you need to put an OLA in place to meet your targets on the total restoration process. This is quite similar to the execution of your DR / continuity plans. They also run a predifined skript (much more detailed than a mere "process" ;-).

Your metrics remark is right. And not only does it make it's own metrics look worse, it can also make other processes look worse (the old "easy incidents are what make IM & SD look good" vs. Solve the problem of the most occuring incident for example). The same is true for your problem manager. (even if the same holds for change-, service level- and a few other -management topics).

The debate goes on

skeptic — Mon, 04 Jan 2010 17:54:34 +0000

No. But thanks for trying.

I think everyone understands the schema. The debate rests on other issues

WHEN should a problem record be created?
Is there a problem record for every incident? What if the underlying problem was quickly and permanently resolved as part of restoring the service?
etc

Viewing from the standpoint of a schema for a simple database

bunkermentality — Mon, 04 Jan 2010 14:51:15 +0000

The comments here (and in the mammoth Linkedin conversation) provide a surprising insight into the level of confusion, and almost religious zeal, about the relationship between Incident Management and Problem Management.

I absolutely agree with the line of reason which argues that all Incident Records must be linked to Problem Records, and this linkage must be established sooner, rather than later.

Does it help to clarify the argument by viewing it from the standpoint of a schema for a simple database? First an Incident Record is created. In database terms the objective is to link this to a known, or new Work-Around Record. When this is done, implying that the Incident is resolved, then the Incident Record can be closed. When a subsequent Incident Record is created (of the same type) then this must also be linked to the known Work-Around.

So how to we organise these known Work-Arounds? The obvious way is to link them to the Problem Record that describes the underlying problem that is causing the Incidents. Whether the root cause of the Problem is known (an ITIL Known-Error) or not is immaterial. Thus a Problem record is linked to one-or-more Incident Records and also to one-or-more Work-Arounds.

The goal for Incident Management is to match Incident Records with Known-Errors as fast as possible. The goal for Problem Management is to determine the root cause, identify a “fix” and have this applied via Change Management and Release Management. Thereafter no further Incidents occur and the Problem Record and associated Work-Arounds become redundant.

Arguments such as the “which team does this?”, “is this L2 or L3?” etc are kept to one side. Furthermore it also puts the Major Incident debate as the schema applies to all Incidents and Problems. Does this provide a simple way to address the confusion?

Paul

PM design and specification

JamesFinister — Mon, 04 Jan 2010 11:16:43 +0000

First of all I think we have to recognise that there is a crossover between PM and other processes/functions - CSI/quality included.

Secondly trying to design/specify problem management runs the danger that we get a framework for PM but not the substance. There have been other posts on this site about justifiably highly recommended reading in this area, most of it from a non IT perspective. And it is important to remember that just because ITIL doesn't include something doesn't mean it isn't available elsewhere.

At the heart of problem management is a good problem manager. It makes or breaks PM.

The shift from an incident focus to a problem focus is one of the most important cultural shifts in mature ITSM. As with much of ITIL though it is hard to build the business case because part of the implementation is the development of the metrics that would prove the case - but obviously that comes after the decision to do it. A useful starting point is to do a one off PM project to look at something that is causing the customer pain but is low on the IT team's agenda - printer issues for instance.

Fixing a problem once is obviously a much better use of resources than fixing the same incident multiple times.

Organising PM across a supply chain is a major topic in its own right, as is how you capture PM in OLAs and contracts. You can’t set a target for how long a problem will take in the same way that you can for an incident, and other things become important, such as the transparency of the process.

Another point about metrics for PM is what I frequently talk about at conferences on metrics is “The Problem Manager’s Dilemma” This is that the metrics that prove you have done a good job one year are actually counterproductive the next year, so cannot be used to measure year on year improvement. Take something simple like the number of problems identified per year. If that number goes up does it show that problem management is working well, or that it isn’t?

to restore service or to get the diagnostics

skeptic — Mon, 04 Jan 2010 10:13:00 +0000

IM should no more ask PM permission than vice versa. There are conflicting goals here - that is one of the first things ITIL taught me. it is management's role to decide what is more important: to restore service or to get the diagnostics. if this is the third time it has hit and management tells IM to wait, then they wait. if no sales are being made and management tell PM to pull their heads in, they pull them in. It is these rare moments when managers actually earn their fat salaries.

The Problem is most often the organization or the process

mbuzina — Mon, 04 Jan 2010 10:06:00 +0000

Hi Rob, Hi Vinod,

-- a whole lot of text removed again --

Sometimes writing about something gets you thinking. I started writing in support of the L1-L2-L3 path for incidents, but while I did that, some ideas formed which need some more time. So let me use these famous words "I'll be back".

Marc

OLAs on PM

mbuzina — Mon, 04 Jan 2010 08:22:20 +0000

Hi James,

I still am a bit torn between having a deeper PM involvement and the requirements of IM. IMs focus is on restoring quickly and within dedicated time frames. Availability, Capacity and SLM design processes should make sure that a service is recoverable within a dedicated time frame. There is no design process for PM (maybe there should be), so with most of the clients I work for, PM is an activity that is done on a "if resource permits" scheduling, meaning that some have allocated resources (some don't) but nowhere I have been are there enough resources to guarantee any kind of response times, which would be required in such a case. In common cases of off-shoring ITSM execution this really is an issue, since often PM staff is not off-shored, but not available enough.

What are your experiances in getting OLA like commitments from PM?