Great myths of ITIL #1: "You can't manage what you can't measure".
"You can't manage what you can't measure". Who on earth came up with that one? One of my big concerns about the application of ITIL is it's emphasis on KPIs. Useful but dangerous.
Management is about the operation of a function or process. Even though I bet somewhere we have sanitary disposal managers these days, back in the real world "manager" means someone who manages people. They build processes around people and they make them follow those processes but at the end of the day managers manage people.
And you can't measure people. Oh, we try. But you can't. Einstein got unimpressive grades. So many modern corporate ills stem from managers trying to manage people by numbers.
First, I have seen brilliant people (read: valuable corporate resources) laid off because the numbers fingered them. Bad managers applying bad numbers.
Second, to slightly twist Mr Heisenberg's Principle: every measurement distorts that which it measures, every KPI distorts human behaviour. No KPI perfectly reflects the desired outcome - it always leads the measured subjects off the desired path. Management by measurement leads to some weird results (every civil service is awash with them).
Third, the KPI hasn't been invented that measures "good". I used to have to rank my staff, 1 to 12, no equals. What a nonsense. The ranking would shift from day to day depending on the challenges the team faced. It would change from hour to hour depending on who had pissed me off last.
Management by measurement is only a substitute for inept management. Put another way, it is an organisation's compensation for its inability to attract, create and retain good managers. Any decent manager can make correct assessment of staff based on instinct and subjective observation. Numbers help, and they particularly help fire someone, but you can manage what you can't measure. Good managers do it every day.
Now, governance - that's another matter. Perhaps you can't govern what you can't measure...?
![Syndicate content](../misc/feed.png "Comments for \"Great myths of ITIL #1: "You can't manage what you can't measure". \""){kind=small}
Made in New Zealand 
Comments
"You can't manage..."
It's close to Peter Drucker: "You can only manage what you can measure."
I'd recommend Phil Verghis' recent work on "Metrics, Methods and Madness..." which warns strongly about putting goals on activities. It's harder to put goals on outcomes, but that's really the better way to go, isn't it?
So many measure activities (call/contact volume, for example) because they are easy. Good management is not easy.
regards,
Roy
it's all Drucker's fault
Yes it's all Drucker's fault. (Is that sacrilege?)
Verghis' book sounds interesting.
driving with your face in the dashboard
Majid Iqbal on Twitter said:
Isn't that what pilots do
Isn't that what pilots do in rain, fog etc.
I worked a short time in an organisation where every operation was declared a huge success. Very simple KPI's would have stopped the nonsense.
Aale
What Pilots Do...
Aale,
No, they don't. Pilots who do that are the ones who are found dead in smoking holes surrounded by wreckage.
The mantra is - aviate, navigate, communicate - in that order. The flight deck is designed to enable these activities and compensate for the lack of visual references.
The system is built to enable this type of activity. Under instrument flight rules, you can even lose the third and still have a successful outcome. This is something we train and build procedures for.
kengon
Dashboard
I think the original comment though was meant to convey the danger of looking at dials that aren't central to the key tasks, whatever they are - and I was focusing on the "aviate" bit.
Interesting to consider what the IT "..ates" should be:
operate, co-operate, innovate?
No, not happy with that.
J
Like The Direction
James,
I think I got the intent of the original post. Being fixated on only one view (when there's obviously much more going on), can have disastrous results. I was actually elaborating on Aale's post.
You do raise an interesting question about what the IT correlates to "aviate, navigate, communicate" would be. I would propose that communicate stay in place. We need more communication, not less.
kengon
...ate
So "operate, co-operate, communicate" then?
So "operate, co-operate, communicate" then?
I would propose the 3 ates of IT follow the standard IT management ates lest the lesser IT resources be downtrodden further:
lie-ate, kiss-ass-ate, claim-the-kudos-ate
Regards.
MasturB
original ates
Having slept on it perhaps the original three with operate instead of aviate would do?
Incidentally the aviating bit does not have to mean looking out the window, it can be more reliable to use the key instruments that provide situational awareness, which on modern aircraft are located in the same "T" arrangment so you know where to find them.
What about ISO/IEC 38500 as the top level view? What measures would you use to measure ISO/IEC 38500 performance?
Assessments
There is an issue that has been worrying me about ITSM maturity assessments for sometime, essentially are we looking at the right dimensions?
Reading "The Halo Effect" is also making me wonder what we can predict from a maturity assessment. I did some work on a risk based approach assessment a few years ago that tried to home in on key predictors of both current and future performance, but I haven't re-visited it post v3
no measurement
I'm no pilot but in Helsinki during winter most planes land in dark and snow fall is common. Staring at the snow cannot help much.
In my local market I have been studying the phenomena that according to surveys a majority of IT-units have implemented ITIL processes but do not have any measurements at all. Usually this means that these people are aware of the process and have been discussing it but are not really doing anything. It is quite possible that the managers who answer the surveys think that they have a process but usually the fact is that there is no process in place. I have been trying to send the message that there must be some measurement that at least show that the process is alive and kicking. In that sense I do not think this is a myth.
Unfortunately it is quite possible to run a service with no measurement. Many years ago I was working for a Finnish governement IT department. My job did not have any measurement so I did what I thought was useful. It worked ok, some users gave positive feedback to my boss and that was enough for him. After I left the job I heard that my successor complained that I had created too much work for him as the users were accustomed to ask for help. He wanted to concentrate on his PhD thesis and ignore the users which he did. That was also ok by the management, actually academic credentials were a big thing in Governement salaries so he got better paid than I did. I do not think that anybody was actually managing that IT service.
The clearness of measures is one the best things of running a small business. You know what you are achieving. In a big corporation it is far more difficult.
Aale
PS. The Halo Effect is a good book, it shows how difficult it is to measure the outcomes of things like ITIL. Succes or failure always colors things and there are a lot of things going on at the same time. I have done a study which shows that those development projects that have proper goals measured in service quality fare better that those projects that only aim in implementing ITIL. The succes was measured by ISO 20000 maturity. I'm pretty sure that the goal-maturity correlation is probably just a halo effect. Setting proper goals is part of generally better management practices and they tend to succeed better in all things they do.
Aale
instruments and measurement
Re flying by instruments, I think the truth is somewhere in the middle as usual. there's a reason pilots have all those instruments and there is a reason they still have cockpit windows as i think James mentioned recently. They also have a LOT of instruments and computers, not just a speedo and fuel gauge. No instuments is as bad as relying solely on instruments. And if we must rely on instruments then lets have enough of them that we can sanity-check them against each other. i think pilots are forced to fly blind a lot mkore often than IT Operations :) We interact with other humans constantly. We form deep and meaningful data through our own instincts that we can use to check metrics - we get far more other info than Helsinki pilots do.
Wholeheartedly agree with earlier point about measuring with somethign other than ITIL - it is a favourite point of mine. I think even ISO20000 or risk are too close to home. In my book Owning ITIL I propose measuring a service culture audit. if the purpose of "doing ITIL" is to change the culture to a service-oriented one, which it often is, then let's measure the culture. And if it is to reduce costs them let's measure costs. Stand back and neasure the big picture. THEN drill in.
Blind Isn't Really Blind...
When you're in a cloud, a window isn't of much use to you.
Flight by reference to instruments allow you to keep your orientation without having visual reference to the ground, understand where you are relative to where you want to be, and be in communication with others who are participating in the system (both ATC and other pilots/aircraft).
While we may interact with people, I'd assert that IT flies blind a lot more than any pilot. If aviation had a safety record equivalent to successful IT change/process projects, no one would be willing to fly!
kengon
Read the right dials
http://en.wikipedia.org/wiki/Eastern_Airlines_Flight_401
First rule of flying, particularly when things are going wrong, is "fly the plane"
Don't get distracted by peripheral data
Very nice.
Very nice.
Model, Measure, Manage - Definitions needed
Skeptic: W. Edwards Deming (http://en.wikipedia.org/wiki/W._Edwards_Deming) is often incorrectly quoted as saying, "You can't manage what you can't measure." In fact, he stated that one of the seven deadly diseases of management is "running a company on visible figures alone".
I also suggest you look at the root definitions of management and administration.
I use the following definitions.
Administer: "To establish and enforce policy and procedure."
Manage: "To change disorder into order."
You "administer" the order established by managing, to reduce the inevitable disintegration into disorder/chaos/creativity/variance/error over time. (Chaos = infinitely complex order. Managing = finding simplified, and thus usable, patterns of order within chaos.)
Managing requires one to model to some cost-worthy level the current or past "disorder", model to some cost-worthy level the intended "order" to be administered, and model to some cost-worthy level the plan to transition from the current disorder to the intended order, updating all as time passes.
Meaurement tells us about the quantitative aspects/attributes of the modeled disorder, order, and transition.
You can neither measure, administer, nor manage until you "model".
Roy
Modeling the dynamics of carbon-based life forms
Life and existence is chaotic and infinitely complex - period, but that does not mean random. Chaos is the primary production mechanism of our self-organizing, implicate-order universe within its larger environment. Read about David Bohm, Ilya Progogine, and Bernard Haisch, e.g., on Wikipedia.com, for more detail.
But life can still be modeled (as a representation or architecture of the reality, not an absolute replica - the map is not the territory) without getting into the quantification of operations research, management science, simulations, business intelligence, and decision support. As a physicist and management analyst I've used all of these methods - and they are very limited, giving a constrained, closed-world view, without more modeling at the beginning of their use, and continuing during their use.
Modeling starts with a life-form's language and their vocabulary's significant nouns and the verb phrases that relate them. To see a model of a life-form's viewpoint, look at their writing. Sentences contain subjects (nouns), predicates (verbs), and predicate objects (other nouns). Writing anything is intended to record and share a viewpoint. The first level of modeling is to create lists of words used, identifying terms of single and compound words, defining the terms from individual/local/group/organization/global contexts, grouping the terms into broader and narrower meanings (as a taxonomy), and building a concept map of the nouns and their relationships.
Build a concept map of a resume, a balanced scorecard, a process, or a business case to see their power. Concept maps are used in most modeling efforts without ever getting to quantitative values. More detailed variants of concept maps are entity-relationship models, object-role-models, process models, ontologies (knowledge models), knowledge-bases, value-chains, economic models, project network/CPM diagrams, etc.
The term that broadly names the above viewpoint is "terminology", encompassing the various stages of organized "semantics". Terminology is also described quite well in Wikipedia.com. My own approach to building a terminology, so that governance methods (i.e., modeled, measured, managed) like ITIL and ITSM can be their most effective, efficient, and responsive to change, is documented at http://gem-ema.one-world-is.org.
map humans
Agreed we can map humans - what they are saying. We can build a picture of their ideas. Not the same as measuring them though.
Measuring Humans
We can measure the "recorded" and observed aspects of humans and life in general (that's social and biological science). See, comment, and vote on my ideas for Recovery.gov at http://tinyurl.com/ctgr6d. Deadline is end of 3 May. My theme is terminology.
Its About Exercising Judgement
Great thread, this response goes madly off in all directions but.....
Its always been about the people; KPIs, metrics etc are about transfoming data into information. Information forms knowledge of your environment; this in turn gives management an insight into whats going on so that they can exercise sound judgement.
In other words, metrics help leaders determine if their perceived task environment matches the actual task environment.
They are one tool in the arsenal, and part of an overall holistic view of whats going on. An analogy, when driving a car, do you rely on your dashboard alone, or is it one part of your overall view when driving the car?
The seven steps to improvement start with What should we measure?... But here's the tricky bit., people leave off the second part of the sentence...Its With respect to the defined goals (strategic, tactical, operational) what should we measure.
That shapes your context for the KPI's, data etc to gather.
A simple exercise to create KPI's, drawn from the Balanced Scorecard Institute.
Step 1) What is the goal / mission / vision of the organization. (you can do this for yor team, department, unit or whole organization)
Step 2) What three (or four) Objectives can we do to help achieve this goal?
Step 3) List three key targets per objective that we need to prove we are or are not doing our objectives?
Step 4) Identify where you are going to gather the data to prove the targets are being met.
Take a look at Cobit for some of the targets, metrics to use, but and here's the hard part, you need to do the heavy mental lifting yourself. It takes effort to build an appropriate measurment framework. And in the end, the best leaders I have worked for use metrics to validate their intuition.
Side bar, take a look at Deming in the CSI book, his 14 steps are there.
task environment vs people
Ah you have clarified what i am saying with a key point here: metrics are for managing the "task environment". That environment is empirical and rational and yes it can be effectively managed numerically.
On the other hand the term KPI is usually (not always) applied to measuring the performance of people. This is what I object to, when it becomes slavish. Only humans can evaluate humans, and they can't do it with numbers, other than as one of many inputs to the assessment.
Managers manage both the task environment and the people in it. The tools and techniques of one don't work for the other. Thanks for crystalising that.
As buraddo said above "Its a mixture of data, experience and leadership (along with other things) that makes good management"
carbon-based life forms
...then Deming is regularly misquoted by the ITSM community.
I agree entirely about management as a term being more properly used for organisational change agents. however in the modern world the word is utterly debased to the point where we have front-of-office managers greet us and sanitation managers filling the dumpster. I understand what you are trying to do, to hold the line on the meaning of the word. Good luck with that.
ideally yes a manager enacts change and an administrator makes it so. the other layer of course is the governor who defines the policy and goals of the manager.
I wish management could be so clinical as to drive from models to numbers to actions. There is unfortunately the small matter of carbon-based life forms. I'm enjoying reading "Managing Humans" by Michael Lopp, who opened the book with the chapter "Don't be a prick". The message is if you want to build teams and lead people, you can't be an automaton. You must see and relate to and react to the people reporting to you, and you must fulfil their needs. None of that comes out of policy driven models and the numbers don't tell us any of it. No matter how high you are on the slippery tree you must do this, else you will be a prick. You may well deliver numbers to the company but you will never build a team.
If everyone was Einstein, or even Bill Gates
You can measure, and doing so improves both our ability to manage, and our performance, especially when the people doing the work measure themseleves, manage themselves, and improve their own performance.
Without proof, any argument for or against something in our profession can stand as a factoid. That's what this so called myth-buster is, just another myth. Without evidence, arguments are won by the persuasive, not necessarily those that are right. Until we can learn to use mesurement and data we will always be slaves to the latest, most popular myth about how to do our work as told by the most persuasive. If this is how you want it, consider a career in polictics, the fashion industry, or entertainment. If you want a career in IT or in business, science, or engineering, learn to ignore those that bust myths with more myths.
You think Einstein and Heisenberg make your point? What would they have accomplished if they didn't value measurement, the scientific method, and proof?
The problem with Science
The problem with Science is it is an industry designed to disprove itself.. An atom is an atom until a sub-atomic particle exists. The data in which alot of scientific "proofs" are based is often tainted by the data collect paradigm, or even "mathematical assumptions" (it doesn't quite fit, so lets fit a really interesting constant in there and see if that works)..
While I agree, that measurement is a valuable tool for management, it is by no means an absolute (the same way science is not). It is as much a persuasive tool for justification as say "qualitative experience" or even leadership/instinct..
Newton's theories on gravity are powerful because of the maths and data, but even more so, because an apple hits you on the head if you sit under an apple tree.
Management decision making should not be crippled by data, but management should always strive for appropriate and effective measurement of business as a quality discipline.
Its a mixture of data, experience and leadership (along with other things) that makes good management, disregard any of these things at your peril, focus on 1, to the detriment of others and failure is assured.
$0.02
Brad Vaughan
http://blogs.sun.com/buraddo
IT is a superstitious art
Jim, I reckon yours is one of the best posts on this blog, even if I'm on the receiving end of it.
I would dearly love to see more science around IT. I think the slowly growing understanding that IT is a superstitious art is one of the first signs of some real maturity. "If they built buildings the way they build IT..."
So, touché! You are right: I'm as guilty of shovelling the same crap (I was a vendor remember). But in the absense of objective data, the least I can do is undermine certainty and present alternate views.
Heisenberg
I don't know about you, but in principle I'm uncertain about Heisenberg.
Sorry,couldn't resist it.
Interesting read is Julian Havil's "Nonplussed!" I like the Anon quote in it 'This is a one line proof...if we start sufficently far to the left.'
Management without Measurement = Moving Target
First of all, managing relates to people and budgets. In both cases, you set goals and measure progress toward those goals.
The author states, “You can’t measure people.” However, you certainly can measure whether they do what is expected of them, you certainly can measure their performance against standards, and you also can measure their relative performance against their co-workers. For the last measure, in high-performance cultures the bar is high, and for low-performance cultures the bar is low. In either case, you benchmark (measure) performance standards to see if you need to raise the bar for you team or not.
If this author’s twelve workers really are subject to his whims about who is “good” or “bad” which changes as he states, “from hour to hour depending on who had pissed me off last,” then I really feel sorry they have to deal with this incompetent of a manager. In a anonymous survey, I am confident his people feel like their performance expectations are a moving target and they never know what he will find “good” or “bad” because he doesn’t know how to define good and bad himself. It is likely to be a function of the external pressures he is facing rather than any concrete objectives he has laid out for his team like a competent manager would.
This article was obviously written by one of the masses of inept managers out there that don’t know what to do, don’t like to be held to goals themselves, and don’t like their poor performance against objectives measured at all. These are the same desires of people in ghettos and trailer parks all over this country; they don’t want to be held to others’ ideas about performance standards either, and with good reason – they don’t measure up.
There are ways to measure people’s performance in objective ways. There are ways of capturing the inherent variability in a measurement system and determining how much variability the measurement system itself contributes to the measurement and how much is the actual process performance itself.
cogs in the machine
Thanks for replying Joe. I always welcome reasoned debate. Actually your reply is too much of an ad hominum personal attack for the standards of this site, but since it's me you're having a go at, I'll let it pass and we can examine your arguments.
"You can't measure people but you can measure people"? You'll need stronger reasoning than that Joe. My whole arguement is that measures exist, and are used excessively by inept managers who can't deal with real live peopel without putting numbers in the way (not that I'm for a moment suggesting you're an inept manager Joe), but that in and of themselves numbers don't measure people.
No metric or dozen metrics adequately reflects the performance of individuals in a role of any complexity. That sort of time-and-motion thinking might work for unskilled workers in an old-world industrial task, but in modern service industries all but the most basic of roles such as call-centre operator or data entry clerk have too many aspects and subtleties for numbers to fairly and truly reflect an employee's contribution. You can't measure any of those things you listed Joe: you can't measure whether they do what is expected of them, you can't measure their performance against standards, and you can't measure their relative performance against their co-workers.
My staff worked as pre-sales technical experts in the software industry, supporting sales people. They had no regular output product of their own; they engaged in entirely different transactions every time; they added value in complex, subtle and often unnoticed ways; and they functioned as part of a complex team with no individual accountability for the outcome. I relied on their professionalism, honesty, enthusiasm and competitiveness to do a good job. How do you measure those Joe? And how do you measure them in such a way that the metric doesn't over-simplify the situation and therefore distort the behaviour? All metrics do.
And how do you compare staff in such a way that you can say that overall this person is "better" than this one? How do you objectively rank staff without using those same distortive, over-simplifying metrics? Nobody is better or worse than anyone else on paper. When the call comes from above to lay off one of them because it is time for the corporation to demonstrate its loyalty to its most valued resources once again, the decision ought to be made looking forward based on an infintely complex web of anticipated requirements from the business and estimated future potentials of the staff, not on some arbitrary numbers that reflect one sad little side of the past story.
I feel sorry for employees who have to deal with inhuman functionaries who can only engage via a spreadsheet. I'm proud that I dealt with my staff as people not resources. Oh, I had spreadsheets and metrics, and I used them as I should have: as one more source of information to use when deciding the fate of friends and colleagues who deserve better than being treated as cogs in the machine.
measuring managers
And of course you can begin to question how good a manager is if their staff consistently under perform, or if their churn rate is above average. Personally I hate the tyranny of the annual appraisal. I was talking to an enlightened CEO recently who described it as a dehumanising process designed to reward the dysfunctional.
who's fault is it if that potential isn't realised?
I have done a lot of work on professional development of staff. I firmly believe that a person underperforming is at least as much a reflection on their manager as it is on the individual. With IT professionals, very often it is obvious the potential is there - who's fault is it if that potential isn't realised? It is the manager's job to bring it out. If the metrics are bad, the boss also has some explaining to do.
I agree that churn is actually a good indicator of a manager. People join jobs. They leave managers.
Metric Monkeys
Hi Skep,
You are quite right. If you train your staff to "hit the numbers" without looking at them holistically, then they'll learn to make you happy (get the annual salary increase) by ensuring the metrics are good, most likely at the cost of working as a team and retaining any interpersonal [why are they called soft?] skills... etc.
To give an example; I used to have a team leader who, on night shift, would call the incoming Helpdesk number from one phone, pick it up at the adjacent desk and then immediately drop both handsets back into the cradles. Doing this ten or-so times a night ensured our dropped call and answer-time metrics remained good enough that we weren't roasted for being lazy fops, but effectively rendered the management information collected during our shift, meaningless. I'm not convinced this made him a good leader.
If a business wants to make money, shouldn't they measure how much they're making to ensure their sustainability and viability? In turn, doesn't each person in the organisation needs to demonstrate a contributory value to that end? Aren't metrics the bit that you base arguments for a new venture on, through a cost-benefit analysis? Aren't we supposed somehow to demonstrate a return on investment, not just for tin, string and vapour, but for the bodies too?
I think that numbers are a valid part of the story, not just for measuring the provision of service; Did your parcel get to where it needed to be, undamaged and in time?, but also for measuring the performance of an individual; As a driver, why is it that nearly 25% of your parcels are returned because they're damaged, yet for the same category of parcel, vehicle and terrain, the nearest of your colleagues rates is 7%?
The answer may have many components and factors, but the metric has prompted the question "what's wrong" and in managing people that can sometimes be a very helpful way of beginning to help someone realise their potential.
numbers are a valid part of the story
Agreed. As you say "numbers are a valid part of the story", and the valid part they play is the one you highlight: asking why.
A good manager says "why is it that nearly 25% of your parcels are returned because they're damaged?". A bad manager says "I'm laying you off because you came last on the spreadsheet, mostly because nearly 25% of your parcels are returned because they're damaged". The good one manages using numbers. The bad one manages by numbers.
Well... just as you wouldn't shoot the messenger...
Hi Skeptic,
I always love a good controversial point of view, makes everybody think.
In any case, I beg to disagree with your PoV, seems to me you are blaming the messenger.
You state "the KPI hasn't been invented yet that measures 'good'".
Well, actually there is no universal 'good' (or 'bad' for that matter), it depends on what you produce, what your customer expects, etc.... So the first question one should ask is "What is good in my context?"
Indeed, a lot of bad managers misuse metrics and KPIs.
In my experience this is mostly the result of
(a) the manager simply not understanding what he/she is managing (it is more than just people btw)
Making it hard for the manager to know what he/she needs to measure exactly (and what not). So he/she takes the easy route, buys a few books (e.g. ITIL v3) and just blindly copies what is in there. End result: a lot of numbers are being collected and crunched everyday, but there is little or no effect on what actually counts: the quality of what is being produced.
(b) mixing up metrics and goals
Metrics are there to help you understand what is going on. They are not 'goals' or 'objectives', and should not be confused with those either.
Maybe do a google on "Deming's 14 Points"
Good managers don't manage by numbers
I think we agree on all that :) Good managers don't manage by numbers, and bad managers misuse numbers.
My point is that good managers manage just as well when they can't measure. [And many things that matter - like people - can't be measured]. Therefore the statement "you can't manage what you can't measure" is B.S.
P.S. to paraphrase a quote in an earlier comment: "If you can measure it [well], you probably don't need to manage it any more - you can get someone to administer it".
Jumping to conclusions?
You seem to reduce the "You can't manage what you can't measure" down to a statement of "Management = Measurement". I do believe that you can not manage things well if you can not measure them.
You need to get some feedback on how well you are manageing and not only by your peers or from your employees (as both might be to good), but also from a more absolute standpoint. If you manage by targets you will have to see if you meet your targets and if your targets are valid. A good manager has a gut feeling about all of this as well, enabling him/her to look where trouble may be and to improve on more intangable things, but really sound management is only valid if you can measure something.
Your ciritique on KPI is (to my humble opinion) more valid. Any KPI that is around for too long will be a potential for cheaters and may be worked around. Good management will find this out and adapt. Even better manegement will lead their employees in a way that they do not even think about working around a measurement (binding bonuses to KPI targets is OK, but it should not be the only criteria).
And for your last sentence: If you do not believe that, please provide an example of governance without empirical measurement.
If you have a hammer everything is a nail
There was a white paper doing the rounds not long ago that said that a 'fool with a tool is still a fool'. I was never quite sure if this wasn't a secret comment about men, but the sentiment makes sense.
Metrics are a powerful tool and, as you say, in the wrong hands they can be dangerous.
Dangerous tools should only be used by people who are well trained in their use and use care and craftsmanship in their deployment.
Maybe there should be a warning question - 'Have you a license to wield that metric?'.
I think that this is a possible additional value in using metrics. They do force you not only to consider what can be measured, but also to think about what exactly you intend to discover from using them and what behaviour change you hope them to produce. Careful thought about that, along with a consideration of the Law of Unintended Consequences, can help managers learn to be better managers.
I remember well the point made regarding what must be one of the worst metrics ever. 'We have found that you have a bad attitude'. How is 'attitude' measured? What factors determine if it is 'bad'? It is the perfect example of bad management - the manager can't be bothered, or hasn't the ability, to say exactly this, or exactly that in your behaviour is a problem. Nobody can fix a 'bad attitude' because it is an opinion, not a measure.
Yes, of course, we all have met people with 'bad attitudes' and, to some people at least, all of us must, at some time, had a 'bad attitude' (turning up eagerly to work when the cultural norm was to slack off would be a 'bad attitude' to some). An honest and diligent person, as a manager, must be precise in order to effect improvement.
It reminds me of the poem by Robert Frost, 'Building a wall'. To paraphrase, good metrics make good workers'. And good managers...
Now that Peter was most
Now that Peter was most certainly hitting the nail on the head. Very valid points indeed.
Bad managers always find a way to manage badly
One of the things I enjoy about reading Deming’s books is the elaborate dance he executes in endorsing measurement and cautioning against the abuse of measurement. Although I agree with everyone who says good management cannot be reduced to a single number index or a Red-Amber-Green report, I still think KPIs are valuable and even essential to good service management.
Whenever I need an example KPI, I go to the impressive array of KPIs in COBIT, not the ITIL literature. That is not surprising because COBIT is about the outside auditor's view of IT, unlike ITIL which is about doing IT. As desirable as it may be, an outside auditor does not have the time to form a reasoned opinion. An auditor has to make a few quick observations and move on. Those observations have to be well-chosen, but in the end, the observations must be completed in time to sign off on the audit.
IT managers face the same problem, on a different scale. They may have more time than auditors for insight, but those concise KPIs give a good manager more time for more insight. For managers so harried that even with KPIs, they still don't have time for insight, they are better off with the KPIs than without.
Of course, that still leaves the question of KPI quality. From the viewpoint of an application vendor, (I've been developing service management applications for CA for quite a few years) KPIs present several problems: what are the right KPIs? How do you gather them? How do you present them? Who do you present them to?
COBIT has helped with these problems. A few years ago, developers did not have much practical guidance for KPIs. We could go to our best and most forward thinking customers, we could glean ideas from ITIL, a few luminaries and some of the services folks in the field could all help, but you were still pretty much on your own.
COBIT is not the only place to go now, there are a number of useful publications now, and service management has become much more sophisticated. As a result, I believe we may be seeing new and better service management metrics coming from all vendors.
Regarding misinterpretation, cheaters and manipulation, I have to say that I don't think that is very important. Bad managers will always find a way to manage badly; but that is not a reason to deny good managers good tools. Marv http://community.ca.com/blogs/itservice
If you can measure it, you can administer it
Today I got a lovely quote from an ISACA training day: "If you can measure it, you don't need to manage it: you can administer it". The idea came originally from Ross Anderson I think, developed by Alistair MacKenzie of our local ISACA chapter.
Skep - Your argument
Skep -
Your argument confuses *what* to measure with *how* to measure.
The real obstacles to measurement are conceptual, not the methods. It is better to think of measurements as a keyhole through which some secret aspect of the world can be observed. Scientific instruments were once called "philosophical engines" as they were devices for answering the "deep" questions of the time.
Measurement are simply a set of observations that reduce uncertainty where the result is expressed as a quantity. The "uncertainty reduction" is what is critical to managers. A manager asked to make decisions under a state of uncertainty and risk can improve his performance, often financially, by using measurements. The bigger, riskier and more uncertain the decisions, then the greater the importance of measurements for competent managers.
[Myth Buster: Einstein did not get unimpressive grades. He was in fact a gifted student. After graduating 2nd in his class at Aarau school in Germany he was admitted to Zurich Polytechnic at age 17 where he remained at the top of his class until his research thesis. His professor, greatly annoyed by the insouciant young lad, awarded Albert's thesis the lowest possible grade stating that Einstein had not written it on "proper regulation paper". Thus explaining how Einstein graduated near the bottom of his class.]
I've read that book too!
>Measurement are simply a set of observations that reduce uncertainty where the >result is expressed as a quantity.
Hey visitor!
I've read that book too, and it's really interesting! ;-)
Antonio Valle
G2, Gobierno y Gestión de TI
http://www.gedos.es
Interesting
Hi guys
I found the topic an interesting one, Especially from the skeptics perspective. I personally feel that KPI's are crucial in service delivery. Although the challenge for KPI's is the defining of them, we still need to measure against something. How will we draw up a strong SLA if we do not have KPI's. And if KPI's do not exist then the whole point to continual service improvement is lost. CSI is based purely on metrics. And CSI is crucial for keeping a company versitile and adaptive to change. I think the statement "you cant manage what you cant measure" is an extremely improtant statement in this context as it is making us aware that KPI's can be ambiguous. It is up to an experienced manager to know what truly valuable KPI's are. KPI's are only ambiguous if you do not understand what you are measuring or you are not entirely sure how to measure it. If this is the case DONT MEASURE IT.
Theory v reality
I've spoken a lot in public about the value of KPIs
http://www.itsmf.com/events/event.asp?EventID=203
But has anyone experience of KPIs that have actually altered the way a service is delivered for the better?
Value of KPI's
Spent alot of time in SixSigma projects (good ones surprisingly) which have delivered service level improvements, higher service quality, cost reduction, the whole nine yards..
Bad execution of SixSigma can cause all the problems that "Skep" outlines regarding inappropriate application of KPI's..
Brad Vaughan
http://blogs.sun.com/buraddo
Performance and Perception
In the last 10ish years of being a consultant in this field I've had many discussions on metrics, measurements and KPI's. In many cases it is all a matter of semantics. What do you mean when you say KPI? And what is measurement according to you?
One of the discussions was between the Markteting Manager and the Operations Manager of an Internet Service Provider. The Marketing Manager wanted the IT department to measure the systems from his customers perspective: can they access the Internet, the Portal or their mail? While the Operations Manager wanted to measure the availability of the individual systems: is the specific mail server running and is the link to the Internet Exchange still active? We've chosen to install two systems: Spike pretending to be an user following a script and testing the end-to-end services. And Track monitoring the individual systems performance. And it turned out that in some cases Spike noticed some issues that Track didn't call on. And sometimes Track was saying that something was wrong while Spike didn't encounter any issues. In most cases though they would confirm each other. Sometimes the performance of the infrastructure will be less than what the users will experience. Sometimes it will be the other way around. Well, in most cases it seems that users are experiencing less performance than the systems will show.
Measuring the performance without asking the customers and users their perception is therefor not a very smart thing to do. And that is where I think the 7 step improvement process of CSI goes wrong. Yes, you'll need to measure your performance based on the KPI's you've set. These KPI's should be derived from the IT Strategy. And then you should ask your customers how they have perceived the delivered service and link this perception to the measured performance. Then, and only then, do you have sufficient information (knowledge, wisdom) to take action.
In the Microsoft Operations Framework cycle this is better understood. There are two monthly reviews: the internal Operations Review looking at the performance of the services and the external Service Level Review looking at the perception of the customers. The second Review together with representatives of the customers. Both Reviews lead to possible improvements
Perception
I would add that it is worth finding out what aspects of the service most affect the customer and user perception of service before setting up the metrics. A lot of the time we fail to measure and report on the things that cause most concern to our stakeholders
ITIL Myths
Don't think ITIL can claim #1.. Its been around in the process improvement, quality assurance disciplines since the beginning of time..
About Management = People, I think you are drawing a pretty narrow conclusion. Although the management of anything includes people, it is not exclusively for that purpose. Our relatives in "Project Management" might have a definition that is broader, or perhaps anything who is involved in "Systems Management" market. Management is about getting the best out of anything, or more specifically getting the "appropriate performance" (assuming excellence is not always required). Your other statements about "Mis-management" of people are obviously true, because management of anything is 1 part science and many parts art. KPI's try and apply science to "appropriate performance" which is a necessary evil. Of course, poor application of science is dangerous.
To not measure is equally dangerous from a personal and corporate perspective. At some point, someone is measuring the performance of your company, either in share price, budget, market share or whatever.. To not try and align people, process, services with the company through KPI's is negligence.. Just don't live and die by the sword..
Myths of ITIL
#1 "ITIL Implementation Project"
#2 "ITIL Certification"
#3 IT Services Management = ITIL
#4 ITIL V3 (you telling me its a product = "as if service management best practice is evolving to a release schedule")
#5 ....
$0.02 (and no reference to my blog this time :)
Brad Vaughan
http://blogs.sun.com/buraddo
#5 itSMF - a foundation ?
A non-profit organization serving some altruistic purpose, or a cartel driving revenue for the bosses ?
A bit synical..
Brad Vaughan
http://blogs.sun.com/buraddo
Can't top that
darn it Brad, how am I going to top those five?
Management, governance and measurement
You're right that measurement is no substitute for management. Good management is not easy and, consequently, not often found. There are many ways in which people given the job of 'manager' try to duck out of their responsibilities and relying on measurement as substitute for judgement and understanding what is going on is just one of them.
It is also true that metrics, rightly, alter what they measure. If they are poorly chosen they distort what is measured. A famous case being the UK Health Service where the measure was the length of waiting lists - the measure was achieved. Congratulations all round. Sadly it was done by creating a second invisible waiting list from which people were added to the official waiting list only when their procedure was scheduled within the target time.
In 'Metrics for IT Service Management', I argue for using balancing metrics to try to detect and minimise this distortion - if they'd measured the length of waiting lists and the user satisfaction with the time they waited, then they'd have seen a magical improvement in one stat with none in the other, giving the game away.
I'm a bit wary of the term KPI too. You only really know what your Key Indicators are after you've been measuring them for a while - it took some time for the car dashboard to settle down to speed, distance and fuel levels as the ubiquitous standard minimum indicators. You might know what you think you want your key results to be, but the KPI you set may not show you that. Since human behavior is complex and people are not stupid, particularly when dealing with their own performance measures, setting good metrics is a subtle management task, not a suitable job for simplistic command and control freaks!
If you want to prove that you've improved something, though, you need to show how, and metrics are good for that - to tell you whether you've succeeded; if you choose them wisely.
Same difference
Hi Skeptic,
I can agree with you, but it is very obvious, isn't it. Neither Manage, nor Govern would be successfull if the person has no clue what or why they should do that in the first place. Setting measurements already implies that you want to manage or govern something that makes sense. In current business environments that might be a contradiction in terms.
I am on my way to the HP OV Universe in Barcelona and most properly I will hear that statement once every half hour and most probably by individuals who can not stay within the 30 or 45 minutes of their allocated time. Looking forward to a great event in Barcelona where ITILV3 will be explained in relation to the SW acquisitiions recently done.
CU?
What is HP announcing in
What is HP announcing in Barcelona???
That they've finally figured
That they've finally figured out what they're going to to with OV Service Desk and (Peregrine) Service Center.
Or that have have regained focus on their ITSM approach.
Hopefully....
or perhaps admit
or perhaps admit that all they really want to sell is tin???
Message from Barcelona
The impression they left with most of the people I spoke to is one of uncertainty. Lets first look at what is new on the block.
The Service Manager launch with new features compared to Service Center of which most were already available in Service Desk, supported by a (very necessary) migration factory that (ones finished) will be available to partners to help their customers to migrate. Unfortunatly no pricing on licensing or migration. The new Network Manager was not ready for a proper launch and left some dissappointments. Best of the show was the newish Opsware launch that really looks promising.
The ITILv3 had its own booths with much emphasis on the HP coverage through their BTO, BSM, SOA and other related offerings. Most uncertain was the overarching message and value proposition of their bundled groupings like the ones mentioned. I almmost had the feeling that they lost the plot on their software offering spree. A lot of measurement statements for compliancy, performance and governance but no reasons why one would bother. The focus on business delivery and value seems to be copy pasted across all offerings without a clear central console or structure that could make sense out of the whirlpool of solutions. The sentiments of the old building blocks got a new positive impulse.
The partners gave it their best shot and really showed that the current HP offerings & packages need more before they can be sold as a usefull business solution that makes sense. Maybe that is what we can expect from HP. They offer the broad message and grouping of solutions and the partners create the practical connections in between and to the business side.
Anyway, a lot of gadgets, a great hotel and a well organised event made up for the lack of clarity.
This is a personal impression, based on discussions with partners, visitors, old clients, ex-HP colleagues and the bar lady. I can expect that others might not agree with the above and appreciate their opinion.
Cheers.
I have implemented and
I have implemented and trained OV Service Desk up to v4.5 when it was rock solid and straight forward.
Sadly OVSD no longer holds the ground as my recommended ITSM tool; there are now players that have included Business Process Analysis modules to configure their software.
Basically the latest (and better) SM tools have forked away from HP...
Being a Gaudi fan
Being a Gaudi fan, Barcelona is high on my list of dream destinations, but not this time :-D