Sizzling criticism of the ITIL Version 3 Foundation exam
Some sizzling criticism of the ITIL Version 3 Foundation exam was posted on this blog recently by Ian Clayton, reproduced here:
...how can you test an individual's basic grasp (their understanding - if we are truly using Blooms taxonomy here), of 380+ figures, 130+ tables and 2000 pages with 40 questions? Simple math and a vanilla application of Blooms gives us an interestingly bigger number.
I'll use a basic formula and bow to any better: Number of subject areas x importance factor x Blooms taxonomy factor to verify type of knowledge required (lets assume for Foundation its # areas x 1.0 x 3 questions.
Lets exclude the items in the appendices for now and focus on the main v3 content, for many that may be the humble 'process'.
We have 5 books + @25 processes + 5 floating topics = 35 'subject areas'. Multiplied by an even importance factor = 35 multiplied by a Blooms factor of 3 (3 questions per area, goal, key concept or two) = 105 questions and a few control or test questions thrown in for good measure to ensure there is a quality management element - giving us around 120 questions.
I am keen to understand what thinking or math was used to leave the current exam at 40 questions, especially when even the Refresh folks admit up to 50% new content...! We have heard 'unofficially' that during this period exams have been subject to 'moderation' - thats laudible but what logic is being used?
Who performed the moderation - the examiner panel? If so, this is akin to the change submitter testing their own change! Did they identify rogue questions? What adjustments were made and why? Does it affect the syllabus, if not then how were questions written that were out of line with the syllabus? Has moderation ended? Were the 'refreshed' exams subject to further moderation? What quality of exam do we have right now?
Where is the transparency we are led to believe we are due as members of the itSMF - the primary marketing channel? What I can say here is that our representatives - the CSME (US) and ISEB have been brilliant in trying to represent our concerns. Yes, a 'major review' is scheduled for Q1 2008, meanwhile we continue to operate at the 'sharp end of the ITIL spear', putting our individual reputations on the line. Simple questions:
What happens in 2008, a quiet withdrawal of exam papers/questions and a new round of 'crash tests'?
What if its actually admitted there were mistakes made - what credibility then for the credential?
Will v2 Foundation get another lease of life as the real 'Foundation', with the v3 Foundation having to morph into some intermediary credential as part of diploma frequent flyer club scheme?
What compensation if any will be offered to the 20-40% who may have failed the exam in the interim?
Anyway, back to the basic math. Not enough exam questions leaves us trainers in an inwinnable position of guessing where the exam emphasis will be - and a temptation of returning to the suspect habit of v2 - coaching to the exam - definitely NOT in the best interests of the student unless the certificate is all we are focused on.If that were not enough, the strange taste possibly being left in candidate mouths right now severely impacts follow-on business in the form of lost consulting, training and book upsell opportunity.
If the recent itSMF USA Conference chatter was anything to go by - we are not alone in our concern for this examination... trouble is - to be forthright directly affects our own business - ouch - conflict of interest anyone - or are we to claim 'caveat emptor" (buyer beware) once again!
Readers wishing to read more about the application of Bloom's taxonomy to ITIL may like to check out my recent article on ITSMWatch
Made in New Zealand 
Comments
Post-Conference gloom!
I have attended several UK ITSMF conferences over the years, and have always enjoyed them. This year, however, I came away quite depressed. Just some of the reasons -
1. Nowhere on the agenda was there an opportunity to have a critical analysis of current developments (V3, APMG etc.) We had the usual, bland "This is how we implemented ITIL successfully at NNN Ltd". The V3 sessions were to tell people how wonderful it was. We used to have a debate at conference - if ever there was a year when a debate was required, this was it, yet there was no debate scheduled.
2. I was told by someone (well known and respected in the Service Management world) that he had wanted to do a slightly tongue-in-cheek presentation to be called "Oh NO! What have they done to ITIL!", but was turned down.
3. Perhaps my reading of this site over the last year has made me more sensitive to it than previously, but itsmf just seemed like a cheer-leader for ITIL, with no representative role for its members. Other frameworks are only recognised as support for ITIL, not as valuable in themselves.
4. Every time I spoke to other trainers, they had major worries too, but this was not reflected in the conference.
5. Many trainers were considering getting out of the ITIL training market altogether, moving into other areas. Even those who did not go that far, said they were not enjoying their work much at the moment.
6. At the session about the new qualification scheme, we were told that "many people felt that the V2 Foundation was dangerous, as people came out of it thinking they could implement ITIL processes", whereas V3 Foundation does not - as it is only an overview, and you have to do other courses before attempting implementation. When I asked who were these people who regarded V2 Foundation as dangerous, there was no response, although several members of the audience sought me out later to say they agreed with me that it showed an incredible arrogance that people can't be trusted unless we indoctrinate them with several courses first. Most people do not do the Managers course, and many very successful implementations have been done with a Foundation qualification and the red and blue books!
Other comments at the end of that session were from people who said they came to find out what training they should be sending their staff on, and had ended up even more confused than they started - delayed dates for release of qualifications, distrust of complex multiple choice etc. etc.
So, although it is a great networking experience still, how much am I actually gaining other than that from attending? The longer I stayed, the more depressed I got - so I left early, armed with information about the new EXIN ISO/IEC 20000 exams- maybe that is where the future lies!
Liz Gallacher
Freelance Trainer and Consultant
halved the size of the engines in all their cars
Hi Liz
Although I share your views, I have in the past pointed out the danger of newly-sheep-dipped Foundations graduates heading off to do ITIL. They are as dangerous as newly minted MBAs thinking they can run a company, more so because MBAs do actually vget a couple of years of intensive education.
So I think it is in fact true to say that a weaker V3 Foundations might reduce the risk. However this makes as much sense as Toyota declaring that they've halved the size of the engines in all their cars in order to reduce the road toll.
Courses+Experience is what
Courses+Experience is what is required.
I would absolutely agree that a Foundation by itself is not enough to implement - but training courses alone will not be enough, even if you attend more. It is the combination of the training course and the industry experience which makes this possible. I hate it when a customer sends new recruits, with little or no IT knowledge or experience on a course - the best courses are where people recognise the issues which ITIL addresses as issues they experience in their daily work.
I worry that ITIL qualifications will become part of graduate induction programmes - and multiple choice, however complex, benefits book-learning over experience. Written exams (such as Practitioner and Manager levels in V2) quickly showed who really understood and could apply their knowledge
Liz Gallacher
Freelance Trainer and Consultant
Agreement
The last few posts have been dwelling on my mind all day. I didn't get to this year's conference because I was busy doing real life service management*, but last year there were already rumours of content control. I also only tend to go every other year because of the parade of "we've just started doing ITIL and we are going to do all these wonderful things" type sessions. Funny how few of them come back four years later to tell you what actually happens.
Being one of those people who bridges the itSMF/ISACA gap I know which body I believe is more professional in every way, and which is listened to in more boardrooms.
Anyway, I digress. I've also probably blown any chance of speaking at another itSMF event, but that seems to happen to me every four years or so as a matter of routine.
I don't think it is the V2 Foundation course per se which is dangerous, but a certain type of delegate. There are technicians who come on it because they've been sent. Some good might rub off on them, when we come to their specialist area they might take away a few good ideas, and there a few who go through a personal revelation. There are good managers who come along because they know they are missing something. They are the challenging but fun students and at the end of the day they both take away some quick wins from the Foundation, but also realise that they need to know more. The dangerous delegate is the pointy haired boss, who either thinks they already know everything, or think they've grasped a concept because their brilliant trainer can sum it using four bullet points and a couple of anecdotes.
The good managers will understand where ITIL v3 is trying to go, but might be more challenging than before. The techie people are going to be lost, and the pointy haired bosses are going to have a field day and make their staff's life h**l when they get back supercharged with concepts and theories but with no realisation the reason their IT department is failing is because they are bad managers. That lies at the heart of it. As trainers (ex already in my case) we can engender step change in technical staff, but we can't do the same for bad managers.
As for teaching ITIL to graduate starters - why on earth not, I would rather they knew best practice and took it for granted than them dismissing ITSM as unimportant because it wasn't taught as part of their MBA. I was lucky enough to get on an audit training programme as a young graduate, after doing a couple of years at the coal face, and I still think it was the best thing that evcer happened to me, being exposed to management techniques and theories whilst still having to be rooted in the real world because you saw so much of how an organisation really works.
I've said this before, I'm sure, but I feel those running the ITIL show have forgotten that it isn't about massaging their egos. It is about the people who need to understand how to deliver effective IT to an increasingly sceptical business.
James
* It is so, so tempting to mention some names who might benefit from doing this, but it would be a cheap shot.
I agree with every word you
I agree with every word you say, James - you will be a loss to the training world ;~).
And Ian - I agree 100% with your definition of what changes are required to the V3 exams!!
Liz Gallacher
Freelance Trainer and Consultant
Ex-trainer
Liz,
Thanks, I've actually been an ex-trainer for a number of years, because of returning to real world service management, and then joing a big name consultancy that didn't believe in ITIL training (My bad) I was actually thinking about returning to the training fold in the next year or so, because I think the mid term market belongs to those of us who have been there and done it, and can highlight what matters in ITIL 3.
Oh no, not again!
Oh no, not again!
Finger trouble
Apologies for a double posting. Finger trouble ;-)
We need alignment
Liz
What is needed are sample exams that represent what candidates will encounter in the real exam. We also need these exams to consist of more questions, and to have emphasis in specific areas, for example a percentage of questions for Strategy, perhaps more for the subject areas of Service Level Management. This would allow us to teach across the board to an even and consistent level of knowledge, or emphasize certain topics deemed more relevant to someone taking the Foundation class.
At the moment the syllabus is flat, recommends a training company NOT follow the order or content flow it is described in, and basically make it a crap shoot for us all by an imbalance of questions and just not enough questions, seemingly forcing the examiners hand to plant a few that require the candidate to recall a diagram layout!
The examiner panel is at fault and APMG must reshuffle matters and include folks who are skilled in designing exams - PhDs of Learning. I for one would not put anyone who designed a V2 Foundation exam on the panel - eleven years of dodgy exams is enough proof for me.
Worst case they should buy a book that folks used to study for school exams or copy something from another industry that is regarded as working. As it stands the ATOs confidence is rocky and I am afraid it will not get any better if the Manager Bridge sample exam is anything to go by. The lack of an answer key explanation is a killer....
Has anyone noticed that the Foundation syllabus/exam is setup to be delivered online...... I feel the days of an ATO being the primary source for this training are numbered...
Ian M. Clayton, ITIL Service Manager, CSMP, ITSM Master
itSMF USA Lifetime Award Recipient 2005
Practitioners are the TRUE Masters - Part 2
Having read through the 'Final' ITIL v3 Qualification Scheme again and reading between the lines its now making more sense. Note my rough figures were wrong, because I assumed the full bredth of the Lifecycle was required.
Take a close look at the points for Practitioner and you'll see that you can proceed to v3 Managers Bridge after doing not ALL but MOST processes via v2 Practitioner courses. In the case of Clustered Practitioners this is 3 of the 4 available (3 x 3.5) + v2 Foundation (1.5) = 12 points. For single process Practitioners this is 6 (6 x 2)of them + v2 Foundation (1.5) = 13.5 points, which covers again MOST of the bredth of v2.
The role of Managing Across the Lifecycle then seems to be to add the overall 'managerial' aspects of introducing ITSM in an organisation, which is arguably the piece of training that Manager's have but Practitioners miss out on via the v2 options currently available, despite going deeper into the actual processes.
So whilst I still think that for Practitioners there will be some duplication in doing BOTH the Manager's Bridge AND Managing Through the Lifecycle, it does make more sense. Most Practitioners will currently only be through some of the v2 Practitioner courses and so can quickly complete some more v2 courses and then go onto v3 Manager's Bridge fairly quickly.
Practitioners are the TRUE Masters
Has anyone read the so called 'Final' ITIL v3 Qualification Scheme from the APMG web site ?
Looks like the scheme has changed again in a big way, especially for Practitioners. Sharon Taylor presented not long ago at the itSMF USA Conference. The 'map' for qualifications then showed a 'v3 Practitioner Bridge', 1 box. The recent APMG statement also stated that the Practitioner Bridge would be 'very similar' to the Manager's Bridge. Well the final Qualification scheme released by APMG just days ago shows a very different Practitioners path that now requires Practitioners to complete BOTH the Manager's Bridge AND (yes AND not OR !) the Managing Through the Lifecycle !
Can someone please attempt to explain the logic behind this ? Or explain why candidates completing the same ATO training, and passing the same exam end up with 2 different results ie. v2 Manager's Cert holder goes straight to ITIL Expert, but any v2 Practitioner has to then do Managing Through the Lifecycle, which conceptually covers and tests very similar material !
From my rough calculations this is the relative effort:
1 - Manager's Cert holder has spent approx. 20 days (3 Foundation, 12 Managers and 5 Mangers Bridge) of training
2 - say a Clustered Practitioner has spent approx. 33 days (3 Foundation, 4 x 5 Clustered Practitioner, 5 Managers Bridge and 5 Managing through the Lifecycle)
3 - Starting v3 from scratch you spend approx. 24/23 days (3 Foundation, 16/15 depending on Capability or Lifecycle modules and 5 Managing through the Lifecycle)
Is it just me or is 33 days alot more than 20 - 24 ? I find it ironic that v2 Clustered Practitioners are being unfairly treated, when the clustering approach is the core of the v3 scheme.
What do people think ?
Are all ITIL V3 Foundation Class Materials now out of date????
Mike
Its a challenge for sure - I'll be issuing a detailed analysis of this and the quality of the Manager Bridge sample and real examinations by year's end. Very useful insights - thank you. Just a quick beating of my chest here, if you take a moment to check page 6 of the published scheme, it lists 35 'subject areas', these seem to replace the previous 20+ 'processes' and largely invalidate or at least bring into suspicion any course materials already submitted to the examination institutes!
Its not a coincidence my earlier blog at this site arrived at a similar number.
Its also ironic that the scheme now uses the subject area term, more in line with the knowledge area term used within the ANSI approved service management qualification scheme.
This is actually a very good thing - we now need to wait for the Examiner Panel to realize that you also need to define 'knowledge domains' - such as Strategy, Design etc, and then determine what percentage of questions are asked in each domain and subject area......
As for the whole Practitioner debacle.... its clear to me that Managers are getting preferential treatment, perhaps to establish an early V3 'landing party' to act as the much needed marketing and sales force!!!!!
Watch out everyone - we have a 'major review' of the ITIL Foundation exams and syllabus due January 2008..... whatever next?
Ian M. Clayton, ITIL Service Manager, CSMP, ITSM Master
itSMF USA Lifetime Award Recipient 2005
v3 Foundation - Let it settle
Ian,
I agree totally, the flux in the v3 Foundation is just too great at the moment for my organisation to even look at it. As you've pointed out, the changes expected in January 2008 for the v3 Foundation look like being radical. I also agree with your general point that despite the 'flux' things are improving and heading in a better direction.
See my updated comments in Part 2 of Practitioners are the TRUE Masters, but I still agree that Managers are getting preferential treatment, as it is still the fastest, cheapest and I would argue 'easiest' path to ITIL v3 Expert/Diploma. But as you've pointed out, there could be other valid reasosns for this and one path or the other has to be the 'easiest'.
Now that the scheme is sort of 'finalised' I would expect a rush on v2 Managers Certificate courses, because if I was at the beginning or having just done v2 Foundation I would go down the Manager's path without hesitation before it starts to become unavailable near the end of 2008.
How is V3 Foundation being marketed?
Just a question for any other ATOs out there. With the current state of the V3 syllabus and exams, how are you describing the general outcomes of your V3 Foundation course? I just got back from my 6th or 7th V3 Foundation course (16 participants) and I continue to find that with the pace of the new course their initial expectations are not really being met. It is only after the first morning that they actually understand that for the most part, most people will achieve: a basic understanding about the benefits of ITSM, an ability to speak a common language describing ITIL processes and concepts, Foundation certification and perhaps a broadened perspective about the role of IT in the context of their organisation.
Although in many cases, the participants on the course have no information given to them other than the direction to attend, they really need to have been communicated what they should expect to achieve. While our Sales department does a fantastic job, I'm concerned the information they work by may be setting the benchmark too high.
So how are you dealing with this issue and how much have you changed since V3?