A Tale of Two Donalds

It’s a week since I posted a comment on Donald Clark’s blog, but comments there require moderation, and so far, at least, Donald hasn’t published my comment. As I mentioned it in my last post here, this is what I said:

Insightful critique, Donald - I found myself nodding enthusiastically at many of your points. But I find your conclusions a tad harsh. The main problem with Kirkpatrick is too many people regard it as "the only game in town", when it is one (just one) of a range of models and tools that have their place in different situations.

Brinkerhoff's Success Case Method (for example) is brilliant, but I wouldn't recommend it for every situation. The same with Kirkpatrick. I don't think we have to dismiss it altogether, just put it in the toolbox alongside all the other options. And I'm not convinced we need a single new all-embracing model to replace it (not sure you are advocating that, but that's one reading of an "overhaul").

Further discussion of situational approaches here: http://www.airthrey.com/papers.htm

On checking back to Donald’s blog, I found he posted an even more devastating critique of Kirkpatrick, back in 2006 – Donald Talks Bollocks! Perhaps Donald Clark’s views have moderated a little since then? I think the problem is too many people are looking at evaluation through the prism of Kirkpatrick, and perhaps that means they need the jolt of this sort of polemic.

Further thoughts:

1. "Happy sheets" shouldn’t be primarily to test how happy learners were: they are a simple quality test, an opportunity to check that the basics of a learning initiative are working – e.g., the trainer is effective, the course materials are relevant and comprehensible, etc. I realise this example is specific to a training course – arguably the sort of application where Kirkpatrick works best.

2. I don’t agree that Kirkpatrick asks “all the wrong questions”, but I think there are plenty of other questions, and many situations where other questions are more pertinent.
3. I don’t want to seem like an apologist for Kirkpatrick, but I think his model still has its place, as long as it’s deliberately selected as the most relevant model, as an informed choice over other models. It would rarely be my first preference, but I can still see it can work for some of the people some of the time.

Theory and practice

Donald Clark’s Plan B blog is excellent, and I commend it to you.  His recent endeavour, to identify the 50 most influential theorists in the learning and development canon, and blog about each of them (in 50 days!) is unparalleled, and deserves to be published to a much wider audience.  Donald eventually blogged about 51, with 2 deletions and 3 additions from the original list, following lively debate.  The 51 included not just the usual suspects like Bloom, Kolb and Maslow, but the likes of Locke, Marx and Freud, all the way back to Socrates, Confucius and Jesus.  A veritable tour de force.

His last post, on Kirkpatrick, caught my eye.  The only evaluation theorist to make it onto Donald’s list.  And for good reason, as whatever the merits of Phillips, Brinkerhoff, Basarab and others, nobody else has enjoyed the sustained influence of Kirkpatrick and his “family business”, as Donald recently described it.

My response to Donald is on his blog, but one other comment he made rang very true for me:

"Evaluation should be done externally. The rewards to internal evaluators for producing a favourable evaluation report vastly outweigh the rewards for producing an unfavourable report”.

I have long held this to be the case, and it was my primary motivation in setting up the independent learning evaluation consultancy, Airthrey.

The MBA

According to Wikipedia, the Master of Business Administration (MBA) degree is “designed to introduce students to the various areas of business such as accounting, finance, marketing, human resources, operations management, etc”.  This is one design, suitable for new graduates seeking entry to a management career, but equally common is the design to validate management experience with credible accreditation, and this was my own motivation in completing, at age 40, a part-time MBA, spread over a number of years.

The MBA remains the most prestigious business/management qualification worldwide, and a focus for a lot of media attention, but its purpose and meaning remains confused.  I subscribe to Management Today, and this excellent magazine seems to carry an MBA feature in every other issue, to no greater enlightenment (it’s really just advertising).

The latest issue of Professional Manager magazine carries one of the recurring pieces questioning the value of the qualification, and as usual, there seems precious little reason for employers to fund MBAs, as the whole point seems to be for individuals to “invest in yourself”.  The survey evidence I see quoted most frequently is about salary increases post-MBA qualification, and the Professional Manager article doesn’t disappoint.  Sourced from the MBAs 2010 career survey, it claims average worldwide salary increases for MBA graduates of 33% immediately post-MBA, 92% 3-5 years post-MBA, and 151% 6-10 years post-MBA.

I wonder to what extent these figures are inflated by young graduates with little or no income suddenly getting their first big management job.  My own (i.e., my personal) experience certainly doesn’t chime with this, as immediately post-MBA my earnings actually fell, then rose by about 50% (from the baseline) in the 3-5 years range, then fell back again.

So what is the point of an MBA?  I’d be interested to hear a case for this most generic of qualifications, in preference to more specific and more targeted higher degree programmes.  In my own case, I’d probably have been better doing a Masters in management consulting, had such an option been apparent to me.  Is the widespread recognition of MBAs enough to outweigh this sort of consideration?  Especially when this recognition is often misplaced?  Do MBAs actually deliver what they promise?

Left nostril learning?

One of the key questions raised by the CIPD 2012 Learning and Talent Development Survey has been why so many favoured ideas in learning and development are so old.  Among those cited have been Belbin’s Team Roles (1981), Honey & Mumford’s Learning Styles (1970s) and the Myers-Briggs Type Indicator (1962).  I could add Kirkpatrick’s four levels of learning evaluation, which dates from even earlier (1950s).

But better questions might be why so few theories stand the test of time, or why more recent theories are not so well regarded.  The evidence suggests there are few popular theories in learning and development (L&D) that truly bear close scrutiny – at best many are useful constructs for opening debate about an issue, but no more.

Honey and Mumford’s learning styles theory, along with the competing theory of Colin Rose, has been widely discredited, at least in academic circles (see the range of sources here), yet remains popular.

Neuro Linguistic Programming is at best deeply flawed, if not downright fraudulent, at least in the opinion of respected commentator Donald Clark, and yet still has many vociferous adherents among L&D professionals.

And left-brain versus right-brain hocus pocus (for that’s all it is, really) is still given credence.  I can’t work out whether James Brockett was kidding in his People Management blog of 13 April, when he considered left ear versus right ear (was his tongue in his cheek, at least?)  What next – creative use of the left nostril?

Five ostriches

Why is learning evaluation so widely ignored? Too many people are sticking their heads in the sand, and I think there are five types.

It’s too hard!
Many people think it’s too difficult to measure the outcomes of learning and their contribution to business performance, but as Peter Drucker said “if you can’t measure it, you can’t manage it”. I think this fear derives mainly from the unfamiliarity of HR and L&D professionals with research methods and statistical analysis – but this specialised help is readily available. A more sophisticated version of this argument is that people development is hard to measure because it’s behavioural and doesn’t lend itself to quantification. Yet qualitative data is also measurable, with just a little more thought. This really shouldn’t be an excuse – if it’s too hard, get help!

It’s too expensive!
This is both an extreme generalisation and a myth, akin to claims that all training is expensive. It can be, but equally it can be managed economically. Online surveys can be distributed cheaply; telephone interviews are usually much cheaper than face-to-face, and just as useful; judicious sampling can significantly reduce costs; and regular practice makes evaluation quicker and easier. The alternative is much more expensive – continuously repeating training that doesn’t pay. The reality is that lack of evaluation is wasteful, and not evaluating is too expensive.

Nobody cares!
They should. At some point, they will. And when they do, there had better be a case for L&D ready in response. Otherwise it’ll be cut. Just because senior management may have a temporary blind spot doesn’t justify this lazy approach. All managers need foresight and planning, and if HR managers aren’t measuring the cost, value, and business contribution of L&D, they’re not doing their job properly.

All learning is valuable!
Of course it is. But so are many things organisations do, and ultimately learning is competing for scarce resources. It’s not enough to be A Good Thing, it has to be demonstrably more valuable than alternative investments. Certainly, there can be debate about how to measure value, and economic/financial considerations may not be the only ones, perhaps not even the most important, but measures still need to be made.

Mañana…
With apologies to Hispanic readers (a joke in the Highlands of Scotland is that there’s no word in the Gaelic language that conveys the same desperate sense of urgency). The problem is that evaluation is rarely seen as a pressing priority, and no matter how important it is judged to be, may continue to be put off till “tomorrow” indefinitely. This is a dangerous way to operate, and more enlightened organisations will weigh urgency and importance to determine when to allocate time and resources to evaluation. The antidote to mañana is to allocate a proportion of L&D budgets (both time and money) to evaluation – 10% to 15% is the usual amount – and ensure that it is used.

Do you recognise behaviour from your organisation as any of these five types? Are you an ostrich with your head in the sand?

Evaluation Debate in People Management

Bill Parsons, CIPD Vice-President, raised an important debate in the most recent issue of People Management (27 March). Under the heading ‘Learning to Leap’, Bill observed that learning and development often fails to correlate with business metrics and contribute to business outcomes.

Unfortunately, the debate hasn’t really taken off. As I write, mine is the only comment (5 April) in reply to Bill’s piece. Keith Ledingham has just written a letter in response, ‘ROI is essential to measure HR’s contribution’ (17 April).

The problem is, the only people who seem interested in debating this are from the supply side: Keith is one example; my recent articles on evaluation at TrainingZone have attracted comments only from Paul Kearns and Stephen Gill, both consultants, despite thousands of “reads”; the overwhelming majority of followers of my evaluation account on Twitter are from the supply side.

It seems far too many HR and L&D professionals are ignoring learning evaluation. My next blog post will explore why this is.

Praise and thanks

My thanks are due to all who have praised my last-but-one book, Delivering E-Learning, in the three years since its publication. I have previously acknowledged Martyn Sloman’s review in Training Journal and Neil Archibald’s in HR Network Scotland magazine, as well as Richard Wright’s original review on amazon.co.uk, who have since added reviews from amazon.com, notably a five-star review by “Mid West Book Review”.

I would also like to thank John P Wilson for his review in Industrial and Commercial Training, Paul Justice for his review in e-learningnow.co.uk, Sebah Al-Ali for her review in Goodreads, and Christopher Pappas for naming it in his blog as one of his top five e-learning books for beginners.

The book seems to continue to attract interest, with a new version in Arabic due out this year, which I’m looking forward to.

Thanks, everyone!