My new business partner

Dr Alasdair Rutherford is a research economist I first met two years ago, when we were both working for Volunteer Development Scotland; I was Learning & Development Manager, and Alasdair was Research & Development Manager. We collaborated on a number of learning evaluation projects, such as for the NHS in Scotland.

Alasdair has a string of impressive academic qualifications, including two first degrees (if that’s not an oxymoron), a study exchange in Germany, a Masters and a PhD, not to mention a sheaf of published papers. He is currently a Research Fellow at the University of Stirling.

We calculate that our respective knowledge and skill sets, one in learning and development, the other in research and development, complement each other perfectly for a business specialising in the evaluation of learning and development.

Alasdair’s interests and skills include econometrics, statistics, formative and summative evaluation, economic and social research, quantitative and qualitative research, data analysis, and database design and administration. I commend Alasdair to you, and I look forward to working with him in applying his extensive range of skills and methods to the learning activities of our corporate clients.

Three theses on learning evaluation

Not enough is happening. Not enough learning and development (L&D) is being evaluated, and when it is being done, it is not being done robustly, systematically, or effectively. As a rule of thumb, organisations should spend 10-15% of their L&D budget on evaluation, but how many do that? Evaluation is insufficiently understood and applied.

There is too much dogma. There are a great many models and tools for L&D evaluation, and just as many petty disagreements about what works and what doesn’t. ROI doesn’t negate Kirkpatrick, any more than ROE negates ROI, and yet many L&D professionals take a partisan view supporting a particular tool, when the truth is that every model and tool has its place – the trick is to be able to select the right one for the right situation.

It’s not just about costs, but value. Many organisations are cost aware, but hardly any are value aware. To appreciate the full impact of L&D, we need to recognise all the different kinds of value it adds, both quantitatively and qualitatively. Organisations need to become more conscious of value, and strive for total value add.

Learning evaluation solutions

Keen readers of this blog may have noticed a shift of emphasis in recent posts. I've apparently become much more interested in the evaluation of learning and development. In fact, this has been a career-long interest, and has always formed a part of my professional work. However, I have let the cat out of the bag with my latest LinkedIn update - "planning a new venture!"

I've written about learning evaluation for CIPD's HR Inform website, and a quarter of my new book, 101 Learning & Development Tools, is about evaluation. Now I'm setting up a new firm, with a partner, in the business incubator at the University of Stirling. We'll be offering learning evaluation solutions to corporate clients from next month, October 2011.

More to follow, and in the meantime, I'd be interested in any questions, comments, or especially invitations to discuss how we may be able to help your business.

Fee, Kenneth: 101 Learning & Development Tools

My new book, 101 Learning and Development Tools, will be published on 3 September by Kogan Page.

It's available for pre-order on Amazon now, and if you're visiting that site, why not check out my other books, at the Amazon author page for Kenneth Fee? Comments, reviews and 'like!' markers always welcome.

Judging by the release of my last book, if you order now, you'll probably get the book right away. I'm about to go on holiday, so I expect to see it on my return. And if I return to comments about the book on this blog, that would be even more welcome!

I hope you enjoy the book, and find it useful.

Lack of evaluation

I’m underwhelmed by the response to my last blog post.

I know many followers/readers of this blog are interested in e-learning, and I thought there might have been a lot of examples of attempts to evaluate e-learning, not least to show where it is more efficient than face-to-face learning – although I believe the focus is increasingly, and correctly, shifting from efficiency to effectiveness.

I’m planning a new venture in evaluation of learning and development, and I was hoping for more informal feedback via this route, aside from the more formal market research and business development I’m undertaking. I would still welcome any examples or anecdotes anyone is willing to share.

However, I wonder whether the lack of comments (or any other responses) is indicative of a lack of examples? This is part of my thesis about L&D evaluation: I believe there is too little of it being undertaken at the moment; I believe there is too little understanding of evaluation and its importance; I don’t know of any consultancies or service providers specialising in it; and I know there has been no new book of any significance published on the subject in nearly a decade. I’d be interested to hear any contradiction of this thesis (or any more supporting evidence).

As always, comments would be very welcome!

evaluation examples

I'm looking for examples of learning evaluation in practice - both good and bad. I'd welcome anything from a bank of case studies to any small anecdote anyone is willing to share. And I'm interested in any context of learning and development implementation.

Can anyone help? Please use the comments option, below, to post links, or email me via the learnforever website.

Thanks.

Reasoned evaluation

A client asked me today about approaches to evaluation, and this got me thinking about how advocates of different approaches get very dogmatic about what works and what doesn’t work.

Kaliym Islam, in Developing and Measuring Training the Six Sigma Way, says “none of the four levels in the Kirkpatrick model capture business feedback or business reaction to the training product”.

Jack Phillips, the Return on Investment guru, argues Kirkpatrick’s four levels are at best inadequate and need ROI as a fifth level.

Paul Kearns, author of Evaluating the ROI from Learning, goes further and describes Kirkpatrick as just “wrong”, but then he draws the same conclusion about ROI as the fifth level – “wrong again”. (See http://www.evidencebasedhr.com/?p=275)

And in The Value of Learning, Valerie Anderson advocates moving away from ROI to ROE, Return on Expectations.

It seems no sooner is a new approach in use than someone is rubbishing it in print. I don’t wish to join this chorus, and I believe each of the commentators attributed above has something important to offer. But do we have to dismiss the Kirkpatrick levels to embrace ROI? Or abandon ROI for ROE? Is it possible Six Sigma has its place in certain contexts but Kirkpatrick still has lessons to offer?

I think we need to recognise that there’s a lot of useful thinking about learning evaluation, and a variety of approaches, even if some of them are contradictory or even mutually exclusive. Organisations need specialist help to consider their options and identify the best way forward for them.