evaluation examples

I'm looking for examples of learning evaluation in practice - both good and bad. I'd welcome anything from a bank of case studies to any small anecdote anyone is willing to share. And I'm interested in any context of learning and development implementation.

Can anyone help? Please use the comments option, below, to post links, or email me via the learnforever website.

Thanks.

Reasoned evaluation

A client asked me today about approaches to evaluation, and this got me thinking about how advocates of different approaches get very dogmatic about what works and what doesn’t work.

Kaliym Islam, in Developing and Measuring Training the Six Sigma Way, says “none of the four levels in the Kirkpatrick model capture business feedback or business reaction to the training product”.

Jack Phillips, the Return on Investment guru, argues Kirkpatrick’s four levels are at best inadequate and need ROI as a fifth level.

Paul Kearns, author of Evaluating the ROI from Learning, goes further and describes Kirkpatrick as just “wrong”, but then he draws the same conclusion about ROI as the fifth level – “wrong again”. (See http://www.evidencebasedhr.com/?p=275)

And in The Value of Learning, Valerie Anderson advocates moving away from ROI to ROE, Return on Expectations.

It seems no sooner is a new approach in use than someone is rubbishing it in print. I don’t wish to join this chorus, and I believe each of the commentators attributed above has something important to offer. But do we have to dismiss the Kirkpatrick levels to embrace ROI? Or abandon ROI for ROE? Is it possible Six Sigma has its place in certain contexts but Kirkpatrick still has lessons to offer?

I think we need to recognise that there’s a lot of useful thinking about learning evaluation, and a variety of approaches, even if some of them are contradictory or even mutually exclusive. Organisations need specialist help to consider their options and identify the best way forward for them.

Adventures in Moodle

Picking up from some of my previous blog posts, it’s now over a year since my organisation established an online learning environment, based on a Moodle platform. It’s been an interesting journey.

Initially I expected to get three things:
1. A learner management system, enabling us to store information about learners and generate reports.
2. A learning content management system, to author, edit and launch courses and other forms of online content.
3. A virtual classroom tool, to enable meetings of learners in multiple locations.

We don’t have the first thing – Moodle seems pretty limited in this capability, and I wonder whether others have found solutions to this? We’ve learned to experiment with plug-ins, trying the Book Module, before settling on Xerte as our default course authoring tool; we’ve plugged in the OU blog, following my amazed discovery that the native Moodle blog tool didn’t allow comments (although Moodle 2.0 apparently allows comments on all features, so we probably just needed to upgrade the whole platform); and we’ve opted for a plug-in for the virtual classroom functionality (hang on a minute – getting ahead of myself). But I’m not convinced any plug-in is going to help us with reporting – unless anyone can tell me different?

We’ve got the second thing, although we didn’t understand at the beginning that Moodle is designed essentially to support tutor-led cohorts of learners following a blended learning model, and any other capability is really just stuck on as an option.

And we don’t have the third thing. We plugged in Dimdim, but since their acquisition, they no longer offer an open source solution (our deal runs out at the end of August) and we need to identify an alternative. WiZiQ looked attractive, but we seem to get more features from Megameeting, with which we’ve already experimented. I’d be interested to hear from others who have plugged in different solutions.

What sort of adventures have you had with Moodle?