evaluation examples

I'm looking for examples of learning evaluation in practice - both good and bad. I'd welcome anything from a bank of case studies to any small anecdote anyone is willing to share. And I'm interested in any context of learning and development implementation.

Can anyone help? Please use the comments option, below, to post links, or email me via the learnforever website.

Thanks.

Reasoned evaluation

A client asked me today about approaches to evaluation, and this got me thinking about how advocates of different approaches get very dogmatic about what works and what doesn’t work.

Kaliym Islam, in Developing and Measuring Training the Six Sigma Way, says “none of the four levels in the Kirkpatrick model capture business feedback or business reaction to the training product”.

Jack Phillips, the Return on Investment guru, argues Kirkpatrick’s four levels are at best inadequate and need ROI as a fifth level.

Paul Kearns, author of Evaluating the ROI from Learning, goes further and describes Kirkpatrick as just “wrong”, but then he draws the same conclusion about ROI as the fifth level – “wrong again”. (See http://www.evidencebasedhr.com/?p=275)

And in The Value of Learning, Valerie Anderson advocates moving away from ROI to ROE, Return on Expectations.

It seems no sooner is a new approach in use than someone is rubbishing it in print. I don’t wish to join this chorus, and I believe each of the commentators attributed above has something important to offer. But do we have to dismiss the Kirkpatrick levels to embrace ROI? Or abandon ROI for ROE? Is it possible Six Sigma has its place in certain contexts but Kirkpatrick still has lessons to offer?

I think we need to recognise that there’s a lot of useful thinking about learning evaluation, and a variety of approaches, even if some of them are contradictory or even mutually exclusive. Organisations need specialist help to consider their options and identify the best way forward for them.

Adventures in Moodle

Picking up from some of my previous blog posts, it’s now over a year since my organisation established an online learning environment, based on a Moodle platform. It’s been an interesting journey.

Initially I expected to get three things:
1. A learner management system, enabling us to store information about learners and generate reports.
2. A learning content management system, to author, edit and launch courses and other forms of online content.
3. A virtual classroom tool, to enable meetings of learners in multiple locations.

We don’t have the first thing – Moodle seems pretty limited in this capability, and I wonder whether others have found solutions to this? We’ve learned to experiment with plug-ins, trying the Book Module, before settling on Xerte as our default course authoring tool; we’ve plugged in the OU blog, following my amazed discovery that the native Moodle blog tool didn’t allow comments (although Moodle 2.0 apparently allows comments on all features, so we probably just needed to upgrade the whole platform); and we’ve opted for a plug-in for the virtual classroom functionality (hang on a minute – getting ahead of myself). But I’m not convinced any plug-in is going to help us with reporting – unless anyone can tell me different?

We’ve got the second thing, although we didn’t understand at the beginning that Moodle is designed essentially to support tutor-led cohorts of learners following a blended learning model, and any other capability is really just stuck on as an option.

And we don’t have the third thing. We plugged in Dimdim, but since their acquisition, they no longer offer an open source solution (our deal runs out at the end of August) and we need to identify an alternative. WiZiQ looked attractive, but we seem to get more features from Megameeting, with which we’ve already experimented. I’d be interested to hear from others who have plugged in different solutions.

What sort of adventures have you had with Moodle?

Whither CMI?

Recently I questioned the relevance of the CIPD to learning and development professionals who are not part of a corporate HR department. My conclusion was that, while we have been relatively neglected, the organisation is still well worth membership.

To be consistent, I should comment on the other body of which I am a Fellow, the Chartered Management Institute (CMI).

On the face of it, the critique I offered of the CIPD (robustly defended by a CIPD representative in the comments to that blog post) applies even more so to CMI. And yet it’s all about expectations – I don’t expect the same close attention to my own professional specialism from a body as broad-based as CMI.

Bizarrely, given that it is open to all managers, CMI has a smaller membership than CIPD – about 80,000 members compared to over 120,000. There may be many reasons for this, but perhaps close attention to professional specialisms is part of the explanation.

Under Ruth Spellman’s leadership, CMI has been modernising and improving. There seem to be greater efforts to develop branches, and I can see improvements to publications. Management Today has always been a top class magazine – much more readable and professionally produced than any other business magazine in the UK. Now the other main CMI publication, Professional Manager, has stopped looking so antiquated and is actually worth reading.

But there has to be more to CMI than magazines, and I see little branch or networking activity that’s of benefit to me. I wonder if others have similar experiences? (Sits back and awaits response from CMI staff scanning the web for criticism to combat…)

Commercial break

There will now be a short commercial break...

I have been impressed by a free booklet from Reed Learning, The Little Book of the Future: a guide to collaborative learning. Use this link to request your own copy.

It’s not really free, of course, as you exchange your contact details and allow Reed to pursue you with other offers, but so far I reckon it’s been worth it. The booklet includes short articles on leading-edge digital tools for learning, along with a bit of future-gazing. Among the latter commentaries, I particularly enjoyed Debbie Carter’s vision of phones printed on wrists and video screens in contact lenses – it chimed with some of my own predictions in the final chapter of my last book, Delivering E-Learning.

I’ve been getting agitated to mention Springest, which describes itself as an independent comparison website for learning and development programmes. Originating in Holland, it offers listings for all sorts of courses, organised in subject categories, along with reviews, links to other social media, and features for learners to store their own favourites. Worth a look, I'd suggest.

OK, Martijn?

While I'm mentioning other people's products I may as well mention my own.

My new book, 101 Learning and Development Tools, is not due for publication for another three months, but it’s available to pre-order on Amazon now.

Comments, as always, welcome.

Manifesto

As we have just held elections in Scotland and the UK, it seems apposite to restate what this blog is about. I want to assert some clearer themes, and encourage readers to subscribe to follow, and followers to invite others to follow.

The key themes are my favourite issues in learning and development, including:

e-learning and blended learning
work-based learning (and volunteering-based learning)
talent management and development
corporate universities or academies
resource-based learning and learning centres
leadership and management development
knowledge management

...plus whatever else may be topical in the world of occupational learning.

Please let me know if there are other, related, themes you would like to see addressed.

Are you my friend?

Driving to work the other day, I heard an item on Radio Scotland about how many online “friends” we can realistically maintain. I’m not sure how this became a topical item for commuter news, as I have investigated further and discovered Robin Dunbar identified the number – it’s 150 – way back in 2008. (See this New Scientist link.)

The point is that those who have hundreds of “friends” on Facebook, or other social networking sites, are kidding themselves, as they simply can’t maintain sufficient contact with so many people to justify describing them as friends.

As one of the dwindling refuseniks who still haven’t joined Facebook or Twitter, I gleaned some foolish comfort from this stat, but of course I have more than 150 people in my LinkedIn network, including many of the followers of this blog, so am I one of the deluded?

I don’t think so. One of the things I like about LinkedIn is that it makes no pretence that your network of contacts are friends. Its clear purpose is networking for work-related reasons, and it holds onto that objectivity.

I’ve just invited all my LinkedIn contacts to follow this blog, as a contribution to my previously posted intention to expand this community and generate more debate. But I won’t be offended by those who don’t sign up to follow, and simply continue to read when it suits them – that’s their choice.