Learning evaluation solutions

Keen readers of this blog may have noticed a shift of emphasis in recent posts. I've apparently become much more interested in the evaluation of learning and development. In fact, this has been a career-long interest, and has always formed a part of my professional work. However, I have let the cat out of the bag with my latest LinkedIn update - "planning a new venture!"

I've written about learning evaluation for CIPD's HR Inform website, and a quarter of my new book, 101 Learning & Development Tools, is about evaluation. Now I'm setting up a new firm, with a partner, in the business incubator at the University of Stirling. We'll be offering learning evaluation solutions to corporate clients from next month, October 2011.

More to follow, and in the meantime, I'd be interested in any questions, comments, or especially invitations to discuss how we may be able to help your business.

Fee, Kenneth: 101 Learning & Development Tools

My new book, 101 Learning and Development Tools, will be published on 3 September by Kogan Page.

It's available for pre-order on Amazon now, and if you're visiting that site, why not check out my other books, at the Amazon author page for Kenneth Fee? Comments, reviews and 'like!' markers always welcome.

Judging by the release of my last book, if you order now, you'll probably get the book right away. I'm about to go on holiday, so I expect to see it on my return. And if I return to comments about the book on this blog, that would be even more welcome!

I hope you enjoy the book, and find it useful.

Lack of evaluation

I’m underwhelmed by the response to my last blog post.

I know many followers/readers of this blog are interested in e-learning, and I thought there might have been a lot of examples of attempts to evaluate e-learning, not least to show where it is more efficient than face-to-face learning – although I believe the focus is increasingly, and correctly, shifting from efficiency to effectiveness.

I’m planning a new venture in evaluation of learning and development, and I was hoping for more informal feedback via this route, aside from the more formal market research and business development I’m undertaking. I would still welcome any examples or anecdotes anyone is willing to share.

However, I wonder whether the lack of comments (or any other responses) is indicative of a lack of examples? This is part of my thesis about L&D evaluation: I believe there is too little of it being undertaken at the moment; I believe there is too little understanding of evaluation and its importance; I don’t know of any consultancies or service providers specialising in it; and I know there has been no new book of any significance published on the subject in nearly a decade. I’d be interested to hear any contradiction of this thesis (or any more supporting evidence).

As always, comments would be very welcome!

evaluation examples

I'm looking for examples of learning evaluation in practice - both good and bad. I'd welcome anything from a bank of case studies to any small anecdote anyone is willing to share. And I'm interested in any context of learning and development implementation.

Can anyone help? Please use the comments option, below, to post links, or email me via the learnforever website.

Thanks.

Reasoned evaluation

A client asked me today about approaches to evaluation, and this got me thinking about how advocates of different approaches get very dogmatic about what works and what doesn’t work.

Kaliym Islam, in Developing and Measuring Training the Six Sigma Way, says “none of the four levels in the Kirkpatrick model capture business feedback or business reaction to the training product”.

Jack Phillips, the Return on Investment guru, argues Kirkpatrick’s four levels are at best inadequate and need ROI as a fifth level.

Paul Kearns, author of Evaluating the ROI from Learning, goes further and describes Kirkpatrick as just “wrong”, but then he draws the same conclusion about ROI as the fifth level – “wrong again”. (See http://www.evidencebasedhr.com/?p=275)

And in The Value of Learning, Valerie Anderson advocates moving away from ROI to ROE, Return on Expectations.

It seems no sooner is a new approach in use than someone is rubbishing it in print. I don’t wish to join this chorus, and I believe each of the commentators attributed above has something important to offer. But do we have to dismiss the Kirkpatrick levels to embrace ROI? Or abandon ROI for ROE? Is it possible Six Sigma has its place in certain contexts but Kirkpatrick still has lessons to offer?

I think we need to recognise that there’s a lot of useful thinking about learning evaluation, and a variety of approaches, even if some of them are contradictory or even mutually exclusive. Organisations need specialist help to consider their options and identify the best way forward for them.

Adventures in Moodle

Picking up from some of my previous blog posts, it’s now over a year since my organisation established an online learning environment, based on a Moodle platform. It’s been an interesting journey.

Initially I expected to get three things:
1. A learner management system, enabling us to store information about learners and generate reports.
2. A learning content management system, to author, edit and launch courses and other forms of online content.
3. A virtual classroom tool, to enable meetings of learners in multiple locations.

We don’t have the first thing – Moodle seems pretty limited in this capability, and I wonder whether others have found solutions to this? We’ve learned to experiment with plug-ins, trying the Book Module, before settling on Xerte as our default course authoring tool; we’ve plugged in the OU blog, following my amazed discovery that the native Moodle blog tool didn’t allow comments (although Moodle 2.0 apparently allows comments on all features, so we probably just needed to upgrade the whole platform); and we’ve opted for a plug-in for the virtual classroom functionality (hang on a minute – getting ahead of myself). But I’m not convinced any plug-in is going to help us with reporting – unless anyone can tell me different?

We’ve got the second thing, although we didn’t understand at the beginning that Moodle is designed essentially to support tutor-led cohorts of learners following a blended learning model, and any other capability is really just stuck on as an option.

And we don’t have the third thing. We plugged in Dimdim, but since their acquisition, they no longer offer an open source solution (our deal runs out at the end of August) and we need to identify an alternative. WiZiQ looked attractive, but we seem to get more features from Megameeting, with which we’ve already experimented. I’d be interested to hear from others who have plugged in different solutions.

What sort of adventures have you had with Moodle?

Whither CMI?

Recently I questioned the relevance of the CIPD to learning and development professionals who are not part of a corporate HR department. My conclusion was that, while we have been relatively neglected, the organisation is still well worth membership.

To be consistent, I should comment on the other body of which I am a Fellow, the Chartered Management Institute (CMI).

On the face of it, the critique I offered of the CIPD (robustly defended by a CIPD representative in the comments to that blog post) applies even more so to CMI. And yet it’s all about expectations – I don’t expect the same close attention to my own professional specialism from a body as broad-based as CMI.

Bizarrely, given that it is open to all managers, CMI has a smaller membership than CIPD – about 80,000 members compared to over 120,000. There may be many reasons for this, but perhaps close attention to professional specialisms is part of the explanation.

Under Ruth Spellman’s leadership, CMI has been modernising and improving. There seem to be greater efforts to develop branches, and I can see improvements to publications. Management Today has always been a top class magazine – much more readable and professionally produced than any other business magazine in the UK. Now the other main CMI publication, Professional Manager, has stopped looking so antiquated and is actually worth reading.

But there has to be more to CMI than magazines, and I see little branch or networking activity that’s of benefit to me. I wonder if others have similar experiences? (Sits back and awaits response from CMI staff scanning the web for criticism to combat…)