I have long advocated dumping all of the detailed questions in Level 1 questionnaires and, instead using one question, with a follow-on:

Would you recommend this course to a colleague? Why or why not?

This has, in my experience, produced more actionable data than any 'smile sheet' I have ever worked with. It may be harder to aggregate than all those multiple choices and Lickerts, but it is quick to answer and really gets to the heart of whether the training was considered useful (without the annoying self-reference worries).

I know this approach is being used in many large companies (Ford, for example, has used this for years). Does anyone on this list know of specific other companies using it? Does anyone have any references to this approach in the evaluation literature?

Your input would be much appreciated.

Godfrey Parkin

Godfrey,

I agree and have been using the same question
for a very long time. Further, I would argue that all
levels be abandoned.They communicate very little to
management at best.

Nero Wolfe

I disagree. I think the idea of standardizing evaluations (either by asking one question or asking multiple questions or whatever) misses the point.

We first (before we decide what questions to ask or even if we ask questions of participants) need to ask the question--why are we evaluating? Or, what is the purpose of the data?

For instance, if the course will only be offered one time and the instructor will only be here that one time, what do we care if people recommend the course or not? Or, if the course is required by law or company policy? Do we really care if people would "recommend" the employee orientation.

I don't have a problem with the question itself--I think it's a good one. But I think ALL evaluation has to start by determining the purpose of the evaluation. If we're not seeking to improve the course (b/c the formative evaluation period has passed), if the course isn't voluntary, or if it isn't going to be repeated, we don't need to ask what people thought of the materials or instructor or the course overall. In fact, I can think of some instances were reaction will be negative (people don't want to be there or don't want to hear the message in the course or meeting) yet I was happy with the product/course/session.

There are obviously other times when reaction is critical. That kind of level 1 data can be very useful in formative evaluation.

I agree that evaluation has to start with determining what it is we intend to do with the resulting data. I suspect that 95% of the data gathered is not actionable at all, and "nice to know" data has no value, other than perhaps ego-stroking.

One of the most common problems with multi-question evaluations (particularly those using closed-ended responses) is that it is assumed that the issue being asked about is important to the learner. They also tend to "lead" learners to focus on only those issues asked about and do not encourage (or, often, allow) learners to communicate what's on their mind.

When people are under time pressure to get out of a classroom or shut down a browser and move on, having them go through a dozen (to them) tedious and irrelevant questions gives you bad data -- and by the time they get to the really useful last question ("other comments?") they are not predisposed to be very forthcoming.

As for formative evaluation, I try to recommend piloting and testing of courses before they "go live" -- and post-course surveys of only a sample of learners for ongoing tracking and investigation of issues raised by the end-of-course question.

Godfrey,

I do believe that your one question would yield a good deal of value. Another question that I've received unexpected information and learning from is the question,

"What will you take from this class and use tomorrow to improve your performance or impact your work group?"

The answers that I receive are often surprising. When we develop or deliver a course we can get absorb by so many things. The one important take away for students is often something that I/we may not have considered vital.

It also helps because it tells us about the challenges that they are facing in the workplace and how we might increase the ability of our training to impact results better and faster.

Because of the input I've received (at the end) I decided to begin classes with the question, "What is the one thing that I could teach you or tell you (about topic) that would benefit you the most on your return to work?"


Valarie A. Washington
Knowledge Broker
Think 6 Results
http://www.think6results.com

Results Matter!