I am hoping you could help me out. We have recently begun a new leadership program across the country and have had some low scores in Level 1 surveys specifiaclly around tarining effectivenss, and overalll program design and activities. The thing is we traditionally only look at the highest score as the goal. So if I rated a 4 out of five that would not count.

- How do you structure your scorecards regarding participant satisfaction? How do you define success for leadership training on this?
- What type of questions do you include on level 1's? Do you include stuff about learning activities, or just look to level 2 results to determine effectiveness in this area?
- What are the typical level 1 results for supervisor and manager level training programs (mean results, percentage of top rated scores, etc.)? Do results typically differ by audience, industry, and/or program content (i.e., hard skills vs. soft)?
- To what degree do the best training functions emphasize level 1 results relative to other metrics (learning, performance, etc.)?

MJP-

I am actually resubscribing to the effectiveness Level 1 because of socialization and social networks. I let the participants determine whether they think the training is worthwhile or not. Collectively, I find they provide as accurate a set of data as I could find trying to indetify level 3 or 4 effectiveness.

That being said, I only ask 2 questions: Would you come back? Why or why not?
Would you reccomend this to others? Why or why not?

I find this answers most of my questions regardless of the type of training etc.

You always have to start out any evaluation effort by asking: why am I evaluating--to what purpose?

If this is NOT a pilot program, you're not trying to determine if you should change the content or keep the instructor, I suggest you drop level I evals. Why? If something goes wrong (the instructor is bad, some are offended by the content), they'll tell you. But if you're past the pilot stage and you're not actively seeking to change the instructor or site or materials or content, then why ask questions about that stuff? It like me asking others if we should keep you on when you've passed your probationary stage of employment.

Now, some would argue that Level I evals suggest other stuff (learning, application, business results). I say "bunk!". If you want to see if the content is making a difference in terms of business goals, go with an ROE (return on expectations--see Toni Hodges-DeToncq's work on this).

Level I evaluations are superb at seeking input on improvements of a course--great for the pilot stage. But beyond that, all it is, is democracy in action. It's like me asking "okay, show of hands here--how many of you think George got smarter as a result of that class he just took?" Whether a majority of hands go up or not does not prove that George learned something significant. You can get low evals and still ahve good classes, great evals and have useless classes. Level I evals basically assess peoples' opinions.

mjp,

Comments above are right on target. The key take-away for this discussion is to keep your eye on the prize. Level 1 is nothing more than a reaction to a recent experience. Place more emphasis on it than that and you are just going to create an interesting but non-value added chunk of work.

The gift your participants can give you is their feedback on the experience. They are not designers or ISD practitioners. What you are looking for are themes and trends in the feedback that would suggest an improvement opportunity exists.

For a multiple day learning activity I end every day with a simple facilitated +/- dialogue to check in on "what's working and what you would do differently?" I start each day with a review of that feedback and an explanation of what we have changed (if anything) and what goes in the bin for future consideration. But I never leave feedback un-addressed.

End of session feedback is not much more complicated...and will use a more targeted approach. The idea here is not to jump on a comment and make an immediate change. Some issues relate to a situation specific context and do not necessarily have meaning beyond that session. Look for themes.

Topic areas generally include questions around:

pre-work/advanced info/scheduling/notification
expectations (met/unmet)
trainer/delivery
activities (exercises, demos, et al)
resources (manuals, handouts)
logistics (room, breaks, lunch, etc)
how I plan to use what I have learned

Jeff