More thoughtful questions!

quote:

Originally posted by KaliKo:
How would you show that a performance improvement did or did not occur due to the training, as opposed to some other intervention, change in conditions, or even chance?

To do this you can do these two things.

1) Partner with the requestor of the training from the start (before the training is designed & implemented). Together you can determine:

A) What performance metrics are you wanting to improve?
B) What is the current level of performance?
C) What business metrics is the performance linked to (and thus will impact)?
D) What else is the business doing that could impact the performance? (motivation, the supervisor, micro-messaging, the environment, non-training related job-aides, technological improvements, etc.)
E) What is the purpose of the requested training?
F) Is training or some other performance solution needed?

2) Conduct a Brinkerhoff style follow-up survey 60-90 days post event. Ask 3 - 7 questions such as:

A) As a result of the training, I learned something new and/or worthwhile.
B) I have used the training back on the job to achieve a concrete & valuable result.

3) Then conduct follow-up interviews with a sample of the top tier & bottom tier respondants. Some questions you might ask are:

A) Why did/didn't you use the training back on the job?
B) How did/didn't you use the training back on the job?
C) What performance & business results did you achieve as a result of the training?

Now you have some stories from real people (leaders like those!) showing how the training made an impact and reasons why it was or wasn't applied. You also now know if and how the training improved performance and impacted business results.

Remember that most businesses do not care if the learners liked the training. And, unless it's compliance training, whether or not they learned something is most likely a minor concern. Most business leaders are concerned with performance and business results.

There was also earlier in this disccusion the question "does there have to be learning in order for behavior to occur?" I would say no, there does not. Behavior can occur for a variety of reasons without learning such as motivation, the supervisor, micro-messaging, the environment, non-training related job-aides, technological improvements, etc.

Try Performance Consulting from Robinson and Robinson. Great tool for analysis and tying training solutions (when/if that is part of the solution) to business goals. Training's purpose is to provide mechanisms for employees to learn skills and knowledge for filling a performance gap or acheiving a new performance goal (opportunity). Thing is, training isn't the only way to acquire that, and usually, gaps are caused by many things other than or in addition to lack of knowledge/skill. So, proving that training had anything to do with performance improvement is about as worthwhile as when people argue it was training's fault that someone is not performing. Training is but one possible tool for addressing only one small factor that is part of the much larger performance picture.

I've been in this business for almost 20 years. I love what I do. I don't have a lot of patience, though, when people try to attribute more to it than it is warranted.

In my experience, a lot of internal training departments reference Kirkpatrick. It has effectively become a standard in the industry for training evaluation.

That said, it's a very ineffective approach/model. First, it's not even a model but a typology (ie: it doesn't explain HOW you evaluate). Second, it's only relevant to evaluation of training. For instance, how would you use it to measure the effectiveness of new work space or enhanced worker benefits packages?

And no, you don't need to change behavior to change performance. For instance, some training (sexual harassment, ethics) is not done in order to change behavior, it's done to reduce liability. Also, what if I replace an ineffective team with a more productive one? I can try to measure how they behave differently (difficult and probably ineffective in the end because there are multiple ways to achieve the same end) or just look at the results. The idea that you need to look at reactions to training, than assess what people learned, than look at how they behave differently in a sequence makes sense only if there was no front-end analysis prior to the training (or whatever intervention you use). If I discover that sales are down because my reps can't close and the reason they can't close is because they listen poorly, I can simply judge if sales went up to assess if the program worked.

I would argue that Kirkpatrick is a common reference for most internal training departments but is mostly irrelevant for most organizational executives. Senior people or folks that populate the "C-suite" don't care for the most part if people liked something or if they learned something or if behavior changed. They're interested in making things better and moving the numbers.

There are a number of evaluation models (Hale, either of Brinkerhoff's models) that are far more relevant because they apply to much more than training and also address evaluation issues during the entire process (such as implementation and design factors).

Sarah, we have a lot of research on this topic, much of it is on our website - and The Training Measurement book (available from Amazon and our website) has lots of statistics on the use of Kirkpatrick and other models. Its still the widest used model, but its slowly being replaced by more modern approaches (including the one we developed - Impact Measurment Model, Success Case Method, and others).

quote:

Behavior can occur for a variety of reasons without learning such as motivation, the supervisor, micro-messaging, the environment, non-training related job-aides, technological improvements, etc.

Behavior occurs always. *Change* in behavior occurs only when the individual chooses to change his/her behavior. There are many ways to help someone through that change, but ultimately, any change that occurs has absolutely nothing to do with anything but the person choosing to change. The only possible exception to this is if someone is quite literally threatening the person to "change or else" via manipulation, but even then, the individual being manipulated can make the choice NOT to be manipulated.