We're working on a long-term assessment for the first time at our agency. Laura, you replied to this post:

"I think it's important to note that because job performance and training transfer are affected by a large number of factors, there is no way to conclusively prove that a change in performance after training was indeed the result (or solely the result) of that training program. Likewise, if performance doesn't improve, it's not necessarily due to a problem with the training program."

That's so true but what is the best way to measure training's role in changing behavior/improving performance? I liked what Bob Blake included about assesing what the learner knew already about the topic because we hadn't thought of that. However, couldn't that be measured in a pre-test or informal query by the instructor at the beginning of class?

Our task is seemingly simple: to provide upper management a comprehensive report on whether our department, with its current staffing, is offering relevant, effective trainings? We just completed a needs assessment survey to decide what trainings to offer and which to drop. I'll include what we've put together and I'd appreciate any feedback on whether the tool could be effective at providing information to management.
_______________________________________________
Long Term Training Assessment Tool

Employee Section:
Please answer every question. We need to know your honest opinion.

1. How relevant do you feel the training was to your position? Please explain.

2. How prepared for the training do you feel that you were? (For example, were you asked to bring paper for notes, did you receive handouts that allowed you to effectively follow along with the presenters, was the purpose of the training fully described to you, etc.)? Please explain.

3. How well was the training organized to address the learning objectives? Please explain.

4. How much of the course content were you familiar with prior to training? Please explain.

5. How conducive to learning do you feel the training area was? Please explain.

6. How effective were the handouts and presentation materials, if any were used? Please explain.

7. How have you used the handouts and presentation materials since attending the training? Please explain.

8. How well did the training meet your expectations? Please explain.

My only concern is if a really nonchalant employee writes “not well” or “very well” to all the questions. Does “Please explain” address that or would “Please cite an example be even better?

Then we list the training objectives on the other side and ask: Since taking the above-mentioned training, please explain why you feel that your skill level has or has not improved in the following areas.

What I'd also like to know what percentage of responses is not only reasonable but also valid? We have about 400 employees. Do people participate and give quality answers on these? We only got back 38 needs assessments from this same group.

One of the most effective questions to gauge learning and retention is simple:

Please list everything you remember from the training.

This forces the person to reflect back on the session, and is only a list so not an overwhelming narrative.

While it may seem counter-intuitive, my experience is that people are more appreciative of the open-endedness of the question. I have gotten a higher response rate from something this simple, as opposed to a questionaire. Primarily because the writer is engaged in the process and not merely filling out a form.

This is a practical application of the type of training that is so effective.

The lists of remembrances will provide you with examples of what they retained, which allows you to target areas that might not have "stuck" with the participants.

If you have 5 goals and the replies only list 3 of them, you know where your efforts should lie.

quote:

I liked what Bob Blake included about assesing what the learner knew already about the topic because we hadn't thought of that.

That's all part of analysis, and something you can/should find easily in just about any resource about instructional design, and more specifically - performance consulting. Perhaps it uses different terminology, but it's all the same thing, repackaged so that people can make money off marketing the same stuff in a different way... (but that's another discussion)

Going back to the objective stated by icyharris:

quote:

Our task is seemingly simple: to provide upper management a comprehensive report on whether our department, with its current staffing, is offering relevant, effective trainings?

It seems simple if you can answer with a simple yes or no, however to support or explain such an answer to management it can be a bit more complex.

The objective seems to be to determine whether the department is overstaffed or understaffed, and whether the course offerings are relevant or not. As Always Learning points out this is part of the initial analysis of developing training programs to begin with, and more importantly part of the learning strategy in aligning corporate objectives with training.

I think that regardless of the answer to this question, it would be beneficial to consider a proposal to examine the job functions that are closest to the revenue stream and define the proficiency of those functions which will define the training objectives to improve performance of those people in those roles. This has the most impact to the business and thus elevates the importance of the training function.

In the overall strategy, all job roles should be considered because regardless of the job within a small or large company, the customer (where the revenue originates) is ultimately impacted in some way even from job functions that are seemingly farthest away from the customer.

We have standard follow up evaluations that any customer who users our learning analytics technology can customize. The benefit is that it has about 200 million benchmark data points behind it in a normative database so you can compare the results of the evaluation from your own training to both internal and external benchmarks within the learning analytics technology. The follow up was designed around the spirit of the Kirkpatrick model and Phillips model and will actually calculate a financial ROI.

For more information go to http://www.knowledgeadvisors.com or you can email me at jberk@knowledgeadvisors.com.

This site wont let me add the survey to this message but if you email me I will send it to you.

Jeff