We rolled out our new hire training curriculum for Customer Care in March. My managers would like to conduct a formal review to determine:
1. Are the trainers using the curriculum?
2. Are the trainers using the curriculum in the manner intended?
3. Which parts are working - which parts aren't working?

I've been asked to draft some survey questions - this survey would go out to trainers and training leadership.

Can anyone point me to some resources to assist me?

We will most likely use Survey Monkey.

You will do well to do a Google search for Kirkpatrick evaluation questions and in particular level 3 and 4.

Also, we have a free pre and post course evaluation document at our website and another here..

Any evaluation should always start with the question: "Why evaluate?" You answer that question (be clear on the purpose) and you know what to ask/observe and how to do it.

It's not clear to me why your managers want to know these things. Are they just curious? Do they suspect that the trainers resisted the new curriculum? Was the old curriculum a failure or inadequate and you need to improve? Are there particular business results you need to achieve?

I know, it's tempting to answer "all of the above!" which if that's the case, than start over--that reflects a lack of real understanding of the purpose.

If it's just curiousity, than ask the trainers: are you using this? It's a far simpler approach than anything else you could use. If you think the trainers would lie (say they're using it when they aren't) then this complicates matters--it implies that the real reason for evaluation is to answer a control question and has little to do with effectiveness. If you think the trainers may be using it but doing so ineffectively or wrong, there are lots of ways to test this: the equivalent of a mystery shopper (have someone who knows the correct way to observe the curriculum or pre-train a class participant and then have them report back to you on what they saw). If you're concerned about trainers using the curriculum right for legal reasons, than have someone from legal affairs sit in and observe (which that plus a signed statement by the trainers are the two best ways to deal with the legal "are we delivering it the way this is supposed to be delivered?" like sexual harassment or worker safety content). But ultimately, I think there is almost no value in determining if the curriculum is being used "right" unless it's a formative evaluation and you will freely admit to still being in the demo/pilot stages and the program is still being revised (rather than you've concluded that it's "done"--afterall, you did said you had "rolled it out" which implies you're past the pilot/demo stage).

I would argue that unless the program is still being revised, formative questions are mostly a waste of time. Why ask if they're using the curriculum or if they're using it the right way (unless there are legal or resource issues at stake)--instead, just look at what the impact is on the business.

That's relatively easy to measure: two groups, one with the old curriculum and one with the new than measure level 3 or level 4 or level 5 (behavior on the job, business results, or ROI). If if you're not sure how to do ROI or think that's too complicated than take a look at ROE (developed by Toni Hodges-DeToncq and in her book "Linking Learning and Performance").

Final point, if managers respond with "well, we don't want to try and measure the impact on the business because we're not concerned with that or we can't measure the impact on the business" than run.

Joe:

You bring up some very interesting points. I believe my answer would be "all of the above" and that, to me, reflects a lack of front-end analysis.
I am not leadership, however, so I had little input into the process

Our organization (which is rather large) has been going through growing pains. Standardizing our curriculum was our first charge. Every little nook and cranny had their own ideas and methods. Change is hard, so we suspect that lip service was paid to the new curriculum, but that trainers are continuing with the status quo.

A formal review/survey may our leadership's way of respectfully policing our training department - there's a lot of ex-military people here - but, it may not be the best way.

Thanks for your input.

I appreciate the honest reply. Here's my additional taken given what you said:

1. If the honest answer to "why evaluate?" is "for political reasons" than you should look for an evaluation method that is politically palletable and credible to senior management that doesn't take up too much time and resources. For instance, you could survey managers of people attending courses to ask if they're happy with the courses, if they seem consistent to them, and if their people are benefiting from the training. Pretty quick and painless. Or, you could simply survey the trainers. A question I often ask clients as a way of getting to the unspeakable question of "why in the heck are we REALLY doing this?" is to ask "what would success look like?". They'll probably respond: "if everyone is implementing this uniformly and correctly" (to which you ask "and what's the cost to us if that isn't so?") or "gee, that's a really good question" (which is what people say when they don't have an answer). If their answer is about "better classes/better learning" than just focus on measuring that.

2. Another approach (a highly cynical one but I've used it/recommended it to clients at times) is if you think over time the senior leaders will lose interest, than put something in motion (so you can report on activity) but with no intention of following through and over a couple of months, the focus of senior management moves on.

3. Related to that question of "why evaluate?" (assuming the real answer is: "it is important for our training methodology/content to be consistent--it affects insurance rates and liability on some issues") then you may discover that you need to EXPAND the evaluation to look at more training subjects (to determine if you have consistency) rather than just one group of courses. Think about it: if the issue is consistency, than isn't it inconsistent to insist that some courses be uniform and other delivery methods (orientations, manager meetings, townhall sessions, whatever) don't follow the same methodology?

Ultimately, there are a lot of ways to measure what the trainers do and how consistently/correctly they do it (bug the classrooms is another way). But you've got to try and get a better sense on why this is being pushed so you can figure out:
(a) how to evaluate and
(b) how much energy and resources to devote to this.

Good luck!