The division of the company which I serve provides audio-visual rental and staging services for conventions, tradeshows, meetings and other entertainment events. My department is 2 years into a entry-level technical training/education program for our operations employees regarding safety, theory, and proper procedures for the different aspects of presentation technology. For each of the seven modules of the program, there is a "Level-2" post-test which participants must score 80% or better to pass the course. Until now, the assessment has been closed book, but we're toying with the idea of allowing participants to use their own notes on the exam which they create during the course.

The primary benefit we see from allowing participants to use notes is the greater probability of knowledge transfer by incentivizing them to take notes. (In the two trials we've done since November, we've had -0- failures, while previously, we averaged 15-25% failure rate.) On the negative side, we're wondering if allowing them to use notes on such a level-2 assessment tool will give us a true representation of participants' understanding as opposed to them just copying what they have in their notes. We are exploring re-writing the exams to provide more "synthesis/application" style questions, however because of the educational nature of the courses, some objective-style questions are inevitable.

Has anyone out there come across or perhaps been involved with a similar situation regarding Level-2 assessments? Does anyone else allow the use of participants' personal notes on such an assessment or have strong feelings or research that might direct us toward feeling better about leaning one way or another?

Thanks in advance...

Are they allowed to use notes while performing under real time?? What have your trials in real time indicated? What do the supervisor's contend??


Nero

Since you have been running the training for two years you should be able to answer another question or so about how these people who have taken the training actually perform on the job. What kind of relationship is there between test scores and job performance? If there is a direct correlation, then that might tell you one thing while no correlation might tell you something else.

One thing you should be able to validate at this point is the congruence, or lack thereof, between objectives tested in the post test and actual job requirements. Unless you find a direct relationship between test scores and on-job performance, I'd question the legitimacy of the test to begin with. In this case, going for an open book will not really have any effect.

Another way to state this is "don't mess with (or put to much faith in) level two until you have verified level three."

I think there's a gap regarding what a level two assessment is really supposed to be. Assuming we're not talking about doctoral students in their residency taking a test on how to perform a life saving procedure, and that we are talking about typical corporate stuff -- a level two is simply supposed to assess "is the person getting it?" This can happen in little pieces throughout training and then with one big one that puts it all together at the end, for example. It's not supposed to be like a final exam in high school or college. It's not supposed to be "put the person on the spot and make them feel like they have to remember everything by heart that they just learned."

So yes -- participants should be able to use whatever resources they have in order to respond to your level 2 tools because they *just* learned it. When have you (not you personally, but the general you) ever had to take a test immediately following the learning? Didn't you have the opportunity to study for a couple days at least? I don't know about the rest of you, but usually, we had assignments (another form of level two) to see how we were doing as we went, and then quizzes and exams to check our knowledge retention.

I wish I could understand why I see so many people swing the pendulum (not you, just in general) so drastically. It used to be that we couldn't even get others to even consider other forms of evaluation. Now, they insist on it, but don't really seem to understand what it means.

According the definitions I've worked with for years, level two determines if the learner has met the stated objectives of the learning event. Level 3 is a measurement of workplace performance achieved as a result of training. Level 4 is a measurement of the business results achieved by the training.

If the stated objective was for the learner to "get it," then that sort of post test is fine. Again, the method of testing should be congruent with the verb or verbs used in the objective itself.