quote:

Originally posted by mrs.mizo:
More specifically, I am looking to see if program theory (for the volunteer program) and outcomes match.

mrs.mizo,

Thanks for the additional information. Your project should be fairly easy to evaluate. As an example, here's what was done to measure theory and its outcomes in one project. (This was for a team-driven OJT program based on the psychological theory of Alfred Adler which included 5 fundamental concepts of human behavior and 4 human interfaces. that impact job-related conditions.)

Very briefly, the Adlerian theory includes these fundamental concepts: 1) all human behavior is goal-directed, the primary goal being to belong; 2) people are creative decision makers - i.e., having an active role in solving problems leads to greater job satisfaction and higher self-esteem; 3) human behavior occurs in a social context and has to be interpreted from within that context; 4) use is more important than posession (i.e., skill and knowledge alone are not sufficient for real human performance improvement - there must be commitment to an attitude that values work, worker participation, and job satisfaction over and above the possession of skills and knowledge; and 5) people and organizations function holistically.

And the 4 interface issues that affect human performance are: 1) the people-people interface; 2) people-information interface; 3) people-environment interface; and 4) people-equuipment interface.

To see if theory matched outcomes we devised a "measurement menu" as part of the project plan that included a systematic method for data collection. Each item we measured was tied back to some aspect of the theory. (First you need to break down the theory into discrete items that you can work with - sort of like a task analysis of the theory - break it down into bite sized pieces to measure.) We collected both objective and subjective performance indicators.

We recorded baseline data, post-implementation data, as well as data points between those two end points. For example, we wanted to measure not only attitude change, but the overall team effectiveness as the teams matured. So we did periodic observations of design team meetings at various stages to compare with the pre and post data. We also had supervisors and trainers tracking the data and providing it to the program facilitator for compilation later on. Everyone involved in the training program had a part to play in the evaluation - supervisors and managers, trainers, trainees, team facilitators, design team members, approval team members, training administrators, and employees in other parts of the organization who were impacted in any way by the OJT program.

We measured both immediate benefits such as improved attitude and morale for system participants as well as long-term benefits that included improved attitude/morale for system participants AND people in other parts of the organization, increased job satisfaction, enhanced team effectiveness, and improved job-related conditions affecting task accomplishment.

To measure immediate benefits, we did a self-report written survey before program implementation followed by repeating the survey at various points during and following implementation. The survey contained items related to general attitude and morale, as well as specific aspects of the program. We gathered anecdotal evidence through interviews with supervisors and employees. We observed employees and design teams both live and via tape to obtain additional data on attitude change. The specific items we measured came from the theoretical basis of the OJT system.

For long-term benefits, we measured changes in attitude and morale in other parts of the organization that showed increased cooperation and coordination attributable to design team efforts. This was done by administering self-report surveys before and after program implementation.

Also for long-term benefits, we measured team effectiveness by observing design team sessions and employee interactions on the job. We looked for changes in communication, decision making, problem solving, and conflict management. These observations were conducted by supervisors, design team facilitators, and trainers at the beginning of the project and various times during implementation. We also conducted interviews at various stages of development and implementation.

To show improved job-related conditions affecting performance (the impact on the 4 human interfaces), we used pre and post measures from each interface area. They were identified in writing (at the beginning and also various points during the implementation of the program) by design teams and later tracked by facilitators and supervisors. These interface areas, included such specifics as faulty equipment, parts unavailability, inconsistent procedures across departments, and poor lighting; they were identified and resolved through coordination and cooperation with those involved.

Well, I guess this wasn't too brief - hope I haven't muddied the waters! Anyway, the important thing is to break down the theoretical items you want to measure and then devise a method to collect the data that will answer your question - i.e., here's the theory - numbers 1 through whatever - the theory predicts this..........; do we get the hoped for outcomes that the theory predicts?

In the above case, I got research psychologists from NASA to help with the measurement menu - especially the subjective part. If you have access to a university, why not ask them for help in devising the surveys, interviews, etc.

Best of luck!

P.S. This evaluation covered all of Kirkpatrick's levels (in case someone is forcing you to use Kirkpatrick, you can tell them you did........) However, the evaluation was not devised with Kirkpatrick or any other "model" of evaluation in mind. The NASA scientists did know of Kirkpatrick, but agreed that it was not the way to approach the evaluation. It just puts blinders on if you're stuck on four levels. You miss the important stuff, the specific aspects of your unique project, and possibly the essence of what you're trying to answer in doing the evaluation in the first place.

This message has been edited. Last edited by: KaliKo,

mrs.miso,

I am curious to find out if you found the above info helpful.

Absolutely, "miss the important stuff".

Nero

Hey, thanks Kaliko. If I can't use this exactly, you gave some great insight and theory to back it up! Kudos to you!