Colleagues,

In a business organization the methodology
for at least roi evaluation already exsists
and is used daily.

Nero

Hi Laura L.,

Regarding your comment earlier in this thread...
"I'm not sure I'm understanding your comment... There are so many factors that affect a person's performance -- how can you guarantee that training (or not) is what impacted his/her success/failure? I never take credit for the success or failure of a trainee once they leave the controlled environment. Out in the real world, no less than 15-20 other factors can and do affect an individual's job performance, not the least of which is that individual's internal motivation (which you have nothing to do with)."

Let me respond (at the risk of getting us off of the subject of evaluation literature).

1. You're correct--there are a gazillion variables that affect performance. And they vary in each situation and with each performer. That is why generic ROI data ("the ROI for leadership courses is 40%") is bogus. A training course for one group of performers in a company could have a high ROI. The same course, same delivery, same facilitator for another group of performers could have a negative ROI. And that is not an atypical example. To deliver training (or any intervention) and then start to evaluate the impact is almost (but not quite) impossible. Unfortunately, that is how most training and HR operations work--deliver the intervention and then try to figure out if it works.

2. You can try to separate variables or control for them. Phillips talks about some of these. By doing things like using shifts (which establish control groups), it is amazingly not that impossible to identify some pretty strong evidence that the shift with the training saw performance go up--and the other shift at the same factory but without the training didn't see an increase.

3. But from a performance perspective, we're not really trying to prove that the training (or whatever solution we use) solved the problem. If you do a good job analyzing the problem up front and get to the root cause, then the issue is: can we believe that data--that the cause of this performance gap is whatever you came up with? If that data is convincing enough, we then pick a solution that addresses the root cause. And if results get better, we're not measuring the impact of the solution, what we're measuring is the degree to which our front-end work was accurate.

Let me give you an analogy. Let's suppose that I prescribed vitamin supplements to everyone using this list. How would I now if they made a difference? I could ask each of you ("yeah Joe--I feel better") but that proves nothing (just like most level I evals prove nothing) because maybe everyone liked the taste of the vitamin supplement I gave out. If I tested how fast each of you were or your endurance before the supplements I could measure afterwards and see if you improved (which could be b/c of the supplements or also b/c of a Hawthorne effect). I could just wait to see how long each of you live (again, confounding variables--maybe each of you gave up smoking or begain driving a motorcyle without a helmet). Tough to claim any impact one way or the other.

Or, I could establish health (ie: performance) standards for each of you. I discover that Laura L. doesn't meet her health standards. I do my analysis (ie: medical tests) and discover--she has iron-poor blood. If my evidence is convincing (ie: it's not just me saying "gee Laura, you look like you have iron poor blood!" but instead I have blood work and have accounted for other factors--you aren't anemic, don't have a bleeding problem or some other condition), then I can with good confidence prescribe vitamin supplements. Your iron will improve, your health is likely to get better and you'll meet your performance standards.

In fact (in a desperate attempt to link this back to the original topic of this thread), the Alan Ramias white paper I mentioned earlier explains this concept relatively clearly (much better than I do for instance). If we start our measurement AFTER we do the solution, then you're right--controlling for other variables is messy at best. But if we analyze up front, then our solution has already been identified as the controlling variable. We recognize that many things affect performance but in this case, what is causing the performance gap emerges in the root cause analysis. Evaluation then is much simplier. And we measure to determine if another factor has intervened.