My organization is looking at incorporating confidence factors into all levels of our training evaluation. We are trying to determine the easiest way to gather and report the data. We use Perception for assessments. Does anyone have any experience with this who could offer some guidance as we move forward?

Hmmmm....I'm not sure why you're doing this (especially if it involves Level I data). But that's not what you asked.

I suggest you take a look at what Toni Hodges-DeToncq refers to as an ROE (Return on Expectations). If you look at the methodology for determining the ROE, it's basically a case of using a confidence factor (in this case, a percentage) to figure out a rough ROI or business impact of an intervention.

I am curious about your remark on not being sure why we are doing this. You don't believe confidence factors have value?

It's not that I don't think confidence factors have value. It's that I think Level-I data has minimal value. It's extremely valuable for pilots and demos--because you want quick feedback to change the material.

But let's suppose you conduct a level-I analysis (not for a demo or pilot but for an established course with a proven instructor) and you attempt to establish a confidence factor for that data--I'm not sure what value you'd get from that information. On a level-I, the data is what it is--people's opinions of what they think is true. You can always use a Likert scale to assess the degree of commitment to a particular answer.

To my understanding, a confidence factor or level is useful for two reasons:
--it distinguishes between a strong commitment/believe versus a weak on. And data has repeatedly shown confidence in a particular position is easily manipulated (the more data I give you, the more confident you become, the more willing you are to "bet the rent" even though the outcome may not be any different).
--it (depending upon what you're assessing) can be a means of determining a risk level or outcome level. But my understanding is that works primarily when you have a trained analyst evaluated a specific situation. For instance, President Obama asking for confidence levels behind predictions of CIA analysts looking at Iran as to how strongly they hold their beliefs or see their recommendations. But that is distinguished from asking people off the street "what do you think is going to happen in Tehran?" and then asking them to assign a confidence factor. Just because I took the training doesn't mean I'm highly knowledgeable in my ability to accurately assess the likelihood of various scenarios (such as my ability to generate particular results with these new skills).

If you're talking about perceptions on what Kirkpatrick would call a Level-IV or Level-III, than the ROE will work just as well. If you're just trying to distinguish between people who think that the class might help versus those who feel strongly it will, then use a Likert scale. But I'd argue that attempting to determine levels of confidence for training value or outcomes is probably beyond the competence of the trainees (not inability to assess probability, but inability to determine the impact of the class).