Hi all,

I’m part of a learning organization that’s looking to move its measurement of effectiveness beyond counting how many students took training or how many hours of training they completed. Eventually we’d like to get to the point where we’re measuring ROI or impact on business, but for now our execs would be happy to show that our students are learning what we teach them and successfully applying it on the job.

Our challenge is that this is the first time that many in our organization have been asked to think about measuring their programs in this fashion. What approach(es) do you recommend we take to get everybody on the same page and thinking about measurement in a more advanced fashion? Train everybody or just pick a few people per learning group to serve as measurement SMEs for their group? What resources (books? training courses? white papers? guest speakers?) do you recommend?

Thanks in advance for your input.

Frank

High Impact Learning by Brinkerhoff

How did you know a particular class was needed without the research and analysis that pointed to a particular issue that needed resolution and the assessment that training would help rectify the situation? All "formal" learning opportunities/events should be linked directly to performance goals which should come directly from business goals.

Thanks SSFN -- will look at that book. I've heard Brinkerhoff speak and was impressed.

Our folks do a good job of working with stakeholders to identify training opportunities, and we have anecdotal evidence that the org is doing a good job, so we're pretty confident that we're on the right path. What we want to do now is come up with something more objective and less subjective, something that we can use to show that we're improving as a learning organization.

Any other thoughts (from anybody)?

Frank

Frank, let me expand by giving you an analogy. Most training is open enrollment. And we've traditionally evaluated our "success" by people's reaction to the training and also if the client was happy with the training (with the mindset that our job was to keep the client happy). But tougher, more competitive environments have pretty much eliminated as an effective way for most training departments to operate.

Even operating by identifying clients who request specific training is an issue. That's because clients still lack the objectivity and ability to determine if it's a "training problem."

Here's the analogy. Let's suppose you ask for vitamins. I give the to you. A month later, your insurance company says "how can we tell that these vitamins Frank is paying for should be reimbursed? How do we know they're improving his health?" So I ask you and you say "I feel great--I bet it's these vitamins." When I respond to the insurance company with this, they say "how do we know it's the vitamins? It could be some other factor. And how do we know he needed them in the first place?" At which point I either give up, or go through a very rigorous and hard process (somewhat similar to what Jack Phillips recommends for determining ROI) to demonstrate that the vitamins probably contributed to you feeling better.

Or, there is another approach. A front-end-analysis (or to use a more generic term: a performance analysis that goes beyond knowledge-skills issues). You come in and request vitamins. Rather than sell them to you (or even ask basic questions "are you feeling run down? Need some pep?"), instead I run some tests on you. I determine you've got low iron levels in your blood (which are associated with lower energy levels). I give you the vitamins. You energy gets better. The insurance company repeats their refrain of "why should we reimburse Frank for the vitamins" but in this case I can say: "we identified that Frank had low energy which as affecting his work hours. We tested and found he had iron-poor blood. I prescribed iron-supplements and now his performance is better."

Now let's apply this to training. Client requests training saying "my team fights and bickers all the time--they need team skills." We don't confirm that they fight and bicker--that doesn't prove that training would work. We identify the business goals of the team and what performance is critical to achieve those goals (because we may discover this team doesn't need to work effectively in order to get their work done--so the fighting and bickering is unfortunate but has no impact on performance). We identify what is causing or likely contributing to the fighting and bickering (it may not be a knowledge-skills issue in which case training will fail completely). If at this point we've documented that the team is critical to a business objective, that collaboration determines if that team will achieve that objective, if the team doesn't collaborate because they don't know how to work together, then we can do team skills training, should see the team start hitting their results and we're able to easily (yes, easily) document our impact and even the ROI.

There used to be an auto parts commercial that used the phrase of "you can pay me now (up front) or you can pay me later (the costs of not doing the upfront work)." Well, you can either do the FEA (invest some time up front). Or do the training and then have to struggle to document business impact and ROI--not impossible but very tough.

Brinkerhoff is good. I also suggest you read Toni Hodges-DeToncq's book "Linking Learning and Performance"--she has some very practical evaluation methods (like ROE).

Likewise - Performance Consulting by Robinson and Robinson.

The Brinkerhoff book is about leveraging transfer. The Robinson book helps you create the links.