One idea is to send follow up surveys 30 days post and ask specific questions to determine if the participants are using the knowledge. That works well with one initiative, like a webcast, but any ideas how to do Level 3 eval for static content, like a video series?

The initiative is to educate customers and alleviate some of sales' and support's burden. Level 4 is pretty straight forward (lower sales cycle, reduce support calls on specific topics, etc ...).

I'm starting to think level 3 is not worth it in this situation, but thought I would throw it out there to see if anyone else has any ideas.

I'm not a big fan of the Kirkpatrick typology for evaluation. I would say that the best reasons for doing a level-3 evaluation in most cares are:
--either you didn't do a really good front-end analysis so you need to establish a chain of inference in order to get to Level-4 and eventually ROI (ie: people liked it, you can show they learned content, there is evidence they're applying it and behaving differently on the job, we're getting better results so it sure looks like the training is what caused or contributed to the better results). Or....
--The change in behavior is a critical piece of the work. For instance, you have truck drivers and they need to be certified. Or meat packers. Or some other job where perhaps a government agency has to certify particular performance and/or behavior levels.

So my question to you would be: why get Level-3 data, to what purpose? If you need to build a case for inferential connection to impact, okay. If you are going to be assessed to some extent (like through certification), than Level-3 makes a lot of sense.

Now, here are some other thoughts (and to some extent, the "con" to doing a Level-3). Unless you control who watches the webcast and you accurately assess who needs to watch it, you'll likely get some bad scores on Level-3. Why? You may get some people who watch who don't need it and are already exhibiting those behaviors. So the webcast would appear to have failed (no behavior change) or in some case succeeded (participants are following the behavior exhibited in the webcast) but either might be deceptive (no behavior change because they were behaving this way pre-webcast, no behavior change because they were behaving this way pre-webcast--so a lot depends upon what you try to evaluate at level-3).

Additionally, if you've done enough front-end work to argue convincingly that there is a performance gap and the cause is a lack of skills/knowledge in this particular area, than you show the material and look for a change in results. For most executives that's going to be good enough--did the organization get better results after doing the program?

Kirkpatrick's (and Phillips') levels of eval do have their issues, but they are proven and accepted methods of measuring the impact of our efforts. I can think of a few reasons for doing a level 3 evaluation:
-- The ROI measurement will take 3-4 months to measure, while behavior change will be instant. Why wait a full quarter to find out if the expected ROI change took place when you'll know right away by evaluating the behavior which leads to the ROI.
-- For customer based training, level 3 data evaluates the key takeaway for the participant. What does your customer care if you show internal ROI to the executives? The customer wants to understand what value he/she can obtain from the training (basic WIIFM), and that what they can do with the knowledge they'll learn.

Even a well defined program which has a full analysis with needs assessment should still have a level 3 eval in place. One reason is the wide range of variables impacting the ROI measurement. I've yet to see a ROI model for a training engagement which can eliminate other variables in play, so you need a way to balance the data you're collecting. The level 3 data provides you a sort of checksum to determine how much of the ROI change is a result of your efforts.

Not sure I agree with your webcast example. Most (of our) customers are very good at self-selecting what will be valuable to them. If the title and abstract of your webcasts and videos are appropriate, your customer base will learn to decide which ones they will benefit from. Another way to ensure you are surveying the correct folks is to only send it to those who stayed until the end of the event. If someone joins who shouldn't be there (ie: they already have the knowledge and are already exhibiting the behaviors) they'll most likely drop off before the end. A lot of it also has to do with having a great marketing resource at your disposal. Hitting the target audience with the message is part of the success story here.

Thanks for the thoughts, but I'm really looking for tactical ideas on what people have done to measure level 3 when you have so many engagement points. I don't want to send a survey for a video series, because the individuals who watch them will be bombarded with emails. I think in this case, level 3 is probably better left alone, but not because I don't see value in it. Level 4 in this case seems better suited to being rolled into the larger initiative rather than specific to the video series.

Stoopalini,

Regardless of who is being asked the feedback you seek remains the same: Are the newly acquired skills, knowledge, or attitude being used in the everyday environment of the learner?

Not sure if these thoughts will help....consider any or all:

- A few questions at end of each session to get that initial reaction.

- A opt-in option (also included at end of session) requesting permission to send along a follow-up survey 30 days later. Most of us who use the web for learning and research are pretty well conditioned to expect these.

- If customers can be linked to sales persons, find a way to check in with the sales folks to see if they are noticing change in level of customer knowledge, savvy, et al...etc.

This "may" help get a better feel for the downstream effect...but you are correct in your observation that there are also other interim variables that can have an impact as well.

More art than science here....:-)

Jeff

Thanks Jeff. The opt-in idea is ineresting. It would be fairly easy to implement with webcasts, but a bit more challenging for posted video series. If I could figure out a way to make it automated, it would work well.

Completely agree, definitely more art than science. Especially when you factor in company culture and executive personalities