In our organization, each year, we train 1000+ employees, hold 500+ training classes, and train at diverse, global locations. We currently evaluate at level 1 only.

How can we effectively and efficiently conduct evaluations at the other three levels also, particularly level 4? What data do we gather? How do we do it?

More and more I'm seeing the use of pre and post training assessments of the participants' skills/behaviors (competencies)that are taught. This gives you both benchmark data based on level of demonstrated skill before the training compared then to their level 60 days after the training.

Check out this Moviehttp://www.progrowthassociates.com/2020insight/movie or go to http://www.progrowthassociates.com

It doesn't sound like you are anywhere near ready to gather data. And none of us can tell you what would be appropriate data, let alone the most effective and efficient way for your organization to do the evaluations. Someone in your organization will have to figure that out. It's not a generic, one-size-fits-all thing.

I would suggest you read something on evaluation by Kirkpatrick or Philips. And ASTD has lots of good books on evaluation methodologies. There's much background work that needs to be done before deciding on data and how to collect it. There is no short cut.

Consider tying your training efforts to specific outcomes that affect the business.
For example, from the Peer Mentoring Workshop we can track these metrics:

Reduce ramp up to productivity (count days from start date to independent use of the skills in the training plan)
Reduce turn-over from burnout (track attrition, especially for recent hires)
Tool usage (count how many people are using the tools months after the workshop)
Reduce safety incidents for new employees (count safety incidents per participant against a control group)
Develop a known pipeline of next generation leaders/managers (Generate a list of qualified mgmt candidates with proven leadership skills using the PM tools)
Reduce rework stemming from lack of skills (measure hours of rework on a project, and/or scrap in industrial setting)
Reduce risk by methodically replicating skills of at-risk workers (workers with singular knowledge can be listed, their skills inventoried, and then training plans executed to prove replicated skills)
Improve morale because communication is better and expectations are clear (employee satisfaction survey)

Each of these measurements requires a bit of effort but they all show ROI to people outside as well as inside the training group.

Good luck
Steve Trautman
Peer Mentoring Workshop

I see this question on these boards a lot, so let me take this opportunity to try and simplify this for everyone.

Kirkpatrick's 4 Levels:
Did they get it?
How do I know they got it?
What are they doing differently now?
So what?

Now let's dig a little deeper.
1)To borrow from Covey, "Begin with the end in sight". Never offer or launch training without first considering level 4 (aka "So what?"). "So what?" is a metric: money saved/earned, time to productivity, employee satisfaction. You musthave a metric in order for you to justify having people take time to go through training.
2)What do you need people to do differently in order for you to achieve your "so what" metric? Work harder? Faster? Smile more? Come to work earlier? Whatever it is, "What will they do differently now" must directly influence "so what".
3)How do I know they go it? Before (or very soon after) participants leave your training they must be able to show that they go it. Test, case study, whatever. It is important for them to be able to articulate what they have learned.
4)Did they get? If they show up and have a pulse you have achieved Level 1. Just kidding! The training should be well liked, it won't mean that you have created a behavioral change, but at least you know people will show up and participate. You will have an opportunity to transfer the knowledge.

Example:
Customer service people who smile are 15% more likely to have a positive result with the customer. (made this up)
Therefore, we want to train all our customer service people to smile with the goal of increasing customer satisfaction by 10%.
The different behavior (level 3) would be smiling more.
We would test everyone for smiling (level 2) at the end of the class and make sure they have it right.
We would bring in professional clowns (level 1) to properly teach everyone how to smile.

Hope this helps!