Hi All,

I've been musing about the issue of isolating the impact of training. My musings are in the text that follows (It's quite long and I couldn't upload it as an attached file).

Essentially I'm exploring traditional approaches to this, and suggesting a 'new' one taken from the world of six sigma and process improvement.

I wonder what others think?

Kind regards

Martin
====================

Isolating The Effects Of Training From Other Influences On Performance
See if the following sounds a little familiar…

A company invests time and money in training its sales force in certain sales techniques. It is early August, traditionally a quiet time of year, and so training at this time is assumed to have limited impact on sales figures.

Time moves on. It is now late September and the sales figures are rising. The question on many managers’ minds, including the training manager, is “is this rise due to the seasonal nature of the business, or due to the effects of the sales training back in August?”

The training manager is interested because she needs to know that the training she is providing is adding value – ideally more value than the costs and troubles associated with actually running the training.

Regional managers are interested because they want to know that the money from their budgets that has gone to pay for the sales training, the time and opportunities lost by way of sales staff being on training and not out selling, and the actual skills learned on the training will actually put them in a better position to reach or even exceed targets and ‘put one over the competition.’

Senior managers are interested because they are ultimately responsible for performance and the strategies they have put in place to achieve their strategic objectives.

The individual sales people are interested – have they acquired additional ‘killer sales skills’ that will help them get to the top of the sales league tables, perhaps even make them more attractive to other employers that may offer better terms? Are the improving sales figures down to the training or the seasonal nature of the business? They have a view, of course, but this is mostly based in intuition.

It does sound familiar, even to those of us who have little or no experience of the sales training arena. This desire to improve performance, marketability, and to make sure that the decisions made by management are the right ones, is endemic: you will find it in any organisation.

Over the years, since Kirkpatrick first had his article on training evaluation published in Training Magazine back in 1959, a number of approaches have been presented to try and help answer these questions. The majority of these approaches it seems have focused on the perspective of the trainer and the trainee, with some attention also being given to the reactions and learning of individual trainees. Typically these approaches involve a basket of techniques, including the use of control groups, where one group receives the training, another doesn’t, and the assumption is that the training is responsible for any differences in performance between the two groups.

Another technique often found in this ‘basket’ is the use of expert estimation – experts, in the above case it would be sales personnel, are asked to estimate the impact the training has had on their performance (e.g. 35% of their performance improvement for example is due to the training they attended). Some more sophisticated approaches also ask the line managers of these experts to do the same – to estimate the impact the training has had on the performance of those who attended the training. An even greater level of sophistication exists in some approaches, where those involved are asked to estimate how good their estimates are (in percentage terms), and to multiply the 2 together. For example, the 35% estimate of the contribution the training has to performance improvement is felt to be ‘accurate’ by the expert concerned to the value of say 70%. In other words the expert feels 70% sure that the estimate he has provided is ‘right’. So 70% multiplied by 35% gives an overall of 24.5%. If the performance improvement is valued at $4000 in the first month then the value added by the training is 24.5% of this or $980. If the training cost $500 for this person to attend then it seems to be money well spent. If we find that by adding in all the other costs – labour, lost opportunity etc – i.e. the ‘fully loaded costs’ we may have a slightly different story, say costs of $700, but none the less it seems to be money well spent. But for how long should we claim the benefits? That’s an issue for another time.

The other technique often found in the ‘basket’ of tools is trend line estimates. By tracking performance before the training and on through and after the training has taken place, it can be asserted that some of the improvement in performance occurring after the training is due to this training.

These estimation methods are often used with so-called ‘soft skills’ training. This is training that is focused on difficult to measure skills, such as management, leadership, assertiveness, and many more of a similar ‘ilk’. Being able to type at a specific speed is easy to measure. Selling and reaching a particular target. But what of the skills, knowledge and attitudes used to achieve such a target? Is it all down to the process, or is it down to other things as well? Things like relationship management and the behaviours that go with it – behaviours that seem to be difficult to define and measure? These examples, and others like it, are where these estimation methods are used, because until now there have been no alternatives. And sadly, as history has shown all too often, when times get tough, and the clear link between such ‘soft skills’ and performance, cannot be seen, at least by those accounting types that we often find at the top of organisations, who see things much more in concrete, measurable, black and white terms, then we see such training curtailed, and training staff let go. Ask anybody what the first public sign of hard times in an organisation is, and they are likely to reply with ‘cuts in training’.

So here’s the killer question: Do these methods, when used in any combination, provide a credible and definitive answer to the original question in our example above? You may recall that the question on many managers’ minds, including the training manager, is “is this rise [in performance] due to the seasonal nature of the business, or due to the effects of the sales training back in August?”

If you feel the answer to this question is ‘yes’ and the approaches descried above would be accepted in your organisation, then don’t waste any more time reading on. My advice would be to spend the time you save by making sure your resume is up to date. Seriously. I took a similar view in the past. And I lost my job as a training manager because my CEO did not share my view – he wanted something far more concrete, something that would stand up to the intense scrutiny he knew he would get from his shareholders. And I couldn’t give him that – not with any combination of the techniques in the ‘basket’ described above.

You see, the missing ‘ingredient’ is a clear, explicit causal link between the training and the performance. Now before you all rush in with some argument along the lines of “hang on, the training was done, there was, soon after, an improvement in sales performance, there’s a clear correlation here” let me present one simple fact: causation IS NOT correlation. There is an assumption that we humans make that cause and apparent effect, closely separated in time, are connected. This is not always the case. I shan’t go in to the specifics here, but suffice to say this involves systems theory, and research on the Internet in to the work of John Sterman and Stafford Beer to name but a few will tell you more. See http://www.exponentialimprovement.com/cms/BusinessDynamics.shtml and http://www.staffordbeer.com

Now, this is not to say that the ‘basket’ of techniques I described earlier is junk – far from it. If a customer or stakeholder is happy to accept these methods when considering the worth of a particular training activity, there is little point in going down another avenue that frankly, will likely take more time and effort to do. And let’s face it, the ‘blue-eyed, golden-haired boy’ of many organisations, the marketing department, has been ‘dining out’ on its claimed successes and values for decades on just this kind of approach!

However, if you find yourself in a real ‘cat fight’ over the true value that training brings to the organisation, you can do little better than to have in your armoury an explicit, clear causal links and the data to support it – it’s difficult to argue with facts, no matter how uncomfortable the story they tell.

So, how do you go about achieving this? Simple, at least in concept. You determine the root causes of a particular performance indicator. For example, what are all the factors that affect sales? Some of these will be quite ‘off the wall’, and some will have a much bigger contribution than others. In reality, no matter what the organisation, no matter how unique, there will likely be many 10s, if not 100s of factors, though typically there will only be some 5-10 that account for the bulk of the performance, say 95% of it. Of these 5-10, there will probably be 1 or 2 that account for 50-65%. What is also quite possible is that these dominant factors may not be the obvious ones you thought they were. Either way, there is only one way to be sure – work out the contributions of each factor. Of course, this could take an age and a pile of scarce resources. But at least you could make decisions on future use of resources, e.g. to fund sales training, based on full knowledge of the facts and the relationships between these factors and performance. You could potentially forecast with great accuracy the likely impact on performance due to an intervention on one or more of these factors. How useful would that be? (Don’t worry, it’s a rhetorical question. The answer is usually going to be “extremely useful”!)

You may not have the time, resources, or inclination, to be this comprehensive in determining the contribution of each root cause factor. So don’t do them all. Get some experts involved, experts in the processes and behaviours that deliver the performance indicator in question. There is no harm in taking a considered and learned view as to which root causes or factors are likely to be the main contributors. There are processes available that make it possible to gather data and through analysis confirm or otherwise the contribution that each root cause makes. If you end up with less than say 80% of the performance being explained by the selected root causes, then this suggests you have missed a couple, so select and analyse a few more. You may need to go around this loop once or twice. Once you can account for at least 80% of the performance, move on. By the way, this 80% is a purely arbitrary figure. We’ve already agreed we probably don’t or can’t account for 100% of the performance. Nature has this so-called ’80-20’ rule (the Pareto Effect) so I have suggested that to go much beyond accounting for 80% of performance takes us in to realms of diminishing returns. There are other approaches that are more involved, which I won’t go in to here, but the Six Sigma world will provide plenty more on this and other aspects.

What you now have is a pretty clear picture as to where at least 80% of the performance indicator comes from – it may be a combination of processes, it may involve a combination of behaviours too.

You now have the information you need to identify the kinds of interventions needed to deliver measurably improved performance. With this level of clarity you can also determine the training needed to either enable such performance improvement, or to support such performance improvement. In the first instance, training that enables is a ‘must have’ and it makes little sense to look at this training in isolation, meaning that it makes little sense to try to isolate the impact of the training – it is integral to the intervention. In the second instance the training is not a ‘must have’, but is now a ‘value-add’ intervention. Because we have clarity about which root cause factor is being affected in order to deliver the performance improvement, it does make sense to separate the training from the rest of the intervention and look at its impact.

Now, here is the ‘clever bit’. The training should itself be one of the factors! We’ve already isolated its impact. If we haven’t, why are we even considering it, if we have no idea what impact it is going to have. That’s just falling back in to the old ways – training for the sake of training – a very ‘trainer-centric’ view far removed from the issues of performance and customer.

There are plenty of tools within the arenas of process improvement (try the Six Sigma world for starters) that can be brought to bare. I’m doing just that and there is more info on my web site.

Why is it so important to isolate it? The issue I have (not with you, but with this issue in general) is that it is assumed that it's necessary to isolate it. All of the arguments for how to do it beg the question.

Hi Laura,

Thanks for your response.

I quite agree with you, which was one of the points I was trying to (badly!) make.

There seems to be a massive 'group think' about this, not just in the industry, but also by some CFOs and others at that level, in my experience.

I wonder if this last aspect is due in part to the fact that in many organisations the training function is explicitly and separately organised - there is a training dept, manager and staff, and people assume that this dept is responsible for all aspects of training, including the results to be extracted from it, while the line managers get on with the operational stuff. This is strengthened by the training dept holding the training budget and having to respond to the CFO and shareholders requesting ROI-type information.

Cheers

Martin

Any investment in evaluattion should be proportionate so I've always used the 'civil Vs criminal court' analogy ...

i.e. for some you should seek to prove that L&D was responsible for the impact 'on the balance of probability' and for others you should seek proof 'beyond reasonable doubt'.

For instance, if I'm evaluating the pilot phase of a programme that is anticipated to cost significant investment and the risks of it not succeeding are also significant I will look for proof beyond reasonable doubt.

If however, I am looking at the efficacy of a mid-range course with medium to low risk than the balance of probability will do.

In either scenario, triangulate your evidence, review and analyse simultaneous factors (including their interaction with each other) and ask the business experts (they're in a better position than you to know what impact those other factors would have had without the L&D).

The bottom line is that we don't live in a science lab, we can't control all the variables, we don't have all the time in the world and in most cases the exact effects of the training will be different for each trainee!

I agree with many of the sentiments in Kizzy's response. I do like the court analogy and use this with clients to position the evaluation work I'm doing and help them decide what approach they require - proof or balance of probabilities. I get them to literally sign up to this just in case they come back later demanding a 'proof' based approach.

The one that aspect that I'd like to pick up on is highlighted by this quote from Kizzy:

quote:

and ask the business experts (they're in a better position than you to know what impact those other factors would have had without the L&D).

It is an assumption that the business experts actually KNOW what is going on in terms of the factors at play and the dynamics between them and consequent outcomes and impact. Most of the time, with most of the major factors, they are right. It's when they are wrong, but believe they are right, and challenge less 'expert' people on this that things can go wrong (I have seen this myself as an employee within a number of global organisations - who shall remain nameless!). It is one possible reason why managers for example, perceived as having a good track record, intelligent, experienced and well trained, get some of their decisions disastrously wrong. We see this at every level (e.g. high CEO churn rates when things change in the marketplace and the old assumptions no longer work).

We don't live in a science lab, nor can we control all the variables, or have all the time in the world, and the effects of training will be different for each employee... BUT ... there will always be some key stakeholder that is more interested in the effects of the training on results in toto rather than individual trainees, will want to know exactly which variables can be controlled and what contribution each makes to the outcome. It comes back to the courtroom analogy I guess.

For perhaps too many key stakeholders, the actual bottom line is "was it worth the time, money and effort doing this training?" and "was it good use of shareholder funds, or would they have been better advised to put their money in low yield, low risk government bonds?"