Epic Think Tank

Evaluation and ROI

The advent of e-learning and blended learning has shaken up the way training departments are required to operate. The most widely used form of training delivery, classroom, is cheap to originate and expensive to deliver: with e-learning things are the other way around - and this encounter with front-loaded costs has brought a whole new set of requirements to the learning professional's role. Sound business cases need to be created, and positive outcomes demonstrated in tangible terms.

But evaluation is difficult, expensive and time-consuming.

These new requirements are exposing a well-recognised weakness in the traditional model of organisational learning. Organisations don't evaluate effectively. By and large they don't have the tools for evaluating, the budgets for evaluating - nor, in most cases, the conceptual wherewhithal to go about originating effective processes for evaluating.

Research data from Training magazine in the US shows how the incidence of evaluation drops off sharply through the four Kirkpatrick levels.

Many people believe that Kirkpatrick Level Four is best expressed as a return-on-investment (ROI) calculation - i.e. how much did the business benefit from the training in cash money? However, here the problem is not so much the difficulty and cost of the exercise, so much as the multiplicity of ways that the calculation can be done.

If we take the three key steps in calculating ROI to be:

• Collecting data
• Isolating the effects of data
• Converting data to monetary value

We find that there are at least a dozen ways to do each of these steps. And a dozen, times a dozen, times a dozen adds up to more than a thousand different ways to make the calculation!

So it's a difficult business - but at the same time an essential one.

This Think tank examined the issues around evaluation and ROI - drawing on the
experience of those who have to deal with the practical issues on the ground, addressing the following questions:

• Why evaluate?
• How and when should evaluation best be undertaken?
• Given the limitations of evaluation, how do we prove the value of learning to the organisation?

(See also Epic white papers: Evaluation of learning, E-learning: return on investment)

top

Why evaluate?

Some see the increasing requirement on training departments to evaluate everything that moves as somewhat odd. After all, within the wider context of Human Resources, one wouldn't evaluate a new share scheme, for instance, or a new admin system. The requirement seems to become even more de rigeur when it comes to e-learning - yet, how many face-to-face courses get effectively evaluated?

Still there is a widespread feeling, reflected around the table in this discussion, that 'what gets measured gets done' and what we can't measure, we can't manage. An important part of the value proposition of e-learning, as propounded back in the heady days around the turn of the century, was a promise of more 'manageable' learning - with the provision of a higher level of management information almost a by-product of delivering training online. Ironically however, according to one delegate, evaluation was almost an afterthought back then, with everybody rushing to do their pilots. It is only now, in tighter times, that the focus is turning to more rigourous evaluation.

Many organisations place mandatory requirements on those managing training and development to evaluate. The CIPD (under whose auspices this Think Tank was held) conducts a feasibility study and an evaluation for every course they run. In the public sector, especially, it is often an entrenched feature of the mindset (one delegate, from education, said she would not think of doing a programme without the thought that it will be evaluated). Within the Ministry of Defence (MOD), where training contracts can be part of larger tenders with a lifespan of twenty years or more, payment is often dependent on the results of evaluation, and managers can be mandated to evaluate to all four Kirpatrick levels.

On large, highly visible projects, where public money is being spent, this is no doubt as it should be. However, a rigorous commitment to evaluation comes with a hefty price tag. In some cases, the budget for procurement and evaluation on MOD contracts can equal that spent on the programme itself, which would certainly not be practical in the corporate arena.

There are circumstances, as well, where severe drawbacks can result from an over-reliance on quantitative evaluation.

It can focus attention on the wrong things. For instance, one delegate from a health background recounted the experience where an award-winning e-learning programme, which had been extraordinarily well-received by its target learners, nevertheless achieved low scores for completions in evaluation. It turned out that the learners, highly time-pressured health professionals, were using the programme to get the 'nuggets' of just-in-time learning they needed to help them do their jobs more effectively, but they ignored the evaluation tests and rarely bothered to work through the programme from beginning to end. The programme was clearly a success on the ground, and having a noticeable beneficial effect, but the criteria set to evaluate it failed to pick up on this success.

Similarly, evaluation systems set up to assess fact-based learning might not cope so well when it comes to learning around behavioural change.

In the week before this Think Tank a BBC Documentary 'The Secret Policeman', filmed with the use of undercover cameras, had exposed racist attitudes in the police during a training session at the national training centre in Warrington. It was pointed out by one of our delegate that all of the police involved (six of whom subsequently resigned) could have been said to have received the training - showing that in such cases a mere 'tick in the box' proves next to nothing!

top

How should we evaluate?

The soft stuff is the hard stuff. Clearly it is important that evaluation should be sophisticated enough to be a real measure of a programme's success (or otherwise). As one delegate put it, 'we can't assume that what we are talking about with evaluation is a number. An organisation may be hitting its targets with regard to training delivered but at the same time can have an atrocious culture… with bad bullying, lack of leadership, etc.'

The culture of an organisation is not an intangible thing: it is clear as day to the people who work in that organisation, and improvements can often be positively felt. Nevertheless, such changes often elude attempts to evaluate the means that were used to bring them about.

Another problem for evaluation lies in timescale. Even where positive change is taking place, it may take time to show up in evaluation. Where training is used to address issues of organisational culture and behaviours, it can take between three and five years to show a result - which is outside the scope of many evaluations.

Various responses were seen to this problem. Some organisations place emphasis on 360° feedback as a good way of assessing whether training has had an effect on the way a person works. The evidence of peers and subordinates often shows up changes in a person that they aren't aware of (or at least don't report) themselves.

Once you scratch the surface it turns out that many organisations use a variety of more or less rough-and-ready evaluative measures, especially when it comes to areas like staff selection. One of the criteria allegedly used by the RAF to decide who should and should not be eligible to train as an officer was to pose the question, 'would you want to stand at the bar and have a beer with this guy?'.

Others espouse more rigorous methods of drilling down into the work situation such as social network analysis - applied over a period of two or more years - a form of evaluation that looks holistically, considering stakeholders including the learner, the person who signed off the budget, and a whole hierarchy of people within the organisation potentially impacted by the programme.

Then again there are those, such as Jack Welch of GE, who take a more ruthlessly pragmatic view and don't try too hard to disentangle cause from effect, 'so long as the number looks good'!

Most evaluation efforts of the more sophisticated kind focus on change of one kind or another. However, we also heard from one newly-founded public sector organisation during this Think Tank that has the luxury of starting from scratch - and was using the opportunity of a new induction programme to ask questions such as, 'what do we want the culture of our company to be? Here a staff questionnaire, carried out yearly, is a qualitative tool that can be used year on year to monitor the success of any learning focused on bringing the desired culture about.

top

The business case and evaluation

Up to now in this report we have focused largely on evaluation carried out 'after the fact'. However a major part of our Think Tank discussion concerned the creation of business cases, which come at the beginning of the life-cycle of a programme and are there principally to evaluate the business opportunity.

Although these two events in a sense book-end the learning programme, it is feared that there is often a disconnect between the two. 'Does anyone ever look at the business case 12 months down the line?'

We heard examples of programmes where the business case, with its ROI calculations, was indeed the basis of evaluation (mostly in the public sector, it should be said). However, many people felt that a type of forgetting went on between the two events.

For some it seemed only right that this should be the case, since the reasons for doing a programme, in terms of the official justification for getting the funding, were often entirely different from the benefits, many of them unforeseen, that would eventually accrue to the organisation.

'You need to produce the business case in the flavour that the person who is paying for it understands,' one delegate remarked; 'many learning projects operate on the stealth level - once you get the business case passed, everyone forgets about it'.

So the business case is there to get the funding.

Since this seems like a fairly cynical exercise - and since no correlation exists, apparently, between the fact of a programme having had a business case and its ultimately being perceived as a success - are business cases, then, a waste of time?

Well, no. If anything goes wrong with a programme, it certainly helps to have a business case handy, to show the reasons behind your decision-making (business case as parachute!). On a more positive note, doing the business case for a programme makes you focus. 'If it was your own money going into the programme,' asked one delegate; 'would you spend it without doing a business case?

And while it is undoubtedly true that many business cases are produced solely as post-rationalisations to justify 'gut feeling', or bright ideas arrived at over dinner, it would be dangerous to gainsay the importance of inspiration to this process: 'You can't produce a good business case if your heart's not in it'.

top

Conclusions

Looking at the issues around evaluation and ROI highlights the fact of how many different levels within the organisation are affected by programmes of learning - and touches on a question often asked in these Think Tank discussions: who, ultimately is the learning for?

Should evaluation focus on the organisational benefits of learning, or the benefit to individuals? Clearly the business case is there to assess the organisation's risks and potential benefits in a given programme. It follows logically that after-the-fact evaluation should, at the very least, assess to what extent the programme in question delivered the hoped-for organisational benefits and avoided the highlighted risks.

However, evaluation that ignores the micro-level and fails to engage with any qualitative measurement of positive change in attitudes and behaviours can run the risk of leaving a substantial tranche of what are, ultimately, organisational benefits out of the picture.

If the business case is there to get the funding, post-programme evaluation asked the question: would I do that again? In order to provide a meaningful answer to this question, evaluation should be effective enough to paint an accurate picture of what actually happened in between.

With the passage of time, perspectives change, priorities shift. When setting the critical success factors for a programme of learning that might take two or three year to have effect, it is worth bearing in mind what might happen along the way.

Without straying into the realm of zen parables, it's worth pointing out that what we get out of our learning isn't always the thing that we went into it for!

See also: