So you know how to develop a mobile training reinforcement plan.
But do you know how to measure it?
You might think you’ve just created the perfect training reinforcement plan to back up your company’s latest safety training session – can you prove it? Do you have the data to support such a claim? The reality is, without reinforcement metrics, the answer to these questions is NO.
Measure these three things and you’ll have all the numbers you need:
- Learner engagement
#1: Learner engagement
Learner engagement is how much a learner takes part in retrieval activities. This measure is crucial because all others depend on it: no engagement means no outcome and no mastery of competencies.
So how can we figure out a learner’s engagement level? By calculating frequency of use and the amount of time spent performing retrieval practices (ideally a blend of the two).
Outcome is how a learner performs on tests and quizzes. You want to know the number of questions answered correctly, accumulated quiz scores, how many awards were earned and what a learner’s position is on a leaderboard.
Use outcomes to boost engagement levels and peak learner interest: make a game out of it using test scores and competitive designs. Then incentivize training reinforcement retrieval practices by posting various team leaderboards.
Mastery is how a learner demonstrates knowledge of a particular competency. Most importantly, we’re talking about the long term – not a short period of time. So then, lasting mastery is proficiency in a balanced (across competencies) and sustained (over time) way. At any given time, an employee has either reached the mastery level for each competency or not. The benchmark between the two can be whatever you want. For example, 3 quiz questions in a row answered correctly means mastery has been achieved. Or, mastery is not achieved until 3 out of 5 questions are answered correctly over a 2-week period. This is entirely up to you, and the formula can change and be refined over time, but it is critical to build this into the design of your training reinforcement program at a granular level.
To accurately measure mastery, each retrieval practice question in your training reinforcement plan must be mapped to one or more competency. This way, you can measure how often a learner correctly answers questions associated with a specific competency over a selected period of time. Then you can use these measures to determine which employees have achieved mastery for each competency and how many have maintained that mastery.
Measurements are two-dimensional
When you’re implementing these metrics, be sure to view them on two dimensions:
- Self – How do each employee’s metrics change over time?
- Cohort – How do each employee’s metrics compare to the average of the group?
By doing both, you give yourself insight not only into an individual learner’s path but also how this path compares to that of others in the same program. You will also have powerful and actionable data into the quality of your training program and a good indication of what materials may need to be taught differently. As an example, if the majority of the cohort continually struggles with the same competency, then perhaps the training itself needs to be revisited.
The Silver Bullet
The most important part of measuring success is, of course, the degree to which mastery of competencies positively impacts the desired behavior change or on-the-job performance (for example, an increase in sales, greater adoption of a new business process or tool, or increased efficiency in team functioning…) These objectives also need to be clearly articulated and built into the ROI of the training. Some things are easier to measure than others of course, and in many cases ensuring that program and/or user data is reliable may require the use of baseline testing for comparison, or A/B testing between a control group and a group with a training reinforcement plan. This is something you will need to evaluate at program level and refine over time.
Use these measurements to your advantage
Whatever the desired learning goal, you are now in a much better position to measure user engagement and performance at a more substantial level than ever before, thanks to the analytics provided by a mobile training reinforcement program.
A training reinforcement plan is a work in progress. Anyone who thinks they’ve hit the nail on the head with the first design is naively mistaken. Even the tiniest of tweaks can make significant improvements in your learners’ ability to retain and apply the information they’ve been taught. With the right metrics in place, you can effectively evaluate, tweak and perfect your reinforcement plan over time.
And this works wonders for your ROI.
For a more comprehensive review of Reinforcement Metrics, read our article on "How to Use Analytics to Optimize the Effectiveness of Your Training Reinforcement Plan."