OSHA and ANSI have both adopted Kirkpatrick's "Four Levels of Evaluation" as the model for evaluating the quality of safety education and training. It's a good model and is used widely.
The model we'll be discussing in this course is slightly modified in that it includes a fifth level that I believe may be quite helpful in the training program evaluation process.
According to this five-level model, methods to evaluate should always include the measurement of student reaction and measure sequentially through learning, application, results and finally return. Now let's take a look at each of these levels of evaluation.
Level One evaluation is extremely important and measures the performance training by gathering data from students about the quality of the content and presentation of the training. Below are some questions we want to answer.
This level of measurement is usually quick and very easy to perform. It doesn't take long for students to provide feedback on the training. It's not expensive to gather and analyze the data. Below are some methods to conduct Level One evaluation.
I'm sure you have all completed a training evaluation form (sometimes called a "Happy Sheet" at the end of a training class.) Sometimes the trainer may ask you to evaluate the training at some time after training has been completed. OSHAcademy courses always ask for a Level One evaluation after each final exam.
A Level Two evaluation measures what the student knows and can do right after training. The vast majority of your safety training will require Level Two evaluation. It may also measure the increase in knowledge - before and after.
What the learner knows or can do may be measured before, during and at the end of training as long as it's in the learning environment. The "learning environment" should not expose the student to actual workplace hazards. Below are the questions the program needs to ask.
Evaluation at this level is suitable for certifying employees as "initially qualified." However, Level Three evaluation will be required to certify the student as "fully qualified." (More on that coming up)
Methods to evaluate knowledge and skills at this level include:
On-the-job training (OJT) is a very effective training strategy to test both knowledge and skills. No matter the training strategy used, be sure evaluation measures are reliable and valid. They are reliable if the results are consistent. They are valid if the results reflect the knowledge and skills specified in the learning objectives.
Test Out option: Evaluation takes place while the learner is in the training environment. Additionally, it may be appropriate, in some instances, to allow learners to "test out" by demonstrating the ability to achieve course objectives without actually being required to complete training. If you allow this option, make sure learners understand test-out criteria, and be careful to ensure training complies with government regulations.
Level Three evaluation is interested in measuring the success that learners demonstrate in applying their newly acquired knowledge and skills to their job. Below are some questions the program will want to answer at this level of evaluation.
Observation of performance over time is the primary technique used for this level of evaluation. Evaluation takes place at some time (days, weeks, months) after the learner leaves the training environment. Typically, a trainer or supervisor will observe the employee at work and rate his or her performance against learning objectives. Certification at this level may be used to verify an employee is "fully qualified." Below are important reasons to include Level 3 evaluation.
Information from each prior level serves as a base for the next level's evaluation. It's also important to understand that measurement of employee behavior change typically requires cooperation and skill of line-managers.
Level Four evaluation represents a change of focus. Now we're interested in the degree to which training has had an impact or effectively contributed to the overall success of the company. The performance of employees who have received training is usually contrasted with the performance of a control group that has not had the training. Both leading and trailing indicators are measured in Level Four evaluation.
It's important to remember that we need to measure these variables both before and after the training has occurred.
Although Level Four evaluation is not required by OSHA standards, it is required by ANSI/ASSE Z490.1-2016. Again, it's important to understand the difference between Level Two/Three evaluation and Level Four evaluation: in Level Four evaluation we're no longer measuring the learner or the training process, but rather the impact of safety training on the organization.
Tip: You'll probably be asked about this difference on the final exam :-)
Wait a minute! I bet you thought there were only four levels of evaluation. Well, there are only four levels required by ANSI standards.
Back in the 1990's Jack Phillips, of the ROI Institute, developed what has been referred to as the fifth level of evaluation. To obtain a true training Return on Investment (ROI) evaluation, the monetary or financial benefits of the training program should be compared to the cost of implementing the training.
Thus, the fifth level of training evaluation is developed by collecting Level Four data, converting the data to monetary values, and comparing them to the cost of the training program to represent the return on training investment.
I'm sure you'll agree that it may be beneficial for you to evaluate how safety training impacts the organization's overall return on investment (ROI).
For instance, if the benefits (savings) due to fewer accident costs and workers compensation payments for last year was $400,000 and the cost of conducting training is $40,000, the training ROI (%) for last year would be ($400,000-$40,000)/$40,000)x 100 or 900%. In other words, last year the company saved NINE-TIMES the cost of training ($360,000/$40,000). Now that's ROI!!!
Before beginning this quiz, we highly recommend you review the module material. This quiz is designed to allow you to self-check your comprehension of the module content, but only focuses on key concepts and ideas.
Read each question carefully. Select the best answer, even if more than one answer seems possible. When done, click on the "Get Quiz Answers" button. If you do not answer all the questions, you will receive an error message.