OSHA and ANSI have both adopted, to some degree, the Kirkpatrick/Phillip's model we discussed in Module 5 for evaluating the quality of safety education and training.
To review, the model contains five levels of measurement:
According to this five-level model, methods to evaluate should always include the measurement of student reaction and measure sequentially through learning, application, results, and returns. Now let's look at each of these levels of evaluation.
Level One evaluation is extremely important and measures the performance training by gathering data from students about the quality of the content and presentation of the training. Below are some questions we want to answer.
This level of measurement is usually quick and very easy to perform. It doesn't take long for students to provide feedback on the training. It's not expensive to gather and analyze the data. Below are some methods to conduct Level One evaluation.
I'm sure you have all completed a training evaluation form (sometimes called a "Happy Sheet" at the end of a training class.) Sometimes the trainer may ask you to evaluate the training at some time after training has been completed. OSHAcademy courses always ask for a Level One evaluation after each final exam.
A Level Two evaluation measures what the student knows and can do right after training. The vast majority of your safety training will require Level Two evaluation. It may also measure the increase in knowledge - before and after.
What the learner knows or can do may be measured before, during and at the end of training as long as it's in the learning environment. The "learning environment" should not expose the student to actual workplace hazards. Below are the questions the program needs to ask.
Evaluation at this level is suitable for certifying employees as "initially qualified." However, Level Three evaluation will be required to certify the student as "fully qualified." (More on that coming up)
Methods to evaluate knowledge and skills at this level include:
On-the-job training (OJT) is a very effective training strategy to test both knowledge and skills. No matter the training strategy used, be sure evaluation measures are reliable and valid. They are reliable if the results are consistent. They are valid if the results reflect the knowledge and skills specified in the learning objectives.
Test Out option: Evaluation takes place while the learner is in the training environment. Additionally, it may be appropriate, in some instances, to allow learners to "test out" by demonstrating the ability to achieve course objectives without actually being required to complete training. If you allow this option, make sure learners understand test-out criteria, and be careful to ensure training complies with government regulations.
Level Three evaluation is interested in measuring the success that learners demonstrate in applying their newly acquired knowledge and skills to their job. Below are some questions the program will want to answer at this level of evaluation.
Observation of performance over time is the primary technique used for this level of evaluation. Evaluation takes place at some time (days, weeks, months) after the learner leaves the training environment. Typically, a trainer or supervisor will observe the employee at work and rate his or her performance against learning objectives. Certification at this level may be used to verify an employee is "fully qualified." Below are important reasons to include Level 3 evaluation.
Information from each prior level serves as a base for the next level's evaluation. It's also important to understand that measurement of employee behavior change typically requires cooperation and skill of line-managers.
Level Four evaluation represents a change of focus. Now we're interested in the degree to which training has had an impact or effectively contributed to the overall success of the company. The performance of employees who have received training is usually contrasted with the performance of a control group that has not had the training. Both leading and trailing indicators are measured in Level Four evaluation.
It's important to remember that we need to measure these variables both before and after the training has occurred.
Although Level Four evaluation is not required by OSHA standards, it is required by ANSI/ASSE Z490.1-2016. Again, it's important to understand the difference between Level Two/Three evaluation and Level Four evaluation: in Level Four evaluation we're no longer measuring the learner or the training process, but rather the impact of safety training on the organization.
Tip: You'll probably be asked about this difference on the final exam :-)
Dr. Jack Phillips was not satisfied with Kirkpatrick's four levels. He believed that only by conducting an evaluation to determine the Return on Investment (ROI), could a company discover the monetary or financial benefits of the training program compared to the cost of implementing the training.
Thus, the fifth level of training evaluation is developed by collecting Level Four results, converting the data to monetary values, and comparing them to the cost of the training program to represent the return on training investment.
For instance, if the benefits (savings) due to fewer accident costs and workers compensation payments for last year was $400,000 and the cost of conducting training is $40,000, the training ROI (%) for last year would be ($400,000-$40,000)/$40,000)x 100 or 900%. In other words, last year the company saved NINE-TIMES the cost of training ($360,000/$40,000). Now that's ROI!!!
Before beginning this quiz, we highly recommend you review the module material. This quiz is designed to allow you to self-check your comprehension of the module content, but only focuses on key concepts and ideas.
Read each question carefully. Select the best answer, even if more than one answer seems possible. When done, click on the "Get Quiz Answers" button. If you do not answer all the questions, you will receive an error message.