OSHA and ANSI have both adopted, to some degree, the Kirkpatrick/Phillip's model we discussed in Module 5 for evaluating the quality of safety education and training.
To review, the model contains five levels of measurement:
According to this five-level model, methods to evaluate should always include the measurement of student reaction and measure sequentially through learning, application, results, and returns. Now let's look at each of these levels of evaluation.
1. When evaluating the training program, which level of evaluation ask if there was an increase in the student's KSAs?
a. ReactionLevel One evaluation is extremely important and measures the performance training by gathering data from students about the quality of the content and presentation of the training. Below are some questions we want to answer.
This level of measurement is usually quick and very easy to perform. It doesn't take long for students to provide feedback on the training. It's not expensive to gather and analyze the data. Below are some methods to conduct Level One evaluation.
I'm sure you have all completed a training evaluation form (sometimes called a "Happy Sheet" at the end of a training class.) Sometimes the trainer may ask you to evaluate the training at some time after training has been completed. OSHAcademy courses always ask for a Level One evaluation after each final exam.
2. Which evaluation level uses post-training surveys or questionnaires to gather student feedback on the training?
a. Level 1 evaluation that measures reactionA Level Two evaluation measures what the student knows and can do right after training. The vast majority of your safety training will require Level Two evaluation. It may also measure the increase in knowledge - before and after.
What the learner knows or can do may be measured before, during and at the end of training as long as it's in the learning environment. The "learning environment" should not expose the student to actual workplace hazards. Below are the questions the program needs to ask.
Evaluation at this level is suitable for certifying employees as "initially qualified." However, Level Three evaluation will be required to certify the student as "fully qualified." (More on that coming up)
Methods to evaluate knowledge and skills at this level include:
On-the-job training (OJT) is a very effective training strategy to test both knowledge and skills. No matter the training strategy used, be sure evaluation measures are reliable and valid. They are reliable if the results are consistent. They are valid if the results reflect the knowledge and skills specified in the learning objectives.
Test Out option: Evaluation takes place while the learner is in the training environment. Additionally, it may be appropriate, in some instances, to allow learners to "test out" by demonstrating the ability to achieve course objectives without actually being required to complete training. If you allow this option, make sure learners understand test-out criteria, and be careful to ensure training complies with government regulations.
3. Evaluation at Level 2 is suitable for certifying employees as _____.
a. fully qualifiedLevel Three evaluation is interested in measuring the success that learners demonstrate in applying their newly acquired knowledge and skills to their job. Below are some questions the program will want to answer at this level of evaluation.
Observation of performance over time is the primary technique used for this level of evaluation. Evaluation takes place at some time (days, weeks, months) after the learner leaves the training environment. Typically, a trainer or supervisor will observe the employee at work and rate his or her performance against learning objectives. Certification at this level may be used to verify an employee is "fully qualified." Below are important reasons to include Level 3 evaluation.
Information from each prior level serves as a base for the next level's evaluation. It's also important to understand that measurement of employee behavior change typically requires cooperation and skill of line-managers.
4. Which of the five levels of evaluation looks at the success that learners demonstrate in applying their newly acquired knowledge and skills to their job?
a. Level 1: ReactionLevel Four evaluation represents a change of focus. Now we're interested in the degree to which training has had an impact or effectively contributed to the overall success of the company. The performance of employees who have received training is usually contrasted with the performance of a control group that has not had the training. Both leading and trailing indicators are measured in Level Four evaluation.
It's important to remember that we need to measure these variables both before and after the training has occurred.
Although Level Four evaluation is not required by OSHA standards, it is required by ANSI/ASSE Z490.1-2016. Again, it's important to understand the difference between Level Two/Three evaluation and Level Four evaluation: in Level Four evaluation we're no longer measuring the learner or the training process, but rather the impact of safety training on the organization.
Tip: You'll probably be asked about this difference on the final exam :-)
5. Which of the five levels of evaluation looks at the degree to which training has had an impact or effectively contributed to the overall success of the company?
a. Level 1: ReactionDr. Jack Phillips was not satisfied with Kirkpatrick's four levels. He believed that only by conducting an evaluation to determine the Return on Investment (ROI), could a company discover the monetary or financial benefits of the training program compared to the cost of implementing the training.
Thus, the fifth level of training evaluation is developed by collecting Level Four results, converting the data to monetary values, and comparing them to the cost of the training program to represent the return on training investment.
For instance, if the benefits (savings) due to fewer accident costs and workers compensation payments for last year was $400,000 and the cost of conducting training is $40,000, the training ROI (%) for last year would be ($400,000-$40,000)/$40,000)x 100 or 900%. In other words, last year the company saved NINE-TIMES the cost of training ($360,000/$40,000). Now that's ROI!!!
6. Which of the five levels of evaluation looks at the degree to which training has had an impact on the company's training ROI?
a. Level 2: LearningRead the material in each section to find the correct answer to each quiz question. After answering all the questions, click on the "Check Quiz Answers" button to grade your quiz and see your score. You will receive a message if you forgot to answer one of the questions. After clicking the button, the questions you missed will be listed below. You can correct any missed questions and check your answers again.
This video explains the Phillips Five Level ROI Model for Training Evaluation. Organizations around the world are spending billions of dollars on trainings. They need to know if this money is spend effectively and is providing any measurable benefits. That’s the reason training evaluation is required.