Five Levels of Evaluation

Kirkpatrick/Phillips Model

OSHA and ANSI have both adopted, to some degree, the Kirkpatrick/Phillip's model we discussed in Module 5 for evaluating the quality of safety education and training.

To review, the model contains five levels of measurement:

Kirkpatrick - Levels 1-4

1. Reaction - did the learner like the safety training?
2. Learning - did the training increase the learner's safety knowledge, skills, and abilities (KSAs)?
3. Application - did the learner successfully apply the KSAs to the job?
4. Results - did the safety training have a positive impact on business key performance indicators (KPIs)?

Phillips - Level 5

5. Return - did the company show a positive return on the training investment (ROI) expressed as a cost/benefit ratio?

According to this five-level model, methods to evaluate should always include the measurement of student reaction and measure sequentially through learning, application, results, and returns. Now let's look at each of these levels of evaluation.

Level 1 Evaluation: Measures Reaction

Did the learners like the training?
(Click to enlarge)

Level One evaluation is extremely important and measures the performance training by gathering data from students about the quality of the content and presentation of the training. Below are some questions we want to answer.

• Did the student enjoy the training?
• Did they think training was applicable and useful?
• Was the training a good use of their time?
• Did they like the facilities, room setup, etc...?

This level of measurement is usually quick and very easy to perform. It doesn't take long for students to provide feedback on the training. It's not expensive to gather and analyze the data. Below are some methods to conduct Level One evaluation.

• Complete 'Happy Sheets'
• Complete feedback forms based on their personal reaction to the training experience
• Provide verbal reaction to the instructor
• Complete post-training surveys or questionnaires
• Complete online evaluation or grading by students
• Provide verbal or written reports to managers back at work

I'm sure you have all completed a training evaluation form (sometimes called a "Happy Sheet" at the end of a training class.) Sometimes the trainer may ask you to evaluate the training at some time after training has been completed. OSHAcademy courses always ask for a Level One evaluation after each final exam.

Level 2 Evaluation: Learning

Did the students learn anything?
(Click to enlarge)

A Level Two evaluation measures what the student knows and can do right after training. The vast majority of your safety training will require Level Two evaluation. It may also measure the increase in knowledge - before and after.

What the learner knows or can do may be measured before, during and at the end of training as long as it's in the learning environment. The "learning environment" should not expose the student to actual workplace hazards. Below are the questions the program needs to ask.

• Did the student learn what was intended to be taught?
• Did the student experience what was intended for them to experience?
• What is the extent of improvement in the trainee' knowledge or skill after the training?

Evaluation at this level is suitable for certifying employees as "initially qualified." However, Level Three evaluation will be required to certify the student as "fully qualified." (More on that coming up)

Methods to evaluate knowledge and skills at this level include:

• written or verbal pre/post training exams to measure knowledge; and
• interview or observation of performance may also be used to measure skills.

On-the-job training (OJT) is a very effective training strategy to test both knowledge and skills. No matter the training strategy used, be sure evaluation measures are reliable and valid. They are reliable if the results are consistent. They are valid if the results reflect the knowledge and skills specified in the learning objectives.

Test Out option: Evaluation takes place while the learner is in the training environment. Additionally, it may be appropriate, in some instances, to allow learners to "test out" by demonstrating the ability to achieve course objectives without actually being required to complete training. If you allow this option, make sure learners understand test-out criteria, and be careful to ensure training complies with government regulations.

Level 3 Evaluation: Application

Can workers apply what they learned?
(Click to enlarge)

Level Three evaluation is interested in measuring the success that learners demonstrate in applying their newly acquired knowledge and skills to their job. Below are some questions the program will want to answer at this level of evaluation.

• Did the student put their learning into use back on the job?
• Was there noticeable and measurable improvement in the performance of the student back in their job?
• Was the change in the students behavior and their new level of knowledge sustained?
• Would the student be able to transfer what they learned to a co-worker?

Observation of performance over time is the primary technique used for this level of evaluation. Evaluation takes place at some time (days, weeks, months) after the learner leaves the training environment. Typically, a trainer or supervisor will observe the employee at work and rate his or her performance against learning objectives. Certification at this level may be used to verify an employee is "fully qualified." Below are important reasons to include Level 3 evaluation.

• Measures long-term retention of knowledge: When you ask the employee to explain procedures at some point in time after training, you can determine how well the employee has retained necessary knowledge.
• Measures skills after training: You can determine the degree to which the employee has retained the skills necessary to perform the procedure.
• Validates the safety training: Successful performance in the work environment is a solid indicator that the safety training was adequate.
• Validates the safety culture: Successful performance at work generally indicates the safety management system and culture are supporting the safety training program.
• It's efficient: Supervisors can perform this level of evaluation during the normal course of their daily supervision. No special procedure is required.

Information from each prior level serves as a base for the next level's evaluation. It's also important to understand that measurement of employee behavior change typically requires cooperation and skill of line-managers.

Level 4 Evaluation: Results

What were the results of training?
(Click to enlarge)

Level Four evaluation represents a change of focus. Now we're interested in the degree to which training has had an impact or effectively contributed to the overall success of the company. The performance of employees who have received training is usually contrasted with the performance of a control group that has not had the training. Both leading and trailing indicators are measured in Level Four evaluation.

Leading Indicators

• Attitudes: What did surveys and interviews reveal about employee thoughts, opinions, and attitudes?
• Conditions: What were the total and average number of hazardous conditions (findings) found during safety inspections?
• Behaviors: What was the rate of unsafe behaviors experienced? What was the rate at which near miss reports occurred?
• Activities: How much safety training was performed? How often are safety meetings held?

Trailing Indicators

• Injury rates: Did injury rates go up or down after training?
• Accident costs: What was the impact of training on the average and total amount spent on medical expenses?
• Workers' compensation costs: Did workers' compensation rates go up after training?
• OSHA violations: What was the change in the frequency and results of OSHA inspections?

It's important to remember that we need to measure these variables both before and after the training has occurred.

Although Level Four evaluation is not required by OSHA standards, it is required by ANSI/ASSE Z490.1-2016. Again, it's important to understand the difference between Level Two/Three evaluation and Level Four evaluation: in Level Four evaluation we're no longer measuring the learner or the training process, but rather the impact of safety training on the organization.

Level 5 Evaluation: Returns

Dr. Jack Phillips was not satisfied with Kirkpatrick's four levels. He believed that only by conducting an evaluation to determine the Return on Investment (ROI), could a company discover the monetary or financial benefits of the training program compared to the cost of implementing the training.

Thus, the fifth level of training evaluation is developed by collecting Level Four results, converting the data to monetary values, and comparing them to the cost of the training program to represent the return on training investment.

For instance, if the benefits (savings) due to fewer accident costs and workers compensation payments for last year was $400,000 and the cost of conducting training is $40,000, the training ROI (%) for last year would be ($400,000-$40,000)/$40,000)x 100 or 900%. In other words, last year the company saved NINE-TIMES the cost of training ($360,000/$40,000). Now that's ROI!!!

Instructions

Before beginning this quiz, we highly recommend you review the module material. This quiz is designed to allow you to self-check your comprehension of the module content, but only focuses on key concepts and ideas.

Read each question carefully. Select the best answer, even if more than one answer seems possible. When done, click on the "Get Quiz Answers" button. If you do not answer all the questions, you will receive an error message.

Good luck!