Evaluating the Training

Evaluate RKSAs

To make sure that the training program is accomplishing its goals, an evaluation of the training can be valuable. Training should have, as one of its critical components, a method of evaluating the student's RKSAs:

• The Reaction of students to training - how the student felt in a relative positive or negative manner.
• The impact of training on student Knowledge - an organized body of information, usually factual or procedural in nature.
• The impact change in student Skills - the proficient manual, verbal, or mental manipulation of data or things.
• The change in Abilities - The power or capacity to perform a task.

The evaluation also include an analysis of the quality of the training process and impact of training on the organization.

It's important that a written plan for evaluating the training be developed when the course objectives and content are developed. It should not be delayed until the training has been completed.

Purpose of Training Evaluation

Evaluation will help determine the amount of learning achieved by training and whether an employee’s performance has improved on the job as result of training. Among the methods of evaluating training are:

1. Employee opinion. Questionnaires or informal discussions with employees can help employers determine the relevance and appropriateness of the training program.
2. Supervisor observations. Supervisors are in good positions to observe an employee’s performance both before and after the training and note improvements or changes.
3. Workplace improvements. The ultimate success of a training program may be changes throughout the workplace that result in reduced injury or accident rates.

However it is conducted, an evaluation of training can give employers the information necessary to decide whether the employees achieved the desired results, and whether the training session should be offered again at some future date.

ANSI Z490.1 Training Evaluation Guidelines

As outlined in detail in ANSI Z490.1, Section 3.4, the employer should design evaluation strategies to measure the performance of the safety training program. To do this, one or more evaluation techniques may be used to evaluate training. The evaluation techniques mentioned in the ANSI standard mirror those developed by Donald Kirkpatrick, author of Evaluating Training Programs: The Four Levels. As the title of the book implies, he developed a four-level model for evaluating training. The first three levels of evaluation, those that you'll be involved with most of the time, measure the impact of training on the leaner. The fourth level measures the impact on the organization. Let's take a look at these four levels of evaluation.

The Four Levels of Evaluation

Level One Evaluation: Measures Reaction

Level One evaluation measures the performance of the training event by gathering data from students and others about the quality of the training process. We're interested in learning what the trainee, or other evaluator, thought about the content and trainer's presentation of the training. Learner evaluation of training typically occurs at the end of training, but may also be conducted during the training session or at some time after the completion of training. The typical tool to measure reaction is a survey.

Level Two Evaluation: Measures knowledge, skills, and abilities

Level Two evaluation measures what the learner knows and can do after training. The vast majority of your safety training will require this level of evaluation. What the learner knows or can do can be measured during and at the end of training as long as it's in the learning environment that does not exposure the learner to actual hazards. Trainer evaluation assesses the extent to which students have advanced in skills, knowledge, or abilities (SKAs). Evaluation at this level is suitable for certifying employees as "initially qualified." Methods to evaluate SKAs at this level include:

• written or verbal pre/post training exams,
• skill/performance demonstrations

On-the-job training (OJT) is a very effective training strategy to test SKAs. No matter the training strategy used, be sure evaluation measures are reliable and valid. They are reliable if the results are consistent. They are valid if the results reflect the SKAs specified in the learning objectives.

Test Out option. Evaluation takes place while the learner is in the training environment. Additionally, it may be appropriate, in some instances, to allow learners to "test out" by demonstrating the ability to achieve course objectives without actually being required to complete training. If you allow this option, make sure learners understand test-out criteria, and be careful to ensure training complies with government regulations.

Level Three Evaluation: Measures application of KSAs

Level Three evaluation is interested in measuring success the learner has had in applying acquired knowledge and skill to their job. It answers the question - Are the newly acquired skills, knowledge, or abilities being successfully used in the workplace? Observation of performance is the technique used for this level of evaluation. Evaluation takes place at some time (days, weeks, months) after the learner leaves the training environment. Typically, a trainer or supervisor will observe the employee at work and rate his or her performance against learning objectives. Certification at this level may be used to certify an employee "fully qualified." It's important to include Level 3 evaluation because it:

• Measures long-term retention of knowledge. When the evaluator asks the employee to explain procedures at some point in time after training, we can determine how well the employee has retained necessary knowledge.
• Measures skills after training. Through observation, the evaluator can determine the degree to which the employee has retained the skills necessary to perform the procedure.
• Validates the safety training. Successful performance in the work environment is a solid indicator that the safety training received by the employee was adequate.
• Validates the safety culture. Successful performance at work generally indicates the safety management system and culture is supporting the safety training program.

According to the Four-Level Evaluation model, techniques (tests) to evaluate should always include level one techniques and move sequentially through levels two, three, and four. Information from each prior level serves as a base for the next level's evaluation.

Level Four Training - Measures Results

Level Four evaluation represents a change of focus. At this level, we're interested in the degree to which training has effectively contributed to the overall success of the company rather than the learner. At this level, the performance of employees who have received training is usually contrasted with the performance of a control group. Although Level Four evaluation is not required by OSHA standards, it's still considered quite important and should be thought of as a wise business practice. We're no longer measuring the learner or the presentation, we're evaluating the impact of safety training on a number of results, including:

• compliance behaviors
• hazard identification and control skills
• frequency of non-injury incidents
• workers compensation claim costs
• direct/indirect accident costs
• production costs
• turnover and absenteeism
• frequency of grievances
• profitability
• morale

ANSI Z490.1 Guidelines for evaluating training programs, process, and results

ANSI Z490.1 recommends evaluating three important elements of a safety training program.

• How the training program is managed
• The quality of the training process
• The results of training

Let's look at each of these three important elements more closely.

Training program management. Training works best when it's designed and performs as part of an integrated system management system rather than a series of unrelated training sessions. Elements to evaluate include:

• Responsibility and accountability
• Quality of facilities and equipment
• Adequacy of program development
• Quality of delivery
• Completeness of documentation and records
• Adequacy of the safety training program evaluation process

The training process. Training should be conducted using a systematic process that includes a needs assessment, objectives, course materials, lesson plans, evaluation strategies, and criteria for successful completion. Areas of emphasis include:

• Training goals
• Learning environment
• Learning objectives
• Training effectiveness

Training results. By evaluating the results of training, it's possible to make improvements to existing plans and gain awareness of the need for new training. Items to evaluate include:

• Quality of the training action-plan
• Adequacy of long-term planning
• Use of needs assessment
• Appropriateness of prioritizing training
• Adequacy of support and funding

OSHA Guidelines for Training Program Evaluation

OSHA's Occupational Safety and Health Standards, CFR 29, Subpart: H - Hazardous Materials 1910.120 App E Training Curriculum Guidelines, serves as an effective guideline for evaluating all safety training. According to these guidelines:

• The employer should conduct or direct an annual written audit of the safety training program.
• Program modifications to address deficiencies, if any, should be documented, approved, and implemented by the training provider.
• The audit and the program modification documents should be maintained at the training facility.

Suggested Program Quality Control Criteria

Factors listed here are suggested criteria for determining the quality and appropriateness of employee health and safety training.

A. Training Plan. Adequacy and appropriateness of the training program's curriculum development, instructor training, distribution of course materials, and direct student training should be considered, including:

• The duration of training, course content, and course schedules/agendas
• The different training requirements of the various target populations, as specified in the appropriate generic training curriculum
• The process for the development of curriculum, which includes appropriate technical input, outside review, evaluation, program pretesting
• The adequate and appropriate inclusion of hands-on, demonstration, and instruction methods
• Adequate monitoring of student safety, progress, and performance during the training

B. Program management, Training Director, staff, and consultants. Adequacy and appropriateness of staff performance and delivering an effective training program should be considered, including:

1. Demonstration of the training director's leadership in assuring quality of health and safety training
2. Demonstration of the competency of the staff to meet the demands of delivering high quality hazardous waste employee health and safety training
3. Clearly defined staff duties including the relationship of the training staff to the overall program
4. Evidence that the training organizational structure suits the needs of the training program
5. Appropriateness and adequacy of the training methods used by the instructors.
6. Sufficiency of the time committed by the training director and staff to the training program.
7. Availability and commitment of the training program of adequate human and equipment resources in the areas of:
• Health effects
• Safety
• Personal protective equipment (PPE)
• Operational procedures
• Employee protection practices/procedures
8. Appropriateness of management controls
9. Adequacy of the organization and appropriate resources assigned to assure appropriate training
10. In the case of multiple-site training programs, adequacy of satellite centers management

C. Training facilities and resources. Adequacy and appropriateness of the facilities and resources for supporting the training program should be considered, including:

• Space and equipment to conduct the training
• Facilities for representative hands-on training
• In the case of multiple-site programs, equipment and facilities at the satellite centers
• Adequacy and appropriateness of the quality control and evaluations program to account for instructor performance
• Adequacy and appropriateness of the quality control and evaluation program to ensure appropriate course evaluation, feedback, updating, and corrective action
• Adequacy and appropriateness of disciplines and expertise being used within the quality control and evaluation program
• Adequacy and appropriateness of the role of student evaluations to provide feedback for training program improvement

D. Quality control and evaluation. Adequacy and appropriateness of quality control and evaluation plans for training programs should be considered, including:

• A balanced advisory committee and/or competent outside reviewers to give overall policy guidance
• Clear and adequate definition of the composition and active programmatic role of the advisory committee or outside reviewers
• Adequacy of the minutes or reports of the advisory committee or outside reviewers' meetings or written communication
• Adequacy and appropriateness of the quality control and evaluations program to account for instructor performance
• Adequacy and appropriateness of the quality control and evaluation program to ensure appropriate course evaluation, feedback, updating, and corrective action
• Adequacy and appropriateness of disciplines and expertise being used within the quality control and evaluation program
• Adequacy and appropriateness of the role of student evaluations to provide feedback for training program improvement

E. Students. Adequacy and appropriateness of the program for accepting students should be considered, including:

1. Assurance that the student already possess the necessary skills for their job, including necessary documentation
2. Appropriateness of methods the program uses to ensure that recruits are capable of satisfactorily completing training
3. Review and compliance with any medical clearance policy

F. Institutional Environment and Administrative Support. The adequacy and appropriateness of the institutional environment and administrative support system for the training program should be considered, including:

• Adequacy of the institutional commitment to the employee training program
• Adequacy and appropriateness of the administrative structure and administrative support

G. Summary of Evaluation Questions. Key questions for evaluating the quality and appropriateness of an overall training program should include the following:

1. Are the program objectives clearly stated?
2. Is the program accomplishing its objectives?
3. Are appropriate facilities and staff available?
4. Is there an appropriate mix of classroom, demonstration, and hands-on training?
5. Is the program providing quality employee health and safety training that fully meets the intent of regulatory requirements?
6. What are the program's main strengths?

Source: OSHAcademy

Certisafety Section Home Page