What is "evaluation" and what do we "evaluate" Evaluation defined. Webster defines the term, evaluate, as "to judge the worth of." Evaluation is a systematic, objective process for determining the success of a policy or program. It addresses questions about whether and to what extent the program is achieving its goals and objectives. Its primary attributes of most SMS evaluations include objectivity, standardization, systematic, and formal.

Evaluation has several distinguishing characteristics. An evaluation:

1. assesses the effectiveness of an ongoing program in achieving its objectives,
2. relies on the standards of project design to distinguish a program's effects from those of other forces, and
3. aims at program improvement through a modification of current operations.

• what will be done
• how it will be done
• who will do it
• when it will be done
• why the evaluation is being conducted.

The purpose of an evaluation

System Evaluations generally have four basic purposes:

Evaluate the design. Examination of the written plans, policies, procedures, and other documents to determine how clearly they are written and if they contain the necessary information. For instance, during the SMS evaluation, an evaluator would examine the written hazard communications program to make sure it contained the required information.

Evaluate the process. Another primary consideration in an evaluation is to assess the quality of SMS activities. For example, an evaluator might observe trainers using the program and write a descriptive account of how employee respond and then provide feedback to instructors.

Evaluate results. It's important for an evaluation to study the immediate or direct results of the SMS and its programs on employees. For example, the evaluator may conduct a walk-around inspection to determine the safety status of tools, equipment, and materials in the workplace.

Evaluate impact. An effective evaluation looks beyond the immediate conditions and behaviors representing the results of policies, instruction, or services. It also identifies longer-term as well as unintended program effects. It may also examine what happens when several programs operate in unison. For example, an impact evaluation might examine whether a safety program's immediate positive effects on behavior were sustained over time.

Regardless of the primary focus of the evaluation, they all use data collected in a systematic manner. These data may be:

• quantitative, such as counts of safe/unsafe behaviors, or
• qualitative, such as descriptions of the effectiveness of an incentive and recognition program.

It is not the purpose of an SMS evaluation to place blame

This is important. Do not conduct a SMS evaluation to determine the inherent value of a person. We don't evaluate to find out who is mad, bad, evil, lazy, crazy, stupid, or otherwise flawed. Do not make value judgments that attack a person or group. A key principle to understand, here, is that if you attack people, they attack back.

If the purpose of an evaluation is to "fix the system," playing the "blame game" is not effective precisely because it does not achieve the desired effect. Actually, the evaluation may be counter-productive.

If we evaluate to place blame, we'll stop the process once blame has been determined. As a result, we'll never get past blame to evaluate the system. In an effective SMS evaluation, our objective is to discover the effectiveness of the system.

If the purpose is to fix the blame, you are not going to ask this critical question. Why? Because...

When the purpose of a process has been achieved, the process stops!

Safety Committees Should Help Evaluate the SMS

The safety committee can help by evaluating the employer's accident and illness prevention program, and making written recommendations to improve the program where applicable. This best practice emphasizes the fact that a very important safety committee responsibility is to help the employer evaluate the SMS. The safety committee should also be able to write quality recommendations to improve the SMS.

Determine the Benchmark

To conduct an evaluation, we need to take the information gathered from the baseline survey and rate it against an established benchmark. A benchmark is a standard by which the system can be measured or judged, for instance, we might say XYZ's SMS is "benchmark of quality" in our industry. In the optional modules of this course (Modules 5-12); you will be introduced to the OSHA Safety and Health Program Assessment Worksheet which may be used as a benchmark. This audit evaluates the same 58 elements of a SMS also used by OSHA to evaluate companies participating in the Safety and Health Achievement Program (SHARP). You may also be interested in using other evaluation standards as benchmarks such as:

• Voluntary Protection Program (VPP)
• ANSI/AIHA Z10, Occupational Health and Safety Management Systems

Now that we've discussed the analysis and evaluation processes, let's take a look at a simple example how all it works.

Donn and Doff Analyze an Apple "Hey Donn, what the heck is that bump on this apple!" "What's with that, Doff? Let's take a look." Identification: Oh oh, we have a problem. Don and Doff see an apple in the basket that looks different. It looks like the apple has a bump on it. They identified a possible problem. Analysis: Hey, what does it look like? To better understand why the apple looks like it does, they decide to cut it up, take a look at the seeds, the core, the flesh and the skin. They gather the following facts about the apple: The core and seeds look just fine. The bump is "smooshy" There are many little discolored "tunnels" throughout the fleshy part. Flesh surrounding the tunnels appears rotted The apples tastes very good The skin of the apple is discolored in places Evaluation: OK, how "good" is the apple? Now that Donn and Doff have gathered information, they are able to evaluate the quality of the apple based on facts. They determine the apple is flawed. Now that they know they have a real problem, they decide to figure out what the cause is so the rest of their apples don't spoil. They've got to conduct a cause analysis. They understand that everything they've identified so far represent only the observable, measurable effects of some cause. Cause Analysis: OK, what's the cause? The question, now, is, "what is the cause." There are two basic types of causes Donn and Doff identify in their analysis: surface and root (very appropriate in our apple example ;-). Surface causes. It's obvious to Donn and Doff the damage is caused by a bug of some kind. Considering all the information gathered helps them search the internet and determine that an Apple Maggot has deposited eggs under the skin of the apple and fed on the flesh of the apple. They're quite happy about discovering the obvious surface cause, but why is the Apple Maggot causing a problem. It never has before! They've got to figure out the root cause. Root cause. Donn and Doff know the maggot did its damage, but why? Asking "why" a number of times, will help them eventually determine the less obvious underlying contributing causes of the spoiled apple. During root cause analysis they determine that: the pesticide used on the apples was not effective against the Apple Maggot the Apple Maggot, which is native to the eastern part of the country has somehow migrated to With this information in hand, they will be able to develop strategies to overcome this infestation.

Using the Failure Matrix to Evaluate the Safety Management System

The matrix below reflects the idea that the negative effects of a flawed system are due to inadequate resources, system design, and/or system performance. If one or more of these three categories are flawed, the effect will be flawed conditions and behaviors.

What's the failure mode for the scenario below?

Bob, a maintenance worker who has been working for the company for 10 years, received a serious electrical shock while working on a conveyor belt motor. When asked why he did not use the company's established lockout/tagout procedures he acknowledged that he had thought about it, but the procedures were not current since the new equipment had been installed last year. He also indicated that most other maintenance workers usually bypassed lockout/tagout procedures because everyone is constantly being told to "hurry up" to get the job finished.

1. Did Bob have adequate physical resources and psychosocial support to do the job?
2. Was the design of the lockout/tagout program adequate? What about other programs. Can you think of other programs that might have "caused" an inadequate design of the lockout/tagout program?
3. Were program policies and procedures being performed adequately? Did other programs affect the quality of lockout/tagout program performance?

Last Words

I hope this information on cause analysis has been helpful. In the next module, we'll discuss a few analysis tools and techniques. Remember, all this information on analysis will help you make factual conclusions about the quality of your SMS. Time to answer the review questions.

Take the review Quiz