Steps for Evaluating Change

Evaluation should be a part of workplace changes from the start, through the process of finding an idea, trying it, testing to see if it results in positive outcomes, and if it does, adopting it. You can evaluate a plan or idea for change just as well as you can evaluate the actual change. Evaluation isn’t necessarily expensive or complicated. Here are four steps to follow:

Step 1: Form a team

Evaluations are best done by a team. When forming a team, keep in mind that workers are often the best source of information about the workplace. Wherever possible, the team should include people who are:

• affected by the safety and health change,
• responsible for implementing and maintaining the change, and/or
• involved in future decisions about changes.

The team should start by defining the problem as clearly as possible. The more specifically that the problem is defined, the better that a solution can be designed to fit, and the easier it is to test the solution. Brainstorm a solution for the problem and one or two ways to measure the effectiveness of the change. Using more than one way to measure the change will increase your confidence in the results.

Step 2: Collect Relevant Data

Collect accurate data on workplace safety and health. Consider collecting three kinds of data: conditions before the change, information about how the change was put in place, and what happened after the change was made. Good records of how the change was made will help if you want to do it again because they tell you what happened. Data on the outcome will tell you whether to bother doing it again.

How, how often, and when should you collect data?

One way is to measure an outcome (like employee injuries) before and after the change. Typically, injuries are infrequent occurrences; therefore, you may need to collect data for quite awhile before and afterwards to make sure that any changes you detect are real. On the other hand, if you are measuring changes in employee knowledge and attitudes after a training session, you may not need to measure for as long a period of time. You may measure immediately before the training, right after the training, and then one or two more times 3 months to 1 year after the training.

Another model for gathering information is to arrange the change so that it does not affect everyone in the operation at one time. By introducing the change to several different groups over a period of several months or a year, you can get a better idea of effectiveness. Simply measure all the groups (those who receive the change immediately and those who receive it later) at regular intervals as you gradually introduce the change to the selected groups. This takes more effort, but it gives you added confidence that any detected effects are from the change and not some other event.

What to measure depends on the following:

1. Expected outcomes – what will happen as a result of the safety and health change? Whatever you expect – reduced injuries or absenteeism, increased productivity, -- measure those things.
2. Nature of the safety and health change – for example, if the change is plant exhaust ventilation, then you may want to measure air quality before and after. If the change is increased rest breaks for employees, then you may want to measure productivity.

If the change is new training, then you may want to look for performance indicators or success/failure rates. Think of measurements that are practical. Remember that someone or something has to collect and analyze the measurements you choose. Here are some simple ways of collecting data. If you use more than one of these methods to measure each change, you will be more certain of the results.

• Records
• Surveys
• Interviews
• Focus Groups
• Observation
• Environmental measures

Several types of records can provide data on the effectiveness of safety and health changes. Useful data include the following:

• Injury frequency and rates
• Workers compensation costs
• Lost workdays and absenteeism due to work-related injuries
• Profit and loss
• Air sampling data, noise readings
• Production errors or waste
• Cost of respiratory equipment or ear protection

Records should be checked for completeness and accuracy before they are used. Surveys are useful for determining workers’ perceptions both before and after a change:

Pre-change assessment:

• What is working well?
• What is working poorly?
• Where in the work process are there delays in production?
• Where can quality be improved?
• What procedures place workers at risk of illness or injury?
• What changes can be made to correct existing problems?

Post-change assessment:

• Is the change effective?
• How has the workplace been changed?
• Are things better or worse?
• How could the change be improved?

You can also use surveys to measure knowledge, attitudes, and behaviors. Write the survey in such a way that all those who read or hear it will understand the questions. Test the survey with a sample of workers before using it to find out whether the questions are understood as you intend. Surveys can be distributed in a number of ways: in person, with paychecks, by mail, e-mail, Internet, or a combination of methods. Followups are usually necessary to increase participation and get the completed forms back.

It is always best to encourage the participation of your entire workforce in a survey. Often surveys lead to planned change and survey participants are more likely to feel a sense of ownership in the change. However, if you cannot afford to survey everyone in a very large workplace, select a sample that represents all important groups such as departments within a hospital or plant, and be sure to include all the groups that are anticipated to be included in the workplace change.

It is very important to establish a system that assures the confidentiality of each person's responses, including both the collection and the reporting of the survey data, i.e., anonymous surveys and results. This can go a long way toward encouraging employees to provide honest feedback on the survey.

Interviews are one-on-one, face-to-face conversations. In these conversations, the interviewer(s) asks the same set of questions to a number of people, but talking to only one person at a time. Interviews require more time and resources than surveys, but provide more detailed information. Record interviews so that statements can be accurately reviewed and analyzed. Since interviews usually involve fewer people than surveys, you should interview a range of people who represent all the important groups associated with the change. One drawback to interviews is that the data can be challenging to analyze because it may be quite detailed or because there may be extra material that is not important to the evaluation.

Focus groups involve gathering information and opinions that are “focused” on one subject from a small group of people (about 8 to 10 per group). These group discussions often provide insights that might not emerge in interviews. Focus groups can be used in all stages of making a safety and health change — from planning to determining effectiveness. You may want to recruit participants with similar characteristics (such as job classification) since group discussions often thrive on common experience. Hold separate group discussions when you want to solicit opinions from supervisors and the employees they supervise. Assuring the confidentiality of such discussions is essential.

Below are tips for planning and conducting a focus group:

• Develop a list of discussion topics ahead of time.
• Determine the amount of time to be spent on each topic. (Good discussions usually require about 1 ½ – 2 hours. )
• Design questions in such a way that they encourage discussion. (Don’t ask questions that get short answers like “yes” or “true.”)
• Start the session with an “ice breaker” that gets everyone to talk. Make sure that everyone knows that they are expected to contribute.
• Have an experienced note taker attend. If you can get all participants’ permission, record the session on audio or video tape.

Afterward, review the tape or transcript and summarize major points. If a transcript is prepared, be sure to substitute fictional names for the names of actual participants. Watching how people behave in the workplace before and after you make a safety and health change may give you good evidence about the effects of the change. Observation allows you to collect fairly continuous information about how the work is done and about the change in the workplace procedures while it is in use.

Follow these guidelines when you observe people:

• Decide in advance which behaviors to look for.
• Vary your observations so that they are a more-representative sample of people and time periods.
• Record your observations. This may involve watching people in person and recording their actions on a checklist. It could also involve recording people on video tape and then categorizing their actions later with the checklist.
• Whether you record people on tape or not, explain what you’re doing and why, and get their permission before you observe. Even though they know you’re watching, most people will quickly forget about it. Remember that the presence of an observer may affect what people do.

Some safety and health changes are designed to improve the physical work environment. Two examples are noise baffles around loud equipment and extra ventilation to remove contaminants. Environmental measures can help determine whether these changes are effective. Environmental measures include air sampling and noise monitoring. This type of data gathering may require people with special equipment and training, but the data they can collect may help pinpoint the value of the change.

Step 3: Analyze Data

Data must be systematically analyzed to determine whether the changes are effective. Data analysis can be simple or complex depending on the questions asked, the completeness of the data, and the experience of the evaluation team. When analyzing data:

• Allow team members to interpret findings independently.
• Compare members’ findings with the goals set by the team.
• Arrive at conclusions about the change on the basis of the findings.
• Summarize the results in such a way that everyone involved understands them.

After the initial analysis is complete, you may want to continue collecting data about the change. These data can serve as baseline measures for future interventions. They are also useful for determining the long-term impact of the change.

How to calculate injury rates

The number of new injuries per 100 workers in a given period of time is called the incidence rate. Here is how to calculate it:

Note: The figure “200,000 hours” will always be the same in your calculations. It represents 2,000 hours per worker, worked per year, times 100 workers.. Example: During 2015, there were five new injuries at a plant. The total number of hours worked that year was 250,000. The incidence rate is calculated as follows:

As shown, there were 4 new injuries per 100 workers at the plant in 2015. The incidence rate is 4.

Step 4: Share Your Results

First, share your results with those affected by the change. The more you tell them about what you did and what you found, the more they will support your conclusions and your future efforts to evaluate safety and health changes in the workplace. A good way to start the feedback process is to hold work-group meetings to present the results. Don't worry about drawing conclusions from the results, just start with the basic findings. This approach will permit everyone to begin thinking about what the results suggest. You could also post the results in worker areas like break rooms, locker rooms, and cafeterias.

During the data analysis and results steps, be careful to maintain confidentiality. Breeches in confidentiality can affect your credibility. Do not report results in such a way that individual responses can be identified. This is especially important where there are workgroups of 3-6 people. In such circumstances, it may be necessary to report results for several workgroups, combined.

Source: CDC/NIOSH

Certisafety Section Home Page