Evaluation should be a part of workplace changes from the start, through the process of finding an idea, trying it, testing to see if it results in positive outcomes, and if it does, adopting it. You can evaluate a plan or idea for change just as well as you can evaluate the actual change. Evaluation isn’t necessarily expensive or complicated. Here are four steps to follow:
Evaluations are best done by a team. When forming a team, keep in mind that workers are often the best source of information about the workplace. Wherever possible, the team should include people who are:
The team should start by defining the problem as clearly as possible. The more specifically that the problem is defined, the better that a solution can be designed to fit, and the easier it is to test the solution. Brainstorm a solution for the problem and one or two ways to measure the effectiveness of the change. Using more than one way to measure the change will increase your confidence in the results.
Collect accurate data on workplace safety and health. Consider collecting three kinds of data: conditions before the change, information about how the change was put in place, and what happened after the change was made. Good records of how the change was made will help if you want to do it again because they tell you what happened. Data on the outcome will tell you whether to bother doing it again.
How, how often, and when should you collect data?
One way is to measure an outcome (like employee injuries) before and after the change. Typically, injuries are infrequent occurrences; therefore, you may need to collect data for quite awhile before and afterwards to make sure that any changes you detect are real. On the other hand, if you are measuring changes in employee knowledge and attitudes after a training session, you may not need to measure for as long a period of time. You may measure immediately before the training, right after the training, and then one or two more times 3 months to 1 year after the training.
Another model for gathering information is to arrange the change so that it does not affect everyone in the operation at one time. By introducing the change to several different groups over a period of several months or a year, you can get a better idea of effectiveness. Simply measure all the groups (those who receive the change immediately and those who receive it later) at regular intervals as you gradually introduce the change to the selected groups. This takes more effort, but it gives you added confidence that any detected effects are from the change and not some other event.
What to measure depends on the following:
If the change is new training, then you may want to look for performance indicators or success/failure rates. Think of measurements that are practical. Remember that someone or something has to collect and analyze the measurements you choose. Here are some simple ways of collecting data. If you use more than one of these methods to measure each change, you will be more certain of the results.
Several types of records can provide data on the effectiveness of safety and health changes. Useful data include the following:
Records should be checked for completeness and accuracy before they are used. Surveys are useful for determining workers’ perceptions both before and after a change:
You can also use surveys to measure knowledge, attitudes, and behaviors. Write the survey in such a way that all those who read or hear it will understand the questions. Test the survey with a sample of workers before using it to find out whether the questions are understood as you intend. Surveys can be distributed in a number of ways: in person, with paychecks, by mail, e-mail, Internet, or a combination of methods. Followups are usually necessary to increase participation and get the completed forms back.
It is always best to encourage the participation of your entire workforce in a survey. Often surveys lead to planned change and survey participants are more likely to feel a sense of ownership in the change. However, if you cannot afford to survey everyone in a very large workplace, select a sample that represents all important groups such as departments within a hospital or plant, and be sure to include all the groups that are anticipated to be included in the workplace change.
It is very important to establish a system that assures the confidentiality of each person's responses, including both the collection and the reporting of the survey data, i.e., anonymous surveys and results. This can go a long way toward encouraging employees to provide honest feedback on the survey.
Interviews are one-on-one, face-to-face conversations. In these conversations, the interviewer(s) asks the same set of questions to a number of people, but talking to only one person at a time. Interviews require more time and resources than surveys, but provide more detailed information. Record interviews so that statements can be accurately reviewed and analyzed. Since interviews usually involve fewer people than surveys, you should interview a range of people who represent all the important groups associated with the change. One drawback to interviews is that the data can be challenging to analyze because it may be quite detailed or because there may be extra material that is not important to the evaluation.
Focus groups involve gathering information and opinions that are “focused” on one subject from a small group of people (about 8 to 10 per group). These group discussions often provide insights that might not emerge in interviews. Focus groups can be used in all stages of making a safety and health change — from planning to determining effectiveness. You may want to recruit participants with similar characteristics (such as job classification) since group discussions often thrive on common experience. Hold separate group discussions when you want to solicit opinions from supervisors and the employees they supervise. Assuring the confidentiality of such discussions is essential.
Below are tips for planning and conducting a focus group:
Afterward, review the tape or transcript and summarize major points. If a transcript is prepared, be sure to substitute fictional names for the names of actual participants. Watching how people behave in the workplace before and after you make a safety and health change may give you good evidence about the effects of the change. Observation allows you to collect fairly continuous information about how the work is done and about the change in the workplace procedures while it is in use.
Follow these guidelines when you observe people:
Some safety and health changes are designed to improve the physical work environment. Two examples are noise baffles around loud equipment and extra ventilation to remove contaminants. Environmental measures can help determine whether these changes are effective. Environmental measures include air sampling and noise monitoring. This type of data gathering may require people with special equipment and training, but the data they can collect may help pinpoint the value of the change.
Data must be systematically analyzed to determine whether the changes are effective. Data analysis can be simple or complex depending on the questions asked, the completeness of the data, and the experience of the evaluation team. When analyzing data:
After the initial analysis is complete, you may want to continue collecting data about the change. These data can serve as baseline measures for future interventions. They are also useful for determining the long-term impact of the change.How to calculate injury rates
The number of new injuries per 100 workers in a given period of time is called the incidence rate. Here is how to calculate it:
Note: The figure “200,000 hours” will always be the same in your calculations. It represents 2,000 hours per worker, worked per year, times 100 workers.. Example: During 2015, there were five new injuries at a plant. The total number of hours worked that year was 250,000. The incidence rate is calculated as follows:
As shown, there were 4 new injuries per 100 workers at the plant in 2015. The incidence rate is 4.
First, share your results with those affected by the change. The more you tell them about what you did and what you found, the more they will support your conclusions and your future efforts to evaluate safety and health changes in the workplace. A good way to start the feedback process is to hold work-group meetings to present the results. Don't worry about drawing conclusions from the results, just start with the basic findings. This approach will permit everyone to begin thinking about what the results suggest. You could also post the results in worker areas like break rooms, locker rooms, and cafeterias.
During the data analysis and results steps, be careful to maintain confidentiality. Breeches in confidentiality can affect your credibility. Do not report results in such a way that individual responses can be identified. This is especially important where there are workgroups of 3-6 people. In such circumstances, it may be necessary to report results for several workgroups, combined.
Copyright ©2000-2019 Geigle Safety Group, Inc. All rights reserved. Federal copyright prohibits unauthorized reproduction by any means without permission. Disclaimer: This material is for training purposes only to inform the reader of occupational safety and health best practices and general compliance requirement and is not a substitute for provisions of the OSH Act of 1970 or any governmental regulatory agency. CertiSafety is a division of Geigle Safety Group, Inc., and is not connected or affiliated with the U.S. Department of Labor (DOL), or the Occupational Safety and Health Administration (OSHA).