Resources - Benchmarking

The DON Benchmarking Model for Conducting a Benchmarking Study: The 10 Steps

Step 4: Collect and analyze the data.

"If you Don't measure it, you don't manage it." (Juran, 1989)

Input to Step 4: The input to Step 4 is the output from Step 3.

  • A list of possible benchmarking partners from research sources (approximately 15).
  • A blind list of potential best practices (approximately 5-15).
  • A list of the final partners selected (approximately 1-5).
  • Team copies of the Benchmarking Code of Conduct.

A. Determine the data collection plan and method.

Now the BMK Team needs to determine a plan and agree on a method to collect data about the benchmarking process and any performance measures to be used for comparison(s), from within their own organization as well as from their benchmarking partner(s). The goal is to collect valid, reliable, objective performance data on the internal process first. Examples adapted from the American Productivity and Quality Center's The Benchmarking Management Guide (1993) of how you might measure data within a given process follow:

  • Productivity, by transactions per unit.
  • Accuracy, by the error rates.
  • Responsiveness, by the time intervals.
  • Speed, by the cycle time.
  • Product stability, by the engineering change orders per month.
  • Process financial contribution, by the value-to-cost ratio.
  • Product availability, by fill rate.
  • Product quality, by first-pass yield.
  • Capacity, by volume managed.
  • Service, by the on-time delivery.

In developing a data collection plan, the BMK Team should answer the following questions:

  • What data will give us what we need to know to compare our process with the best-in-class (inventory control, recruitment, procurement, education and training, marketing, etc.) process?
  • What kind of information/measurement is necessary (accuracy, quality, customer satisfaction, speed, dependability, etc.)?
  • What data exist on the internal process?
  • How should the BMK Team collect the data?
  • Who on the BMK Team will collect the data?
  • How will the BMK Team check its results?
  • How much time will be needed to collect the data?
  • How will the data be consolidated and analyzed?
  • How will we evaluate the satisfaction of the customer(s) of the process?
  • What method(s) should be used to collect data from partner(s)?
  • hard copy correspondence (mail/e-mail/fax)?
  • telephone interviews?
  • publications/other media?
  • library and database research?
  • personal interviews/meetings?

Methods of Data Collection

One (or more) of the following methods can be used to collect the data. Following are some guidelines for each method and some advantages and disadvantages of each.

Correspondence. Using hard copy correspondence such as the U.S. Mail service, electronic mail, or fax to collect data is an inexpensive, easy, and time-efficient way to gather this information. However, correspondence limits the ability to probe, and may require follow-on questions. Be aware that some organizations may not give "answering the mail" a high priority.

Telephone. A telephone call is easy to plan and conduct. It facilitates contact with a large number of partners and can be relatively inexpensive. It provides a direct, personal contact with your partner(s). It also provides the ability to get a better sense of the organization and the individual with whom you are dealing. A common problem with telephoning, however, is that it can be difficult to .connect. with the person you wish to speak to (a.k.a. phone tag). In addition, a "cold call" can be time consuming and frustrating for all parties. It is recommended that you send a read-ahead package to prepare your partner(s). Include a suggested date and time and an estimate of the time required for the call to increase your chances of finding your point of contact available and informed. Contact a specific individual and maintain a good working relationship with this person. Explain again who you are and why you are calling. Mention any referrals. Exchange information where appropriate. Establish a follow-up session where necessary.

Publications. Publications and other forms of media, including World Wide Web sites, hold vast amounts of useful information, provide many opportunities to advertise for a partner, and often provide clues as to who may be considered the best-in-class. Magazines and journals often have articles on the pacesetters in a particular process. An ad in the newspaper or a trade paper can be minimally expensive and might solicit some surprising partner(s).

Research. By this point, the BMK Team has already done research to identify partner(s) via the library and database research. The BMK Team members can sift through this information to see what may already be contained and useful for this particular step of the process.

Interviews. Face-to-face contacts through personal interviews and meetings represent a powerful methodology. Conferences, meetings, training sessions, etc., provide informal opportunities to talk to others about what they do and how they do it. But this can become a resource-intensive method of gathering information from possible benchmarking partner(s), and, most importantly, it doesn't guarantee that you will find the recognized world-class organizations. It can also become awkward if the partnership doesn't work out as anticipated.

Site visit. It is possible to have a successful benchmarking study without a site visit. Sometimes through the use of technology, such as teleconferences and a groupware system, the information you need can be acquired at low cost. However, if it is necessary to go to a partner's location, here are some guidelines for the visit:

  1. Make contact with the appropriate person to provide the proper authority for the visit.
  2. State the purpose of the visit.
  3. Verify the suitability of the site.
  4. Offer a reciprocal visit.
  5. Identify mutually agreeable date(s) with start and end times.
  6. Select a limited number of benchmarking team members to visit (2 to 5).
  7. Send a letter to confirm the visit and include:
  8. the date(s)
  9. those who will be making the site visit
    • those who you propose to visit at the partner's site
    • a proposed agenda]
    • a proposed time frame for visit
    • a flowchart and/or explanation of the process you plan to benchmark
    • the data collection process you plan to use
    • any ground rules.
  10. Be sure to check on any security check-through procedures.
  11. Get clear directions on how to get there and where to park.

A Word of Advice: Don't rush off and do the site visit before the benchmarking team is adequately prepared. You want to be sure to use your time (as well as your partner's time) during the site visit effectively and efficiently. Send the right people and be prepared to provide business cards. Listen. Stay focused. Test for a common understanding among all internal and external parties throughout the site visit. Debrief as soon as possible; always debrief one site before you go to another. Neutralize emotions and be objective. You may find some great personalities at some great locations, but how great is their actual process?

Survey. Many organizations use survey instruments or a questionnaire to help focus the effort and standardize the information collected from various partners. A survey should consist of open-ended questions developed by the BMK Team. The questionnaire should be limited to no more than 15 questions that would take no more than one hour to answer.

Regardless of how contacts with potential partners are made, the same questions should be asked of each partner. This will enable the team to have like-responses for better comparisons. You should be able to answer the same survey questions for your own process. The answers to the survey reveal a lot about an organization's understanding of the benchmarking process as well as about their own business process.

B. Collect and rank the data.

Now the team can actually begin the comparison of its business process against those of a world-class organization. Designated BMK Team members should contact the partner(s) and collect the data based on the plan and methodology developed by the team. After the data are collected, each partner is ranked in performance measurement order. This identifies where your partner's performance is significantly above and below your current performance level.

After collecting the data about the process from the benchmarking partner(s), establish your own ranking and any performance gaps. This provides the basis for performance goal-setting (Step 6). Having a measurement system in place allows you to measure your progress toward the goals.

Partner(s) should be blindly ranked; that is the actual names of the organizations should be replaced with symbols such as Company #1, Company #2, and Company #3, etc., or Organization A, Organization B, Organization C, etc. The example that follows is a simple matrix showing generic performance measures and where each organization, as well as your own, ranks. It uses a 1 (best) to 5 (worst) numbering system. With this method, the team can quickly see which organizations are the best of breed. In this case, the lower the number the better.

Ranking the performance measures of a pharmacy

A Word of Advice: The goal in ranking performance measures is to seek direction and categorize partners. Don't spend an inordinate amount of time splitting hairs between which organizations should be rated number 4 or number 5.

Look at the gaps in rankings and try to determine some of the reasons for the gaps. Project any future competitive gaps you may be aware of due to things such as evolving technologies. Camp stresses to look for balance in measures, not just cost (Camp, 1996). Things like quality, accuracy, delivery time, asset utilization, and the level of customer satisfaction in products should also be measured and ranked. Summarize the findings for the benchmarking report (Step 5).

C. Train the BMK Team with just-in-time skills/tools as needed.

When you assign the roles and responsibilities that each BMK Team member will have in researching, collecting, analyzing, and documenting the internal and external benchmarking data, consider if they will require specific training to participate. For example, training in: Many quality tools are available to assist the BMK Team in data collection and analysis. Those in your organization who have successfully completed the DON TQL Team Skills and Concepts course and/or the Systems Approach to Process Improvement course are trained in the use of these tools. Contact your TQL coordinator or specialist as needed. Some tools commonly used in benchmarking studies are:

  • survey instrument development.
  • use of databases/technology/World Wide Web sites.
  • matrix development.
  • data collection methods.
  • statistical analysis.
  • fishbone diagram.
  • matrix.
  • pareto analysis.
  • histogram.
  • affinity diagram.
  • scatter diagram.
  • run chart.
  • storyboarding.
  • integrated computer-aided manufacturing definition language (IDEF) modeling.
  • cost/benefit analysis.

Output from Step 4: The output from Step 4 is the input for Step 5.

  • A data collection plan and method.
  • Quantitative (just the numbers) and qualitative (what the numbers mean) performance measures of the process.
  • A blind list of partners in order of where they rank in each performance measure.
Quality Advisor's Checklist

Before moving to the next step, the quality advisor should review the following checklist:

  • Will the data collection plan and method provide valid, reliable, and objective performance information on the process?
  • Are all BMK Team members collecting data in a similar fashion?
  • Is a site visit necessary?
  • Were potential partners ranked blindly?
  • Does the BMK Team have all the tools and skills necessary to collect and analyze the data?
  • Did the BMK Team document, document, and document some more? z Is there software available that could simplify the data collection and analysis?
  • Is there information that can be collected through secondary research without wasting resources?
  • Is there information that can be collected through secondary research without wasting resources?

Source: USN Benchmarking Handbook

Certisafety Section Home Page

Copyright ©2000-2019 Geigle Safety Group, Inc. All rights reserved. Federal copyright prohibits unauthorized reproduction by any means without permission. Disclaimer: This material is for training purposes only to inform the reader of occupational safety and health best practices and general compliance requirement and is not a substitute for provisions of the OSH Act of 1970 or any governmental regulatory agency. CertiSafety is a division of Geigle Safety Group, Inc., and is not connected or affiliated with the U.S. Department of Labor (DOL), or the Occupational Safety and Health Administration (OSHA).