Checklist Analysis

Checklist analysis is a systematic evaluation against preestablished criteria in the form of one or more checklists.

Brief summary of characteristics

• A systematic approach built on the historical knowledge included in checklist questions
• Used for high-level or detailed analysis, including root cause analysis
• Applicable to any activity or system, including equipment issues and human factors issues
• Generally performed by an individual trained to understand the checklist questions. Sometimes performed by a small group, not necessarily risk analysis experts
• Based mostly on interviews, documentation reviews, and field inspections
• Generates qualitative lists of conformance and nonconformance determinations, with recommendations for correcting nonconformances
• The quality of evaluation is determined primarily by the experience of people creating the checklists and the training of the checklist users

Most common uses

• Used most often to guide boarding teams through inspection of critical vessel systems
• Also used as a supplement to or integral part of another method, especially what-if analysis, to address specific requirements
• A special, graphical type of checklist called a Root Cause Map™ is particularly effective for root cause analysis. (A Root Cause Map is included at the end of this chapter)

Example

Limitations of Checklist Analysis

Although checklist analysis is highly effective in identifying various system hazards, this technique has two key limitations:

Likely to miss some potential problems. The structure of checklist analysis relies exclusively on the knowledge built into the checklists to identify potential problems. If the checklist does not address a key issue, the analysis is likely to overlook potentially important weaknesses.

Traditionally only provides qualitative information. Most checklist reviews produce only qualitative results, with no quantitative estimates of risk-related characteristics. This simplistic approach offers great value for minimal investment, but it can answer more complicated risk-related questions only if some degree of quantification is added, possibly with a relative ranking/risk indexing approach.

Procedure for Checklist Analysis

The procedure for performing a checklist analysis consists of the following seven steps. Each step will be further explained on the following pages.

1.0 Define the activity or system of interest. Specify and clearly define the boundaries for which risk-related information is needed.

2.0 Define the problems of interest for the analysis. Specify the problems of interest that the analysis will address. These may include safety problems, environmental issues, economic impacts, etc.

3.0 Subdivide the activity or system for analysis. Section the subject into its major elements. These may include locations on the waterway, tasks, or subsystems. The analysis will begin at this level.

4.0 Gather or create relevant checklists for the problems of interest. Identify and collect lists of important questions or issues related to the type of potential problems within the scope of the analysis. If useful checklists are not available, consider developing your own checklists with the assistance of subject matter experts.

5.0 Respond to the checklist questions. Use a team of subject matter experts to respond to each of the checklist questions. Develop recommendations for improvement wherever the risk of potential problems seems uncomfortable or unnecessary.

6.0 Further subdivide the elements of the activity or system (if necessary or otherwise useful). Further subdivision of selected elements of the activity or system may be necessary if more detailed analysis of one or more elements is desired. Section those elements into successively finer levels of resolution until further subdivision will (1) provide no more valuable information or (2) exceed the organization's control or influence to make improvements. Generally, the goal is to minimize the level of resolution necessary for an analysis.

7.0 Use the results in decision making. Evaluate the recommendations from the analysis and implement those that will bring more benefits than costs over the life cycle of the activity or system.

1.0 Define the activity or system of interest

Intended functions. Because all risk assessments look at ways in which intended functions can fail, a clear definition of these intended functions is an important first step in any risk assessment. This step does not have to be formally documented in most checklist analyses.

Boundaries. Few activities or systems operate in isolation. Most interact with others. Boundaries may include areas where a vessel will transit or boundaries with support systems such as electric power and compressed air. By clearly defining the boundaries of the study, the analyst helps to avoid the following:

• Overlooking key elements of an activity or system at interfaces
• Penalizing an activity or system by associating other equipment with the subject of the study

Examples

2.0 Define the problems of interest for the analysis

Safety problems. The risk assessment team may be asked to look for ways in which improper performance of a marine activity or failures in a hardware system may result in personnel injury. These injuries may be caused by many mechanisms, including the following:

• Vessel collisions or groundings
• Person overboard
• Exposure to high temperatures (e.g., steam leaks)
• Fires or explosions

Environmental issues. The risk assessment team may be asked to look for ways in which the conduct of a particular activity or the failure of a system can adversely affect the environment. These environmental issues may be caused by many mechanisms, including the following:

• Discharge of material, intentionally or unintentionally, into the water
• Equipment failures, such as seal failures, that result in a material spill
• Overutilization of a marine area, resulting in a disruption of the ecosystem

Economic impacts. The analysis team may be asked to look for ways in which the improper conduct of a particular activity or the failure of a system can have undesirable economic impacts. These economic risks may be categorized in many ways, including the following:

• Business risks such as vessels detained at port, contractual penalties, or lost revenue
• Environmental restoration costs
• Replacement costs for damaged equipment

A particular analysis may focus only on events above a certain threshold of concern in one or more of these categories.

3.0 Subdivide the activity or system for analysis

An activity or system may be divided at many levels of resolution. Generally speaking, analysts should try to describe risk-related characteristics for an activity or system at the broadest level possible. The procedure for subdividing an activity or system for risk assessment is typically repetitive, beginning with a broad subdivision into major sections or tasks.

This strategy of beginning at the highest level helps promote effective and efficient risk assessment by (1) ensuring that all key attributes are considered in the risk assessment, (2) encouraging analysts to avoid unnecessary detail, and (3) using a structure that helps to avoid overlooking individual components or steps if further subdivision is necessary.

Example

Systems associated with the vessel's compressed air system

• Compressor system
• Dryer system
• Distribution system

4.0 Gather or create relevant checklists

Following are the three major types of checklists that you will likely be able to use in your risk assessment:

Internal checklists. Many formal and informal checklists commonly exist internally. In some cases, Coast Guard or regulatory standards mandate the use of specific checklists at key points. Examples include boarding checklists, design checklists, fabrication or installation checklists, pre-startup checklists, etc. These checklists may be updated regularly to help build organizational knowledge and to prevent problems from recurring. Frequently, there are less formal checklists used within selected geographic, functional, or organizational groups. The following are some examples:

• Checklists of key equipment that must be inspected on foreign flagged vessels while they are in port
• Checklists of key equipment specification and configuration requirements for selected applications. These are often based on vendor-specific design standards
• Checklists of best practices for making systems more maintainable
• Checklists of best practices for making systems easier to operate. These would include human factors and ergonomic issues

Many of these checklists may be general purpose and applicable to a variety of situations; others will be for more specific applications.

Checklists should generally be created and maintained by a team of experts. This is especially true of checklists that will be broadly applied. This team approach builds the checklists from many years of experience and forces consensus on important issues rather than relying on one person's ideas about what is best or necessary.

External checklists. When internal checklists do not exist or additional ideas about potential issues must be considered, external checklists may be used. External checklists may come from a variety of sources, including the following:

• Requirements in codes, standards, and regulations
• Industry best practices and guidelines
• Application guidelines from vendors
• Checklists gathered from other companies or organizations with similar applications

Of course, the key issue with external checklists is to be certain that they are applicable to your specific situation. If not, they may overlook important issues or may drive you to implement unnecessary changes.

Customized checklists. For many risk-based decisions for which a checklist analysis is appropriate, no suitable previously developed checklist will be available. In these cases, a customized checklist must be developed.

Questions for customized checklists should be derived from suitable existing checklists as much as possible. Where other checklists are not helpful, the analyst or the analysis team should discuss important issues and compose specific checklist questions to structure the risk assessment. Frequently, these questions ask whether particular safeguards are in place to protect against key weaknesses. The questions should then be sorted according to subject area and incorporated with other checklist questions obtained from other sources. If the checklist may be used for many applications in the future, you may want to use a more structured risk assessment tool, such as what-if analysis, to help build a reasonably complete checklist of important issues.

Example

5.0 Respond to the checklist questions

Each checklist question must be answered by people who are knowledgeable about the subject of the risk assessment, including the design, operation, and maintenance of associated systems.

Answering checklist questions generally involves two decisions:

1. Is the question applicable to this situation?
2. If so, are there weaknesses related to this question? This is typically
3. indicated by "no" answers to checklist questions.

When weaknesses are identified, the respondents generate recommendations for improvements to address those weaknesses.

There are three basic levels of documentation possible for a checklist analysis, as shown in the following table.

Example of complete checklist documentation

6.0 Further subdivide the elements of the activity or system (if necessary or otherwise useful)

Further subdivision of activities or systems occurs only under the following conditions:

• Applicable data at the higher levels are not available
• Decision makers need information at a more detailed level

Often, only a few activities or systems must be subdivided.

If the above criteria apply to one or more subsystems, they may be further divided into components. In a similar manner, broad activities or tasks may be divided into individual steps. At each level, the process of performing the checklist analysis is repeated.

Example

Subsystems associated with the vessel's compressor system

• Electrical supply to the compressor
• Lubrication system
• Seal system
• Drive system, including the motor
• Mechanical compression system
• Control system
• Relief system
• Filter system

Checklist analyses of any or all of these subsystems might occur if they were important from a risk perspective.

7.0 Use the results in decision making

Judge acceptability. Decide whether the activity or system meets established requirements.

Identify improvement opportunities. Identify the elements of the activity or system most likely to contribute to future risk-related problems, based on identified deficiencies.

Evaluate recommendations for improvements. Evaluate the specific suggestions for improving the activity or system performance, including any of the following:

• Equipment modifications
• Procedural changes
• Administrative policy changes such as planned maintenance tasks, operator training, etc.

Justify allocation of resources for improvements. Estimate how implementation of expensive or controversial recommendations will affect future performance. Compare the risk-related benefits of these improvements to the total life-cycle costs of implementing each recommendation.

Source: USCG Risk-based Decision-making (RBDM) Guidelines.

Certisafety Section Home Page