An Introduction to Measuring Reliability

Measuring reliability can improve the quality and value of our health care systems and quality improvement projects. Reliability measures how consistent the quality and safety of health care systems or processes perform over a required period of time. A highly reliable system has a lower risk of errors and process failures that can cause patients harm. Routine anesthesia, for example, is considered very reliable. It’s associated with very few errors and process failures, and there are seldom deaths resulting from those errors or failures.

Reliability matters in population health initiatives too. For example, a reliable developmental screening system would result in a consistent process for providing developmental screenings to young children and then, when necessary, referring them to interventions.  

Ensuring reliable systems, whether at the hospital, state, or community level, requires measuring it. Measuring reliability enables us to understand and learn from variability in our systems and discover whether our improvements result in higher reliability.

Evaluating current processes

In order to measure reliability, teams must first observe and evaluate the system processes. In a developmental screening system, one process may be training health care professionals to conduct screenings during well-visits, and ensuring those screenings align with the American Academy of Pediatrics (APP) developmental milestones.

The Institute of Medicine (IOM) identified six dimensions for health care improvement. Use these aims to evaluate the effectiveness and reliability of each of your processes:

• Safety: Patients should not be harmed by the care that is intended to help them.
• Effective: Care should be based on scientific knowledge and offered to all who could benefit, and not to those not likely to benefit.
• Patient-Centered: Care should be respectful of and responsive to individual patient preferences, needs, and values.
• Timely: Waits and sometimes-harmful delays in care should be reduced both for those who receive care and those who give care.
• Efficient: Care should be given without wasting equipment, supplies, ideas, and energy.
• Equitable: Care should not vary in quality because of personal characteristics such as gender, ethnicity, geographic location, and socio-economic status.

Focusing on processes rather than people, is important, says NICHQ Quality Improvement Advisor, Jane Taylor, EdD, MBA, MHA, “By focusing on processes, we avoid assigning blame. We shouldn’t be looking for who’s at fault for a failure because from the patient’s perspective, it doesn’t matter who’s at fault. All that matters is what happened or didn’t happen. We must look at systems and process design and their reliability.”

How is reliability measured

The effectiveness of these processes and occurrence of failures is the key to measuring the reliability of your system. One way of measuring reliability involves comparing the number of actions that achieved the intended results to the total number of actions taken.

One example of a failure may be a roadblock in your processes that then prevents systems from achieving their intended outcomes. Referring to developmental screenings, you might discover that the current practices for screening children does not follow AAP recommendations on how often screening should take place. If the screenings are not done, then the referral isn’t made. The result of fewer screenings and referrals than recommended is that children miss opportunity for early interventions and optimal outcomes. Every missed screening would be considered a failure. In the developmental screening example, you would track how many screenings and referrals were made. . To measure reliability, you would compare the number of infants who received a screening against the number of infants due for a screening who did not receive a screening. This helps you determine your failure rate, and the higher the failure rate, the less reliable your system. For example, if 80% of infants are receiving the screening, then 20 out of 100 times a failure occurs, and the process is considered unreliable and chaotic.

Steps to increase reliability

To increase the reliability, it’s helpful to backtrack by identifying where a failure occurred, then analyzing each process and outcome that led to that failure. The first step is to collect data over time and display it in a run or control chart to understand the process or system performance, signaling when a system moves from stability to instability or vice versa. During this process, it’s important to pay attention to human factors and poor system and process design. The IOM dimensions may be useful in understanding the type of failure, such as a failure in safety, timeliness, etc.

Then the second step is to prevent future failure with tools like standard work, checklist, prompts, or memory aides. Develop a way to identify failures in real time and use error-proofing strategies to mitigate harm from the failure.

The third step is for critical process that cause serious harm or injury. It requires redesign of the process using human factors engineering and to design robust systems and processes that rarely fail.  

You may find you need a project team to support improving reliability. In doing so, you can begin by identifying causes for failures and developing options to improve these processes, test the ideas, and track data to determine whether your changes are improving reliability. This strategy will help you create stronger processes, result in fewer failures, and create lasting change.

In this article, learn more about developing a successful measurement strategy.