An Introduction to Measuring Reliability
Measuring reliability can improve the quality and value of our health care systems and quality improvement projects. Reliability measures how consistent the quality and safety of health care systems or processes perform over a required period of time. A highly reliable system has a lower risk of errors and process failures that can cause patients harm. Routine anesthesia, for example, is considered very reliable. It’s associated with very few errors and process failures, and there are seldom deaths resulting from those errors or failures.
Reliability matters in population health initiatives too. For example, a reliable developmental screening system would result in a consistent process for providing developmental screenings to young children and then, when necessary, referring them to interventions.
Ensuring reliable systems, whether at the hospital, state, or community level, requires measuring it. Measuring reliability enables us to understand and learn from variability in our systems and discover whether our improvements result in higher reliability.
Evaluating current processes
In order to measure reliability, teams must first observe and evaluate the system processes. In a developmental screening system, one process may be training health care professionals to conduct screenings during well-visits, and ensuring those screenings align with the American Academy of Pediatrics (APP) developmental milestones.
The Institute of Medicine (IOM) identified six dimensions for health care improvement. Use these aims to evaluate the effectiveness and reliability of each of your processes:
- Safety: Patients should not be harmed by the care that is intended to help them.
- Effective: Care should be based on scientific knowledge and offered to all who could benefit, and not to those not likely to benefit.
- Patient-Centered: Care should be respectful of and responsive to individual patient preferences, needs, and values.
- Timely: Waits and sometimes-harmful delays in care should be reduced both for those who receive care and those who give care.
- Efficient: Care should be given without wasting equipment, supplies, ideas, and energy.
- Equitable: Care should not vary in quality because of personal characteristics such as gender, ethnicity, geographic location, and socio-economic status.
Focusing on processes rather than people, is important, says NICHQ Quality Improvement Advisor, Jane Taylor, EdD, MBA, MHA, “By focusing on processes, we avoid assigning blame. We shouldn't be looking for who's at fault for a failure because from the patient's perspective, it doesn't matter who's at fault. All that matters is what happened or didn’t happen. We must look at systems and process design and their reliability.”
How is reliability measured
The effectiveness of these processes and occurrence of failures is the key to measuring the reliability of your system. One way of measuring reliability involves comparing the number of actions that achieved the intended results to the total number of actions taken.
One example of a failure may be a roadblock in your processes that then prevents systems from achieving their intended outcomes. Referring to developmental screenings, you might discover that the current practices for screening children does not follow AAP recommendations on how often screening should take place. If the screenings are not done, then the referral isn’t made. The result of fewer screenings and referrals than recommended is that children miss opportunity for early interventions and optimal outcomes. Every missed screening would be considered a failure. In the developmental screening example, you would track how many screenings and referrals were made. . To measure reliability, you would compare the number of infants who received a screening against the number of infants due for a screening who did not receive a screening. This helps you determine your failure rate, and the higher the failure rate, the less reliable your system. For example, if 80% of infants are receiving the screening, then 20 out of 100 times a failure occurs, and the process is considered unreliable and chaotic.
Steps to increase reliability
To increase the reliability, it’s helpful to backtrack by identifying where a failure occurred, then analyzing each process and outcome that led to that failure. The first step is to collect data over time and display it in a run or control chart to understand the process or system performance, signaling when a system moves from stability to instability or vice versa. During this process, it’s important to pay attention to human factors and poor system and process design. The IOM dimensions may be useful in understanding the type of failure, such as a failure in safety, timeliness, etc.
Then the second step is to prevent future failure with tools like standard work, checklist, prompts, or memory aides. Develop a way to identify failures in real time and use error-proofing strategies to mitigate harm from the failure.
The third step is for critical process that cause serious harm or injury. It requires redesign of the process using human factors engineering and to design robust systems and processes that rarely fail.
You may find you need a project team to support improving reliability. In doing so, you can begin by identifying causes for failures and developing options to improve these processes, test the ideas, and track data to determine whether your changes are improving reliability. This strategy will help you create stronger processes, result in fewer failures, and create lasting change.
In this article, learn more about developing a successful measurement strategy.
TRANSCRIPT | Sickle Cell Awareness Month: Transitioning to College, Equity Considerations, and Resource Sharing
Our main story this episode highlights the need for resources for people living with sickle cell disease and strategies for transitioning from pediatric to adult care. We also reflect on National Infant Mortality Awareness Month and hear from NICHQ team member about connecting their personal and professional passion for equity. Thanks for joining us!
For our main story in this episode we are joined by Summer 2023 NICHQ Communications & Digital Strategy interns Amalia Hirschhorn-Martinez and Katie McCormick. They speak with members of the NICHQ team about our upcoming webinar focused on infant safe sleep and breastfeeding messaging and NICHQ’s continued work to ensure health professionals and the communities they serve are supported with the necessary tools and resources to create safe sleeping environments and improve breastfeeding and chestfeeding rates.
Inviting and Engaging Family Partners in Your Work
Patient and family partnerships are an essential element of health equity. By supporting patient and family voices and encouraging space for collaboration, public health professionals can help ensure shared vision and values are at the forefront of determining solutions to improve a community’s health outcomes.
Importance of Engaging Fathers in MCH Initiatives
Evidence demonstrates that fathers play a critical role in children’s health and development, beginning in the prenatal period and continuing through early childhood and adolescence. On this episode of Before Birth & Beyond, NICHQ honors Men's Health Month and Father's Day, both celebrated in June, by sitting down with two Black fatherhood experts to discuss the importance of including fathers in maternal and child health work, policy considerations to improve father engagement, and men's mental health and its impact on outcomes for children and families.
After Action Reviews Improve Systems, Strengthen Teams
The National Institute for Children's Health Quality (NICHQ) has incorporated AARs as a quality improvement tool for the past four years. Read a conversation with NICHQ Project Director Sandra Widland, MPH, and Associate Project Director Eliza Williamson about the ways NICHQ utilizes AARs in various projects and its benefits to healthcare professionals and others interested in improving systems.
Look for NICHQ at Upcoming Spring Maternal Child Health Conferences
Teams at the National Institute for Children’s Health Quality are preparing for an exciting spring 2023 conference season, where staff will give poster presentations and facilitate workshops at a variety of national maternal and child health conferences.