So, your rules auto-verification program is up and running. Your auto-verification rate is 85 percent or better. Your lab productivity has dramatically improved and you are happy! It has been over a year since you verified your rules. You’ve made a few changes, inactivated and added some rules, edited your value lists and now your CAP inspection is coming up. Now what? Do you have a plan to make sure that your rules changes are effective? Will they have the desired outcome with the same or improved auto-verification rate?
The quality of the auto-verification rules when implemented has a direct impact on the efficiency of the rule logic and outcome of patient results. However, post-implementation and over time, it becomes a more challenging task to maintain consistent quality of those decision rules. Changes are made to rules for specific clinical reasons, but not always in the context of the full rules set. Inadvertent changes can creep into the rules software that can have unintended consequences on rule performance. Often rules oversight is not centralized; several staff could be involved in rules configuration, and, although well-intentioned, could introduce issues that go unnoticed.
A rule quality monitoring program can help laboratories establish better controls that continuously assess rule quality and performance. Rule performance monitoring should be incorporated into the laboratory’s overall quality assurance program to ensure the decision rules are meeting a high auto-verification rate all of the time. Even the smallest of changes can impact the quality of your rules and erode your auto-verification rate with intended or often inadvertent changes to the rule base. A rule quality monitoring program has the benefit of being able to pinpoint problems and detect subtle changes to your rules operation to ensure no unintended modifications have been introduced into your rule sets.
The following are recommended components of an auto-verification quality assurance program that can assist in maintaining the integrity of your auto-verification rules set over the long-term. Rules sets are never static and require a multi-layered approach to keep a rule base intact and functional to support your patient quality and safety goals.
Rule quality assurance program recommendations
- Auto-verification rate monitoring: Periodic review of your auto-verification rate (AVR) is highly recommended to identify any rule degradation or opportunities for improvement. There is a direct relationship between rule integrity and performance. Each lab or site should establish an AVR that is achievable based on the lab’s clinical objectives. An AVR is a key laboratory operational metric to ensure that when rules are added, adjusted, or changed based on clinical practices that the AVR is not impacted or reduced as a result of rule maintenance activities. Checking your AVR on a consistent basis will detect any latent or unexpected changes that could impact rule performance.
- Verify rule values: Periodic review of rule input values are important to confirm they are current with the lab’s clinical practices. On an annual basis, a process must be in place to confirm that the data variables and value lists are current and accurate within the rule set. Rule inputs such as reference ranges, critical values, delta logic, and instrument flagging are important to verify to protect against any missing or inadvertent changes to the rule set that could degrade the AVR. Create a source of truth document to establish a baseline for on-going reference and for use in verifying your rules inputs.
- Check rule build logic: An important component of rule quality is the configuration of the rule according to the correct software syntax and the order of the rule actions. Because rule executions may occur in steps with rule parent-child relationships, the rule operation workflow must be reviewed periodically to identify any gaps and to avoid multiple failures in the rule processing workflow. An annual process of verification of mapping of the parent-child relationship is recommended before rule verification testing to ensure no changes have occurred that might interrupt the rule prioritization workflow.
- Maintain rule traceability matrix: Each laboratory should have a definitive list of its auto-verification rules. All rule changes should be documented and tested in a ‘test’ environment before placed into production using good rule testing practices. Rules that are no longer needed should be inactivated (never deleted) to maintain rule traceability over time. A traceability matrix of each rule by number or identifier should be maintained to certify that all rules can be accounted for and no inadvertent actions have removed or deleted a rule or its contents.
- Establish a rule verification testing schedule: Auto-verification rules should be verified at key points to ensure viability. Rules should be tested at least annually to meet regulatory requirements, when there are changes to the rule components and when new rules are introduced. Creating a rule testing schedule for the year ahead will keep you on track to ensure your rules are always operating at peak performance.
