Antimicrobial susceptibility test breakpoint updates: Challenges and considerations for laboratory validation

Antimicrobial susceptibility testing (AST) is one of the most complex actions performed in the clinical microbiology laboratory.^1-3 The regulatory environment surrounding the update and clearance of minimum inhibitory concentration (MIC) interpretive criteria is equally complicated.² Following a recent update to the College of American Pathologists (CAP) inspection checklist and the growing threat of antimicrobial resistance (AMR),^4-8 many labs are faced with the daunting task of reviewing the breakpoints used on their AST test systems, discontinuing the use of any that may be obsolete, and updating to current AST breakpoints. This process involves local gap analyses and risk assessments, extensive communication with manufacturers, and verification and validation studies.

Laboratory-based validation for AST breakpoint updates

Clinical laboratorians are no strangers to quality control or the analytic methods required to ensure that diagnostic tests function and align with a positive impact on patient care. While test validation and verification are cornerstones of any clinical microbiology laboratory, the requirements for performing these studies, particularly validations, are not always clear. The lack of guidelines is especially true for complex test methodologies like AST, where validation or verification may be required for a situation such as minimum inhibitory concentration (MIC) breakpoint updates, even though the commercial antimicrobial susceptibility test (cAST) instrument has previously been established and verified in the laboratory. The terms verification and validation have been used interchangeably, but they serve fundamentally different purposes. The Clinical and Laboratory Standards Institute (CLSI) M52 document defines verification as “confirmation, through the provision of objective evidence, that specified requirements have been fulfilled (ISO 9000).”⁹ For AST breakpoint updates, this means that if the manufacturer of a test system has received U.S. Food and Drug Administration (FDA) clearance for the new breakpoints on their test system, the laboratory only needs to perform minimal isolate and quality control (QC) testing to ensure that the test system performs locally with the new breakpoints comparable to the performance demonstrated during the FDA clearance process.

Figure 1. Example MIC distribution and calculation of an ECV. Espinel-Ingroff A, Turnidge J. The role of epidemiological cutoff values (ECVs/ECOFFs) in antifungal susceptibility testing and interpretation for uncommon yeasts and moulds. Rev Iberoam Micol. 2016;33(2):63-75. doi:10.1016/j.riam.2016.04.001. — Figure 1. Example MIC distribution and calculation of an ECV.
Espinel-Ingroff A, Turnidge J. The role of epidemiological cutoff values (ECVs/ECOFFs) in antifungal susceptibility testing and interpretation for uncommon yeasts and moulds. Rev Iberoam Micol. 2016;33(2):63-75. doi:10.1016/j.riam.2016.04.001.

Validation is defined as “the term used to describe the process that manufacturers follow to set test performance specifications, also used to describe the process laboratories need to follow to determine test performance specifications for laboratory-developed tests or when FDA-cleared tests are modified by the laboratory.”⁹ For AST breakpoint updates, a validation study establishes test performance data for the AST test system since it has not yet been done by the manufacturer or cleared by the FDA, meaning it is a more rigorous process than a verification study. It is important to note that while AST validation studies are more rigorous than verification studies, they will rarely be as rigorous or complex as the validation study performed by a manufacturer to achieve FDA clearance for a new test or breakpoints.^9,10 Validation studies performed by manufacturers typically involve testing hundreds of isolates across at least three different clinical sites, which involves the analysis of a copious amount of data.9 In addition, clinical laboratories must perform local risk assessments before validation to ensure that the test isolates are representative of local epidemiology and performance of the test system using new breakpoints is accurately assessed. While resources are available to help guide laboratorians through the development of their validation protocols, they are generally nonspecific, and protocol details (i.e., how many isolates to test and what the comparator method should be) are left to the discretion of the laboratory director.

Determination of epidemiologic cutoff values

During the data collection period of the evaluation of a new antimicrobial, hundreds of isolates are tested against the drug to obtain MICs in vitro. The distribution of hundreds of these MICs tells us the organism’s wild-type (or naturally occurring) susceptibility. Using these MIC data, an epidemiologic cutoff value (ECV) can be calculated,^11,12 which separates the population of organisms into those that are wild-type and those that have acquired resistance. While the ECV can help guide the development of a breakpoint, it is strictly experimental and does not consider important factors like clinical or pharmacokinetic/pharmacodynamic data.¹³ While organisms with MICs above the ECV are likely to have acquired resistance and those with MICs below the ECV are likely to represent the wild-type, clinical decisions about the administration of therapy cannot reliably be made from the ECV.¹⁴ See Figure 1.

How breakpoints are determined

In addition to the MIC distribution, multiple other factors are considered when determining breakpoints for a bug/drug combination. For example, data on how the antimicrobial gets distributed throughout the body and how long it stays at certain levels in the blood are evaluated, as well as any clinical data that reflect treatment successes and failures when the antibiotic is administered in certain dosages.¹³ These data help determine an interpretive value that translates a MIC value into either susceptible, intermediate, susceptible dose-dependent, or resistant categories, representing the probability of treatment failure or success in vivo.

Why breakpoints are revised

As an antimicrobial agent is used in clinical practice, new data may emerge that suggest the current breakpoints are not ideal and should be updated (either raised or lowered) to accommodate more appropriate treatment and better patient care.¹⁵ In addition to clinical concerns, breakpoints that don’t effectively capture resistance may inadvertently impact public health and antimicrobial resistance surveillance efforts.^3,8 Each year, CLSI reviews clinical, PK/PD (pharmacokinetic/pharmacodynamic), and other microbiological data to assess if the current breakpoints for a given organism–antibiotic combination are appropriate or need to be updated.¹ This exercise is essential for the practice of clinical infectious disease and microbiology since microorganisms and their susceptibility profiles may change frequently. It is important to note that the breakpoint process reflects this and is not a static process. Clinical microbiologists, infectious disease clinicians, and AST device manufacturers should be aware that modifications may be made regularly, and these groups should be prepared to accommodate and adjust to these changes.¹

Figure 2. Example of piperacillin-tazobactam MIC distribution for Escherichia coli and Klebsiella pneumoniae isolates in the MERINO trial, with current CLSI breakpoint overlayed. Note that the 2022 piperacillin-tazobactam breakpoint falls within the wild-type MIC range. Adapted from CLSI. Piperacillin-Tazobactam Breakpoints for Enterobacterales. 1st Ed. CLSI Rationale Document MR14. Clinical and Laboratory Standards Institute; 2022. — Figure 2. Example of piperacillin-tazobactam MIC distribution for Escherichia coli and Klebsiella pneumoniae isolates in the MERINO trial, with current CLSI breakpoint overlayed. Note that the 2022 piperacillin-tazobactam breakpoint falls within the wild-type MIC range.
Adapted from CLSI. Piperacillin-Tazobactam Breakpoints for Enterobacterales. 1st Ed. CLSI Rationale Document MR14. Clinical and Laboratory Standards Institute; 2022.

Why a full validation is needed for new breakpoints

Many microbiology laboratories have been significantly impacted by the COVID-19 pandemic and are experiencing high-testing volumes and staffing shortages.^16-19 Amidst these challenges, it may feel daunting to perform labor-intensive validations for each organism–antibiotic combination that has not gained FDA clearance for new breakpoints on a cAST device. In addition, since new AST breakpoints impact the interpretation of MIC values and only the categorical agreement between a comparator method and the current cAST is required, microbiologists may wonder why it is necessary to test multiple isolates rather than reanalyze existing data. While the details of a validation protocol are entirely at the discretion of the laboratory director and will differ depending on laboratory resources, patient population, and local epidemiology, here are some important points to consider:

During the initial FDA-clearance process, the analytical performance characteristics of a cAST are established using obsolete breakpoints, correlated to broth microdilution

When a cAST system is validated and cleared at the FDA level, performance characteristics are established using current breakpoints at the time, which are likely tightly correlated with broth microdilution.¹ These breakpoints are obsolete at the time a laboratory is validating new breakpoints, and it is recommended that modern isolates that are representative of isolates regularly tested in the laboratory or isolates with a well-characterized susceptibility profile (i.e., Antibiotic Resistance Isolate Bank)20 be tested on the system to ensure acceptable performance, particularly since correlation to broth microdilution may differ between breakpoints.
Variability of cAST performance may have a greater impact when the breakpoints cut into the MIC distribution

When the MIC distribution of an organism is evaluated and breakpoints are set, it is ideal that the breakpoints do not fall within the wild-type range. When the breakpoints are outside of the wild-type range, interpretation is clear-cut (i.e., isolates with acquired resistance will be in the higher MIC tail, away from most isolates that represent the wild type). Sometimes clinical needs dictate that the breakpoints be set within the wild-type distribution,¹⁵ but this can be challenging because “susceptible” and “resistant” organisms are not truly separated by acquired resistance. The variability in AST test system performance can introduce unique challenges for validation and establishing performance.^15,21,22 Essentially, there is less “wiggle” room for variability in this scenario, and performing a validation by testing bacterial isolates is recommended to assess performance in each local laboratory setting.

3. Using existing cAST data as a comparator may result in unacceptable levels of errors

According to current guidance documents,⁹ a comparator method is required when performing laboratory validation. In the case of applying new breakpoints and reanalyzing AST data, the comparator is AST data with old breakpoints applied. This method may result in unacceptable levels of major and very major errors, particularly when the new breakpoints fall within the wild-type MIC range. For example, in a recent CLSI webinar,²³ Dr. Romney Humphries provided an example by reanalyzing data from the MERINO trial²⁴ using new piperacillin-tazobactam breakpoints. Doing this resulted in a 66% very major error rate, a rate unacceptable for test validation in the microbiology laboratory. See Figure 2.

Future steps

AST is complex and further complicated by the breakpoint update process and regulatory environment. However, routine AST and breakpoint updates are crucial for optimal patient care and public health. Clinical microbiology laboratories should aim to perform prudent validations that accurately assess and implement new AST breakpoints as soon as possible, but this will require extensive support from scientific agencies and cAST manufacturers. Continued education, communication, and streamlining of complex processes will help ensure that accurate AST breakpoints are implemented and utilized, improving patient management, and supporting public health AMR prevention efforts.

References

Humphries RM, Abbott AN, Hindler JA. Understanding and addressing CLSI breakpoint revisions: A primer for clinical laboratories. J Clin Microbiol. 2019;57(6). doi: 10.1128/JCM.00203-19.
Humphries RM, Hindler JA. Emerging Resistance, New Antimicrobial Agents … but No Tests! The Challenge of Antimicrobial Susceptibility Testing in the Current US Regulatory Landscape. Clin Infect Dis. 2016;63(1):83-88. doi: 10.1093/cid/ciw201.
Prinzi AM. Updating Breakpoints in Antimicrobial Susceptibility Testing. In: McAdam A, ed. American Society for Microbiology 2022.
Newitt VN. AST and safety at core of microbiology checklist changes. CAP Today 2021. Accessed September 22, 2022. https://www.captodayonline.com/ast-and-safety-at-core-of-microbiology-checklist-changes/.
AST news update June 2022: Updating breakpoints–new developments from CAP. Clinical & Laboratory Standards Institute. Accessed September 23, 2022. https://clsi.org/about/blog/ast-news-update-june-2022-updating-breakpoints-new-developments-from-cap/.
Titus K. Leaving behind outdated AST breakpoints. CAP Today. 2022. Accessed September 22, 2022. https://www.captodayonline.com/leaving-behind-outdated-ast-breakpoints/.
Titus K. AST breakpoints: a case of not aging gracefully. CAP Today. 2020. Accessed September 22, 2022. https://www.captodayonline.com/ast-breakpoints-a-case-of-not-aging-gracefully/.
Prinzi A, Hunt, Lauren N, Goff, Debra, Rohde, Rodney E. Antimicrobial Resistance: A Review of a Broad-Spectrum Problem and Future Needs. InfectionControlTips. 2022. Published April 27, 2022. Accessed September 22, 2022. https://infectioncontrol.tips/2022/04/27/antimicrobial-resistance-a-review-of-a-broad-spectrum-problem-and-future-needs/.
CLSI. Verification of Commercial Microbial Identification and Antimicrobial Susceptibility Test Systems. 1st ed. CLSI guideline M52. Wayne, PA: Clinical and Laboratory Standards Institute; 2015.
U.S. Food and Drug Administration. Antimicrobial Susceptibility Test (AST) Systems - Class II Special Controls Guidance for Industry and FDA. Guidance Documents (Medical Devices and Radiation Emitting Products) Web site. Published 2018. Accessed September 22, 2022. https://www.fda.gov/medical-devices/guidance-documents-medical-devices-and-radiation-emitting-products/antimicrobial-susceptibility-test-ast-systems-class-ii-special-controls-guidance-industry-and-fda.
CSLI. ECOFFinder. CLSI. Published 2022. Accessed September 22, 2022. https://clsi.org/meetings/susceptibility-testing-subcommittees/ecoffinder/.
Schuetz AN. Role of Bacterial and Fungal Epidemiological Cutoff Values for Antimicrobial Susceptibility Testing in the Microbiology Laboratory. Clinical Microbiology Newsletter. 2020;42(2):9-17.
Monoguen MS, James Clark, Andrew. Breakpoint Breakdown. In. Lablogatory: American Society for Clinical Pathologists (ASCP); 2020.
Edelstein P. Epidemiologic Cutoff Values. University of Pennsylvania. Published 2017. Accessed September 22, 2022. https://www.uphs.upenn.edu/bugdrug/antibiotic_manual/ECV.htm.
Turnidge J, Paterson DL. Setting and revising antibacterial susceptibility breakpoints. Clin Microbiol Rev. 2007 Jul;20(3):391-408, table of contents. doi: 10.1128/CMR.00047-06.
Durant TJS, Peaper DR, Ferguson D, Schulz WL. Impact of COVID-19 Pandemic on Laboratory Utilization. J Appl Lab Med. 2020 Nov 1;5(6):1194-1205. doi: 10.1093/jalm/jfaa121.
Ahmed S, Jahan F, Naeem Effendi MU, Ghani F. Impact of COVID-19 on the pre and post analytical clinical laboratory testing processes- A performance evaluation study using six sigma. Ann Med Surg (Lond). 2021 Oct; 70:102842. doi: 10.1016/j.amsu.2021.102842.
Jones A. Laboratory Morale During the Covid-19 Pandemic. ASCLS Today. 2021;35. Accessed September 22, 2022. https://ascls.org/laboratory-morale-during-the-covid-19-pandemic/.
Olayinka O, Odujoko O, Barasch S, Farley J, Woodruff C, Chan S. A Retrospective Review of the Effect of COVID-19 Pandemic on Laboratory Utilization. American Journal of Clinical Pathology. 2021;156(Supplement_1):S112-S112.
CDC and FDA. How the AR Isolate Bank helps combat antibiotic resistance. Centers for Disease Control and Prevention. Published August 9, 2022. Accessed September 23, 2022. https://www.cdc.gov/drugresistance/resistance-bank/index.html.
Humphries RM, Ambler J, Mitchell SL, Castanheira M, Dingle T, Hindler JA, Koeth L, Sei K; CLSI Methods Development and Standardization Working Group of the Subcommittee on Antimicrobial Susceptibility Testing. CLSI Methods Development and Standardization Working Group Best Practices for Evaluation of Antimicrobial Susceptibility Tests. J Clin Microbiol. 2018 Mar 26;56(4):e01934-17. doi: 10.1128/JCM.01934-17.
Henderson A, Paterson DL, Chatfield MD, et al. Association between minimum inhibitory concentration, beta-lactamase genes and mortality for patients treated with piperacillin/tazobactam or meropenem from the MERINO study. Clin Infect Dis. 2021;73(11):e3842-e3850. doi:10.1093/cid/ciaa1479.
(CLSI) CaLSI. June 2022 AST Education Session: Updating Breakpoints—Challenges and Solutions for Various Stakeholders. CLSI. Published 2022. Accessed September 22, 2022. https://clsi.org/standards/products/webinars/education/astedujune22wr/.
Harris PNA, Tambyah PA, Lye DC, et al. Effect of piperacillin-tazobactam vs meropenem on 30-day mortality for patients with E coli or Klebsiella pneumoniae bloodstream infection and ceftriaxone resistance: A randomized clinical trial. JAMA. 2018 Sep 11;320(10):984-994. doi: 10.1001/jama.2018.12163.