LEARNING OBJECTIVES 1. Discuss CLIA’s role in the assurance of quality testing in laboratories. 2. Identify the major topics of concern that PT users discussed in the focus group. 3. Describe the constraints involved with each topic of concern that PT users have identified. 4. Identify the many benefits that proficiency testing programs have to offer.
As an external assessment of quality, proficiency testing (PT) is recognized as one essential component for assuring quality testing in clinical laboratories.1-3 Each laboratory that performs non-waived testing is required by the Clinical Laboratory Improvement Amendments of 1988 (CLIA) to perform PT. The CLIA regulations specify requirements for PT program provider approval by the Department of Health and Human Services (HHS).
The CLIA law changed the paradigm for providers from an educational to a more regulatory role. Consequently, there are often misperceptions about providers’ operations and their limitations. This report presents some of the common viewpoints held by clinical laboratory professionals and the corresponding perspectives shared by some PT providers. The report is intended to assist clinical laboratories in understanding the constraints faced by PT program providers, learning about potentially helpful provider services, communicating with their providers, and making knowledgeable inquiries as they search for PT services.
In 2012, the Centers for Disease Control and Prevention (CDC), working in collaboration with the Association of Public Health Laboratories, conducted a series of focus groups that included clinical and public health laboratory professionals to explore how they use and perceive PT.4-5 Some discussions concerned participants’ satisfaction with PT provider services. Participants spoke of issues they had experienced and made recommendations for providers to consider. As a follow-up to the focus groups, in 2015 CDC developed several open-ended questions about some of these issues and sent them to all 11 HHS-approved PT program providers that provide, at a minimum, chemistry analytes. Seven providers furnished either written or verbal answers, which are summarized in this article. Respondents included providers which offer a wide range of programs: programs affiliated with accreditation organizations; those that offer a small number of programs to specific types of laboratories (e.g. physician office laboratories); and some independent and state-affiliated programs.
The 2012 focus groups clearly expressed a desire for additional commercially available PT analytes or modules. CDC asked the providers to describe their criteria for adding new analytes, as clinical laboratory professionals typically are not aware of the processes used to choose new analytes and develop PT modules. All of the providers stated that the most important factor is the level of interest because a certain threshold is required for sample grading, acceptable performance statistics, and reasonable cost. Therefore, they may not offer a particular analyte because the potential pool of users is too small. The providers determine the level of interest by directly asking customers, tracking requests for the new analyte, or utilizing a scientific advisory committee. Providers must consider analyte stability, methodologies, and sample matrices used for testing. Because most providers do not prepare samples but rather contract this service, they must also consider whether there are reliable suppliers of the materials.
Takeaway: Laboratory professionals can potentially influence their PT programs by asking for new analytes.
Ungradable samples were another concern for the focus groups. Some participants stated there should be more quality control during sample development and manufacturing. While providers may not be directly involved in preparing samples, they do have processes in place to help ensure the best possible product. Our follow-up survey indicated that the providers select manufacturers that follow good manufacturing processes, provide evidence that they can prepare quality samples, and adhere to strict specifications for analyte levels, stability, and storage. To address potential problems with PT samples, each respondent retains extra samples for off-schedule shipping, training, and validation for as long as the analyte is stable. If a problem does occur, the providers try to be as fair as possible when grading. Laboratory professionals should contact their PT provider as soon as possible if there are any problems with the shipment, storage, or package contents.
Most providers first offer the analyte as a pilot program or as educational samples and collect data regarding specimen performance for the test methods used. A few send the potential specimens to collaborating laboratories to test. One provider requires in vitro diagnostics (IVD) manufacturers to purchase test materials if they want to be listed as discrete method group (peer group) rather than being compared to all test methods. This allows the provider to discover problems and to work with the sample manufacturers for a resolution.
Another concern for the focus groups was that some samples required complex reconstitution procedures. However, all seven providers responded that they ship most samples frozen or as a liquid. The exceptions are the coagulation analytes because they are more stable when lyophilized, analytes that are not available in human whole blood or serum, and live organisms that could infect the end user.
The focus groups indicated they expected PT samples to behave like patient specimens on new instruments or methods that have been validated with patient specimens. Providers were asked how they ensure that PT samples will perform correctly for their clients’ instruments or methods, whether extra samples are retained for use after the PT results have been graded, and if they allow manufacturers to purchase specimens or subscribe to the provider. Because it is impossible to pre-test PT samples on each of the instruments and methods used in U.S. clinical laboratories, most providers rely on feedback from the sample manufacturers, instrument manufacturers, and customers. Again, voluntary feedback from laboratory professionals helps identify problems with PT as quickly as possible.
The providers in this survey strongly encourage IVD manufacturers to purchase PT materials to test or to subscribe to a PT program to check their instrument or method performance. However, according to one provider, IVD manufacturers appear to have scaled back on this practice. Some PT program providers post their summary statistics online so IVD manufacturers can check results for their instrument or method, even if they do not subscribe to the PT program. Providers also look at new and upcoming instrumentation and methodologies at conferences and trade shows to help them determine if PT samples will require modification as more laboratories adopt new platforms. One provider asks instrument manufacturers what materials will work best for their instrument. For an international laboratory that requests subscription to PT programs with sample-specific materials, such as hematology, one provider uses the most common sample material already used for PT samples and, based on performance, will decide if another material would work better. Generally, providers will note any instrument incompatibility in their catalogs or provide various modules to allow for the differences in methodologies.
Takeaway: Providers evaluate sample material performance with new instrumentation and methodologies by communicating with their participants and with instrument manufacturers.
Regarding microbiology PT, focus groups voiced concerns about the quality and types of photographs used, the quality of stained slides, and the absence of multiple fields and views when photographs are provided. Some providers send only hard copies of digital photos, some use digital photos in electronic format only, and some use both. Providers favor digital photos because all participants receive the same image, placing everyone on a level playing field. Two providers noted that the use of arrows to point out a particular cell or organism can cause the participant to “overthink” the result and report an abnormality for a normal sample. Alternatively, not having an arrow to point out the cell of interest can also cause overthinking and lead to incorrect results.
Stained slide samples pose issues as well. One provider stated that there can be laboratory-specific preference for the staining intensity or, occasionally, the cells on the sample slide may not absorb the stain well. Consequently, this provider uses experts to check the slide quality. Sometimes unexpected factors affect the quality of stained slides. For example, another provider related that a participant noticed that KOH slides performed better after being stored at room temperature instead of refrigerated. The provider asked the manufacturer to perform a study which demonstrated that prolonged refrigeration led to decreased performance of the samples. The provider changed the instructions to reflect this finding so all participants would benefit.
Three providers remarked that parasitology is particularly challenging because finding a manufacturer that has quality specimens in a large enough volume is difficult, and patient specimens with multiple parasites are not as useful for PT. Many specimens are acquired in Africa, and unexpected events, such as the recent Ebola outbreak, may result in a temporary need to find new specimen sources. Specimens containing parasites are rare in the U.S. because U.S. patients tend to be treated quickly and have very few detectable parasites in their specimens.
Takeaway: Laboratory professionals may benefit by comparing programs and obtaining advice from their peers.
Grading is an important issue to laboratory professionals, yet many in the focus groups were uninformed about how their provider determines their scores. We have previously described the fundamentals of peer-grouping, which sorts the PT users into groups for scoring based upon their testing platform.6 CLIA allows peer-grouping on the supposition that differences between PT results on different IVD platforms may be due to peculiarities inherent in the PT material (a matrix effect) rather than inaccuracy that would necessarily occur when testing patient specimens. Matrix effects can be especially problematic for grading when there are very few laboratories in a peer group. For example, if a potential peer group contains only two or three laboratories and the results trend together but away from the other peer groups, then they would be ungraded until there were enough participants. According to one PT provider, sometimes five is adequate for a qualitative method, but at least 10 would be necessary for quantitative tests. Some providers use a group of referees if there is no consensus for grading a sample, but finding available referees can be difficult.
Takeaway: Laboratories that are enrolled in programs that have too few participants may want to consider finding another PT provider.
Focus group participants expressed consternation about discrepancies between the units they report for patient testing and those of the provider. Some providers allow for alternate units based on the methodology being used. Their software will convert the measurements to standard units. For those providers that do not allow alternate units, the default is the units used by most participants, and those using alternate units are responsible for converting their results into the default units. One provider can make a change to its software to account for alternate units if it is notified before participants report the data. Another provider stated that it determines whether its system could convert the results to a standard unit or if these participants should be placed in a separate group when inquiries are received from multiple participants. One provider reported that new reagents are reviewed by technical staff, and if a difference in units is found, it automatically takes the appropriate steps to ensure participants can report their results.
The focus groups recommended that the paperwork associated with reporting PT results be reduced because it was easy to make slight transcription errors. In response to this, all seven providers said they offer electronic reporting. One provider allows electronic reporting only; the other six providers continue to accept hard copy or faxed results. While human error is always possible, the providers commented that with electronic reporting the laboratory professional can check results for accuracy or validate the data before submitting it to the provider. Also, data fields for standard information, such as method code, may be programmed into the system or may be added to drop-down lists to further reduce the chance of a clerical error. Currently, electronic data reporting appears to be the most effective way to reduce clerical errors.
Takeaway: Reporting is another area in which providers differ, so laboratories should compare them.
Focus group participants generally acknowledged that the report turnaround time was satisfactory, yet some believed that getting results sooner would improve the utility of PT. CDC asked providers about the process and examples of what may cause delays. All of the providers agreed that reporting online made analysis easier because all results are received in a standard format ready for analysis. A common misperception noted by the providers is that each analyte has target and acceptance criteria, permitting analysis as soon as the participants’ results are received. In reality, the statistics, including peer group means to determine targets, are based on all results from all participants, and analysis cannot begin until all results are submitted.
Perhaps the most important determinant of turnaround time for scoring is how long it takes for providers to obtain all of the participants’ results. Reporting via hard copy delays analysis of the results because the information must be scanned, checked, and uploaded into the provider’s system. One provider estimated that for each batch of 500 participant reports, it would take approximately one week to manually process faxed or hard copy data, and additional time is required to complete other manual processes, including outlier removal. Microbiology reports from one provider sometimes include a detailed critique that is written like a tutorial to help laboratories in future PT events. Depending upon the size of the provider, turnaround time ranged from five days to three weeks from event closure to the online release of scores. As an incentive, one provider allows additional time to submit results for those laboratories that report online.
Takeaway: Providers cannot analyze PT results until all reports have been received, which may affect turnaround times.
Many of the focus group participants felt that they could reduce their costs if providers would provide customized modules. All providers stated that they offer as much customization as possible, including tiered pricing based on the number of analytes tested, large modules to cover most of the PT needs in one module, instrument-specific modules, and a variety of culture or method groupings. The biggest obstacles are the production cost of the samples and the limited number of manufacturers that make PT samples. The fundamental issue is that samples are prepared in large batches, making it extremely difficult to produce the different combinations that laboratories want. Manufacturers often produce samples on a yearly cycle and are reluctant to make a new sample in a small batch in the middle of the year. Additionally, hundreds of different combinations would be created, which would be impossible to track and evaluate using current computer systems.
If it were possible to provide customized modules, the price would increase and would be passed on to the laboratories. To keep costs reasonable, providers receive price quotes from each manufacturer. Offering modules with many different analytes reduces the expense for the providers, thus keeping costs down for their clients. Providers try to group analytes into appropriate modules so they are amenable to as many customers as possible. For example, in microbiology, combinations of matrices such as throat swabs and urine are offered to reduce the required number of PT modules. Also, related organisms are grouped into one module (e.g., enteric pathogens). Providers can help the laboratories decide which modules will best fit their needs, especially if a laboratory’s testing menu has changed.
Takeaway: There are differences in costs of PT modules, so laboratory professionals may wish to compare the PT providers.
Ancillary services that providers may offer include CME credit offerings, telephone access to a laboratory professional familiar with testing methodologies for technical advice, and telephone or email access to the staff member who scored the PT results if the laboratory has a question about its evaluation. Providers may offer program guides with detailed information about the individual programs or have staff available to offer guidance throughout the entire PT process, from choosing the best program and how to order it to explanations of the evaluation report.
One provider offers the ability to upload data directly from a laboratory information system (LIS) rather than manually entering data online. For laboratories that require very specialized PT (laboratory-developed test analytes, rare analytes, etc.), one provider offered to act as an “honest broker.” Instead of the laboratory blind-coding, testing, and evaluating samples internally, it could send samples and a sample key to the PT provider, who would randomize and blind-code them and send them back to the laboratory for testing. The laboratory would send back the results, and the provider would score them and deliver an evaluation.
Although PT can be expensive for laboratories, it is a valuable investment and tool for ongoing monitoring of test performance. Making informed decisions about the programs and services providers offer helps laboratories find the best provider to fit their needs. Aspects of PT that cause frustration among laboratory professionals may have resulted from the providers’ operational processes and their limitations. The value of the educational component of PT may be overlooked sometimes because of PT’s regulatory implications. Reviewing PT results over time to track trends and instrument performance and using leftover samples for analyst competency tests are some other uses beyond the regulatory component. It is important for laboratories to assess their PT programs periodically because, as laboratories’ requirements change, a different PT program or PT program provider may better fit their needs.
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Marie C. Earley, PhD, and J. Rex Astles, PhD, serve as health scientists at the Centers for Disease Control and Prevention (CDC), Laboratory Practice Standards Branch (LPSB), and work in support of the Clinical Laboratory Standards Amendments of 1988 (CLIA) regulatory program.
CDC asked providers: If you had an opportunity to say one thing to your clients, what would you want them to know? Here are some of the answers:
