The most essential aspect of any laboratory test is that it provides correct results. Perfection cannot be achieved, but it should be continuously sought. For this reason, diagnostic scientists utilize a framework of quality control.
The advent of nucleic acid-based testing has been a significant addition to the field of diagnostic science. Such testing, often termed, “molecular,” includes the exploitation of the coding sequence of nucleic acid as a source of derived specificity. Molecular testing has been particularly impactful in the field of infectious disease, where organisms can be readily differentiated by DNA or RNA sequences that are unique to their species (or even sub-species). This means that molecular tests can be designed so that they theoretically have singular specificity for only their intended target. Additionally, many molecular tests operate on the principle of “target amplification,” which denotes that detection of analytes is performed through amplification of the specific actual nucleic acid sequence target. Examples of such methods include polymerase chain reaction, transcription mediated amplification, and strand displacement amplification, to name a few.
Target amplification has resulted in tests of exquisite sensitivity, with certain tests being capable of detecting less than a single organism. The exquisite sensitivity of target amplification can also be an Achilles’ heel, however. With such sensitivity, even the minutest amount of contamination with a target sequence, or one quite similar to it in some cases, can provide false positive results. Moreover, the enzymes that perform target amplification are themselves temperamental little machines, in highly balanced chemical environs which are inhibited often by simple and common molecules and can result in failed amplifications and false negative results. Verification and validation—perhaps no two words are more discussed among laboratory professionals. That is not because either of these nouns refer to a frequent event in a laboratory, but because few of us have a firm understanding of what the differences in their meaning are.
Laboratory tests generally fall into two categories: those that are cleared by the U.S. Food and Drug Administration (FDA) and those that are not. The tests that are not are often referred to as laboratory developed tests or LDTs. These two classes of tests are treated differently when they are assessed for use in the diagnostic laboratory environment.
The process of verification
FDA-cleared assays are vetted through extensive internal or clinical trial processes that include the assembly of a large amount of performance data for a test. When a test is cleared, and subsequently marketed and sold, it must include a document (called a “package insert”) which includes the most relevant data from such trials, including specificity and sensitivity. For molecular tests, there will always also be an assessment of “inclusivity” and “exclusivity,” which are assessments of the ability of the test to react with many strains of a particular species (inclusivity); and not to react with species of organisms that are not the intended target (exclusivity). Regulation requires that such FDA cleared assays must be verified prior to use at a diagnostic laboratory site. In verifying an assay, a lab director is verifying the performance claims of the product’s package insert at his or her
diagnostic site.
In the process of verification, one is determining whether a particular product works as expected, using the equipment, personnel, and geography of one’s own laboratory. Geography can bring a host of physical principles into play that can affect the performance of molecular tests and their associated equipment, including humidity and temperature factors that can most assuredly affect test results. It is very important to note that a test performed in a laboratory is only FDA cleared when the test is run according to the strict direction of the package insert. Any deviation from such instructions immediately causes a test to be classified as an LDT.
The CLIA regulations are clear with regard to what assessments must be made for a verification study. (It must be recognized that molecular tests can be both qualitative and quantitative, and this is commented upon, below.) The requirements for a verification study include:
Accuracy. This includes an assessment of how well the test correctly determines the disposition of a specimen. Specimens can be purchased calibrators, quality control materials, or patient specimens for which a result is already determined and traceable back to a reference standard or method. The number of such specimens is not indicated in the regulations. Most laboratorians consider the number twenty to be a minimum number of specimens evaluated. Viewed from the scientific or statistical perspective, more is always better in the course of generating a data set in which there is high confidence.
Precision. Accuracy assesses how “on target” a test is, relative to a gold standard; precision assesses how repeatable a test result is. It is often assessed by testing a singular specimen (of adequate volume) to be evaluated on the test platform on multiple instances, preferably by different operators on different days of the week. This accounts for any variability in test performance or environment. Specimens may include patient samples or quality control materials.
Reportable range. For qualitative tests, this is essentially negligible, as test results for a qualitative molecular test are positive or negative. However, it is important to spell that out in this part of the verification report. Such results were evaluated in the Accuracy phase when positive and negative specimens were assessed. For quantitative tests, this is a more intensive operation. It includes the assessment of enough specimens with adequate loads to show that a test can accurately assess results across a necessary range. For example, a typical viral load test that claims to have a reportable quantification range of 50 to 10,000,000 would have to be assessed to show that all specimens of various loads across that range, inclusive of the boundaries, are quantifiable.
Reference range. This is the value that the test result would provide for a “normal” or non-afflicted individual. In qualitative molecular tests for infectious agents, this value is “negative.” For quantitative molecular tests, this might most likely be “target not detected” or zero.
The process of validation
When a test system that is being assessed for use is non-FDA cleared, or if it is a modification of an FDA cleared test, the test system must be validated for medical use. Validation is a process that is very similar to verification, requiring everything in a verification as described above, but it also includes the following:
Analytical sensitivity. This is the process whereby the limits of detection are assessed for a particular test. Whether qualitative or quantitative, this minimal concentration of analyte(s) that a particular test can detect must be determined. For qualitative tests, this is often best performed using quality control or calibrator materials (purchased from a vendor) that have been accurately quantified. Dilutions of such materials across a broad range inclusive of a hypothesized threshold of detection would be adequate. The lowest dilution detectable would contain the assessed analytical sensitivity. In molecular tests like PCR, there is a theoretical limit of one target nucleic acid molecule, but inefficiencies in extraction and amplification
often make that number much higher.
Analytical specificity. Molecular tests are of considerable value not only for their ability to sensitively detect targets, but to discriminate on the basis of nucleic acid sequence. While theoretically this builds high specificity into a test, it is not guaranteed. For that reason, such tests must be further assessed. This is best achieved in two ways: a) by challenging the test with specific organisms that may be found at the same physiologic environment as a typical tested specimen, and b) by challenging the test with specimens of such physiologic regions known not to contain the target.
Achieving “a” above involves generating a list of known organisms, either closely related to, or found in physiologic concordance with, the target of the test in question. Those organisms are obtained/purchased, cultured, and then diluted into the appropriate testing matrix. Quality control materials are of value here as well, if available. Diluted specimens are subject to the test in question and results are tabulated. A test specific for a particular target should not react with close relatives, or other organisms in the same physiologic site.
Achieving “b” above is also very helpful in assessing specificity and is perhaps more meaningful scientifically. It includes obtaining known-negative specimens from the physiologic sites relevant to the test and assessing them. Such specimens should contain the realistic amount and ecologic variety of the flora of the tested site, and thus should challenge the test in question with the most relevant materials.
Three closing points
- Many molecular methods include discrete steps of nucleic acid extraction and amplification. It is imperative that both aspects of the test are challenged by validation and verification. An amplification platform cannot be “fronted” with a variety of extraction methods unless each and every extraction method has been included in the verification or validation.
- In many molecular LDTs, analyte specific reagents for a test would include nucleic acid primers and/or probes (“oligos” as they are often called) that are manufactured in batches upon request by an external agency. It is crucial that quality control measures be established to continuously validate the functionality of batches of such materials.
- The verification or validation of a particular specimen type does not guarantee functionality of any specimen type. Assessing the accuracy of a test for detecting an analyte in stool does not validate or verify a test’s ability to detect the same analyte in urine, for example.
Mark W. Pandori, PhD, HCLD(ABB), serves as Director of Laboratory Services at Alameda County Department of Public Health and Associate Professor at the University of California, San Francisco.
