Does reagent quality play a role in reproducibility of experimental data?

Currently, there is a great deal of (necessary) attention focused on the lack of reproducibility of scientific studies, often caused by bias due to small sample size, cherry picking data, and selective reporting. Confidence in published research comes with independent replication of findings. The impact of these findings on the scientific community can be significant—particularly as downstream research undertakings can potentially lead to avenues for innovation of diagnostics, or disease treatments, for example.

The consequence of irreproducible results in the clinical setting can—at a minimum—require retesting of patients. At worst, it can yield inaccurate patient results, with potentially life-threatening consequences.

Reagent quality is key

While many variables affect the reproducibility of experimental data, reagent quality should be considered one of the most fundamental requirements.

In the field of life sciences, a high-quality reagent is one that generates reproducible results in an otherwise identical experiment. Reagent production in Good Manufacturing Practice (GMP) facilities ensures a standard that minimizes variation during assay or drug development and, ultimately, an introduction into the clinical setting. It covers all aspects of production, including the location, starting resources, equipment, and staff training.

However, manufacturers of life science reagents more often supply products for “research use only” in academic settings, contract research organizations (CROs), pharmaceutical companies, and other biotechnology companies. In these instances, reagent quality is just as critical, since reproducibility of results can underpin downstream research efforts. Research findings are often scrutinized in peer-reviewed journals, and the researcher’s reputation and the lasting significance of his or her findings rests on the reproducibility of the protocols in an entirely independent setting, by an independent researcher, using independent equipment.

Thus, choosing a reagent that performs reliably, particularly in more sensitive applications, is a critical first step toward improving the reproducibility of research. Considerations such as manufacturing processes, quality control testing, sourcing of raw materials, packaging, and shipping all contribute to the development of a high-quality product.

In 1978, New England Biolabs (NEB) produced a recombinant protein for research purposes, which enables streamlined consistency of product quality. The cloning of restriction endonucleases and many other enzymes for use in recombinant DNA technologies has enabled the implementation of stringent control measures. Native bacterial strains often do not produce sufficient amounts of the enzyme-of-interest. Additionally, other enzymes may be co-purified in the final product, and there may be impurities in the native strain, such as non-specific endo- and exo-nucleases. We can address these challenges by choosing a specific expression vector and host strain for consistent expression and production of recombinant enzymes in a tightly controlled system. Not only can impurities—such as DNases, RNases, or phosphatases—be reduced, but also yields are higher, resulting in greater product consistency and less lot-to-lot variation.

Once a product is manufactured, enzyme quality is continually monitored through a set of quality controls (QCs) that address specific attributes, such as concentration, purity, functionality, and reliability. Are there contaminants (i.e., DNase or proteins in DNA products)? Does the enzyme maintain consistent activity over an extended period of time?

QCs also need to be continually reassessed and improved as technologies improve and sensitivities increase. For example, a contaminant that was previously undetectable may now be identified and measured due to the rapid technological advances of sequencing instruments.

Product application in QC

Product application is another factor to consider when developing and reassessing a set of QCs. In fact, a QC test that was developed many years ago may no longer be sensitive enough to satisfy today’s molecular diagnostic applications. It is therefore important to continually refine QC practices when developing and maintaining reagents through their lifecycle. For example, residual E.coli DNA contamination from a host strain might not affect a targeted qPCR experiment and can be identified and ignored in many sequencing applications. But it needs to be removed entirely for a pathogenic E.coli test.

Quality measures sometimes require the complete rethinking of production methods—for example, in cases such as the re-engineering of clones to produce more product or eliminate targeted background contaminants (proteases) in a particular strain, or the development of different fermentation processes. Sourcing materials from outside vendors can introduce another variable in the process of reagent production. It is thus vitally important that vendors have in place a sound quality management system.

Several platform tests from NEB that cover multiple reagents have been developed or refined as part of a quality improvement effort that helps to bring about best practices. The analyses include qPCR, RNase detection, capillary electrophoresis (CE) exonuclease detection, CE phosphatase detection, CE polymerase detection, protein purity, protein concentration, and identification by mass spectrometry.

The use of mass spectrometry for the detection and monitoring of mutations caused by toxic overexpression of a recombinant product is an essential layer of information for the maintenance of a high-quality product, regardless of whether the mutation affects the activity of the end-product. Further, customers who are developing a platform for a regulated market are supplied with three independent lots to validate their test with NEB products and demonstrate reproducible results.

Case study: QC practices when manufacturing T4 DNA Ligase

As an example, NEB considers all these factors when it manufactures its T4 DNA Ligase. Ligases are an important tool in molecular biology that facilitate the joining of DNA strands for the construction of recombinant DNA. NEB’s ligase production methodologies incorporate extensive QCs that include testing for:

• Endonuclease activity using agarose gel electrophoresis to examine nicking of supercoiled DNA
• Exonuclease activity by assessing the release of radioactive nucleotides following incubation of ligase with radiolabeled single- and double-stranded DNA
• Non-specific DNase activity, which is evaluated by agarose gel electrophoresis following incubation with a DNA substrate
• Protein purity using SDS-PAGE to compare contaminating protein bands to the protein-of-interest
• RNase activity, which is assessed by gel electrophoresis following incubation (for two and 16 hours) with an RNA substrate
• Functional testing, which encompasses blunt- and cohesive-end ligation and subsequent transformation efficiency

The extreme purity of NEB’s T4 DNA Ligase can be seen in Figure 1, where equivalent amounts of protein from alternate suppliers were analyzed side-by-side, and in Figure 2, where T4 DNA ligase samples from different suppliers were examined for contaminating nucleases.

The importance of corporate culture

Finally, corporate culture is paramount to the efforts to provide quality products. An overarching focus on customers and their needs, coupled with a commitment to achieving and maintaining the highest possible quality—from product inception through development, production, packaging, and shipping protocols—ensures the optimal activity and consistency of the product. A company that continually incorporates feedback from customers is also in a better position to rectify any product shortcomings. A reasonable expectation of the consumer is a level of transparency regarding QCs, exceptional technical support, and timely response to feedback.

Life science research is demanding, and many factors can hinder scientific advancements. Furthermore, publications and new attention to erroneous, or irreproducible findings erode the credibility of research. The responsibility of life science reagent providers in contributing to reliable, trustworthy research is to provide the scientific community with the highest quality products so that this variable is taken out of the equation at the outset.