NEW YORK (360Dx) – Comparing a new assay to a gold standard is a way to assess performance, but what if the comparator test turns out to have variability between labs or experiments? Accelerate Diagnostics reported earlier this month it has had to slightly delay the start of clinical trials for a pneumonia assay in development because the firm has discovered the gold standard culture methods, which are commonly used by microbiology labs, are too highly variable.
Accelerate received US Food and Drug Administration clearance last year for a groundbreaking rapid, automated, direct-from-sample platform that combines genotypic pathogen ID and phenotypic antimicrobial susceptibility testing (AST).
Investment analysts have repeatedly affirmed the assessment that Accelerate's technology is "disruptive" in the microbiology space. It's Pheno system was cleared with a test that can detect — and determine antimicrobial susceptibility of — organisms that cause bloodstream infections, or sepsis, but the Tucson, Arizona-based company is also developing an assay for lower respiratory infections that uses bronchoalveolar lavage (BAL) samples as well as the firm's proprietary morphokinetic analysis of live cultures for AST.
Romney Humphries, Accelerate's CSO, explained in an interview with 360Dx that the pneumonia test essentially looks for the typical pathogens that cause hospital-onset pneumonia and it performs in vitro susceptibility testing for antibiotics directly from BAL.
"What differentiates us really from the molecular players that are also looking at pneumonia is that our tests look specifically for living organisms," Humphries explained.
Therefore, the test does not rely on serendipitous targeting of the genetic element that leads to resistance, but rather measures behavioral changes in pathogens in the presence of different antimicrobials. In this way, it can generate what is termed the "susceptible, intermediate, and resistant call" for a bacteria, as well as minimum inhibitory concentrations for the antibiotics that halt bacterial growth.
The pneumonia assay is in the FDA pre-submission process, but on a recent conference call to discuss the firm's third quarter earnings — including 63 percent revenue growth year over year — Accelerate's President and CEO Lawrence Mehren said that the clinical trials for the pneumonia assay have been slightly delayed. Â
"In preparations to begin the trial, we conducted a study of the reproducibility of the reference method, the 'gold standard,' so to speak, for identification and quantification of pathogens [and] the results of the study were quite surprising," Mehren said during the call.
In an interview, Humphries explained that, particularly for ventilator assisted pneumonia, it can be challenging to parse out whether a particular species of bacteria is simply colonizing the lung or if its presence represents an actual infectious process.
Microbiologists typically use a technique called quantitative culture to estimate the density of bacteria in a sample from the lung, and researchers have established cutoff points above which something can be deemed an infection.
Specifically, if a culture-based method determines there are more than 105 colony-forming units in a milliliter of sample, then a person should be treated for the infection, while fewer than 104 CFU/mL would likely not be treated.
Accelerate would like to compare its pneumonia test to a quantitative or semi-quantitative method, but because the test deals with live bacteria, the firm can't use a gold standard test like bi-directional sequencing or PCR, since those require lysis and can pick up dead bacteria as well.
Humphries said the firm tried a technique called quadrant testing, which is a commonly-used semi-quantitative culture method.
Quadrant testing involves streaking a sample into four quadrants on a culture plate, then estimating the CFUs per mL based on how far the subsequent bacterial growth extends beyond the first quadrant.
Labs can also learn about the amount of bacteria present in a sample by using fully quantitative methods, such as a calibrated loop to extract an exact amount of BAL to add to the culture plate, or techniques involving dilution and serial plating, Humphries said.
Interestingly, even though there are plenty of studies suggesting that quantitative culture methods can be used to tell simple bacterial colonization from infection in ventilator-assisted pneumonia, not many labs are actually using these methods to guide treatment, Humphries said.
She suggested that one reason for this is that others may have found what the Accelerate lab found – that the results of the comparator culture methods, and quadrant testing in particular, can be highly variable.
With the same technologist repeatedly plating the same sample, she said, the firm saw variability ranging from one to two logs difference in quantitation.
"Likely, that has to do with the viscosity of the specimen and how bacteria are distributed in it," Humphries said, adding, "Even though we did our best to vortex well and do a good job of homogenizing the sample, we still saw this variability."
Furthermore, another red flag with the comparator culture methods popped up when the firm was conducting other studies, comparing results that its collaborators obtained on the Pheno system with Accelerate's own testing of the same patient samples performed the following day in its reference labs.
In this case, it was looking at the standard of care, which is not a quantitative test, but rather simply a culture determining whether bacteria are present or absent.
Although one might predict shipping a patient's BAL sample across the country would have resulted in a decreased viability of bacteria, especially more fastidious organisms, and consequent decreased growth and detection in culture, Humphries and her colleagues found the opposite to be true.
Testing in the firm's lab — using a culture process it has standardized that involves a variety of selective media to make sure it can pick up important pathogens — Accelerate scientists were able to detect pathogens the other labs had missed. Specifically, this amounted to detection in approximately 30 percent of samples that had previously been missed, Mehren said in the earnings call.
"These were not inconsequential — they were things like Pseudomonas or Staph aureus," Humphries noted in the interview.
Accelerate has been discussing ways to cope with the variability in gold standard testing with the FDA as part of the presubmission process.
"Given that this is the method to which we score our performance during the trial," Mehren said during the call, "we must again align with the FDA on a study design which will not unduly penalize us for superiority to current methods."
The culture test the firm will now use, according to Humphries, is a modification of the full quantitative method that is being used by the clinical labs and is published in the clinical microbiology procedures handbook from the American Society for Microbiology.
"We've taken that method and we're modifying it a little bit to look at some more selective media … so that we can have the best chance to recover the organisms," she said.
Accelerate now plans to begin clinical trials of the lower respiratory tract AST test in the first quarter of 2019.
Furthermore, the firm has also developed an upfront processing module to ensure that samples are well homogenized, Humphries said, which also makes the test slightly different than the Pheno sepsis assay.
The processor also includes a proprietary method to maintain the viability of the organisms during the homogenization process and it can remove inhibitory substances, like antimicrobials, that may be present in the BAL.
In a note published by Piper Jaffray yesterday, analyst William Quirk noted that Accelerate has 40 sample prep instruments available for the upcoming clinical trials.
Going forward, Mehren suggested on the call that the forthcoming pneumonia test would "demonstrate the versatility and platform potential of the Pheno, and its ability to replace significant portions of the current micro workflow."
And, the challenges with the gold standard also present "a tremendous opportunity" to structure the clinical trial in a way that generates a data set which "indicts the current standard of care," Mehren said.