NEW YORK (GenomeWeb) – In an effort to systematically characterize major technologies for cancer mutation testing, researchers at AstraZeneca have published an evaluation of 13 assays for KRAS mutations.
The study, published last month in the BMJ ESMO Open journal Cancer Horizons, stressed the technologies by using very small amounts of mutation-containing sample and then assessed differences in sensitivity and specificity as well as metrics such as turnaround time, expertise required to run the tests, and multiplexing level.
The platforms evaluated included Qiagen's therascreen KRAS RGQ PCR Kit; Roche Diagnostics' cobas KRAS Mutation Test; Biocartis' Idylla; Agena Bioscience's iPLEX Pro tests; RainDance's ThunderBolts Cancer Panel; Thermo Fisher Scientific's Oncomine Focus Assay and Ion AmpliSeq Cancer Hotspot Panel v2; Vela Diagnostics' Sentosa SQ NSCLC Panel; Illumina's Nextera Rapid Capture Custom Lung Panel and TruSight Tumor 15; Bio-Rad's PrimePCR ddPCR Mutation Assays KRAS; and a Sanger sequencing test using a system from Applied Biosystems.
"All the tests are great — they're designed for specific purposes, so we are just highlighting the differences," James Sherwood, corresponding author and a senior clinical diagnostic scientist at AstraZeneca's precision medicine and genomics group, said in an interview.
The study does not, however, determine a "winner" amongst the 13 technologies, Sherwood emphasized. "We didn't really intend it as a competition. … They all have different strengths which would be useful in different circumstances depending on the needs of each patient," he said. "I would like to see [the study] being used to help labs who are doing clinical testing understand the potential pitfalls of using clinical samples which have not been specifically validated for a given platform."
One major limitation he noted was that the researchers used high molecular weight DNA from cell lines, not patient samples. However, that also ensured that each platform received the same number of DNA copies.
Clinicians, technology companies, and other pharmaceutical industry researchers have already been providing feedback on the study, Sherwood noted.
To the best of his knowledge, this was the largest head-to-head comparison of KRAS detection technologies in the same set of samples that has ever been done. The study was also particularly notable because it compared assays across different core technology types, from PCR-based to NGS and proteomics-based assays, which is not common.
"There's such a huge array of platforms and assays that are available for the detection of somatic mutations in the market, but in practice certain sample types present an awful lot of difficulty," Sherwood said. "We wanted to highlight what happens if you analyze the same samples on different assays and platforms, to emphasize the need for proper validation and optimization of these kits and tests."
AstraZeneca is agnostic in its testing preferences, Sherwood said, but the firm was keen to share its results because it has a unique perspective. "We get to see an awful lot of testing from different places," he said, including patient selection testing from around the world and results of the firm's own clinical trials. "Our motivation was to help labs with identification of a suitable testing platform for the samples that they're likely to be receiving from their patients," he said.
Besides comparing across technology domains, the study was also novel in the way it challenged the assays. In typical comparisons, researchers often also use "oodles of DNA," said Sherwood, but that's not normally what one would see in the clinic. "We wanted to see what happens when you give them just a little bit of DNA," he said.
Some of the platforms AstraZeneca had internally, but the researchers also wanted to work with external collaborators, including manufacturers, hospitals, and labs that do testing.
Results were comparable in many respects. However, a few key differences stood out.
Turnaround time varied considerably, for example. The longer turnarounds tended to occur on the NGS platforms, whereas single-gene tests, particularly those designed to work on point-of-care devices, had shorter turnaround times. This parameter can be important for a patient with late-stage lung cancer, for example, Sherwood said. In such a case, "A test with a shorter turnaround time may be a higher priority than a test which takes four weeks to get a result back," he noted.
On the other hand, a one-gene test isn't so useful to characterize several genes and it might use up a good proportion of the available sample and prevent further testing for that patient, Sherwood said. "But if you wanted a fast, accurate result for one actionable gene that was already approved for decision-making, then it would be good to consider that technology," he explained.
Maximum samples per run had a clear standout in the Agena Bioscience platform. Its kit involves matrices of 384 spots and uses MALDI-TOF technology. DNA is amplified and subsequently ionized on the platform, with time of flight used to calculate the weight and identify the mutation. "It can do that on the fly, really, because you're just basically weighing it, and then the software is able to analyze that straight away – it gives you quite a fast turnaround time and high throughput," Sherwood said.
AstraZeneca originally approached Agena about participating in the study based on a previously successful collaborative study using the MALDI-TOF-based UltraSEEK chemistry to detect BRAF-V600E on metastatic melanoma samples, Peter Dansky, CEO of Agena, said in an email.
The study did not compare cost of the platforms and assays, but Agena's Dansky said his firm's technology has some advantage there, because it "can offer UltraSEEK at a fraction of the price of other technologies," and maintain single-day time to result. Agena plans to share the results of the study with its lab and hospital customers.
Meantime, Bart Claes, a scientist at Biocartis, said his firm tested AstraZeneca's KRAS mutation-containing samples in-house using its qPCR-based Idylla KRAS mutation test and then reported the obtained KRAS genotypes back. "An important aspect was that we, and all other technologies, were blinded to the true genotype status of the samples, which is certainly a strength of this study," Claes said in an email.
The Biocartis device had a quick turnaround on its point-of-care device because it uses FFPE specimens directly, Sherwood said. "For those patients that need really fast results, and for those sites that maybe don't have a huge molecular lab, that could be very interesting," he said.
After unblinding the results, Biocartis collaborated on the manuscript as did other technology providers, but AstraZeneca had final responsibility for the content of the manuscript in order to safeguard the independence of the study. Claes also noted that the assay used was a first-generation version, and the firm has since introduced an enhanced version with improvements in the detection of minor variants. "We believe that were we to run the samples again, we would identify the two lowest frequency KRAS Q61H mutations that were missed in the study," he said.
IMGM Laboratories performed the Bio-Rad testing. "They used the Prime PCR off-the-shelf assays," Sherwood said, as per the manufacturer's instructions. The assays "had good sensitivity, but there was a slight problem with specificity that we observed in our study," Sherwood said.
This result was part of the phenomena the AstraZeneca researchers wanted to highlight with their study, he added. "If you have a test you've already been running in a research setting, don't just assume that it's going to be adequate to make a clinical decision," he said. If it isn't validated and optimized for the number of copies of DNA in clinical samples by end users, they may get spurious results.
Bio-Rad representatives said in an email that the firm has submitted a letter to the journal to critique the study and request that it be considered for retraction if not amended. They perceived a number of faults.
The firm found the results to be at odds with more than 2,000 ddPCR publications from the last six years, for example. "Apart from stark disagreement with Bio-Rad's own validation data, it is most surprising that the authors do not cite nor compare their own anomalous results with abundant contrary publication examples that demonstrate very different findings with the QX200 system," George Karlin-Neumann, director of scientific affairs at Bio-Rad's Digital Biology Center, said in the letter.
Bio-Rad also questioned why the study didn't show quantitation with the QX system, since that is one of its main attributes, yet showed minor allele frequency titrations for all NGS systems. "This strongly suggests that the experimental runs done by IMGM were seriously flawed, uncharacteristic of the technology, and that they should have been very wary of their published results," the firm suggested.Â
The company also found it noteworthy that the "best performing systems — Idylla, Oncomine, and UltraSEEK — were all run and analyzed by the vendors themselves, who were also co-authors on the manuscript, whereas Bio-Rad was never contacted nor invited to participate in the study."
Bio-Rad requested that the journal editor both publish its letter and also urged the authors to more thoughtfully critique the results and reconcile them with the prevailing literature, or, failing that, retract the paper, the firm said. The outcome of the dispute is still pending.
AstraZeneca has no preferred assay from the study, but Sherwood would like to see the study used to help elucidate issues that arise in patient testing and therefore highlight to laboratories the need for validation to deliver the best kind of testing for patients.