NEW YORK – A consortium led by US Food and Drug Administration investigators has evaluated the analytical performance of five research-use-only liquid biopsy next-generation sequencing (NGS) assays for different clinical applications and to help inform practice guidelines for the precision oncology space.
The team also saw that the reliable sampling of rare circulating tumor DNA fragments served as the major challenge for developing novel liquid biopsy assays.
Joshua Xu, a computer scientist in the FDA's division of bioinformatics and biostatistics, explained that his team launched a project in 2016 called Sequencing Quality Control Phase 2 (SEQC2) to develop standard analysis protocols and quality control metrics for the use of NGS data to enhance regulatory science research and precision medicine.
The FDA has currently approved two ctDNA-based liquid biopsy tests that have high sensitivities for allele frequencies (AF) of about 0.3 percent, Foundation Medicine's FoundationOne Liquid CDx and Guardant Health's next-generation sequencing (NGS) Guardant360 CDx pan-cancer assay.
Xu said that manufacturers of the RUO sequencing kits have published studies that claim to reach lower allele frequencies — even as far down as 0.01 percent — than currently approved assays. However, he noted that reports exist that also claim there is a lack of concordance between what manufacturers claim and the performance that clinical labs see.
In the analytical validation study, published on Monday in Nature Biotechnology, Xu's team examined the ability of four capture-based kits (Burning Rock Biotech Dx's Lung Plasma V4, Integrated DNA Technologies' xGen Non-Small Cell Lung Cancer, Illumina's TruSight Tumor 170 panel, and Roche Diagnostics' Avenio ctDNA Expanded Kit) and one amplicon-based RUO kit (Thermo Fisher Scientific's Oncomine Lung Cell-free DNA (cfDNA) assay) to detect mutations in ctDNA at different variant allele frequencies (VAF).
In addition, Xu and his colleagues measured the tests' reproducibility between runs and different labs, their false positivity rate, and the impact of different DNA inputs. They also measured the impact of using DNA extracted from synthetic plasma samples.
"We need comprehensive quality control with reference samples [that] is crucial for translating [cancer panels] from lab development to clinical applications … so that the users, physicians, and patients have a better understanding of the test's reliability," Xu said.
The researchers developed contrived genomic samples using control genomic material supplied by Agilent Technologies. Using one sample containing a mixture of genomic DNA from 10 cancer cell lines with a male DNA control sample, the team created six samples with different DNA concentrations and VAFs that allowed them to maximize homogeneity. To mimic cfDNA, the group also performed enzymatic fragmentation and gel-based size selection.
After Xu's team sent the six samples to independent partners proficient at running the different assays being compared, each test lab sequenced the samples containing different DNA input amounts and concentrations. The labs then submitted the sequencing results to the respective assay providers for analysis with their own variant calling pipelines.
Following variant call analysis, the providers returned the results to the participating labs. The team then examined intra-lab and cross-lab reproducibility and used its list of known mutations to examine the assays' sensitivity and false positivity rate.
Xu and his colleagues saw that all four of the capture-based assays broadly produced high sensitivity for variants with VAF above 0.5 percent using 25 ng of DNA. Below 0.5 percent VAF, however, the group saw degradation in DNA material and a drop in sensitivity. The most sensitive assays, from IDT and Burning Rock, produced a sensitivity of 90 percent for variants with 0.3 to 0.5 percent VAF.
The researchers also found that DNA input amount impacted the tests' sensitivity. Median fragment depth was linearly correlated with the DNA amount, and some panels had better effective depth at different DNA inputs. While differences in assay sensitivity partially reflected the differences observed among assays in coverage depth, Xu noted that coverage depth did not always serve as a good predictor of sensitivity.
"Essentially, each unique molecular identifier (UMI) read corresponds to a fragment of the DNA, and they overlap with each other," Xu said. "The fragment depth is a very important surrogate … [since] when you want to improve the assay performance, you [need to] improve the fragment depth."
Overall, Xu and his colleagues saw that higher DNA input amount led to better technical performance, as an increased input appeared to offer a greater median fragment depth. Higher input also improved sensitivity and reproducibility for variants with VAF below 0.5 percent. While the Illumina and Burning Rock capture-based panels maintained their false positivity rate, the false positivity rate of the panels from Roche and IDT rose with VAF below 0.5 percent.
In addition, the group saw that Thermo's assay produced similar sensitivity and specificity when compared to the hybrid capture panels. The amplicon-capture assay showed equivalent sensitivity to the Roche and Burning Rock assays at 25 and 50 ng of ctDNA. While the Thermo assay's sensitivity fell compared to that of Roche and Burning Rock when using 10 ng, it still outperformed Illumina's assay at the sample amount. In addition, the test did not show any differences in intra-lab and inter-lab reproducibility.
"With 50 ng of DNA input material, good performance can be achieved with a VAF above 0.3 percent, which is very similar to FDA-approved ctDNA assays," Xu said. "Fragment depth was a critical variable, with high coverage essential for sensitive detection of low-frequency mutations."
Because of the results, Xu's team believes that improved sensitivity for mutations below 0.5 VAF should a priority for developing ctDNA assays in the future.
"The purpose of the project was not to have a horse race, but we really wanted to have a transparent open assessment … where [companies] had to agree to provide all the raw data and the mutation calling results, [and] all this would be released to the public," Xu said. "It was not our purpose to show which was the best, because they all have different characteristics and coverage using different technologies and have the potential [for improvement]."
A Roche spokesperson said in an email that the performance of its assay is "as expected given the number of mutant molecules" present at the different VAFs and DNA inputs. The firm noted that input mass, amount of sequencing, panel size, and sample type all have a significant impact on the assay's observed performance.
Kevin Keegan, senior director of global oncology marketing at Illumina, noted in an email that several investigators are using the firm's platform in different ways to develop liquid biopsy assays with "sufficient sensitivity" to detect circulating ctDNA. He pointed out that Illumina's TruSight Oncology 500 ctDNA panel allows users to perform comprehensive genomic profiling from blood samples and detects mutations at 0.5 percent VAF, with 95 percent sensitivity and greater than 99 percent specificity.
Xu also highlighted the need for improved and standardized methods for DNA extraction in the cancer space. While the researchers used a combination of synthetic DNA controls and cell-line-derived reference samples in the study, he said that the extraction yield from synthetic plasma widely fluctuated across labs between the different panels when using different methods for DNA extraction.
"We reported yields ranging from 29 to 77 percent, except one outlier at 96 percent, [while] most were between 30 and 60 percent, which was after we asked the lab to do Qubit high-sensitivity quantification," Xu said. "This led to variance in effective depth and thus performance."
By standardizing cfDNA extraction methods, Xu believes researchers will be able to increase the number of copies for library prep and produce more precise DNA quantification.
"Missed mutations (false negatives) were more common than erroneous candidates (false negatives) due to random sampling," Xu said. "To really improve this technology, the tests will need to improve DNA extraction to improve capture rate and sensitivity."
Because the enzymatic method did not perfectly mimic natural cell-free DNA fragmentation, Xu pointed out that fragment capture and library conversion steps for the sample may be lower or higher for the assays than for natural cfDNA. In addition, he warned that the measures for each assay's precision are inflated and should not be interpreted as realistic measurements of performance on clinical samples.
The Roche spokesperson said that fragmented cell line DNA or synthetic cfDNA can perform significantly worse than real cfDNA — leading to reduced sensitivity — based on how the sample was prepared. The firm instead recommends using blended plasma samples from healthy donors as the best approximation "of a true ctDNA sample," where single nucleotide polymorphisms from one healthy donor are used to simulate SNVs in the recipient's ctDNA.
"We are hoping that [the scientific] community would have a better understanding of performance and the state of liquid biopsy [testing]," Xu said. "This study also demonstrated that reference samples that we developed, through the consortium, is a very valuable resource and can be used to benchmark other liquid biopsy assays, as well, either during development or reproducibility."
Xu's team plan to benchmark more assays and DNA extraction methods later this year.