The story has been updated to include comments from Quidel.
NEW YORK – A rapid SARS-CoV-2 antigen assay from Becton Dickinson has been shown to produce similar results as a rapid antigen test from Quidel despite sensitivity differences reported in initial FDA filings.
There are currently a total of four commercially available tests for SARS-CoV-2 infection that detect proteins or antigens on the surface of the virus in patient samples. The Quidel antigen test was the first to obtain Emergency Use Authorization from the US Food and Drug Administration. It received EUA in May. The BD assay was granted EUA in July.
Both tests have been ordered in large quantities by the US Department of Health and Human Services to support testing in nursing homes as well as by some US states.
But in the original filing documents with the FDA, the Quidel rapid test claims a sensitivity of 97 percent within the first five days after symptom onset while the BD test claims 84 sensitivity, also through day five.
Troy Kirkpatrick, a spokesperson at BD, said in an email that the firm undertook a study after reviewing data from other antigen tests.
"The difference in published sensitivity claims didn't make sense to us, given that the limit of detection for the tests were nearly identical," he said. In addition, the patient populations used in the EUA studies were different, "so we wanted to test the performance of both platforms in the same patient population," Kirkpatrick added.
As described in a MedRxiv preprint, the evaluation compared the BD Veritor System for Rapid Detection of SARS-CoV-2 Assay to the Quidel Sofia 2 SARS Antigen Fluorescent Immunoassay, with the comparator test being the Lyra SARS-CoV-2 PCR Assay, also an EUA test from Quidel.
Researchers from BD collaborated on the study with research teams at TriCore Reference Laboratories, Louisiana State University Health Science Center, and contract research organizations STAT Research and Comprehensive Clinical Research.
In the evaluation, the group performed two studies of the BD antigen test.
In the first, they compared the performance of the BD test to the Lyra reference PCR test using 251 nasal specimens paired with either nasopharyngeal or oropharyngeal specimens from patients who were symptomatic for less than seven days. Discordant results were resolved with PCR testing using the BD Max system.
The second study compared the BD antigen test to the Quidel Sofia test using 361 nasal specimens collected from patients five days or less after symptom onset with discrepant results resolved through PCR testing using the Lyra assay.
Compared to the RT-qPCR test, the BD antigen test had positive percent agreement ranging from approximately 82 percent to 88 percent. Compared the Quidel antigen test, the BD test had a positive percent agreement of 97 percent, a negative percent agreement of 98 percent, and an overall percent agreement of 98 percent.
The first study also found nine false-negative BD antigen test results that were positive using the Lyra PCR test. The authors used the BD Max PCR assay to resolve the discrepancy and found a positive result for only two of the nine BD antigen test negative-discordant results. Six of the remaining seven discordant samples were associated with a negative BD Max assay result and one was associated with an unresolved result, namely no detection of the internal control in the Max assay.
In the second study comparing the BD and Quidel antigen tests head to head, one PCR-positive sample was missed by the BD antigen assay and five PCR-positive samples were missed by the Quidel antigen assay. There was also a sample that had a positive BD antigen test result but was negative by PCR.
Overall, Kirkpatrick said BD concluded that the tests were essentially equivalent, "which means that the study doesn’t support the difference in published sensitivity claims."
The firm thinks this could be driven by the difference in the EUA studies.
"The patient population chosen for BD Veritor EUA study was intended to reflect real world performance of the BD Veritor for Rapid Detection of SARS-CoV-2 assay in true point -of-care settings such as drive-through testing, tents, and outpatient clinics, which largely include individuals with milder severity illness," Kirkpatrick said. On the other hand, "Quidel's EUA study used patients from the ER and those who were admitted to the hospital, which means they more likely had patients with higher severity of illness," he said.
Stephanie Taylor, an infectious disease specialist at Louisiana State University and study co-author commented in a press release that the study shows, "EUA labeled sensitivity and specificity should not be the sole basis for understanding antigen assay performance."
Other factors, such as clinical trial enrollment criteria, study population characteristics, clinical setting, reference method sensitivity, overall prevalence in the trial population, and the limit of detection for each antigen test are all important considerations in order to arrive at a fuller understanding of overall performance, Taylor said.
Tamara Ranalli, senior vice president of the molecular diagnostics business unit at Quidel, said that although the MedRxiv study mentions that other antigen tests have primarily used hospital and emergency department specimens, that is not true for Quidel's assay.
"All of our specimens for our EUA were obtained from point-of-care settings at five geographically diverse locations," Ranalli said, including urgent care and drive-through testing sites. None of the data was from emergency departments or hospitalized patients, she asserted.
Ranalli also commented that the study does not mention how the samples were randomized, which could be important. "The first swab always has a better chance of obtaining more virus," she said. And, the BD assay protocol involves swabbing both nostrils, while the Quidel protocol uses a single nostril, which might account for additional amounts of virus obtained, Ranalli said.
In addition, the study involved freezing the samples to transport them, she noted. "Our EUA study used freshly collected specimens that were not frozen," she said, adding, "We have not determined sensitivity on frozen specimens."
In terms of the analysis of discrepancies, Ranalli said that the study only compared the discrepant samples to PCR, not every sample, and it did not provide Ct values on the discrepant samples. "I’d like to see a true head to head study, with a data set that is large enough to provide good statistical control and [where] the actual protocols are similar, [such as] both tests use a swab that has swabbed both nostrils, and on all fresh samples processed where they are actually taking the specimen, rather than freezing and flying them to be tested elsewhere," she said.
Quidel updated the sensitivity in its product insert a few months ago, Ranalli said, to reflect additional fresh nasal specimen data that the firm had continued to collect after the initial EUA. When the test was launched, the firm provided some data on direct swab samples but had included samples collected in transport media as well, in part because it hadn't yet reached the threshold of a total of 30 positive samples guided by FDA. However, transport media dilutes the amount of virus in a sample, and likely lowers the sensitivity of a test. As the firm collected more data from fresh samples, the PPA increased from 80 percent to approximately 97 percent. "We submitted that information to the FDA and we were able to amend our package insert," Ranalli said. This translated to updating the sensitivty as well as removing the transport media indication, which the firm said was more in line with the intended use at the point of care.
Nathan Ledeboer, medical director of the microbiology laboratory at the Medical College of Wisconsin and Froedtert Hospital who was not involved in the BD evaluation, noted that labs do typically consider the data in a test's instructions for use together with peer-reviewed data in choosing to adopt a test, but that this is amplified by the emergency circumstances.
In the case of the current EUA for SAR-CoV-2, "the normal rigor and number of clinical specimens that are required for 510k or [pre-market approval] has been reduced in favor of expediency," Ledeboer said. This in turn makes interpretation of IFU data more complicated "as the typical transport stability studies, inclusivity studies, [and] inhibition studies, may be absent or abbreviated," he said.
Furthermore, "In FDA clinical trials, the enrollment criteria for patient specimens is usually quite strict, which may not resemble current clinical practice," Ledeboer said.
Taken together, the EUA clinical trial data may not be a suitable replacement for peer-reviewed, independent evaluation studies of real-world experiences, he said, so in this case labs will use both the independent studies and the IFU data to inform which products they may choose to implement.
Ledeboer pointed out that approximately 1 in 6 patients in the first study had an oropharyngeal swab collected for the reference PCR. These specimens are "notably less sensitive" than nasopharyngeal specimens, he said, adding, "That may have had a minor impact that may have been favorable to the Veritor performance."
Finding similar performance between the two antigen tests is "not overly surprising," he said, since the technologies are similar and both systems use a reader to detect and report results.
In terms of the discordance, Ledeboer said he personally finds this type of analysis to be of limited utility.
"The challenge with discordant analysis is that it is biased in favor of the test device, as you only run it when the test device does not agree with the reference standard," he said, noting that because of the inherent bias that is introduced via discordant resolution testing, many journals in laboratory medicine will no longer accept this data as a part of manuscript submissions. This will also limit the usefulness of this data to practitioners and labs, he said
Ledeboer said his health system has adequate PCR turnaround times and capacity currently, so it is not using rapid SARS-CoV-2 tests. Furthermore, "In our patient population we value the improved sensitivity of PCR over antigen testing in making infection prevention, PPE, and patient care decisions," he said.
More antigen tests are likely on the horizon. A test from OraSure is anticipated by the end of the year, for example. In its most recent virtual town hall meetings, FDA has noted that it is eager to authorize more rapid antigen tests and would even be willing to accept tests with lower sensitivity provided they could improve the PPA through using two tests administered sequentially on different days.
Specifically, Timothy Stenzel, director of the Office of In Vitro Diagnostics and Radiological Health at FDA's Center for Devices and Radiological Health, said on a call last week, "For home-use or non-lab tests for both molecular and antigen testing, we are very open to serial testing protocols in order to increase the PPA or sensitivity of the assays." He later clarified that this would likely translate to asymptomatic and symptomatic people testing themselves at home.
"If the sensitivity for a single test result is not high enough for us to be able to be comfortable making the authorization for that test, it may be that if you test days one and two, or days one and three [such that] you provide the consumer or patient with two self-tests and it is the combined result that matters ... then that may improve performance of a very widely available test to where we're comfortable," Stenzel said.
This likely would not apply to currently authorized rapid antigen test systems. Of the four, only the BinaxNow test from Abbott can be run without a reader and might be suitable to run at home, while the BD and Quidel tests as well as one from LumiraDx require reader instruments. Stenzel commented that all of the antigen tests authorized so far perform well and above the agency's recommended levels for authorization.
This week, Stenzel emphasized again that the FDA is "eager to authorize as many antigen tests and molecular diagnostics point-of-care and home tests as possible." The agency has opened up the possibility for point-of-care test validation to use banked samples and has halved the number of samples that would need to be evaluated for a test for asymptomatic claims in the premarket phase.
Stenzel reiterated, "There could be significant trade-offs in test accuracy that may be appropriate, where the need for availability in test results is not being met." He noted that, for example, a hypothetical paper strip home test that doesn't meet FDA's recommended levels of sensitivity, but "utilizing serial testing, a less sensitive test could be deployed." Specifically, he said a two-pack of tests where each test has, for example, 70 percent sensitivity, could meet the threshold provided one or the other test turning out positive translates to a positive result. "We are open to that kind of testing format," Stenzel said.
Ledeboer said this serial approach is "absolutely viable," however, it is important to note that the strategy needs to be paired with good contact tracing. Nevertheless, "there are currently no rapid tests that are FDA approved for asymptomatic screening with serial testing, so this is a welcome update from the FDA."
For the BD and Quidel test study, Kirkpatrick said the results won't really impact the way BD markets the Veritor SARS-CoV-2 antigen test.
"We have stood behind the performance of our test from the beginning, but this may help healthcare professionals better understand that you can't evaluate test performance on sensitivity claims alone," Kirkpatrick said. "To get closer to the truth, you have to look at all data and understand the study design that may influence true performance of the test," he added.
BD does not plan to revise the test's IFU based on this study alone, but the firm is looking to conduct additional studies in patients with more severe illness to determine if that results in higher sensitivity of the test as compared to the original EUA study, which was done at point-of-care sites on patients with milder severity of illness. "We will determine if an update to our IFU is warranted after those studies are completed," Kirkpatrick said.
The firm's initial ramp up was to produce 10 million rapid antigen tests in total between July and September. As of Oct. 1, BD will have capacity to produce 8 million tests per month, and by March 1, 2021, it will have capacity to produce 12 million tests per month, Kirkpatrick said.
"We have completed our obligation to the HHS nursing home initiative, where BD provided BD Veritor analyzers to 11,000 nursing homes along with more than 3.75 million test kits," he said, noting this represents about 70 percent of all nursing homes in the US.
"We have shipped out all units and tests, and all nursing homes will have those in hand by the end of the week," he said, which is ahead of the scheduled delivery date as of the end of September. "This also means that we are now able to replenish our distributors to increase access to BD Veritor devices and test kits beyond nursing homes," Kirkpatrick said.