NEW YORK – Against a backdrop of dynamic US Food and Drug Administration regulations and a quickly expanding commercial landscape, labs are facing unusual obstacles in validating serology testing for COVID-19. Issues with test accuracy and supply, combined with a paucity of independent validations, mean most labs must basically start from scratch in vetting tests on their own.
Along with lab-based and high-throughput serology tests that can be validated as lab-developed tests, seven commercial serology assays have obtained Emergency Use Authorization from the FDA – including rapid lateral flow tests from Cellex, Chembio Diagnostics Systems, and Autobio Diagnostics.
Additionally, there are currently 141 manufacturers that have notified FDA that they have validated and are offering serology tests in the agency's Policy D — which are necessarily deemed high-complexity tests until they are evaluated in the EUA process. These serology tests are listed on the FDA's website.
Serology tests, and particularly the lateral flow immunoassays, may or may not be available in large quantities as labs around the world scramble to acquire them, and few have been tested in large controlled trials. Sensitivity and specificity concerns — particularly in light of an unclear prevalence of SARS-CoV-2 infection in the US — mean labs need to validate any test they bring before they use it on patients.
Indeed, a lengthy report issued this week from the Center for Health Security at Johns Hopkins University calls out the fact that validated, accurate serology tests are currently in short supply.
"These tests will be in high demand, and manufacturing should be scaled appropriately, but the first steps will be to ensure accuracy, validity, and comparability of available tests," the authors wrote.
A special article in the American Journal of Clinical Pathology noted, however, that COVID-19 serology assay validation is "a particularly daunting task" made more challenging because usefulness of the data hinges on knowing the prevalence of disease. For now, the authors conclude the situation requires a stance of "caveat emptor," or buyer beware.
Currently, there are 21 labs that are registered on the FDA website as offering serology testing, which means they have validated a test in the lab.
Labs at ARUP, the Medical College of Wisconsin, and the Mayo Clinic's Arizona location are also validating a variety of serology tests, and their lab directors have offered insights into the process.
As a large national reference lab, ARUP expects to soon be running a high volume of lab-based serology testing sent in from all over the US, according to Patricia Slev, medical director of ARUP's immunology core laboratory, where she and her team recently extensively validated a high-throughput serology test from Abbott Laboratories. The SARS-CoV-2 IgG assay from Abbott is available on the Architect i2000SR system and can produce more than 4,000 results in 24 hours. The assay is also available on the i1000SR system, according to Abbott's website.
But the Abbott serology test has not yet been granted EUA, Slev noted in an interview.
So, not only does a CLIA-certified lab like ARUP need to validate it extensively, it is also somewhat unclear whether labs are then required to notify the FDA in any way if they are using a test from a manufacturer who has already notified FDA. "It is a real question in the lab community," Slev said.
Having EUA of high-throughput serology tests is urgently needed, she said, because it is likely a challenge for smaller hospitals and labs to do the types of validation studies required.
ARUP is also looking into a strategy of running confirmatory serology tests to accommodate for the low positive predictive value associated with low prevalence, and it is looking into tests from multiple manufacturers, including a semi-quantitative test from PerkinElmer's Euroimmun. As a national reference lab, it would not be feasible to determine the local prevalence for each sample sent in, Slev said.
Generally, the evolving FDA guidelines have also been challenging to stay on top of, particularly the Pathway D track that differs from the typical EUA process of the past. Access to positive serum or plasma samples needed for validation can also be a challenge for a lot of labs, Slev noted.
At the Medical College of Wisconsin, Nathan Ledeboer, concurred that sourcing specimens to conduct a thorough validation, particularly for cross-reactivity studies, has been a challenge.
Even more challenging, however, has been obtaining a sufficient supply of reagents within a time frame that was compatible with the increasing demand for testing, Ledeboer said in an email.
With his team at MCW, Ledeboer is now evaluating serology tests from Abbott, Roche, DiaSorin, Euroimmun, United Biomedical's Covaxx division, and Cellex, among others. The majority are ELISA and automated immunoassays, he said.
"Earlier in the outbreak we evaluated one lateral flow test and found the specificity of the test to exceed 90 percent, but the sensitivity was 45 percent," Ledeboer said.
Thus, the lab is now limiting its evaluations to vendors that either have EUA or are in the process of obtaining EUA. It is also focusing on tests that have existing data to support performance, particularly independent data, and it is only looking at IgG tests.
"We are also expecting a significant level of demand, so our decisions are also being driven to methods that can support the volume of several thousand tests per day," Ledeboer said.
He has found the FDA's approach to be helpful in that it likely weeds out really bad tests. However, he added that the lab "recognized very early that broad and comprehensive validation would be required" for any serology test that it might begin using.
For molecular testing, the Mayo Clinic's Rochester location got EUA for a molecular test, while the Mayo Clinic's Arizona site had run the US Centers for Disease Control and Prevention EUA test, then switched to the Abbott m2000 test. For serology, the Rochester site is validating plate-based assays, and Mayo-Arizona is doing the same, according to Thomas Grys, director of microbiology at Mayo Arizona.
But Grys said the lab also wants to bring on lateral flow tests since having a validated point-of-care test could potentially allow the health system to reach more patients, including healthcare workers and employees. The lab is currently in the validation process, beginning with the first step — trying to figure out which of the tests that are currently available might be any good, and also reliably available in large enough quantities.
That said, "the guidance for how to use these tests, and what the results mean, is definitely evolving," Grys noted. The fact that notification of validation is the only requirement for Pathway D is "a very low bar," he said.
"Everyone appreciates that there are efforts being made to make tests available, but that puts the onus on clinical labs — that are already busy — to figure out what's good and bad," he added.
While, Grys said he thinks the US emergency regulatory strategy in the COVID-19 crisis will likely be debated for years to come, right now, "just trying to get our hands on some of the lateral flows has been difficult."
Companies seem to be trying to supply hard-hit areas first, and other countries as well, some of which have also endorsed certain tests.
To select tests that it could attempt to validate, Grys said his lab started with the FDA EUA list, and consulted preprints and colleagues, with the reputation of the company being a factor, as well.
Grys said he has been in touch with eight manufacturers and expects to actually get tests from half of those. He said that some companies wanted specimens from his lab to validate testing themselves, and making arrangements for that right now is difficult. Others have told him they are having shipping problems, or that their supply chains are disrupted. And some of the tests require a reader, which can lead to pricing differences that instigate another set of tradeoffs.
The lab is also considering the likelihood that it will need to validate a second serology test as a confirmatory test, but both tests would probably have to be validated together. "Operationally, it is another nightmare — do you need to confirm by a different antigen, or will any second test do?" Grys said. "Does it have to be plate based test, or another lateral flow? And you would have to validate the two as a pair — is [the second test] the same sample, or is it a new draw?"
Furthermore, in the current emergency, manufacturers are developing tests, but the designs may not be completely finalized, Grys said. He received some kits from a manufacturer who told him it was still kind of tweaking the test, but that he could have the first batch. The positive samples looked fine, but negative samples – including plain buffer alone – also tested positive, presumably due to some cross-reactivity in the reagents. Grys alerted the company and it is now reformulating.
"That's where some of these companies are at — they're like, 'We'll give you kits but we kind of want to know what you find, because if it's not working in your hands then we're going to change it.'" In this context, it is tough to publicly report results, or know whether if the lab chooses to go with a test, that the scenario will repeat itself, he suggested.
Word of mouth between lab managers is a source of leads on accurate tests, but Grys said he has not yet seen any specific kit being recommended enthusiastically. Admittedly, it is also hard to know if any companies have the capacity to scale testing, so there is a possibility that labs aren't sharing preliminary good results for fear that there will be a spike in demand and they won't be able to get the test after all the validation work.
Point-of-care lateral flow tests are still considered high-complexity tests if they have not been reviewed in an EUA application, and some test makers who notified FDA under Policy D are marketing them saying they are authorized. "There has been a lot of confusion for everybody," Grys said.
Some Policy D serology tests also have product inserts showing they have tested 60 to 100 samples, which Grys said is not that extensive. Coupled with the many FDA-mandated disclaimers, "it is hard to look at that and feel like you're impacting patient care in a positive way," he said.
"For me it's almost like a few years ago when direct-to-consumer tests were 'for entertainment purposes.' That's almost where we're at, and it's like, if we're going to be there, then why are we doing this? What is the advantage of having allowed all of this?" he said.
Recent head-to-head studies
End users often rely on published validations in selecting and vetting assays. But designing, conducting, and publishing studies takes time and resources and may not be the top priority for labs in the current crisis. Grys noted that cross-referencing preprints and other public data with information in package inserts can be helpful to winnowing the list of tests.
A preliminary report earlier this month from a group in Peru used an internet search and product information to find there were 226 tests from 20 countries, 180 of which were antibody tests. Based on the product inserts, the overall reported accuracy of the tests was high — the average sensitivity was approximately 92 percent and average specificity was 97 percent.
There have also been a few preprints of head-to-head serology evaluations published recently.
A preliminary study in Denmark, for example, looked at six commercial point-of-care serology tests and three ELISA tests, and found an ELISA from Beijing Wantai had superior sensitivity and specificity.
The POC test results in the Danish study varied, with the best performance observed for a test produced by AutoBio Diagnostics, followed by tests from Dynamiker Biotechnology and CTK Biotech. In a study of 30 serum samples from known COVID-19 positive patients, the AutoBio test was 93 percent sensitive, while the other two both had sensitivities of 90 percent. All three were found to be 100 percent specific when tested on sera of patients who had other viral infections. Importantly, these results were dependent on the phase of the disease, with highest sensitivities in a window of 14 to 20 days after symptom onset.
A preliminary study in the UK also compared nine lateral flow immunoassays. The study concluded that "the performance of current LFIA devices is inadequate for most individual patient applications." The analysis revealed the unnamed tests had sensitivity between 55 and 70 percent when compared to the reference standard RT-PCRs, and 65 to 85 percent compared to ELISA, with specificity of 95 percent 100 percent and 93 to 100 percent, respectively.
In a model scenario using these data, the UK team estimated that in the context of a 5 percent COVID-19 seroprevalence, a test that is 70 percent sensitive and 98 percent specific would have a false positive rate of 35 percent. That rate drops as seroprevalence rises, down to 3 percent in a scenario of 50 percent seroprevalence.
A group in California called the COVID-19 Testing Project also recently posted preliminary evaluations of 10 lateral flow tests and two ELISA tests online. The study used 290 samples, including 130 samples collected at different time points from 80 SARS-CoV-2 RT-PCR-positive individuals.
Test specificity ranged from 84 percent to 100 percent, with 39 out of 108 samples demonstrating false positive results by at least one of the 10 lateral flow assays. Among the LFAs, tests from Wondfo and Sure Biotech were the most sensitive, although the majority of tests had sensitivities above 95 percent. The group also found that people reading the visually read tests could disagree on whether a band was a positive or not, and suggested reader training is key to reliable results.
Other independent evaluations on the horizon
As more countries eye serology testing as a tool for measuring the spread of COVID-19 and determining whether individuals have developed antibodies to the coronavirus, other organizations have taken the lead to evaluate such tests.
For example, the Foundation for Innovative New Diagnostics issued an expression of interest for immunoassays on March 13, and received tests from more than 70 manufacturers of antibody tests.
According to its website, FIND is running a first round of assessment on 26 serology-based rapid tests from 24 manufacturers, including the Dynamiker and Wondfo tests. Cross-referencing these tests with the FDA website, it appears 17 of these manufacturers are also listed as having notified FDA that they wish to sell in the US, while seven are not currently listed.
FIND's mission is to use diagnostics to improve global health, and Cassandra Kelly-Cirino, the director of emerging threats, noted in a recent interview that the global health nonprofit prioritizes evaluating tests that are intended to be marketed everywhere, not just in the US, since the US could presumably run its own head-to-head comparisons.
Indeed, the FDA announced two weeks ago that it would be forming an interagency task force to validate the accuracy of serology tests. Additional details were provided last week, including that the partnership will include the National Cancer Institute and the National Institute of Allergy and Infectious Diseases.
The FDA validation is welcome, according to Mayo's Grys, because even though some labs are now, unfortunately, awash in positive specimens, not every lab will necessarily have enough serum collected and organized to run a good study.
This week in a virtual town hall, the FDA said the interagency group has begun the first testing, but it still has not determined whether and how it will make the results public. Tim Stenzel, director of the US Food and Drug Administration's Office of In Vitro Diagnostics and Radiological Health, said the agency will "in all likelihood offer those manufacturers who pass a certain bar of performance in this interagency testing a somewhat streamlined approach to EUA," noting that this is "one clear way we can make the testing performance measured by this interagency team available to the public."
Until fully authorized or independently well-validated tests are available in substantial amounts, most labs will need to persist in their own validations.
"Although the need for antibody testing may be great, we cannot sacrifice quality," ARUP's Slev said.