NEW YORK – Diagnostics makers have begun bringing serology tests for SARS-CoV-2 to market, raising the prospect that these assays could help scientists and policy makers better assess the scope of the pandemic and perhaps identify individuals with antibodies against the virus who could safely return to work.
The likely low prevalence of COVID-19 in the general population presents a challenge to such testing, however, as tests without extremely high accuracy could lead to large numbers of incorrect results, resulting in dangerous consequences.
Given the use cases commonly discussed for serology testing, assay specificity is a particular concern. Many European countries and, to a lesser extent, the US, are contemplating using serology tests to identify people who have had the disease and recovered, and who, therefore, can reenter society. As such, serology testing could prove a key tool for planning as countries seek to ease the shutdown measures taken to stop the virus' spread.
The challenge is that when testing for a disease with low prevalence in the tested population, even a test with relatively high specificity will generate large numbers of false positives. In the case of SARS-CoV-2 serology testing, this could mean that many people being told they have protective antibodies do not in fact have them and are still susceptible to infection.
The basic process underlying the development of a serology test is straightforward — developers identify proteins or protein fragments specific to the target virus and generate them via a recombinant protein expression system. In practice, though, it can be somewhat more complicated as the ideal viral proteins from a specificity standpoint might not be the best proteins in terms of generating an immune response, or might not be the easiest to express.
In the case of SARS-CoV-2, the question of specificity is a challenging one given that there are several other common coronaviruses to which many people probably already have antibodies. These existing antibodies can cross-react with the antigens used in the SARS-CoV-2 test, leading to false positive results.
"When you are looking at a large sample size with a lower prevalence, those false positives start to become much more of an issue," said Kaitlyn Sadtler, chief of the section for immunoengineering at the National Institute of Biomedical Imaging and Bioengineering and one of the leaders of a study launched by the National Institutes of Health that aims to test 10,000 health volunteers for anti-SARS-CoV-2 antibodies.
Robert Garry, professor of microbiology at the Tulane University School of Medicine, likewise noted the issue of potential false positives.
"Prevalence is the key," he said. "In low prevalence populations there will be more false positives than true positives."
Currently, there are no good estimates of COVID-19 prevalence in the broader population, but existing surveys suggest it is relatively low. Screening of 215 pregnant women delivering babies at New York–Presbyterian Allen Hospital and Columbia University Irving Medical Center found that 15 percent were positive for COVID-19 based on PCR testing for the virus. A recent survey of residents of the German town Gangelt using serology testing similarly found that around 15 percent of the population had been infected by the virus. On the other hand, serology testing in Colorado's San Miguel County put infection levels at around 1 percent to 3 percent of the population.
Even assuming a 15 percent prevalence, some serology tests could generate large numbers of false positives. Take, for instance, the COVID-19 serology test recently launched by Morrisville, North Carolina-based test developer BioMedomics and Becton Dickinson. The test measures anti-SARS-CoV-2 IgG and IgM antibodies in patient blood to determine if they have been exposed to the virus and developed antibodies against it. In a study of the test's performance done in a cohort of 397 confirmed COVID-19 cases and 128 uninfected patients, the test showed a sensitivity of 89 percent and specificity of 91 percent.
At those performance levels, and assuming a COVID-19 prevalence in the tested population of 15 percent, the test would have a positive predictive value (PPV) of around 60 percent, meaning that around 60 percent of patients testing positive would actually be true positives. Drop the disease prevalence to 10 percent and the PPV falls to around 50 percent. At 5 percent prevalence, PPV could fall into the 30 percent range.
Test developers who intend for their serology assays to be used alone to determine infection status are still required to take their tests through the US Food and Drug Administration's Emergency Use Authorization process.
Even at higher performance levels this remains a potential issue. This month, Cellex became the first company to receive EUA from the FDA for a coronavirus serology test. In validation studies, the test demonstrated 94 percent sensitivity and 96 percent specificity. Assuming a 15 percent disease prevalence, PPV would be around 80 percent, which would still result in a substantial number of false positive results.
Chembio has also received EUA for a serology test that in validation studies performed with sensitivity ranging between 94 percent and 100 percent and specificity ranging between 90 percent and 100 percent depending on the cohort.
BioMedomics, Cellex, and Chembio note that their tests are not intended to be used as the sole basis of a diagnosis for COVID-19.
During a Wednesday conference call for test developers, Timothy Stenzel, director of the US Food and Drug Administration's Office of In Vitro Diagnostics and Radiological Health, acknowledged the challenges facing serology tests, noting that "as potential uses of the results of this testing may inform important decisions, specificity, in particular, is important."
"False positives might lead somebody to believe that they are immune when they are not, and therefore there are significant risks in interpreting the results of these serology tests," he said.
Stenzel added that "in a setting of very low prevalence, even a very specific test can have a number of false positives, and the positive predictive value is relatively low," and noted that the prevalence of SARS-CoV-2 in the US is currently unknown.
The FDA, he said, is "looking for ways to highlight these concerns," though he did not specify any particular actions the agency was taking.
A number of new serology tests have come to market since the FDA announced on March 16, 2020 that it would be substantially loosening its regulations for SARS-CoV-2 serology tests, alerting test makers that it would not "object to the development and distribution by commercial manufacturers or development and use by laboratories of serology tests to identify antibodies to SARS-CoV-2." Test makers must validate their assay, notify the FDA of the assay, and include with the assay notice that the test has not been reviewed by the FDA, that false negatives and false positives are possible, and that the results should not be used alone to determine infection status.
More than 70 serology tests are available in the US under this regulatory process, called Policy D.
While specificity is an issue for serology testing, there are indications that test makers will manage to achieve suitable performance levels. Ortho Clinical has received EUA for a serology test that the company said demonstrated 100 percent specificity and 87 percent sensitivity. Mount Sinai Laboratories has also received EUA for its test, which demonstrated 100 percent specificity in a small sample cohort used to assess its performance. Abbott Laboratories on Wednesday launched a serology test that the company said performed with 100 percent sensitivity and 99.5 percent specificity in a validation cohort of 1,000 samples.
Such levels of performance could make for effective assays even given a relatively low prevalence of infection.
The Abbott test has not received EUA yet. Nevertheless, on the company's first quarter conference call on Thursday, its President and CEO Robert Ford said that in just one day, Abbott had received about 1 million orders for the test.
Abbott is also working on a point-of-care version of the test that could improve access and throughput, provided the POC assay is able to maintain the high performance of the lab-based test.
One question, though, is whether the validation data presented by Ortho Clinical, Mount Sinai, and Abbott, as well as other diagnostic firms developing serology tests, will hold up in populations with less severe disease.
Tulane's Garry said that while he believed his lab would be able to produce a serology test with high performance in patients who had been hospitalized, "the question is the milder or asymptomatic subjects."
He said that based on research out of China he suspected antibody titers would be lower in those groups than in those with more serious infectious, which could make distinguishing them from uninfected subjects more difficult.
He said that he thought it would ultimately be doable but noted that this was "going out a bit on a limb."
Sadtler said that the test she and her colleagues have developed for the NIH serology study also has a very small false positive rate, though she noted that the test is for research use only.
She said her team was following a test development protocol developed by the NIH's Vaccine Research Center, which has developed a COVID-19 serology test that it is using as part of clinical trials for a COVID-19 vaccine.
"They have run this test many, many times," she said. "It has been validated very thoroughly. Our teams at the [National Institute of Allergy and Infectious Diseases] are very confident that the false positives are minimal, if any."