Skip to main content

NCI Validation Pointed to Problems With Chembio Dx's COVID-19 Serology Test Before EUA Revocation

NEW YORK ─ An independent validation that found problems with Chembio Diagnostics' coronavirus serology test set the stage for the US Food and Drug Administration to revoke the Emergency Use Authorization it initially issued in April, according to documents filed with the agency.

The FDA revoked the EUA for Chembio's DPP COVID-19 IgM/IgG test earlier this month after the National Cancer Institute performed a validation of the test as a condition for the EUA, according to the FDA documents. And while Chembio Diagnostics claimed a modification could improve the performance of the test, the FDA refuted the company's claims, and along with revoking Chembio's EUA, the agency told the firm to stop distributing its coronavirus antibody test in the US: The test, the FDA said, generated a higher-than-expected rate of false results and posed a risk to public health. 

The revocation of the EUA is the first, and only one so far, by the FDA for a coronavirus test and was a major setback for the Hauppauge, New York-based firm, which in the days after receiving the EUA, rode high on rosy investor sentiments.

Since the beginning of the pandemic, the agency's opinion about coronavirus antibody tests and how it should regulate them have evolved amid concerns about the quality of some of the tests. In early May, the FDA issued new guidance that refined its stance on coronavirus serology testing and told test developers they could no longer simply notify the agency, but instead had to file for Emergency Use Authorization and meet certain performance thresholds.

Additionally, the FDA has worked with the NCI, the Centers for Disease Control and Prevention, and the US Department of Health and Human Services' Biomedical Advanced Research and Development Authority to help establish a testing capability at the NCI to independently validate certain antibody tests, including antibody tests that are not the subject of an EUA and those that are under FDA review.

The FDA said that it is using the NCI data to inform future decision-making, such as whether to authorize a test; to guide it as it works with test developers who want to keep their tests on the market; or to take other action regarding tests that do not perform adequately, including stopping their marketing in the US.

Some companies are voluntarily submitting their tests to NCI for evaluation after they have done their own validation. In addition, most COVID-19 serology tests that have received EUA include a stipulation that the developer participate in the NCI study and to "submit lots for testing by NCI, some upon FDA’s request," an FDA spokesperson said.

The EUAs also include a condition requiring the developer to collect and report performance information to the FDA, including suspected false positive or negative results and any significant deviations from the established performance characteristics of the test, the FDA said.

A condition of authorization in Chembio’s EUA required that it participate in the NCI study, the FDA added.

In a town hall meeting in April, Timothy Stenzel, director of the FDA's Office of In Vitro Diagnostics and Radiological Health, said "We will in all likelihood offer those manufacturers who pass a certain bar of performance in this interagency testing a somewhat streamlined approach to EUA authorization."  

In another town hall meeting last week, Stenzel bluntly stated, "We have raised the bar on expectations for serology performance, and Chembio, unfortunately, was no longer able to reach that bar."

According to Chembio, its immunochromatographic technology, called the dual-path-platform (DPP), which underlies its coronavirus antibody test, offers advantages over single-path lateral flow technology, including the ability to detect multiple analytes on a single cassette, run a variety of sample types, and deliver improved sensitivity and specificity.

When the FDA granted Chembio the EUA for the coronavirus antibody test, the agency did so on the presumption that the test could be effective for the detection of IgM and IgG antibodies against SARS-CoV-2 in serum and plasma, venous whole blood, or fingerstick whole blood specimens collected from people suspected of COVID-19 by their healthcare provider. It based the authorization on clinical performance data that reflected estimates of 77.4 positive-percent agreement for IgM; 87.1 positive-percent agreement for IgG; 93.5 positive-percent agreement for combined IgM and IgG; and 94.4 negative-percent agreement for combined IgM and IgG.

New information from three evaluations done subsequent to the EUA, however, determined that the test's "performance may be both inconsistent and lower than that described in your original submission," the FDA said in its letter to Chembio.

In response to the FDA's concerns, Chembio proposed a new cut-off value of 35 for its Micro Reader II platform, up from 25 used in validation testing. Chembio told the FDA that the new cut-off suggested that the specificity of the device could be improved from 81.2 percent to 93.5 percent.

But the firm's proposal for the new cut-off value, after analyzing NCI performance results for its test, would require a change to the scope of the Emergency Use Authorization and represented a "significant modification that affects the sensitivity and specificity of the device," the agency told the firm.

"The change in the cut-off was a significant modification that affected the sensitivity and specificity of the device," an FDA spokesperson told 360Dx. "As a general matter, the FDA is always concerned about modifications that change the performance characteristics of the device."

Chembio, the FDA noted, hadn't requested such changes to the scope of the authorization, and the agency hadn't agreed to the changes.

"More fundamentally, your proposed modification of the device has not resolved the poor clinical performance observed, as demonstrated in the reanalysis of the NCI evaluation results that you provided on May 24," the FDA said in its letter. In that reanalysis, specificity improved from 81.2 percent to 93.5 percent, but the sensitivity for IgG decreased from 78.6 percent to 75 percent and the sensitivity for IgM, at 57.1 percent, and combined IgM/IgG, at 82.1 percent, were unchanged.

The FDA further noted in its letter to Chembio that, generally, SARS-CoV-2 antibody tests with 30 positive samples and 75 negative samples should demonstrate a minimum combined positive-percent agreement of 90 percent and a minimum negative-percent agreement of 95 percent. For tests that report IgM or IgG, a minimum positive-percent agreement for IgG of 90 percent and a minimum positive-percent agreement for IgM of 70 percent are expected.

Chembio's test fell short for each of the minimum thresholds.

The details of Chembio's request to modify its device are described in an FDA letter to the point-of-care diagnostic company dated June 16 and posted to the agency's website.

Ripple effects

Chembio did not respond to a request for comment or to questions emailed to the company. 

In a filing filed with the US Securities and Exchange Commission last week, the firm said that it intends "to continue working with the FDA with respect to the modification of the DPP COVID-19 System and of the revocation of the EUA for our test system."

Investors, however, were spooked by FDA's action and responded with a massive sell-off of Chembio's stock. Since the FDA's revocation of the EUA, Chembio's share price has dropped by almost 68 percent.

The FDA takesy-backsy is likely to create a domino effect. Chembio's DPP COVID-19 system was granted EUA in April by Brazilian regulators, and it was granted a CE mark in early May. In a research noted last week, Canaccord Genuity analyst Max Masucci said it is unclear whether the CE mark or Brazil authorization will be pulled, but he anticipated a "ripple effect" on its business. The bank downgraded its rating from Buy to Hold, and then terminated its coverage of Chembio altogether.

The effects are also unclear for the firm's partnership with private diagnostics company LumiraDx. The firms had inked a deal to develop point-of-care diagnostic tests for the detection of SARS-CoV-2 and IgM and IgG antibodies to the coronavirus that would run on both firm's diagnostic platforms.

The Chembio experience could be a warning shot to other serology test developers, as well. Last week, the FDA issued a letter to US clinical laboratory staff and healthcare providers saying certain COVID-19 serology/antibody tests should not be used. The agency has removed dozens of names from a list of developers that had provided notification that they had validated and intended to distribute a serology test.

To date, the FDA has granted EUAs for 23 serology tests. However, its recent warning about serology tests has raised concerns about whether other such tests that have received EUA could be revoked when NCI evaluation data are made available.

The FDA has in the past granted authorizations to some developers of tests that are amenable to testing at NCI, Stenzel said during the town hall meeting last week. It decides as soon as possible on whether to authorize serology tests for which it has received a notification from a developer that its validation has been completed.

"When we get NCI data, whether it supports authorization or not, we've asked our reviewers to make that a priority," he said. "There is obviously a lag between getting some NCI data, making a regulatory decision, making that regulatory decision public, and then posting that information."

Additionally, the FDA is looking to have NCI testing performed on "pretty much all" serology tests that it authorizes for emergency use going forward, Stenzel added.