NEW YORK – Despite careful planning and oversight, when a large health system in the Detroit area made the jump from rapid immunoassays to point-of-care molecular diagnostics using the Roche Cobas Liat platform for influenza testing, almost everything that could go wrong did.
Three flu seasons later, the health system is seeing benefits in terms of cost and patient management, but its initial experience is a cautionary tale for other lab managers considering the switch to molecular.
In 2015, Henry Ford Health System (HFHS) made up of six large hospitals, 12 urgent care centers and walk-in clinics, and approximately 40 small medical centers decided to make the switch to a molecular system for flu testing.
Linoj Samuel, the director of clinical microbiology at HFHS, described the process of rolling out molecular flu testing across this system in a presentation at the American Society of Microbiology (ASM) Microbe conference in June.
Along with the enormous size of the system, the Henry Ford Medical Group, part of HFHS, employs more than 1,200 physicians, making it one of the nation's largest group practices. The health system also serves a huge population spread across Southeast Michigan. It had 4.2 million outpatient visits in 2017, and more than 480,000 visits to its nine emergency departments.
In an interview, Samuel said that as a system, HFHS had typically run about 1,000 rapid influenza detection tests, or RIDTs, per week in the peak of flu season prior to implementing molecular testing. They were also doing a low volume of lab-based molecular flu testing.
That all changed in 2015 when HFHS decided to go all-in with POC flu testing.
At the time, the lab had some experience with the different point-of-care molecular platforms available, and it considered workflow, the literature on performance, experience of colleagues, and what was then available as a CLIA-waived assay during the evaluation process.
Around the time that HFHS was going molecular on flu testing, a number of platforms received CLIA waiver, including the Alere i Influenza A&B in January 2015, becoming the first POC molecular diagnostic test and molecular diagnostic platform ever to receive such designation.
A flu A/B test on the Roche Cobas Liat platform was then CLIA waived in September, and Cepheid subsequently got CLIA-waiver on the Xpert Flu+RSV Xpress test for flu A/B and respiratory syncytial virus on the GeneXpert Xpress system three months later. CLIA-waived flu/RSV tests from Alere and Roche soon followed.
With the premium on "simplicity and performance," all the data streams and opinions came together in such a way that the HFHS microbiology lab settled on advocating for the adoption of the Roche Liat, Samuel said.
The Liat uses PCR-based assays, and it is has been shown to have a higher sensitivity and specificity than Alere i — now dubbed the Abbot ID Now — as demonstrated, for example, in a 2016 head-to-head evaluation. The Cepheid platform, which was CLIA waived after HFHS brought on POC molecular, also performs quite well, Samuel noted.Â
The Liat instrument was more expensive than competitors, although it was also not the most expensive, but because of the throughput required at HFHS, there was bound to be significant instrument cost, Samuel said. Among CLIA-waived POC MDx instruments, there has also been competition in terms of menu, but Samuel said a broader menu was not necessarily a big selling point for HFHS given its typical testing needs at the point of care.
What was important was input from the system's emergency room physicians, who are the highest-volume users of flu tests, Samuel said, and their perspectives on whether molecular testing would change their practices was also important.
They got on board with the decision to do molecular flu testing, believing "they would get a definitive answer," with such tests whether someone had the virus, Samuel said.
The decision was also informed by the inpatient side, particularly with management of isolation rooms in the hospitals.Â
"We place patients who are positive for influenza in airborne isolation, and with the RIDT you couldn't tell for certain," Samuel said. For definitive confirmation, PCR testing in the main lab was needed, but that was performed in batch testing, up to twice a day. Furthermore, off-site hospitals couldn't really benefit from lab-based molecular testing because of the distance between sites.
During peak flu season, isolation rooms were often in short supply. With less sensitive RIDT testing, Samuel said it was likely that patients in whom flu could not be ruled out in a timely manner were unnecessarily placed in isolation. Isolation rooms cost more, and so should be used only for people who actually are infected with influenza virus, he noted.
Lastly, the lab at HFHS studied the available literature on assay performance as it prepared its cost justification to present to hospital administrators, Samuel said.
Oversight strategies
From the beginning of POC MDx, there has been a vigorous discussion among clinical lab experts about whether and how labs should oversee decentralized testing.
From the lab manager's perspective, especially, vendors sometimes market CLIA-waived POC molecular testing directly to physician offices at the peripheries of  large health systems, and it can be tough to exert authority once the systems are in place.
Lab directors have long argued that CLIA-waived molecular testing, which can presumably be performed by non-experts, still needs to be treated as highly sensitive, subjected to rigorous quality controls, and overseen by the lab.
Rolling out molecular testing from the lab was, in essence, also a strategy to maintain a centralized authority, because "Without appropriate oversight, who knows what is happening?," Samuel said. As part of its oversight and monitoring, he noted that HFHS adopted a number of different strategies to try to reduce any variability and minimize potential for errors in its molecular flu testing.
For example, although CLIA-waived molecular tests can theoretically be performed by anyone, at HFHS the tests are done only by med techs.
This strategy was motivated in part by the fact that the health system had seen issues in the past with point-of-care immunoassay testing performed by non-lab personnel, particularly problems with quality control and records maintenance.
Samuel was also influenced by anecdotal reports of contamination issues. "With these newer molecular platforms, there are still a lot of unknowns," he said, so designating med techs as the only operators was meant to provide an added level of quality control.
The cost justification that the lab presented to the HFHS administration included the potential impact on patient care. Yes, there would be an increased capital outlay, but this would be offset by increased sensitivity, the lab argued: While 80 percent of RIDT results were negative, "those results had no value because you can't trust a negative antigen test," Samuel said.
"We were spending a lot of tech time and a lot of money on something that had relatively little value — why not redirect those resources to something that does have value for patient care, where you can have a meaningful negative result, as well as a reliable positive result?" Samuel said.
The lab's modeling predicted reduced use of isolation rooms, decreased inpatient length of stay, more appropriate antibiotic use, and more appropriate use of antiviral medication.
Moving to a molecular model for all flu testing, it appeared, was a no-brainer.
The problems
For his presentation at ASM Microbe, Samuel chose the tongue-in-cheek title, "What could possibly go wrong?"
The answer: A lot.
After testing the Roche Liat flu assay in the central lab and getting a green light from HFHS administration to roll it out for the 2016/2017 flu season, "Everything went horribly wrong," Samuel told his fellow laboratorians during his presentation.
It began with changes to the Liat platform that "impacted the reliability of the instrument when it was rolled out," Samuel later said to 360Dx. Compared to the instruments tested in the lab, the ones Roche supplied for the rollout — which, it later turned out, had been manufactured at a different site — had an increased number of invalid results.
This was was ultimately traced to two defects that had not been present before. There were invalid results due to issues with the software, Samuel said, as well as a secondary problem in which a small number of patients with flu B had invalid results due to SNP level mismatches, but not false negatives. Samuel also said that this secondary problem was noted by the techs running the testing who spotted the unusual curves, "Something that a non-lab person may not have noticed," he added.
The problem was corrected with some changes to the software script, and Roche worked closely with the HFHS to resolve it. But in the meantime, the health system still needed to perform between 1,000 and 2,000 flu tests each week during peak flu season.
If there were invalid POC MDx results — and there were — they had to be repeated, causing a major bottleneck. There were about 25 Liat instruments installed in the emergency departments at HFHS, Samuel said, with each Cobas Liat taking about 20 minutes to run. Each instrument can only run one test at a time, or about 72 tests in a 24-hour period. Simply put, demand outstripped supply.
While there were no problems with false negative results, Samuel said, "repeating testing on the same specimen over and over" led to a backlog that then led to "angry mobs at the door" of the lab emanating from the emergency department as well as the administrative offices, Samuel said at ASM Microbe.
There were no fallback options, either. The lab had decided that as part of the rollout it would eliminate RIDTs as well as lab-based flu testing in the core lab.
Roche worked with the lab and supplied new instruments, but initially even those instruments did not pass quality control, Samuel said. To make matters worse, for a period of time during the initial rollout, the lab discovered it sometimes temporarily lost track of instruments because it became hard to track them by serial number for warranty purposes when they were frequently being replaced.
Further adding chaos to the confusion, just as things started to die down, an enterprising med tech decided to swab the inside of a Liat instrument and run a flu test on the swab.
The results came back positive for flu, leading to another cascade of panicked interrogation that nearly halted the POC MDx flu testing entirely until Roche was able to demonstrate that a highly contaminated instrument is unlikely to contaminate the patient sample, a result the company has since published.
It was only because of the early discussion that Samuel had with clinical colleagues and a concerted effort to keep the lines of communication open that he was able to assuage frustrated physicians and administrators, he said. Â
"The providers knew who to call when things were going wrong ... and we were able to give them a heads up. They have a little more understanding when they know who you are and you're not a nameless, faceless box that does lab testing," Samuel said.
Different perspectivesÂ
To be sure, HFHS' experience may be an aberration, and it sharply contrasts the situation at one other health system, Evanston, Illinois-based NorthShore University HealthSystem, which also rolled out the Liat for molecular flu testing in the 2016/2017 flu season.
"We did not have any problems that were unexpected or not easily solved," Sanchita Das, a clinical and molecular microbiologist, and Robert Benirschke, the director of point of care testing at NorthShore, told 360Dx in an email.
NorthShore's initial rollout of the Liat flu assay was at one immediate care center. For the 2017/2018 flu season, NorthShore expanded testing to five additional immediate care centers and four hospital laboratories — although in the latter it was restricted to use with ED patients only — for a total of 12 instruments. In the most recent flu season, the system expanded further, and it now has over 40 instruments at different urgent care and medical group offices.
As the lab recently published in the Journal of Clinical Microbiology, NorthShore already had an established centralized flu PCR assay, and for outpatients and immediate care it reflexed all negative RIDTs to PCR. This centralized PCR system was used as backup for problem samples during POC MDx roll out, specifically those that were repeatedly invalid, though it has only been required in a few cases.
In terms of quality control, NorthShore said it follows manufacturer recommendations and also runs QC on all new shipments. "We subscribe to external [College of American Pathologists] proficiency testing even though this test is CLIA waived ... [and] we performed a validation and comparison of the Liat assay to our centralized PCR assay before we rolled this out," the laboratorians said to 360Dx.
The laboratory also provides overall guidance on use of the tests, such as when to begin testing and the workflow in both the urgent care and medical group settings. Training and ongoing competency assessment is performed by pathology staff in the immediate care centers while the medical group offices manage this themselves.
To prevent the contamination issues, NorthShore also trained staff to clean all surfaces at regular intervals and created maintenance logs to ensure this was being done. Â It also instituted the use of shields in front of the instrument, and separated the collection and preparation areas from the testing areas. So far, there has been no evidence of any contamination.
Positive outcomes, and a moral
Fortunately for HFHS, its problems were constrained to the first year and invalid rates on the Liat have stayed below 2 percent. "We've been running this since the 2016/2017 flu season, and that was the only season we had those issues," Samuel said. "Since then, it's been very smooth." The system is also continuing to expand the number of sites that have POC MDx flu testing available.
Occasionally there are reagent supply issues, he said, but that happens with all the platforms and, "Managing reagent supplies is always challenging for high-volume assays that are complex," he said.
There are no immediate plans to add menu to the existing Liats at HFHS, Samuel said, but it is always a possibility. For example, sexually transmitted disease testing might have some utility, but he also noted that some STI targets are typically tested in high volumes, and the instruments are low throughput.Â
In the end, Samuel said he would advise colleagues to first get hands-on experience with whatever platforms they are considering.
The changes in instrument manufacturing weren't possible to anticipate, Samuel said. "It wasn't like we rolled out an unknown quantity — we knew the machine we were working with, but when changes are happening on the manufacturer's end, it is important for [them] to communicate with the clients," he asserted.
A company official was not available to speek with 360Dx in time for the publication of this article, but a company official confirmed that Samuel presented a fair picture of the situation.
Another key is monitoring performance in real time, rather than monthly, for example, in order to respond and react to problems quickly. "When you decentralize, it is harder to keep track of how test performance is going ... and [users] can be 20 or 30 miles away," Samuel said.
At HFHS even though the testing is CLIA-waived and truly point of care in the sense that tests are not run in small peripheral lab spaces, the HFHS decentralized testing model has a strong oversight component.
"We have a systems-level committee that has representatives from most of the testing locations ... including point-of-care coordinators, lab managers who have oversight of the testing locations, and once a month we get on a conference call and discuss issues, and we're constantly sending emails as a means of collecting feedback," Samuel said.
The lab also tries to standardize the testing protocol across the health system as much as possible. The Liat is connected to the LIS, and the lab aggregates results of testing on each location and monitors the number of invalids particularly closely.
The outcomes have been significant, and positive.
The lab has presented data at recent conferences documenting the effects it has seen using molecular instead of RIDTs at the point of care, such as a 40 percent drop in the number of tests being performed while also diagnosing many more than twice as many cases of influenza.
HFHS has also found that patients tested with the molecular technology were less likely to be admitted to the hospital and less likely to revisit the emergency room in the subsequent 30 days. When they were admitted, they had a shorter length of stay, and there was a lower cost associated with their care.
Stewardship was also improved with POC MDx testing at HFHS. There was an increase in appropriate use of antivirals, and while antibiotics had been prescribed in 70 percent of positive RIDT cases, they were only prescribed in 25 percent of MDx flu cases.
Like HFHS, NorthShore has also seen improvements in stewardship. As described in JCM, antivirals were prescribed for 82 percent of patients who tested positive for flu by POC MDx, but for only 70 percent of the patients who tested positive with RIDT and reflex PCR. And while 25 percent of patients with a negative RIDT still got antivirals only 2 percent of patients with a negative POC molecuar test did. Antibacterial prescribing was not significantly impacted in NorthShore's evaluation, however, but the medical staff reported increased physician and patient satisfaction.Â
The only concern has been the turnaround time, but overall NorthShore believes that the better performance of the assay outweighs this.
Every situation is different, but ideally, Samuel said he would have preferred to have a high-volume PCR assay available in the main lab during the rollout, so it could still offer "like for like" in a crisis. Barring that, being able to offer an antigen test would be better than nothing.
That said, switching to point-of-care molecular testing is "definitely worth the time and effort," Samuel said, even though "You can never underestimate how badly things can go wrong."