By Turna Ray
Consumer genomics firms that were the subject of last week's damaging Government Accountability Office investigation into their practices went on the offensive this week and accused the federal agency's report of being "flawed," "misleading," and "slanderous."
At the same time, Navigenics, 23andMe, and Decode Genetics all welcomed greater regulatory oversight for the nascent industry and vowed to work with the US Food and Drug Administration to develop standards for their offerings.
After a year-long investigation into the direct-to-consumer genomics industry, the GAO issued a report last week in which it concluded that the genetic test results provided by DTC genomics companies were "misleading and of little or no practical use to consumers." After having a week to look over the report, which the GAO presented at a congressional hearing (PGx Reporter 7/23/2010), several DTC genomics firms have taken issue with the agency's findings and tactics.
In conducting its investigation from June 2009 to June 2010 at the request of the House Committee on Energy and Commerce, the GAO deployed five fictitious customers to look into the marketing practices of 15 firms selling tests to consumers online. These customers also purchased gene scans from four DTC firms — 23andMe, Navigenics, Decode Genetics, and Pathway Genomics — to compare how closely the disease predisposition results agreed between companies and to evaluate advice provided by company representatives in follow-up phone calls.
In a review of the donors' test results across all four companies, the GAO found contradictory risk predictions for at least nine diseases for each donor. The agency also concluded that follow-up consultations "provided only general information and not the expert advice the companies promised to provide."
However, several DTC genetics firms this week pointed out weaknesses in the GAO's methodology. For example, because the four companies used varying language to describe risk predictions, GAO investigators used three categories —"average," "above average," and "below average" — in order to compare the risk assessments from the four firms. Kari Stefansson, president of research at Decode, said that with this approach, GAO investigators were "making a game out of the numbers for the purpose of arriving at a self-fulfilling conclusion."
23andMe also found the GAO's report "deeply flawed" due to its "unscientific nature," while Navigenics noted on its blog that "many of the report’s findings are anecdotal, partially informed, or incomplete."
Indeed, the GAO admitted in its report that the investigation it conducted was not grounded in science. "It is important to emphasize that we did not conduct a rigorous scientific study; our observations are those that could be made by any customer," it explained.
Meanwhile, in light of the GAO report, an ongoing congressional investigation into DTC genetics testing, and a two-day meeting held by the US Food and Drug Administration last week to discuss its regulatory approach for laboratory-developed tests, in vitro diagnostics, and DTC genetic tests (PGx Reporter 7/21/2010), the three DTC firms all acknowledged this week that the industry needs regulatory oversight, particularly guidance with regard to clinical validation of its services.
Navigenics, 23andMe, and Decode's Stefansson all said this week that they will work with the FDA to develop standards for the industry. However, in the absence of federal regulations, all three companies maintained that their genetic analyses are robust and based on sound science, even though risk predictions between the various firms may differ based on the SNPs they analyze.
[ pagebreak ]
The FDA is in the process of meeting with DTC firms to carve out a regulatory pathway that addresses the unique features of the industry. In addition to sending letters in June to Pathway, Illumina, 23andMe, Navigenics, Knome, and Decode asking them to discuss regulation of their tests with the agency, FDA sent similar letters to 14 additional genetic testing firms ahead of the congressional hearing last week.
As the agency works with industry in developing standards, there will be areas of agreement and disagreement. For example, 23andMe has openly disagreed with FDA's classification of DTC firms as medical devices. FDA officials at the congressional hearing said that the agency may require DTC firms to garner clearance or approval, and may even restrict them to marketing tests only through physicians.
Although the Google-backed California firm met with the agency last week to discuss its own set of regulatory suggestions, the company wouldn't discuss them with PGx Reporter since the agency is currently reviewing them. "We are encouraged by our discussions with FDA and confident we will be able to meet requirements," said Ashley Gould, general counsel of 23andMe.
Meanwhile, Navigenics has posted on its blog a 10-point proposal for FDA regulation of the industry that it originally proposed in 2008. And Decode, the Iceland-based firm that couldn't make it to the Congressional hearing due to the short notice it was given, told PGx Reporter that it will meet with FDA to discuss industry guidelines on Aug. 4.
Pathway Genomics — the company that kicked off an avalanche of regulatory and congressional scrutiny when it announced in May that it would market its saliva collection kits through Walgreens and CVS pharmacies — was not available for comment to PGx Reporter. However, at the congressional hearing, Pathway Chief Scientific Officer David Becker expressed support for a preliminary proposal from Jeff Shuren, director of FDA's Center for Devices and Radiological Health, suggesting the agency might clear tests marketed by DTC firms based on its review of a subset of markers the companies test for.
Reliable but Different
At the congressional hearing, committee members expressed the most concern over the GAO report's finding that disease predisposition assessments from the three firms disagreed.
In his testimony to the House Committee on Energy and Commerce, Gregory Kutz, managing director of forensic audits and special investigations at the GAO, revealed that he was one of the five fictitious consumers and received conflicting risk predictions for prostate cancer.
Kutz learned from 23andMe and Decode that he had average risk for prostate cancer. Meanwhile, Pathway told him he had below average risk and Navigenics determined that Kutz had above average risk for the disease.
However, as Stefansson pointed out, the investigated companies calculate genetic risk for diseases in terms of odds ratios and do not necessarily interpret the data using the same terms as in the GAO report. While the GAO grouped assessments as "average," "below average," and "above average," several of the firms in the report opt for terms such as "increased risk," "decreased risk", or "typical risk." One company characterizes risk in terms of "strength of susceptibility."
The GAO "looked at the odds ratio, but they took the results and put them into categories," Stefansson said. "If we had reported odds ratio of 1.0 and 23andMe had reported an odds ratio of 1.01, [GAO would have] said that we reported average risk and 23andme reported above average risk, whereas 1.0 and 1.01 are really the same."
[ pagebreak ]
Furthermore, the GAO, not having conducted a scientific investigation, did not account for a margin of error in the companies' analyses, Stefansson added.
"The report talks a lot about misleading results. If I've ever seen a misleading result, it was the analyses that [the GAO] did on the differences in the results from these companies," he noted. "They reported it in a manner that would result in the maximum discrepancy. You have to give yourself a certain error in measurement … and they don't do that. Everyone who does any measurement" leaves room for a margin of error.
The GAO report noted that risk predictions provided by the four firms conflicted with donors' actual illnesses and medical histories. The agency also pointed out that since most gene-risk associations are based on research involving Caucasian populations, none of the companies were able to provide complete results to customers with fictitious profiles claiming to be African American or Asian. And yet, these firms did not "explicitly" inform customers of this limitation ahead of purchasing the tests.
When GAO's fictitious customers pointed out this limitation to the companies, only two firms – Pathway and Decode – refunded customers the money for testing. At the congressional hearing, 23andMe's Gould defended that the company does inform customers when gene-disease associations are based on Caucasian populations and that not all 23andMe's studies have such limitations.
The GAO's customers also made secret calls to several companies and identified instances when customer service representatives gave wrong information or suggested a course of action that would conflict with the company's privacy policies.
In one case, a Navigenics representative told one of GAO's fictitious donors that her "above average" risk for breast cancer meant she was "in the high risk of pretty much getting the disease." At the hearing, Navigenics CEO Vance Vanier bemoaned this revelation as enormously disappointing.
In another call, a GAO donor asked a Pathway Genomics representative whether she could surprise her fiancé with his genetic test results if she could secretly acquire a saliva sample from him. The Pathway representative told the customer that it was a good idea to do so, even though this action is illegal in 33 states. According to Pathway's Becker, this customer rep has been informed of her error and educated about the company's privacy policies.
When another customer pressed a Pathway representative in a follow-up call about her qualifications as a genetic counselor, the representative admitted that she was not a licensed counselor yet, although she had gone to school to become one. Meanwhile, in marketing its service, Pathway promises its customers access to counselors certified by the American Board of Genetic Counseling.
In its broader investigation of the other 11 DTC firms from which it did not order tests, the GAO found several "flagrant" examples of deceptive marketing, such as claims that a customer's DNA could be used to create personalized supplements to cure disease, or claims that DNA testing would reveal in which sport a child might excel.
"The experts we spoke to agreed that the companies' claims are both ambiguous and misleading," the GAO said in its report. "Further, they felt that consumers who are concerned about their health should consult directly with their physicians instead of purchasing these kinds of DTC genetic tests."
Not All DTC Firms Are Alike
Not surprisingly, DTC firms weren't happy with the GAO's findings, which took them by surprise since the agency didn't allow the firms to view the report ahead of the congressional hearing.
[ pagebreak ]
"When last week’s events kicked off with a two-day meeting called by the FDA, we were pleased that the need for industry standards has been acknowledged at a high level. At the gathering of experts in health, genetics, science, and the law, many good points were raised and excellent ideas exchanged," Navigenics stated on its blog. "The next day, however, saw a very different — and less productive — atmosphere come to light … on Capitol Hill."
The main contention of these DTC firms is that the report does not differentiate between the policies and scientific methods employed at the 15 companies it investigated. Furthermore, the companies assert that the GAO approached its investigation with the supposition that unmediated access to test results will invariably result in consumer harm.
They also pointed out the agency's failure to acknowledge that some of the differences between the companies' results may be due to the lack of overarching standards from federal regulators that would have provided a baseline for comparison.
The GAO report did not employ "a particularly honest method, nor does it make a lot of sense to try to evaluate quality by taking these four companies and comparing the results," Stefansson said. "Among these companies there may be one or two that do this work at an extremely high quality but others that work at a lower quality, and to judge everyone on the basis of harmony of results is not a particularly good method."
Operating in the current climate of regulatory uncertainty, Navigenics pointed out in its blog that although there is no federal oversight of DTC firms, Navigenics is the only firm to have garnered a license to operate in all 50 states.
23andMe pointed out in its blog that the GAO report did not review all the services provided by the firms it investigated. Choosing to focus only on genetic tests that gauge disease predisposition, the GAO's report makes little acknowledgement of the fact that several of these firms provide genetic data related to ancestry, carrier status for single-gene diseases, and drug response. "This suggests that GAO found no problems with these parts of our service," 23andMe asserts.
23andMe was also quick to differentiate itself from Navigenics and Pathway, two firms where customer representatives were found to provide misleading advice to GAO's donors through secretly taped phone calls. "Other than saying that we believe customers should consult with their physician or other healthcare professional when they have questions about their data, 23andMe is not implicated in any wrongdoing" in the GAO report, the company states in its blog.
Stefansson took issue with the GAO's discussion of the service provided by consumer genomics firms in the context of nutrigenomics firms. In 2006, the GAO investigated several nutrigenomics companies and testified before a Senate committee that these firms were making claims that were scientifically unproven and misleading to the public. GAO's Kutz, in presenting the report on the consumer genomics industry last week to the House Committee on Energy and Commerce, started off by discussing the GAO's investigation of nutrigenomics firms.
Kutz "put the consumer genomics industry in the context of a fraudulent discipline that has little scientific foundation," Stefansson said. "The science at DTC genomics firms is as high as in any discipline of biology today. It's absolutely outrageous that this report begins in that context."
While consumer genomics firms may differ at this point in terms of the SNPs they analyze, the level of privacy they provide, and the policies that drive their research, they argued that once federal regulations are established for the industry there will be a base set of standards by which they will have to operate.
Navigenics has some advice for health regulators in this regard. In 2008, the company issued 10 standards as a guide for companies operating in this space. The suggestions focus on establishing clinical validity of predisposition data; assuring quality of testing laboratories through CLIA certification; providing data with a 99 percent or more analytical validity and a 100 percent call rate; and reporting on genes that have been replicated in multiple populations of the same ethnicity.
[ pagebreak ]
According to Navigenics, DTC firms should also strive to provide their customers with continued access to genetic counseling, establish security and privacy standards that give customers control over who can view their genetic information, acknowledge that customers have ownership over their genetic information; engage in physician education, be transparent about the limitations of the service, and conduct studies to assess the impact genetic information has on behavior over time.
Ensuring that the genetic information provided by DTC firms is actionable in terms of healthcare decisions or lifestyle choices is a premise that Navigenics has always supported through its service and feels the entire industry should adopt.
Meanwhile, Navigenics tests its customers' genetic risk for Alzheimer's, a disease for which there is no cure. Experts have questioned the clinical utility of knowing one's genetic risk for Alzheimer's, in the absence of meaningful early interventions. Some of the preventative strategies for Alzheimer's discussed on Navigenics website, include taking up a musical instrument or doing Sudoku puzzles.
During the hearing, members of Congress repeatedly spoke of rash decisions and unnecessary worry that consumers could experience from incorrect disease risk predictions. One congressman dramatized in his opening statements during the hearing that not only irreparable harm, but even death, could result from patients getting inaccurate disease susceptibility estimates.
While there may be scant and preliminary evidence suggesting that genetic risk assessments spur people to make positive lifestyle choices, it is worth noting that there is also a lack of evidence suggesting that direct access to genetic test results causes patients psychological stress, and there has been no recorded instance of death resulting from less than stellar gene risk assessments from DTC firms.
Still, DTC firms are walking a fine line. Consumer genomics firms maintain that the genetic data provided by their services are for education and research use only, but many of these companies have partnered with doctors to incorporate their customers' genetic data into medical decisions.
Concerned medical professionals and genetic researchers believe that these firms are providing medical advice based on gene associations with unproven clinical utility, and that this could cause patients and doctors to make unwise healthcare decisions.
While there is agreement within the life sciences community that consumer genomics firms offer a viable research model to speed the incorporation of genomics into medical care, there is broad disagreement on whether disease-risk predictions from these companies should be incorporated into patient care at this time.
Within this context, at least two firms have backed away from the DTC model. Navigenics has put its DTC service on hold, and is operating only through wellness programs and physician groups. Pathway has also taken the physician-based marketing route and shelved DTC access for now. 23andMe and Decode continue to operate under the DTC model.
"I don't think we make false claims on our website and we don't think we went over the line. I think the information [on Decode's website] is very responsibly presented by experts in the field," Stefansson said. "If the concern is [that it's] going to be dangerous for people to learn about the risk of diseases or [whether we] should prevent people from buying information about their genetic risk for diseases, I think [restricting access to such information] would go against everything that American culture stands for."
He added that it’s "patronizing" to tell consumers that information about their genetic risk for disease is dangerous. "And I think it's also patronizing to tell people that they have to learn about their risk for diseases from a third party, a healthcare provider of some sort."
[ pagebreak ]
Echoing similar sentiments is 23andMe, a company that through the recent maelstrom of regulatory and congressional scrutiny, has maintained its motto that "individuals have a right to their genetic information" without doctors as intermediaries.
However, at the congressional hearing, James Evans, director of adult genetics services at the University of North Carolina and a consultant to GAO in drafting its report, asserted that the gene risk data provided by these services are of little value.
"It's clear from the GAO report [that these companies] really don't know how to interpret the data," Evans said. "In the advent of this kind of testing … we don't understand how to interpret the data, and this can lead to harm." While Evans acknowledged that the analytical validity of the tests is strong, he found the clinical validity of risk assessments lacking.
While DTC firms have acknowledged the dearth of clinical validity for their services, Navigenics' Vanier noted at the hearing last week that there is no "uniformity in the academic community" that genetic test results cause harm. Furthermore, Vanier noted that Eric Topol, director of the Scripps Translational Science Institute, will soon release preliminary results from a study showing that genomic information does not have negative psychological effects on consumers and encourages them to take positive action.
Topol told PGx Reporter that the results of this study are currently under review for publication. He said he would discuss the preliminary findings of this ongoing, 20-year behavior study Scripps is conducting with Navigenics, Affymetrix, and Microsoft (PGx Reporter 10/15/2008) upon publication of the results.
Despite the apparent differences between the GAO and DTC firms, it appears that both sides are joined in their concern to protect public health.
Although he disagrees with the findings in the GAO report, Stefansson recognized that the GAO pursued its investigation out of a "genuine desire" to protect the consumer.
"The lack of a regulatory oversight has incited concern and those concerns pushed GAO and the [congressional] committee over the line into making statements that are untrue and slanderous," he said. "But I'm convinced that out of all this will come a very good regulatory framework for these tests."
Similarly, at the hearing GAO's Kutz expressed his belief that the four DTC genomics firms for which the GAO ordered tests are not intentionally attempting to deceive the public. "I think they believe what they're doing is good," he told the committee.