Skip to main content
Premium Trial:

Request an Annual Quote

MSK Researchers Describe Clinical-Grade Deep Learning Approach to Assist Pathologists


NEW YORK – Researchers at Memorial Sloan Kettering Cancer Center today described an approach that they said speeds up the development of clinical-grade, deep learning-based computational pathology systems by evading the time-consuming process of manually annotating slide images.

The approach, published in Nature Medicine, addresses a need for more effective decision support systems that cancer pathologists can use in clinical practice, Thomas Fuchs, one of the developers of the new approach and a director of the Computational Pathology Lab at Memorial Sloan Kettering Cancer Center, said in an interview.

Manual annotation slows the preparation of slide images representing patient tissue samples that are then used to develop decision support systems, such as those that use artificial intelligence to augment a pathologist's work and enable better diagnoses, Fuchs said.

For example, a single slide image associated with prostate cancer, he said, can take up to 45 minutes of a trained pathologists' time prior to feeding the slide into the computation system.

In a study, the MSK researchers reported using the approach to develop a multiple instance learning-based deep learning framework on a dataset of 44,732 whole-slide images from 15,187 patients, representing cancer cases from more than 800 centers globally, without any form of data curation.

The research team — two of whom are affiliated with both the Weill Cornell Graduate School of Medical Sciences and MSK — reported that tests on slide images associated with prostate cancer, skin cancer, and breast cancer resulted in areas under the curve above 0.98.

The Nature Medicine paper describes "a seminal piece of work that will transform the future practice of pathology," said Beatrice Knudsen, a professor of biomedical sciences and pathology and director of translational pathology at Cedars-Sinai Medical Center in Los Angeles, California, and who is not involved in the study described in the Nature Medicine paper.

"In my opinion, this paper demonstrates for the first time that it is feasible to develop a decision support tool that can assist pathologists in their daily work," she said. "Training a computer with an unprecedented large number of slides and using a new modeling approach achieved a software product that works well on challenging diagnostic cases from all around the world."

Further, the approach allows pathologists to spend more time on analyzing the details of each cancer, Knudsen said.

According to Fuchs, the method developed by him and his collaborators, reflects the enormous potential "for artificial intelligence and decision support systems that can help pathologists to not only be faster and more efficient but also to be more confident and consistent."

The big problem until now, largely because of the requirement to do painstaking annotation, is that the development of most computational decision support systems has involved the use of relatively small curated datasets — from a few hundred to about 1,000 slides, at most, he said. Such small dataset fail to capture the variability found in cancer.

Pathologists and the computational systems they use to diagnose cancers need to be able to learn from thousands of slides; obtain enough data that represent diversified cancers influenced by variations associated with different ethnicities and geographic regions; leverage novel ways of training machine learning models; and use a high-performance computer infrastructure, Fuchs said.

In clinical application, the MSK system would enable pathologists to exclude up to 75 percent of slides normally needed to train such decision support systems and still retain 100 percent sensitivity, he said. The large-scale training of accurate classification models lays the foundation for the deployment of computational decision support systems in clinical practice.

Such systems are being developed to assist but not replace pathologists. "A pathologist with an AI is so much more powerful and has so much more confidence in decisions that it is in patients' interests that they use the system," Fuchs said. "It is also in the interest of a good clinician, because the primary goal is to get better, more reproducible results for the patient."

Ulysses Balis, a professor of pathology and director of the division of pathology information at Michigan Medicine, said in an interview that while it is important to note that advanced decision support systems cannot replace skilled pathologists, the MSK study shows that "we will have the use of computational tools able to render a full set of diagnostic information in the not-too-distant future."

Sub-visual information unobtainable without the use of computation will soon become a recognized component of a diagnosis, he said.

MSK and other investigators at the forefront of developing pathology systems using advanced informatics are "chipping away at this objective one organ system at a time, one cancer at a time, and one clinical diagnosis at a time," Balis said. Such systems may take many years to implement clinically, but "when it is done, we will have a machine learning solution for just about any diagnostic entity," he said.

Liron Pantanowitz, vice chairman for pathology informatics at the University of Pittsburgh Medical Center, which is affiliated with Shadyside Hospital, noted the MSK study highlights a bottleneck that exists in finding experts who can work on "supervising" algorithms — the process associated with doing slide annotation.

"We probably need more than a thousand algorithms or software applications in our profession to completely assist a pathologist," Pantanowitz said. "At MSK, they have a good digital pathology infrastructure and IT support system; they have some of the world's best pathologists; and they have 25 million slides that can be digitized," he said.

What this all means is that the MSK researchers can expedite the development of algorithms, but it doesn't mean for certain that the algorithms will work in clinical practice, Pantanowitz said. The question of whether it will be possible to generalize the algorithms — applying them clinically in many settings and for many cancers — still needs to demonstrated, he said. "If it works at MSK, will it work at the UPMC, and will it work in African countries where it is really needed because they don't have the same level of digital pathology expertise?" Pantanowitz said.

Another open question is how well the deep learning-based systems will operate at scale in large health centers and integrated health systems, Pantanowitz said, adding the regulatory environment associated with providing oversight of these systems is still evolving.

The US Food and Drug Administration has shown a willingness to approve artificial intelligence-based pathology systems for use in specific circumstances.

According to Pantanowitz, pathologist have been given the green light to use such technologies, digital pathology vendors see that they will be used clinically, and those who have worked on AI "see that their inventions can be plugged in."

Balis added that more than an approach to assist pathologists in diagnosing cancers, deep learning will be increasingly deployed to help clinicians predict patient outcomes. "What we are unsure of is who will be most critically involved in implementing and interpreting results with these systems," he said, adding that pathologists should be integral to the process but whether they will be is still an open question.

In that context, the MSK study is a "wonderful wakeup call that demonstrates their approaches are valid," which brings to attention a lack of readiness among pathologists to implement such technologies. "We need skilled pathologists that know how to use these technologies and approaches and we don't have that right now," Balis said.

Further, these developments in computational pathology systems are coming at a time when pathology lab technicians are seeing dramatic changes to the types and depth of analyses they perform on a routine basis.

Navigating regulations

To navigate regulatory hurdles among other activities, Fuchs has cofounded a company, Paige.AI, that has licensed computational pathology technology from MSK and is developing it for commercial clinical applications. In March, the firm received a US Food and Drug Administration Breakthrough Device Designation for use of its computational pathology tools in prostate cancer.

Technically, the system for which the firm received Breakthrough Device Designation is related to its academic study, but "the specific algorithm is different," Fuchs, who is CSO of Paige.AI, said.

He added that in its commercial development work, Paige.AI has advanced the work described in the Nature Medicine paper. To put its approaches into the hands of pathologists, the firm is developing commercial grade code, building quality management systems, and embarking on regulatory and business activities, he said.

In addition to leveraging slide images for cancer cases at MSK, the firm and the center's researchers have been able to tap into an archive of slides accumulated from cancer cases obtained at other research institutions globally that were escalated to MSK for a second opinion.

Paige.AI anticipates eventually making its computational pathology systems available to hospitals worldwide, Fuchs said.