Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer

Babak Ehteshami Bejnordi; Mitko Veta; Paul Johannes van Diest; Bram van Ginneken; Nico Karssemeijer; Geert Litjens; Jeroen A W M van der Laak; Meyke Hermsen; Quirine F Manson; Maschenka Balkenhol; Oscar Geessink; Nikolaos Stathonikos; Marcory Crf van Dijk; Peter Bult; Francisco Beca; Andrew H Beck; Dayong Wang; Aditya Khosla; Rishab Gargeya; Humayun Irshad; Aoxiao Zhong; Qi Dou; Quanzheng Li; Hao Chen; Huang-Jing Lin; Pheng-Ann Heng; Christian Haß; Elia Bruni; Quincy Wong; Ugur Halici; Mustafa Ümit Öner; Rengul Cetin-Atalay; Matt Berseth; Vitali Khvatkov; Alexei Vylegzhanin; Oren Kraus; Muhammad Shaban; Nasir Rajpoot; Ruqayya Awan; Korsuk Sirinukunwattana; Talha Qaiser; Yee-Wah Tsang; David Tellez; Jonas Annuscheit; Peter Hufnagl; Mira Valkonen; Kimmo Kartasalo; Leena Latonen; Pekka Ruusuvuori; Kaisa Liimatainen; Shadi Albarqouni; Bharti Mungal; Ami George; Stefanie Demirci; Nassir Navab; S Watanabe; Shigeto Seno; Yoichi Takenaka; Hideo Matsuda; Parham Phoulady; Vassili Kovalev; Alexander Kalinovsky; Vitali Liauchuk; Gloria Bueno; M Milagro Fernandez-Carrobles; Ismael Serrano; Oscar Deniz; Daniel Racoceanu; Rui Venâncio

doi:10.1001/jama.2017.14585

Back

Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer

Journal article

Peer reviewed

Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer

Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes van Diest, Bram van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen A W M van der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balkenhol, …

JAMA : the journal of the American Medical Association, Vol.318(22), pp.2199-2210

12/12/2017

DOI: https://doi.org/10.1001/jama.2017.14585

Handle:

https://hdl.handle.net/20.500.12741/rep:6857

PMCID: PMC5820737

PMID: 29234806

Abstract

Lymphatic Metastasis - diagnosis

Breast Neoplasms - pathology

Humans

Lymphatic Metastasis - pathology

Female

Pathologists

Pathology, Clinical

ROC Curve

Algorithms

Machine Learning

Application of deep learning algorithms to whole-slide pathology images can potentially improve diagnostic accuracy and efficiency. Assess the performance of automated deep learning algorithms at detecting metastases in hematoxylin and eosin-stained tissue sections of lymph nodes of women with breast cancer and compare it with pathologists' diagnoses in a diagnostic setting. Researcher challenge competition (CAMELYON16) to develop automated solutions for detecting lymph node metastases (November 2015-November 2016). A training data set of whole-slide images from 2 centers in the Netherlands with (n = 110) and without (n = 160) nodal metastases verified by immunohistochemical staining were provided to challenge participants to build algorithms. Algorithm performance was evaluated in an independent test set of 129 whole-slide images (49 with and 80 without metastases). The same test set of corresponding glass slides was also evaluated by a panel of 11 pathologists with time constraint (WTC) from the Netherlands to ascertain likelihood of nodal metastases for each slide in a flexible 2-hour session, simulating routine pathology workflow, and by 1 pathologist without time constraint (WOTC). Deep learning algorithms submitted as part of a challenge competition or pathologist interpretation. The presence of specific metastatic foci and the absence vs presence of lymph node metastasis in a slide or image using receiver operating characteristic curve analysis. The 11 pathologists participating in the simulation exercise rated their diagnostic confidence as definitely normal, probably normal, equivocal, probably tumor, or definitely tumor. The area under the receiver operating characteristic curve (AUC) for the algorithms ranged from 0.556 to 0.994. The top-performing algorithm achieved a lesion-level, true-positive fraction comparable with that of the pathologist WOTC (72.4% [95% CI, 64.3%-80.4%]) at a mean of 0.0125 false-positives per normal whole-slide image. For the whole-slide image classification task, the best algorithm (AUC, 0.994 [95% CI, 0.983-0.999]) performed significantly better than the pathologists WTC in a diagnostic simulation (mean AUC, 0.810 [range, 0.738-0.884]; P < .001). The top 5 algorithms had a mean AUC that was comparable with the pathologist interpreting the slides in the absence of time constraints (mean AUC, 0.960 [range, 0.923-0.994] for the top 5 algorithms vs 0.966 [95% CI, 0.927-0.998] for the pathologist WOTC). In the setting of a challenge competition, some deep learning algorithms achieved better diagnostic performance than a panel of 11 pathologists participating in a simulation exercise designed to mimic routine pathology workflow; algorithm performance was comparable with an expert pathologist interpreting whole-slide images without time constraints. Whether this approach has clinical utility will require evaluation in a clinical setting.

Files and links (1)

url

https://doi.org/10.1001/jama.2017.14585View

Published (Version of record)

Metrics

1 File views/ downloads

29 Record Views

See more details

Details

Title: Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer
Creators: Babak Ehteshami Bejnordi (Author) - Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands
Mitko Veta (Author) - Medical Image Analysis Group, Eindhoven University of Technology, Eindhoven, the Netherlands
Paul Johannes van Diest (Author) - Department of Pathology, University Medical Center Utrecht, Utrecht, the Netherlands
Bram van Ginneken (Author) - Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands
Nico Karssemeijer (Author) - Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands
Geert Litjens (Author) - Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
Jeroen A W M van der Laak (Author) - Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
Meyke Hermsen (Author) - Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
Quirine F Manson (Author) - Department of Pathology, University Medical Center Utrecht, Utrecht, the Netherlands
Maschenka Balkenhol (Author) - Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
Oscar Geessink (Author) - Laboratorium Pathologie Oost Nederland, Hengelo, the Netherlands
Nikolaos Stathonikos (Author) - Department of Pathology, University Medical Center Utrecht, Utrecht, the Netherlands
Marcory Crf van Dijk (Author) - Rijnstate Hospital, Arnhem, the Netherlands
Peter Bult (Author) - Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
Francisco Beca (Author) - BeckLab, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts
Andrew H Beck (Author) - PathAI, Cambridge, Massachusetts
Dayong Wang (Author) - PathAI, Cambridge, Massachusetts
Aditya Khosla (Author) - Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, Massachusetts
Rishab Gargeya (Author) - Harker School, San Jose, California
Humayun Irshad (Author) - BeckLab, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts
Aoxiao Zhong (Author) - Center for Clinical Data Science, Gordon Center for Medical Imaging, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
Qi Dou (Author) - Chinese University of Hong Kong, Hong Kong, China
Quanzheng Li (Author) - Center for Clinical Data Science, Gordon Center for Medical Imaging, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
Hao Chen (Author) - Chinese University of Hong Kong, Hong Kong, China
Huang-Jing Lin (Author) - Chinese University of Hong Kong, Hong Kong, China
Pheng-Ann Heng (Author) - Chinese University of Hong Kong, Hong Kong, China
Christian Haß (Author) - ExB Research and Development GmbH, Munich, Germany
Elia Bruni (Author) - ExB Research and Development GmbH, Munich, Germany
Quincy Wong (Author) - Munich Business School, Munich, Germany
Ugur Halici (Author) - Neuroscience and Neurotechnology, Graduate School of Natural and Applied Sciences, Middle East Technical University, Ankara, Turkey
Mustafa Ümit Öner (Author) - Department of Electrical and Electronics Engineering, Middle East Technical University, Ankara, Turkey
Rengul Cetin-Atalay (Author) - Cancer System Biology Laboratory, Graduate School of Informatics, Middle East Technical University, Ankara, Turkey
Matt Berseth (Author) - NLP LOGIX, Jacksonville, Florida
Vitali Khvatkov (Author) - Smart Imaging Technologies, Houston, Texas
Alexei Vylegzhanin (Author) - Smart Imaging Technologies, Houston, Texas
Oren Kraus (Author) - Department of Electrical and Computer Engineering, University of Toronto, Toronto, Ontario, Canada
Muhammad Shaban (Author) - Tissue Image Analytics Lab, Department of Computer Science, University of Warwick, Coventry, United Kingdom
Nasir Rajpoot (Author) - Department of Pathology, University Hospitals Coventry and Warwickshire National Health Service Foundation Trust, Coventry, United Kingdom
Ruqayya Awan (Author) - Department of Computer Science and Engineering, Qatar University, Doha, Qatar
Korsuk Sirinukunwattana (Author) - Tissue Image Analytics Lab, Department of Computer Science, University of Warwick, Coventry, United Kingdom
Talha Qaiser (Author) - Tissue Image Analytics Lab, Department of Computer Science, University of Warwick, Coventry, United Kingdom
Yee-Wah Tsang (Author) - Department of Pathology, University Hospitals Coventry and Warwickshire National Health Service Foundation Trust, Coventry, United Kingdom
David Tellez (Author) - Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
Jonas Annuscheit (Author) - Hochschule für Technik und Wirtschaft, Berlin, Germany
Peter Hufnagl (Author) - Hochschule für Technik und Wirtschaft, Berlin, Germany
Mira Valkonen (Author) - BioMediTech Institute and Faculty of Medicine and Life Sciences, Tampere University of Technology, Tampere, Finland
Kimmo Kartasalo (Author) - BioMediTech Institute and Faculty of Biomedical Science and Engineering, Tampere University of Technology, Tampere, Finland
Leena Latonen (Author) - Prostate Cancer Research Center, Faculty of Medicine and Life Sciences and BioMediTech, University of Tampere, Tampere, Finland
Pekka Ruusuvuori (Author) - Faculty of Computing and Electrical Engineering, Tampere University of Technology, Pori, Finland
Kaisa Liimatainen (Author) - Hochschule für Technik und Wirtschaft, Berlin, Germany
Shadi Albarqouni (Author) - Technical University of Munich, Munich, Germany
Bharti Mungal (Author) - Technical University of Munich, Munich, Germany
Ami George (Author) - Technical University of Munich, Munich, Germany
Stefanie Demirci (Author) - Technical University of Munich, Munich, Germany
Nassir Navab (Author) - Technical University of Munich, Munich, Germany
S Watanabe (Author) - Department of Bioinformatic Engineering, Osaka University
Shigeto Seno (Author) - Department of Bioinformatic Engineering, Osaka University
Yoichi Takenaka (Author) - Department of Bioinformatic Engineering, Osaka University
Hideo Matsuda (Author) - Department of Bioinformatic Engineering, Osaka University
Parham Phoulady (Author)
Vassili Kovalev (Author) - Biomedical Image Analysis Department, United Institute of Informatics Problems, Belarus National Academy of Sciences, Minsk, Belarus
Alexander Kalinovsky (Author) - Biomedical Image Analysis Department, United Institute of Informatics Problems, Belarus National Academy of Sciences, Minsk, Belarus
Vitali Liauchuk (Author) - Biomedical Image Analysis Department, United Institute of Informatics Problems, Belarus National Academy of Sciences, Minsk, Belarus
Gloria Bueno (Author) - Visilab, University of Castilla-La Mancha, Ciudad Real, Spain
M Milagro Fernandez-Carrobles (Author) - Visilab, University of Castilla-La Mancha, Ciudad Real, Spain
Ismael Serrano (Author) - Visilab, University of Castilla-La Mancha, Ciudad Real, Spain
Oscar Deniz (Author) - Visilab, University of Castilla-La Mancha, Ciudad Real, Spain
Daniel Racoceanu (Author) - Pontifical Catholic University of Peru, San Miguel, Lima, Peru
Rui Venâncio (Author) - Sorbonne University, Pierre and Marie Curie University, Paris, France
Academic Unit: Computer Science Department
Publication Details: 12/12/2017
Grant note: Stichting IT Projecten and by the Fonds Economische Structuurversterking (tEPIS/TRAIT project; LSH-FES Program 2009; DFES1029161 and FES1103JJT8U) By Grant 601040 from the Seventh Framework Programme for Research–funded VPH-PRISM project of the European Union.
Identifiers: 99257869978301671; https://hdl.handle.net/20.500.12741/rep:6857; https://doi.org/10.1001/jama.2017.14585
Language: English