DeepCell Kiosk: scaling deep learning-enabled cellular image analysis with Kubernetes.
Bannon Dylan,Moen Erick,Schwartz Morgan,Borba Enrico,Kudo Takamasa,Greenwald Noah,Vijayakumar Vibha,Chang Brian,Pao Edward,Osterman Erik,Graf William,Van Valen David
Deep learning is transforming the analysis of biological images, but applying these models to large datasets remains challenging. Here we describe the DeepCell Kiosk, cloud-native software that dynamically scales deep learning workflows to accommodate large imaging datasets. To demonstrate the scalability and affordability of this software, we identified cell nuclei in 10 1-megapixel images in ~5.5 h for ~US$250, with a cost below US$100 achievable depending on cluster configuration. The DeepCell Kiosk can be downloaded at https://github.com/vanvalenlab/kiosk-console ; a persistent deployment is available at https://deepcell.org/ .
nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation.
Isensee Fabian,Jaeger Paul F,Kohl Simon A A,Petersen Jens,Maier-Hein Klaus H
Biomedical imaging is a driver of scientific discovery and a core component of medical care and is being stimulated by the field of deep learning. While semantic segmentation algorithms enable image analysis and quantification in many applications, the design of respective specialized solutions is non-trivial and highly dependent on dataset properties and hardware conditions. We developed nnU-Net, a deep learning-based segmentation method that automatically configures itself, including preprocessing, network architecture, training and post-processing for any new task. The key design choices in this process are modeled as a set of fixed parameters, interdependent rules and empirical decisions. Without manual intervention, nnU-Net surpasses most existing approaches, including highly specialized solutions on 23 public datasets used in international biomedical segmentation competitions. We make nnU-Net publicly available as an out-of-the-box tool, rendering state-of-the-art segmentation accessible to a broad audience by requiring neither expert knowledge nor computing resources beyond standard network training.
An Observational Study of Deep Learning and Automated Evaluation of Cervical Images for Cancer Screening.
Hu Liming,Bell David,Antani Sameer,Xue Zhiyun,Yu Kai,Horning Matthew P,Gachuhi Noni,Wilson Benjamin,Jaiswal Mayoore S,Befano Brian,Long L Rodney,Herrero Rolando,Einstein Mark H,Burk Robert D,Demarco Maria,Gage Julia C,Rodriguez Ana Cecilia,Wentzensen Nicolas,Schiffman Mark
Journal of the National Cancer Institute
BACKGROUND:Human papillomavirus vaccination and cervical screening are lacking in most lower resource settings, where approximately 80% of more than 500 000 cancer cases occur annually. Visual inspection of the cervix following acetic acid application is practical but not reproducible or accurate. The objective of this study was to develop a "deep learning"-based visual evaluation algorithm that automatically recognizes cervical precancer/cancer. METHODS:A population-based longitudinal cohort of 9406 women ages 18-94 years in Guanacaste, Costa Rica was followed for 7 years (1993-2000), incorporating multiple cervical screening methods and histopathologic confirmation of precancers. Tumor registry linkage identified cancers up to 18 years. Archived, digitized cervical images from screening, taken with a fixed-focus camera ("cervicography"), were used for training/validation of the deep learning-based algorithm. The resultant image prediction score (0-1) could be categorized to balance sensitivity and specificity for detection of precancer/cancer. All statistical tests were two-sided. RESULTS:Automated visual evaluation of enrollment cervigrams identified cumulative precancer/cancer cases with greater accuracy (area under the curve [AUC] = 0.91, 95% confidence interval [CI] = 0.89 to 0.93) than original cervigram interpretation (AUC = 0.69, 95% CI = 0.63 to 0.74; P < .001) or conventional cytology (AUC = 0.71, 95% CI = 0.65 to 0.77; P < .001). A single visual screening round restricted to women at the prime screening ages of 25-49 years could identify 127 (55.7%) of 228 precancers (cervical intraepithelial neoplasia 2/cervical intraepithelial neoplasia 3/adenocarcinoma in situ [AIS]) diagnosed cumulatively in the entire adult population (ages 18-94 years) while referring 11.0% for management. CONCLUSIONS:The results support consideration of automated visual evaluation of cervical images from contemporary digital cameras. If achieved, this might permit dissemination of effective point-of-care cervical screening.
Automated acquisition of explainable knowledge from unannotated histopathology images.
Yamamoto Yoichiro,Tsuzuki Toyonori,Akatsuka Jun,Ueki Masao,Morikawa Hiromu,Numata Yasushi,Takahara Taishi,Tsuyuki Takuji,Tsutsumi Kotaro,Nakazawa Ryuto,Shimizu Akira,Maeda Ichiro,Tsuchiya Shinichi,Kanno Hiroyuki,Kondo Yukihiro,Fukumoto Manabu,Tamiya Gen,Ueda Naonori,Kimura Go
Deep learning algorithms have been successfully used in medical image classification. In the next stage, the technology of acquiring explainable knowledge from medical images is highly desired. Here we show that deep learning algorithm enables automated acquisition of explainable features from diagnostic annotation-free histopathology images. We compare the prediction accuracy of prostate cancer recurrence using our algorithm-generated features with that of diagnosis by expert pathologists using established criteria on 13,188 whole-mount pathology images consisting of over 86 billion image patches. Our method not only reveals findings established by humans but also features that have not been recognized, showing higher accuracy than human in prognostic prediction. Combining both our algorithm-generated features and human-established criteria predicts the recurrence more accurately than using either method alone. We confirm robustness of our method using external validation datasets including 2276 pathology images. This study opens up fields of machine learning analysis for discovering uncharted knowledge.
Nucleus segmentation across imaging experiments: the 2018 Data Science Bowl.
Caicedo Juan C,Goodman Allen,Karhohs Kyle W,Cimini Beth A,Ackerman Jeanelle,Haghighi Marzieh,Heng CherKeng,Becker Tim,Doan Minh,McQuin Claire,Rohban Mohammad,Singh Shantanu,Carpenter Anne E
Segmenting the nuclei of cells in microscopy images is often the first step in the quantitative analysis of imaging data for biological and biomedical applications. Many bioimage analysis tools can segment nuclei in images but need to be selected and configured for every experiment. The 2018 Data Science Bowl attracted 3,891 teams worldwide to make the first attempt to build a segmentation method that could be applied to any two-dimensional light microscopy image of stained nuclei across experiments, with no human interaction. Top participants in the challenge succeeded in this task, developing deep-learning-based models that identified cell nuclei across many image types and experimental conditions without the need to manually adjust segmentation parameters. This represents an important step toward configuration-free bioimage analysis software tools.
Deep Learning Localizes and Identifies Polyps in Real Time With 96% Accuracy in Screening Colonoscopy.
Urban Gregor,Tripathi Priyam,Alkayali Talal,Mittal Mohit,Jalali Farid,Karnes William,Baldi Pierre
BACKGROUND & AIMS:The benefit of colonoscopy for colorectal cancer prevention depends on the adenoma detection rate (ADR). The ADR should reflect the adenoma prevalence rate, which is estimated to be higher than 50% in the screening-age population. However, the ADR by colonoscopists varies from 7% to 53%. It is estimated that every 1% increase in ADR lowers the risk of interval colorectal cancers by 3%-6%. New strategies are needed to increase the ADR during colonoscopy. We tested the ability of computer-assisted image analysis using convolutional neural networks (CNNs; a deep learning model for image analysis) to improve polyp detection, a surrogate of ADR. METHODS:We designed and trained deep CNNs to detect polyps using a diverse and representative set of 8,641 hand-labeled images from screening colonoscopies collected from more than 2000 patients. We tested the models on 20 colonoscopy videos with a total duration of 5 hours. Expert colonoscopists were asked to identify all polyps in 9 de-identified colonoscopy videos, which were selected from archived video studies, with or without benefit of the CNN overlay. Their findings were compared with those of the CNN using CNN-assisted expert review as the reference. RESULTS:When tested on manually labeled images, the CNN identified polyps with an area under the receiver operating characteristic curve of 0.991 and an accuracy of 96.4%. In the analysis of colonoscopy videos in which 28 polyps were removed, 4 expert reviewers identified 8 additional polyps without CNN assistance that had not been removed and identified an additional 17 polyps with CNN assistance (45 in total). All polyps removed and identified by expert review were detected by the CNN. The CNN had a false-positive rate of 7%. CONCLUSION:In a set of 8,641 colonoscopy images containing 4,088 unique polyps, the CNN identified polyps with a cross-validation accuracy of 96.4% and an area under the receiver operating characteristic curve of 0.991. The CNN system detected and localized polyps well within real-time constraints using an ordinary desktop machine with a contemporary graphics processing unit. This system could increase the ADR and decrease interval colorectal cancers but requires validation in large multicenter trials.
Image-based consensus molecular subtype (imCMS) classification of colorectal cancer using deep learning.
Sirinukunwattana Korsuk,Domingo Enric,Richman Susan D,Redmond Keara L,Blake Andrew,Verrill Clare,Leedham Simon J,Chatzipli Aikaterini,Hardy Claire,Whalley Celina M,Wu Chieh-Hsi,Beggs Andrew D,McDermott Ultan,Dunne Philip D,Meade Angela,Walker Steven M,Murray Graeme I,Samuel Leslie,Seymour Matthew,Tomlinson Ian,Quirke Phil,Maughan Timothy,Rittscher Jens,Koelzer Viktor H,
OBJECTIVE:Complex phenotypes captured on histological slides represent the biological processes at play in individual cancers, but the link to underlying molecular classification has not been clarified or systematised. In colorectal cancer (CRC), histological grading is a poor predictor of disease progression, and consensus molecular subtypes (CMSs) cannot be distinguished without gene expression profiling. We hypothesise that image analysis is a cost-effective tool to associate complex features of tissue organisation with molecular and outcome data and to resolve unclassifiable or heterogeneous cases. In this study, we present an image-based approach to predict CRC CMS from standard H&E sections using deep learning. DESIGN:Training and evaluation of a neural network were performed using a total of n=1206 tissue sections with comprehensive multi-omic data from three independent datasets (training on FOCUS trial, n=278 patients; test on rectal cancer biopsies, GRAMPIAN cohort, n=144 patients; and The Cancer Genome Atlas (TCGA), n=430 patients). Ground truth CMS calls were ascertained by matching random forest and single sample predictions from CMS classifier. RESULTS:Image-based CMS (imCMS) accurately classified slides in unseen datasets from TCGA (n=431 slides, AUC)=0.84) and rectal cancer biopsies (n=265 slides, AUC=0.85). imCMS spatially resolved intratumoural heterogeneity and provided secondary calls correlating with bioinformatic prediction from molecular data. imCMS classified samples previously unclassifiable by RNA expression profiling, reproduced the expected correlations with genomic and epigenetic alterations and showed similar prognostic associations as transcriptomic CMS. CONCLUSION:This study shows that a prediction of RNA expression classifiers can be made from H&E images, opening the door to simple, cheap and reliable biological stratification within routine workflows.
Clinically applicable histopathological diagnosis system for gastric cancer detection using deep learning.
Song Zhigang,Zou Shuangmei,Zhou Weixun,Huang Yong,Shao Liwei,Yuan Jing,Gou Xiangnan,Jin Wei,Wang Zhanbo,Chen Xin,Ding Xiaohui,Liu Jinhong,Yu Chunkai,Ku Calvin,Liu Cancheng,Sun Zhuo,Xu Gang,Wang Yuefeng,Zhang Xiaoqing,Wang Dandan,Wang Shuhao,Xu Wei,Davis Richard C,Shi Huaiyin
The early detection and accurate histopathological diagnosis of gastric cancer increase the chances of successful treatment. The worldwide shortage of pathologists offers a unique opportunity for the use of artificial intelligence assistance systems to alleviate the workload and increase diagnostic accuracy. Here, we report a clinically applicable system developed at the Chinese PLA General Hospital, China, using a deep convolutional neural network trained with 2,123 pixel-level annotated H&E-stained whole slide images. The model achieves a sensitivity near 100% and an average specificity of 80.6% on a real-world test dataset with 3,212 whole slide images digitalized by three scanners. We show that the system could aid pathologists in improving diagnostic accuracy and preventing misdiagnoses. Moreover, we demonstrate that our system performs robustly with 1,582 whole slide images from two other medical centres. Our study suggests the feasibility and benefits of using histopathological artificial intelligence assistance systems in routine practice scenarios.
Intelligent image-based deformation-assisted cell sorting with molecular specificity.
Nawaz Ahmad Ahsan,Urbanska Marta,Herbig Maik,Nötzel Martin,Kräter Martin,Rosendahl Philipp,Herold Christoph,Toepfner Nicole,Kubánková Markéta,Goswami Ruchi,Abuhattum Shada,Reichel Felix,Müller Paul,Taubenberger Anna,Girardo Salvatore,Jacobi Angela,Guck Jochen
Although label-free cell sorting is desirable for providing pristine cells for further analysis or use, current approaches lack molecular specificity and speed. Here, we combine real-time fluorescence and deformability cytometry with sorting based on standing surface acoustic waves and transfer molecular specificity to image-based sorting using an efficient deep neural network. In addition to general performance, we demonstrate the utility of this method by sorting neutrophils from whole blood without labels.
Rapid vessel segmentation and reconstruction of head and neck angiograms using 3D convolutional neural network.
Fu Fan,Wei Jianyong,Zhang Miao,Yu Fan,Xiao Yueting,Rong Dongdong,Shan Yi,Li Yan,Zhao Cheng,Liao Fangzhou,Yang Zhenghan,Li Yuehua,Chen Yingmin,Wang Ximing,Lu Jie
The computed tomography angiography (CTA) postprocessing manually recognized by technologists is extremely labor intensive and error prone. We propose an artificial intelligence reconstruction system supported by an optimized physiological anatomical-based 3D convolutional neural network that can automatically achieve CTA reconstruction in healthcare services. This system is trained and tested with 18,766 head and neck CTA scans from 5 tertiary hospitals in China collected between June 2017 and November 2018. The overall reconstruction accuracy of the independent testing dataset is 0.931. It is clinically applicable due to its consistency with manually processed images, which achieves a qualification rate of 92.1%. This system reduces the time consumed from 14.22 ± 3.64 min to 4.94 ± 0.36 min, the number of clicks from 115.87 ± 25.9 to 4 and the labor force from 3 to 1 technologist after five months application. Thus, the system facilitates clinical workflows and provides an opportunity for clinical technologists to improve humanistic patient care.
Utilisation of artificial intelligence for the development of an EUS-convolutional neural network model trained to enhance the diagnosis of autoimmune pancreatitis.
Marya Neil B,Powers Patrick D,Chari Suresh T,Gleeson Ferga C,Leggett Cadman L,Abu Dayyeh Barham K,Chandrasekhara Vinay,Iyer Prasad G,Majumder Shounak,Pearson Randall K,Petersen Bret T,Rajan Elizabeth,Sawas Tarek,Storm Andrew C,Vege Santhi S,Chen Shigao,Long Zaiyang,Hough David M,Mara Kristin,Levy Michael J
OBJECTIVE:The diagnosis of autoimmune pancreatitis (AIP) is challenging. Sonographic and cross-sectional imaging findings of AIP closely mimic pancreatic ductal adenocarcinoma (PDAC) and techniques for tissue sampling of AIP are suboptimal. These limitations often result in delayed or failed diagnosis, which negatively impact patient management and outcomes. This study aimed to create an endoscopic ultrasound (EUS)-based convolutional neural network (CNN) model trained to differentiate AIP from PDAC, chronic pancreatitis (CP) and normal pancreas (NP), with sufficient performance to analyse EUS video in real time. DESIGN:A database of still image and video data obtained from EUS examinations of cases of AIP, PDAC, CP and NP was used to develop a CNN. Occlusion heatmap analysis was used to identify sonographic features the CNN valued when differentiating AIP from PDAC. RESULTS:From 583 patients (146 AIP, 292 PDAC, 72 CP and 73 NP), a total of 1 174 461 unique EUS images were extracted. For video data, the CNN processed 955 EUS frames per second and was: 99% sensitive, 98% specific for distinguishing AIP from NP; 94% sensitive, 71% specific for distinguishing AIP from CP; 90% sensitive, 93% specific for distinguishing AIP from PDAC; and 90% sensitive, 85% specific for distinguishing AIP from all studied conditions (ie, PDAC, CP and NP). CONCLUSION:The developed EUS-CNN model accurately differentiated AIP from PDAC and benign pancreatic conditions, thereby offering the capability of earlier and more accurate diagnosis. Use of this model offers the potential for more timely and appropriate patient care and improved outcome.
Development and Validation of a Deep Neural Network for Accurate Evaluation of Endoscopic Images From Patients With Ulcerative Colitis.
Takenaka Kento,Ohtsuka Kazuo,Fujii Toshimitsu,Negi Mariko,Suzuki Kohei,Shimizu Hiromichi,Oshima Shiori,Akiyama Shintaro,Motobayashi Maiko,Nagahori Masakazu,Saito Eiko,Matsuoka Katsuyoshi,Watanabe Mamoru
BACKGROUND & AIMS:There are intra- and interobserver variations in endoscopic assessment of ulcerative colitis (UC) and biopsies are often collected for histologic evaluation. We sought to develop a deep neural network system for consistent, objective, and real-time analysis of endoscopic images from patients with UC. METHODS:We constructed the deep neural network for evaluation of UC (DNUC) algorithm using 40,758 images of colonoscopies and 6885 biopsy results from 2012 patients with UC who underwent colonoscopy from January 2014 through March 2018 at a single center in Japan (the training set). We validated the accuracy of the DNUC algorithm in a prospective study of 875 patients with UC who underwent colonoscopy from April 2018 through April 2019, with 4187 endoscopic images and 4104 biopsy specimens. Endoscopic remission was defined as a UC endoscopic index of severity score of 0; histologic remission was defined as a Geboes score of 3 points or less. RESULTS:In the prospective study, the DNUC identified patients with endoscopic remission with 90.1% accuracy (95% confidence interval [CI] 89.2%-90.9%) and a kappa coefficient of 0.798 (95% CI 0.780-0.814), using findings reported by endoscopists as the reference standard. The intraclass correlation coefficient between the DNUC and the endoscopists for UC endoscopic index of severity scoring was 0.917 (95% CI 0.911-0.921). The DNUC identified patients in histologic remission with 92.9% accuracy (95% CI 92.1%-93.7%); the kappa coefficient between the DNUC and the biopsy result was 0.859 (95% CI 0.841-0.875). CONCLUSIONS:We developed a deep neural network for evaluation of endoscopic images from patients with UC that identified those in endoscopic remission with 90.1% accuracy and histologic remission with 92.9% accuracy. The DNUC can therefore identify patients in remission without the need for mucosal biopsy collection and analysis. Trial number: UMIN000031430.
Spike Encoding with Optic Sensory Neurons Enable a Pulse Coupled Neural Network for Ultraviolet Image Segmentation.
Wu Quantan,Dang Bingjie,Lu Congyan,Xu Guangwei,Yang Guanhua,Wang Jiawei,Chuai Xichen,Lu Nianduan,Geng Di,Wang Hong,Li Ling
Drawing inspiration from biology, neuromorphic systems are of great interest in direct interaction and efficient processing of analogue signals in the real world and could be promising for the development of smart sensors. Here, we demonstrate an artificial sensory neuron consisting of an InGaZnO (IGZO)-based optical sensor and NbO-based oscillation neuron in series, which can simultaneously sense the optical information even beyond the visible light region and encode them into electrical impulses. Such artificial vision sensory neurons can convey visual information in a parallel manner analogous to biological vision systems, and the output spikes can be effectively processed by a pulse coupled neural network, demonstrating the capability of image segmentation out of a complex background. This study could facilitate the construction of artificial visual systems and pave the way for the development of light-driven neurorobotics, bioinspired optoelectronics, and neuromorphic computing.
Automated deep-neural-network surveillance of cranial images for acute neurologic events.
Titano Joseph J,Badgeley Marcus,Schefflein Javin,Pain Margaret,Su Andres,Cai Michael,Swinburne Nathaniel,Zech John,Kim Jun,Bederson Joshua,Mocco J,Drayer Burton,Lehar Joseph,Cho Samuel,Costa Anthony,Oermann Eric K
Rapid diagnosis and treatment of acute neurological illnesses such as stroke, hemorrhage, and hydrocephalus are critical to achieving positive outcomes and preserving neurologic function-'time is brain'. Although these disorders are often recognizable by their symptoms, the critical means of their diagnosis is rapid imaging. Computer-aided surveillance of acute neurologic events in cranial imaging has the potential to triage radiology workflow, thus decreasing time to treatment and improving outcomes. Substantial clinical work has focused on computer-assisted diagnosis (CAD), whereas technical work in volumetric image analysis has focused primarily on segmentation. 3D convolutional neural networks (3D-CNNs) have primarily been used for supervised classification on 3D modeling and light detection and ranging (LiDAR) data. Here, we demonstrate a 3D-CNN architecture that performs weakly supervised classification to screen head CT images for acute neurologic events. Features were automatically learned from a clinical radiology dataset comprising 37,236 head CTs and were annotated with a semisupervised natural-language processing (NLP) framework. We demonstrate the effectiveness of our approach to triage radiology workflow and accelerate the time to diagnosis from minutes to seconds through a randomized, double-blinded, prospective trial in a simulated clinical environment.
Automatically Designing CNN Architectures Using the Genetic Algorithm for Image Classification.
Sun Yanan,Xue Bing,Zhang Mengjie,Yen Gary G,Lv Jiancheng
IEEE transactions on cybernetics
Convolutional neural networks (CNNs) have gained remarkable success on many image classification tasks in recent years. However, the performance of CNNs highly relies upon their architectures. For the most state-of-the-art CNNs, their architectures are often manually designed with expertise in both CNNs and the investigated problems. Therefore, it is difficult for users, who have no extended expertise in CNNs, to design optimal CNN architectures for their own image classification problems of interest. In this article, we propose an automatic CNN architecture design method by using genetic algorithms, to effectively address the image classification tasks. The most merit of the proposed algorithm remains in its "automatic" characteristic that users do not need domain knowledge of CNNs when using the proposed algorithm, while they can still obtain a promising CNN architecture for the given images. The proposed algorithm is validated on widely used benchmark image classification datasets, compared to the state-of-the-art peer competitors covering eight manually designed CNNs, seven automatic + manually tuning, and five automatic CNN architecture design algorithms. The experimental results indicate the proposed algorithm outperforms the existing automatic CNN architecture design algorithms in terms of classification accuracy, parameter numbers, and consumed computational resources. The proposed algorithm also shows the very comparable classification accuracy to the best one from manually designed and automatic + manually tuning CNNs, while consuming fewer computational resources.
Automated design of a convolutional neural network with multi-scale filters for cost-efficient seismic data classification.
Geng Zhi,Wang Yanfei
Geoscientists mainly identify subsurface geologic features using exploration-derived seismic data. Classification or segmentation of 2D/3D seismic images commonly relies on conventional deep learning methods for image recognition. However, complex reflections of seismic waves tend to form high-dimensional and multi-scale signals, making traditional convolutional neural networks (CNNs) computationally costly. Here we propose a highly efficient and resource-saving CNN architecture (SeismicPatchNet) with topological modules and multi-scale-feature fusion units for classifying seismic data, which was discovered by an automated data-driven search strategy. The storage volume of the architecture parameters (0.73 M) is only ~2.7 MB, ~0.5% of the well-known VGG-16 architecture. SeismicPatchNet predicts nearly 18 times faster than ResNet-50 and shows an overwhelming advantage in identifying Bottom Simulating Reflection (BSR), an indicator of marine gas-hydrate resources. Saliency mapping demonstrated that our architecture captured key features well. These results suggest the prospect of end-to-end interpretation of multiple seismic datasets at extremely low computational cost.
Deep-learning algorithm helps to standardise ATS/ERS spirometric acceptability and usability criteria.
Das Nilakash,Verstraete Kenneth,Stanojevic Sanja,Topalovic Marko,Aerts Jean-Marie,Janssens Wim
The European respiratory journal
RATIONALE:While American Thoracic Society (ATS)/European Respiratory Society (ERS) quality control criteria for spirometry include several quantitative limits, it also requires manual visual inspection. The current approach is time consuming and leads to high intertechnician variability. We propose a deep-learning approach called convolutional neural network (CNN), to standardise spirometric manoeuvre acceptability and usability. METHODS AND METHODS:In 36 873 curves from the National Health and Nutritional Examination Survey USA 2011-2012, technicians labelled 54% of curves as meeting ATS/ERS 2005 acceptability criteria with satisfactory start and end of test, but identified 93% of curves with a usable forced expiratory volume in 1 s. We processed raw data into images of maximal expiratory flow-volume curve (MEFVC), calculated ATS/ERS quantifiable criteria and developed CNNs to determine manoeuvre acceptability and usability on 90% of the curves. The models were tested on the remaining 10% of curves. We calculated Shapley values to interpret the models. RESULTS:In the test set (n=3738), CNN showed an accuracy of 87% for acceptability and 92% for usability, with the latter demonstrating a high sensitivity (92%) and specificity (96%). They were significantly superior (p<0.0001) to ATS/ERS quantifiable rule-based models. Shapley interpretation revealed MEFVC<1 s (MEFVC pattern within first second of exhalation) and plateau in volume-time were most important in determining acceptability, while MEFVC<1 s entirely determined usability. CONCLUSION:The CNNs identified relevant attributes in spirometric curves to standardise ATS/ERS manoeuvre acceptability and usability recommendations, and further provides individual manoeuvre feedback. Our algorithm combines the visual experience of skilled technicians and ATS/ERS quantitative rules in automating the critical phase of spirometry quality control.
Near real-time intraoperative brain tumor diagnosis using stimulated Raman histology and deep neural networks.
Hollon Todd C,Pandian Balaji,Adapa Arjun R,Urias Esteban,Save Akshay V,Khalsa Siri Sahib S,Eichberg Daniel G,D'Amico Randy S,Farooq Zia U,Lewis Spencer,Petridis Petros D,Marie Tamara,Shah Ashish H,Garton Hugh J L,Maher Cormac O,Heth Jason A,McKean Erin L,Sullivan Stephen E,Hervey-Jumper Shawn L,Patil Parag G,Thompson B Gregory,Sagher Oren,McKhann Guy M,Komotar Ricardo J,Ivan Michael E,Snuderl Matija,Otten Marc L,Johnson Timothy D,Sisti Michael B,Bruce Jeffrey N,Muraszko Karin M,Trautman Jay,Freudiger Christian W,Canoll Peter,Lee Honglak,Camelo-Piragua Sandra,Orringer Daniel A
Intraoperative diagnosis is essential for providing safe and effective care during cancer surgery. The existing workflow for intraoperative diagnosis based on hematoxylin and eosin staining of processed tissue is time, resource and labor intensive. Moreover, interpretation of intraoperative histologic images is dependent on a contracting, unevenly distributed, pathology workforce. In the present study, we report a parallel workflow that combines stimulated Raman histology (SRH), a label-free optical imaging method and deep convolutional neural networks (CNNs) to predict diagnosis at the bedside in near real-time in an automated fashion. Specifically, our CNNs, trained on over 2.5 million SRH images, predict brain tumor diagnosis in the operating room in under 150 s, an order of magnitude faster than conventional techniques (for example, 20-30 min). In a multicenter, prospective clinical trial (n = 278), we demonstrated that CNN-based diagnosis of SRH images was noninferior to pathologist-based interpretation of conventional histologic images (overall accuracy, 94.6% versus 93.9%). Our CNNs learned a hierarchy of recognizable histologic feature representations to classify the major histopathologic classes of brain tumors. In addition, we implemented a semantic segmentation method to identify tumor-infiltrated diagnostic regions within SRH images. These results demonstrate how intraoperative cancer diagnosis can be streamlined, creating a complementary pathway for tissue diagnosis that is independent of a traditional pathology laboratory.
Hierarchical Scene Parsing by Weakly Supervised Learning with Image Descriptions.
Zhang Ruimao,Lin Liang,Wang Guangrun,Wang Meng,Zuo Wangmeng
IEEE transactions on pattern analysis and machine intelligence
This paper investigates a fundamental problem of scene understanding: how to parse a scene image into a structured configuration (i.e., a semantic object hierarchy with object interaction relations). We propose a deep architecture consisting of two networks: i) a convolutional neural network (CNN) extracting the image representation for pixel-wise object labeling and ii) a recursive neural network (RsNN) discovering the hierarchical object structure and the inter-object relations. Rather than relying on elaborative annotations (e.g., manually labeled semantic maps and relations), we train our deep model in a weakly-supervised learning manner by leveraging the descriptive sentences of the training images. Specifically, we decompose each sentence into a semantic tree consisting of nouns and verb phrases, and apply these tree structures to discover the configurations of the training images. Once these scene configurations are determined, then the parameters of both the CNN and RsNN are updated accordingly by back propagation. The entire model training is accomplished through an Expectation-Maximization method. Extensive experiments show that our model is capable of producing meaningful scene configurations and achieving more favorable scene labeling results on two benchmarks (i.e., PASCAL VOC 2012 and SYSU-Scenes) compared with other state-of-the-art weakly-supervised deep learning methods. In particular, SYSU-Scenes contains more than 5,000 scene images with their semantic sentence descriptions, which is created by us for advancing research on scene parsing.
Trunk-Branch Ensemble Convolutional Neural Networks for Video-Based Face Recognition.
Ding Changxing,Tao Dacheng
IEEE transactions on pattern analysis and machine intelligence
Human faces in surveillance videos often suffer from severe image blur, dramatic pose variations, and occlusion. In this paper, we propose a comprehensive framework based on Convolutional Neural Networks (CNN) to overcome challenges in video-based face recognition (VFR). First, to learn blur-robust face representations, we artificially blur training data composed of clear still images to account for a shortfall in real-world video training data. Using training data composed of both still images and artificially blurred data, CNN is encouraged to learn blur-insensitive features automatically. Second, to enhance robustness of CNN features to pose variations and occlusion, we propose a Trunk-Branch Ensemble CNN model (TBE-CNN), which extracts complementary information from holistic face images and patches cropped around facial components. TBE-CNN is an end-to-end model that extracts features efficiently by sharing the low- and middle-level convolutional layers between the trunk and branch networks. Third, to further promote the discriminative power of the representations learnt by TBE-CNN, we propose an improved triplet loss function. Systematic experiments justify the effectiveness of the proposed techniques. Most impressively, TBE-CNN achieves state-of-the-art performance on three popular video face databases: PaSC, COX Face, and YouTube Faces. With the proposed techniques, we also obtain the first place in the BTAS 2016 Video Person Recognition Evaluation.
Deep learning-based cross-classifications reveal conserved spatial behaviors within tumor histological images.
Noorbakhsh Javad,Farahmand Saman,Foroughi Pour Ali,Namburi Sandeep,Caruana Dennis,Rimm David,Soltanieh-Ha Mohammad,Zarringhalam Kourosh,Chuang Jeffrey H
Histopathological images are a rich but incompletely explored data type for studying cancer. Manual inspection is time consuming, making it challenging to use for image data mining. Here we show that convolutional neural networks (CNNs) can be systematically applied across cancer types, enabling comparisons to reveal shared spatial behaviors. We develop CNN architectures to analyze 27,815 hematoxylin and eosin scanned images from The Cancer Genome Atlas for tumor/normal, cancer subtype, and mutation classification. Our CNNs are able to classify TCGA pathologist-annotated tumor/normal status of whole slide images (WSIs) in 19 cancer types with consistently high AUCs (0.995 ± 0.008), as well as subtypes with lower but significant accuracy (AUC 0.87 ± 0.1). Remarkably, tumor/normal CNNs trained on one tissue are effective in others (AUC 0.88 ± 0.11), with classifier relationships also recapitulating known adenocarcinoma, carcinoma, and developmental biology. Moreover, classifier comparisons reveal intra-slide spatial similarities, with an average tile-level correlation of 0.45 ± 0.16 between classifier pairs. Breast cancers, bladder cancers, and uterine cancers have spatial patterns that are particularly easy to detect, suggesting these cancers can be canonical types for image analysis. Patterns for TP53 mutations can also be detected, with WSI self- and cross-tissue AUCs ranging from 0.65-0.80. Finally, we comparatively evaluate CNNs on 170 breast and colon cancer images with pathologist-annotated nuclei, finding that both cellular and intercellular regions contribute to CNN accuracy. These results demonstrate the power of CNNs not only for histopathological classification, but also for cross-comparisons to reveal conserved spatial behaviors across tumors.
Improved Accuracy in Optical Diagnosis of Colorectal Polyps Using Convolutional Neural Networks with Visual Explanations.
Jin Eun Hyo,Lee Dongheon,Bae Jung Ho,Kang Hae Yeon,Kwak Min-Sun,Seo Ji Yeon,Yang Jong In,Yang Sun Young,Lim Seon Hee,Yim Jeong Yoon,Lim Joo Hyun,Chung Goh Eun,Chung Su Jin,Choi Ji Min,Han Yoo Min,Kang Seung Joo,Lee Jooyoung,Chan Kim Hee,Kim Joo Sung
BACKGROUND & AIMS:Narrow-band imaging (NBI) can be used to determine whether colorectal polyps are adenomatous or hyperplastic. We investigated whether an artificial intelligence (AI) system can increase the accuracy of characterizations of polyps by endoscopists of different skill levels. METHODS:We developed convolutional neural networks (CNNs) for evaluation of diminutive colorectal polyps, based on efficient neural architecture searches via parameter sharing with augmentation using NBIs of diminutive (≤5 mm) polyps, collected from October 2015 through October 2017 at the Seoul National University Hospital, Healthcare System Gangnam Center (training set). We trained the CNN using images from 1100 adenomatous polyps and 1050 hyperplastic polyps from 1379 patients. We then tested the system using 300 images of 180 adenomatous polyps and 120 hyperplastic polyps, obtained from January 2018 to May 2019. We compared the accuracy of 22 endoscopists of different skill levels (7 novices, 4 experts, and 11 NBI-trained experts) vs the CNN in evaluation of images (adenomatous vs hyperplastic) from 180 adenomatous and 120 hyperplastic polyps. The endoscopists then evaluated the polyp images with knowledge of the CNN-processed results. We conducted mixed-effect logistic and linear regression analyses to determine the effects of AI assistance on the accuracy of analysis of diminutive colorectal polyps by endoscopists (primary outcome). RESULTS:The CNN distinguished adenomatous vs hyperplastic diminutive polyps with 86.7% accuracy, based on histologic analysis as the reference standard. Endoscopists distinguished adenomatous vs hyperplastic diminutive polyps with 82.5% overall accuracy (novices, 73.8% accuracy; experts, 83.8% accuracy; and NBI-trained experts, 87.6% accuracy). With knowledge of the CNN-processed results, the overall accuracy of the endoscopists increased to 88.5% (P < .05). With knowledge of the CNN-processed results, the accuracy of novice endoscopists increased to 85.6% (P < .05). The CNN-processed results significantly reduced endoscopist time of diagnosis (from 3.92 to 3.37 seconds per polyp, P = .042). CONCLUSIONS:We developed a CNN that significantly increases the accuracy of evaluation of diminutive colorectal polyps (as adenomatous vs hyperplastic) and reduces the time of diagnosis by endoscopists. This AI assistance system significantly increased the accuracy of analysis by novice endoscopists, who achieved near-expert levels of accuracy without extra training. The CNN assistance system can reduce the skill-level dependence of endoscopists and costs.
Applying cascaded convolutional neural network design further enhances automatic scoring of arthritis disease activity on ultrasound images from rheumatoid arthritis patients.
Christensen Anders Bossel Holst,Just Søren Andreas,Andersen Jakob Kristian Holm,Savarimuthu Thiusius Rajeeth
Annals of the rheumatic diseases
OBJECTIVES:We have previously shown that neural network technology can be used for scoring arthritis disease activity in ultrasound images from rheumatoid arthritis (RA) patients, giving scores according to the EULAR-OMERACT grading system. We have now further developed the architecture of this neural network and can here present a new idea applying cascaded convolutional neural network (CNN) design with even better results. We evaluate the generalisability of this method on unseen data, comparing the CNN with an expert rheumatologist. METHODS:The images were graded by an expert rheumatologist according to the EULAR-OMERACT synovitis scoring system. CNNs were systematically trained to find the best configuration. The algorithms were evaluated on a separate test data set and compared with the gradings of an expert rheumatologist on a per-joint basis using a Kappa statistic, and on a per-patient basis using a Wilcoxon signed-rank test. RESULTS:With 1678 images available for training and 322 images for testing the model, it achieved an overall four-class accuracy of 83.9%. On a per-patient level, there was no significant difference between the classifications of the model and of a human expert (p=0.85). Our original CNN had a four-class accuracy of 75.0%. CONCLUSIONS:Using a new network architecture we have further enhanced the algorithm and have shown strong agreement with an expert rheumatologist on a per-joint basis and on a per-patient basis. This emphasises the potential of using CNNs with this architecture as a strong assistive tool for the objective assessment of disease activity of RA patients.
MonoCap: Monocular Human Motion Capture using a CNN Coupled with a Geometric Prior.
Zhou Xiaowei,Zhu Menglong,Pavlakos Georgios,Leonardos Spyridon,Derpanis Konstantinos G,Daniilidis Kostas
IEEE transactions on pattern analysis and machine intelligence
Recovering 3D full-body human pose is a challenging problem with many applications. It has been successfully addressed by motion capture systems with body worn markers and multiple cameras. In this paper, we address the more challenging case of not only using a single camera but also not leveraging markers: going directly from 2D appearance to 3D geometry. Deep learning approaches have shown remarkable abilities to discriminatively learn 2D appearance features. The missing piece is how to integrate 2D, 3D, and temporal information to recover 3D geometry and account for the uncertainties arising from the discriminative model. We introduce a novel approach that treats 2D joint locations as latent variables whose uncertainty distributions are given by a deep fully convolutional neural network. The unknown 3D poses are modeled by a sparse representation and the 3D parameter estimates are realized via an Expectation-Maximization algorithm, where it is shown that the 2D joint location uncertainties can be conveniently marginalized out during inference. Extensive evaluation on benchmark datasets shows that the proposed approach achieves greater accuracy over state-of-the-art baselines. Notably, the proposed approach does not require synchronized 2D-3D data for training and is applicable to "in-the-wild" images, which is demonstrated with the MPII dataset.
Gastroenterologist-Level Identification of Small-Bowel Diseases and Normal Variants by Capsule Endoscopy Using a Deep-Learning Model.
Ding Zhen,Shi Huiying,Zhang Hao,Meng Lingjun,Fan Mengke,Han Chaoqun,Zhang Kun,Ming Fanhua,Xie Xiaoping,Liu Hao,Liu Jun,Lin Rong,Hou Xiaohua
BACKGROUND & AIMS:Capsule endoscopy has revolutionized investigation of the small bowel. However, this technique produces a video that is 8-10 hours long, so analysis is time consuming for gastroenterologists. Deep convolutional neural networks (CNNs) can recognize specific images among a large variety. We aimed to develop a CNN-based algorithm to assist in the evaluation of small bowel capsule endoscopy (SB-CE) images. METHODS:We collected 113,426,569 images from 6970 patients who had SB-CE at 77 medical centers from July 2016 through July 2018. A CNN-based auxiliary reading model was trained to differentiate abnormal from normal images using 158,235 SB-CE images from 1970 patients. Images were categorized as normal, inflammation, ulcer, polyps, lymphangiectasia, bleeding, vascular disease, protruding lesion, lymphatic follicular hyperplasia, diverticulum, parasite, and other. The model was further validated in 5000 patients (no patient was overlap with the 1970 patients in the training set); the same patients were evaluated by conventional analysis and CNN-based auxiliary analysis by 20 gastroenterologists. If there was agreement in image categorization between the conventional analysis and CNN model, no further evaluation was performed. If there was disagreement between the conventional analysis and CNN model, the gastroenterologists re-evaluated the image to confirm or reject the CNN categorization. RESULTS:In the SB-CE images from the validation set, 4206 abnormalities in 3280 patients were identified after final consensus evaluation. The CNN-based auxiliary model identified abnormalities with 99.88% sensitivity in the per-patient analysis (95% CI, 99.67-99.96) and 99.90% sensitivity in the per-lesion analysis (95% CI, 99.74-99.97). Conventional reading by the gastroenterologists identified abnormalities with 74.57% sensitivity (95% CI, 73.05-76.03) in the per-patient analysis and 76.89% in the per-lesion analysis (95% CI, 75.58-78.15). The mean reading time per patient was 96.6 ± 22.53 minutes by conventional reading and 5.9 ± 2.23 minutes by CNN-based auxiliary reading (P < .001). CONCLUSIONS:We validated the ability of a CNN-based algorithm to identify abnormalities in SB-CE images. The CNN-based auxiliary model identified abnormalities with higher levels of sensitivity and significantly shorter reading times than conventional analysis by gastroenterologists. This algorithm provides an important tool to help gastroenterologists analyze SB-CE images more efficiently and more accurately.
HCP: A Flexible CNN Framework for Multi-label Image Classification.
Wei Yunchao,Xia Wei,Lin Min,Huang Junshi,Ni Bingbing,Dong Jian,Zhao Yao,Yan Shuicheng
IEEE transactions on pattern analysis and machine intelligence
Convolutional Neural Network (CNN) has demonstrated promising performance in single-label image classification tasks. However, how CNN best copes with multi-label images still remains an open problem, mainly due to the complex underlying object layouts and insufficient multi-label training images. In this work, we propose a flexible deep CNN infrastructure, called Hypotheses-CNN-Pooling (HCP), where an arbitrary number of object segment hypotheses are taken as the inputs, then a shared CNN is connected with each hypothesis, and finally the CNN output results from different hypotheses are aggregated with max pooling to produce the ultimate multi-label predictions. Some unique characteristics of this flexible deep CNN infrastructure include: 1) no ground-truth bounding box information is required for training; 2) the whole HCP infrastructure is robust to possibly noisy and/or redundant hypotheses; 3) the shared CNN is flexible and can be well pre-trained with a large-scale single-label image dataset, e.g., ImageNet; and 4) it may naturally output multi-label prediction results. Experimental results on Pascal VOC 2007 and VOC 2012 multi-label image datasets well demonstrate the superiority of the proposed HCP infrastructure over other state-of-the-arts. In particular, the mAP reaches 90.5% by HCP only and 93.2% after the fusion with our complementary result in  based on hand-crafted features on the VOC 2012 dataset.
Triple U-net: Hematoxylin-aware nuclei segmentation with progressive dense feature aggregation.
Zhao Bingchao,Chen Xin,Li Zhi,Yu Zhiwen,Yao Su,Yan Lixu,Wang Yuqian,Liu Zaiyi,Liang Changhong,Han Chu
Medical image analysis
Nuclei segmentation is a vital step for pathological cancer research. It is still an open problem due to some difficulties, such as color inconsistency introduced by non-uniform manual operations, blurry tumor nucleus boundaries and overlapping tumor cells. In this paper, we aim to leverage the unique optical characteristic of H&E staining images that hematoxylin always stains cell nuclei blue, and eosin always stains the extracellular matrix and cytoplasm pink. Therefore, we extract the Hematoxylin component from RGB images by Beer-Lambert's Law. According to the optical attribute, the extracted Hematoxylin component is robust to color inconsistency. With the Hematoxylin component, we propose a Hematoxylin-aware CNN model for nuclei segmentation without the necessity of color normalization. Our proposed network is formulated as a Triple U-net structure which includes an RGB branch, a Hematoxylin branch and a Segmentation branch. Then we propose a novel feature aggregation strategy to allow the network to fuse features progressively and to learn better feature representations from different branches. Extensive experiments are performed to qualitatively and quantitatively evaluate the effectiveness of our proposed method. In the meanwhile, it outperforms state-of-the-art methods on three different nuclei segmentation datasets.
Trophectoderm segmentation in human embryo images via inceptioned U-Net.
Rad Reza Moradi,Saeedi Parvaneh,Au Jason,Havelock Jon
Medical image analysis
Trophectoderm (TE) is one of the main components of a day-5 human embryo (blastocyst) that correlates with the embryo's quality. Precise segmentation of TE is an important step toward achieving automatic human embryo quality assessment based on morphological image features. Automatic segmentation of TE, however, is a challenging task and previous work on this is quite limited. In this paper, four fully convolutional deep models are proposed for accurate segmentation of trophectoderm in microscopic images of the human blastocyst. In addition, a multi-scaled ensembling method is proposed that aggregates five models trained at various scales offering trade-offs between the quantity and quality of the spatial information. Furthermore, synthetic embryo images are generated for the first time to address the lack of data in training deep learning models. These synthetically generated images are proven to be effective to fill the generalization gap in deep learning when limited data is available for training. Experimental results confirm that the proposed models are capable of segmenting TE regions with an average Precision, Recall, Accuracy, Dice Coefficient and Jaccard Index of 83.8%, 90.1%, 96.9%, 86.61% and 76.71%, respectively. Particularly, the proposed Inceptioned U-Net model outperforms state-of-the-art by 10.3% in Accuracy, 9.3% in Dice Coefficient and 13.7% in Jaccard Index. Further experiments are conducted to highlight the effectiveness of the proposed models compared to some recent deep learning based segmentation methods.
Automated Recognition of Regional Wall Motion Abnormalities Through Deep Neural Network Interpretation of Transthoracic Echocardiography.
Huang Mu-Shiang,Wang Chi-Shiang,Chiang Jung-Hsien,Liu Ping-Yen,Tsai Wei-Chuan
BACKGROUND:Automated interpretation of echocardiography by deep neural networks could support clinical reporting and improve efficiency. Whereas previous studies have evaluated spatial relationships using still frame images, we aimed to train and test a deep neural network for video analysis by combining spatial and temporal information, to automate the recognition of left ventricular regional wall motion abnormalities. METHODS:We collected a series of transthoracic echocardiography examinations performed between July 2017 and April 2018 in 2 tertiary care hospitals. Regional wall abnormalities were defined by experienced physiologists and confirmed by trained cardiologists. First, we developed a 3-dimensional convolutional neural network model for view selection ensuring stringent image quality control. Second, a U-net model segmented images to annotate the location of each left ventricular wall. Third, a final 3-dimensional convolutional neural network model evaluated echocardiographic videos from 4 standard views, before and after segmentation, and calculated a wall motion abnormality confidence level (0-1) for each segment. To evaluate model stability, we performed 5-fold cross-validation and external validation. RESULTS:In a series of 10 638 echocardiograms, our view selection model identified 6454 (61%) examinations with sufficient image quality in all standard views. In this training set, 2740 frames were annotated to develop the segmentation model, which achieved a Dice similarity coefficient of 0.756. External validation was performed in 1756 examinations from an independent hospital. A regional wall motion abnormality was observed in 8.9% and 4.9% in the training and external validation datasets, respectively. The final model recognized regional wall motion abnormalities in the cross-validation and external validation datasets with an area under the receiver operating characteristic curve of 0.912 (95% CI, 0.896-0.928) and 0.891 (95% CI, 0.834-0.948), respectively. In the external validation dataset, the sensitivity was 81.8% (95% CI, 73.8%-88.2%), and specificity was 81.6% (95% CI, 80.4%-82.8%). CONCLUSIONS:In echocardiographic examinations of sufficient image quality, it is feasible for deep neural networks to automate the recognition of regional wall motion abnormalities using temporal and spatial information from moving images. Further investigation is required to optimize model performance and evaluate clinical applications.
Holistic decomposition convolution for effective semantic segmentation of medical volume images.
Zeng Guodong,Zheng Guoyan
Medical image analysis
Convolutional Neural Networks (CNNs) have achieved state-of-the-art performance in many different 2D medical image analysis tasks. In clinical practice, however, a large part of the medical imaging data available is in 3D, e.g, magnetic resonance imaging (MRI) data, computed tomography (CT) data and data generated by many other modalities. This has motivated the development of 3D CNNs for volumetric image segmentation in order to benefit from more spatial context. Due to GPU memory restrictions caused by moving to fully 3D, state-of-the-art methods depend on subvolume/patch processing and the size of the input patch is usually small, limiting the incorporation of larger context information for a better performance. In this paper, we propose a novel Holistic Decomposition Convolution (HDC), which learns a number of separate kernels within the same layer and can be regarded as an inverse operation to the previously introduced Dense Upsampling Convolution (DUC), for an effective and efficient semantic segmentation of medical volume images. HDC consists of a periodic down-shuffling operation followed by a conventional 3D convolution. HDC has the advantage of significantly reducing the size of the data for sub-sequential processing while using all the information available in the input irrespective of the down-shuffling factors. We apply HDC directly to the input data, whose output will be used as the input to sub-sequential CNNs. In order to achieve volumetric dense prediction at final output, we need to recover full resolution, which is done by using DUC. We show that both HDC and DUC are network agnostic and can be combined with different CNNs for an improved performance in both training and testing phases. Results obtained from comprehensive experiments conducted on both MRI and CT data of different anatomical regions demonstrate the efficacy of the present approach.