Application of Deep Learning in Medical Imaging Analysis: A Survey

doi:10.11871/jfdc.issn.2096-742X.2019.02.004

Abstract

Abstract:

[Objective] This paper reviews the recent progress of deep learning researches and applications in medical image analysis. [Coverage] Relevant papers were first retrieved by keyword search and then by citation screening. [Methods] Deep learning based on convolutional neural networks is briefly introduced. Then, we review the diagnostic performance of deep learning on medical images in recent years with respect to different types of diseases, such as stroke, pulmonary nodules and bone age estimation. [Results] Deep learning for medical image interpretation has demonstrated advantages in many aspects, including accuracy, speed, stability and scalability. Meanwhile, existing problems may hinder clinical adoption of deep learning, such as dependence on a large amount of labelled data, inconsistent labeling standards, poor generalizability and interpretability of deep learning methods. [Limitations] There may be omissions of the retrieved literature, and it is impossible to compare the performance of the same deep learning model across different diseases. [Conclusions] Powerful artificial intelligence can improve the efficiency and accuracy of image interpretations for radiologists, but artificial intelligence is not perfect. Before being widely adopted in medical image interpretation, deep learning methods need more verification in real applications.

Key words: deep learning, medical imaging, survey

Yu Yizhou, Ma Jiechao, Shi Dejun, Zhou Zhen. Application of Deep Learning in Medical Imaging Analysis: A Survey[J]. Frontiers of Data and Computing, 2019, 1(2): 37-52.

Figures/Tables 6

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Table 1

References 101

[1]	王弈, 李传富 . 人工智能方法在医学影像处理中的研究新进展[J]. 中国医学物理学杂志, 2013(3).
[2]	丘文峰, 鲍苏苏, 彭丰平 , 等. Jolly:基于Python的医学影像处理框架[J]. 现代计算机:专业版, 2010(5):62-67.
[3]	Papademetris X DeLorenzo C, Flossmann S , et al. From medical image computing to computer‐aided intervention: development of a research interface for image-guided navigation[J]. The International Journal of Medical Robotics and Computer Assisted Surgery, 2009,5(2):147-157.
[4]	LeCun Y, Bengio Y, Hinton G . Deep learning[J]. nature, 2015,521(7553):436.
[5]	Lakhani P, Sundaram B . Deep Learning at Chest Radiography: Automated Classification of Pulmonary Tuberculosis by Using Convolutional Neural Networks.[J]. Radiology, 2017: 574-582.
[6]	Zhu W., Liu C., Fan W ., et al. DeepLung: Deep 3D Dual Path Nets for Automated Pulmonary Nodule Detection and Classification. IEEE WACV, 2018.
[7]	张巧丽, 赵地, 迟学斌 . 基于深度学习的医学影像诊断综述[J]. 计算机科学, 2017(S2):11-17.
[8]	夏黎明, 沈坚, 张荣国 , 等. 深度学习技术在医学影像领域的应用[J]. 协和医学杂志, 2018(1):10-14.
[9]	Geert Litjens, Thijs Kooi , et al. A survey on deep learning in medical image analysis[J]. Medical Image Analysis, 2017: 60-88.
[10]	Dalal Navneet, and Bill Triggs . Histograms of oriented gradients for human detection[C]. Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. Vol. 1. IEEE, 2005.
[11]	Lowe David G . Distinctive image features from scale-invariant keypoints[J].International journal of computer vision 60. 2(2004):91-110.
[12]	Smistad E, Falch T L, Bozorgi M , et al. Medical image segmentation on GPUs - A comprehensive review[J]. Medical Image Analysis, 2015,20(1):1-18.
[13]	Esteva Andre , et al. Dermatologist-level classification of skin cancer with deep neural networks[J].Nature 542. 7639(2017):115.
[14]	Lo S.-C., Lou S.-L., Lin J.-S., Freedman M.T., Chien M.V., Mun S.K ., 1995. Artificial convolution neural network techniques and applications for lung nodule detection. IEEE Trans. Med. Imaging 14, 711-718. doi: 10.1109/42.476112.
[15]	Ren S, He K, Girshick R, et al. Faster r-cnn: Towards real-time object detection with region proposal networks [C]// Advances in neural information processing systems, 2015: 91-99.
[16]	Liu W, Anguelov D, Erhan D, et al. Ssd: Single shot multibox detector [C]// European conference on computer vision. Springer, Cham, 2016: 21-37.
[17]	Redmon J, Divvala S, Girshick R, et al. You only look once: Unified, real-time object detection [C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2016: 779-788.
[18]	Kong B, Zhan Y, Shin M C, et al. Recognizing End-Diastole and End-Systole Frames via Deep Temporal Regression Network[C]. medical image computing and computer assisted intervention, 2016: 264-272.
[19]	Sundermeyer M, Schlüter R, Ney H . LSTM neural networks for language modeling [C]// Thirteenth annual conference of the international speech communication association, 2012.
[20]	Yan K, Bagheri M, Summers R M . 3d context enhanced region-based convolutional neural network for end-to-end lesion detection [C]// International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2018: 511-519.
[21]	Ronneberger O, Fischer P, Brox T . U-net: Convolutional networks for biomedical image segmentation [C]// International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015: 234-241.
[22]	Milletari F, Navab N, Ahmadi S A . V-net: Fully convolutional neural networks for volumetric medical image segmentation [C]// 2016 Fourth International Conference on 3D Vision (3DV). IEEE, 2016: 565-571.
[23]	Çiçek Ö, Abdulkadir A, Lienkamp S S, et al. 3D U-Net: learning dense volumetric segmentation from sparse annotation [C]// International conference on medical image computing and computer-assisted intervention. Springer, Cham, 2016: 424-432.
[24]	Rueckert D, Schnabel J A . Medical image registration[M] //Biomedical image processing. Springer, Berlin, Heidelberg, 2010: 131-154.
[25]	Simonovsky M, Gutierrezbecker B, Mateus D, et al. A Deep Metric for Multimodal Registration[C]. medical image computing and computer assisted intervention, 2016: 10-18.
[26]	Miao S, Wang Z J, Liao R , et al. A CNN Regression Approach for Real-Time 2D/3D Registration[J]. IEEE Transactions on Medical Imaging, 2016,35(5):1352-1363.
[27]	Jaderberg M, Simonyan K, Zisserman A, et al. Spatial transformer networks[C]. neural information processing systems, 2015: 2017-2025.
[28]	Haskins G, Kruger U, Yan P . Deep Learning in Medical Image Registration: A Survey[J]. arXiv:1903.02026. 2019.
[29]	Krupinski J, Kaluza J, Kumar P , et al. Role of angiogenesis in patients with cerebral ischemic stroke[J]. Stroke, 1994,25(9):1794-1798.
[30]	Hu X De Silva T M, Chen J , et al. Cerebral Vascular Disease and Neurovascular Injury in Ischemic Stroke[J]. Circulation Research, 2017,120(3):449-471.
[31]	Chilamkurthy S, Ghosh R, Tanamala S , et al. Development and Validation of Deep Learning Algorithms for Detection of Critical Findings in Head CT Scans[J]. arXiv preprint arXiv:1803.05854, 2018.
[32]	Chang P D, Kuoy E, Grinband J , et al. Hybrid 3D/2D convolutional neural network for hemorrhage evaluation on head CT[J]. American Journal of Neuroradiology, 2018,39(9):1609-1616.
[33]	Zhang R, Zhao L, Lou W , et al. Automatic Segmentation of Acute Ischemic Stroke from DWI using 3D Fully Convolutional DenseNets[J]. IEEE Transactions on Medical Imaging, 2018: 1-1.
[34]	Liu Q, Cheng X, Zhang Q, Zhou C, Li X, Wang S, Kongming Liang, Guangming Lu , Automatic Segmentation for Acute Ischemic Stroke from DWI Using Deep Convolutional Neural Networks[J]. European Society of Radiology (2019).
[35]	Moccia S, Momi E D, Hadji S E , et al. Blood vessel segmentation algorithms — Review of methods, datasets and evaluation metrics[J]. Computer Methods & Programs in Biomedicine, 2018,158:71-91.
[36]	Tetteh G, Efremov V, Forkert N D , et al. DeepVesselNet: Vessel Segmentation, Centerline Prediction, and Bifurcation Detection in 3-D Angiographic Volumes[J]. 2018.
[37]	Fong D S, Aiello L P, Gardner T W , et al. Retinopathy in Diabetes[J]. Diabetes Care, 2004,27(1).
[38]	Jones S, Edwards R T . Diabetic retinopathy screening: a systematic review of the economic evidence[J]. Diabet Med, 2010,27(3):249-256.
[39]	Nielsen K B, Lautrup M L Andersen J K H , et al. Deep Learning-based Algorithms in Screening of Diabetic Retinopathy: A Systematic Review of Diagnostic Performance.[J]. Ophthalmology Retina, 2018.
[40]	Quellec G, Charriere K, Boudi Y , et al. Deep image mining for diabetic retinopathy screening[J]. Medical Image Analysis, 2017,39:178-193.
[41]	Takahashi H, Tampo H, Arai Y , et al. Applying artificial intelligence to disease staging: deep learning for improved staging of diabetic retinopathy[J]. PLoS ONE, 2017,12:e0179790.
[42]	Jiang Z., Zhang H., Wang Y ., et al. Retinal blood vessel segmentation using fully convolutional network with transfer learning[J]. Computerized Medical Imaging and Graphics, 2018,68:1-15.
[43]	Fraz M M, Remagnino P, Hoppe A , et al. Blood vessel segmentation methodologies in retinal images-a survey[J]. Computer methods and programs in biomedicine, 2012,108(1):407-433.
[44]	Xiao Z T, Zhang X P, Geng L , et al. Automatic non-proliferative diabetic retinopathy screening system based on color fundus image[J]. BioMedical Engineering OnLine, 2017,16:122.
[45]	Prentašić P., Lončarić S . Detection of exudates in fundus photographs using deep neural networks and anatomical landmark detection fusion[J]. Computer Methods and Programs in Biomedicine, 2016,137:281-292.
[46]	Bray F I, Ferlay J, Soerjomataram I , et al. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries[J]. CA: A Cancer Journal for Clinicians, 2018,68(6):394-424.
[47]	Siegel R L, Miller K D, Ahmedin Jemal . Cancer statistics, 2018[J]. Ca A Cancer Journal for Clinicians, 2018,68(1):11.
[48]	Van Ginneken B, Armato III S G, de Hoop B , et al. Comparing and combining algorithms for computer-aided detection of pulmonary nodules in computed tomography scans: the ANODE09 study[J]. Medical image analysis, 2010,14(6):707-722.
[49]	Setio A A A, Jacobs C, Gelderblom J , et al. Automatic detection of large pulmonary solid nodules in thoracic CT images[J]. Medical Physics, 2015,42.
[50]	Zhu W, Liu C, Fan W, et al. DeepLung: Deep 3D Dual Path Nets for Automated Pulmonary Nodule Detection and Classification[C]. IEEE Winter Conference on Applications of Computer Vision (WACV), 2018: 673-681.
[51]	Ma J, Li X, Li H , et al. Group-Attention Single-Shot Detector (GA-SSD): Finding Pulmonary Nodules in Large-Scale CT Images[J]. 2018.
[52]	Setio A A A, Traverso A, De Bel T , et al. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge[J]. Medical image analysis, 2017,42:1-13.
[53]	李欣菱, 郭芳芳, 周振 , et al. 基于深度学习的人工智能胸部CT肺结节检测效能评估[J]. 中国肺癌杂志, 2019(6).
[54]	Wang S, Zhou M, Gevaert O, et al. A multi-view deep convolutional neural networks for lung nodule segmentation [C]// 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 2017.
[55]	Nam C, Kim J, Lee K . Lung nodule segmentation with convolutional neural network trained by simple diameter information. [C]// 1st Conference on Medical Imaging with Deep Learning, 2018.
[56]	Haichao C, fHong L, Enmin S, Chih-Cheng H, Guangzhi M , et al. Dual-branch residual network for lung nodule segmentation. Arxiv 2019 [1905.080413].
[57]	Shen S., Han S., Aberle D ., et al. An Interpretable Deep Hierarchical Semantic Convolutional Neural Network for Lung Nodule Malignancy Classification. Arxiv 2018 [1806.00712].
[58]	Shen W, Zhou M, Yang F , et al. Multi-crop Convolutional Neural Networks for Lung Nodule Malignancy Suspiciousness Classification[J]. Pattern Recognition, 2016,61(61):663-673.
[59]	Wu B, Zhou Z, Wang J , et al. Joint Learning for Pulmonary Nodule Segmentation, Attributes and Malignancy Prediction[J]. 2018.
[60]	Mundher A, Boon L, Wai Y , et al. Lung Nodule Classification using Deep Local-Global Networks. Arxiv 2019 [1904.10126].
[61]	Blackmon K N, Florin C, Bogoni L , et al. Computer-aided detection of pulmonary embolism at CT pulmonary angiography: can it improve performance of inexperienced readers?[J]. European Radiology, 2011,21(6):1214-1223.
[62]	Rucco M, Sousarodrigues D, Merelli E , et al. Neural hypernetwork approach for pulmonary embolism diagnosis[J]. BMC Research Notes, 2015,8(1):617-617.
[63]	Bi J, Liang J . Multiple Instance Learning of Pulmonary Embolism Detection with Geodesic Distance along Vascular Structure[C]. computer vision and pattern recognition, 2007: 1-8.
[64]	Chen M C, Ball R L, Yang L , et al. Deep Learning to Classify Radiology Free-Text Reports[J]. Radiology, 2017.
[65]	World Health Organization . Global tuberculosis report 2018[R]. Geneva, Switzerland: World Health Organization, 2018.
[66]	Pande T, Cohen C, Pai M , et al. Computer Aided Detection of Pulmonary Tuberculosis on Digital Chest Radiographs: a systematic review[J]. The International Journal of Tuberculosis and Lung Disease, 2016,20(9):1226.
[67]	Antani S . Automated Detection of Lung Diseases in Chest X-Rays. A Report to the Board of Scientific Counselors[R]. US National Library of Medicine. https://lhncbc.nlm.nih.gov/system/files/pub9126.pdf. Published April 2015. Accessed September 20, 2016.
[68]	Jaeger S, Karargyris A, Candemir S , et al. Automatic screening for tuberculosis in chest radiographs: a survey[J]. Quantitative imaging in medicine and surgery, 2013,3(2):89-99.
[69]	Hwang S, Kim H E, Jeong J, et al. A novel approach for tuberculosis screening based on deep convolutional neural networks [C]// Medical Imaging 2016: Computer-Aided Diagnosis. International Society for Optics and Photonics, 2016.
[70]	Jemal A, Siegel R, Ward E , et al. Cancer statistics, 2008[J]. Ca A Cancer Journal for Clinicians, 2008,58(2):71.
[71]	Gram I T, Funkhouser E, Tabár L . The Tabar classification of mammographic parenchymal patterns[J]. European Journal of Radiology, 1997,24(2):131-6.
[72]	Kooi T, Litjens G, Van Ginneken B , et al. Large scale deep learning for computer aided detection of mammographic lesions[J]. Medical image analysis, 2017,35:303-312.
[73]	Arevalo J, González F A, Ramos-Pollán R , et al. Representation learning for mammography mass lesion classification with convolutional neural networks[J]. Computer methods and programs in biomedicine, 2016,127:248-257.
[74]	Fotin S V, Yin Y, Haldankar H, et al. Detection of soft tissue densities from digital breast tomosynconfproc: comparison of conventional and deep learning approaches [C]// Medical Imaging 2016: Computer-Aided Diagnosis. International Society for Optics and Photonics, 2016,9785:97850X.
[75]	B Buciu I , Gacsadi A. Directional features for automatic tumor classification of mammogram images[J]. Biomedical Signal Processing & Control, 2011,6(4):370-378.
[76]	Shastri A A, Tamrakar D, Ahuja K . Density-wise two stage mammogram classification using texture exploiting descriptors[J]. Expert Systems with Applications, 2018,99:71-82.
[77]	Bria A, Karssemeijer N, Tortorella F . Learning from unbalanced data: a cascade-based approach for detecting clustered microcalcifications[J]. Medical image analysis, 2014,18(2):241-252.
[78]	Cai G, Guo Y, Zhang Y, et al. A fully automatic microcalcification detection approach based on deep convolution neural network [C]// Medical Imaging 2018: Computer-Aided Diagnosis. International Society for Optics and Photonics, 2018,10575:105752Q.
[79]	Zhang F, Luo L, Sun X, et al. Cascaded Generative and Discriminative Learning for Microcalcification Detection in Breast Mammograms [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019: 12578-12586.
[80]	顾景范 . 中国居民营养与慢性病状况报告(2015)解读[J]. 营养学报, 2016,38(6):525-529.
[81]	郭静 . 骨龄检测的方法及其应用[J]. 实用预防医学, 2009, 16(06):1995-1996+1987.
[82]	Iglovikov V I, Rakhlin A, Kalinin A A , et al. Paediatric Bone Age Assessment Using Deep Convolutional Neural Networks[J]. arXiv: Computer Vision and Pattern Recognition, 2018: 300-308.
[83]	Son S J, Song Y, Kim N , et al. TW3-Based Fully Automated Bone Age Assessment System Using Deep Neural Networks[J]. IEEE Access, 2019: 33346-33358.
[84]	Mayeux R, Stern Y . Epidemiology of Alzheimer Disease[J]. Cold Spring Harbor Perspectives in Medicine, 2012,2(8):1-19.
[85]	Durstewitz D, Koppe G, Meyer-Lindenberg A . Deep neural networks in psychiatry[J]. Molecular Psychiatry, 2019.
[86]	Liu X, Chen K, Wu T, Weidman D, Lure F, Li J . Use of multi-modality imaging and artificial intelligence for diagnosis and prognosis of early stages of alzheimer’s disease. Transl Res. 2018,194:56-67.
[87]	Liu S, Liu S, Cai W, Che H, Pujol S, Kikinis R , et al. Multimodal neuroimaging feature learning for multiclass diagnosis of Alzheimer’s disease. IEEE Trans Biomed Eng. 2015,62:1132-40.
[88]	Ma J, Zhang R . Automatic Calcium Scoring in Cardiac and Chest CT Using DenseRAUnet[J]. arXiv: Image and Video Processing, 2019.
[89]	Dou Q, Chen H, Jin Y, et al. 3D deeply supervised network for automatic liver segmentation from CT volumes [C]// International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2016: 149-157.
[90]	Gregor U, Priyam T, Talal A , et al. Deep Learning Localizes and Identifies Polyps in Real Time With 96% Accuracy in Screening Colonoscopy[J]. Gastroenterology, 2018: S0016508518346596.
[91]	Havaei M, Davy A, Warde-Farley D , et al. Brain tumor segmentation with deep neural networks[J]. Medical image analysis, 2017,35:18-31.
[92]	Menze B H, Jakab A, Bauer S , et al. The multimodal brain tumor image segmentation benchmark (BRATS)[J]. IEEE transactions on medical imaging, 2014,34(10):1993-2024.
[93]	Wang D, Khosla A, Gargeya R , et al. Deep learning for identifying metastatic breast cancer[J]. arXiv preprint arXiv:1606.05718, 2016.
[94]	Gulshan V, Peng L, Coram M , et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs[J]. Jama, 2016,316(22):2402-2410.
[95]	Hansen M B, Abràmoff Michael D, Folk J C , et al. Results of Automated Retinal Image Analysis for Detection of Diabetic Retinopathy from the Nakuru Study, Kenya[J]. PLOS ONE, 2015,10(10):e0139148.
[96]	Li W, Jia F, Hu Q . Automatic segmentation of liver tumor in CT images with deep convolutional neural networks[J]. Journal of Computer and Communications, 2015,3(11):146.
[97]	Shen D, Wu G, Suk H , et al. Deep Learning in Medical Image Analysis[J]. Annual Review of Biomedical Engineering, 2017,19(1):221-248.
[98]	Simonyan K, Vedaldi A, Zisserman A, et al. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps[C]. international conference on learning representations, 2013.
[99]	Lee H, Tajmir S, Lee J S , et al. Fully Automated Deep Learning System for Bone Age Assessment[J]. Journal of Digital Imaging, 2017,30(4):427-441.
[100]	Erickson B J, Korfiatis P, Kline T L , et al. Deep Learning in Radiology: Does One Size Fit All?[J]. Journal of The American College of Radiology, 2018,15(3):521-526.
[101]	Kaiser L, Gomez A N, Shazeer N , et al. One Model To Learn Them All[J]. arXiv: Learning, 2017.

任务和评价指标	数据模态	DL	人工特征方法	人类/专家	DL+人类	数据集
脑肿瘤分割/Dice	核磁	88.0%^[91]	79.0%^[92]	-	-	2013 BRATS
转移乳腺癌检测/准确度^[93]	病理切片	92.5%	-	96.60%	99.50%	Camelyon16
视网膜血管分割/准确度	眼底图片	96.0%~97.3%^[40]	92.7~94.5%	94.70%^[43]	-	DRIVE/STARE
糖网筛查/AUC	眼底图片	99.0%^[94]	87.8%^[95]	-	-	Kaggle’s dataset
肺结节筛查/敏感度	CT	95.00%^[52]	63.20%^[48]	-	-	LUNA16
肝分割/体积重叠误差^[89]	CT	5.37%	7.73%	-	-	SLIVER07
皮肤癌分类/准确度^[13]	皮肤图片	72.10%	-	66.00%	-	Public+Private
乳腺良恶识别/敏感性^[74]	DBT*	93.00%	85.20%	-	-	Private
肝肿瘤分割/Dice^[96]	CT	80.1%	75.67%~79.78%	-	-	Private
肠息肉筛查/检出率^[90]	结肠镜	96.40%	-	7%~53%	-	Private
*DBT:数字乳腺断层合成显像