Nothing Special   »   [go: up one dir, main page]

You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 

Deep Learning in Medical Image Analysis

A special issue of Journal of Imaging (ISSN 2313-433X). This special issue belongs to the section "Medical Imaging".

Deadline for manuscript submissions: closed (31 December 2020) | Viewed by 143953

Printed Edition Available!
A printed edition of this Special Issue is available here.

Special Issue Editors


grade E-Mail Website
Guest Editor
Informatics Building School of Informatics, University of Leicester, Leicester LE1 7RH, UK
Interests: deep learning; artificial intelligence; machine learning
Special Issues, Collections and Topics in MDPI journals

E-Mail Website
Guest Editor
1. Molecular Imaging and Neuropathology Division, Columbia University and New York State Psychiatric Institute, New York, NY 10032, USA
2. New York State Psychiatric Institute, New York, NY 10032, USA
Interests: structural mechanics; computational mechanics; contact mechanics; efficient solvers; interfaces; modeling; applications in mechanical and civil engineering
Special Issues, Collections and Topics in MDPI journals

Special Issue Information

Dear Colleagues,

Over the past years, deep learning has established itself as a powerful tool across a broad spectrum of domains in imaging, e.g., classification, prediction, detection, segmentation, diagnosis, interpretation, reconstruction, etc. While deep neural networks initially found nurture in the computer vision community, they have quickly spread over medical imaging applications.

The accelerating power of deep learning in diagnosing diseases will empower physicians and speed-up decision making in clinical environments. Applications of modern medical instruments and digitalization of medical care generated enormous amounts of medical images in recent years. In this big data arena, new deep learning methods and computational models for efficient data processing, analysis, and modeling of the generated data is crucially important for clinical applications and in understanding the underlying biological process.

The purpose of this Special Issue “Deep Learning in Medical Image Analysis” is to present and highlight novel algorithms, architectures, techniques, and applications of deep learning for medical image analysis.

Prof. Dr. Yudong Zhang
Prof. Dr. Juan Manuel Gorriz
Prof. Dr. Zhengchao Dong
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Journal of Imaging is an international peer-reviewed open access monthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 1800 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Keywords

  • artificial intelligence
  • deep learning
  • transfer learning
  • deep neural network
  • convolutional neural network
  • multi-task learning
  • biomedical engineering
  • multimodal imaging
  • semantic segmentation
  • image reconstruction
  • explainable AI
  • healthcare

Benefits of Publishing in a Special Issue

  • Ease of navigation: Grouping papers by topic helps scholars navigate broad scope journals more efficiently.
  • Greater discoverability: Special Issues support the reach and impact of scientific research. Articles in Special Issues are more discoverable and cited more frequently.
  • Expansion of research network: Special Issues facilitate connections among authors, fostering scientific collaborations.
  • External promotion: Articles in Special Issues are often promoted through the journal's social media, increasing their visibility.
  • e-Book format: Special Issues with more than 10 articles can be published as dedicated e-books, ensuring wide and rapid dissemination.

Further information on MDPI's Special Issue polices can be found here.

Related Special Issue

Published Papers (23 papers)

Order results
Result details
Select all
Export citation of selected articles as:

Editorial

Jump to: Research, Review

4 pages, 193 KiB  
Editorial
Deep Learning in Medical Image Analysis
by Yudong Zhang, Juan Manuel Gorriz and Zhengchao Dong
J. Imaging 2021, 7(4), 74; https://doi.org/10.3390/jimaging7040074 - 20 Apr 2021
Cited by 44 | Viewed by 4415
Abstract
Over recent years, deep learning (DL) has established itself as a powerful tool across a broad spectrum of domains in imaging—e [...] Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)

Research

Jump to: Editorial, Review

49 pages, 3950 KiB  
Article
Quantitative Comparison of Deep Learning-Based Image Reconstruction Methods for Low-Dose and Sparse-Angle CT Applications
by Johannes Leuschner, Maximilian Schmidt, Poulami Somanya Ganguly, Vladyslav Andriiashen, Sophia Bethany Coban, Alexander Denker, Dominik Bauer, Amir Hadjifaradji, Kees Joost Batenburg, Peter Maass and Maureen van Eijnatten
J. Imaging 2021, 7(3), 44; https://doi.org/10.3390/jimaging7030044 - 2 Mar 2021
Cited by 28 | Viewed by 8061
Abstract
The reconstruction of computed tomography (CT) images is an active area of research. Following the rise of deep learning methods, many data-driven models have been proposed in recent years. In this work, we present the results of a data challenge that we organized, [...] Read more.
The reconstruction of computed tomography (CT) images is an active area of research. Following the rise of deep learning methods, many data-driven models have been proposed in recent years. In this work, we present the results of a data challenge that we organized, bringing together algorithm experts from different institutes to jointly work on quantitative evaluation of several data-driven methods on two large, public datasets during a ten day sprint. We focus on two applications of CT, namely, low-dose CT and sparse-angle CT. This enables us to fairly compare different methods using standardized settings. As a general result, we observe that the deep learning-based methods are able to improve the reconstruction quality metrics in both CT applications while the top performing methods show only minor differences in terms of peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). We further discuss a number of other important criteria that should be taken into account when selecting a method, such as the availability of training data, the knowledge of the physical measurement model and the reconstruction speed. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>Parallel beam geometry. Adopted from [<a href="#B34-jimaging-07-00044" class="html-bibr">34</a>].</p>
Full article ">Figure 2
<p>The angular sampling ranges employed for sparse image reconstructions for (<b>a</b>) 50 (full), (<b>b</b>) 10 (subset of 50 angles), (<b>c</b>) 5 (subset of 50 angles) and (<b>d</b>) 2 angles (subset of 10 angles). The black arrows show the position of the X-ray source (dot) and the position of the detector (arrowhead). For the sparse-angle scenario, the unused angles are shown in light gray.</p>
Full article ">Figure 3
<p>A horizontal cross-section of a healthy slice in an apple is shown on the <b>left</b>, and another cross-section with the bitter pit defects in the same apple on the <b>right</b>.</p>
Full article ">Figure 4
<p>Reconstructions on the challenge set from the LoDoPaB-CT dataset. The window <math display="inline"><semantics> <mrow> <mo>[</mo> <mn>0</mn> <mo>,</mo> <mn>0.45</mn> <mo>]</mo> </mrow> </semantics></math> corresponds to a HU range of ≈[−1001, 831] .</p>
Full article ">Figure 5
<p>Mean data discrepancy <math display="inline"><semantics> <mrow> <mo>−</mo> <msub> <mo>ℓ</mo> <mi>Pois</mi> </msub> </mrow> </semantics></math> between the noisy measurements and the forward-projected reconstructions, respectively the noise-free measurements. Evaluation is done on the LoDoPaB challenge images.</p>
Full article ">Figure 6
<p>Visual overview of one apple slice with bitter pit for different learned methods. Evaluated on Gaussian noise. The quality of the reconstruction deteriorates very quickly for a reduced number of angles. For the 2-angle case, none of the methods can reconstruct the exact shape of the apple.</p>
Full article ">Figure 7
<p>Mean squared error (MSE) data discrepancy between the measurements and the forward-projected reconstructions for the noise-free (<b>left</b>) and Gaussian noise (<b>right</b>) dataset. The MSE values are plotted against the number of angles used for the reconstruction. For the Gaussian dataset, the mean data discrepancy between noisy and noise-free measurements is given for reference. Evaluation is done on 100 Apple CT test images. See <a href="#jimaging-07-00044-t0A6" class="html-table">Table A6</a> for the exact values.</p>
Full article ">Figure A1
<p>Architecture of the learned primal dual algorithm unrolled for <math display="inline"><semantics> <mrow> <mi>K</mi> <mo>=</mo> <mn>5</mn> </mrow> </semantics></math> iterations. We used a zero initialization for <math display="inline"><semantics> <msup> <mi>h</mi> <mrow> <mo>[</mo> <mn>0</mn> <mo>]</mo> </mrow> </msup> </semantics></math> and the FBP reconstruction for <math display="inline"><semantics> <msup> <mi>x</mi> <mrow> <mo>[</mo> <mn>0</mn> <mo>]</mo> </mrow> </msup> </semantics></math>. Adapted from [<a href="#B19-jimaging-07-00044" class="html-bibr">19</a>].</p>
Full article ">Figure A2
<p>Architecture of the mutli-scale, post-processing U-Net. The general architecture of a U-Net consists on a downsampling path on the left and a upsampling path on the right with intermediate connection between similar scales. Adapted from [<a href="#B40-jimaging-07-00044" class="html-bibr">40</a>].</p>
Full article ">Figure A3
<p>Architecture of the MS-D neural network for width of 1 and depth of 3, feature maps are drawn as light blue squares. Colored lines represent dilated convolutions, different colors correspond to different dilation values. Black lines represent 1 × 1 convolutions that connect the input and all feature maps to the output image. Adapted from [<a href="#B21-jimaging-07-00044" class="html-bibr">21</a>].</p>
Full article ">Figure A4
<p>Architecture of the conditional invertible neural network. The ground truth image <span class="html-italic">x</span> is transformed by <math display="inline"><semantics> <msub> <mi>F</mi> <mi mathvariant="sans-serif">Θ</mi> </msub> </semantics></math> to a Gaussian distributed <span class="html-italic">z</span>. Adapted from [<a href="#B48-jimaging-07-00044" class="html-bibr">48</a>].</p>
Full article ">Figure A5
<p>Architecture of the ISTA U-Net adapted from [<a href="#B42-jimaging-07-00044" class="html-bibr">42</a>]. The sparse code <math display="inline"><semantics> <mi>α</mi> </semantics></math> replaces the downsampling part in the standard U-Net (cf. <a href="#jimaging-07-00044-f0A2" class="html-fig">Figure A2</a>).</p>
Full article ">Figure A6
<p>Architecture of the iCTU-Net.</p>
Full article ">Figure A7
<p>PSNR and SSIM depending on the number of angles on the Apple CT datasets.</p>
Full article ">Figure A8
<p>PSNR and SSIM compared for all noise settings and numbers of angles.</p>
Full article ">Figure A9
<p>Example of an artifact produced by DIP + TV, which has only minor impact on the evaluated metrics (especially the SSIM). The area containing the artifact is marked with a red circle.</p>
Full article ">Figure A10
<p>Training curves of Learned Primal-Dual on the Apple CT dataset. Dashed lines: average validation loss computed after every full training epoch; solid lines: running average of training loss since start of epoch. Duration of 20 epochs on full dataset: ≈10–17 days, varying with the number of angles.</p>
Full article ">Figure A11
<p>Training curves of ISTA U-Net on the Apple CT dataset. Dashed lines: average validation PSNR in decibel computed after every full training epoch; marks: selected model. Duration of 20 epochs on full dataset: ≈10 days for hidden layer width 32+ and 5 scales, respectively ≈5.5 days for hidden layer width 16+ and 6 scales.</p>
Full article ">Figure A12
<p>Training curves of U-Net on the Apple CT dataset. Dashed lines: average validation loss computed after every full training epoch; solid lines: running average of training loss since start of epoch. Duration of 20 epochs on full dataset: ≈1.5 days.</p>
Full article ">Figure A13
<p>Training curves of MS-D-CNN on the Apple CT dataset. Dashed lines: average validation loss computed after every full training epoch; solid lines: running average of training loss since start of epoch. Duration of 20 epochs on full dataset: ≈20 days.</p>
Full article ">Figure A14
<p>Training curves of CINN on the Apple CT dataset. Dashed lines: average validation loss computed after every full training epoch; solid lines: running average of training loss (at every 50-th step) since start of epoch. For some of the trainings, the epochs were divided into multiple shorter ones. Duration of 20 epochs on full dataset: ≈2.5 days (using 2 GPUs).</p>
Full article ">Figure A15
<p>Training curves of iCTU-Net on the Apple CT dataset. Opaque lines: loss for a validation sample (after every 500-th step); semi-transparent lines: training loss (at every 500-th step). Duration of 20 epochs on full dataset: ≈3 days.</p>
Full article ">
10 pages, 695 KiB  
Article
Accelerating 3D Medical Image Segmentation by Adaptive Small-Scale Target Localization
by Boris Shirokikh, Alexey Shevtsov, Alexandra Dalechina, Egor Krivov, Valery Kostjuchenko, Andrey Golanov, Victor Gombolevskiy, Sergey Morozov and Mikhail Belyaev
J. Imaging 2021, 7(2), 35; https://doi.org/10.3390/jimaging7020035 - 13 Feb 2021
Cited by 12 | Viewed by 3839
Abstract
The prevailing approach for three-dimensional (3D) medical image segmentation is to use convolutional networks. Recently, deep learning methods have achieved human-level performance in several important applied problems, such as volumetry for lung-cancer diagnosis or delineation for radiation therapy planning. However, state-of-the-art architectures, such [...] Read more.
The prevailing approach for three-dimensional (3D) medical image segmentation is to use convolutional networks. Recently, deep learning methods have achieved human-level performance in several important applied problems, such as volumetry for lung-cancer diagnosis or delineation for radiation therapy planning. However, state-of-the-art architectures, such as U-Net and DeepMedic, are computationally heavy and require workstations accelerated with graphics processing units for fast inference. However, scarce research has been conducted concerning enabling fast central processing unit computations for such networks. Our paper fills this gap. We propose a new segmentation method with a human-like technique to segment a 3D study. First, we analyze the image at a small scale to identify areas of interest and then process only relevant feature-map patches. Our method not only reduces the inference time from 10 min to 15 s but also preserves state-of-the-art segmentation quality, as we illustrate in the set of experiments with two large datasets. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>Time-performance trade-off for different convolutional neural network models under 8 GB of RAM and eight central processing unit thread restrictions. We evaluate models on two clinically relevant datasets with lung nodules (LUNA16) and brain metastases in terms of the average object-wise recall (LUNA16 competition metric [<a href="#B17-jimaging-07-00035" class="html-bibr">17</a>]). Our model spends less than 15 s per study on processing time while preserving or even surpassing the performance of the state-of-the-art models.</p>
Full article ">Figure 2
<p>Diameter distribution of tumors in the chosen datasets. On both plots, the distribution is presented separately for each subset for which we split the data. The median value is highlighted with orange. In addition, medical studies [<a href="#B31-jimaging-07-00035" class="html-bibr">31</a>,<a href="#B32-jimaging-07-00035" class="html-bibr">32</a>] recommend choosing a 10 mm threshold for the data that contain lung nodules and 5 mm threshold for multiple brain metastases, when classifying the particular component of a target as small.</p>
Full article ">Figure 3
<p>The proposed architecture is a two-stage fully convolutional neural network. It includes <span class="html-italic">low-resolution segmentation</span> (blue), which predicts the <math display="inline"><semantics> <msup> <mn>8</mn> <mn>3</mn> </msup> </semantics></math> times downsampled mask, and <span class="html-italic">detailed segmentation</span> (red), which <span class="html-italic">iteratively</span> and <span class="html-italic">locally</span> aggregates features from the first stage and predicts the segmentation map in the original resolution. Speedup comes from two main factors: the lighter network with early downsampling in the first stage and the heavier second part that typically processes only 5% of the image.</p>
Full article ">
14 pages, 2289 KiB  
Article
Domain Adaptation for Medical Image Segmentation: A Meta-Learning Method
by Penghao Zhang, Jiayue Li, Yining Wang and Judong Pan
J. Imaging 2021, 7(2), 31; https://doi.org/10.3390/jimaging7020031 - 10 Feb 2021
Cited by 18 | Viewed by 5310
Abstract
Convolutional neural networks (CNNs) have demonstrated great achievement in increasing the accuracy and stability of medical image segmentation. However, existing CNNs are limited by the problem of dependency on the availability of training data owing to high manual annotation costs and privacy issues. [...] Read more.
Convolutional neural networks (CNNs) have demonstrated great achievement in increasing the accuracy and stability of medical image segmentation. However, existing CNNs are limited by the problem of dependency on the availability of training data owing to high manual annotation costs and privacy issues. To counter this limitation, domain adaptation (DA) and few-shot learning have been extensively studied. Inspired by these two categories of approaches, we propose an optimization-based meta-learning method for segmentation tasks. Even though existing meta-learning methods use prior knowledge to choose parameters that generalize well from few examples, these methods limit the diversity of the task distribution that they can learn from in medical image segmentation. In this paper, we propose a meta-learning algorithm to augment the existing algorithms with the capability to learn from diverse segmentation tasks across the entire task distribution. Specifically, our algorithm aims to learn from the diversity of image features which characterize a specific tissue type while showing diverse signal intensities. To demonstrate the effectiveness of the proposed algorithm, we conducted experiments using a diverse set of segmentation tasks from the Medical Segmentation Decathlon and two meta-learning benchmarks: model-agnostic meta-learning (MAML) and Reptile. U-Net and Dice similarity coefficient (DSC) were selected as the baseline model and the main performance metric, respectively. The experimental results show that our algorithm maximally surpasses MAML and Reptile by 2% and 2.4% respectively, in terms of the DSC. By showing a consistent improvement in subjective measures, we can also infer that our algorithm can produce a better generalization of a target task that has few examples. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>The diversity of image features in two example tissue types: heart (<b>left</b>) and spleen (<b>right</b>). The image features are segmented by green and red lines in the heart and spleen, respectively. (<b>a</b>) Heart; (<b>b</b>) spleen.</p>
Full article ">Figure 2
<p>Optimization-based meta-learning algorithms. We use black and red arrows to represent the gradient steps in the inner loop and the outer loop, respectively. The blue arrow represents the direction of model update. (<b>a</b>) MAML; (<b>b</b>) Reptile; (<b>c</b>) our algorithm.</p>
Full article ">Figure 3
<p>The selection of examples in the meta-training phase. Left: MAML. Right: Our algorithm. H, S, and P represent the MRI images from the heart, spleen, and prostate, respectively. Each example is displayed by a particular color. (<b>a</b>) MAML; (<b>b</b>) ours.</p>
Full article ">Figure 4
<p>U-Net architecture.</p>
Full article ">Figure 5
<p>The three pipelines. (<b>a</b>) Pipeline I; (<b>b</b>) pipeline II; (<b>c</b>) pipeline III.</p>
Full article ">Figure 6
<p>The convergence of three different approaches under the few-shot scenario of colon.</p>
Full article ">Figure 7
<p>Subjective measures of different approaches in the established segmentation scenarios (top: colon, bottom: liver). Each row presents one typical task, from left to right: (<b>a</b>) raw scans; (<b>b</b>) ground truth labels; (<b>c</b>) pipeline II (multi-source Training); (<b>d</b>) MAML + layer-freezing; (<b>e</b>) Reptile + layer-freezing; (<b>f</b>) ours + layer-freezing. White areas represent the image features of colon and liver.</p>
Full article ">
17 pages, 1864 KiB  
Article
Personal Heart Health Monitoring Based on 1D Convolutional Neural Network
by Antonella Nannavecchia, Francesco Girardi, Pio Raffaele Fina, Michele Scalera and Giovanni Dimauro
J. Imaging 2021, 7(2), 26; https://doi.org/10.3390/jimaging7020026 - 5 Feb 2021
Cited by 23 | Viewed by 4222
Abstract
The automated detection of suspicious anomalies in electrocardiogram (ECG) recordings allows frequent personal heart health monitoring and can drastically reduce the number of ECGs that need to be manually examined by the cardiologists, excluding those classified as normal, facilitating healthcare decision-making and reducing [...] Read more.
The automated detection of suspicious anomalies in electrocardiogram (ECG) recordings allows frequent personal heart health monitoring and can drastically reduce the number of ECGs that need to be manually examined by the cardiologists, excluding those classified as normal, facilitating healthcare decision-making and reducing a considerable amount of time and money. In this paper, we present a system able to automatically detect the suspect of cardiac pathologies in ECG signals from personal monitoring devices, with the aim to alert the patient to send the ECG to the medical specialist for a correct diagnosis and a proper therapy. The main contributes of this work are: (a) the implementation of a binary classifier based on a 1D-CNN architecture for detecting the suspect of anomalies in ECGs, regardless of the kind of cardiac pathology; (b) the analysis was carried out on 21 classes of different cardiac pathologies classified as anomalous; and (c) the possibility to classify anomalies even in ECG segments containing, at the same time, more than one class of cardiac pathologies. Moreover, 1D-CNN based architectures can allow an implementation of the system on cheap smart devices with low computational complexity. The system was tested on the ECG signals from the MIT-BIH ECG Arrhythmia Database for the MLII derivation. Two different experiments were carried out, showing remarkable performance compared to other similar systems. The best result showed high accuracy and recall, computed in terms of ECG segments and even higher accuracy and recall in terms of patients alerted, therefore considering the detection of anomalies with respect to entire ECG recordings. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>Tracing of a normal electrocardiogram (ECG) including P wave, QRS complex, and T wave. Reproduced with permission from Gordon Betts et al., Anatomy and Physiology, Connexions Website. <a href="http://cnx.org/content/col11496/1.6/" target="_blank">http://cnx.org/content/col11496/1.6/</a>; published by OpenStax, 19 June 2013.</p>
Full article ">Figure 2
<p>Examples of ECG recordings. (<b>Top</b>) Premature ventricular contraction (V). (<b>Bottom</b>) Normal record.</p>
Full article ">Figure 3
<p>Architecture of the proposed 1D-convolutional neural network (CNN) model.</p>
Full article ">Figure 4
<p>(<b>a</b>) Training and validation loss and (<b>b</b>) training and validation accuracy for the first experimentation. Accuracy metric is defined in Equation (1). Epoch is commonly referred to as the number of a full training pass over the entire dataset.</p>
Full article ">Figure 5
<p>(<b>a</b>) Training and validation loss and (<b>b</b>) training and validation accuracy for the second experimentation.</p>
Full article ">
15 pages, 920 KiB  
Article
Testing Segmentation Popular Loss and Variations in Three Multiclass Medical Imaging Problems
by Pedro Furtado
J. Imaging 2021, 7(2), 16; https://doi.org/10.3390/jimaging7020016 - 27 Jan 2021
Cited by 12 | Viewed by 2692
Abstract
Image structures are segmented automatically using deep learning (DL) for analysis and processing. The three most popular base loss functions are cross entropy (crossE), intersect-over-the-union (IoU), and dice. Which should be used, is it useful to consider simple variations, such as modifying formula [...] Read more.
Image structures are segmented automatically using deep learning (DL) for analysis and processing. The three most popular base loss functions are cross entropy (crossE), intersect-over-the-union (IoU), and dice. Which should be used, is it useful to consider simple variations, such as modifying formula coefficients? How do characteristics of different image structures influence scores? Taking three different medical image segmentation problems (segmentation of organs in magnetic resonance images (MRI), liver in computer tomography images (CT) and diabetic retinopathy lesions in eye fundus images (EFI)), we quantify loss functions and variations, as well as segmentation scores of different targets. We first describe the limitations of metrics, since loss is a metric, then we describe and test alternatives. Experimentally, we observed that DeeplabV3 outperforms UNet and fully convolutional network (FCN) in all datasets. Dice scored 1 to 6 percentage points (pp) higher than cross entropy over all datasets, IoU improved 0 to 3 pp. Varying formula coefficients improved scores, but the best choices depend on the dataset: compared to crossE, different false positive vs. false negative weights improved MRI by 12 pp, and assigning zero weight to background improved EFI by 6 pp. Multiclass segmentation scored higher than n-uniclass segmentation in MRI by 8 pp. EFI lesions score low compared to more constant structures (e.g., optic disk or even organs), but loss modifications improve those scores significantly 6 to 9 pp. Our conclusions are that dice is best, it is worth assigning 0 weight to class background and to test different weights on false positives and false negatives. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>Example magnetic resonance image (MRI) segmentation of independent test images using DeepLabv3 segmentation network. The left of each image is the groundtruth on a black background, the right is the segmentation: (<b>a</b>) is a slice showing the liver and spleen; (<b>b</b>) is another slice showing the kidneys and a small extremity of the liver.</p>
Full article ">Figure 2
<p>Illustrative examples from the three datasets used. Left: MRI segmentation of liver, spleen, left and right kidneys (groundtruth + segmented); Center: computer tomography (CT) segmentation of liver (groundtruth + segmented); Right: eye fundus image (EFI) with indication of some lesions and example groundtruth.</p>
Full article ">
13 pages, 10882 KiB  
Article
Hand Motion-Aware Surgical Tool Localization and Classification from an Egocentric Camera
by Tomohiro Shimizu, Ryo Hachiuma, Hiroki Kajita, Yoshifumi Takatsume and Hideo Saito
J. Imaging 2021, 7(2), 15; https://doi.org/10.3390/jimaging7020015 - 25 Jan 2021
Cited by 11 | Viewed by 2798
Abstract
Detecting surgical tools is an essential task for the analysis and evaluation of surgical videos. However, in open surgery such as plastic surgery, it is difficult to detect them because there are surgical tools with similar shapes, such as scissors and needle holders. [...] Read more.
Detecting surgical tools is an essential task for the analysis and evaluation of surgical videos. However, in open surgery such as plastic surgery, it is difficult to detect them because there are surgical tools with similar shapes, such as scissors and needle holders. Unlike endoscopic surgery, the tips of the tools are often hidden in the operating field and are not captured clearly due to low camera resolution, whereas the movements of the tools and hands can be captured. As a result that the different uses of each tool require different hand movements, it is possible to use hand movement data to classify the two types of tools. We combined three modules for localization, selection, and classification, for the detection of the two tools. In the localization module, we employed the Faster R-CNN to detect surgical tools and target hands, and in the classification module, we extracted hand movement information by combining ResNet-18 and LSTM to classify two tools. We created a dataset in which seven different types of open surgery were recorded, and we provided the annotation of surgical tool detection. Our experiments show that our approach successfully detected the two different tools and outperformed the two baseline methods. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>The comparison figure between the images of endoscopic surgery [<a href="#B18-jimaging-07-00015" class="html-bibr">18</a>] <b>(left</b>) and open surgery (<b>right</b>).</p>
Full article ">Figure 2
<p>The comparison figure between the scissors (<b>left</b>) and the needle holders (<b>right</b>).</p>
Full article ">Figure 3
<p>The comparison of the hands’ motion between the surgical tools: (<b>a</b>) scissors, (<b>b</b>) needle holder.</p>
Full article ">Figure 4
<p>The overview of the proposed surgical tool detection and classification.</p>
Full article ">Figure 5
<p>The abstract of the proposed method.</p>
Full article ">Figure 6
<p>Images of videos we used for the experiment: (<b>a</b>) Stahl’s ear, (<b>b</b>) lipoma, (<b>c</b>) skin tumor, (<b>d</b>) cryptotia, (<b>e</b>) cleft palate, (<b>f</b>) open fracture, and (<b>g</b>) nevus cell nevus.</p>
Full article ">Figure 7
<p>Examples of results.</p>
Full article ">Figure 8
<p>ROC curve.</p>
Full article ">Figure 9
<p>Learning curve.</p>
Full article ">
14 pages, 2059 KiB  
Article
Bayesian Learning of Shifted-Scaled Dirichlet Mixture Models and Its Application to Early COVID-19 Detection in Chest X-ray Images
by Sami Bourouis, Abdullah Alharbi and Nizar Bouguila
J. Imaging 2021, 7(1), 7; https://doi.org/10.3390/jimaging7010007 - 10 Jan 2021
Cited by 9 | Viewed by 2961
Abstract
Early diagnosis and assessment of fatal diseases and acute infections on chest X-ray (CXR) imaging may have important therapeutic implications and reduce mortality. In fact, many respiratory diseases have a serious impact on the health and lives of people. However, certain types of [...] Read more.
Early diagnosis and assessment of fatal diseases and acute infections on chest X-ray (CXR) imaging may have important therapeutic implications and reduce mortality. In fact, many respiratory diseases have a serious impact on the health and lives of people. However, certain types of infections may include high variations in terms of contrast, size and shape which impose a real challenge on classification process. This paper introduces a new statistical framework to discriminate patients who are either negative or positive for certain kinds of virus and pneumonia. We tackle the current problem via a fully Bayesian approach based on a flexible statistical model named shifted-scaled Dirichlet mixture models (SSDMM). This mixture model is encouraged by its effectiveness and robustness recently obtained in various image processing applications. Unlike frequentist learning methods, our developed Bayesian framework has the advantage of taking into account the uncertainty to accurately estimate the model parameters as well as the ability to solve the problem of overfitting. We investigate here a Markov Chain Monte Carlo (MCMC) estimator, which is a computer–driven sampling method, for learning the developed model. The current work shows excellent results when dealing with the challenging problem of biomedical image classification. Indeed, extensive experiments have been carried out on real datasets and the results prove the merits of our Bayesian framework. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>Graphical representation of our developed Bayesian finite shifted-scaled Dirichlet mixture model. Fixed hyperparameters are indicated by rounded boxes and random variables by circles. <span class="html-italic">Y</span> is the observed variable, <span class="html-italic">Z</span> represents the latent variable, the large box indicates repeated process, and the arcs show the dependencies between variables.</p>
Full article ">Figure 2
<p>Illustrative sample of Chest X-Rays image with COVID-19 [<a href="#B43-jimaging-07-00007" class="html-bibr">43</a>].</p>
Full article ">Figure 3
<p>Illustrative examples of augmented Chest X-Rays with COVID-19 from the dataset [<a href="#B44-jimaging-07-00007" class="html-bibr">44</a>].</p>
Full article ">Figure 4
<p>Illustrative samples of chest-xray-pneumonia from the dataset in [<a href="#B45-jimaging-07-00007" class="html-bibr">45</a>].</p>
Full article ">Figure 5
<p>The pipeline of the proposed method. First, the lungs are segmented, then robust visual features are extracted. Features are modelled using the proposed mixture model (SSDDMM) and a Bayesian framework is applied to estimate the parameters of the model. Finally, images are classified on the basis of Bayes rule.</p>
Full article ">Figure 6
<p>Process of lungs regions extraction applied on image sample from [<a href="#B42-jimaging-07-00007" class="html-bibr">42</a>].</p>
Full article ">
15 pages, 46221 KiB  
Article
Data Augmentation Using Adversarial Image-to-Image Translation for the Segmentation of Mobile-Acquired Dermatological Images
by Catarina Andrade, Luís F. Teixeira, Maria João M. Vasconcelos and Luís Rosado
J. Imaging 2021, 7(1), 2; https://doi.org/10.3390/jimaging7010002 - 24 Dec 2020
Cited by 8 | Viewed by 3448
Abstract
Dermoscopic images allow the detailed examination of subsurface characteristics of the skin, which led to creating several substantial databases of diverse skin lesions. However, the dermoscope is not an easily accessible tool in some regions. A less expensive alternative could be acquiring medium [...] Read more.
Dermoscopic images allow the detailed examination of subsurface characteristics of the skin, which led to creating several substantial databases of diverse skin lesions. However, the dermoscope is not an easily accessible tool in some regions. A less expensive alternative could be acquiring medium resolution clinical macroscopic images of skin lesions. However, the limited volume of macroscopic images available, especially mobile-acquired, hinders developing a clinical mobile-based deep learning approach. In this work, we present a technique to efficiently utilize the sizable number of dermoscopic images to improve the segmentation capacity of macroscopic skin lesion images. A Cycle-Consistent Adversarial Network is used to translate the image between the two distinct domains created by the different image acquisition devices. A visual inspection was performed on several databases for qualitative evaluation of the results, based on the disappearance and appearance of intrinsic dermoscopic and macroscopic features. Moreover, the Fréchet Inception Distance was used as a quantitative metric. The quantitative segmentation results are demonstrated on the available macroscopic segmentation databases, SMARTSKINS and Dermofit Image Library, yielding test set thresholded Jaccard Index of 85.13% and 74.30%. These results establish a new state-of-the-art performance in the SMARTSKINS database. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>Illustrative examples of macroscopic (row above) and dermoscopic (below) skin lesions. (<b>A</b>) above: SMARTSKINS (Set M); below: ISIC (set D); (<b>B</b>) above: Dermofit (Set M); below: PH2 (set D); (<b>C</b>) EDRA; (<b>D</b>) SMARTSKINS 2014/2015.</p>
Full article ">Figure 2
<p>CycleGAN framework and training strategy. <span style="color:blue">Blue</span>—Forward cycle-consistency loss; <span style="color:red">Red</span>—Backwards cycle-consistency loss.</p>
Full article ">Figure 3
<p>Examples of the translation between domains in EDRA (<b>a</b>) and SMARTSKINS 2014/2015 (<b>b</b>) tests subsets. For each subfigure, from left to right: pair Dermo→TransMacro and pair Macro→TransDermo.</p>
Full article ">Figure 4
<p>Examples of successful (<b>top row</b>) and failure cases (<b>bottom row</b>) of translation from the dermoscopic domain to the macroscopic domain in ISIC test subset. The letters A, B, C and D represent pairs of Dermo→TransMacro.</p>
Full article ">Figure 5
<p>Examples of successful (column 1 and 2) and failure cases (column 3 and 4) of translation from the dermoscopic domain to the macroscopic domain in the PH2 test subset. From Left to right: pair Dermo→TransMacro, cropped pair Dermo→TransMacro.</p>
Full article ">Figure 6
<p>Examples of questionable segmentation labels of the Dermofit Database.</p>
Full article ">Figure 7
<p>Segmentation results of the Set M + Set M<math display="inline"><semantics> <msub> <mrow/> <mrow> <mi>a</mi> <mi>r</mi> <mi>t</mi> <mi>i</mi> <mi>f</mi> <mi>i</mi> <mi>c</mi> <mi>i</mi> <mi>a</mi> <mi>l</mi> </mrow> </msub> </semantics></math> from the tests subsets. In the comparison images: <span style="color:yellow">yellow</span>—true positives; <span style="color:red">red</span>—false positives; <span style="color:green">green</span>—false negatives; black—true negatives.</p>
Full article ">
14 pages, 550 KiB  
Article
Musculoskeletal Images Classification for Detection of Fractures Using Transfer Learning
by Ibrahem Kandel, Mauro Castelli and Aleš Popovič
J. Imaging 2020, 6(11), 127; https://doi.org/10.3390/jimaging6110127 - 23 Nov 2020
Cited by 22 | Viewed by 3826
Abstract
The classification of the musculoskeletal images can be very challenging, mostly when it is being done in the emergency room, where a decision must be made rapidly. The computer vision domain has gained increasing attention in recent years, due to its achievements in [...] Read more.
The classification of the musculoskeletal images can be very challenging, mostly when it is being done in the emergency room, where a decision must be made rapidly. The computer vision domain has gained increasing attention in recent years, due to its achievements in image classification. The convolutional neural network (CNN) is one of the latest computer vision algorithms that achieved state-of-the-art results. A CNN requires an enormous number of images to be adequately trained, and these are always scarce in the medical field. Transfer learning is a technique that is being used to train the CNN by using fewer images. In this paper, we study the appropriate method to classify musculoskeletal images by transfer learning and by training from scratch. We applied six state-of-the-art architectures and compared their performance with transfer learning and with a network trained from scratch. From our results, transfer learning did increase the model performance significantly, and, additionally, it made the model less prone to overfitting. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>A sample of the MURA dataset.</p>
Full article ">
12 pages, 1351 KiB  
Article
Fully 3D Active Surface with Machine Learning for PET Image Segmentation
by Albert Comelli
J. Imaging 2020, 6(11), 113; https://doi.org/10.3390/jimaging6110113 - 23 Oct 2020
Cited by 13 | Viewed by 3097
Abstract
In order to tackle three-dimensional tumor volume reconstruction from Positron Emission Tomography (PET) images, most of the existing algorithms rely on the segmentation of independent PET slices. To exploit cross-slice information, typically overlooked in these 2D implementations, I present an algorithm capable of [...] Read more.
In order to tackle three-dimensional tumor volume reconstruction from Positron Emission Tomography (PET) images, most of the existing algorithms rely on the segmentation of independent PET slices. To exploit cross-slice information, typically overlooked in these 2D implementations, I present an algorithm capable of achieving the volume reconstruction directly in 3D, by leveraging an active surface algorithm. The evolution of such surface performs the segmentation of the whole stack of slices simultaneously and can handle changes in topology. Furthermore, no artificial stop condition is required, as the active surface will naturally converge to a stable topology. In addition, I include a machine learning component to enhance the accuracy of the segmentation process. The latter consists of a forcing term based on classification results from a discriminant analysis algorithm, which is included directly in the mathematical formulation of the energy function driving surface evolution. It is worth noting that the training of such a component requires minimal data compared to more involved deep learning methods. Only eight patients (i.e., two lung, four head and neck, and two brain cancers) were used for training and testing the machine learning component, while fifty patients (i.e., 10 lung, 25 head and neck, and 15 brain cancers) were used to test the full 3D reconstruction algorithm. Performance evaluation is based on the same dataset of patients discussed in my previous work, where the segmentation was performed using the 2D active contour. The results confirm that the active surface algorithm is superior to the active contour algorithm, outperforming the earlier approach on all the investigated anatomical districts with a dice similarity coefficient of 90.47 ± 2.36% for lung cancer, 88.30 ± 2.89% for head and neck cancer, and 90.29 ± 2.52% for brain cancer. Based on the reported results, it can be claimed that the migration into a 3D system yielded a practical benefit justifying the effort to rewrite an existing 2D system for PET imaging segmentation. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>Comparison of the Positron Emission Tomography (PET) system proposed in the previous work (<b>A</b>) [<a href="#B24-jimaging-06-00113" class="html-bibr">24</a>], and the implementation set out here (<b>B</b>). The proposed implementation substitutes the ‘Slice Marching Segmentation’ block (<b>A</b>) with the fully “3D Segmentation” block (<b>B</b>). Therefore, cross-slice information previously ignored is now being completely exploited. Moreover, the artificial stopping condition (step i) is no longer necessary. Additionally, (steps b, c, f, and g) were modified in order to provide a 3D sampling and training (step b1), a 3D tissue classification (step c1), 3D contour initialization (step f1), and 3D AS (step g1).</p>
Full article ">Figure 2
<p>Classification performance of the 3D DA classifier was evaluated by calculating the area under the receiver operating characteristic (ROC) curve (AUC) using the ROC analysis.</p>
Full article ">Figure 3
<p>Three different tumors reported in the figure show the difference between 2D active contours (<b>a</b>,<b>d</b>,<b>g</b>) [<a href="#B24-jimaging-06-00113" class="html-bibr">24</a>] and 3D active surface (<b>b</b>,<b>e</b>,<b>h</b>) segmentations guided by the 3D classifier. The reconstructed surfaces (green and red) and the gold standards (in yellow) are rendered partially transparent for better comparison. In the last column, the overlap of both methods and the gold standard is shown (<b>c</b>,<b>f</b>,<b>i</b>). Specifically, an over-segmentation of the 2D approach compared to 3D can be observed. (Color images can be found on the electronic version of this article).</p>
Full article ">
17 pages, 15218 KiB  
Article
Morphological Estimation of Cellularity on Neo-Adjuvant Treated Breast Cancer Histological Images
by Mauricio Alberto Ortega-Ruiz, Cefa Karabağ, Victor García Garduño and Constantino Carlos Reyes-Aldasoro
J. Imaging 2020, 6(10), 101; https://doi.org/10.3390/jimaging6100101 - 27 Sep 2020
Cited by 5 | Viewed by 4252
Abstract
This paper describes a methodology that extracts key morphological features from histological breast cancer images in order to automatically assess Tumour Cellularity (TC) in Neo-Adjuvant treatment (NAT) patients. The response to NAT gives information on therapy efficacy and it is measured by the [...] Read more.
This paper describes a methodology that extracts key morphological features from histological breast cancer images in order to automatically assess Tumour Cellularity (TC) in Neo-Adjuvant treatment (NAT) patients. The response to NAT gives information on therapy efficacy and it is measured by the residual cancer burden index, which is composed of two metrics: TC and the assessment of lymph nodes. The data consist of whole slide images (WSIs) of breast tissue stained with Hematoxylin and Eosin (H&E) released in the 2019 SPIE Breast Challenge. The methodology proposed is based on traditional computer vision methods (K-means, watershed segmentation, Otsu’s binarisation, and morphological operations), implementing colour separation, segmentation, and feature extraction. Correlation between morphological features and the residual TC after a NAT treatment was examined. Linear regression and statistical methods were used and twenty-two key morphological parameters from the nuclei, epithelial region, and the full image were extracted. Subsequently, an automated TC assessment that was based on Machine Learning (ML) algorithms was implemented and trained with only selected key parameters. The methodology was validated with the score assigned by two pathologists through the intra-class correlation coefficient (ICC). The selection of key morphological parameters improved the results reported over other ML methodologies and it was very close to deep learning methodologies. These results are encouraging, as a traditionally-trained ML algorithm can be useful when limited training data are available preventing the use of deep learning approaches. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>(<b>a</b>) Stroma region which shows several clusters, regions of fat as well as some lymphocytes. (<b>b</b>) Graphical description of the elements that are analysed in this work. Within a stroma, the connective tissue shown in pink region, ducts appear as clusters stained with haematoxylin and contain several nuclei. Outside these ducts, regions of fat appear white and lymphocytes appear purple.</p>
Full article ">Figure 2
<p>Graphical display of three selected morphological parameters against the cellularity: Global stroma filtered region (o), minor axis (△) and Value concentration in region (x). Cellularity was manually ranked by a pathologist and the parameters automatically extracted. The plots represent average parameter value normalised with maximum value to be compared in the same graph. The graph indicates an inverse relationship between respective parameter and cellularity. Thumbnails show images with cellularity values from zero to one and magnified versions of cases with values 0, <math display="inline"><semantics> <mrow> <mn>0.4</mn> </mrow> </semantics></math>, <math display="inline"><semantics> <mrow> <mn>0.7</mn> </mrow> </semantics></math>, and 1, where the prevalence of cancerous cells can be clearly observed.</p>
Full article ">Figure 3
<p>Graphical illustration of the pipeline of the methodology. Under two operational modes images are processed to extract features at nuclei, regional, cluster, and global image regions either to classify and assign Cellularity or to extract same features in order to an archive output.</p>
Full article ">Figure 4
<p>Enhancement of nuclei area. First row corresponds to a Tumour Cellularity (TC) value of zero, second row is an example of TC <math display="inline"><semantics> <mrow> <mo>=</mo> <mn>0.5</mn> </mrow> </semantics></math> and third row is TC <math display="inline"><semantics> <mrow> <mo>=</mo> <mn>1</mn> </mrow> </semantics></math>. Images in column (<b>a</b>) are the original image, (<b>b</b>) gray level images, (<b>c</b>) enhanced images, notice that the nuclei region is darker whilst background becomes lighter, (<b>d</b>) Binary image obtained with an adaptive threshold value estimated directly from strongest correlated parameter, and (<b>e</b>) validation of nuclei segmentation by Jaccard index. Reference image was manually segmented by using QuPath Platform.</p>
Full article ">Figure 5
<p>Regional analysis of the data. (<b>a</b>) Original image. (<b>b</b>) <math display="inline"><semantics> <mrow> <mi>I</mi> <mi>p</mi> </mrow> </semantics></math>, stroma region image obtained from colour separation. (<b>c</b>) Binarised <math display="inline"><semantics> <mrow> <mi>I</mi> <mi>p</mi> </mrow> </semantics></math> image. (<b>d</b>) <math display="inline"><semantics> <mrow> <mi>I</mi> <mi>b</mi> </mrow> </semantics></math> nuclei image obtained from colour separation. (<b>e</b>) Binarised <math display="inline"><semantics> <mrow> <mi>I</mi> <mi>b</mi> </mrow> </semantics></math> image. (<b>f</b>) Regional image concentration of <math display="inline"><semantics> <mrow> <mi>I</mi> <mi>p</mi> </mrow> </semantics></math> in which brown colour indicates the lowest and white is the highest region concentration. (<b>g</b>) Regional density concentration of <math display="inline"><semantics> <mrow> <mi>I</mi> <mi>b</mi> </mrow> </semantics></math>. This example was processed at <math display="inline"><semantics> <mrow> <mn>30</mn> <mo>×</mo> <mn>30</mn> </mrow> </semantics></math> pixels window.</p>
Full article ">Figure 6
<p>(<b>a</b>) The original patch with clear breast ducts. (<b>b</b>) Clusters detected by the methodology, every duct or cluster is in yellow and cells inside are in red. Background is labelled in white. (<b>c</b>,<b>d</b>) Magnified regions of (<b>a</b>,<b>b</b>).</p>
Full article ">Figure 7
<p>Morphological relationship between the four strongest correlated parameters. (<b>a</b>) Global value from HSV component and Stroma average Concentration. (<b>b</b>) Background Image Concentration and Stroma Filtered Concentration. In both cases, the benign and malignant cells are highlighted with different colours, which indicate a clear separation.</p>
Full article ">Figure 8
<p>(<b>a</b>) Receiver operating characteristic (ROC) curves obtained during training phase for the three selected algorithms. (<b>a</b>) Support Vector Machines (SVM), (<b>b</b>) AdaBoost, and (<b>c</b>) K-Nearest Neighbour (KNN). The higher accuracy value achieved is obtained only with the training set that is highly correlated with Tumour Cellularity (TC).</p>
Full article ">Figure 9
<p>Visual description of the method. Three TC cases are presented in each row: 0, <math display="inline"><semantics> <mrow> <mn>0.5</mn> </mrow> </semantics></math> and 1. (<b>a</b>) The original image, the image is segmented and key parameters are computed, then a classification predictor estimates either malignant or benign cells, shown in red and green, respectively in (<b>b</b>). A dilation of segmented malignant nuclei estimates full cytoplasm of every detected malignant cell (<b>c</b>) and TB region is shown in white in (<b>d</b>). The cellularity metrics calculated by the proposed methodology are: <math display="inline"><semantics> <mrow> <mi>T</mi> <mi>C</mi> <mo>=</mo> <mn>0.0113</mn> </mrow> </semantics></math>, <math display="inline"><semantics> <mrow> <mi>T</mi> <mi>C</mi> <mo>=</mo> <mn>0.5181</mn> </mrow> </semantics></math> and <math display="inline"><semantics> <mrow> <mi>T</mi> <mi>C</mi> <mo>=</mo> <mn>0.9936</mn> </mrow> </semantics></math>.</p>
Full article ">Figure 10
<p>Results of implementation on Training Data. Boxplots for the Cancer Cellularity Reference Value against: (<b>a</b>) SVM estimation, (<b>b</b>) KNN estimation and (<b>c</b>) BT Estimation. It should be noted that large boxplots correspond to large variations of the estimations and as such, SVM shows the lowest variability.</p>
Full article ">Figure 11
<p>Three cases where the TC was incorrectly calculated. Each row corresponds to a patch with incorrect TC. Column (<b>a</b>) illustrates the original image, and columns (<b>b</b>–<b>e</b>) show the step by step process to assess cellularity. (<b>b</b>) Corresponds to the segmented image and classified into benign (green) and malignant (red). Columns (<b>c</b>,<b>d</b>) show expanded region of malignant cells. Column (<b>e</b>) corresponds to cancer cell region in white, used to compute TC. Three worst cases correspond to a TC of zero; this means there would not be any malignant cell and TC image must be completely black. Several cells were miss classified which yields to a TC wrong assessment. Estimated TC values are: 0.75, 0.81, and 0.75, instead of zero.</p>
Full article ">
17 pages, 573 KiB  
Article
Comparative Study of First Order Optimizers for Image Classification Using Convolutional Neural Networks on Histopathology Images
by Ibrahem Kandel, Mauro Castelli and Aleš Popovič
J. Imaging 2020, 6(9), 92; https://doi.org/10.3390/jimaging6090092 - 8 Sep 2020
Cited by 49 | Viewed by 6083
Abstract
The classification of histopathology images requires an experienced physician with years of experience to classify the histopathology images accurately. In this study, an algorithm was developed to assist physicians in classifying histopathology images; the algorithm receives the histopathology image as an input and [...] Read more.
The classification of histopathology images requires an experienced physician with years of experience to classify the histopathology images accurately. In this study, an algorithm was developed to assist physicians in classifying histopathology images; the algorithm receives the histopathology image as an input and produces the percentage of cancer presence. The primary classifier used in this algorithm is the convolutional neural network, which is a state-of-the-art classifier used in image classification as it can classify images without relying on the manual selection of features from each image. The main aim of this research is to improve the robustness of the classifier used by comparing six different first-order stochastic gradient-based optimizers to select the best for this particular dataset. The dataset used to train the classifier is the PatchCamelyon public dataset, which consists of 220,025 images to train the classifier; the dataset is composed of 60% positive images and 40% negative images, and 57,458 images to test its performance. The classifier was trained on 80% of the images and validated on the rest of 20% of the images; then, it was tested on the test set. The optimizers were evaluated based on their AUC of the ROC curve. The results show that the adaptative based optimizers achieved the highest results except for AdaGrad that achieved the lowest results. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>Example of images available in the PatchCamelyon dataset [<a href="#B20-jimaging-06-00092" class="html-bibr">20</a>].</p>
Full article ">Figure 2
<p>InceptionV3 network architecture.</p>
Full article ">
13 pages, 4237 KiB  
Article
Detection of HER2 from Haematoxylin-Eosin Slides Through a Cascade of Deep Learning Classifiers via Multi-Instance Learning
by David La Barbera, António Polónia, Kevin Roitero, Eduardo Conde-Sousa and Vincenzo Della Mea
J. Imaging 2020, 6(9), 82; https://doi.org/10.3390/jimaging6090082 - 23 Aug 2020
Cited by 18 | Viewed by 5127
Abstract
Breast cancer is the most frequently diagnosed cancer in woman. The correct identification of the HER2 receptor is a matter of major importance when dealing with breast cancer: an over-expression of HER2 is associated with aggressive clinical behaviour; moreover, HER2 targeted therapy results [...] Read more.
Breast cancer is the most frequently diagnosed cancer in woman. The correct identification of the HER2 receptor is a matter of major importance when dealing with breast cancer: an over-expression of HER2 is associated with aggressive clinical behaviour; moreover, HER2 targeted therapy results in a significant improvement in the overall survival rate. In this work, we employ a pipeline based on a cascade of deep neural network classifiers and multi-instance learning to detect the presence of HER2 from Haematoxylin–Eosin slides, which partly mimics the pathologist’s behaviour by first recognizing cancer and then evaluating HER2. Our results show that the proposed system presents a good overall effectiveness. Furthermore, the system design is prone to further improvements that can be easily deployed in order to increase the effectiveness score. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>Example of tiles from the HEROHE dataset. The left-most two slides (<b>A</b>,<b>B</b>) have been labelled as positive for HER2; The right-most two slides (<b>C</b>,<b>D</b>) have been labelled as negative.</p>
Full article ">Figure 2
<p>Graphical overview of the multi-stage deep learning pipeline.</p>
Full article ">Figure 3
<p>An example of slides from our dataset in which we applied a rectangular grid (shown with the continuous black lines) to show the division of a slide in tiles.</p>
Full article ">Figure 4
<p>Tiles identified by the cancer classifier as: positive (first two tiles, tile (<b>A</b>) is a true positive, tile (<b>B</b>) is a false positive), and negative (last two tiles, tile (<b>C</b>) is a true negative, tile (<b>D</b>) is a false negative).</p>
Full article ">Figure 5
<p>The basic architecture of Resnet152 (<b>left</b>, from [<a href="#B48-jimaging-06-00082" class="html-bibr">48</a>]), and the building block for the residual learning (<b>right</b>, from [<a href="#B43-jimaging-06-00082" class="html-bibr">43</a>]).</p>
Full article ">Figure 6
<p>Tiles identified by the HER2 classifier as positive (first two tiles, tile (<b>A</b>) is a true positive, tile (<b>B</b>) is a false positive), and negative (last two tiles, tile (<b>C</b>) is a true negative, and tile (<b>D</b>) is a false negative).</p>
Full article ">
18 pages, 5040 KiB  
Article
Full 3D Microwave Breast Imaging Using a Deep-Learning Technique
by Vahab Khoshdel, Mohammad Asefi, Ahmed Ashraf and Joe LoVetri
J. Imaging 2020, 6(8), 80; https://doi.org/10.3390/jimaging6080080 - 11 Aug 2020
Cited by 45 | Viewed by 4618
Abstract
A deep learning technique to enhance 3D images of the complex-valued permittivity of the breast obtained via microwave imaging is investigated. The developed technique is an extension of one created to enhance 2D images. We employ a 3D Convolutional Neural Network, based on [...] Read more.
A deep learning technique to enhance 3D images of the complex-valued permittivity of the breast obtained via microwave imaging is investigated. The developed technique is an extension of one created to enhance 2D images. We employ a 3D Convolutional Neural Network, based on the U-Net architecture, that takes in 3D images obtained using the Contrast-Source Inversion (CSI) method and attempts to produce the true 3D image of the permittivity. The training set consists of 3D CSI images, along with the true numerical phantom images from which the microwave scattered field utilized to create the CSI reconstructions was synthetically generated. Each numerical phantom varies with respect to the size, number, and location of tumors within the fibroglandular region. The reconstructed permittivity images produced by the proposed 3D U-Net show that the network is not only able to remove the artifacts that are typical of CSI reconstructions, but it also enhances the detectability of the tumors. We test the trained U-Net with 3D images obtained from experimentally collected microwave data as well as with images obtained synthetically. Significantly, the results illustrate that although the network was trained using only images obtained from synthetic data, it performed well with images obtained from both synthetic and experimental data. Quantitative evaluations are reported using Receiver Operating Characteristics (ROC) curves for the tumor detectability and RMS error for the enhancement of the reconstructions. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>Simulated Breast Model. Gray, blue, green, and red regions represent air, fat, fibroglandular, and tumor, respectively.</p>
Full article ">Figure 2
<p>Schematic for the proposed U-Net to reconstruct the real part of permittivity. The input to the network is the 3D Contrast-Source Inversion (CSI) reconstruction, and the network is trained to output the corresponding true 3D permittivity map.</p>
Full article ">Figure 3
<p>The detection performance using the reconstructed outputs of the Convolutional Neural Network (CNN) and CSI as well as the intersection of CSI reconstructions at the five chosen frequencies. (<b>a</b>) Receiver Operating Characteristics (ROC) curves derived from the reconstructions. (<b>b</b>) The DMTD curve.</p>
Full article ">Figure 4
<p>Reconstruction results for a particular example with two tumors. The real (<b>a</b>) and imaginary (<b>b</b>) part of CSI reconstruction at 1.1 GHz. (<b>c</b>) CNN reconstruction. (<b>d</b>) Ground truth.</p>
Full article ">Figure 5
<p>CSI reconstructions at four remaining frequencies for the same example as in <a href="#jimaging-06-00080-f004" class="html-fig">Figure 4</a> and resulting images after intersecting images thresholded at 85% of the maximum reconstructed permittivity. (<b>a</b>–<b>d</b>) The real part of CSI reconstructions at 1.2, 1.3, 1.4, and 1.5 GHz. (<b>e</b>) Intersection of real part of CSI reconstructions. (<b>f</b>) Intersection of imaginary part of CSI reconstructions.</p>
Full article ">Figure 6
<p>Reconstruction results for a particular example with one tumor at 1.05 GHz. The real (<b>a</b>) and imaginary (<b>b</b>) part of CSI reconstruction. (<b>c</b>) CNN reconstruction. (<b>d</b>) Ground truth.</p>
Full article ">Figure 7
<p>Reconstruction results for a particular example when the test images are CSI results for a breast phantom having a smaller fibroglandular region than those of the training set. The (<b>a</b>) real and (<b>b</b>) imaginary parts of the CSI reconstructions. (<b>c</b>) CNN reconstruction. (<b>d</b>) Ground truth.</p>
Full article ">Figure 8
<p>Reconstruction results for a particular example with two tumor when the training images are CSI results with perfect prior information, but the neural net was tested on imperfect prior information. The real (<b>a</b>) and imaginary (<b>b</b>) part of CSI reconstruction. (<b>c</b>) CNN reconstruction. (<b>d</b>) Ground truth.</p>
Full article ">Figure 9
<p>Detection performance based on the reconstructed outputs of CNN and CSI. (<b>a</b>) ROC curves derived from the reconstructed real part of the permittivity from CSI and CNN. (<b>b</b>) The DMTD. test cases are: synthetic: imperfect permittivity prior, and true breast phantom with elongated fibroglandular region. Experimental: using data within the frequency band and much higher than the training frequency band.</p>
Full article ">Figure 10
<p>Reconstruction results for a particular example when the training images are CSI results with one or two tumors but the neural net was tested on a phantom with no tumor. The real (<b>a</b>) and imaginary (<b>b</b>) part of CSI reconstruction. (<b>c</b>) CNN reconstruction. (<b>d</b>) Ground truth.</p>
Full article ">Figure 11
<p>The experimental system including the three region breast phantom (Diameter of fat, fibroglanduar and tumor regions are 10, 8 and 2 CM respectively).</p>
Full article ">Figure 12
<p>CNN performance for experimental result when the neural net was trained on Synthetic data. The real (<b>a</b>) and imaginary (<b>b</b>) part of CSI reconstruction. (<b>c</b>) CNN reconstruction. (<b>d</b>) Ground truth.</p>
Full article ">Figure 13
<p>Reconstruction results for a particular example when the test images are CSI results in high frequencies but the neural net was trained on low frequencies. The real (<b>a</b>) and imaginary (<b>b</b>) part of CSI reconstruction. (<b>c</b>) CNN reconstruction. (<b>d</b>) Ground truth. (<b>e</b>) Intersection of real part of CSI reconstruction at all frequencies.(<b>f</b>) Intersection of imaginary part of CSI reconstruction at all frequencies(two intersection images are binary image).</p>
Full article ">
22 pages, 4563 KiB  
Article
Analyzing Age-Related Macular Degeneration Progression in Patients with Geographic Atrophy Using Joint Autoencoders for Unsupervised Change Detection
by Guillaume Dupont, Ekaterina Kalinicheva, Jérémie Sublime, Florence Rossant and Michel Pâques
J. Imaging 2020, 6(7), 57; https://doi.org/10.3390/jimaging6070057 - 29 Jun 2020
Cited by 7 | Viewed by 3439
Abstract
Age-Related Macular Degeneration (ARMD) is a progressive eye disease that slowly causes patients to go blind. For several years now, it has been an important research field to try to understand how the disease progresses and find effective medical treatments. Researchers have been [...] Read more.
Age-Related Macular Degeneration (ARMD) is a progressive eye disease that slowly causes patients to go blind. For several years now, it has been an important research field to try to understand how the disease progresses and find effective medical treatments. Researchers have been mostly interested in studying the evolution of the lesions using different techniques ranging from manual annotation to mathematical models of the disease. However, artificial intelligence for ARMD image analysis has become one of the main research focuses to study the progression of the disease, as accurate manual annotation of its evolution has proved difficult using traditional methods even for experienced practicians. In this paper, we propose a deep learning architecture that can detect changes in the eye fundus images and assess the progression of the disease. Our method is based on joint autoencoders and is fully unsupervised. Our algorithm has been applied to pairs of images from different eye fundus images time series of 24 ARMD patients. Our method has been shown to be quite effective when compared with other methods from the literature, including non-neural network based algorithms that still are the current standard to follow the disease progression and change detection methods from other fields. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>3 of pairs of images acquired six months apart, the GA corresponds to the bright areas. The green arrow in (<b>f</b>) shows a new lesion.</p>
Full article ">Figure 2
<p>Example of Illumination correction. The three images on the top row represent the two original consecutive images (<b>a</b>) and (<b>b</b>), and their raw difference in absolute value (<b>c</b>); on the bottom row: the same images after illumination correction (<b>d</b>) and (<b>e</b>), and the new difference (<b>f</b>).</p>
Full article ">Figure 3
<p>Autoencoder architecture for our algorithm.</p>
Full article ">Figure 4
<p>Patches construction, useful pixels are inside the green area.</p>
Full article ">Figure 5
<p>Structure of the algorithm. Example for set of two images <math display="inline"><semantics> <mrow> <mi>I</mi> <msub> <mi>m</mi> <mn>1</mn> </msub> </mrow> </semantics></math> and <math display="inline"><semantics> <mrow> <mi>I</mi> <msub> <mi>m</mi> <mn>2</mn> </msub> </mrow> </semantics></math> and <span class="html-italic">n</span> the number of patches.</p>
Full article ">Figure 6
<p>Average recall, Precision, and F1 Score depending on the patch size and sigma: <span class="html-italic">σ</span> = 5 <span class="html-italic">in red</span> <span class="html-italic">σ</span> = 7 <span class="html-italic">in green</span> <span class="html-italic">σ</span> = 9 <span class="html-italic">in blue</span>. (<b>a</b>) Recall and Precision depending on the patch size; (<b>b</b>) F1 Score depending on the Patch size.</p>
Full article ">Figure 7
<p>F1-Score as a function of the patch size and the value of sigma <span class="html-italic">σ</span> on patient 005.</p>
Full article ">Figure 8
<p>Difference + Otsu thresholding vs. our approach (AE) on patient 003. (<b>a</b>) Image at <span class="html-italic">t</span> = 0; (<b>b</b>) Image at <span class="html-italic">t</span> + 3 months; (<b>c</b>) Raw difference and Otsu thresholding, F1 score = 0.26; (<b>d</b>) Our method, F1 score = 0.36; (<b>e</b>) Proposed ground truth.</p>
Full article ">Figure 9
<p>Comparison example of the three methods on patient 005. (<b>a</b>) Corrected Image from October 2017; (<b>b</b>) Corrected Image from June 2018; (<b>c</b>) Proposed ground truth; (<b>d</b>) Asako Kanezaki’s approach, F1 score = 0.15; (<b>e</b>) Turgay Celik’s approach, F1 score = 0.35; (<b>f</b>) Our Fully Convolutional AE, F1 score = 0.4.</p>
Full article ">Figure 10
<p>Comparison example of the three methods on patient 001. (<b>a</b>) Corrected Image from April 2017; (<b>b</b>) Corrected Image from October 2017; (<b>c</b>) Proposed ground truth; (<b>d</b>) Asano Kanezaki’s approach, F1 score = 0.17; (<b>e</b>) Turgay Celik’s approach, F1 score = 0.43; (<b>f</b>) Our Fully convolutional AE, F1 score = 0.43.</p>
Full article ">Figure 11
<p>Comparison example of the three methods on patient 010. (<b>a</b>) Corrected Image from November 2017; (<b>b</b>) Corrected Image from May 2018; (<b>c</b>) Proposed ground truth; (<b>d</b>) Otsu thresholding, F1 score = 0.05; (<b>e</b>) Turgay Celik’s approach, F1 score = 0.253; (<b>f</b>) Our Fully convolutional AE, F1 score = 0.38.</p>
Full article ">Figure 12
<p>Example of ground-truth build for patient 010 based on two consecutive masks of segmented lesions at time <span class="html-italic">t</span> and <math display="inline"><semantics> <mrow> <mi>t</mi> <mo>+</mo> <mn>1</mn> </mrow> </semantics></math>: All changes inside the lesions, textural or otherwise, are ignored. (<b>a</b>) Image <span class="html-italic">t</span>; (<b>b</b>) Image <span class="html-italic">t</span> + 1; (<b>c</b>) segmentation mask <span class="html-italic">t</span>; (<b>d</b>) segmentation mask <span class="html-italic">t</span> + 1; (<b>e</b>) Ground truth built from Mask <span class="html-italic">t</span> and <math display="inline"><semantics> <mrow> <mi>t</mi> <mo>+</mo> <mn>1</mn> </mrow> </semantics></math>.</p>
Full article ">Figure 13
<p>Example of a segmentation in (d) where all changes detected in red will be considered false positive since the ground truth does not consider changes within existing lesions regardless of if they are structural of textural. (<b>a</b>) Image of patient 018 at time <span class="html-italic">t</span>; (<b>b</b>) Image of patient 018 at time <span class="html-italic">t</span> + 1; (<b>c</b>) Proposed Ground truth; (<b>d</b>) Proposed segmentation.</p>
Full article ">
15 pages, 2225 KiB  
Article
Classification Models for Skin Tumor Detection Using Texture Analysis in Medical Images
by Marcos A. M. Almeida and Iury A. X. Santos
J. Imaging 2020, 6(6), 51; https://doi.org/10.3390/jimaging6060051 - 19 Jun 2020
Cited by 32 | Viewed by 5286
Abstract
Medical images have made a great contribution to early diagnosis. In this study, a new strategy is presented for analyzing medical images of skin with melanoma and nevus to model, classify and identify lesions on the skin. Machine learning applied to the data [...] Read more.
Medical images have made a great contribution to early diagnosis. In this study, a new strategy is presented for analyzing medical images of skin with melanoma and nevus to model, classify and identify lesions on the skin. Machine learning applied to the data generated by first and second order statistics features, Gray Level Co-occurrence Matrix (GLCM), keypoints and color channel information—Red, Green, Blue and grayscale images of the skin were used to characterize decisive information for the classification of the images. This work proposes a strategy for the analysis of skin images, aiming to choose the best mathematical classifier model, for the identification of melanoma, with the objective of assisting the dermatologist in the identification of melanomas, especially towards an early diagnosis. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>Images of melanoma and nevus tissues: (<b>a</b>) Skin lesion from melanoma image; (<b>b</b>) Nevus Skin.</p>
Full article ">Figure 2
<p>Estimated of incident cases from 2018 to 2040, melanoma of skin, both sexes, all ages. Source: International Agency for Research on Cancer of World Health Organization.</p>
Full article ">Figure 3
<p>Block Diagram of the proposed strategy.</p>
Full article ">Figure 4
<p>Boxplot with average intensities of grayscale images with melanoma and nevus.</p>
Full article ">Figure 5
<p>Receiver operating curve (ROC) curve of logistic regression.</p>
Full article ">Figure 6
<p>Confusion Matrix considering the Logistic Regression Model.</p>
Full article ">Figure 7
<p>Logistic Regression Model of the Proposed Strategy.</p>
Full article ">
11 pages, 1061 KiB  
Article
Deep Multimodal Learning for the Diagnosis of Autism Spectrum Disorder
by Michelle Tang, Pulkit Kumar, Hao Chen and Abhinav Shrivastava
J. Imaging 2020, 6(6), 47; https://doi.org/10.3390/jimaging6060047 - 10 Jun 2020
Cited by 33 | Viewed by 7147
Abstract
Recent medical imaging technologies, specifically functional magnetic resonance imaging (fMRI), have advanced the diagnosis of neurological and neurodevelopmental disorders by allowing scientists and physicians to observe the activity within and between different regions of the brain. Deep learning methods have frequently been implemented [...] Read more.
Recent medical imaging technologies, specifically functional magnetic resonance imaging (fMRI), have advanced the diagnosis of neurological and neurodevelopmental disorders by allowing scientists and physicians to observe the activity within and between different regions of the brain. Deep learning methods have frequently been implemented to analyze images produced by such technologies and perform disease classification tasks; however, current state-of-the-art approaches do not take advantage of all the information offered by fMRI scans. In this paper, we propose a deep multimodal model that learns a joint representation from two types of connectomic data offered by fMRI scans. Incorporating two functional imaging modalities in an automated end-to-end autism diagnosis system will offer a more comprehensive picture of the neural activity, and thus allow for more accurate diagnoses. Our multimodal training strategy achieves a classification accuracy of 74% and a recall of 95%, as well as an F1 score of 0.805, and its overall performance is superior to using only one type of functional data. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p><b>Preprocessing workflow.</b> Construction of correlation matrices, or activation maps (terms are used interchangeably) between fMRI voxels × ROI time series and between pairs of ROI time series alone. Numbers on edges of the objects indicate matrix dimensions at the current step, e.g., the dimensions of the fMRI scan and ROI time series data are <math display="inline"><semantics> <mrow> <mo>(</mo> <mi>H</mi> <mo>=</mo> <mn>61</mn> <mo>,</mo> <mi>W</mi> <mo>=</mo> <mn>73</mn> <mo>,</mo> <mi>D</mi> <mo>=</mo> <mn>61</mn> <mo>,</mo> <mi>T</mi> <mo>=</mo> <mn>296</mn> <mo>)</mo> </mrow> </semantics></math> and <math display="inline"><semantics> <mrow> <mo>(</mo> <mi>N</mi> <mo>=</mo> <mn>116</mn> <mo>,</mo> <mi>T</mi> <mo>=</mo> <mn>296</mn> <mo>)</mo> </mrow> </semantics></math>, respectively. For more detail on ResNet and MLP network architectures, refer to <a href="#jimaging-06-00047-f002" class="html-fig">Figure 2</a>.</p>
Full article ">Figure 2
<p><b>Multimodal network architectures and training phases.</b> Visualization of the structure of the complete deep multimodal network. Numbers denote nodes or input channels in the labeled layer; FC implies fully connected layers. Phase I of training refers to the training of Resnet-18 and MLP for feature extraction; Phase II refers to the end-to-end training of the entire multimodal model after removing output layers from the feature extractors (as shown in the figure, the last 2-node layer of the MLP and the 512-channel layer of the ResNet are bypassed during Phase II of training.)</p>
Full article ">Figure 3
<p><b>Explanations of model predictions.</b> Cross-sections of fMRI scans indicating the regions that the network focused on when making predictions. Left: scans from a patient with autism; right: scans from a healthy individual with normal development. These images depict the regions the model used to predict the diagnosis of the scan, offering transparency on the model’s decision-making process. The visualizations were produced using techniques proposed by Selvaraju et al. [<a href="#B25-jimaging-06-00047" class="html-bibr">25</a>].</p>
Full article ">
22 pages, 10311 KiB  
Article
Explainable Machine Learning Framework for Image Classification Problems: Case Study on Glioma Cancer Prediction
by Emmanuel Pintelas, Meletis Liaskos, Ioannis E. Livieris, Sotiris Kotsiantis and Panagiotis Pintelas
J. Imaging 2020, 6(6), 37; https://doi.org/10.3390/jimaging6060037 - 28 May 2020
Cited by 42 | Viewed by 5772
Abstract
Image classification is a very popular machine learning domain in which deep convolutional neural networks have mainly emerged on such applications. These networks manage to achieve remarkable performance in terms of prediction accuracy but they are considered as black box models since they [...] Read more.
Image classification is a very popular machine learning domain in which deep convolutional neural networks have mainly emerged on such applications. These networks manage to achieve remarkable performance in terms of prediction accuracy but they are considered as black box models since they lack the ability to interpret their inner working mechanism and explain the main reasoning of their predictions. There is a variety of real world tasks, such as medical applications, in which interpretability and explainability play a significant role. Making decisions on critical issues such as cancer prediction utilizing black box models in order to achieve high prediction accuracy but without provision for any sort of explanation for its prediction, accuracy cannot be considered as sufficient and ethnically acceptable. Reasoning and explanation is essential in order to trust these models and support such critical predictions. Nevertheless, the definition and the validation of the quality of a prediction model’s explanation can be considered in general extremely subjective and unclear. In this work, an accurate and interpretable machine learning framework is proposed, for image classification problems able to make high quality explanations. For this task, it is developed a feature extraction and explanation extraction framework, proposing also three basic general conditions which validate the quality of any model’s prediction explanation for any application domain. The feature extraction framework will extract and create transparent and meaningful high level features for images, while the explanation extraction framework will be responsible for creating good explanations relying on these extracted features and the prediction model’s inner function with respect to the proposed conditions. As a case study application, brain tumor magnetic resonance images were utilized for predicting glioma cancer. Our results demonstrate the efficiency of the proposed model since it managed to achieve sufficient prediction accuracy being also interpretable and explainable in simple human terms. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>An abstract overview of our proposed explainable model.</p>
Full article ">Figure 2
<p>Head MRI examples. The red color illustrates the tumor area (<b>a</b>) glioma, (<b>b</b>) meningioma, (<b>c</b>) pituitary.</p>
Full article ">Figure 3
<p>(<b>a</b>) Image with glioma, (<b>b</b>) Region of Interest (ROI) extraction.</p>
Full article ">Figure 4
<p>(<b>a</b>) Binary image, (<b>b</b>) border extraction.</p>
Full article ">Figure 5
<p>A visualization example presenting the decision function of a trained linear model.</p>
Full article ">Figure 6
<p>Two case study instances. Instance A is GLIOMA while Instance B is NON GLIOMA.</p>
Full article ">Figure 7
<p>Explanation output for Instance A.</p>
Full article ">Figure 8
<p>Explanation output for Instance A.</p>
Full article ">

Review

Jump to: Editorial, Research

38 pages, 2866 KiB  
Review
A Survey of Deep Learning for Lung Disease Detection on Medical Images: State-of-the-Art, Taxonomy, Issues and Future Directions
by Stefanus Tao Hwa Kieu, Abdullah Bade, Mohd Hanafi Ahmad Hijazi and Hoshang Kolivand
J. Imaging 2020, 6(12), 131; https://doi.org/10.3390/jimaging6120131 - 1 Dec 2020
Cited by 81 | Viewed by 13126
Abstract
The recent developments of deep learning support the identification and classification of lung diseases in medical images. Hence, numerous work on the detection of lung disease using deep learning can be found in the literature. This paper presents a survey of deep learning [...] Read more.
The recent developments of deep learning support the identification and classification of lung diseases in medical images. Hence, numerous work on the detection of lung disease using deep learning can be found in the literature. This paper presents a survey of deep learning for lung disease detection in medical images. There has only been one survey paper published in the last five years regarding deep learning directed at lung diseases detection. However, their survey is lacking in the presentation of taxonomy and analysis of the trend of recent work. The objectives of this paper are to present a taxonomy of the state-of-the-art deep learning based lung disease detection systems, visualise the trends of recent work on the domain and identify the remaining issues and potential future directions in this domain. Ninety-eight articles published from 2016 to 2020 were considered in this survey. The taxonomy consists of seven attributes that are common in the surveyed articles: image types, features, data augmentation, types of deep learning algorithms, transfer learning, the ensemble of classifiers and types of lung diseases. The presented taxonomy could be used by other researchers to plan their research contributions and activities. The potential future direction suggested could further improve the efficiency and increase the number of deep learning aided lung disease detection applications. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>Flow diagram of the methodology used to conduct this survey.</p>
Full article ">Figure 2
<p>Overview of using deep learning for lung disease detection.</p>
Full article ">Figure 3
<p>Taxonomy of lung disease detection using deep learning.</p>
Full article ">Figure 4
<p>Examples of chest X-ray images.</p>
Full article ">Figure 5
<p>Examples of CT scan images.</p>
Full article ">Figure 6
<p>Examples of sputum smear microscopy images.</p>
Full article ">Figure 7
<p>Examples of histopathology images.</p>
Full article ">Figure 8
<p>Examples of image augmentation: (<b>a</b>) original; (<b>b</b>) 45° rotation; (<b>c</b>) 90° rotation; (<b>d</b>) horizontal flip; (<b>e</b>) vertical flip; (<b>f</b>) positive x and y translation; (<b>g</b>) negative x and y translation; (<b>h</b>) salt and pepper noise; and (<b>i</b>) speckle noise.</p>
Full article ">Figure 9
<p>Example of a CNN structure.</p>
Full article ">Figure 10
<p>(<b>a</b>) The trend of the usage of image types in lung disease detection works in recent years; and (<b>b</b>) the distribution of the image type used in deep learning aided lung disease detection in recent years.</p>
Full article ">Figure 11
<p>(<b>a</b>) The trend of the usage of features in lung disease detection works in recent years; and (<b>b</b>) the distribution of usage of data augmentation in deep learning aided lung disease detection in recent years.</p>
Full article ">Figure 12
<p>(<b>a</b>) The trend of the usage of data augmentation in lung disease detection works in recent years; and (<b>b</b>) the distribution of usage of data augmentation in deep learning aided lung disease detection in recent years.</p>
Full article ">Figure 13
<p>(<b>a</b>) The trend of the usage of deep learning algorithms in lung disease detection works in recent years; and (<b>b</b>) the distribution of the usage of CNN in deep learning aided lung disease detection in recent years.</p>
Full article ">Figure 14
<p>(<b>a</b>) The trend of the usage of transfer learning in lung disease detection works in recent years; and (<b>b</b>) the usage of transfer learning in lung disease detection works using CNN.</p>
Full article ">Figure 15
<p>(<b>a</b>) The trend of the usage of ensemble classifier in lung disease detection works in recent years; and (<b>b</b>) the distribution of the usage of the ensemble in deep learning aided lung disease detection in recent years.</p>
Full article ">Figure 16
<p>(<b>a</b>) The trend of the deep learning aided lung disease detection works in recent years; and (<b>b</b>) the distribution of the diseases detected using deep learning in recent years.</p>
Full article ">
40 pages, 558 KiB  
Review
Deep Learning in Selected Cancers’ Image Analysis—A Survey
by Taye Girma Debelee, Samuel Rahimeto Kebede, Friedhelm Schwenker and Zemene Matewos Shewarega
J. Imaging 2020, 6(11), 121; https://doi.org/10.3390/jimaging6110121 - 10 Nov 2020
Cited by 51 | Viewed by 8025
Abstract
Deep learning algorithms have become the first choice as an approach to medical image analysis, face recognition, and emotion recognition. In this survey, several deep-learning-based approaches applied to breast cancer, cervical cancer, brain tumor, colon and lung cancers are studied and reviewed. Deep [...] Read more.
Deep learning algorithms have become the first choice as an approach to medical image analysis, face recognition, and emotion recognition. In this survey, several deep-learning-based approaches applied to breast cancer, cervical cancer, brain tumor, colon and lung cancers are studied and reviewed. Deep learning has been applied in almost all of the imaging modalities used for cervical and breast cancers and MRIs for the brain tumor. The result of the review process indicated that deep learning methods have achieved state-of-the-art in tumor detection, segmentation, feature extraction and classification. As presented in this paper, the deep learning approaches were used in three different modes that include training from scratch, transfer learning through freezing some layers of the deep learning network and modifying the architecture to reduce the number of parameters existing in the network. Moreover, the application of deep learning to imaging devices for the detection of various cancer cases has been studied by researchers affiliated to academic and medical institutes in economically developed countries; while, the study has not had much attention in Africa despite the dramatic soar of cancer risks in the continent. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>Example of convolution operation from [<a href="#B7-jimaging-06-00121" class="html-bibr">7</a>].</p>
Full article ">Figure 2
<p>Example of down sampling operation using max-pooling and average-pooling.</p>
Full article ">Figure 3
<p>Search criteria framework used for IEEEexplore, PubMed, Google Scholar and Science Direct engines to select papers for review.</p>
Full article ">
24 pages, 947 KiB  
Review
Applications of Computational Methods in Biomedical Breast Cancer Imaging Diagnostics: A Review
by Kehinde Aruleba, George Obaido, Blessing Ogbuokiri, Adewale Oluwaseun Fadaka, Ashwil Klein, Tayo Alex Adekiya and Raphael Taiwo Aruleba
J. Imaging 2020, 6(10), 105; https://doi.org/10.3390/jimaging6100105 - 8 Oct 2020
Cited by 26 | Viewed by 5888
Abstract
With the exponential increase in new cases coupled with an increased mortality rate, cancer has ranked as the second most prevalent cause of death in the world. Early detection is paramount for suitable diagnosis and effective treatment of different kinds of cancers, but [...] Read more.
With the exponential increase in new cases coupled with an increased mortality rate, cancer has ranked as the second most prevalent cause of death in the world. Early detection is paramount for suitable diagnosis and effective treatment of different kinds of cancers, but this is limited to the accuracy and sensitivity of available diagnostic imaging methods. Breast cancer is the most widely diagnosed cancer among women across the globe with a high percentage of total cancer deaths requiring an intensive, accurate, and sensitive imaging approach. Indeed, it is treatable when detected at an early stage. Hence, the use of state of the art computational approaches has been proposed as a potential alternative approach for the design and development of novel diagnostic imaging methods for breast cancer. Thus, this review provides a concise overview of past and present conventional diagnostics approaches in breast cancer detection. Further, we gave an account of several computational models (machine learning, deep learning, and robotics), which have been developed and can serve as alternative techniques for breast cancer diagnostics imaging. This review will be helpful to academia, medical practitioners, and others for further study in this area to improve the biomedical breast cancer imaging diagnosis. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>A simple artificial neural network (ANN) method on breast cancer [<a href="#B103-jimaging-06-00105" class="html-bibr">103</a>].</p>
Full article ">Figure 2
<p>A convolutional neural network (CNN)-based approach for screening mammography [<a href="#B125-jimaging-06-00105" class="html-bibr">125</a>].</p>
Full article ">Figure 3
<p>Structure of a generative adversarial networks (GANs) [<a href="#B142-jimaging-06-00105" class="html-bibr">142</a>].</p>
Full article ">
19 pages, 11380 KiB  
Review
Explainable Deep Learning Models in Medical Image Analysis
by Amitojdeep Singh, Sourya Sengupta and Vasudevan Lakshminarayanan
J. Imaging 2020, 6(6), 52; https://doi.org/10.3390/jimaging6060052 - 20 Jun 2020
Cited by 373 | Viewed by 26489
Abstract
Deep learning methods have been very effective for a variety of medical diagnostic tasks and have even outperformed human experts on some of those. However, the black-box nature of the algorithms has restricted their clinical use. Recent explainability studies aim to show the [...] Read more.
Deep learning methods have been very effective for a variety of medical diagnostic tasks and have even outperformed human experts on some of those. However, the black-box nature of the algorithms has restricted their clinical use. Recent explainability studies aim to show the features that influence the decision of a model the most. The majority of literature reviews of this area have focused on taxonomy, ethics, and the need for explanations. A review of the current applications of explainable deep learning for different medical imaging tasks is presented here. The various approaches, challenges for clinical deployment, and the areas requiring further research are discussed here from a practical standpoint of a deep learning researcher designing a system for the clinical end-users. Full article
(This article belongs to the Special Issue Deep Learning in Medical Image Analysis)
Show Figures

Figure 1

Figure 1
<p>A brief schematic of basics of XAI methods.</p>
Full article ">Figure 2
<p>Taxonomy of XAI methods.</p>
Full article ">Figure 3
<p>Attributions of VGG-16 with images from Imagenet using the methods implemented in [<a href="#B19-jimaging-06-00052" class="html-bibr">19</a>].</p>
Full article ">Figure 4
<p>Example of heat maps from a retinal OCT image [<a href="#B40-jimaging-06-00052" class="html-bibr">40</a>].</p>
Full article ">Figure 5
<p>Classification of explainability methods that are not attribution based.</p>
Full article ">
Back to TopTop