SRGAN Assisted Encoder-Decoder Deep Neural Network for Colorectal Polyp Semantic Segmentation

Colorectal cancer is growing rapidly throughout the world. As per the recent survey, colon cancer is diagnosed in 1,931,590 individuals, which is 10% of the total cancer cases [1]. Adenocarcinomas cause approximately 96% of colorectal cancers, and intestinal stromal tumors (GISTs). A polyp is a tiny development of excess cells, typically expands in the rectum or colon. All of the polyps may not turn into cancer cells, but it might take several years for a polyp to become malignant. The polyps can be detected through the standard biomedical imagining procedures such as Virtual colonoscopy, colon capsule endoscopy (CCE), and optical colonoscopy (OC) [2]. However, the wide variety of sizes and shapes of polyps and the minimal vision of the colon make it challenging to the endoscopists to maintain continual and regular assessments on detection and diagnosis of the polyps. In addition, recent scientific research studies have revealed that (8-37%) of polyps are missed as a result of high background object resemblance.

Over decades, computer vision-based techniques have been proposed for the automatic identification of polyps. Some researchers suggested utilizing the shape features and texture attributes combined with common classifiers for polyp detection. However, these techniques still struggle with a high False-positive rate. Recently, Deep CNNs have shown stupendous success in clinical image analysis, which can aid the colonoscopist to lower their polyp miss-rates in a real-time clinical atmosphere. The deep learning-based Encoder-Decoder network is an inexpensive tool used to segment a colorectal polyp effectively from the ubiquitously available histological images. However, automatic Polyp segmentation is challenging because of high variations in polyp appearance (texture, color, high interclass variations in size and shape), the presence of other endoluminal scene structures (e.g., colon walls and air bubbles), and the small multiple adenomas. Hence, developing an automated polyp segmentation system is necessary to support the gastroenterologists to identify and resect the polyps. The result is GAN-assisted Encoder-Decoder architecture. The contributions of this work are as follows.

1. We propose a novel GAN-assisted encoder-decoder architecture that is effective than compared with SOTA approaches.

2. We present an intelligent decoder design to ensure explicit contour preservation through GAN-assisted up-sampling processes at each decoder stage.

3. Our experimental results demonstrate noticeably better performance compared to current SOTA methods.

The remainder of the paper will be organized as follows. Section 2 explains the relevant work and the gap identification. Section 3 discusses the overview of CNN architecture, Encoder-Decoder network, and GANs. Section 4 introduces the suggested model, and Section 5 discusses the experimental results and analysis with SOTA techniques. Finally, the last section concludes the study in section 6.

CNN [11] is a conventional deep neural network, basically used for object classification problems. The name convolution came from the linear mathematical operations between two matrices. CNN is composed by number of convolutional, pooling, non-linearity layers and is finally attached with a fully connected (FC) layer. The convolutional (conv) and FC layers have learnable parameters, but non-linearity and pooling layers do not have any learnable parameters. Generally, CNN concentrates on image classification problems, where input is an image (matrix) as well as output is one labeled image.

1.png

Figure 1. CNN architecture [12]

As depicted in Figure 1, in a CNN, a series of convolutional procedures are executed to generate the high-level feature maps from the given input image. Traditionally, the first Conv-Layer captures the Low-level features such as contours (edges), gradient orientation, color, etc. However, if the network is mode deeper, the additional Conv Layers adapt the high-level features as well, offering us the wholesome understanding of the processed images. After a series of Conv operations, pooling is performed, responsible for lowering the spatial size of the Convolved features.

In general, max-pooling is performed for image dimensionality reduction of the feature maps for suppressing the noise. For the given input feature map (Tensor), max-pooling gives the optimum value for the portion of the image overlapped by the kernel. After repeating the max pooling operations number of times, as shown in Figure 1, the design succeeds in recognizing the important features. After that, the final output is flattening as well as feed to an FCN for classification purposes. But in biomedical classification problems, along with the identification of disease is there or not, yet also to center the location of the abnormality. To accomplish this, there is a need for Encoder-Decoder based networks.

3.1 Encoder-decoder architecture

2.png

Figure 2. Encoder-Decoder Architecture [13]

Encoder-Decoder network evolved from the traditional convolutional neural network and can localize and identify contours of the objects by classifying each pixel in the image. As shown in Figure 2, in the architecture, the encoder (Left) part is also known as the contracting Path, which is comprised by the general convolutional procedure, and the Decoder (Right) part is also known as expansive Path, performs the Deconvolutions (Transposed convolutions), which seem to be the reverse of the encoder part.

3.2 Generative adversarial networks

Generative Adversarial Networks [14] (GAN) provide a path to innovative domain-specific data augmentation. Traditionally GAN [15] are a strategy to generative modeling making use of deep learning techniques. GANs use two neural networks Generator (G) and Discriminator (D). These two networks (G, D) play a min-max game where one attempts to outsmart the other network. The Generator is remained to be idle while the Discriminator is trained. In this phase, only forward propagation is done. The Discriminator is trained on original data for (n) number of epochs and sees if it can properly forecast them as original (real). Also, in this phase, the Discriminator is likewise trained on the phony (fake) generated data from the Generator (G) and saw if it can appropriately classify them as phony. The discriminator network (D) target is to determine which outputs it obtains have been artificially produced.

3.png

Figure 3. Basic Working model of GAN [16]

The goal of the generator network is to create synthesized data artificially and try to fool the Discriminator network. The Generator network is trained while the Discriminator remains idle. After the Discriminator network is trained by the phony data generated by the Generator, we can get its classification predictions and again use the outcomes to train the Generator network to improve from the previous state (more accurate fake data). The generated fake data is given to the discriminator network along with the original data. The Discriminator needs to classify the input images and returns a probability depend upon the loss function, between 0 and 1. This value represents the authenticity of each image. One (1) associate with original and zero (0) associates with phony images. If the loss is more, the output generated by the Discriminator (D) is again given back to the Generator (G) as input for fine-tuning. This process is repeated until the loss is minimized (The Discriminator fails to identify the fake data). Figure 3 represents the working functionality of a GAN.

The Generator G(z) attempts to reduce this loss function(min-max) value while the Discriminator tries to maximize it, which seems like a min-max game. Thus, Eq. (1) represents the min-max loss function, where Eq. (2) represents the loss at the Discriminator network D(x).

$ Min~\text{max}\left( D,G \right)$                (1)

${{E}_{x}}\left[ \log \left( D\left( x \right) \right) \right]+{{E}_{z}}\left[ \log \left( 1-D\left( G\left( z \right) \right) \right) \right]$                (2)

While the Discriminator classifies the original and phony images, if the loss is more with respect to the Generator, it needs to do back propagation in the Generator network. Both the networks are working adversarial to the other to generate the synthetic data, which can pass for original input data. Initially, GANs are used to generate fake images, which is similar to the original image. Researchers proposed that GANs can also be used mainly in the process of data augmentation. CycleGAN [17] is used to improve the generalizability in CT segmentation tasks. In Ref. [18], GANs are used for Semi-Supervised Semantic Segmentation, and SeGAN was proposed by Xue et al. [19] for Medical Image Segmentation.

In Ref. [20], SRGAN was used for image super-resolution, which surpassed the traditional up-sampling methods (nearest neighbor, bilinear interpolation, bicubic interpolation). Most of the GANs are used to generate the augmented image data used in the training phase of a CNN. To the best of our knowledge, this is the first study to use SRGAN [20] to substitute the traditional Up-sampling process at the decoder phase. The accuracy of the polyp segmentation depends upon the up-sampling process. Whereas the existing up-sampling methods have limitations in generating the accurate high-level segmented masks for the polyps. These drawbacks inspired us to develop an SRGAN assisted automatic Encoder-Decoder based model for accurate polyp detection and diminish the polyp miss rate from the colonoscopic images.

5.1 Dataset

We employ the two basic requirements when taking into consideration a dataset. First is the dataset must be publicly available and properly annotated. We used the publicly available colonoscopy CVC-ColonDB [26] dataset, which consists of a total of 380 colonoscopy frames of 574×500 pixel resolution has been generated extracted from 15 different colonoscopy videos. The sequences are from regular colonoscopies and were selected to represent as much variation in polyp appearance as possible. The CVC-ColonDB dataset is composed of frames from the 15 different video sequences containing at least one polyp. The whole dataset contains different shapes and sizes of polyps. In addition to each polyp image, the appropriate ground truth of the image is provided, which consists of a binary mask representing the area covered by the polyp in the image. We Also consider another benchmark dataset Warwick–QU [27, 28], to compare the efficiency with our previous work with the proposed model. The dataset primarily consists of 85 images in the training part and 80 images in the testing part.

5.2 Data augmentation

In real-time, due to the lack of medical image data, deep learning models face considerable difficulty in medical imagining compared to other object detection problems. Image Data augmentation plays a decisive role in enhancing the number of colon polyp images. This fixes the data deficiency problem, boosts the model's efficiency, and helps to diminish the under-fitting. The images are taken from the CVC-ColonDB dataset (380 images) are have a fixed size resolution of (574×500). After removing the canvas around each image, the complete dataset images are resized into (512 X 512). First, we applied the horizontal & vertical flip on the original dataset images to generate 760 augmented images. Then we applied various data augmentation techniques such as random contrast, random brightness, and random rotation (by 90°, 180°, 270°) to generate 6840 augmented images. To fit and evaluate the proposed deep learning model, the images are divided into training dataset consists of 5472 (80%) images and testing dataset consists of 1368 images (20%). There is no intersection of images between the train and the test dataset. And on the Warwick–QU dataset, the training dataset was only augmented.

5.3 Results and Discussion

There are some basic metrics to evaluate the performance of the proposed model’s implementation on the benchmark colorectal polyp datasets. We considered the essential statistical validation metric, the Dice similarity coefficient (DSC) to evaluate the segmentation accuracy of the model. These metric ranges between (0-1), where zero (0) indicates no overlap and one (1) indicates perfectly overlapping. Eq. (7) describes the dice coefficient at the pixel level. For segmentation tasks, we need to measure the dice index at the object level described in Eqns. (8) and (9).

$Dice\left( G,S \right)=2*\frac{\left| G\cap S \right|}{\left| G \right|+\left| S \right|}$                (7)

$\operatorname{Dice}_{o b j}(G, S)=\frac{1}{2} *\left[\begin{array}{l}n_{G} \gamma^{*} \operatorname{Dice}\left(G_{i}, S_{*}\left(G_{i}\right)\right) \\ \sum_{i=1}^{n}+\sum_{j=1}^{n_{S}} \sigma_{j} * \operatorname{Dice}\left(G_{*}\left(S_{j}\right), S_{j}\right)\end{array}\right]$                   (8)

$\begin{align}  & {{\gamma }_{i}}={}^{\left| {{G}_{i}} \right|}/{}_{\sum\nolimits_{p=1}^{{{n}_{G}}}{\left| {{G}_{p}} \right|}}, \\  & {{\sigma }_{j}}={}^{\left| {{S}_{j}} \right|}/{}_{\sum\nolimits_{q=1}^{{{n}_{S}}}{\left| {{S}_{q}} \right|}} \\ \end{align}$                 (9)

After establishing the model, it is evaluated on the CVC-ColonDB and Warwick–QU benchmark datasets. Due to the existence of the GAN at each level of the decoder network, the proposed model requires high computational power and a huge amount of time for training the network. But it generates more accurate feature maps in the up-sampling process. Table 1 describes the performance evaluation of the various SOTA models with respect to the Dice score on CVC-ColonDB.

To review the performance of the suggested model on the benchmark Warwick–QU dataset, we considered the object-level Dice index (D_obj) for segmentation accuracy and the Hausdorff distance (Eq. (10)) to measure the shape similarity among the ground Truth (G) and segmented images(S). The smaller Hausdorff distance represents the maximal similarity among the borders of S and G.

Table 1. Comparison of SOTA models with the proposed model over CVC-ColonDB

$\text{H}\left( \text{G},\text{S} \right)=\max \left\{ \begin{align}  & \underset{\text{x}\in \text{G}}{\mathop{\sup }}\,\underset{\text{y}\in \text{S}}{\mathop{\inf }}\,\text{d}\left( \text{x},\text{y} \right), \\  & \underset{\text{y}\in \text{s}}{\mathop{\sup }}\,\underset{\text{x}\in \text{G}}{\mathop{\inf }}\,\text{d}\left( \text{x},\text{y} \right) \\ \end{align} \right\}$                (10)

${{H}_{obj}}\left( G,S \right)=\frac{1}{2}*\left[ \begin{align}  & \sum\nolimits_{i=1}^{{{n}_{G}}}{{{\gamma }_{i}}}*H\left( {{G}_{i}},{{S}_{*}}\left( {{G}_{i}} \right) \right)+ \\  & \sum\nolimits_{j=1}^{{{n}_{S}}}{{{\sigma }_{j}}}*H\left( {{G}_{*}}\left( {{S}_{j}} \right),{{S}_{j}} \right) \\\end{align} \right]$                 (11)

Object wise Hausdorff distance (H_obj) is applied as shown in Eq (11) to find the object-wise contour-based shape similarity. The model achieved a dice score of 0.952 on Part A and 0.924 on Part B and H_obj of 76.142 (part A), 81.245(part B), which are more accurate than existing SOTA methods.