Deep Multi-Magnification Networks for Multi-Class Breast Cancer Image Segmentation

1. Introduction

Breast carcinoma is the most common cancer to be diagnosed for women [1]. Approximately 12% of women in the United States will be diagnosed with breast cancer during their lifetime [2]. Pathologists diagnose breast carcinoma based on a variety of morphologic features including tumor growth pattern and nuclear cytologic features. Pathologic assessment of breast tissue dictates the clinical management of the patient and provides prognostic information. Breast tissue from a variety of biopsies and surgical specimens is evaluated by pathologists. For example, patients with early-stage breast cancer often undergo breast-conserving surgery, or lumpectomy, which removes a portion of breast tissue containing the cancer [3]. To determine the completeness of the surgical excision, the edges of the lumpectomy specimen, or margins, are evaluated microscopically by a pathologist. Achieving negative margins (no cancer found touching the margins) is important to minimize the risk of local recurrence of the cancer [4]. Accurate analysis of margins by the pathologist is critical for determining the need for additional surgery. Pathologic analysis of margin specimens involves the pathologist reviewing roughly 20–40 histologic slides per case, and this process can be time-consuming and tedious. With the increasing capabilities of digitally scanning histologic glass slides, computational pathology approaches could potentially improve the efficiency and accuracy of this process by evaluating whole slide images (WSIs) of specimens [5].

Various approaches have been used to analyze WSIs. Most models include localization, detection, classification, and segmentation of objects (i.e. histologic features) in digital slides. Histopathologic features include pattern-based identification, such as nuclear features, cellular/stromal architecture, or texture. Computational pathology has been used in nuclei segmentation to extract nuclear features such as size, shape, and relationship between them [6, 7]. Nuclei segmentation is done by adaptive thresholding and morphological operations to find regions where nuclei density is high [8]. A breast cancer grading method can be developed by gland and nuclei segmentation using a Bayesian classifier and structural constraints from domain knowledge [9]. To segment overlapping nuclei and lymphocytes, an integrated active contour based on region, boundary, and shape is presented in [10]. A gland segmentation and classification method in prostate tissue is introduced where structural and contextual features from nuclei, cytoplasm, and lumen are used to classify artifact, normal gland, and cancer gland [11]. These nuclei-segmentation-based approaches are challenging because shapes of nuclei and structures of cancer regions may have large variations in the tissue samples captured in the WSIs.

Recently, deep learning, a type of machine learning, has been widely used for automatic image analysis due to the availability of large training datasets and the advancement of graphics processing units (GPUs) [12]. Deep learning models composed of deep layers with non-linear activation functions enable to learn more sophisticated features. Especially, convolutional neural networks (CNNs) learning spatial features in images have shown outstanding achievements in image classification [13], object detection [14], and semantic segmentation [15]. Fully Convolutional Network (FCN) in [15] developed for semantic segmentation, also known as pixel-wise classification, can understand location, size, and shape of objects in images. FCN is composed of an encoder and a decoder, where the encoder extracts low-dimensional features of an input image and the decoder utilizes the low-dimensional features to produce segmentation predictions. To improve segmentation predictions, SegNet introduces max-unpooling layers where max-pooling indices in an encoder are stored and used at the corresponding upsampling layers in a decoder [16]. Semantic segmentation has been used on medical images to automatically segment biological structures. For example, U-Net [17] is used to segment cells in microscopy images. U-Net architecture has concatenations transferring feature maps from an encoder to a decoder to preserve spatial information. This architecture has shown more precise segmentation predictions on biomedical images.

Deep learning has recently received high attention in the computational pathology community [18, 19, 20]. Investigators have shown automated identification of invasive breast cancer detection in WSIs by using a simple 3-layer CNN [21]. A method of classifying breast tissue slides to invasive cancer or benign by analyzing stroma regions using CNNs is described in [22]. More recently, a multiple-instance-learning-based CNN achieves 100% sensitivity where the CNN is trained by 44,732 WSIs from 15,187 patients [23]. The availability of public pathology datasets contributes to develop many deep learning approaches for computational pathology. For example, a breast cancer dataset to detect lymph node metastases was released for the CAMELYON challenges [24, 25] and several deep learning techniques to analyze breast cancer datasets are developed [26, 27, 28].

One challenge of using deep learning on WSIs is that the size of a single, entire WSI is too large to be processed into GPUs. Images can be downsampled to be processed by pretrained CNNs [29, 30] but critical details needed for clinical diagnosis in WSIs would be lost. To solve this, patch-based approaches are generally used instead of slide-level approaches. Here, patches are extracted from WSIs to be processed by CNNs. A patch-based process followed by a multi-class logistic regression to classify in slide-level is described in [31]. The winner of the CAMELYON16 challenge uses the Otsu thresholding technique [32] to extract tissue regions and trains a patch-based model to classify tumor and non-tumor patches [26]. To increase the performance, class balancing between tumor and non-tumor patches and data augmentation techniques such as rotation, flip, and color jittering are used in [27]. The winner of the CAMELYON17 challenge additionally develops patch-overlapping strategy for more accurate predictions [28]. In [33], a patch is processed with an additional larger patch including border regions in the same magnification to segment subtypes in breast WSIs. Alternatively, Representation-Aggregation CNNs to aggregate features generated from patches in WSIs are developed to share representations between patches [34, 35]. The main limitations of these patch-based approaches extracted from a single magnification are (1) the field-of-view becomes narrow and (2) morphological features from a lower magnification are not used.

Therefore, we develop segmentation CNNs by inputting a set of patches from multiple magnifications to increase the field-of-view and to provide more information from other magnifications. Figure 1 introduces the main difference between a Deep Single-Magnification Network (DSMN) and a Deep Multi-Magnification Network (DMMN) for tissue segmentation of whole slide images. An input to a DSMN in Figure 1(a) is a patch from a single magnification which limits a field-of-view. An input to a DMMN in Figure 1(b) is a set of patches from multiple magnifications allowing a wider field-of-view. High magnification patches provide details at the cellular level, such as nuclear features, whereas low magnification patches demonstrate distribution of tissue types and architectural growth patterns of benign and malignant processes.

There are several works using multiple magnifications to analyze images from tissue samples. A multi-input multi-output CNN is presented by analyzing an input image in multiple resolutions to segment cells in fluorescence microscopy images [36]. Similarly, a stain-aware multi-scale CNN is further designed for instance cell segmentation in histology images [37]. To segment tumor regions in the CAMELYON dataset [24], a binary segmentation CNN is described in [38]. In this work, four encoders for different magnifications are implemented but only one decoder is used to generate the final segmentation predictions. More recently, a CNN architecture composed of three expert networks for different magnifications, a weighting network to automatically select weights to emphasize specific magnifications based on input patches, and an aggregating network to produce final segmentation predictions is developed in [39]. Here, feature maps are not shared between the three expert networks until the last layer which can limit utilizing feature maps from multiple magnifications. Architectures designed in [38] and [39] center-crop feature maps in lower magnifications and then upsample the cropped feature maps to match the size and magnification during concatenations which can also limit the usage of feature maps on cropped boundary regions in lower magnifications.

In this paper, we present a Deep Multi-Magnification Network (DMMN) to accurately segment multiple subtypes in images of breast tissue. Our DMMN architecture has multiple encoders, multiple decoders, and multiple concatenations between decoders to have richer feature maps in intermediate layers. To fully utilize feature maps in lower magnifications, we center-crops intermediate feature maps during concatenations. By concatenating intermediate feature maps in each layer, feature maps from multiple magnifications can be used to produce accurate segmentation predictions. To train our DMMN, we partially annotate WSIs, similarly done as [40], to reduce the burden of annotations. Our DMMN model trained by our partial annotations can learn not only features of each subtype, but also morphological relationship between subtypes especially transitions from one subtype to another subtype on boundary regions, which leads to outstanding segmentation performance. We test our multi-magnification model on two breast datasets and observe that our model consistently outperforms other architectures. Our method can be used to automatically segment cancer regions on breast images to assist in diagnosis of patients’ status and to decide future treatments.

2. Proposed Method

Figure 2 shows the block diagram of our proposed method. Our goal is to segment multiple subtypes on breast images using our Deep Multi-Magnification Network (DMMN). First of all, manual annotation is done on the training dataset with C classes. Here, this annotation is done partially for an efficient and fast process. To train our multi-class segmentation DMMN, patches are extracted from whole slide images (WSIs) and the corresponding annotations. Before training our DMMN with the extracted patches, we use elastic deformation [17, 41] to multiply patches belonging to rare classes to balance the number of annotated pixels between classes. After the training step is done, the model can be used for multi-class segmentation of breast cancer images. We have implemented our system in PyTorch [42].

3. Experimental Results

Two breast datasets, Dataset-I and Dataset-II, were used to train and evaluate various multi-class tissue segmentation methods. Dataset-I is composed of whole slide images (WSIs) with Triple-Negative Breast Cancer (TNBC) containing high grade invasive ductal carcinoma (IDC). Dataset-II is composed of WSIs from lumpectomy and breast margins containing IDC and ductal carcinoma in situ (DCIS) of various histologic grades. All WSIs in Dataset-I and Dataset-II were from different patients, were hematoxylin and eosin (H&E) stained, and were digitized at Memorial Sloan Kettering Cancer Center. Dataset-I was digitized by Aperio XT where microns per pixel (MPP) in 20× is 0.4979 and Dataset-II was digitized by Aperio AT2 where MPP in 20× is 0.5021. WSIs in Dataset-I were partially annotated by two pathologists and WSIs in Dataset-II were partially annotated by another pathologist.

Thirty two WSIs from Dataset-I were used to train and validate segmentation models. The number of whole slide images and the number of patches before and after class balancing to train and validate the models are shown in Table 1. No images from Dataset-II were used during training. In our work, only 5.34% of pixels of training WSIs were annotated. Our models can predict 6 classes (C = 6) which are carcinoma, benign epithelial, background, stroma, necrotic, and adipose. Note that background is defined as regions which are not tissue. To balance the number of annotated pixels between classes, we empirically set r₂ = 10, r₁ = 2, r₅ = 3, r₃ = 1, r₄ = 0, and r₆ = 0 where r₁, r₂, r₃, r₄, r₅, and r₆ are rates of elastic deformation of carcinoma, benign epithelial, background, stroma, necrotic, and adipose, respectively. Benign epithelial was selected as the highest class order followed by carcinoma, necrotic, and background, because we want to accurately segment carcinoma regions and separate benign epithelial to reduce false segmentation. Figure 5 shows the number of annotated pixels between classes are balanced using elastic deformation. We trained two Deep Single-Magnification Networks (DSMNs), SegNet [16] architecture and U-Net [17] architecture, and four Deep Multi-Magnification Networks (DMMNs), Single-Encoder Single-Decoder (DMMN-S2) architecture, Multi-Encoder Single-Decoder (DMMN-MS) architecture, Multi-Encoder Multi-Decoder Single-Concatenation (DMMN-M2S) architecture, and our proposed Multi-Encoder Multi-Decoder Multi-Concatenation (DMMN-M3) architecture. The number of convolutional layers, the number of downsampling and upsampling layers, and the number of channels are kept the same between the SegNet architecture used in this experiment and the original U-Net architecture. Also, the number of channels on DMMN-MS, DMMN-M2S, and DMMN-M3 are reduced by a factor of 2 from an original U-Net architecture. Table 2 lists the models we compared, the number of trainable parameters, and segmentation time, where the segmentation time was measured on a whole slide image in Figure 7 whose size is 53,711 × 38,380 pixels with 31,500 patches using a single NVIDIA GeForce GTX TITAN X GPU.

We processed 55 testing images from Dataset-I and 34 testing images from Dataset-II to evaluate various models. Figure 6 depicts multi-class segmentation predictions in a WSI from Dataset-I by SegNet [16] architecture, U-Net [17] architecture, DMMN-S2 architecture, DMMN-MS architecture, DMMN-M2S architecture, and our proposed DMMN-M3 architecture, and Figure 7 depicts multi-class segmentation predictions in a patch containing invasive ductal carcinoma (IDC) with size of 1024×1024 pixels in 10× magnification from the WSI in Figure 6. Similarly, Figure 8 depicts multi-class segmentation predictions in a WSI from Dataset-I, Figure 9 depicts multi-class segmentation predictions in a patch containing benign epithelial from the WSI in Figure 8, Figure 10 depicts multi-class segmentation predictions in a WSI from Dataset-II, and Figure 11 depicts multi-class segmentation predictions in a patch containing ductal carcinoma in situ (DCIS) from the WSI in Figure 10. Tissue subtypes are labeled in distinct colors such as carcinoma in red, benign epithelial in blue, background in yellow, stroma in green, necrotic in gray, and adipose in orange. White regions in Figures 6(b), 7(b), 8(b), 9(b), 10(b), and 11(b) are unlabeled. The Otsu thresholding technique [32] was used to extract patches only on foreground regions of the WSIs from Dataset-II digitized from a different scanner because we observed that models are sensitive to background noise leading mis-segmentation on background regions. White regions in Figure 10(c-h) are removed by the Otsu technique [32].

We evaluated our predictions numerically using intersection-over-union (IOU), recall, and precision which are defined as:

where N_TP, N_FP, and N_FN are the number of pixels for true-positive, false-positive, and false-negative, respectively. Tables 3 and 4 shows mean IOU (mIOU), mean recall (mRecall), and mean precision (mPrecision) on Dataset-I and Dataset-II, respectively, where mIOU is used as our main evaluation metric to select the best performing model. Figures 12 and 13 show confusion matrices from the models on Dataset-I and Dataset-II, respectively. Note that necrotic, adipose, and background were excluded for evaluating Dataset-II because (1) Dataset-II does not contain large necrotic regions and (2) most of adipose and background regions were not segmented due to the Otsu technique [32].

Based on our visual and numerical evaluations on Dataset-I, both DSMNs had blocky boundaries between subtypes, shown in Figures 7(c,d) and 9(c,d) due to their narrow field-of-view. DMMN-S2 also had blocky boundaries between subtypes, shown in Figures 7(e) and 9(e), because patches from multiple magnifications are concatenated early in the model so various features from multiple magnifications could not be fully extracted. These blockly predictions led to low mIOU, low mRecall, and low mPrecision in Table 3. DMMN-MS and DMMN-M2S had smoother boundaries between subtypes, but they did not have consistent predictions throughout subtypes. For example, DMMN-MS and DMMN-M2S cannot predict necrotic successfully according to Figure 12(d,e). Our proposed DMMN-M3 has shown accurate predictions throughout all subtypes, shown in Figure 12(f), leading to the best mIOU, mRecall, and mPrecision in Table 3.

Our models were trained on Dataset-I and we kept aside images in Dataset-II, annotated by a different pathologist, for our testing set. We still observed blocky boundaries on predictions done by SegNet, U-Net, and DMMN-S2 on Dataset-II, shown in Figure 11(c,d,e). We noticed predictions by DMMN-M2S were not successful where large regions are falsely segmented as benign epithelial in Figures 10(g) and 11(g). Figure 11(h) demonstrates that our proposed DMMN-M3 segments subtypes with smoother and clearer boundaries.

According to Figure 13(f), DMMN-M3 segmented many carcinoma pixels as benign epithelial on Dataset-II causing low mIOU in Table 4. We observed that well-differentiated carcinomas were segmented as benign epithelial by DMMN-M3, shown in Figure 14. Well-differentiated carcinomas, known to be morphologically similar to benign cells, were not presented in Dataset-I composed of high grade Triple-Negative Breast Cancer (TNBC). DMMN-M3 trained by Dataset-I alone would not be able to learn morphological features of well-differentiated carcinomas, causing inaccurate segmentation on them in Dataset-II. We numerically evaluated the 6 models on 29 WSIs in Dataset-II by excluding 5 images with well-differentiated carcinomas, and Table 5 shows DMMN-M3 outperforms other methods for higher histologic grades based on mIOU, mRecall, and mPrecision. Although our current DMMN-M3 model may be challenged to segment well-differentiated carcinomas due to their absence in the training set, our proposed model, with additional training annotations of well-differentiated carcinomas, can be used to successfully segment breast whole slide images to assist pathologists for breast cancer assessments.

4. Conclusion

We described a Deep Multi-Magnification Network (DMMN) for an accurate multi-class tissue segmentation of whole slide images. Our model is trained by partially-annotated images to reduce time and effort for annotators. Although the annotation was partially done, our model was able to learn not only spatial characteristics within a class but also spatial relationship between classes. Our DMMN architecture looks at all 20×, 10×, and 5× magnifications to have a wider field-of-view to make more accurate predictions based on feature maps from multiple magnifications. We were able to improve previous DMMNs by transferring intermediate feature maps from decoders in 10× and 5× to a decoder in 20× to enrich feature maps. Our implementation achieved outstanding segmentation performance on breast datasets that can be used to decide patients’ future treatment. One main challenge we encountered is that our model may not successfully segment well-differentiated carcinomas presented in breast images because well-differentiated carcinomas were not included in training annotation. We also observed that our model can be sensitive to background noises potentially leading to mis-segmentation on background regions if whole slide images are digitized by other scanners. In the future, we plan to develop a more accurate DMMN model where various cancer structures and background noise patterns are included during training.