Nothing Special   »   [go: up one dir, main page]

You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Sign in to use this feature.

Years

Between: -

Subjects

remove_circle_outline
remove_circle_outline
remove_circle_outline
remove_circle_outline

Journals

Article Types

Countries / Regions

Search Results (10)

Search Parameters:
Keywords = xBD dataset

Order results
Result details
Results per page
Select all
Export citation of selected articles as:
21 pages, 3402 KiB  
Article
A CNN- and Transformer-Based Dual-Branch Network for Change Detection with Cross-Layer Feature Fusion and Edge Constraints
by Xiaofeng Wang, Zhongyu Guo and Ruyi Feng
Remote Sens. 2024, 16(14), 2573; https://doi.org/10.3390/rs16142573 - 13 Jul 2024
Cited by 2 | Viewed by 1096
Abstract
Change detection aims to identify the difference between dual-temporal images and has garnered considerable attention over the past decade. Recently, deep learning methods have shown robust feature extraction capabilities and have achieved improved detection results; however, they exhibit limitations in preserving clear boundaries [...] Read more.
Change detection aims to identify the difference between dual-temporal images and has garnered considerable attention over the past decade. Recently, deep learning methods have shown robust feature extraction capabilities and have achieved improved detection results; however, they exhibit limitations in preserving clear boundaries for the identified regions, which is attributed to the inadequate contextual information aggregation capabilities of feature extraction, and fail to adequately constrain the delineation of boundaries. To address this issue, a novel dual-branch feature interaction backbone network integrating the CNN and Transformer architectures to extract pixel-level change information was developed. With our method, contextual feature aggregation can be achieved by using a cross-layer feature fusion module, and a dual-branch upsampling module is employed to incorporate both spatial and channel information, enhancing the precision of the identified change areas. In addition, a boundary constraint is incorporated, leveraging an MLP module to consolidate fragmented edge information, which increases the boundary constraints within the change areas and minimizes boundary blurring effectively. Quantitative and qualitative experiments were conducted on three benchmarks, including LEVIR-CD, WHU Building, and the xBD natural disaster dataset. The comprehensive results show the superiority of the proposed method compared with previous approaches. Full article
Show Figures

Figure 1

Figure 1
<p>The overall structure diagram of the proposed model.</p>
Full article ">Figure 2
<p>Main structure of backbone network based on CNN and Transformer.</p>
Full article ">Figure 3
<p>Cross-layer feature fusion module.</p>
Full article ">Figure 4
<p>Channel feature interaction module.</p>
Full article ">Figure 5
<p>Visualization results of different methods on LEVIR dataset.</p>
Full article ">Figure 6
<p>Visualization results of different methods on WHU dataset.</p>
Full article ">Figure 7
<p>Visualization results of different methods on xBD datasets.</p>
Full article ">Figure 8
<p>Visualization results of different methods on LEVIR, WHU Building, and xBD datasets, including TP (white), TN (black), FP (red), and FN (green).</p>
Full article ">Figure 8 Cont.
<p>Visualization results of different methods on LEVIR, WHU Building, and xBD datasets, including TP (white), TN (black), FP (red), and FN (green).</p>
Full article ">Figure 9
<p>Comparison among qualitative results highlighting the contribution of the edge constraint module, including TP (white), TN (black), FP (red), and FN (green).</p>
Full article ">
36 pages, 57413 KiB  
Article
BD-SKUNet: Selective-Kernel UNets for Building Damage Assessment in High-Resolution Satellite Images
by Seyed Ali Ahmadi, Ali Mohammadzadeh, Naoto Yokoya and Arsalan Ghorbanian
Remote Sens. 2024, 16(1), 182; https://doi.org/10.3390/rs16010182 - 31 Dec 2023
Cited by 6 | Viewed by 2375
Abstract
When natural disasters occur, timely and accurate building damage assessment maps are vital for disaster management responders to organize their resources efficiently. Pairs of pre- and post-disaster remote sensing imagery have been recognized as invaluable data sources that provide useful information for building [...] Read more.
When natural disasters occur, timely and accurate building damage assessment maps are vital for disaster management responders to organize their resources efficiently. Pairs of pre- and post-disaster remote sensing imagery have been recognized as invaluable data sources that provide useful information for building damage identification. Recently, deep learning-based semantic segmentation models have been widely and successfully applied to remote sensing imagery for building damage assessment tasks. In this study, a two-stage, dual-branch, UNet architecture, with shared weights between two branches, is proposed to address the inaccuracies in building footprint localization and per-building damage level classification. A newly introduced selective kernel module improves the performance of the model by enhancing the extracted features and applying adaptive receptive field variations. The xBD dataset is used to train, validate, and test the proposed model based on widely used evaluation metrics such as F1-score and Intersection over Union (IoU). Overall, the experiments and comparisons demonstrate the superior performance of the proposed model. In addition, the results are further confirmed by evaluating the geographical transferability of the proposed model on a completely unseen dataset from a new region (Bam city earthquake in 2003). Full article
(This article belongs to the Special Issue Artificial Intelligence for Natural Hazards (AI4NH))
Show Figures

Graphical abstract

Graphical abstract
Full article ">Figure 1
<p>Number of image pairs in each disaster type per group in the xBD dataset.</p>
Full article ">Figure 2
<p>Geographical distribution and type of disasters in the xBD dataset. The corresponding number of building damage classes in each disaster is shown in pie charts. Each disaster is pinned on the map by a relative icon that shows its type. On the pie charts, green, light yellow, orange, and red show no-damage, minor-damage, major-damage, and destroyed classes, respectively. Black represents the unclassified buildings.</p>
Full article ">Figure 3
<p>Location of Bam city in Kerman province of Iran (<b>top left</b>), the municipality blocks along with the building footprints overlaid with the pre-disaster satellite image of the study region (<b>bottom left</b>), and the post-disaster satellite image covered by some sample regions showing damage (red) and no-damaged (green) buildings (<b>right</b>).</p>
Full article ">Figure 4
<p>Overview of pre-disaster image acquired from the study area (<b>left</b>) and three samples of pre-disaster (<b>left squares</b>) and post-disaster (<b>middle squares</b>) images along with their corresponding ground truth data (<b>right squares</b>). Green and red show the buildings with no-damage and damaged, respectively.</p>
Full article ">Figure 5
<p>Ground truth map of the study area, along with six zoomed patches for better visualization. Buildings in red, green, and black demonstrate damaged, not damaged, and unclassified classes, respectively.</p>
Full article ">Figure 6
<p>Augmentation techniques were applied to input images for further regularizing the model. On the left, pre- and post-disaster images are displayed, and other images are the outputs of specific augmentations.</p>
Full article ">Figure 7
<p>The overall workflow of our study consists of three major parts: (1) Data Preparation, (2) Damage assessment, which includes Localization and Classification models, and (3) Transferability Analysis.</p>
Full article ">Figure 8
<p>Schematic diagram of a UNet for our building localization stage, which shows different components of our UNet model, including encoder and decoder paths, skip connections, and pre-trained backbones. The output of this model is a binary segmentation map of building footprints.</p>
Full article ">Figure 9
<p>Schematic diagram of a dual-branch UNet network for the building damage assessment method. Each of the pre- and post-disaster images enters a separate branch with shared weights, and the output feature maps are concatenated and inserted into the segmentation head. The output of this stage is a per-building damage classification map.</p>
Full article ">Figure 10
<p>Diagram of the Selective Kernel Module which shows its three stages of Split, Fuse, and Select, for a sample two-branch selective kernel.</p>
Full article ">Figure 11
<p>(<b>a</b>) Interpretation of the confusion matrix into useful evaluation metrics; and (<b>b</b>) visual comparison of IoU and F1-score based on [<a href="#B69-remotesensing-16-00182" class="html-bibr">69</a>].</p>
Full article ">Figure 12
<p>Demonstration of the image preparation steps, i.e., random 256 × 256 patch extraction, augmentation, and arrangement of masks for the damage assessment stage and the corresponding weights for each class, obtained based on their proportional number of samples.</p>
Full article ">Figure 13
<p>Schematic diagram showing various architectures that were used for comparison.</p>
Full article ">Figure 14
<p>Boxplots of evaluation metrics used to compare the localization models.</p>
Full article ">Figure 15
<p>Visual comparison of different localization methods used in our paper.</p>
Full article ">Figure 16
<p>Comparison of the classification models in the damage assessment stage. Dashed and connected lines are for training and validation sets, respectively. Models 1 to 4 are colored red, cyan, yellow, and black, respectively.</p>
Full article ">Figure 17
<p>Visual comparison of damage assessment results from four models used in our study.</p>
Full article ">Figure 18
<p>Building damage assessment transferability results for seven regions in the Bam earthquake dataset (<b>a</b>–<b>g</b>). Pre- and post-disaster images are shown on the left. Probability maps for no-damage and destroyed classes are shown in the middle with the relevant color map. The building damage classification map is demonstrated on the right, and the ground truth data for each building is overlaid with green or red borders around it.</p>
Full article ">Figure 19
<p>Visualizing imaging properties in 19 disaster events of the xBD dataset, before and after the disasters.</p>
Full article ">Figure 20
<p>Distribution of influencing parameters and the thresholds specified for conducting each experiment. The number of images for each experiment is presented next to the curly brackets. Subfigures are displaying (<b>a</b>) histogram of pre-disaster Sun elevation angles, (<b>b</b>) off-nadir angle vs. ground resolution, (<b>c</b>) histogram of differences between pre- and post-disaster Sun elevation angles, (<b>d</b>) histogram of differences between pre- and post-disaster GSDs, (<b>e</b>) histogram of differences between pre- and post-disaster off-nadir angles. The blue circles are showing the specified threshold which has been used for further analysis.</p>
Full article ">Figure 21
<p>The relation between different parameters and the “localization” quality metrics.</p>
Full article ">Figure 22
<p>The relation between different relative parameters and the “classification” quality metrics.</p>
Full article ">Figure 23
<p>The relation between “disaster types” and the “classification” quality metrics.</p>
Full article ">
23 pages, 21129 KiB  
Article
SegDetector: A Deep Learning Model for Detecting Small and Overlapping Damaged Buildings in Satellite Images
by Zhengbo Yu, Zhe Chen, Zhongchang Sun, Huadong Guo, Bo Leng, Ziqiong He, Jinpei Yang and Shuwen Xing
Remote Sens. 2022, 14(23), 6136; https://doi.org/10.3390/rs14236136 - 3 Dec 2022
Cited by 9 | Viewed by 3320
Abstract
Buildings bear much of the damage from natural disasters, and determining the extent of this damage is of great importance to post-disaster emergency relief. The application of deep learning to satellite remote sensing imagery has become more and more mature in monitoring natural [...] Read more.
Buildings bear much of the damage from natural disasters, and determining the extent of this damage is of great importance to post-disaster emergency relief. The application of deep learning to satellite remote sensing imagery has become more and more mature in monitoring natural disasters, but there are problems such as the small pixel scale of targets and overlapping targets that hinder the effectiveness of the model. Based on the SegFormer semantic segmentation model, this study proposes the SegDetector model for difficult detection of small-scale targets and overlapping targets in target detection tasks. By changing the calculation method of the loss function, the detection of overlapping samples is improved and the time-consuming non-maximum-suppression (NMS) algorithm is discarded, and the horizontal and rotational detection of buildings can be easily and conveniently implemented. In order to verify the effectiveness of the SegDetector model, the xBD dataset, which is a dataset for assessing building damage from satellite imagery, was transformed and tested. The experiment results show that the SegDetector model outperforms the state-of-the-art (SOTA) models such as you-only-look-once (YOLOv3, v4, v5) in the xBD dataset with F1: 0.71, Precision: 0.63, and Recall: 0.81. At the same time, the SegDetector model has a small number of parameters and fast detection capability, making it more practical for deployment. Full article
Show Figures

Graphical abstract

Graphical abstract
Full article ">Figure 1
<p>Example from the xBD dataset showing satellite images before and after a volcanic eruption disaster. (<b>a</b>) shows satellite imagery of buildings before the eruption and (<b>b</b>) shows satellite imagery of buildings destroyed after the eruption.</p>
Full article ">Figure 2
<p>Structure of attention. ((<b>a</b>) is the multi-head attention of vanilla transformer and (<b>b</b>) is the multi-head attention with spatial reduction of transformer).</p>
Full article ">Figure 3
<p>Change of detection frame in the case of target occlusion and non-occlusion. (<b>a</b>) indicates the blue object is occluded and (<b>b</b>) indicates the blue object is not occluded.</p>
Full article ">Figure 4
<p>SegDetector calculates the BCE loss in separate channels and generates the final detection frame in separate channels to reduce the difficulty of detecting obscured targets.</p>
Full article ">Figure 5
<p>Overall model structure of SegDetector, which is based on SegFormer, to implement sub-channel target detection.</p>
Full article ">Figure 6
<p>SegDetector’s horizontal detection implementation. (<b>a</b>) is the pixel-by-pixel result calculated by SegDetector and (<b>b</b>) is the horizontal detection result calculated by SegDetector.</p>
Full article ">Figure 7
<p>Rotation detection implementation of SegDetector. (<b>a</b>) shows the pixel-by-pixel result calculated by SegDetector and (<b>b</b>) shows the rotating detection result calculated by SegDetector.</p>
Full article ">Figure 8
<p>Randomly calculated test dataset of two images for YOLOX detection results and SegDetector horizontal and rotational detection results. (<b>a</b>,<b>b</b>) are Faster R-CNN detection results, (<b>c</b>,<b>d</b>) are YOLOX detection results, (<b>e</b>,<b>f</b>) are SegDetector-HBB detection results, and (<b>g</b>,<b>h</b>) are SegDetector-OBB detection results.</p>
Full article ">Figure 8 Cont.
<p>Randomly calculated test dataset of two images for YOLOX detection results and SegDetector horizontal and rotational detection results. (<b>a</b>,<b>b</b>) are Faster R-CNN detection results, (<b>c</b>,<b>d</b>) are YOLOX detection results, (<b>e</b>,<b>f</b>) are SegDetector-HBB detection results, and (<b>g</b>,<b>h</b>) are SegDetector-OBB detection results.</p>
Full article ">Figure 9
<p>Downsampling the original resolution images in the xBD dataset.</p>
Full article ">Figure 10
<p>YOLOX and SegDetector detection results (<b>a</b>) is the YOLOX detection result and (<b>b</b>) is the SegDetector detection result.</p>
Full article ">Figure 11
<p>YOLOX and SegDetector detection results (<b>a</b>) is the YOLOX detection result and (<b>b</b>) is the SegDetector detection result.</p>
Full article ">Figure 12
<p>Plots of YOLOX and SegDetector detection results. ((<b>a</b>) shows YOLOX detection results and (<b>b</b>) shows SegDetector detection results).</p>
Full article ">
23 pages, 5249 KiB  
Article
Building Damage Assessment Based on Siamese Hierarchical Transformer Framework
by Yifan Da, Zhiyuan Ji and Yongsheng Zhou
Mathematics 2022, 10(11), 1898; https://doi.org/10.3390/math10111898 - 1 Jun 2022
Cited by 15 | Viewed by 2923
Abstract
The rapid and accurate damage assessment of buildings plays a critical role in disaster response. Based on pairs of pre- and post-disaster remote sensing images, effective building damage level assessment can be conducted. However, most existing methods are based on Convolutional Neural Network, [...] Read more.
The rapid and accurate damage assessment of buildings plays a critical role in disaster response. Based on pairs of pre- and post-disaster remote sensing images, effective building damage level assessment can be conducted. However, most existing methods are based on Convolutional Neural Network, which has limited ability to learn the global context. An attention mechanism helps ameliorate this problem. Hierarchical Transformer has powerful potential in the remote sensing field with strong global modeling capability. In this paper, we propose a novel two-stage damage assessment framework called SDAFormer, which embeds a symmetric hierarchical Transformer into a siamese U-Net-like network. In the first stage, the pre-disaster image is fed into a segmentation network for building localization. In the second stage, a two-branch damage classification network is established based on weights shared from the first stage. Then, pre- and post-disaster images are delivered to the network separately for damage assessment. Moreover, a spatial fusion module is designed to improve feature representation capability by building pixel-level correlation, which establishes spatial information in Swin Transformer blocks. The proposed framework achieves significant improvement on the large-scale building damage assessment dataset—xBD. Full article
Show Figures

Figure 1

Figure 1
<p>An overview of the SDAFormer framework. It is composed of two stages: (<b>a</b>) Stage 1: building localization, (<b>b</b>) Stage 2: damage classification.</p>
Full article ">Figure 2
<p>The architecture of U-Net-like network based on Swin Transformer block, which is composed of encoder, bottleneck, decoder and skip connections. Spatial fusion modules are embedded in the encoder.</p>
Full article ">Figure 3
<p>(<b>a</b>) The structure of a standard Transformer block [<a href="#B35-mathematics-10-01898" class="html-bibr">35</a>]. (<b>b</b>) Two consecutive Swin Transformer blocks [<a href="#B41-mathematics-10-01898" class="html-bibr">41</a>], which are called Window-based Transformer block and Shifted Window-based Transformer block, respectively.</p>
Full article ">Figure 4
<p>Swin Transformer blocks with the spatial fusion (SF) module.</p>
Full article ">Figure 5
<p>The structure of SF module.</p>
Full article ">Figure 6
<p>Building damage assessment results. (<b>a</b>,<b>b</b>) respectively show pre- and post-disaster images; (<b>c</b>) shows ground truth; (<b>d</b>,<b>e</b>) are the results of RescueNet and MaskRCNN, respectively; (<b>f</b>) is the prediction of our proposed framework.</p>
Full article ">Figure 7
<p>Visual examples of segmentation results in Stage 1. (<b>a</b>) shows pre-disaster images; (<b>b</b>) shows ground truth; (<b>c</b>,<b>d</b>) are the results of RescueNet and MaskRCNN, respectively; (<b>e</b>) is the segmentation result of our proposed framework.</p>
Full article ">Figure 8
<p>The visual comparison of prediction results on hurricane–Harveyan xBD dataset. (<b>a</b>) Pre-disaster; (<b>b</b>) post-disaster; (<b>c</b>) ground truth; (<b>d</b>) baseline; (<b>e</b>) SDAFormer (without SF); (<b>f</b>) SDAFormer (with SF).</p>
Full article ">Figure 9
<p>The visual comparison of prediction results on hurricane-Michael in xBD dataset. (<b>a</b>) Pre-disaster; (<b>b</b>) post-disaster; (<b>c</b>) ground truth; (<b>d</b>) baseline; (<b>e</b>) SDAFormer (without SF); (<b>f</b>) SDAFormer (with SF).</p>
Full article ">Figure 10
<p>The visual comparison of prediction results on palu-tsunami in xBD dataset. (<b>a</b>) Pre-disaster; (<b>b</b>) post-disaster; (<b>c</b>) ground truth; (<b>d</b>) baseline; (<b>e</b>) SDAFormer (without SF); (<b>f</b>) SDAFormer (with SF).</p>
Full article ">Figure 11
<p>Results of the independent disaster events. First row to fourth row are: tornado in Arkansas and in Kentucky, typhoon on the Saipan Island and on the Tinian Island. (<b>a</b>) Pre-disaster; (<b>b</b>) post-disaster; (<b>c</b>) RescueNet; (<b>d</b>) MaskRCNN; (<b>e</b>) our results.</p>
Full article ">Figure 12
<p>The distribution of disaster categories in xBD dataset.</p>
Full article ">
22 pages, 20753 KiB  
Article
Building Damage Detection Using U-Net with Attention Mechanism from Pre- and Post-Disaster Remote Sensing Datasets
by Chuyi Wu, Feng Zhang, Junshi Xia, Yichen Xu, Guoqing Li, Jibo Xie, Zhenhong Du and Renyi Liu
Remote Sens. 2021, 13(5), 905; https://doi.org/10.3390/rs13050905 - 28 Feb 2021
Cited by 71 | Viewed by 6995
Abstract
The building damage status is vital to plan rescue and reconstruction after a disaster and is also hard to detect and judge its level. Most existing studies focus on binary classification, and the attention of the model is distracted. In this study, we [...] Read more.
The building damage status is vital to plan rescue and reconstruction after a disaster and is also hard to detect and judge its level. Most existing studies focus on binary classification, and the attention of the model is distracted. In this study, we proposed a Siamese neural network that can localize and classify damaged buildings at one time. The main parts of this network are a variety of attention U-Nets using different backbones. The attention mechanism enables the network to pay more attention to the effective features and channels, so as to reduce the impact of useless features. We train them using the xBD dataset, which is a large-scale dataset for the advancement of building damage assessment, and compare their result balanced F (F1) scores. The score demonstrates that the performance of SEresNeXt with an attention mechanism gives the best performance among single models, with the F1 score reaching 0.787. To improve the accuracy, we fused the results and got the best overall F1 score of 0.792. To verify the transferability and robustness of the model, we selected the dataset on the Maxar Open Data Program of two recent disasters to investigate the performance. By visual comparison, the results show that our model is robust and transferable. Full article
Show Figures

Graphical abstract

Graphical abstract
Full article ">Figure 1
<p>Damage levels and their descriptions [<a href="#B29-remotesensing-13-00905" class="html-bibr">29</a>].</p>
Full article ">Figure 2
<p>The overall framework. Localization U-Net is used to locate buildings. After pre-training with pre-disaster image, it shares weight with classification U-Net, which is used to classify damage level. The combination of the two is Siamese neural network. We use three random seeds to train it. After inputting pre- and post- disaster images into Siamese neural network, we get three five channels classification results. The first channel is the result of localization, and the last four channels are the probability of each damage level at each location. In the fusion step, the results corresponding to the three seeds are first weighted and averaged, and the weight is determined by the validation accuracy during the training process. Then, the threshold is used to determine the value of each pixel. Finally, for improving the accuracy, the pre-trained classification U-Net localizes the buildings again and get the localize result which be used for the double check of buildings’ localization.</p>
Full article ">Figure 3
<p>U-Net with the attention mechanism’s structure [<a href="#B34-remotesensing-13-00905" class="html-bibr">34</a>]. The encoder performs 4 downsampling. Symmetrically, its decoder upsamples 4 times to restore the features to the original image resolution. The attention gate is placed at the end of the skip connection.</p>
Full article ">Figure 4
<p>Attention gate structure [<a href="#B34-remotesensing-13-00905" class="html-bibr">34</a>]. <span class="html-italic">F</span>, <span class="html-italic">H</span> and <span class="html-italic">W</span> stand for channel, height and width respectively, and <span class="html-italic">D</span> is the depth of the 3D data block. <math display="inline"><semantics> <mrow> <msup> <mi>x</mi> <mi>l</mi> </msup> </mrow> </semantics></math>, the feature map from the encoder layer, is scaled with the attention coefficients (<span class="html-italic">α</span>), which are computed by <math display="inline"><semantics> <mrow> <msup> <mi>x</mi> <mi>l</mi> </msup> </mrow> </semantics></math> and <span class="html-italic">g</span>. The previous decoder features in <span class="html-italic">g</span> are added to <math display="inline"><semantics> <mrow> <msup> <mi>x</mi> <mi>l</mi> </msup> </mrow> </semantics></math> to determine the focus regions; then, the value of the attention coefficients is between 0 and 1 throughout training.</p>
Full article ">Figure 5
<p>Residual connection. Each residual block is composed by convolutions (Conv), batch normalizations (BN) and rectified linear units (ReLU).</p>
Full article ">Figure 6
<p>Squeeze-and-Excitation (SE) module [<a href="#B39-remotesensing-13-00905" class="html-bibr">39</a>]. The module is mainly composed of three parts: squeeze, excitation and scale. <math display="inline"><semantics> <mrow> <msub> <mi>F</mi> <mrow> <mi>s</mi> <mi>q</mi> </mrow> </msub> <mrow> <mo>(</mo> <mo>·</mo> <mo>)</mo> </mrow> </mrow> </semantics></math> represents the squeeze transformation, <math display="inline"><semantics> <mrow> <msub> <mi>F</mi> <mrow> <mi>e</mi> <mi>x</mi> </mrow> </msub> <mrow> <mo>(</mo> <mrow> <mo>·</mo> <mo>,</mo> <mi>w</mi> </mrow> <mo>)</mo> </mrow> </mrow> </semantics></math> represents the excitation transformation<math display="inline"><semantics> <mrow> <mo> </mo> <mi>and</mi> <mo> </mo> <msub> <mi>F</mi> <mrow> <mi>s</mi> <mi>c</mi> <mi>a</mi> <mi>l</mi> <mi>e</mi> </mrow> </msub> <mrow> <mo>(</mo> <mo>·</mo> <mo>)</mo> </mrow> <mo> </mo> </mrow> </semantics></math>represents the scale transformation.</p>
Full article ">Figure 7
<p>ResNeXt structure [<a href="#B41-remotesensing-13-00905" class="html-bibr">41</a>]. In the figure above, the structure of (<b>a</b>) is the original structure of ResNeXt, and (<b>b</b>,<b>c</b>) are equivalent representations of the structure of (<b>a</b>) in an actual implementation, the structure of (<b>c</b>) which is relatively simple to implement, and the basic block of ResNeXt is realized through the form of grouped convolution.</p>
Full article ">Figure 8
<p>Residual block combined with the SE module. The SE module is composed of the residual block, which was introduced in <a href="#remotesensing-13-00905-f005" class="html-fig">Figure 5</a>, global pooling block, full connection (FC) block, rectified linear units (ReLU) and activation block (Sigmoid).</p>
Full article ">Figure 9
<p>Dual Path Net (DPN) structure diagram [<a href="#B42-remotesensing-13-00905" class="html-bibr">42</a>].</p>
Full article ">Figure 10
<p>Hurricane-michael_00000256 in the verification set. (<b>a</b>) A pre-disaster image, (<b>b</b>) a post-disaster image, (<b>c</b>) the ground truth (GT) and (<b>d</b>) the legend. (<b>e</b>) The result of ResNet (w/o A or without attention), (<b>f</b>) the result of SEresNeXt (w/o A), (<b>g</b>) the result of DPN (w/o A), (<b>h</b>) the result of the Squeeze-and-Excitation network (SENet) (w/o A), (<b>i</b>) the result of ResNet (w A or with attention), (<b>j</b>) the result of SEresNeXt (w A), (<b>k</b>) the result of DPN (w A) and (<b>l</b>) the result of SENet (w A).</p>
Full article ">Figure 10 Cont.
<p>Hurricane-michael_00000256 in the verification set. (<b>a</b>) A pre-disaster image, (<b>b</b>) a post-disaster image, (<b>c</b>) the ground truth (GT) and (<b>d</b>) the legend. (<b>e</b>) The result of ResNet (w/o A or without attention), (<b>f</b>) the result of SEresNeXt (w/o A), (<b>g</b>) the result of DPN (w/o A), (<b>h</b>) the result of the Squeeze-and-Excitation network (SENet) (w/o A), (<b>i</b>) the result of ResNet (w A or with attention), (<b>j</b>) the result of SEresNeXt (w A), (<b>k</b>) the result of DPN (w A) and (<b>l</b>) the result of SENet (w A).</p>
Full article ">Figure 11
<p>Palu_00000004 in the verification set. (<b>a</b>) A pre-disaster image, (<b>b</b>) a post-disaster image and (<b>c</b>) the ground truth (GT). (<b>d</b>) The result of ResNet (w/o A), (<b>e</b>) the result of SEresNeXt (w/o A), (<b>f</b>) the result of DPN (w/o A), (<b>g</b>) the result of SENet (w/o A), (<b>h</b>) the result of ResNet (w A), (<b>i</b>) the result of SEresNeXt (w A), (<b>j</b>) the result of DPN (w A) and (<b>k</b>) the result of SENet (w A).</p>
Full article ">Figure 11 Cont.
<p>Palu_00000004 in the verification set. (<b>a</b>) A pre-disaster image, (<b>b</b>) a post-disaster image and (<b>c</b>) the ground truth (GT). (<b>d</b>) The result of ResNet (w/o A), (<b>e</b>) the result of SEresNeXt (w/o A), (<b>f</b>) the result of DPN (w/o A), (<b>g</b>) the result of SENet (w/o A), (<b>h</b>) the result of ResNet (w A), (<b>i</b>) the result of SEresNeXt (w A), (<b>j</b>) the result of DPN (w A) and (<b>k</b>) the result of SENet (w A).</p>
Full article ">Figure 12
<p>Palu_00000181 in the verification set. (<b>a</b>) A pre-disaster image, (<b>b</b>) a post-disaster image and (<b>c</b>) the GT. (<b>d</b>) The result of ResNet (w/o A), (<b>e</b>) the result of SEresNeXt (w/o A), (<b>f</b>) the result of DPN (w/o A), (<b>g</b>) the result of SENet (w/o A), (<b>h</b>) the result of ResNet (w A), (<b>i</b>) the result of SEresNeXt (w A), (<b>j</b>) the result of DPN (w A) and (<b>k</b>) the result of SENet (w A).</p>
Full article ">Figure 13
<p>The fusion result of hurricane-michael_00000256, palu_00000004 and palu_00000181. (<b>a</b>) The GT of hurricane-michael_00000256, (<b>b</b>,<b>c</b>) the results of the model w/o A and model w A of hurricane-michael_00000256, (<b>d</b>) the GT of palu_00000004, (<b>e</b>,<b>f</b>) the results of the model w/o A and model w A of palu_00000004, (<b>g</b>) the GT of palu_00000181 and (<b>h</b>,<b>i</b>) the results of the model w/o A and model w A of palu_00000181.</p>
Full article ">Figure 14
<p>Results of Beirut explosion. (<b>a</b>) A pre-disaster image, (<b>b</b>) a post-disaster image, and (<b>m</b>) the GT. (<b>c</b>) The result of ResNet (w/o A), (<b>d</b>) the result of SEresNeXt (w/o A), (<b>e</b>) the result of DPN (w/o A), (<b>f</b>) the result of SENet (w/o A), (<b>g</b>) the result of ResNet (w A), (<b>h</b>) the result of SEresNeXt (w A), (<b>i</b>) the result of DPN (w A), (<b>j</b>) the result of SENet (w A), (<b>k</b>) the fusion result without the attention mechanism and (<b>l</b>) the fusion result with the attention mechanism.</p>
Full article ">Figure 14 Cont.
<p>Results of Beirut explosion. (<b>a</b>) A pre-disaster image, (<b>b</b>) a post-disaster image, and (<b>m</b>) the GT. (<b>c</b>) The result of ResNet (w/o A), (<b>d</b>) the result of SEresNeXt (w/o A), (<b>e</b>) the result of DPN (w/o A), (<b>f</b>) the result of SENet (w/o A), (<b>g</b>) the result of ResNet (w A), (<b>h</b>) the result of SEresNeXt (w A), (<b>i</b>) the result of DPN (w A), (<b>j</b>) the result of SENet (w A), (<b>k</b>) the fusion result without the attention mechanism and (<b>l</b>) the fusion result with the attention mechanism.</p>
Full article ">Figure 15
<p>Results of Hurricane Laura. (<b>a</b>) A pre-disaster image, (<b>b</b>) a post-disaster image and (<b>m</b>) is the GT. (<b>c</b>) The result of ResNet (w/o A), (<b>d</b>) the result of SEresNeXt (w/o A), (<b>e</b>) the result of DPN (w/o A), (<b>f</b>) the result of SENet (w/o A), (<b>g</b>) the result of ResNet (w A), (<b>h</b>) the result of SEresNeXt (w A), (<b>i</b>) the result of DPN (w A), (<b>j</b>) the result of SENet (w A), (<b>k</b>) the fusion result without the attention mechanism and (<b>l</b>) the fusion result with the attention mechanism.</p>
Full article ">
20 pages, 10253 KiB  
Article
Transferability of Convolutional Neural Network Models for Identifying Damaged Buildings Due to Earthquake
by Wanting Yang, Xianfeng Zhang and Peng Luo
Remote Sens. 2021, 13(3), 504; https://doi.org/10.3390/rs13030504 - 31 Jan 2021
Cited by 43 | Viewed by 4418
Abstract
The collapse of buildings caused by earthquakes can lead to a large loss of life and property. Rapid assessment of building damage with remote sensing image data can support emergency rescues. However, current studies indicate that only a limited sample set can usually [...] Read more.
The collapse of buildings caused by earthquakes can lead to a large loss of life and property. Rapid assessment of building damage with remote sensing image data can support emergency rescues. However, current studies indicate that only a limited sample set can usually be obtained from remote sensing images immediately following an earthquake. Consequently, the difficulty in preparing sufficient training samples constrains the generalization of the model in the identification of earthquake-damaged buildings. To produce a deep learning network model with strong generalization, this study adjusted four Convolutional Neural Network (CNN) models for extracting damaged building information and compared their performance. A sample dataset of damaged buildings was constructed by using multiple disaster images retrieved from the xBD dataset. Using satellite and aerial remote sensing data obtained after the 2008 Wenchuan earthquake, we examined the geographic and data transferability of the deep network model pre-trained on the xBD dataset. The result shows that the network model pre-trained with samples generated from multiple disaster remote sensing images can extract accurately collapsed building information from satellite remote sensing data. Among the adjusted CNN models tested in the study, the adjusted DenseNet121 was the most robust. Transfer learning solved the problem of poor adaptability of the network model to remote sensing images acquired by different platforms and could identify disaster-damaged buildings properly. These results provide a solution to the rapid extraction of earthquake-damaged building information based on a deep learning network model. Full article
Show Figures

Graphical abstract

Graphical abstract
Full article ">Figure 1
<p>Location of the study area and the remote sensing images after the 2008 M<sub>w</sub>7.9 Wenchuan Earthquake: (<b>A</b>) the composite image of the Northeast of Wenchuan County; (<b>B</b>) images for T1–T6 sub-study areas in the Wenchuan County; (<b>C</b>) the image of the Beichuan County.</p>
Full article ">Figure 2
<p>Workflow for preparing the building samples from the dataset of this study.</p>
Full article ">Figure 3
<p>Flowchart of the work, including four steps: (<b>A</b>) to generate samples from the image datasets described in <a href="#sec2-remotesensing-13-00504" class="html-sec">Section 2</a>; (<b>B</b>) to train the CNN models with the samples generated in Step A; (<b>C</b>) to assess geographic transferability with the samples of Wenchuan, and (<b>D</b>) to assess data transferability with the samples of Beichuan.</p>
Full article ">Figure 4
<p>Results evaluated by (<b>a</b>) Accuracy and (<b>b</b>) F1 score of the adjusted pre-trained VGG16 (CNN-T), Inception V3 (CNN-T), and DenseNet121 (CNN-T) in the six sub-study areas T1–T6 of Wenchuan County.</p>
Full article ">Figure 5
<p>Comparison of the predicted results of three adjusted CNN models in the T1 and T2 sub-study regions.</p>
Full article ">Figure 6
<p>Examples of building misclassification in the Wenchuan area. (<b>a</b>,<b>b</b>) are damaged buildings but wrongly classified as no damage in T2; (<b>c</b>) the building with no damage but was wrongly classified as damaged in T4; (<b>d</b>) the building with no damage but wrongly was classified as damaged in T5.</p>
Full article ">Figure 7
<p>ROC curve for the classification result of all the building samples of Beichuan. (<b>a</b>) prediction from the pre-trained model without fine-tuning process; (<b>b</b>) prediction from the pre-trained model fine-tuned with S1 and S2 samples.</p>
Full article ">Figure 8
<p>Predicted results of earthquake-damaged buildings in Beichuan using the adjusted DenseNet121 model with the fine-tuning process.</p>
Full article ">Figure 9
<p>Confusion matrices of the adjusted DenseNet121 model tested on (<b>A</b>) the xBD-test dataset, (<b>B</b>) the Wenchuan dataset, and (<b>C</b>) the Beichuan dataset.</p>
Full article ">
27 pages, 9491 KiB  
Article
Post-Disaster Building Damage Detection from Earth Observation Imagery Using Unsupervised and Transferable Anomaly Detecting Generative Adversarial Networks
by Sofia Tilon, Francesco Nex, Norman Kerle and George Vosselman
Remote Sens. 2020, 12(24), 4193; https://doi.org/10.3390/rs12244193 - 21 Dec 2020
Cited by 38 | Viewed by 6898
Abstract
We present an unsupervised deep learning approach for post-disaster building damage detection that can transfer to different typologies of damage or geographical locations. Previous advances in this direction were limited by insufficient qualitative training data. We propose to use a state-of-the-art Anomaly Detecting [...] Read more.
We present an unsupervised deep learning approach for post-disaster building damage detection that can transfer to different typologies of damage or geographical locations. Previous advances in this direction were limited by insufficient qualitative training data. We propose to use a state-of-the-art Anomaly Detecting Generative Adversarial Network (ADGAN) because it only requires pre-event imagery of buildings in their undamaged state. This approach aids the post-disaster response phase because the model can be developed in the pre-event phase and rapidly deployed in the post-event phase. We used the xBD dataset, containing pre- and post- event satellite imagery of several disaster-types, and a custom made Unmanned Aerial Vehicle (UAV) dataset, containing post-earthquake imagery. Results showed that models trained on UAV-imagery were capable of detecting earthquake-induced damage. The best performing model for European locations obtained a recall, precision and F1-score of 0.59, 0.97 and 0.74, respectively. Models trained on satellite imagery were capable of detecting damage on the condition that the training dataset was void of vegetation and shadows. In this manner, the best performing model for (wild)fire events yielded a recall, precision and F1-score of 0.78, 0.99 and 0.87, respectively. Compared to other supervised and/or multi-epoch approaches, our results are encouraging. Moreover, in addition to image classifications, we show how contextual information can be used to create detailed damage maps without the need of a dedicated multi-task deep learning framework. Finally, we formulate practical guidelines to apply this single-epoch and unsupervised method to real-world applications. Full article
Show Figures

Graphical abstract

Graphical abstract
Full article ">Figure 1
<p>Skip-GANomaly architecture. Adapted from [<a href="#B24-remotesensing-12-04193" class="html-bibr">24</a>].</p>
Full article ">Figure 2
<p>Example from the xBD dataset showing pre- and post-event satellite images from a location where a volcanic eruption took place. Several buildings and sport facilities are visible. The post-event image shows damage induced by volcanic activity. The buildings are outlined and the damage level is depicted by the polygon color. The scale bars are approximate.</p>
Full article ">Figure 3
<p>Examples of Satellite imagery used for testing: (<b>a</b>) Hurricane Florence (USA), (<b>b</b>) Hurricane Michael (USA), (<b>c</b>) Hurricane Harvey (USA), (<b>d</b>) Hurricane Mathew (Haiti), (<b>e</b>) Volcano (Guatemala), (<b>f</b>) Earthquake (Mexico), (<b>g</b>) Flood (Midwest), (<b>h</b>) Tsunami (Palu, Indonesia), (<b>i</b>) Wildfire (Santa-Rosa USA) and (<b>j</b>) Fire (Socal, USA). Examples of <b>UAV imagery</b> used for testing: (<b>k</b>) Earthquake (Pescara del Tronto, Italy), (<b>l</b>) Earthquake (L’Aquila, Italy), (<b>m</b>) Earthquake (Mirabello, Italy), (<b>n</b>) Earthquake (Taiwan) and (<b>o</b>), Earthquake (Nepal). The scale bars are approximate.</p>
Full article ">Figure 4
<p>Illustration of different cropping strategies for the xBD dataset from the original patch size of 1024 × 1024 to 256 × 256, 64 × 64 and 32 × 32. The scale bars are approximate.</p>
Full article ">Figure 5
<p>Illustration of cropping strategies for the UAV dataset from the original patch size of 4000 × 6000 to 512 × 512, 256 × 256 and 64 × 64. The scale bars are approximate and refer to the front of the scene.</p>
Full article ">Figure 6
<p>Number of samples in each data subset. Original refers to the complete un-preprocessed dataset. Y-axis is in log-scale.</p>
Full article ">Figure 7
<p>Performance of Skip-GANomaly on pre-processed satellite patches of size 256 × 256 (only baseline) 64 × 64 and 32 × 32.</p>
Full article ">Figure 8
<p>Performance of Skip-GANomaly on UAV imagery of size 512 × 512, 256 × 256 and 64 × 64 for different locations.</p>
Full article ">Figure 9
<p>Post-wildfire satellite imagery from the USA showing multiple damaged buildings overlaid with anomaly scores and building polygon. The classification, classification threshold and anomaly scores are indicated (TP = True positive). The scale bars are approximate. High anomaly scores on burnt building surroundings and smaller patch sizes lead to correct classifications.</p>
Full article ">Figure 10
<p>Post-flood satellite imagery from the USA showing a damaged building, overlaid with anomaly scores and building polygons. The classification, classification threshold and anomaly scores are indicated (TP = True positive). The scale bars are approximate. High anomaly scores on flooded areas resulted in correct classifications, regardless of patch size.</p>
Full article ">Figure 11
<p>Post-earthquake satellite scene from Mexico showing undamaged buildings overlaid with anomaly scores and building polygons. The classification, classification threshold and anomaly scores are indicated (FP = False positive). The scale bars are approximate. High anomaly scores induced by varying building surroundings resulted in false positives, regardless of the patch size.</p>
Full article ">Figure 12
<p>Earthquake UAV scene from Nepal showing a damaged building overlaid with anomaly scores. The classification, classification threshold and anomaly scores are indicated (TP = True positive, FN = False Negative). The scale bars are approximate. The 64 × 64 patch yielded a correct classification due to a better understanding of the building image distribution.</p>
Full article ">
20 pages, 11041 KiB  
Article
Pyramid Pooling Module-Based Semi-Siamese Network: A Benchmark Model for Assessing Building Damage from xBD Satellite Imagery Datasets
by Yanbing Bai, Junjie Hu, Jinhua Su, Xing Liu, Haoyu Liu, Xianwen He, Shengwang Meng, Erick Mas and Shunichi Koshimura
Remote Sens. 2020, 12(24), 4055; https://doi.org/10.3390/rs12244055 - 11 Dec 2020
Cited by 37 | Viewed by 4637
Abstract
Most mainstream research on assessing building damage using satellite imagery is based on scattered datasets and lacks unified standards and methods to quantify and compare the performance of different models. To mitigate these problems, the present study develops a novel end-to-end benchmark model, [...] Read more.
Most mainstream research on assessing building damage using satellite imagery is based on scattered datasets and lacks unified standards and methods to quantify and compare the performance of different models. To mitigate these problems, the present study develops a novel end-to-end benchmark model, termed the pyramid pooling module semi-Siamese network (PPM-SSNet), based on a large-scale xBD satellite imagery dataset. The high precision of the proposed model is achieved by adding residual blocks with dilated convolution and squeeze-and-excitation blocks into the network. Simultaneously, the highly automated process of satellite imagery input and damage classification result output is reached by employing concurrent learned attention mechanisms through a semi-Siamese network for end-to-end input and output purposes. Our proposed method achieves F1 scores of 0.90, 0.41, 0.65, and 0.70 for the undamaged, minor-damaged, major-damaged, and destroyed building classes, respectively. From the perspective of end-to-end methods, the ablation experiments and comparative analysis confirm the effectiveness and originality of the PPM-SSNet method. Finally, the consistent prediction results of our model for data from the 2011 Tohoku Earthquake verify the high performance of our model in terms of the domain shift problem, which implies that it is effective for evaluating future disasters. Full article
Show Figures

Graphical abstract

Graphical abstract
Full article ">Figure 1
<p>Example of the xBD dataset: Tsunami in Palu, Indonesia. From left to right: (<b>a</b>) Pre-disaster image, (<b>b</b>) Post-disaster image, (<b>c</b>) Damage scale, and (<b>d</b>) Building footprint.</p>
Full article ">Figure 2
<p>Ratio of damage class at the pixel level.</p>
Full article ">Figure 3
<p>Validation area. (<b>a</b>) Higashi Matsushima in the Tohoku region of Japan; the rectangular areas marked in blue and red are the selected validation areas; (<b>b</b>) The close-up of the blue area as shown in Figure 10a with the ground truth data of building damage; and (<b>c</b>) The close-up of the red area as shown in Figure 10a with the ground truth data of building damage.</p>
Full article ">Figure 4
<p>Dilated convolution with dilated rates of 1 (i.e., normal convolution; left side of the figure) and 2 (right side of the figure). <span class="html-italic">g</span>, <span class="html-italic">h</span>, and <span class="html-italic">u</span> mean the input image (or activation map), convolutional kernel, and output. An output <span class="html-italic">u</span> is calculated by summing the multiplications of each value (<span class="html-italic">i</span>, <span class="html-italic">j</span>) at the kernel <span class="html-italic">h</span> and its corresponding value (<span class="html-italic">x</span>, <span class="html-italic">y</span>) at <span class="html-italic">g</span>.</p>
Full article ">Figure 5
<p>Squeeze-and-excitation (SE) blocks produce and apply channel-wise attention on the activation maps. GAP means global average pooling. <math display="inline"><semantics> <msub> <mi>w</mi> <mi>i</mi> </msub> </semantics></math> denotes the <span class="html-italic">i</span>th linear production layer. ReLU and Sigmoid are employed following <math display="inline"><semantics> <msub> <mi>w</mi> <mn>1</mn> </msub> </semantics></math> and <math display="inline"><semantics> <msub> <mi>w</mi> <mn>2</mn> </msub> </semantics></math> for the activation functions. The columns depicted in different colors represent the activation map of each channel of the input/output tensor.</p>
Full article ">Figure 6
<p>The pyramid pooling module (PPM) <span class="html-italic">g</span> represents an activation map of a single channel. N is the number of cells in a row/column of a pooling grid.</p>
Full article ">Figure 7
<p>The architecture of the proposed network. <span class="html-italic">c</span>, <span class="html-italic">b</span>, <span class="html-italic">d</span>, and <span class="html-italic">r</span> represent the convolutional layer, batch normalization layer, dropout layer, and ReLU layer. SE, RB’, RB, and PPM represent the modules illustrated at the bottom of this figure. The difference between RB’ and RB is that RB’ has an additional convolutional layer + batch normalization layer, which is designed to change the number of channels or size of the input tensor if needed. See <a href="#remotesensing-12-04055-t002" class="html-table">Table 2</a> for more details.</p>
Full article ">Figure 8
<p>FPN R-CNN network.</p>
Full article ">Figure 9
<p>Siam-U-Net-Attention network model.</p>
Full article ">Figure 10
<p>The results from our proposed method and comparisons with others. (<b>a</b>) Image collected before the disaster; (<b>b</b>) Image collected after the disaster; (<b>c</b>) Reference data; (<b>d</b>) Proposed PPM-SSNet model; (<b>e</b>) Siam-U-Net model; and (<b>f</b>) FPN-R-CNN model.</p>
Full article ">Figure 11
<p>Prediction results from our proposed method in the validation areas. (<b>a1</b>,<b>a2</b>) Pre-disaster image; (<b>b1</b>,<b>b2</b>) Post-disaster image; (<b>c1</b>,<b>c2</b>) Predicted damage scale by the PPM-SSNet model; and (<b>d1</b>,<b>d2</b>) Prediction building footprint by the PPM-SSNet model.</p>
Full article ">
25 pages, 14333 KiB  
Article
Technical Solution Discussion for Key Challenges of Operational Convolutional Neural Network-Based Building-Damage Assessment from Satellite Imagery: Perspective from Benchmark xBD Dataset
by Jinhua Su, Yanbing Bai, Xingrui Wang, Dong Lu, Bo Zhao, Hanfang Yang, Erick Mas and Shunichi Koshimura
Remote Sens. 2020, 12(22), 3808; https://doi.org/10.3390/rs12223808 - 20 Nov 2020
Cited by 12 | Viewed by 4853
Abstract
Earth Observation satellite imaging helps building diagnosis during a disaster. Several models are put forward on the xBD dataset, which can be divided into two levels: the building level and the pixel level. Models from two levels evolve into several versions that will [...] Read more.
Earth Observation satellite imaging helps building diagnosis during a disaster. Several models are put forward on the xBD dataset, which can be divided into two levels: the building level and the pixel level. Models from two levels evolve into several versions that will be reviewed in this paper. There are four key challenges hindering researchers from moving forward on this task, and this paper tries to give technical solutions. First, metrics on different levels could not be compared directly. We put forward a fairer metric and give a method to convert between metrics of two levels. Secondly, drone images may be another important source, but drone data may have only a post-disaster image. This paper shows and compares methods of directly detecting and generating. Thirdly, the class imbalance is a typical feature of the xBD dataset and leads to a bad F1 score for minor damage and major damage. This paper provides four specific data resampling strategies, which are Main-Label Over-Sampling (MLOS), Discrimination After Cropping (DAC), Dilation of Area with Minority (DAM) and Synthetic Minority Over-Sampling Technique (SMOTE), as well as cost-sensitive re-weighting schemes. Fourthly, faster prediction meets the need for a real-time situation. This paper recommends three specific methods, feature-map subtraction, parameter sharing, and knowledge distillation. Finally, we developed our AI-driven Damage Diagnose Platform (ADDP). This paper introduces the structure of ADDP and technical details. Customized settings, interface preview, and upload and download satellite images are major services our platform provides. Full article
Show Figures

Graphical abstract

Graphical abstract
Full article ">Figure 1
<p>Disaster events included in the xBD dataset.</p>
Full article ">Figure 2
<p>Example of xBD dataset. From left to right: (<b>a</b>) Pre-Disaster Image, (<b>b</b>) Post-Disaster Image, (<b>c</b>) Damage Scale Label, (<b>d</b>) Building Footprint.</p>
Full article ">Figure 3
<p>Histogram of (<b>a</b>) different disaster types and (<b>b</b>) different sensors images are obtained by in tier 1, tier 3, tier hold and tier test for xBD dataset. The distribution demonstrates the unbalanced distribution both in disaster types and sensor types in the dataset.</p>
Full article ">Figure 4
<p>The overall structure of this paper. It contains three main parts and follows the black arrow line from the start point. The first is the review of two categories of building-damage assessment. The second is about four key challenges in this field and proposes some novel solutions. The third is the web-based platform supporting damage detection after disasters.</p>
Full article ">Figure 5
<p>The flowchart of building-level model.</p>
Full article ">Figure 6
<p>Pixel-level Computation Flow. The images are processed with crop, class balance and augmentation operations. Then, the processed pre-image and post-image are input to the encoder and decoder, and the masks of building segmentation and damage classification are produced.</p>
Full article ">Figure 7
<p>Examples of state-of-the-art end-to-end models. (<b>a</b>) The overall structure of the Siam-U-Net-Attn model [<a href="#B25-remotesensing-12-03808" class="html-bibr">25</a>] which uses a double U-Net model to generate binary masks. (<b>b</b>) The overall structure of the model in Weber et al. [<a href="#B26-remotesensing-12-03808" class="html-bibr">26</a>] which uses Mask R-CNN(Regions with CNN features) with FPN (Feature Pyramid Networks) architecture as the backbone.</p>
Full article ">Figure 8
<p>An example of transfer learning. First, we input the image to DilatedNet to obtain the color mask and binary mask. Then, the original image, color mask and binary mask are input to VGG, VGG and LeNet respectively for extracting corresponding features. Finally, we input three features to regression network and output the estimated damage.</p>
Full article ">Figure 9
<p>A case study of different level models evaluated in the pixel-level.</p>
Full article ">Figure 10
<p>With a given IoU for building contour segmentation, we can define the True Positive (TP), False Positive (FP), and False Negative (FN) to evaluate the detection result. The IoU threshold decides the precision of localization result.</p>
Full article ">Figure 11
<p>Interface of Cloud-Based AI Damage Mapping Online Service.</p>
Full article ">Figure 12
<p>The overall structure of the AI-driven Damage Diagnosis Platform (ADDP). It has a service-oriented architecture containing the following four layers: an application layer, a logic layer, a service layer, and a resource layer.</p>
Full article ">Figure 13
<p>The detailed workflow of the logit layer we design. The structure helps users switch to a preferred mode and handle abnormal cases, hostile attacks, and concurrent assess by a task queue design which makes sure only 5 sub-threads run at the same time.</p>
Full article ">
29 pages, 24498 KiB  
Article
Multi-Hazard and Spatial Transferability of a CNN for Automated Building Damage Assessment
by Tinka Valentijn, Jacopo Margutti, Marc van den Homberg and Jorma Laaksonen
Remote Sens. 2020, 12(17), 2839; https://doi.org/10.3390/rs12172839 - 1 Sep 2020
Cited by 49 | Viewed by 6982
Abstract
Automated classification of building damage in remote sensing images enables the rapid and spatially extensive assessment of the impact of natural hazards, thus speeding up emergency response efforts. Convolutional neural networks (CNNs) can reach good performance on such a task in experimental settings. [...] Read more.
Automated classification of building damage in remote sensing images enables the rapid and spatially extensive assessment of the impact of natural hazards, thus speeding up emergency response efforts. Convolutional neural networks (CNNs) can reach good performance on such a task in experimental settings. How CNNs perform when applied under operational emergency conditions, with unseen data and time constraints, is not well studied. This study focuses on the applicability of a CNN-based model in such scenarios. We performed experiments on 13 disasters that differ in natural hazard type, geographical location, and image parameters. The types of natural hazards were hurricanes, tornadoes, floods, tsunamis, and volcanic eruptions, which struck across North America, Central America, and Asia. We used 175,289 buildings from the xBD dataset, which contains human-annotated multiclass damage labels on high-resolution satellite imagery with red, green, and blue (RGB) bands. First, our experiments showed that the performance in terms of area under the curve does not correlate with the type of natural hazard, geographical region, and satellite parameters such as the off-nadir angle. Second, while performance differed highly between occurrences of disasters, our model still reached a high level of performance without using any labeled data of the test disaster during training. This provides the first evidence that such a model can be effectively applied under operational conditions, where labeled damage data of the disaster cannot be available timely and thus model (re-)training is not an option. Full article
Show Figures

Graphical abstract

Graphical abstract
Full article ">Figure 1
<p>Workflow of DNAs during emergency response operations and the possible role of an automated building damage classification model. Black: items/data; blue: processes; purple: events. Full lines: necessary steps; dashed lines: optional steps.</p>
Full article ">Figure 2
<p>Architecture of the used model. The numbers in the squares indicate the input and output size of each block, where N equals the number of damage classes.</p>
Full article ">Figure 3
<p>Examples of challenges in the data. The type of challenge is indicated by the subcaption. (<b>a</b>) Cloud cover; (<b>b</b>) misaligned building polygons; (<b>c</b>) building with two different damage labels; (<b>d</b>) difference in illumination on before (<b>left</b>) and after (<b>right</b>) images.</p>
Full article ">Figure 4
<p>Before and after imagery and damage labels of part of the areas that were impacted by the Joplin tornado (<b>top row</b>) and the Nepal flooding (<b>bottom row</b>). The colors of the damage labels in figures <span class="html-italic">c</span> and <span class="html-italic">f</span> indicate the different degrees of damage, where green equals <span class="html-italic">no damage</span>, yellow <span class="html-italic">minor damage</span>, orange <span class="html-italic">major damage</span>, and red <span class="html-italic">destroyed</span>.</p>
Full article ">Figure 5
<p>Examples of individual buildings after the disaster impacted by the Joplin tornado (<b>a</b>–<b>d</b>) and the Nepal flooding (<b>e</b>–<b>h</b>). The subcaptions indicate the damage class.</p>
Full article ">Figure 6
<p>Confusion matrices of the model tested on the Nepal flooding (<b>left</b>) and the Joplin tornado (<b>right</b>). Both models were trained on 80% of the data, and tested on 10%.</p>
Full article ">Figure 7
<p>Four examples of buildings in the test set of the Nepal flooding that were misclassified by the model. For each building, the before image is shown on the left and the after image on the right. (<b>a</b>) Label: <span class="html-italic">major damage</span>; prediction: <span class="html-italic">no damage</span>; (<b>b</b>) Label: <span class="html-italic">major damage</span>; prediction: <span class="html-italic">no damage</span>; (<b>c</b>) Label: <span class="html-italic">no damage</span>; prediction: <span class="html-italic">destroyed</span>; (<b>d</b>) Label: <span class="html-italic">no damage</span>; prediction: <span class="html-italic">major damage</span>.</p>
Full article ">Figure 8
<p>Four examples of buildings in the test set of the Joplin tornado that were misclassified by the model. For each building, the before image is shown on the left and the after image on the right. (<b>a</b>) Label: <span class="html-italic">minor damage</span>; prediction: <span class="html-italic">no damage</span>; (<b>b</b>) Label: <span class="html-italic">major damage</span>; prediction: <span class="html-italic">no damage</span>; (<b>c</b>) Label: <span class="html-italic">no damage</span>; prediction: <span class="html-italic">destroyed</span>; (<b>d</b>) Label: <span class="html-italic">no damage</span>; prediction: <span class="html-italic">destroyed</span>.</p>
Full article ">Figure 9
<p>Scatter plot of the percentage of data points belonging to a class versus the recall of that class for each of the 13 tested disasters. The blue line shows the best polynomial fit and the blue area the 95% confidence interval.</p>
Full article ">Figure 10
<p>Distribution plot of the building footprint for buildings up to 700 m<math display="inline"><semantics> <msup> <mrow/> <mn>2</mn> </msup> </semantics></math>, i.e., 95% of the buildings. The lines correspond to the distributions over correctly and incorrectly classified samples.</p>
Full article ">Figure 11
<p>Scatter plots of the value of the parameter versus the AUC. One dot belongs to one disaster.</p>
Full article ">Figure 12
<p>Scatters plot of the value of the parameter versus the AUC. One dot belongs to one pre-disaster–post-disaster satellite image pair. The <span class="html-italic">x</span>-axis of the left column represents the sum of the parameter over the pre and post images. In the right column, the <span class="html-italic">x</span>-axis represents the absolute difference in the parameter value between the pre and post images.</p>
Full article ">Figure 13
<p>Confusion matrix of the model trained on the <span class="html-italic">four wind disasters</span> and tested on the Joplin tornado.</p>
Full article ">Figure 14
<p>Two buildings before and after the Joplin tornado, and the predictions made by the model trained on the Joplin tornado and the model trained on the <span class="html-italic">four wind disasters</span>.</p>
Full article ">Figure 15
<p>The true positive (TP), true negative (TN), false positive (FP), and false negative (FN) predictions on 2395 out of 12,165 buildings of the Joplin tornado by the model trained on a mixture of four disasters with wind damage.</p>
Full article ">Figure 16
<p>Confusion matrix of the model trained on the Midwest flooding and tested on 10% of the Nepal flooding.</p>
Full article ">Figure 17
<p>Two examples of post disaster buildings that were misclassified by the model trained on the Midwest flooding and tested on the Nepal flooding.</p>
Full article ">Figure 18
<p>The predictions, true negatives (TN), and false negatives (FN) on 9 out of the 29,808 buildings of the Nepal flooding by the model trained on the Midwest flooding.</p>
Full article ">
Back to TopTop