Asymmetric Dual-Decoder U-Net for Joint Rain and Haze Removal

The very first single-image rain removal methods were based on a priori knowledge. Morphological component analysis (MCA) [24] employs bilateral filters to extract high-frequency components from rain images, where the high-frequency components are further decomposed into “rain components” and “non-rain components” through dictionary learning and sparse coding. Luo et al. [36] proposed a single-image rain removal algorithm based on mutual exclusion dictionary learning. Gaussian mixture model prior knowledge [31] was utilized to accommodate multiple orientations and scales of rain streaks. In [62], Zhu et al. detected the approximate region, where the rain streaks were located, to guide the separation of the rain layer from the background layer. However, early models based on prior knowledge often suffer from a lack of stability in real scenarios [24, 31, 36]. Since 2017, deep learning approaches have been developed for rain removal tasks. Deep detail networks [16] narrowed the mapping from input to output and combined prior knowledge to capture high-frequency details, making the model stay focused on rain streaks information. By adding an iterative information feedback network, JORDER [53] used a binary mapping to locate rain streaks. A non-locally enhanced encoder-decoder structure [28] was proposed to capture long-range dependencies and leverage the hierarchical features of the convolutional layer. In [30], Li et al. proposed a deep recurrent convolutional neural network to remove rain marks located at different depths progressively. A density-aware multi-stream connectivity network was introduced for rain removal in [58]. By adding constraints to the cGAN [23], Zhang et al. [59] generated more photo-realistic results. A progressive contextual aggregation network [40] was proposed as a baseline for rain removal. A real-world rain dataset was constructed by Wang et al. [49]; they also incorporated spatial perception mechanisms into deraining networks. Recently, Zhu et al. [61] proposed a gated non-local depth residual network for image rain removal. Yu et al. [55] conducted a comprehensive analysis of various aspects of existing rain removal models and their robustness against adversarial attacks. Based on these analyses, they proposed a more robust approach to address this issue.

While significant progress has been made in the research on image rain removal, the existing studies lack consideration for real-world rainy scenarios, limiting their effectiveness in practical applications. In contrast, our methods take a more realistic approach by not only addressing rain streak occlusions commonly encountered in rainy weather but also considering the impact of haze, which is prevalent in the atmosphere, on atmospheric light. By incorporating these factors, our methods offer a more comprehensive and practical solution that better aligns with real-world conditions.

Similar to image rain removal methods, early work on image dehaze tended to employ statistical methods to acquire prior information by capturing patterns in haze-free images. Representative methods include Dark channel prior [19], color-line prior [15], color attenuation prior [63], and so forth. However, prior-based methods tend to distort colors and thus produce undesirable artifacts [15, 19, 63]. In the deep learning era, methods started to not rely on prior knowledge, but to estimate atmospheric light and the transmission map directly. For example, Cai et al. [5] proposed an end-to-end dehazing model named DehazeNet, where haze-free images are produced by learning the transmission rate. Similarly, Ren et al. [41] employed multi-scale deep neural networks to learn the mapping relationship between foggy images and their corresponding transmission maps, aiming to reduce the error in estimating the transmission maps. AODNet [27] reconstructed the atmospheric scattering model by leveraging an improved convolutional neural network to learn the mapping relationship between foggy and clean pairs. In [57], a single network was proposed to simultaneously learn the intrinsic relationship between transmission maps, atmospheric light, and clean images. Ren et al. [42] built an encoder-decoder neural network to enhance the dehazing process. A network with an enhancer and two generators was proposed by Qu et al. [39]. Chen et al. [9] proposed a patch map-based PMS-Net to effectively suppress the distorted color issue. Dong et al. [12] proposed MSBDN (Multi-Scale Boosted Dehazing Network) based on the U-Net architecture, incorporating boosting and error feedback as guiding principles. Although the method achieves good results, it suffers from a large number of parameters. Yeh et al. [54] decomposed hazy images into base components and detail components and proposed MSRL-DehazeNet, which is based on residual learning and U-Net architecture. Sun et al. [46] proposed SADNet based on the attention mechanism using a semi-supervised approach for solving practical problems. Song et al. [45] introduced Swin Transformer into image haze removal and proposed DehazeFormer, which achieved significant improvements on multiple datasets. Unlike image rain removal, image dehazing often considers the impact of haze on atmospheric light intensity, which can compensate for the limitations in rain removal methods. Our methods combine these approaches with the research on rain removal, resulting in a more realistic approach that better aligns with real-world scenarios.

Unlike previous single-task models, some researchers have also explored the simultaneous enhancement of both rain removal and haze removal in images. Hu et al. [21] built an imaging model for rain streaks and haze based on the visual effect of rain and the scene depth map to synthesize a realistic dataset named RainCityscapes. Han et al. [18] constructed a superimposed image dataset and proposed a simple yet general Blind Image Decomposition Network to decompose rain streaks, raindrops, and haze in a blind image decomposition setting. Kim et al. [25] proposed a frequency-based model for removing rain and haze, where the frequency-based model divided the input image into high-frequency and low-frequency parts with a guided filter and then employed a symmetric encoder-decoder network to remove rain and haze separately. Kulkarni and Murala [26] proposed a lightweight network that combines convolutions at different scales with spatial attention and channel attention mechanisms, employing a dual restoration mechanism to handle images affected by various weather conditions. Recently, Li et al. [29] used a neural structure search based approach to handle multiple weather situations; however, it has a large number of parameters as it uses multiple encoders for each weather removal task. Chen et al. [10] proposed a training approach based on knowledge distillation, considering the perspective of training strategies. They introduced a multi-teacher model and a single-student model, enabling a single model to handle various weather conditions without increasing the parameter size. Valanarasu et al. [47] proposed a single transformer-based encoder-decoder network while restored image with a learnable weather type query in the decoder to learn the type of weather degradation. Wang et al. [50] enhanced the U-Net architecture by adding a small decoder and a dilated convolution attention module. This enhancement enabled the network to capture both global information and finer details in high-resolution remote sensing images. After an in-depth study of related works, we have identified two primary factors that contribute to image quality degradation: contamination and scene information affected by atmospheric light. To effectively address these factors, we introduce an innovative asymmetric dual-branch structure, allowing independent processing of each category. By separately optimizing contamination removal and scene information recovery, our method achieves enhanced overall performance and improved image quality.

Let a third-order tensor, \({{\boldsymbol {I}}} \in \mathbb {R}^{C \times H \times W}\), denote an input image, where C, H, and W present the channel, height, and width of the image, respectively. In our application, both rain and haze are synthesized into the origin scene images as input images. Each input image \({\boldsymbol {I}}\) is labeled with its ground truth image \({\boldsymbol {I}}^{\mathrm{gt}}\) without rain and haze in the scene. Our ADU-Net \(f_{\theta }\), consisting of a single branch encoder \(f_{\mathrm{E}}\), and an asymmetric dual-decoder \(f_{\mathrm{AD}}\), can remove the rain and haze in the input image, such that the output of the ADU-Net, \({\boldsymbol {Y}} = f_{\theta }({\boldsymbol {I}})\), can restore its ground truth scene \({\boldsymbol {I}}^{\mathrm{gt}}\). The ADU-Net is trained to learn a set of parameters, \(\theta ^*\), with minimum empirical objective value \(\mathcal {L}({\boldsymbol {I}}^{\mathrm{gt}}, {\boldsymbol {Y}})\).

We first give a sketch of the proposed ADU-Net. In rain and haze removal applications, one ideal option is to employ the deep neural network to understand the scene of the input image and separate the rain and haze from the input image. In our work, we develop the ADU-Net to remove the rain and haze jointly. As shown in Figure 2, the ADU-Net is stacked by a single branch encoder and an asymmetric dual-decoder. In the encoder \(f_{\mathrm{E}}\), we have five convolutional blocks, with each denoted by \(\mathrm{Conv}_{i},~0 \le i \le 4\). The output of each convolutional block is denoted by \({\boldsymbol {F}}_i = \mathrm{Conv}_i({\boldsymbol {F}}_{i-1})\) and \({\boldsymbol {F}}_{-1} = {\boldsymbol {I}}\).

Then a following asymmetric dual-decoder \(f_{\mathrm{AD}}\) aims to recover the scene image without rain and haze (see Figure 2). The proposed asymmetric dual-decoder is stacked of a set of ADBs, which produce two streams of latent features, denoted by \({\boldsymbol {Z}}^{\mathrm{c}}_j\) and \({\boldsymbol {Z}}^{\mathrm{s}}_j\) in the j-th ADB. Specifically, the processing can be formulated asor

After the last ADB, each stream of latent features \({\boldsymbol {Z}}^{\mathrm{c}}_3\) or \({\boldsymbol {Z}}^{\mathrm{s}}_3\) is encoded by a convolutional block to recover the channel dimensions into the image space (e.g., \(C=3\)), as \({\boldsymbol {Y}}^{\mathrm{c}} = \mathrm{Conv}_5({\boldsymbol {Z}}^{\mathrm{c}}_3)\) and \({\boldsymbol {Y}}^{\mathrm{s}} = \mathrm{Conv}_5({\boldsymbol {Z}}^{\mathrm{s}}_3)\). We denote the \({\boldsymbol {Y}}^{\mathrm{c}}\) as the contamination residual, and \({\boldsymbol {Y}}^{\mathrm{s}}\) as the scene residual. Having the \({\boldsymbol {Y}}^{\mathrm{c}}\) and \({\boldsymbol {Y}}^{\mathrm{s}}\) at hand, one can obtain the restored scene image \({\boldsymbol {Y}}\) as

The network is optimized by the negative SSIM loss [51] as \(\mathcal {L}_{\mathrm{SSIM}} = -\mathrm{SSIM} ({\boldsymbol {I}}^{\mathrm{gt}}, {\boldsymbol {Y}})\). Note that the common practice uses both the negative SSIM loss and MSE loss as the objective. Empirically, we observed that a negative SSIM loss works better in the proposed ADU-Net, which will be justified in Section 4.4.

In this part, we will describe the asymmetric dual-decoder \(f_{\mathrm{AD}}\) in ADU-Net. As shown in Figure 2, \(f_{\mathrm{AD}}\) consists of four ADBs and a convolutional block, while the ADBs are stacked by two different instantiations (e.g., \(\mathrm{ADB}_0\) vs. \(\mathrm{ADB}_j,~j = 1,2,3\)). In this following, we will first describe \(\mathrm{ADB}_0\), a simple form of the block. Then with minor modifications, we can realize the \(\mathrm{ADB}_j, \ j = 1,2,3\) on top of the \(\mathrm{ADB}_0\).

The \(\mathrm{ADB}_0\) is a two-branch architecture (see Figure 2), which receives the \({\boldsymbol {F}}_3\) and \({\boldsymbol {F}}_4\) as input, and produces two latent features \({\boldsymbol {Z}}^{\mathrm{c}}_0\) and \({\boldsymbol {Z}}^{\mathrm{s}}_0\). In \(\mathrm{ADB}_0\), the two latent features are respectively encoded by two branches of network, namely, contamination residual net (denoted by \(g^{\mathrm{c}}\)), and scene residual net (denoted by \(g^{\mathrm{s}}\)), given byandContamination Residual Net. In the contamination residual net (\(g^{\mathrm{c}}\)), \({\boldsymbol {F}}_3\) and \({\boldsymbol {F}}_4\) are fed to a cCFF module to localize the rain and haze areas in the scene image, as

The details of CFF are illustrated in Figure 3(a). Given two feature maps \({\boldsymbol {F}}_3\) and \({\boldsymbol {F}}_4\) as input, it first fuses the two inputs by using element-wise addition and then feeds the fused feature maps to two-layer convolutional blocks to obtain the attention weights, formulated bywhere \(\sigma\), \(\mathrm{BN}\), and \(\mathrm{ReLU}\) are sigmoid function, batch normalization, and rectified linear unit activation, respectively. Here, the kernel size of \(\mathrm{Conv}\) is \(1\times 1\), which can be understood as applying a fully connected layer to the channel features.

Then, we can apply the attention weights to the input feature maps and obtain the fused output, as

The CFF module fuses the input feature maps, and the fusion weights are produced via the channel patterns. We further employ a self-attention mechanism to build the spatially long-range dependencies of the fused feature maps \({\boldsymbol {G}}^{\mathrm{s}}_0\), given bywhere W-MSA is the window multi-head self-attention from the Swin Transformer [35].

By fusing the input feature maps and processing them by the attention mechanism, we can obtain the contamination residual feature maps as

The contamination residual net (\(g^{\mathrm{c}}\)) aims to attend to the rainy and hazy regions, thereby highlighting the rain and haze components in the contamination residual feature maps.

Scene Residual Net. Since we can observe from the contamination residual (\({\boldsymbol {Y}}^{\mathrm{c}}\)) that it contains the scene information along with the rain and haze, we develop a scene residual net (\(g^{\mathrm{s}}\)), that can compensate for the removed scene information in the image. In doing so, the GCFF module is proposed to capture valuable global scene information of the image, and fuse features, as

As shown in Figure 3(b), \({\boldsymbol {F}}_3\) and \({\boldsymbol {F}}_4\) are first fused, and summarized to its global feature, aswhere \(\mathrm{GAP}\) indicates the global average pooling and \({\boldsymbol {m}}^{\mathrm{s}}_0\) indicates the resultant vector. Then, a two-layer convolutional block is used to modulate per element of the global feature \({\boldsymbol {m}}^{\mathrm{s}}_0\), written by

We can thereby fuse the input feature maps aswhere \({\boldsymbol {i}}\) indicates a unit vector with the same size as \({\boldsymbol {w}}^{\mathrm{s}}_0\).

After GCFF, we employ the SW-MSA to enhance the spatial interaction of the feature maps and obtain the scene residual features, described byandInstantiation of \({\boldsymbol {ADB}}_j\). The difference between \(\mathrm{ADB}_{j}, \ j \ne 0\) and \(\mathrm{ADB}_0\) is that \(\mathrm{ADB}_0\) receives two feature maps as input, while \(\mathrm{ADB}_j, \ j\ne 0\) includes three feature maps as input. To adapt the architecture of \(\mathrm{ADB}_0\) to \(\mathrm{ADB}_j, \ j \ne 0\), we make minor modifications (see Figure 2). Specifically, for any block, \(\mathrm{ADB}_j\), its input includes the output from the \(\left(j-1\right)\)-th ADB blcok, e.g., \({\boldsymbol {Z}}^{\mathrm{c}}_{j-1}, {\boldsymbol {Z}}^{\mathrm{s}}_{j-1} \in \mathbb {R}^{d \times h\times w}\), and from the \(\left(3-j\right)\)-th convolutional encoder, e.g., \({\boldsymbol {F}}_{3-j}\). We first concatenate the \({\boldsymbol {Z}}^{\mathrm{c}}_{j-1}, {\boldsymbol {Z}}^{\mathrm{s}}_{j-1}\), and reduce its dimension from \(2d \times h \times w\) to \(d \times h \times w\), asandwhere \(\mathrm{Conv_{in}}\) indicates a two-layer convolution block with batch normalization and rectified linear unit activation. Here, the kernel size of convolution layers is \(3 \times 3\).

With \({\boldsymbol {F}}_{3-j}\), the output of \(\mathrm{ADB}_j\) can be obtained asandHere, \(\mathrm{Conv_{out}}\) indicates a convolution layer with the kernel size of \(3 \times 3\) followed by a leaky rectified linear unit and another convolution layer with the kernel size of \(1 \times 1\) also followed by a leaky rectified linear unit.

In this work, we propose a novel architecture for the rain and haze removal task. Considering the network capacity and hardware overhead, we propose two sizes of networks. One is the lite network, called ADU-Net, and the other is the large network, called ADU-Net-plus. In Section 4, we present the details of two architectures. The network performance is also evaluated in Section 4.

Network Architecture. The overall neural architecture of the proposed network is shown in Figure 2. Table 1 lists the kernel size of the convolutional layers. In the encoder block, the feature maps are processed by the Batch Normalization [22] and ReLU [1] after the convolutional layer, i.e., \(\mathrm{Conv_0}\), \(\mathrm{Conv_1}\), \(\mathrm{Conv_2}\), \(\mathrm{Conv_3}\), and \(\mathrm{Conv_4}\). Then the max-pooling layer is employed to down-sample the feature maps in each layer. In the decoder block, we also list the kernel size in the convolutional layers (see Table 1), and employ the Leaky ReLU as the activation function. Having computational efficiency in mind, we develop two neural networks of different scales. The light one is denoted as ADU-Net, while the large one is denoted as ADU-Net-plus. As shown in Table 1, the difference between the two networks is merely the modification to the channel dimensions. The superiority of our network will be evaluated in Section 4.3.

Network Training. We implement our method using the PyTorch deep learning package [37]. All experiments are evaluated on NVIDIA GTX 2080ti GPUs. In the experiments for RainCityscapes [21], BID Rain datasets [18], and NH-HAZE [2], the input images are resized to \(512 \times 256\). For the SPA-Data, we follow the practice in [49], which uses original images with size of \(256 \times 256\). The Adam optimization scheme with an initial learning rate of 0.001 is used to optimize the network. We train the network for 100 epochs for RainCityscapes and BID Rain datasets, and 20 epochs for SPA-Data. The learning rate adjustment strategy is employed to realize the learning rate decay, where the learning rate is decayed by a factor of 0.1 when the accuracy of the network does not improve in five epochs.

We evaluate the proposed methods on two synthetic datasets, i.e., RainCityscapes [21], BID Rain [18], and two real-world datasets, i.e., SPA-Data [49], and NH-HAZE [2]. In the following, we will introduce these datasets and the statistics of each dataset are illustrated in Table 2.

RainCityscapes. The RainCityscapes dataset is synthesized from the Cityscapes dataset [11]. It takes 9,432 images synthesized from 262 Cityscapes images as the training set and 1,188 images synthesized from 33 Cityscapes images as the test set. All the selected images of Cityscapes are overcast, without obvious shadow. Rain streaks and haze are synthesized by different intensity maps. By adjusting the intensity of the rain streaks and haze, each original image can produce 36 different synthesized images. The results of different methods are reported in Table 3.

BID Rain. The BID Rain dataset is also synthesized from the Cityscapes dataset. It samples 2,975 images from the validation set of the Cityscapes dataset as a training set, and 500 images from the test set of the Cityscapes dataset as its test set. This is a complicated dataset as the images contain rain streaks, haze, snow, and raindrops. The rain streaks masks are sampled from Rain100L and Rain100H [53], and the snow masks are sampled from Snow 100K [34]. The haze masks include three different intensities originating from FoggyCityScape [44]. The raindrops are produced from the metaball model [4]. Those weather components are mixed with the images in the Cityscapes dataset using the physical imaging models [4, 19, 34, 44, 53]. In the training set, every image can be mixed with each weather component with random probabilities, and we evaluate our model in six different cases; the combinations of the weather components in each case are as follows (1) rain streaks; (2) rain streaks and snow; (3) rain streaks and light haze; (4) rain streaks and heavy haze; (5) rain streaks, moderate haze, and raindrops; and (6) rain streaks, snow, moderate haze, and raindrops. Refer to [18] for more details of the six settings. The results of different cases are shown in Table 4.

SPA-Data. The SPA-Data is a real-world dataset, which is cropped from 170 real rain videos, of which 86 videos are collected from StoryBlocks or YouTube, and 84 videos are captured by iPhone X or iPhone 6SP. Those videos cover outdoor fields, suburb scenes, and common urban scenes. This dataset contains 638,492 image pairs for training and 1,000 for testing. The results of SPA-Data are shown in Table 5.

NH-HAZE The NH-HAZE [2] is a valuable dataset for non-homogeneous haze research, as it offers ground truth images for evaluation. The dataset comprises 55 pairs of real-world outdoor scenes, where each pair consists of a hazy image and its corresponding haze-free counterpart. The non-homogeneous haze present in these images has been meticulously generated using a professional haze generator, ensuring an accurate representation of real-life haze conditions. The results on the NH-HAZE dataset are presented in Table 6.

Evaluation Protocol. In our experiments, the network performance is quantitatively evaluated by the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) metrics. A higher value of PSNR and SSIM indicates a better image recovery performance of the network.

To verify the advance of our method, we compare the performance of our method with current state-of-the-art methods across three datasets.

RainCityscapes. In the RainCityscapes dataset, we compare our methods to the state-of-the-art rain removal methods including RESCAN [28], PReNet [40], DuRN [33], RCDNet [48], SPANet [49], and MPRNet [56]. We also compare our methods with approaches that jointly remove the rain and haze, i.e., DAF-Net [20], DGNL-Net [21], WiperNet [26], TransWeather [47], and GTRain [3]. The comparison with haze removal methods, like EPDN [39], DCPDN [57], and AECR-Net [52], is also conducted. The results are reported in Table 3. We can find that our vanilla solution, i.e., ADU-Net, outperforms the existing state-of-the-art methods. In particular, it improves the PSNR/SSIM values of the DGNL-Net by 1.45/0.0017, indicating the superior design of our method. The plus version of our method, i.e., ADU-Net-plus, again brings performance gain over the AUD-Net, where the ADU-Net-plus improves the PSNR/SSIM values by 0.81/0.0021.

BID Rain. Since the scene in the RainCityscapes dataset only contains rain and haze information, we further evaluate our methods on the challenging dataset, BID Rain, to verify its generalization of working in complicated weather conditions. Table 4 illustrates the comparison of the model performance in each weather condition. We can observe that the proposed ADU-Net can outperform the BIDeN [18] in each of the cases. Especially in cases (2) and (3), the ADU-Net brings the maximum performance gain. One possible explanation is that the proposed ADU-Net is designed with a dual-branch decoder, which is tailored for the images in case (2) including the rain streaks and snow, or that in (3) including rain streaks and a light haze. However, the improvement in the other cases reveals the generalization of our proposal. Along with the ADU-Net, its plus version can significantly improve both PSNR/SSIM values, showing the superiority of our network architecture. In case (4), the performance of ADU-Net is lower than that of RCDNet [48], BIDeN [18], TransWeather [47], and GTrain [3]. One possible explanation is that the “heavy haze” covers the scenes, which makes it difficult for our network to produce the scene residual. Nevertheless, this issue is addressed by increasing the parameter size, supported by the performance in ADU-Net-plus.

SPA-Data. We also evaluate our methods in the large-scale dataset, SPA-Data. We compare our methods to the existing state-of-the-art methods in Table 5, including RESCAN [30], PReNet [40], SPANet [49], RCDNet [48], and WiperNet [26]. As shown in Table 5, the proposed methods outperform the existing methods by a large margin. For example, the improvements read of 2.72/0.0051 (PSNR/SSIM) from ADU-Net and 4.57/0.0090 from ADU-Net-plus, as compared to RCDNet, showing the strong performance of our network architecture. Indeed, although ADU-Net exhibits a slightly lower SSIM compared to WiperNet by 0.0020, its superiority in PSNR by 2.46 highlights its excellent performance. Furthermore, ADU-Net-plus outperforms WiperNet, leading in both PSNR and SSIM by 4.31 and 0.0019, respectively. These findings affirm the robustness and efficacy of our proposed methods for image rain removal in real-world scenarios.

NH-HAZE. To showcase the effectiveness of our approach, we conducted experiments using the real-world NH-HAZE dataset [2]. In Table 6, we compare our methods with state-of-the-art techniques, including MSBDN [12], FFA-Net [38], AECRNet [52], and DehazeFormer [45]. The results, as presented in Table 6, demonstrate significant performance improvements with our proposed methods surpassing existing approaches by a wide margin. For instance, when compared to DehazeFormer-S, ADU-Net and ADU-Net-plus exhibit remarkable improvements of 7.9/0.156 (PSNR/SSIM) and 8.99/0.159, respectively. These outcomes highlight the strong performance of our network architecture.

Comparison of Model Complexity and Time Cost. In addition to analyzing PSNR and SSIM, we also compare the complexity of our methods with the existing state-of-the-art methods in Table 7, including PReNet [40], RCDNet [48], AECRNet [52], and DGNL-Net [21]. We employ GFLOPs (Giga Floating Point Operations) to quantify the complexity of the model. Additionally, we assess the runtime complexity by averaging the training time per epoch (s/epoch). It is evident from the table that ADU-Net stands out with the smallest GFLOPs and the second shortest s/epoch. Although our model’s runtime is slightly higher compared to DGNL-Net [21], we have demonstrated superior performance on both PSNR and SSIM. In our proposed methods, the convolutional layers play a significant role in the overall computational load. We are proud to share that ADU-Net achieves an impressive 50% reduction in kernel size compared to ADU-Net-plus while maintaining an exceptional performance level of 95%. We believe this significant reduction in complexity makes ADU-Net a more efficient and practical choice for various applications. However, in cases where computational resources are abundant and time is not a constraint, choosing ADU-Net-plus to achieve better performance is also a viable option.

In this section, we conduct thorough ablation studies to verify the effectiveness per component in the proposed network. All studies in this section are conducted using ADU-Net on the RainCityscapes dataset.

Loss Function. In our implementation, the network is optimized by the negative SSIM loss, i.e., \(\mathcal {L}_{\mathrm{SSIM}}\). While in many practices of the low-level computer vision tasks, the MSE loss, i.e., \(\mathcal {L}_{\mathrm{MSE}}\), is employed [14]. In this study, we evaluate the effectiveness of each loss function. As shown in Table 8, we can find that each of the loss functions works better for our rain and haze removal task, and the network performance training from the two-loss functions is similar. However, the multi-task training, which optimizes the loss functions jointly, will degrade the network performance, indicating that the network may be saturated using one loss function, and the joint training will harm the network.

Effect of Dual-branch Architecture. Our work naively proposes a dual-branch architecture, i.e., asymmetric dual-decoder U-Net, for rain and haze removal tasks. In this study, we will justify the effectiveness of the dual-branch design in our task (shown in Figure 4). Table 9 shows the empirical comparison of three architectures, i.e., Residual U-Net, Dual-decoder U-Net, and the proposed ADU-Net. The “Residual U-Net” represents the structure shown in Figure 4(a), while the “Dual-Decoder U-Net” also represents the structure depicted in Figure 4(b). Table 9 verifies our design is reasonable, where the dual-decoder U-Net outperforms the vanilla version of the residual U-Net and our ADU-Net can further bring the performance gain to the dual-decoder U-Net.

The above study shows our design flow is reasonable. We further evaluate the effectiveness of the contamination residual branch and scene residual branch in ADU-Net (see the results in Table 10). As compared to the Residual U-Net, each branch can improve its performance, showing the effectiveness of the proposed residual branch. Also, we can observe that the combination of the proposed residual branches can achieve further improvement, indicating that those two decoders learn complementary features of the image. In Table 10, the first row, “Residual U-Net,” is the same as in Table 9. The second row, “+Contamination residual branch,” represents the model where the decoder of Residual U-Net is replaced with the Contamination residual branch, and similarly, the third row, “+Scene residual branch,” represents the model where the decoder of Residual U-Net is replaced with the Scene residual branch. From the experimental results, it can be observed that each branch contributes to the improvement of PSNR and SSIM. However, combining both branches in the model leads to greater improvement.

Effect of Self-attention Module. As for Table 11, we aim to demonstrate that using W-MSA and SW-MSA in both symmetric decoders is superior to using either one alone. The first row (Dual-decoder U-Net) represents the architecture shown in Figure 4(b). The second row (+W-MSA) indicates that W-MSA is used in both symmetric decoders. Compared to the first row, there is an improvement of 0.44/0.0037 in PSNR/SSIM, indicating that using W-MSA alone brings only limited improvement. Similarly, the third row (+SW-MSA) indicates that SW-MSA is used in both symmetric decoders. Compared to the first row, there is an improvement of 0.51/0.0036 in PSNR/SSIM, showing that using SW-MSA alone also brings a modest improvement. The third row (+W-MSA&SW-MSA) represents the utilization of both W-MSA and SW-MSA in the two branches. There is an improvement of 0.74/0.0040 in PSNR/SSIM compared to the first row. However, it should be noted that in our experiments, we focused on saving the model with the best PSNR during training and did not specifically optimize for SSIM. Therefore, we believe that the marginal decrease in SSIM does not necessarily indicate a decline in model performance. We acknowledge that the benefits brought by simultaneously using both W-MSA and SW-MSA may not be significant.

Effect of Feature Fusion Module. In the proposed architecture of the ADU-Net, each decoder block has two information flows, respectively encoding the contamination residual and the scene residual (see Figure 2). Each information flow yields the feature fusion w.r.t. the concern of physical properties. In this study, we evaluate our design. Table 12 shows ablations of the effectiveness of the feature fusion blocks. Each of the CFF or GCFF can improve the accuracy by about 0.2 PSNR value. However, combining those two blocks can further bring an outstanding performance gain on top of the individual one, around 0.6 PSNR value. This can greatly verify the good practice of the feature fusion blocks in our design.

To demonstrate the effectiveness of our approach for application, we conducted an evaluation of our approach using the RainCityscapes dataset, chosen for its comprehensive assessment of overall image restoration. The experimental results are presented in Table 13. As a baseline, we used DeepLabV3 [8] and performed semantic segmentation on the RainCityscapes dataset. The evaluation metrics included \(IoU_{{class}}\) (Intersection-over-Union for classes), \(iIoU_{class}\) (instance-level Intersection-over-Union for classes), \(IoU_{{category}}\) (Intersection-over-Union for categories), \(iIoU_{category}\) (instance-level Intersection-over-Union for categories), and accuracy.

We conducted four types of experiments: original rainy images from the RainCityscapes dataset (Rainy Images), rainy images derained by ADU-Net (Rain-free Images (ADU-Net)), rainy images derained by ADU-Net-plus (Rain-free Images (ADU-Net-plus)), and ground truth images from the RainCityscapes dataset (Rain-free Images (ground truth)). As shown in Table 13, the utilization of ADU-Net for removing rain and haze from the images resulted in improvements across various evaluation metrics. The \(IoU_{{class}}\) metric showed a notable improvement of 0.1047, while the \(iIoU_{{class}}\) increased by 0.0842. Furthermore, the \(IoU_{{category}}\) experienced a significant boost of 0.1241, and the \(iIoU_{{category}}\) demonstrated an even more substantial enhancement of 0.1896. Additionally, the accuracy metric showed a notable increase of 0.0754. Similar positive advancements were observed for ADU-Net-plus across all metrics.

It is noteworthy that the experimental results of the rainy images derained by ADU-Net (Rain-free Images (ADU-Net)) and ADU-Net-plus (Rain-free Images (ADU-Net-plus)) were comparable. Considering the minimal difference of only 0.81/0.0021 in terms of PSNR and SSIM, it indicates that there is a limited benefit in the application of semantic segmentation models when the image restoration reaches a certain level. Further improvements are necessary to achieve better results.

Along with the quantitative analysis in the above paragraphs, we further conduct qualitative analysis to verify the superiority of our work. In this study, we first illustrate the rain and haze removal performance between our work and existing SOTA methods in synthetic datasets (see Figure 5). Various real-world outdoor scenes are also evaluated (see Figure 6). The generalization of the proposed ADU-Net is further evaluated by removing other contamination, e.g., only rain in Figure 7, or rain and snow in Figure 8.

The first study is evaluated on the RainCityscapes dataset. We compare our method with the state-of-the-art methods, including PReNet [40], AECR-Net [52], and DGNL-Net [21]. As shown in Figure 5, our method can produce a much clearer scene image (see the red box for details). For example, in the fourth row of Figure 5, our method removes most of the haze and produces a clear shape of the tree branches, while other methods fail to recover the tree branches. This clearly shows the superiority of our method.

In the second study, we conduct the analysis on real-world images¹ used in [49], to justify the potential of our method in real scenarios. We again compare our method to PReNet, AECR-Net, and DGNL-Net. For a fair comparison, each method adopts publicly available fine-tuned weights trained on their own datasets. As can be observed from Figure 6, the scene images, generated by our method, are clearer and more realistic than those from other methods. For example, as compared to the rain removal network PReNet, our method can also remove the haze in real-world scenes. The hues of the recovered scene from our method are also more realistic than that from the dehazing network AECR-Net and reflective details of the scenes are maintained by our method. As compared to DGNL-Net, the closest work to ours, our ADU-Net can remove more rain streaks (the second row) or haze (the third row) and retain more scene details (the first row). This study can vividly show the effectiveness of our method in real scenarios.

To demonstrate the generalization of our dual-decoder architecture in separating different contamination, we show the residual produced by different branches. Figure 7 shows the results of our method on the BID Rain dataset. The first row is the input image. The second row and third row present the masks of contamination residual and scene residual, respectively. The fourth row and fifth row are the generated images and the ground truth, respectively. We can find that our method separates the contamination (e.g., snow or haze) and scene clearly, and produces high-quality scene images. A similar observation is also made in the real-world images from Internet-Data in Figure 8. This study also verifies our motivation that most of the contamination components in the image are included in the contamination residual while the scene residual contains more details of the scene including building structures and driveway lines. This analysis again illustrates the superior generalization of the proposed method.