Single-frame-based rain removal via image decomposition

SINGLE-FRAME-BASED RAIN REMOVAL VIA IMAGE DECOMPOSITION+ Yu-Hsiang Fu1, Li-Wei Kang2, Chia-Wen Lin1*, and Chiou-Ting Hsu3 1 Department of Electrical Engineering, National Tsing Hua University, Hsinchu, Taiwan 2 Institute of Information Science, Academia Sinica, Taipei, Taiwan 3 Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan ABSTRACT Rain removal from a video is a challenging problem and has been recently investigated extensively. Nevertheless, the problem of rain removal from a single image has been rarely studied in the literature, where no temporal information among successive images can be exploited, making it more challenging. In this paper, to the best of our knowledge, we are among the first to propose a single-frame-based rain removal framework via properly formulating rain removal as an image decomposition problem based on morphological component analysis (MCA). Instead of directly applying conventional image decomposition technique, we first decompose an image into the low-frequency and highfrequency parts using a bilateral filter. The high-frequency part is then decomposed into “rain component” and “nonrain component” via performing dictionary learning and sparse coding. As a result, the rain component can be successfully removed from the image while preserving most original image details. Experimental results demonstrate the efficacy of the proposed algorithm. Index Terms— Rain removal, sparse coding, dictionary learning, image decomposition, morphological component analysis (MCA). 1. INTRODUCTION Different weather conditions such as rain, snow, haze, or fog will cause complex visual effects of spatial or temporal domains in images or videos [1][4]. Such effects may significantly degrade the performances of outdoor vision systems relying on image/video feature extraction, such as object detection, tracking, recognition, indexing, and retrieval. Removal of rain streaks has recently received much attention. The first work for detecting and removing rain from videos was proposed in [1], where the authors developed a correlation model capturing the dynamics of rain and a physics-based motion blur model characterizing the photometry of rain. Then, the same authors [2] also showed that the camera parameters, such as exposure time and depth of field can be selected to mitigate the effects of rain without altering the appearance of the scene. Moreover, an improved video rain removal algorithm incorporating both temporal and chromatic properties was proposed in [3]. On the other hand, a model of the shape and appearance of a single rain or snow streak in the image space was developed in [4], which can be fit to a video and used to detect rain or snow streaks. Then, the amount of rain or snow in the video can be reduced or increased. So far, the research works on rain removal found in the literature have been mainly focused on video-based approaches that exploit information in multiple successive frames. Nevertheless, when only a single image is available, such as an image captured from a digital camera or downloaded from the Internet, a single-frame based rain removal approach is required, which was rarely investigated before. In this paper, we propose a single-frame-based rain removal framework via formulating rain removal as an image decomposition problem based on morphological component analysis (MCA) [5]-[6]. The major contribution of this paper is three-fold: (i) our scheme is among the first to formulate the single image rain removal problem as an image decomposition problem; (ii) the learning of the dictionary for decomposing rain steaks is self-contained, where no extra training samples are required in the dictionary learning stage; and (iii) no temporal or motion information among successive images is required. The rest of this paper is organized as follows. In Sec. 2, we briefly review the concepts of MCA-based image decomposition, sparse coding, and dictionary learning, respectively. Sec. 3 presents the proposed single-imagebased rain removal framework. In Sec. 4, simulation results are demonstrated. Finally, Sec. 5 concludes this paper. 2. MCA-BASED IMAGE DECOMPOSITION, SPARSE CODING, AND DICTIONARY LEARNING 2.1. MCA-based Image Decomposition Suppose that an image I of N pixels is a linear combination of S layers (called morphological components), denoted by S I ¦ Is , where Is denotes the s-th component, such as s 1 image I into ^Is `s 1 , the MCA algorithms [5], [6] iteratively geometric or textured component of I. To decompose the S minimize the energy function defined as: 2 1 S S S S E ^Is `s 1 ,^T s `s 1 I  ¦ s 1 Is  P ¦ s 1 E s Is ,T s , (1) 2 2 where T s denotes the sparse coefficients corresponding to Is with respect to the dictionary Ds described later, ʅ is a _______________________________________________________________________________________________________________________________ +This work was supported in part by the National Science Council, Taiwan, under Grants NSC99-2218-E-001-010 and NSC99-2811-E-001-006. *Corresponding author: cwlin@ee.nthu.edu.tw. 978-1-4577-0539-7/11/$26.00 ©2011 IEEE 1453 ICASSP 2011 k regularization parameter, and Es is the energy defined according to the type of Ds (global or local). For a global dictionary Ds RNuMs , N ≤ Ms, T s RMs is the sparse learn a dictionary Ds sparsifying y via solving the following optimization problem [7]: coefficients of Is with respect to Ds. The energy function Es for global dictionary is defined in [6] as 1 2 (2) E s Is ,T s Is  DsT s 2  O T s 1 , 2 where ʄ is a regularization parameter. Usually, to decompose an image into its geometric and textured components, traditional basis, such as wavelet or curvelet, is used as the global dictionary for representing the geometric component of the image. For a local dictionary Ds Rnums , n ≤ ms, T sk Rms is the where T k denotes the sparse coefficients of y with respect to Ds and ʄ is a regularization parameter, which can be efficiently solved via performing the K-SVD dictionary learning algorithm [7]. Finally, the image decomposition is achieved via iteratively performing the MCA algorithm to solve Is (while fixing Ds) and the dictionary learning algorithm to learn Ds (while fixing Is) until convergence [6]. sparse coefficients of the patch bsk Rn , k = 1, 2, …, N, extracted from Is, with respect to Ds. Each patch bsk can be extracted centralized with a pixel of Is and overlapped with adjacent patches. The energy function Es for the local dictionary can be defined as [6] E s Is ,T s 1 N §1 k · k 2 k ¦ ¨ bs  DsT s 2  O T s 1 ¸ , n k 1©2 ¹ (3) where the weight 1/n compensates for the redundancy factor introduced by the overlap between patches bsk . Usually, the local dictionary for representing the textured component of an image is constructed from the dictionary learning procedure described in Sec. 2.2. To decompose the image I into ^Is `s 1 , S the MCA algorithm [6] solves (1) by iteratively performing the two steps for each component Is, as follows: (i) update of the sparse coefficients: this step performs sparse coding via solving a convex non-smooth optimization to solve T s or ^T ` k N s k 1 to minimize E s Is ,T s ^ ` while fixing T s or T k N s k 1 Ds R ,T Rms §1 k 1 k 2 ·  DsT k  O T k ¸ , 2 1 ¹ (4) k 3. PROPOSED RAIN REMOVAL FRAMEWORK As shown in Fig. 1, in the proposed framework rain removal is formulated as an image decomposition problem. It can be observed from Fig. 2(b) that directly applying the MCAbased image decomposition algorithm described in Sec. 2 [6] by treating rain streaks as the textured component in an image will seriously blur the image even if the rain streaks can be removed. To prevent original image details from being removed together with rain streaks, we propose to first roughly decompose an image into the low-frequency (LF) part and the high-frequency (HF) part, as illustrated in Figs. 2(c) and 2(d), respectively. Obviously, the most basic information will be included in the LF part while the rain streaks and the other edge/texture information will be included in the HF part of an image. Then, the HF part is further decomposed into the “rain component” and “nonrain component,” as illustrated in Figs. 2(e) and 2(f), respectively, where the training exemplars are extracted from the image itself in the dictionary learning stage. The details and the problem formulation of the proposed scheme are elaborated in the following subsections. ^ ` while fixing Is; and (ii) update of the components: this step updates Is or bsk ¦¨© 2 y p min nums k N Input rain image I - k 1 Smoothing filter . 2.2. Sparse Coding and Dictionary Learning Sparse coding is the technique of finding a sparse representation for a signal with a small number of nonzero or significant coefficients corresponding to the atoms in a dictionary [5]-[7]. Recall from Sec. 2.1, it is required to construct a dictionary Ds containing the local structures of textures for sparsely representing each patch bsk extracted from the textured component Is of an image I. In some applications, we may use a set of available training exemplars (similar to the patches extracted from the component we want to extract) y k Rn , k = 1, 2, …, p, to 1454 High frequency part IHF Detection and extraction of rain patches Low frequency part ILF Dictionary Learning + Non-rain component Output rain-removed image INon-rain Image decomposition via sparse coding Rain component Fig. 1. Proposed rain removal framework. (a) (b) (c) (d) (e) (f) (g) (h) Fig. 2. Illustrations of rain removal for a single image: (a) the original rain image; (b) the rain-removed version of (a) with the image decomposition via MCA [6]; (c) the low-frequency part (separated by the bilateral filter [8]) of (a); (d) the high-frequency part of (a); (e) the rain component of (d); (f) the non-rain component of (d); (g) the learned dictionary for decomposing (d); and (h) the rain-removed version of (a) with the proposed scheme. 3.1. Preprocessing and Problem Formulation For an input rain image I, in the preprocessing stage, we apply an edge-preserving smoothing filter, called bilateral filter [8] to obtain the LF part ILF of I and roughly decompose I into the LF part (ILF) and HF part (IHF), i.e., I = ILF + IHF. Similar to the MCA algorithm proposed in [6], we select the curvelet basis DCurvelet as the global dictionary for representing the geometric component of IHF. For representing the rain component of IHF, we use a dictionary DRain learned from the training exemplars of rain streaks extracted from IHF (described in Sec. 3.2). Hence, we formulate the problem of rain removal for image I of N pixels as an image decomposition problem to minimize the energy function defined as ଶ ଵ ௞ ௞ ௞ ‫ܧ‬ሺ‫ܫ‬ுி ǡ ߠுி ሻ ൌ σே ௞ୀଵ ቀ ฮܾுி െ ‫ܦ‬ுி ߠுி ฮ ൅ ߣฮߠுி ฮ ቁ, (5) where THF denotes the sparse coefficients of IHF with respect to the dictionary ‫ܦ‬ுி ൌ ሾ‫ܦ‬஼௨௥௩௘௟௘௧ ȁ‫ܦ‬ோ௔௜௡ ሿ ‫ א‬௡ൈ௠ , n ≤ m, k bHF Rn represents the k-th patch extracted from IHF, k = 1, ଶ ଶ ଵ k k with 2, …, N, THF Rm is the sparse coefficients of bHF respect to ‫ܦ‬ுி , and ʄ is a regularization parameter. 3.2. Detection of Rain Streaks and Dictionary Learning In the dictionary learning stage, different from the MCA algorithm, where several training patches are usually collected in advance, in the proposed scheme, we extract the rain patches from IHF, to be the training exemplars. First, we detect and extract all the patches containing rain streaks 1455 from IHF. Based on the fact that the edge directions of rain streaks in a patch should be almost consistent, rain patches can be well distinguishable from other texture patches. To detect the rain streaks in a patch, we derive the two criteria: (i) the intensity of each pixel in a rain streak is larger than those of the neighboring pixels outside the streak; and (ii) the pixels in a rain streak can be projected onto a single position if the projection direction is correct. After extracting a set of rain patches y k Rn , k = 1, 2, …, p, from IHF as training exemplars for learning the dictionary DRain, we formulate the dictionary learning problem as: ଵ ௣ (6) ‹ σ௞ୀଵ ቀ ԡ‫ ݕ‬௞ െ ‫ܦ‬ோ௔௜௡ ߠ ௞ ԡଶଶ ൅ ߣԡߠ ௞ ԡଵ ቁ, where T k denotes the sparse coefficients of y with respect to Drain. Similar to [6], we also apply the K-SVD algorithm [7] to solve (6) to obtain the optimized DRain that consists of the atoms of rain streaks, as illustrated in Fig. 2(g). ଶ k 3.3. Removal of Rain Streaks Based on the dictionary DHF, we can perform sparse coding via applying the OMP (orthogonal matching pursuit) k algorithm [9] (also applied in [6]) for each patch bHF extracted from IHF via minimizing (5) to find its sparse k coefficients T HF . Different from the MCA algorithm [6], where the sparse coding and dictionary learning will be iteratively performed, we perform sparse coding only once for each patch with respect to DHF. Then, each reconstructed ௞ patch ܾுி can be used to recover either geometric ே௢௡ିோ௔௜௡ ோ௔௜௡ component ‫ܫ‬ுி or rain component ‫ܫ‬ுி of IHF based ௞ ෨ on the sparse coefficients ߠுி as follows. We let the ௞ to zeros to coefficients corresponding to DCurvelet in ߠ෨ுி ௞ obtain ߠ෨ுி̴ோ௔௜௡ , while the coefficients corresponding to DRain ௞ ௞ in ߠ෨ுி to zeros to obtain ߠ෨ுி̴ே௢௡ିோ௔௜௡ . Therefore, each ௞ ௞ patch ܾுி can be re-expressed as either ܾ෨ுி̴ே௢௡ିோ௔௜௡ ൌ ௞ ௞ ௞ ෨ ෨ , ‫ܦ‬஼௨௥௩௘௟௘௧ ൈ ߠுி̴ே௢௡ିோ௔௜௡ or ܾுி̴ோ௔௜௡ ൌ ‫ܦ‬ோ௔௜௡ ൈ ߠ෨ுி̴ோ௔௜௡ ே௢௡ିோ௔௜௡ ோ௔௜௡ or ‫ܫ‬ுி , which can be used to recover ‫ܫ‬ுி respectively, by averaging the pixel values in overlapping regions. Finally, the rain-removed version of I can be Non Rain obtained via I NonRain =ILF+ IHF , as illustrated in Fig. 2(h). (a) (b) 4. SIMULATION RESULTS To evaluate the performance of the proposed rain removal algorithm, we compared the proposed scheme with the bilateral filter proposed in [8] and the MCA-based image decomposition scheme [6]. The rain removal results obtained from the three evaluated schemes for the three test images are shown in Figs. 2-4, respectively, which demonstrate that the proposed scheme significantly outperforms the other two schemes. (a) (c) (d) Fig. 4. Comparison of rain removal results: (a) the original image; the rain-removed versions with: (b) the bilateral filter [8]; (c) the MCA-based image decomposition [6]; (d) the proposed scheme. 5. CONCLUSION In this paper, we have proposed a single-frame-based rain removal framework via formulating rain removal as an image decomposition problem solved by performing sparse coding and dictionary learning algorithms. Our experimental results show that the proposed scheme can effectively remove rain steaks without significantly blurring the original image details. For future work, the performance may be further improved by enhancing the sparse coding and dictionary learning steps. Moreover, the proposed scheme may be extended to remove other kinds of repeated textures. REFERENCES (b) [1] (c) (d) Fig. 3. Comparison of rain removal results: (a) the original rain image; the rain-removed versions via: (b) the bilateral filter [8]; (c) the MCA-based image decomposition [6]; and (d) the proposed scheme. As illustrated in Fig. 2(c), although the bilateral filter [8] can remove most rain streaks, it simultaneously removes other image detail as well. With the MCA-based image decomposition scheme [6], most image details, together with rain streaks, will be always filtered out in all the three test cases. The proposed scheme successfully removes most rain streaks while preserving most original image details (e.g., the bubbles in the red box in Fig. 3(d)). More test results can be found in [10]. 1456 K. Garg and S. K. Nayar, “Detection and removal of rain from videos,” Proc. CVPR, June 2004, vol. 1, pp. 528-535. [2] K. Garg and S. K. Nayar, “When does a camera see rain?” Proc. ICCV, Oct. 2005, vol. 2, pp. 1067-1074. [3] X. Zhang, et al., “Rain removal in video by combining temporal and chromatic properties,” Proc. ICME, July 2006, pp. 461-464. [4] P. C. Barnum, S. Narasimhan, and T. Kanade, “Analysis of rain and snow in frequency space,” IJCV, vol. 86, no. 2-3, pp. 256-274, 2010. [5] J. M. Fadili, J. L. Starck, J. Bobin, and Y. Moudden, “Image decomposition and separation using sparse representations: an overview,” Proc. of the IEEE, vol. 98, no. 6, pp. 983-994, June 2010. [6] G. Peyré, J. Fadili, and J. L. Starck, “Learning adapted dictionaries for geometry and texture separation,” Proc. SPIE, vol. 6701, 2007. [7] M. Aharon, M. Elad, and A. M. Bruckstein, “The K-SVD: An algorithm for designing of overcomplete dictionaries for sparse representation,” IEEE Trans. Signal Process., vol. 54, 2006. [8] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” Proc. ICCV, Bombay, India, Jan. 1998, pp. 839-846. [9] S. Mallat and Z. Zhang, “Matching pursuits with time-frequency dictionaries,” IEEE Trans. Signal Process., vol. 41, no. 12, Dec. 1993. [10] NTHU Rain Removal project. [Online]. Available: http://www.ee.nthu.edu.tw/cwlin/Rain_Removal/Rain_Removal.htm.