research-article

Unsharp Mask Guided Filtering

Authors:

Efstratios Gavves,

Cees G. M. SnoekAuthors Info & Claims

IEEE Transactions on Image Processing, Volume 30

Pages 7472 - 7485

https://doi.org/10.1109/TIP.2021.3106812

Published: 01 January 2021 Publication History

Abstract

The goal of this paper is guided image filtering, which emphasizes the importance of structure transfer during filtering by means of an additional guidance image. Where classical guided filters transfer structures using hand-designed functions, recent guided filters have been considerably advanced through parametric learning of deep networks. The state-of-the-art leverages deep networks to estimate the two core coefficients of the guided filter. In this work, we posit that simultaneously estimating both coefficients is suboptimal, resulting in halo artifacts and structure inconsistencies. Inspired by unsharp masking, a classical technique for edge enhancement that requires only a single coefficient, we propose a new and simplified formulation of the guided filter. Our formulation enjoys a filtering prior from a low-pass filter and enables explicit structure transfer by estimating a single coefficient. Based on our proposed formulation, we introduce a successive guided filtering network, which provides multiple filtering results from a single network, allowing for a trade-off between accuracy and efficiency. Extensive ablations, comparisons and analysis show the effectiveness and efficiency of our formulation and network, resulting in state-of-the-art results across filtering tasks like upsampling, denoising, and cross-modality filtering. Code is available at <uri>https://github.com/shizenglin/Unsharp-Mask-Guided-Filtering</uri>.

References

[1]

M. Elad and A. Feuer, “Superresolution restoration of an image sequence: Adaptive filtering approach,” IEEE Trans. Image Process., vol. 8, no. 3, pp. 387–395, Mar. 1999.

Digital Library

[2]

M. R. Banham and A. K. Katsaggelos, “Spatially adaptive wavelet-based multiscale image restoration,” IEEE Trans. Image Process., vol. 5, no. 4, pp. 619–634, Apr. 1996.

Digital Library

[3]

S. P. Awate and R. T. Whitaker, “Unsupervised, information-theoretic, adaptive image filtering for image restoration,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 3, pp. 364–376, Mar. 2006.

Digital Library

[4]

W.-Y. Ma and B. S. Manjunath, “EdgeFlow: A technique for boundary detection and image segmentation,” IEEE Trans. Image Process., vol. 9, no. 8, pp. 1375–1388, Aug. 2000.

Digital Library

[5]

Y. Kang, C. Roh, S.-B. Suh, and B. Song, “A lidar-based decision-making method for road boundary detection using multiple Kalman filters,” IEEE Trans. Ind. Electron., vol. 59, no. 11, pp. 4360–4368, Nov. 2012.

[6]

M. Jacob and M. Unser, “Design of steerable filters for feature detection using canny-like criteria,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 8, pp. 1007–1019, Aug. 2004.

Digital Library

[7]

D. Dunn and W. E. Higgins, “Optimal Gabor filters for texture segmentation,” IEEE Trans. Image Process., vol. 4, no. 7, pp. 947–964, Jul. 1995.

Digital Library

[8]

T. P. Weldon, W. E. Higgins, and D. F. Dunn, “Efficient Gabor filter design for texture segmentation,” Pattern Recognit., vol. 29, no. 12, pp. 2005–2015, 1996.

Digital Library

[9]

T. Randen and J. H. Husoy, “Texture segmentation using filters with optimized energy separation,” IEEE Trans. Image Process., vol. 8, no. 4, pp. 571–582, Apr. 1999.

Digital Library

[10]

T. C. Aysal and K. E. Barner, “Quadratic weighted median filters for edge enhancement of noisy images,” IEEE Trans. Image Process., vol. 15, no. 11, pp. 3294–3310, Nov. 2006.

Digital Library

[11]

X. Guo, Y. Li, and H. Ling, “LIME: Low-light image enhancement via illumination map estimation,” IEEE Trans. Image Process., vol. 26, no. 2, pp. 982–993, Feb. 2017.

Digital Library

[12]

D. Min, J. Lu, and M. N. Do, “Depth video enhancement based on weighted mode filtering,” IEEE Trans. Image Process., vol. 21, no. 3, pp. 1176–1190, Mar. 2012.

Digital Library

[13]

K. He, J. Sun, and X. Tang, “Guided image filtering,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 6, pp. 1397–1409, Jun. 2013.

Digital Library

[14]

J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele, “Joint bilateral upsampling,” ACM Trans. Graph., vol. 26, no. 3, p. 96, Jul. 2007.

Digital Library

[15]

C. C. Pham, S. V. U. Ha, and J. W. Jeon, “Adaptive guided image filtering for sharpness enhancement and noise reduction,” in Proc. PSIVT, 2011, pp. 323–334.

[16]

M.-Y. Liu, O. Tuzel, and Y. Taguchi, “Joint geodesic upsampling of depth images,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2013, pp. 169–176.

[17]

M. Gharbi, J. Chen, J. T. Barron, S. W. Hasinoff, and F. Durand, “Deep bilateral learning for real-time image enhancement,” ACM Trans. Graph., vol. 36, no. 4, pp. 1–12, Jul. 2017.

Digital Library

[18]

J. Xie, R. S. Feris, and M.-T. Sun, “Edge-guided single depth image super resolution,” IEEE Trans. Image Process., vol. 25, no. 1, pp. 428–438, Jan. 2016.

Digital Library

[19]

F. Kou, W. Chen, C. Wen, and Z. Li, “Gradient domain guided image filtering,” IEEE Trans. Image Process., vol. 24, no. 11, pp. 4528–4539, Nov. 2015.

Digital Library

[20]

Z. Li, J. Zheng, Z. Zhu, W. Yao, and S. Wu, “Weighted guided image filtering,” IEEE Trans. Image Process., vol. 24, no. 1, pp. 120–129, Jan. 2015.

[21]

Z. Sun, B. Han, J. Li, J. Zhang, and X. Gao, “Weighted guided image filtering with steering kernel,” IEEE Trans. Image Process., vol. 29, pp. 500–508, 2020.

Digital Library

[22]

Y. Li, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Deep joint image filtering,” in Proc. ECCV, 2016, pp. 154–169.

[23]

T.-W. Hui, C. C. Loy, and X. Tang, “Depth map super-resolution by deep multi-scale guidance,” in Proc. ECCV, 2016, pp. 353–369.

[24]

B. Albahar and J.-B. Huang, “Guided image-to-image translation with bi-directional feature transformation,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2019, pp. 9016–9025.

[25]

H. Su, V. Jampani, D. Sun, O. Gallo, E. Learned-Miller, and J. Kautz, “Pixel-adaptive convolutional neural networks,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 11166–11175.

[26]

Y. Li, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Joint image filtering with deep convolutional networks,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 8, pp. 1909–1923, Aug. 2019.

Digital Library

[27]

J. Pan, J. Dong, J. S. Ren, L. Lin, J. Tang, and M.-H. Yang, “Spatially variant linear representation models for joint filtering,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 1702–1711.

[28]

K. Morishita, S. Yamagata, T. Okabe, T. Yokoyama, and K. Hamatani, “Unsharp masking for image enhancement,” U.S. Patent 4 794 531, Dec. 27, 1988.

[29]

A. Polesel, G. Ramponi, and V. J. Mathews, “Image enhancement via adaptive unsharp masking,” IEEE Trans. Image Process., vol. 9, no. 3, pp. 505–510, Mar. 2000.

Digital Library

[30]

G. Deng, “A generalized unsharp masking algorithm,” IEEE Trans. Image Process., vol. 20, no. 5, pp. 1249–1261, May 2011.

Digital Library

[31]

W. Ye and K. K. Ma, “Blurriness-guided unsharp masking,” IEEE Trans. Image Process, vol. 27, no. 9, pp. 4465–4477, Sep. 2018.

Digital Library

[32]

X. Shen, C. Zhou, L. Xu, and J. Jia, “Mutual-structure for joint filtering,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Dec. 2015, pp. 3406–3414.

[33]

R. J. Jevnisek and S. Avidan, “Co-occurrence filter,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 3184–3192.

[34]

X. Guo, Y. Li, J. Ma, and H. Ling, “Mutually guided image filtering,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 42, no. 3, pp. 694–707, Mar. 2020.

[35]

B. Ham, M. Cho, and J. Ponce, “Robust guided image filtering using nonconvex potentials,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 1, pp. 192–207, Jan. 2018.

[36]

H. Yin, Y. Gong, and G. Qiu, “Side window filtering,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 8758–8766.

[37]

Z. Farbman, R. Fattal, and D. Lischinski, “Edge-preserving decompositions for multi-scale tone and detail manipulation,” ACM Trans. Graph., vol. 27, no. 3, pp. 1–10, 2008.

Digital Library

[38]

H. Wu, S. Zheng, J. Zhang, and K. Huang, “Fast end-to-end trainable guided filter,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2018, pp. 1838–1847.

[39]

X. Deng and P. L. Dragotti, “Deep convolutional neural network for multi-modal image restoration and fusion,” IEEE Trans. Pattern Anal. Mach. Intell., early access, Apr. 2, 2020. 10.1109/TPAMI.2020.2984244.

[40]

I. Marivani, E. Tsiligianni, B. Cornelis, and N. Deligiannis, “Multimodal deep unfolding for guided image super-resolution,” IEEE Trans. Image Process., vol. 29, pp. 8443–8456, 2020.

[41]

I. Marivani, E. Tsiligianni, B. Cornelis, and N. Deligiannis, “Learned multimodal convolutional sparse coding for guided image super-resolution,” in Proc. IEEE Int. Conf. Image Process. (ICIP), Sep. 2019, pp. 2891–2895.

[42]

I. Marivani, E. Tsiligianni, B. Cornelis, and N. Deligiannis, “Joint image super-resolution via recurrent convolutional neural networks with coupled sparse priors,” in Proc. IEEE Int. Conf. Image Process. (ICIP), Oct. 2020, pp. 868–872.

[43]

I. Marivani, E. Tsiligianni, B. Cornelis, and N. Deligiannis, “Multimodal image super-resolution via deep unfolding with side information,” in Proc. 27th Eur. Signal Process. Conf. (EUSIPCO), Sep. 2019, pp. 1–5.

[44]

I. Marivani, E. Tsiligianni, B. Cornelis, and N. Deligiannis, “Designing CNNs for multimodal image super-resolution via the method of multipliers,” in Proc. 28th Eur. Signal Process. Conf. (EUSIPCO), Jan. 2021, pp. 780–783.

[45]

X. Deng and P. L. Dragotti, “Deep coupled ISTA network for multi-modal image super-resolution,” IEEE Trans. Image Process., vol. 29, pp. 1683–1698, 2020.

Digital Library

[46]

P. Song, X. Deng, J. F. C. Mota, N. Deligiannis, P. L. Dragotti, and M. R. D. Rodrigues, “Multimodal image super-resolution via joint sparse representations induced by coupled dictionaries,” IEEE Trans. Comput. Imag., vol. 6, pp. 57–72, 2020.

[47]

F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” in Proc. ICLR, 2015, pp. 1–13.

[48]

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 4, pp. 834–848, Apr. 2017.

[49]

V. Belagiannis, C. Rupprecht, G. Carneiro, and N. Navab, “Robust optimization for deep regression,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Dec. 2015, pp. 2830–2838.

[50]

P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 1125–1134.

[51]

N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from RGBD images,” in Proc. ECCV, 2012, pp. 746–760.

[52]

D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black, “A naturalistic open source movie for optical flow evaluation,” in Proc. ECCV, 2012, pp. 611–625.

[53]

D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in Proc. 8th IEEE Int. Conf. Comput. Vis. (ICCV), Jul. 2001, pp. 416–423.

[54]

K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising,” IEEE Trans. Image Process., vol. 26, no. 7, pp. 3142–3155, Jul. 2017.

Digital Library

[55]

X. Xu, Y. Ma, and W. Sun, “Learning factorized weight matrix for joint filtering,” in Proc. ICML, 2020, pp. 10587–10596.

[56]

S. Roth and M. J. Black, “Fields of experts,” Int. J. Comput. Vis., vol. 82, no. 2, p. 205, 2009.

Digital Library

[57]

S. Iizuka, E. Simo-Serra, and H. Ishikawa, “Let there be color!: Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification,” ACM Trans. Graph., vol. 35, no. 4, pp. 1–11, 2016.

Digital Library

[58]

J. Shi, Q. Yan, L. Xu, and J. Jia, “Hierarchical image saliency detection on extended CSSD,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 4, pp. 717–729, Apr. 2016.

Digital Library

[59]

Q. Hou, M.-M. Cheng, X. Hu, A. Borji, Z. Tu, and P. Torr, “Deeply supervised salient object detection with short connections,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 3203–3212.

[60]

S. He and R. W. Lau, “Saliency detection with flash and no-flash image pairs,” in Proc. ECCV, 2014, pp. 110–124.

[61]

G. Petschnigg, R. Szeliski, M. Agrawala, M. Cohen, H. Hoppe, and K. Toyama, “Digital photography with flash and no-flash image pairs,” ACM Trans. Graph., vol. 23, no. 3, pp. 664–672, Aug. 2004.

Digital Library

[62]

Q. Yanet al., “Cross-field joint image restoration via scale map,” in Proc. IEEE Int. Conf. Comput. Vis., Dec. 2013, pp. 1537–1544.

[63]

M. Brown and S. Süsstrunk, “Multi-spectral SIFT for scene category recognition,” in Proc. CVPR, Jun. 2011, pp. 177–184.

[64]

X. Wang, F. Dai, Y. Ma, J. Guo, Q. Zhao, and Y. Zhang, “Near-infrared image guided neural networks for color image denoising,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), May 2019, pp. 3807–3811.

Cited By

Kumar MBhandari A(2024)Unsupervised Real-Time Mobile-Based Highly Dark Image Texture Enhancement AppIEEE Transactions on Consumer Electronics10.1109/TCE.2024.335171170:1(608-616)Online publication date: 9-Jan-2024
https://dl.acm.org/doi/10.1109/TCE.2024.3351711
Shi ZMettes PSnoek C(2024)Focus for Free in Density-Based CountingInternational Journal of Computer Vision10.1007/s11263-024-01990-3132:7(2600-2617)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s11263-024-01990-3
Wang LMa C(2024)Adapting Pretrained Large-Scale Vision Models for Face Forgery DetectionMultiMedia Modeling10.1007/978-3-031-53302-0_6(71-85)Online publication date: 29-Jan-2024
https://dl.acm.org/doi/10.1007/978-3-031-53302-0_6
Show More Cited By

Recommendations

Guided Image Filtering

In this paper, we propose a novel explicit image filter called guided filter. Derived from a local linear model, the guided filter computes the filtering output by considering the content of a guidance image, which can be the input image itself or ...
Guided 3D point cloud filtering

3D point cloud has gained significant attention in recent years. However, raw point clouds captured by 3D sensors are unavoidably contaminated with noise resulting in detrimental efforts on the practical applications. Although many widely used point ...
Guided image filtering
ECCV'10: Proceedings of the 11th European conference on Computer vision: Part I

In this paper, we propose a novel type of explicit image filter - guided filter. Derived from a local linear model, the guided filter generates the filtering output by considering the content of a guidance image, which can be the input image itself or ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Image Processing

IEEE Transactions on Image Processing Volume 30, Issue

2021

5053 pages

ISSN:1057-7149

Issue’s Table of Contents

1941-0042 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Press

Publication History

Published: 01 January 2021

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kumar MBhandari A(2024)Unsupervised Real-Time Mobile-Based Highly Dark Image Texture Enhancement AppIEEE Transactions on Consumer Electronics10.1109/TCE.2024.335171170:1(608-616)Online publication date: 9-Jan-2024
https://dl.acm.org/doi/10.1109/TCE.2024.3351711
Shi ZMettes PSnoek C(2024)Focus for Free in Density-Based CountingInternational Journal of Computer Vision10.1007/s11263-024-01990-3132:7(2600-2617)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s11263-024-01990-3
Wang LMa C(2024)Adapting Pretrained Large-Scale Vision Models for Face Forgery DetectionMultiMedia Modeling10.1007/978-3-031-53302-0_6(71-85)Online publication date: 29-Jan-2024
https://dl.acm.org/doi/10.1007/978-3-031-53302-0_6
Song SZhong FWang TQin XTu C(2023)Guided Linear UpsamplingACM Transactions on Graphics10.1145/359245342:4(1-12)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592453
Zhong ZLiu XJiang JZhao DJi X(2023)Guided Depth Map Super-Resolution: A SurveyACM Computing Surveys10.1145/358486055:14s(1-36)Online publication date: 17-Jul-2023
https://dl.acm.org/doi/10.1145/3584860
Deng XXu JGao FSun XXu M(2023)Deep$\mathrm {M^{2}}$M2CDL: Deep Multi-Scale Multi-Modal Convolutional Dictionary Learning NetworkIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.333462446:5(2770-2787)Online publication date: 20-Nov-2023
https://dl.acm.org/doi/10.1109/TPAMI.2023.3334624
Sheng ZLiu XCao SShen HZhang H(2023)Frequency-Domain Deep Guided Image DenoisingIEEE Transactions on Multimedia10.1109/TMM.2022.321437525(6767-6781)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TMM.2022.3214375
Sun ZZhang MSun HLi JLiu TGao X(2023)Multi-modal deep convolutional dictionary learning for image denoisingNeurocomputing10.1016/j.neucom.2023.126918562:COnline publication date: 28-Dec-2023
https://dl.acm.org/doi/10.1016/j.neucom.2023.126918
Wang JZhang XXu XSong K(2022)Underwater Acoustic Point-cloud Filtering via Adaptive Unsharp MaskingProceedings of the 5th International Conference on Computer Science and Software Engineering10.1145/3569966.3570052(269-275)Online publication date: 21-Oct-2022
https://dl.acm.org/doi/10.1145/3569966.3570052

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents