A Spatial-Spectral Decoupling Fusion Framework for Visible and Near-Infrared Images
Article No.: 43, Pages 1 - 7
Abstract
The visible and near-infrared image fusion aims to generate an image that integrates complementary information from images captured in different spectral bands. However, existing fusion methods either only focus on the fusion of spatial information, or fuse spatial and spectral information without decoupling, resulting in undesirable effects such as halo artifacts, information loss, and inferior visual quality. To address these issues, we propose a Spatial-Spectral Decoupling Fusion (SSDF) framework that can effectively fuse the spatial and spectral information of visible and near-infrared images. The SSDF framework decomposes the image pairs into two main branches: the Spatial Feature Enhancement (SFE) branch and the Spectral Characteristic Preservation (SCP) branch. The SFE branch enhances the salient details in the fused image by exploiting the contrast between spatial features and generating region-based fusion weights, while the SCP branch preserves the intrinsic spectral characteristics of the scene by fusing the reflectance characteristics of visible and near-infrared images. The final image is obtained by combining the spatial and spectral information. We conduct extensive experiments to show that our SSDF method can achieve superior fusion performance in subjective visual quality and objective metrics compared with state-of-the-art methods.
References
[1]
Mohamed Awad, Ahmed Elliethy, and Hussein A Aly. 2019. Adaptive near-infrared and visible fusion for fast image enhancement. IEEE Transactions on Computational Imaging 6 (2019), 408–418.
[2]
C Bellotti, F Bellotti, A De Gloria, L Andreone, and Michele Mariani. 2004. Developing a near infrared based night vision system. In IEEE Intelligent Vehicles Symposium, 2004. IEEE, 686–691.
[3]
Edward R Dougherty and Roberto A Lotufo. 2003. Hands-on morphological image processing. Vol. 59. SPIE press.
[4]
Chen Feng, Shaojie Zhuo, Xiaopeng Zhang, Liang Shen, and Sabine Süsstrunk. 2013. Near-infrared guided color image dehazing. In 2013 IEEE international conference on image processing. IEEE, 2363–2367.
[5]
Bo-Cai Gao. 1996. NDWI—A normalized difference water index for remote sensing of vegetation liquid water from space. Remote sensing of environment 58, 3 (1996), 257–266.
[6]
Nadine Gobron, Bernard Pinty, and Michel M Verstraete. 1997. Theoretical limits to the estimation of the leaf area index on the basis of visible and near-infrared remote sensing data. IEEE Transactions on Geoscience and Remote Sensing 35, 6 (1997), 1438–1445.
[7]
Yu Han, Yunze Cai, Yin Cao, and Xiaoming Xu. 2013. A new image fusion performance metric based on visual information fidelity. Information fusion 14, 2 (2013), 127–135.
[8]
David Hasler and Sabine E Suesstrunk. 2003. Measuring colorfulness in natural images. In Human vision and electronic imaging VIII, Vol. 5007. SPIE, 87–95.
[9]
Kaiming He, Jian Sun, and Xiaoou Tang. 2010. Single image haze removal using dark channel prior. IEEE transactions on pattern analysis and machine intelligence 33, 12 (2010), 2341–2353.
[10]
María Herrera-Arellano, Hayde Peregrina-Barreto, and Iván Terol-Villalobos. 2021. Visible-NIR image fusion based on top-hat transform. IEEE Transactions on Image Processing 30 (2021), 4962–4972.
[11]
María A Herrera-Arellano, Hayde Peregrina-Barreto, and Iván Terol-Villalobos. 2019. Color outdoor image enhancement by V-NIR fusion and weighted luminance. In 2019 IEEE International Autumn Meeting on Power, Electronics and Computing (ROPEC). IEEE, 1–6.
[12]
Chan-Gi Im, Dong-Min Son, Hyuk-Ju Kwon, and Sung-Hak Lee. 2022. Tone Image Classification and Weighted Learning for Visible and NIR Image Fusion. Entropy 24, 10 (2022), 1435.
[13]
Dong-Won Jang and Rae-Hong Park. 2017. Colour image dehazing using near-infrared fusion. IET Image Processing 11, 8 (2017), 587–594.
[14]
Jionghui Jiang, Fen Liu, Yingying Xu, Hui Huang, 2019. Multi-spectral RGB-NIR image classification using double-channel CNN. IEEE Access 7 (2019), 20607–20613.
[15]
Hyungjoo Jung, Youngjung Kim, Hyunsung Jang, Namkoo Ha, and Kwanghoon Sohn. 2020. Unsupervised deep image fusion with structure tensor representations. IEEE Transactions on Image Processing 29 (2020), 3845–3858.
[16]
FJ Kriegler, WA Malila, RF Nalepka, and W Richardson. 1969. Preprocessing transformations and their effects on multispectral recognition. Remote sensing of environment, VI (1969), 97.
[17]
Rahul Kumar, Brajesh Kumar Kaushik, and R Balasubramanian. 2019. Multispectral transmission map fusion method and architecture for image dehazing. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 27, 11 (2019), 2693–2697.
[18]
Hyuk-Ju Kwon and Sung-Hak Lee. 2021. Visible and near-infrared image acquisition and fusion for night surveillance. Chemosensors 9, 4 (2021), 75.
[19]
Shutao Li, Xudong Kang, Leyuan Fang, Jianwen Hu, and Haitao Yin. 2017. Pixel-level image fusion: A survey of the state of the art. information Fusion 33 (2017), 100–112.
[20]
Zhuo Li, Hai-Miao Hu, Wei Zhang, Shiliang Pu, and Bo Li. 2020. Spectrum characteristics preserved visible and near-infrared image fusion algorithm. IEEE Transactions on Multimedia 23 (2020), 306–319.
[21]
Yu Liu, Xun Chen, Juan Cheng, and Hu Peng. 2017. A medical image fusion method based on convolutional neural networks. In 2017 20th international conference on information fusion (Fusion). IEEE, 1–7.
[22]
Yu Liu, Lei Wang, Juan Cheng, Chang Li, and Xun Chen. 2020. Multi-focus image fusion: A survey of the state of the art. Information Fusion 64 (2020), 71–91.
[23]
Yue M Lu, Clément Fredembach, Martin Vetterli, and Sabine Süsstrunk. 2009. Designing color filter arrays for the joint capture of visible and near-infrared images. In 2009 16th IEEE International Conference on Image Processing (ICIP). IEEE, 3797–3800.
[24]
Yun Luo, Jeffrey Remillard, and Dieter Hoetzer. 2010. Pedestrian detection in near-infrared night vision system. In 2010 IEEE Intelligent Vehicles Symposium. IEEE, 51–58.
[25]
Aziz Makandar and Bhagirathi Halalli. 2015. Image enhancement techniques using highpass and lowpass filters. International Journal of Computer Applications 109, 14 (2015), 12–15.
[26]
Harvey B Mitchell. 2010. Image fusion: theories, techniques and applications. Springer Science & Business Media.
[27]
Nirmala Paramanandham and Kishore Rajendiran. 2018. Multi sensor image fusion for surveillance applications using hybrid image fusion algorithm. Multimedia Tools and Applications 77 (2018), 12405–12436.
[28]
Akikazu Sakudo. 2016. Near-infrared spectroscopy for medical applications: Current status and future perspectives. Clinica Chimica Acta 455 (2016), 181–188.
[29]
Neda Salamati, Zahra Sadeghipoor, and Sabine Süsstrunk. 2011. Analyzing near-infrared images for utility assessment. In Human Vision and Electronic Imaging XVI, Vol. 7865. SPIE, 62–70.
[30]
James R Shell II 2005. Polarimetric remote sensing in the visible to near infrared. (2005).
[31]
Chang-Hwan Son and Xiao-Ping Zhang. 2016. Near-infrared image dehazing via color regularization. arXiv preprint arXiv:1610.00175 (2016).
[32]
Chang-Hwan Son and Xiao-Ping Zhang. 2017. Near-infrared coloring via a contrast-preserving mapping model. IEEE Transactions on Image Processing 26, 11 (2017), 5381–5394.
[33]
Tania Stathaki. 2011. Image fusion: algorithms and applications. Elsevier.
[34]
Ashish V Vanmali and Vikram M Gadre. 2017. Visible and NIR image fusion using weight-map-guided Laplacian–Gaussian pyramid for improving scene visibility. Sādhanā 42 (2017), 1063–1082.
[35]
Ashish V Vanmali, Samrudha G Kelkar, and Vikram M Gadre. 2015. A novel approach for image dehazing combining visible-NIR images. In 2015 Fifth National Conference on Computer Vision, Pattern Recognition, Image Processing and Graphics (NCVPRIPG). IEEE, 1–4.
[36]
Farshad G Veshki and Sergiy A Vorobyov. 2022. Convolutional Simultaneous Sparse Approximation with Applications to RGB-NIR Image Fusion. In 2022 56th Asilomar Conference on Signals, Systems, and Computers. IEEE, 872–876.
[37]
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600–612.
[38]
Pu Xia, Xiaolai Chen, and Xiaohui Gao. 2021. Polarimetric dehazing utilizing Vis-NIR spectral image fusion. In 2021 International Conference on Intelligent Computing, Automation and Systems (ICICAS). IEEE, 48–54.
[39]
Li Xu, Qiong Yan, Yang Xia, and Jiaya Jia. 2012. Structure extraction from texture via relative total variation. ACM transactions on graphics (TOG) 31, 6 (2012), 1–10.
[40]
Costas S Xydeas, Vladimir Petrovic, 2000. Objective image fusion performance measure. Electronics letters 36, 4 (2000), 308–309.
[41]
Qiong Yan, Xiaoyong Shen, Li Xu, Shaojie Zhuo, Xiaopeng Zhang, Liang Shen, and Jiaya Jia. 2013. Cross-field joint image restoration via scale map. In Proceedings of the IEEE International Conference on Computer Vision. 1537–1544.
[42]
Guanyu Zhang, Beichen Sun, Yuehan Qi, and Yang Liu. 2022. Visible and Near Infrared Image Fusion Based on Texture Information. arXiv preprint arXiv:2207.10953 (2022).
[43]
Xingchen Zhang. 2021. Deep learning-based multi-focus image fusion: A survey and a comparative study. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 9 (2021), 4819–4838.
[44]
Yufeng Zheng. 2011. Image fusion and its applications. BoD–Books on Demand.
Index Terms
- A Spatial-Spectral Decoupling Fusion Framework for Visible and Near-Infrared Images
Recommendations
Fusion method for infrared and visible images based on improved quantum theory model
Quantum theory model (QTM) owns a superior characteristic to turn a complex problem into the form of the linear combination of several much simpler components. Therefore, a novel fusion method based on improved quantum theory model (IQTM) is proposed in ...
A Fourier Transform-based Approach to Fusion High Spatial Resolution Remote Sensing Images
ICVGIP '08: Proceedings of the 2008 Sixth Indian Conference on Computer Vision, Graphics & Image ProcessingThe modern remote sensing imaging sensors, like those in the IKONOS and QuickBird satellites, are capable of generating panchromatic images with one meter spatial resolution and multiespectral images with good spectral information. The principal ...
Fusion of near-infrared and visible images based on saliency-map-guided multi-scale transformation decomposition
AbstractIn this research, we propose a near-infrared (NIR) and visible (VIS) image fusion method based on saliency-map-guided multi-scale transform decomposition (SMG-MST) to solve the problem of color distortion. Although the existing NIR and VIS image ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2023
745 pages
ISBN:9798400702051
DOI:10.1145/3595916
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 01 January 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- the National Natural Science Foundation of China
- the Pioneer and Leading Goose R&D Program of Zhejiang
- the Fundamental Research Funds for the Central Universities
Conference
MMAsia '23
Sponsor:
Acceptance Rates
Overall Acceptance Rate 59 of 204 submissions, 29%
Upcoming Conference
MM '24
- Sponsor:
- sigmm
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 61Total Downloads
- Downloads (Last 12 months)61
- Downloads (Last 6 weeks)3
Reflects downloads up to 26 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format