Abstract
The integration of miniaturized spectrometers into mobile devices offers new avenues for image quality enhancement and facilitates novel downstream tasks. However, the broader application of spectral sensors in mobile photography is hindered by the inherent complexity of spectral images and the constraints of spectral imaging capabilities. To overcome these challenges, we propose a joint RGB-Spectral decomposition model guided enhancement framework, which consists of two steps: joint decomposition and prior-guided enhancement. Firstly, we leverage the complementarity between RGB and Low-resolution Multi-Spectral Images (Lr-MSI) to predict shading, reflectance, and material semantic priors. Subsequently, these priors are seamlessly integrated into the established HDRNet to promote dynamic range enhancement, color mapping, and grid expert learning, respectively. Additionally, we construct a high-quality Mobile-Spec dataset to support our research, and our experiments validate the effectiveness of Lr-MSI in the tone enhancement task. This work aims to establish a solid foundation for advancing spectral vision in mobile photography. The code is available at https://github.com/CalayZhou/JDM-HDRNet.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Gaiasky-mini hyperspectral imaging camera. https://www.dualix.com.cn/en/Goods/desc/id/123/aid/954.html
Spectral Sensing — ams.com. https://ams.com/spectral-sensing
Barrow, H., Tenenbaum, J., Hanson, A., Riseman, E.: Recovering intrinsic scene characteristics. Comput. vis. Syst. 2(3–26), 2 (1978)
Cai, J., Gu, S., Zhang, L.: Learning a deep single image contrast enhancer from multi-exposure images. IEEE Trans. Image Process. 27(4), 2049–2062 (2018)
Chen, X., et al.: Intrinsic decomposition from a single spectral image. Appl. Opt. 56(20), 5676–5684 (2017)
Chen, Y.S., Wang, Y.C., Kao, M.H., Chuang, Y.Y.: Deep photo enhancer: unpaired learning for image enhancement from photographs with GANs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6306–6314 (2018)
Cheng, Z., Zheng, Y., You, S., Sato, I.: Non-local intrinsic decomposition with near-infrared priors. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2521–2530 (2019)
Erba, I., et al.: Computational color constancy beyond RGB images: multispectral and temporal extensions (2024)
Finlayson, G.D., Zhu, Y.: Designing color filters that make cameras more colorimetric. IEEE Trans. Image Process. 30, 853–867 (2020)
Fu, X., Ying, Y.: Food safety evaluation based on near infrared spectroscopy and imaging: a review. Crit. Rev. Food Sci. Nutr. 56(11), 1913–1924 (2016)
Gharbi, M., Chen, J., Barron, J.T., Hasinoff, S.W., Durand, F.: Deep bilateral learning for real-time image enhancement. ACM Trans. Graph. (TOG) 36(4), 1–12 (2017)
Gijsenij, A., Gevers, T., Van De Weijer, J.: Computational color constancy: survey and experiments. IEEE Trans. Image Process. 20(9), 2475–2489 (2011)
Glatt, O., et al.: Beyond RGB: a real world dataset for multispectral imaging in mobile devices. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4344–4354 (2024)
He, J., Liu, Y., Qiao, Yu., Dong, C.: Conditional sequential modulation for efficient global image retouching. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12358, pp. 679–695. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58601-0_40
He, Q., Wang, R.: Hyperspectral imaging enabled by an unmodified smartphone for analyzing skin morphological features and monitoring hemodynamics. Biomed. Opt. Express 11(2), 895–910 (2020)
He, Q., Wang, R.K.: Analysis of skin morphological features and real-time monitoring using snapshot hyperspectral imaging. Biomed. Opt. Express 10(11), 5625–5638 (2019)
Huang, Q., et al.: Multispectral image intrinsic decomposition via subspace constraint. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6430–6439 (2018)
Jiang, J., Liu, D., Gu, J., Süsstrunk, S.: What is the space of spectral sensitivity functions for digital color cameras? In: 2013 IEEE Workshop on Applications of Computer Vision (WACV), pp. 168–179. IEEE (2013)
Karaimer, H.C., Brown, M.S.: A software platform for manipulating the camera imaging pipeline. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision – ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I, pp. 429–444. Springer International Publishing, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_26
Kim, S.J., Lin, H.T., Lu, Z., Süsstrunk, S., Lin, S., Brown, M.S.: A new in-camera imaging model for color computer vision and its application. IEEE Trans. Pattern Anal. Mach. Intell. 34(12), 2289–2302 (2012)
Kim, Y.T.: Contrast enhancement using brightness preserving bi-histogram equalization. IEEE Trans. Consum. Electron. 43(1), 1–8 (1997)
Land, E.H., McCann, J.J.: Lightness and retinex theory. Josa 61(1), 1–11 (1971)
Lindholm, V., et al.: Differentiating malignant from benign pigmented or non-pigmented skin tumours-a pilot study on 3D hyperspectral imaging of complex skin surfaces and convolutional neural networks. J. Clin. Med. 11(7), 1914 (2022)
Liu, C., Yang, H., Fu, J., Qian, X.: 4D LUT: learnable context-aware 4D lookup table for image enhancement. IEEE Trans. Image Process. 32, 4742–4756 (2023)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)
Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004). https://doi.org/10.1023/B:VISI.0000029664.99615.94
Van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(11) (2008)
Mantiuk, R., Daly, S., Kerofsky, L.: Display adaptive tone mapping. In: ACM SIGGRAPH 2008 papers, pp. 1–10 (2008)
McGonigle, A.J., et al.: Smartphone spectrometers. Sensors 18(1), 223 (2018)
Ng, P.C., Chi, Z., Verdie, Y., Lu, J., Plataniotis, K.N.: Hyper-skin: a hyperspectral dataset for reconstructing facial skin-spectra from RGB images. Adv. Neural Inf. Process. Syst. 36 (2024)
Rao, S., Huang, Y., Cui, K., Li, Y.: Anti-spoofing face recognition using a metasurface-based snapshot hyperspectral image sensor. Optica 9(11), 1253–1259 (2022)
e Silva, G.M., et al.: Smartphone-based spectrometry system as a prescreening assessment of copper and iron for real time control of water pollution. J. Environ. Manag. 323, 116214 (2022)
Thomas, J.B.: Illuminant estimation from uncalibrated multispectral images. In: 2015 Colour and Visual Computing Symposium (CVCS), pp. 1–6. IEEE (2015)
Wang, R., Zhang, Q., Fu, C.W., Shen, X., Zheng, W.S., Jia, J.: Underexposed photo enhancement using deep illumination estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6849–6857 (2019)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Yang, C., Jin, M., Xu, Y., Zhang, R., Chen, Y., Liu, H.: SepLUT: separable image-adaptive lookup tables for real-time image enhancement. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision – ECCV 2022, pp. 201–217. Springer Nature Switzerland, Cham (2022). https://doi.org/10.1007/978-3-031-19797-0_12
Yang, Z., Albrow-Owen, T., Cai, W., Hasan, T.: Miniaturization of optical spectrometers. Science 371(6528), eabe0722 (2021)
Yuan, L., Sun, J.: Automatic exposure correction of consumer photographs. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7575, pp. 771–785. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33765-9_55
Zeng, H., Cai, J., Li, L., Cao, Z., Zhang, L.: Learning image-adaptive 3D lookup tables for high performance photo enhancement in real-time. IEEE Trans. Pattern Anal. Mach. Intell. 44(4), 2058–2073 (2020)
Zhang, F., You, S., Li, Y., Fu, Y.: HSI-guided intrinsic image decomposition for outdoor scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 313–322 (2022)
Zhang, F., Zeng, H., Zhang, T., Zhang, L.: CLUT-Net: learning adaptively compressed representations of 3DLUTs for lightweight image enhancement. In: Proceedings of the 30th ACM International Conference on Multimedia, pp. 6493–6501 (2022)
Zheng, Y., Sato, I., Sato, Y.: Illumination and reflectance spectra separation of a hyperspectral image meets low-rank matrix factorization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1779–1787 (2015)
Acknowledgments
This research was supported by National Key Research and Development Program of China (2023YFF0713300), National Natural Science Foundation of China under Grant (62071216), Leading Technology of Jiangsu Basic Research Plan under Grant (BK20192003), and National Research Foundation Singapore Competitive Research Program (award number CRP29-2022-0003). We thank the device support from NJU-ZPTECH University-Enterprise Joint Laboratory for Computational Spectral Imaging.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhou, K. et al. (2025). Joint RGB-Spectral Decomposition Model Guided Image Enhancement in Mobile Photography. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15071. Springer, Cham. https://doi.org/10.1007/978-3-031-72624-8_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-72624-8_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72623-1
Online ISBN: 978-3-031-72624-8
eBook Packages: Computer ScienceComputer Science (R0)