Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3581783.3612436acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Fearless Luminance Adaptation: A Macro-Micro-Hierarchical Transformer for Exposure Correction

Published: 27 October 2023 Publication History

Abstract

Photographs taken with less-than-ideal exposure settings often display poor visual quality. Since the correction procedures vary significantly, it is difficult for a single neural network to handle all exposure problems. Moreover, the inherent limitations of convolutions, hinder the models ability to restore faithful color or details on extremely over-/under-exposed regions. To overcome these limitations, we propose a Macro-Micro-Hierarchical transformer, which consists of a macro attention to capture long-range dependencies, a micro attention to extract local features, and a hierarchical structure for coarse-to-fine correction. In specific, the complementary macro-micro attention designs enhance locality while allowing global interactions. The hierarchical structure enables the network to correct exposure errors of different scales layer by layer. Furthermore, we propose a contrast constraint and couple it seamlessly in the loss function, where the corrected image is pulled towards the positive sample and pushed away from the dynamically generated negative samples. Thus the remaining color distortion and loss of detail can be removed. We also extend our method as an image enhancer for low-light face recognition and low-light semantic segmentation. Experiments demonstrate that our approach obtains more attractive results than state-of-the-art methods quantitatively and qualitatively.

References

[1]
Mahmoud Afifi, Konstantinos G Derpanis, Bjorn Ommer, and Michael S Brown. 2021. Learning multi-scale photo exposure correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9157--9167.
[2]
Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo Durand. 2011. Learning photographic global tonal adjustment with a database of input/output image pairs. In CVPR 2011. IEEE, 97--104.
[3]
Bolun Cai, Xianming Xu, Kailing Guo, Kui Jia, Bin Hu, and Dacheng Tao. 2017. A joint intrinsic-extrinsic prior model for retinex. In Proceedings of the IEEE international conference on computer vision. 4000--4009.
[4]
Jiawen Chen, Andrew Adams, Neal Wadhwa, and Samuel W Hasinoff. 2016. Bilateral guided upsampling. ACM Transactions on Graphics (TOG), Vol. 35, 6 (2016), 1--8.
[5]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 1597--1607.
[6]
Yu-Sheng Chen, Yu-Ching Wang, Man-Hsin Kao, and Yung-Yu Chuang. 2018. Deep photo enhancer: Unpaired learning for image enhancement from photographs with gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6306--6314.
[7]
Ziteng Cui, Kunchang Li, Lin Gu, Shenghan Su, Peng Gao, Zhengkai Jiang, Yu Qiao, and Tatsuya Harada. 2022. Illumination Adaptive Transformer. arXiv preprint arXiv:2205.14871 (2022).
[8]
Ziteng Cui, Guo-Jun Qi, Lin Gu, Shaodi You, Zenghui Zhang, and Tatsuya Harada. 2021. Multitask aet with orthogonal tangent regularity for dark object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2553--2562.
[9]
Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. 2021. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, Vol. 34 (2021), 3965--3977.
[10]
Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafał K Mantiuk, and Jonas Unger. 2017. HDR image reconstruction from a single exposure using deep CNNs. ACM transactions on graphics (TOG), Vol. 36, 6 (2017), 1--15.
[11]
F Eyiokur, Dogucan Yaman, Hazim Kemal Ekenel, and Alexander Waibel. 2022. Exposure Correction Model to Enhance Image Quality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 676--686.
[12]
Xueyang Fu, Delu Zeng, Yue Huang, Xiao-Ping Zhang, and Xinghao Ding. 2016. A weighted variational model for simultaneous reflectance and illumination estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2782--2790.
[13]
Chunle Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong. 2020. Zero-reference deep curve estimation for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1780--1789.
[14]
Xiaojie Guo. 2016. LIME: A method for low-light image enhancement. In Proceedings of the 24th ACM international conference on Multimedia. 87--91.
[15]
David Hasler and Sabine E Suesstrunk. 2003. Measuring colorfulness in natural images. In Human vision and electronic imaging VIII, Vol. 5007. SPIE, 87--95.
[16]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, Vol. 30 (2017).
[17]
Yuanming Hu, Hao He, Chenxi Xu, Baoyuan Wang, and Stephen Lin. 2018. Exposure: A white-box photo post-processing framework. ACM Transactions on Graphics (TOG), Vol. 37, 2 (2018), 1--17.
[18]
Jie Huang, Yajing Liu, Xueyang Fu, Man Zhou, Yang Wang, Feng Zhao, and Zhiwei Xiong. 2022a. Exposure Normalization and Compensation for Multiple-Exposure Correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 6043--6052.
[19]
Jie Huang, Feng Zhao, Man Zhou, Jie Xiao, Naishan Zheng, Kaiwen Zheng, and Zhiwei Xiong. 2023. Learning Sample Relationship for Exposure Correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 9904--9913.
[20]
Jie Huang, Man Zhou, Yajing Liu, Mingde Yao, Feng Zhao, and Zhiwei Xiong. 2022b. Exposure-Consistency Representation Learning for Exposure Correction. In Proceedings of the 30th ACM International Conference on Multimedia. 6309--6317.
[21]
Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey, and Luc Van Gool. 2017. Dslr-quality photos on mobile devices with deep convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision. 3277--3285.
[22]
Zhiying Jiang, Zhuoxiao Li, Shuzhou Yang, Xin Fan, and Risheng Liu. 2022. Target oriented perceptual adversarial fusion network for underwater image enhancement. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 32, 10 (2022), 6584--6598.
[23]
Feng Zhao Keyu Yan Jinghao Zhang Yukun Huang Man Zhou Zhiwei Xiong Jie Huang, Yajing Liu. 2022. Deep Fourier-based Exposure Correction Network with Spatial-Frequency Interaction. In Proceedings of the European Conference on Computer Vision (ECCV).
[24]
Hanul Kim, Su-Min Choi, Chang-Su Kim, and Yeong Jun Koh. 2021. Representative color transform for image enhancement. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4459--4468.
[25]
Dokyeong Kwon, Guisik Kim, and Junseok Kwon. 2020. DALE: Dark region-aware low-light image enhancement. arXiv preprint arXiv:2008.12493 (2020).
[26]
Hongjae Lee, Changwoo Han, and Seung-Won Jung. 2022. GPS-GLASS: Learning Nighttime Semantic Segmentation Using Daytime Video and GPS data. arXiv preprint arXiv:2207.13297 (2022).
[27]
Attila Lengyel, Sourav Garg, Michael Milford, and Jan C van Gemert. 2021. Zero-shot day-night domain adaptation with a physics prior. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4399--4409.
[28]
Jiawei Li, Jinyuan Liu, Shihua Zhou, Qiang Zhang, and Nikola K Kasabov. 2022. Learning a coordinated network for detail-refinement multiexposure image fusion. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 33, 2 (2022), 713--727.
[29]
Jinxiu Liang, Jingwen Wang, Yuhui Quan, Tianyi Chen, Jiaying Liu, Haibin Ling, and Yong Xu. 2021. Recurrent exposure generation for low-light face detection. IEEE Transactions on Multimedia, Vol. 24 (2021), 1609--1621.
[30]
Jinyuan Liu, Xin Fan, Zhanbo Huang, Guanyao Wu, Risheng Liu, Wei Zhong, and Zhongxuan Luo. 2022a. Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5802--5811.
[31]
Jinyuan Liu, Xin Fan, Ji Jiang, Risheng Liu, and Zhongxuan Luo. 2021a. Learning a deep multi-scale feature ensemble and an edge-attention guidance for image fusion. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 32, 1 (2021), 105--119.
[32]
Jinyuan Liu, Jingjie Shang, Risheng Liu, and Xin Fan. 2022d. Attention-guided global-local adversarial learning for detail-preserving multi-exposure image fusion. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 32, 8 (2022), 5026--5040.
[33]
Jinyuan Liu, Guanyao Wu, Junsheng Luan, Zhiying Jiang, Risheng Liu, and Xin Fan. 2023. HoLoCo: Holistic and local contrastive learning network for multi-exposure image fusion. Information Fusion, Vol. 95 (2023), 237--249.
[34]
Jinyuan Liu, Yuhui Wu, Zhanbo Huang, Risheng Liu, and Xin Fan. 2021d. Smoa: Searching a modality-oriented architecture for infrared and visible image fusion. IEEE Signal Processing Letters, Vol. 28 (2021), 1818--1822.
[35]
Risheng Liu, Zhiying Jiang, Shuzhou Yang, and Xin Fan. 2022b. Twin adversarial contrastive learning for underwater image enhancement and beyond. IEEE Transactions on Image Processing, Vol. 31 (2022), 4922--4936.
[36]
Risheng Liu, Jinyuan Liu, Zhiying Jiang, Xin Fan, and Zhongxuan Luo. 2020. A bilevel integrated model with data-driven layer ensemble for multi-modality image fusion. IEEE Transactions on Image Processing, Vol. 30 (2020), 1261--1274.
[37]
Risheng Liu, Long Ma, Tengyu Ma, Xin Fan, and Zhongxuan Luo. 2022c. Learning with nested scene modeling and cooperative architecture search for low-light vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 45, 5 (2022), 5953--5969.
[38]
Risheng Liu, Long Ma, Jiaao Zhang, Xin Fan, and Zhongxuan Luo. 2021c. Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10561--10570.
[39]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021b. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 10012--10022.
[40]
Feifan Lv, Feng Lu, Jianhua Wu, and Chongsoon Lim. 2018. MBLLEN: Low-Light Image/Video Enhancement Using CNNs. In BMVC, Vol. 220. 4.
[41]
Long Ma, Risheng Liu, Yiyang Wang, Xin Fan, and Zhongxuan Luo. 2022a. Low-light image enhancement via self-reinforced retinex projection model. IEEE Transactions on Multimedia (2022).
[42]
Long Ma, Tianjiao Ma, Xinwei Xue, Xin Fan, Zhongxuan Luo, and Risheng Liu. 2022b. Practical Exposure Correction: Great Truths Are Always Simple. arXiv preprint arXiv:2212.14245 (2022).
[43]
Sean Moran, Pierre Marza, Steven McDonagh, Sarah Parisot, and Gregory Slabaugh. 2020. Deeplpf: Deep local parametric filters for image enhancement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12826--12835.
[44]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234--241.
[45]
Christos Sakaridis, Dengxin Dai, and Luc Van Gool. 2021. ACDC: The adverse conditions dataset with correspondences for semantic driving scene understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 10765--10775.
[46]
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1874--1883.
[47]
Haoyuan Wang, Ke Xu, and Rynson WH Lau. 2022c. Local color distributions prior for image enhancement. In European Conference on Computer Vision. Springer, 343--359.
[48]
Ruixing Wang, Qing Zhang, Chi-Wing Fu, Xiaoyong Shen, Wei-Shi Zheng, and Jiaya Jia. 2019. Underexposed photo enhancement using deep illumination estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6849--6857.
[49]
Wenjing Wang, Xinhao Wang, Wenhan Yang, and Jiaying Liu. 2022b. Unsupervised face detection in the dark. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 45, 1 (2022), 1250--1266.
[50]
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. 2022a. Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 17683--17693.
[51]
Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. 2018. Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv:1808.04560 (2018).
[52]
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. 2021. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 22--31.
[53]
Jun Xu, Yingkun Hou, Dongwei Ren, Li Liu, Fan Zhu, Mengyang Yu, Haoqian Wang, and Ling Shao. 2020. Star: A structure and texture aware retinex model. IEEE Transactions on Image Processing, Vol. 29 (2020), 5022--5037.
[54]
Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu. 2020a. From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3063--3072.
[55]
Wenhan Yang, Ye Yuan, Wenqi Ren, Jiaying Liu, Walter J Scheirer, Zhangyang Wang, Taiheng Zhang, Qiaoyong Zhong, Di Xie, Shiliang Pu, et al. 2020b. Advancing image understanding in poor visibility environments: A collective benchmark study. IEEE Transactions on Image Processing, Vol. 29 (2020), 5737--5752.
[56]
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. 2021. Incorporating convolution designs into visual transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 579--588.
[57]
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. 2022. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5728--5739.
[58]
Haokui Zhang, Wenze Hu, and Xiaoyu Wang. 2022a. EdgeFormer: Improving Light-weight ConvNets by Learning from Vision Transformers. arXiv preprint arXiv:2203.03952 (2022).
[59]
Jinghao Zhang, Jie Huang, Mingde Yao, Man Zhou, and Feng Zhao. 2022b. Structure- and Texture-Aware Learning for Low-Light Image Enhancement. In Proceedings of the 30th ACM International Conference on Multimedia. 6483--6492.
[60]
Qing Zhang, Ganzhao Yuan, Chunxia Xiao, Lei Zhu, and Wei-Shi Zheng. 2018. High-quality exposure correction of underexposed photos. In Proceedings of the 26th ACM international conference on Multimedia. 582--590.
[61]
Yonghua Zhang, Xiaojie Guo, Jiayi Ma, Wei Liu, and Jiawan Zhang. 2021a. Beyond brightening low-light images. International Journal of Computer Vision, Vol. 129 (2021), 1013--1037.
[62]
Yonghua Zhang, Jiawan Zhang, and Xiaojie Guo. 2019. Kindling the darkness: A practical low-light image enhancer. In Proceedings of the 27th ACM international conference on multimedia. 1632--1640.
[63]
Zhaoyang Zhang, Yitong Jiang, Jun Jiang, Xiaogang Wang, Ping Luo, and Jinwei Gu. 2021b. Star: A structure-aware lightweight transformer for real-time image enhancement. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4106--4115.
[64]
Naishan Zheng, Jie Huang, Feng Zhao, Xueyang Fu, and Feng Wu. 2022. Unsupervised Underexposed Image Enhancement via Self-Illuminated and Perceptual Guidance. IEEE Transactions on Multimedia (2022), 1--16. https://doi.org/10.1109/TMM.2022.3193059

Cited By

View all
  • (2024)MECNet: Multi-Scale Exposure-Consistency Learning via Fourier Transform for Exposure Correction2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC)10.1109/SMC54092.2024.10831842(2440-2446)Online publication date: 6-Oct-2024
  • (2024)Illumination-guided dual-branch fusion network for partition-based image exposure correctionJournal of Visual Communication and Image Representation10.1016/j.jvcir.2024.104342(104342)Online publication date: Nov-2024
  • (2024)Leveraging a self-adaptive mean teacher model for semi-supervised multi-exposure image fusionInformation Fusion10.1016/j.inffus.2024.102534112:COnline publication date: 18-Oct-2024

Index Terms

  1. Fearless Luminance Adaptation: A Macro-Micro-Hierarchical Transformer for Exposure Correction

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '23: Proceedings of the 31st ACM International Conference on Multimedia
    October 2023
    9913 pages
    ISBN:9798400701085
    DOI:10.1145/3581783
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 27 October 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. exposure correction
    2. image restoration
    3. low-light enhancement
    4. low-light face detection
    5. low-light semantic segmentation

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    MM '23
    Sponsor:
    MM '23: The 31st ACM International Conference on Multimedia
    October 29 - November 3, 2023
    Ottawa ON, Canada

    Acceptance Rates

    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)94
    • Downloads (Last 6 weeks)8
    Reflects downloads up to 16 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)MECNet: Multi-Scale Exposure-Consistency Learning via Fourier Transform for Exposure Correction2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC)10.1109/SMC54092.2024.10831842(2440-2446)Online publication date: 6-Oct-2024
    • (2024)Illumination-guided dual-branch fusion network for partition-based image exposure correctionJournal of Visual Communication and Image Representation10.1016/j.jvcir.2024.104342(104342)Online publication date: Nov-2024
    • (2024)Leveraging a self-adaptive mean teacher model for semi-supervised multi-exposure image fusionInformation Fusion10.1016/j.inffus.2024.102534112:COnline publication date: 18-Oct-2024

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media