research-article

Video Inverse Tone Mapping Network with Luma and Chroma Mapping

Authors:

Guoping QiuAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 1383 - 1391

https://doi.org/10.1145/3581783.3612199

Published: 27 October 2023 Publication History

Abstract

\beginabstract With the popularity of consumer high dynamic range (HDR) display devices, video inverse tone mapping (iTM) has become a research hotspot. However, existing methods are designed based on a perceptual non-uniformity color space (e.g., RGB and YC_BC_R), resulting in limited quality of HDR video rendered by these methods. Considering the two key factors involved in the video iTM task: luma and chroma, in this paper, we design an IC_TC_P color space based video iTM model, which reproduces high quality HDR video by processing luma and chroma information. Benefitting from the decorrelated perception of luma and chroma in the IC_TC_P color space, two global mapping networks (INet and TPNet) are developed to enhance the luma and chroma pixels, respectively. However, luma and chroma mapping in the iTM task may be affected by color appearance phenomena. Thus, a luma-chroma adaptation transform network (LCATNet) is proposed to process the luma and chroma pixels affected by color appearance phenomena, which can complement the local details to the globally enhanced luma and chroma pixels. In the LCATNet, either the luma mapping or the chroma mapping is adaptively adjusted according to both the luma and the chroma information. Besides, benefitting from the perceptually consistent property of the IC_T C_P color space, the same pixel errors can draw equal model attentions during the training. Thus, the proposed model can correctly render luma and chroma information without highlighting special regions or designing special training losses. Extensive experimental results demonstrate the effectiveness of the proposed model. \endabstract

Supplemental Material

MP4 File

A brief exposition of the paper.

Download
47.64 MB

References

[1]

Gaofeng Cao, Fei Zhou, Kanglin Liu, and Liu Bozhi. 2021. A brightness-adaptive kernel prediction network for inverse tone mapping. Neurocomputing 464 (2021), 1--14.

Digital Library

[2]

Gaofeng Cao, Fei Zhou, Kanglin Liu, Anjie Wang, and Leidong Fan. 2023. A Decoupled Kernel Prediction Network Guided by Soft Mask for Single Image HDR Reconstruction. ACM Transactions on Multimedia Computing, Communications, and Applications 19, 2s (2023), 1--23.

Digital Library

[3]

Gaofeng Cao, Fei Zhou, Han Yan, Anjie Wang, and Leidong Fan. 2022. KPN-MFI: A Kernel Prediction Network with Multi-frame Interaction for Video Inverse Tone Mapping. In Proceedings of the Thirty-First International Joint Conference onArtificial Intelligence, IJCAI-22. International Joint Conferences on Artificial Intelligence Organization, 806--812.

[4]

Guanying Chen, Chaofeng Chen, Shi Guo, Zhetong Liang, Kwan-Yee K Wong, and Lei Zhang. 2021. Hdr video reconstruction: A coarse-to-fine network and a real-world benchmark dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2502--2511.

[5]

Xiangyu Chen, Yihao Liu, Zhengwen Zhang, Yu Qiao, and Chao Dong. 2021. HDRUnet: Single Image HDR Reconstruction with Denoising and Dequantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 354--363.

[6]

Xiangyu Chen, Zhengwen Zhang, Jimmy S. Ren, Lynhoo Tian, Yu Qiao, and Chao Dong. 2021. A New Journey from SDRTV to HDRTV. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10--17, 2021. IEEE, 4480--4489.

[7]

Min Dai and Ai-Mei Huang. 2013. Study on panel sharpening in different color spaces. In Applications of Digital Image Processing XXXVI, Vol. 8856. International Society for Optics and Photonics, SPIE, 88560H.

[8]

Dolby. 2016. ICtCp Dolby White Paper. Dolby Vision.

[9]

Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2014. Learning a Deep Convolutional Network for Image Super-Resolution. In Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6--12, 2014, Proceedings, Part IV (Lecture Notes in Computer Science, Vol. 8692). Springer, 184--199.

[10]

Mark D. Fairchild and Garrett M. Johnson. 2004. The iCAM Framework for Image Appearance, Image Differences, and Image Quality. Journal of Electronic Imaging 13, 1 (2004), 126--138.

[11]

Xavier Glorot and Y. Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. Journal of Machine Learning Research 9 (01 2010), 249--256.

[12]

Cheng Guo, Leidong Fan, Ziyu Xue, and Xiuhua Jiang. 2023. Learning a Practical SDR-to-HDRTV Up-Conversion Using New Dataset and Degradation Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 22231--22241.

[13]

Gang He, Kepeng Xu, Li Xu, Chang Wu, Ming Sun, Xing Wen, and Yu-Wing Tai. 2022. SDRTV-to-HDRTV via Hierarchical Dynamic Context Feature Mapping. In MM '22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022. ACM, 2890--2898.

Digital Library

[14]

Gang He, Kepeng Xu, Li Xu, Chang Wu, Ming Sun, Xing Wen, and Yu-Wing Tai. 2022. SDRTV-to-HDRTV via Hierarchical Dynamic Context Feature Mapping. In MM '22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022. ACM, 2890--2898.

Digital Library

[15]

Jingwen He, Yihao Liu, Yu Qiao, and Chao Dong. 2020. Conditional Sequential Modulation for Efficient Global Image Retouching. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XIII (Lecture Notes in Computer Science, Vol. 12358). 679--695.

[16]

Robert William Gainer Hunt. 2005. The reproduction of colour. Wiley-IS&T Series in Imaging Science and Technology. 342 pages.

[17]

ITU-R. 2015. Colour conversion from Recommendation ITU-R BT.709 to Recommendation ITU-R BT.2020. ITU-R Rec, BT.2087 (2015).

[18]

ITU-R. 2015. Parameter values for the HDTV standards for production and international programme exchange. ITU-R Rec, BT.709--6 (2015).

[19]

ITU-R. 2015. Parameter values for ultra-high definition television systems for production and international programme exchange. ITU-R Rec, BT.2020--2 (2015).

[20]

ITU-R. 2020. High dynamic range television for production and international programme exchangee. ITU-R Rec, BT.2390--8 (2020).

[21]

ITU-R. 2021. Methods for conversion of high dynamic range content to standard dynamic range content and vice-versa. ITU-R Rec, BT.2446--1 (2021).

[22]

ITU-R BT.2124. 2019. Objective Metric for the Assessment of the Potential Visibility of Colour Differences in Television. (2019).

[23]

Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. 2015. Spatial transformer networks. Advances in neural information processing systems 28 (2015).

[24]

Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. 2003. High Dynamic Range Video. ACM Transactions on Graphics 22, 3 (2003), 319--325.

Digital Library

[25]

Soo Ye Kim, Dae-Eun Kim, and Munchurl Kim. 2019. ITM-CNN: Learning the inverse tone mapping from low dynamic range video to high dynamic range displays using convolutional neural networks. In Computer Vision--ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2--6, 2018, Revised Selected Papers, Part III 14. Springer, 395--409.

[26]

Soo Ye Kim, Jihyong Oh, and Munchurl Kim. 2019. Deep SR-ITM: Joint Learning of Super-Resolution and Inverse Tone-Mapping for 4K UHD HDR Applications. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 3116--3125.

[27]

Soo Ye Kim, Jihyong Oh, and Munchurl Kim. 2020. JSI-GAN: GAN-Based Joint Super-Resolution and Inverse Tone-Mapping with Pixel-Wise Task-Specific Filters for UHD HDR Video. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI, New York, NY, USA, February 7--12, 2020. AAAI Press, 11287-- 11295.

[28]

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7--9, 2015, Conference Track Proceedings.

[29]

Min Lin, Qiang Chen, and Shuicheng Yan. 2014. Network In Network. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14--16, 2014, Conference Track Proceedings.

[30]

David L MacAdam. 1942. Visual sensitivities to color difference in daylight. Journal of the Optical Society of America 32, 5 (1942), 247--274.

[31]

David L MacAdam. 1942. Visual sensitivities to color differences in daylight. Josa 32, 5 (1942), 247--274.

[32]

Rafat Mantiuk, Kil Joong Kim, Allan G Rempel, and Wolfgang Heidrich. 2011. HDR-VDP-2: A Calibrated Visual Metric for Visibility and Quality Predictions in all Luminance Conditions. ACM Transactions on Graphics 30, 4 (2011), 1--14.

Digital Library

[33]

Pedram Mohammadi, Mahsa T Pourazad, and Panos Nasiopoulos. 2020. A perception-based inverse tone mapping operator for high dynamic range video applications. IEEE Transactions on Circuits and Systems for Video Technology 31, 5 (2020), 1711--1723.

Digital Library

[34]

Erik Reinhard, GregWard, Sumanta N. Pattanaik, Paul E. Debevec, andWolfgang Heidrich. 2010. High Dynamic Range Imaging - Acquisition, Display, and Image- Based Lighting (2. ed.). Academic Press.

[35]

Marcel Santana Santos, Tsang Ing Ren, and Nima Khademi Kalantari. 2020. Single Image HDR Reconstruction Using a CNN With Masked Features and Perceptual Loss. ACM Transactions on Graphics 39, 4 (2020), 1--10.

Digital Library

[36]

Tong Shao, Deming Zhai, Junjun Jiang, and Xianming Liu. 2022. Hybrid Conditional Deep Inverse Tone Mapping. In MM '22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022. ACM, 1016--1024.

[37]

SMPTE Standard. 2014. High dynamic range electro-optical transfer function of mastering reference displays. SMPTE ST 2084, 2014 (2014), 1--14.

[38]

Wilhelm von Bezold. 1873. Über das Gesetz der Farbenmischung und die physiologischen Grundfarben. Annalen der Physik 226, 10 (1873), 221--247.

[39]

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image Quality Assessment: from Error Visibility to Structural Similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600--612.

Digital Library

[40]

Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. 2018. CBAM: Convolutional Block Attention Module. In Proceedings of the European Conference on Computer Vision (ECCV).

Digital Library

[41]

Gang Xu, Qibin Hou, Le Zhang, and Ming-Ming Cheng. 2022. FMNet: Frequency-Aware Modulation Network for SDR-to-HDR Translation. In Proceedings of the 30th ACM International Conference on Multimedia. 6425--6435.

Digital Library

Cited By

Liu PLi JWang LZha ZXiong ZCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)MLP Embedded Inverse Tone MappingProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680937(1283-1291)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680937
Zhou FZheng ZQiu G(2024)Removing Banding Artifacts in HDR Videos Generated From Inverse Tone MappingIEEE Transactions on Broadcasting10.1109/TBC.2024.339429770:2(753-762)Online publication date: Jun-2024
https://doi.org/10.1109/TBC.2024.3394297

Index Terms

Video Inverse Tone Mapping Network with Luma and Chroma Mapping
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Reconstruction

Recommendations

A framework for inverse tone mapping
Abstract
In recent years many tone mapping operators (TMOs) have been presented in order to display high dynamic range images (HDRI) on typical display devices. TMOs compress the luminance range while trying to maintain contrast. The inverse of tone ...
Lighting condition adaptive tone mapping method
SIGGRAPH '18: ACM SIGGRAPH 2018 Posters

We propose an adaptive tone mapping method for displaying HDR images according to ambient light conditions. To compensate the loss of perceived luminance in brighter viewing conditions, we enhance the HDR image by an algorithm based on the Naka-Rushton ...
Zonal brightness coherency for video tone mapping

Tone Mapping Operators (TMOs) compress High Dynamic Range (HDR) contents to address Low Dynamic Range (LDR) displays. While many solutions have been designed over the last decade, only few of them can cope with video sequences. Indeed, these TMOs tone ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Guangdong Basic and Applied Basic Research Foundation
the National Natural Science Foundation of China
Shenzhen R&D Program

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
161
Total Downloads

Downloads (Last 12 months)151
Downloads (Last 6 weeks)8

Reflects downloads up to 23 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu PLi JWang LZha ZXiong ZCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)MLP Embedded Inverse Tone MappingProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680937(1283-1291)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680937
Zhou FZheng ZQiu G(2024)Removing Banding Artifacts in HDR Videos Generated From Inverse Tone MappingIEEE Transactions on Broadcasting10.1109/TBC.2024.339429770:2(753-762)Online publication date: Jun-2024
https://doi.org/10.1109/TBC.2024.3394297

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents