Nothing Special   »   [go: up one dir, main page]

skip to main content
research-article

High dynamic range and super-resolution from raw image bursts

Published: 22 July 2022 Publication History

Abstract

Photographs captured by smartphones and mid-range cameras have limited spatial resolution and dynamic range, with noisy response in underexposed regions and color artefacts in saturated areas. This paper introduces the first approach (to the best of our knowledge) to the reconstruction of highresolution, high-dynamic range color images from raw photographic bursts captured by a handheld camera with exposure bracketing. This method uses a physically-accurate model of image formation to combine an iterative optimization algorithm for solving the corresponding inverse problem with a learned image representation for robust alignment and a learned natural image prior. The proposed algorithm is fast, with low memory requirements compared to state-of-the-art learning-based approaches to image restoration, and features that are learned end to end from synthetic yet realistic data. Extensive experiments demonstrate its excellent performance with super-resolution factors of up to ×4 on real photographs taken in the wild with hand-held cameras, and high robustness to low-light conditions, noise, camera shake, and moderate object motion.

Supplemental Material

MP4 File
presentation
SRT File
presentation
ZIP File
supplemental material

References

[1]
Cecilia Aguerrebere, Julie Delon, Yann Gousseau, and Pablo Musé. 2014. Best Algorithms for HDR Image Generation. A Study of Performance Bounds. SIAM Journal on Imaging Science 7, 1 (2014), 1--34.
[2]
Paul E. Anuta. 1970. Spatial Registration of Multispectral and Multitemporal Digital Imagery Using Fast Fourier Transform Techniques. IEEE Transactions on Geoscience eletronics 8, 4 (1970), 353--368.
[3]
Tunç Ozan Aydin, Rafal Mantiuk, and Hans-Peter Seidel. 2008. Extending quality metrics to full luminance range images. In Proceedings of Human Vision and Electronic Imaging (SPIE Proceedings), Bernice E. Rogowitz and Thrasyvoulos N. Pappas (Eds.), Vol. 6806. SPIE, 68060B.
[4]
Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. 2018. Automatic differentiation in machine learning: a survey. Journal of Machine Learning Research (JMLR) 18 (2018), 1--43.
[5]
Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. 2021a. Deep Burst Super-Resolution. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 9209--9218.
[6]
Goutam Bhat, Martin Danelljan, Fisher Yu, Luc Van Gool, and Radu Timofte. 2021b. Deep Reparametrization of Multi-Frame Super-Resolution and Denoising. (2021), 2460--2470.
[7]
Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T. Barron. 2019. Unprocessing Images for Learned Raw Denoising. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 11036--11045.
[8]
Che-Han Chang, Chun-Nan Chou, and Edward Y Chang. 2017. CLKN: Cascaded lucaskanade networks for image alignment. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 2213--2221.
[9]
Jongseong Choi, Min Kyu Park, and Moon Gi Kang. 2009. High Dynamic Range Image Reconstruction with Spatial Resolution Enhancement. Computer Journal 52, 1 (2009), 114--125.
[10]
Roger N. Clark. 2006. Digital Camera Reviews and Sensor Performance Summary. "https://clarkvision.com/articles/digital.sensor.performance.summary/".
[11]
Ryan Dahl, Mohammad Norouzi, and Jonathon Shlens. 2017. Pixel Recursive Super Resolution. In Proceedings of the International Conference on Computer Vision (ICCV).
[12]
Paul E. Debevec and Jitendra Malik. 1997. Recovering high dynamic range radiance maps from photographs. In SIGGRAPH. ACM, 369--378.
[13]
Xin Deng, Yutong Zhang, Mai Xu, Shuhang Gu, and Yiping Duan. 2021. Deep Coupled Feedback Network for Joint Exposure Fusion and Image Super-Resolution. IEEE Transactions on Image Processing (TIP) 30 (2021), 3098--3112.
[14]
Akshay Dudhane, Syed Waqas Zamir, Salman Khan, Fahad Khan, and Ming-Hsuan Yang. 2021. Burst Image Restoration and Enhancement. arXiv preprint arXiv:2110.03680 (2021).
[15]
Gabriel Eilertsen, Saghi Hajisharif, Param Hanji, Apostolia Tsirikoglou, Rafal K. Mantiuk, and Jonas Unger. 2021. How to cheat with metrics in single-image HDR reconstruction. In Proceedings of the workshops of the International Conference on Computer Vision (ICCVW). 3981--3990.
[16]
Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafal K. Mantiuk, and Jonas Unger. 2017. HDR image reconstruction from a single exposure using deep CNNs. ACM Transactions on Graphics 36, 6 (2017), 178:1--178:15.
[17]
Yuki Endo, Yoshihiro Kanamori, and Jun Mitani. 2017. Deep reverse tone mapping. ACM Transactions on Graphics (ToG) 36, 6 (2017), 177:1--177:10.
[18]
Manfred Ernst and Bartlomiej Wronski. 2021. HDR+ with Bracketing on Pixel Phones. "https://ai.googleblog.com/2021/04/hdr-with-bracketing-on-pixel-phones.html".
[19]
Sina Farsiu, Michael Elad, and Peyman Milanfar. 2006. Multiframe demosaicing and super-resolution of color images. IEEE Transactions on Image Processing (TIP) 15, 1 (2006), 141--159.
[20]
Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen O. Egiazarian. 2008. Practical Poissonian-Gaussian Noise Modeling and Fitting for Single-Image Raw-Data. IEEE Transactions on Image Processing (TIP) 17, 10 (2008), 1737--1754.
[21]
Orazio Gallo, Marius Tico, Roberto Manduchi, Natasha Gelfand, and Kari Pulli. 2012. Metering for Exposure Stacks. Computer Graphics Forum 31, 2 (2012), 479--488.
[22]
Orazio Gallo, Alejandro J. Troccoli, Jun Hu, Kari Pulli, and Jan Kautz. 2015. Locally non-rigid registration for mobile HDR photography. In (CVPRW). IEEE Computer Society, 48--55.
[23]
Donald Geman and Chengda Yang. 1995. Nonlinear image recovery with half-quadratic regularization. IEEE Transactions on Image Processing (TIP) 5, 7 (1995), 932--946.
[24]
Miguel Granados, Boris Ajdin, Michael Wand, Christian Theobalt, Hans-Peter Seidel, and Hendrik P. A. Lensch. 2010. Optimal HDR reconstruction with linear digital cameras. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, 215--222.
[25]
Bahadir K. Gunturk and Murat Gevrekci. 2006. High-resolution image reconstruction from multiple differently exposed images. IEEE Signal Processing Letters 13, 4 (2006), 197--200.
[26]
Param Hanji, Fangcheng Zhong, and Rafal K. Mantiuk. 2020. Noise-Aware Merging of High Dynamic Range Image Stacks Without Camera Calibration. In Proceedings of the workshops of the European Conference on Computer Vision (ECCVW). 376--391.
[27]
Samuel W. Hasinoff, Frédo Durand, and William T. Freeman. 2010. Noise-optimal capture for high dynamic range photography. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 553--560.
[28]
Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Transactions on Graphics (ToG) 35, 6 (2016), 192:1--192:12.
[29]
Felix Heide, Markus Steinberger, Yun-Ta Tsai, Mushfiqur Rouf, Dawid Pajak, Dikpal Reddy, Orazio Gallo, Jing Liu abd Wolfgang Heidrich, Karen Egiazarian, Jan Kautz, and Kari Pulli. 2014. FlexISP: A flexible camera image processing framework. ACM Transactions on Graphics (ToG) 33, 6 (2014), 231:1--231:13.
[30]
Andrey Ignatov, Luc Van Gool, and Radu Timofte. 2020. Replacing mobile camera isp with a single deep learning model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 536--537.
[31]
Nima Khademi Kalantari and Ravi Ramamoorthi. 2017. Deep high dynamic range imaging of dynamic scenes. ACM Transactions on Graphics (ToG) 36, 4 (2017), 144:1--144:12.
[32]
Soo Ye Kim, Jihyong Oh, and Munchurl Kim. 2019. Deep SR-ITM: Joint Learning of Super-Resolution and Inverse Tone-Mapping for 4K UHD HDR Applications. In Proceedings of the International Conference on Computer Vision (ICCV). 3116--3125.
[33]
Bruno Lecouat, Jean Ponce, and Julien Mairal. 2021. Lucas-Kanade Reloaded: End-to-End Super-Resolution from Raw Image Bursts. In Proceedings of the International Conference on Computer Vision (ICCV).
[34]
Anat Levin, Robert Fergus, Frédo Durand, and William T. Freeman. 2007. Image and depth from a conventional camera with a coded aperture. ACM Transactions on Graphics (ToG) 26, 3 (2007), 70.
[35]
Orly Liba, Kiran Murthy, Yun-Ta Tsai, Tim Brooks, Tianfan Xue, Nikhil Karnad, Qiurui He, Jonathan T. Barron, Dillon Sharlet, Ryan Geiss, Samuel W. Hasinoff, Yael Pritch, and Marc Levoy. 2019. Handheld mobile photography in very low light. ACM Transactions on Graphics (ToG) 38, 6 (2019), 164:1--164:16.
[36]
Yu-Lun Liu, Wei-Sheng Lai, Yu-Sheng Chen, Yi-Lung Kao, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang. 2020. Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 1648--1657.
[37]
Bruce D. Lucas and Takeo Kanade. 1981. An Iterative Image Registration Technique with an Application to Stereo Vision. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). 674--679.
[38]
Guillermo Luijk. 2007. Dcraw tutorial. "http://guillermoluijk.com/tutorial/dcraw/index_en.htm".
[39]
Ziwei Luo, Lei Yu, Xuan Mo, Youwei Li, Lanpeng Jia, Haoqiang Fan, Jian Sun, and Shuaicheng Liu. 2021. EBSR: Feature enhanced burst super-resolution with deformable alignment. In Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops. 471--478.
[40]
Kede Ma, Hui Li, Hongwei Yong, Zhou Wang, Deyu Meng, and Lei Zhang. 2017. Robust Multi-Exposure Image Fusion: A Structural Patch Decomposition Approach. IEEE Transactions on Image Processing (TIP) 26, 5 (2017), 2519--2532.
[41]
Henrique S. Malvar, Li-wei He, and Ross Cutler. 2004. High-quality linear interpolation for demosaicing of Bayer-patterned color images. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP). 485--488.
[42]
Steve Mann and Rosalind W. Picard. 1995. On being 'undigital' with digital cameras: Extending dynamic range by combining differently exposed pictures. In Proceedings of Is&T. 442--448.
[43]
Julien NP Martel, Lorenz K Mueller, Stephen J Carey, Piotr Dudek, and Gordon Wetzstein. 2020. Neural sensors: Learning pixel exposures for HDR imaging and video compressive sensing with programmable sensors. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 7 (2020), 1642--1653.
[44]
Emil Martinec. 2008. Noise, Dynamic Range and Bit Depth in Digital SLRs. "https://photonstophotos.net/EmilMartinec/noise.html".
[45]
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. 2020. PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
[46]
Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul Srinivasan, and Jonathan T Barron. 2021. NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images. arXiv preprint arXiv:2111.13679 (2021).
[47]
Antoine Monod, Julie Delon, and Thomas Veit. 2021. An Analysis and Implementation of the HDR+ Burst Denoising Method. Image Processing On Line 11 (2021), 142--169.
[48]
Manish Narwaria, Rafal K. Mantiuk, Matthieu Perreira Da Silva, and Patrick Le Callet. 2015. HDR-VDP-2.2: a calibrated method for objective quality prediction of high-dynamic range and standard images. Journal on Electronic Imaging 24, 1 (2015), 010501.
[49]
Shree K Nayar and Tomoo Mitsunaga. 2000. High dynamic range imaging: Spatially varying pixel exposures. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1. IEEE, 472--479.
[50]
Yuzhen Niu, Jianbin Wu, Wenxi Liu, Wenzhong Guo, and Rynson W. H. Lau. 2021. HDR-GAN: HDR Image Reconstruction From Multi-Exposed LDR Images With Large Motions. IEEE Transactions on Image Processing (TIP) 30 (2021), 3885--3896.
[51]
Neal Parikh and Stephen P. Boyd. 2014. Proximal Algorithms. Foundations and Trends in Optimization 1, 3 (2014), 127--239.
[52]
Eduardo Pérez-Pellitero, Sibi Catley-Chandar, Ales Leonardis, and Radu Timofte. 2021. NTIRE 2021 Challenge on High Dynamic Range Imaging: Dataset, Methods and Results. In CVPR Workshops. 691--700.
[53]
Tobias Plötz and Stefan Roth. 2017. Benchmarking Denoising Algorithms with Real Photographs. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 2750--2759.
[54]
Ali Ajdari Rad, Laurence Meylan, Patrick Vandewalle, and Sabine Süsstrunk. 2007. Multidimensional image enhancement from a set of unregistered and differently exposed images. In Computational Imaging (SPIE Proceedings), Vol. 6498. SPIE, 649808.
[55]
Erik Reinhard, Michael M. Stark, Peter Shirley, and James A. Ferwerda. 2002. Photographic tone reproduction for digital images. ACM Transactions on Graphics (ToG) 21, 3 (2002), 267--276.
[56]
Javier Sanchez. 2016. The inverse compositional algorithm for parametric registration. Image Processing On Line (2016).
[57]
Marcel Santana Santos, Tsang Ing Ren, and Nima Khademi Kalantari. 2020. Single image HDR reconstruction using a CNN with masked features and perceptual loss. ACM Transactions on Graphics (ToG) 39, 4 (2020), 80.
[58]
Pradeep Sen, Nima Khademi Kalantari, Maziar Yaesoubi, Soheil Darabi, Dan B. Goldman, and Eli Shechtman. 2012. Robust patch-based HDR reconstruction of dynamic scenes. ACM Transactions on Graphics (ToG) 31, 6 (2012), 203:1--203:11.
[59]
Ana Serrano, Felix Heide, Diego Gutierrez, Gordon Wetzstein, and Belen Masia. 2016. Convolutional sparse coding for high dynamic range imaging. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 153--163.
[60]
Hiroyuki Takeda, Sina Farsiu, and Peyman Milanfar. 2007. Kernel Regression for Image Processing and Reconstruction. IEEE Transactions on Image Processing (TIP) 16, 2 (2007), 349--366.
[61]
Yann Traonmilin and Cecilia Aguerrebere. 2014. Simultaneous High Dynamic Range and Superresolution Imaging without Regularization. SIAM Journal on Imaging Science 7, 3 (2014), 1624--1644.
[62]
Okan Tarhan Tursun, Ahmet Oguz Akyüz, Aykut Erdem, and Erkut Erdem. 2016. An Objective Deghosting Quality Metric for HDR Images. Computer Graphics Forum 35, 2 (2016), 139--152.
[63]
Patrick Vandewalle, Sabine Süsstrunk, and Martin Vetterli. 2006. A Frequency Domain Approach to Registration of Aliased Images with Application to Super-resolution. EURASIP Journal on Advances in Signal Processing 2006 (2006).
[64]
Subeesh Vasu, Abhijeet Shenoi, and A. N. Rajagopalan. 2018. Joint HDR and SuperResolution Imaging in Motion Blur. In Proceedings of the International Conference on Image Processing (ICIP). 2885--2889.
[65]
Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. 2013. Plug-and-play priors for model based reconstruction. In Proceedings of the Global Conference on Signal and Information Processing. 945--948.
[66]
Greg Ward. 2003. Fast, Robust Image Registration for Compositing High Dynamic Range Photographs from Hand-Held Exposures. Journal on Graphics, GPU, & Game Tools 8, 2 (2003), 17--30.
[67]
Bartlomiej Wronski, Ignacio Garcia-Dorado, Manfred Ernst, Damien Kelly, Michael Krainin, Chia-Kai Liang, Marc Levoy, and Peyman Milanfar. 2019. Handheld multiframe super-resolution. ACM Transactions on Graphics (ToG) 38, 4 (2019), 28:1--28:18.
[68]
Shangzhe Wu, Jiarui Xu, Yu-Wing Tai, and Chi-Keung Tang. 2018. Deep High Dynamic Range Imaging with Large Foreground Motions. In Proceedings of the European Conference on Computer Vision (ECCV). 120--135.
[69]
Qingsen Yan, Dong Gong, Javen Qinfeng Shi, Anton van den Hengel, Chunhua Shen, Ian Reid, and Yanning Zhang. 2021. Dual-attention-guided network for ghost-free high dynamic range imaging. International Journal of Computer Vision (IJCV) (2021), 1--19.
[70]
Qingsen Yan, Lei Zhang, Yu Liu, Yu Zhu, Jinqiu Sun, Qinfeng Shi, and Yanning Zhang. 2020. Deep HDR Imaging via A Non-Local Network. IEEE Transactions on Image Processing (TIP) 29 (2020), 4308--4322.
[71]
Kai Zhang, Luc Van Gool, and Radu Timofte. 2020. Deep unfolding network for image super-resolution. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 3217--3226.
[72]
Henning Zimmer, Andrés Bruhn, and Joachim Weickert. 2011. Freehand HDR Imaging of Moving Scenes with Simultaneous Resolution Enhancement. Computer Graphics Forum 30, 2 (2011), 405--414.

Cited By

View all
  • (2024)Advances of low-level vision reconstruction in raw domainJournal of Image and Graphics10.11834/jig.23079429:6(1646-1666)Online publication date: 2024
  • (2024)Self-supervised High Dynamic Range Imaging: What Can Be Learned from a Single 8-bit Video?ACM Transactions on Graphics10.1145/364857043:2(1-16)Online publication date: 23-Mar-2024
  • (2024)Training Adaptive Reconstruction Networks for Blind Inverse ProblemsSIAM Journal on Imaging Sciences10.1137/23M154562817:2(1314-1346)Online publication date: 21-Jun-2024
  • Show More Cited By

Index Terms

  1. High dynamic range and super-resolution from raw image bursts

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Graphics
    ACM Transactions on Graphics  Volume 41, Issue 4
    July 2022
    1978 pages
    ISSN:0730-0301
    EISSN:1557-7368
    DOI:10.1145/3528223
    Issue’s Table of Contents
    Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 22 July 2022
    Published in TOG Volume 41, Issue 4

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. computational photography
    2. high-dynamic range imaging
    3. raw bursts
    4. super-resolution

    Qualifiers

    • Research-article

    Funding Sources

    • French government
    • ANR 3IA MIAI@Grenoble Alpes
    • ERC
    • Louis Vuitton/ENS chair

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)87
    • Downloads (Last 6 weeks)10
    Reflects downloads up to 05 Mar 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Advances of low-level vision reconstruction in raw domainJournal of Image and Graphics10.11834/jig.23079429:6(1646-1666)Online publication date: 2024
    • (2024)Self-supervised High Dynamic Range Imaging: What Can Be Learned from a Single 8-bit Video?ACM Transactions on Graphics10.1145/364857043:2(1-16)Online publication date: 23-Mar-2024
    • (2024)Training Adaptive Reconstruction Networks for Blind Inverse ProblemsSIAM Journal on Imaging Sciences10.1137/23M154562817:2(1314-1346)Online publication date: 21-Jun-2024
    • (2024)DeepDuoHDR: A Low Complexity Two Exposure Algorithm for HDR Deghosting on Mobile DevicesIEEE Transactions on Image Processing10.1109/TIP.2024.349783833(6592-6606)Online publication date: 1-Jan-2024
    • (2024)Collaborative Blind Image Deblurring2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00792(7943-7952)Online publication date: 17-Jun-2024
    • (2024)High Quality Reference Feature for Two Stage Bracketing Image Restoration and Enhancement2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00630(6267-6276)Online publication date: 17-Jun-2024
    • (2024)NTIRE 2024 Challenge on Bracketing Image Restoration and Enhancement: Datasets, Methods and Results2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00620(6153-6166)Online publication date: 17-Jun-2024
    • (2024)Bracketing Image Restoration and Enhancement with High-Low Frequency Decomposition2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00616(6097-6107)Online publication date: 17-Jun-2024
    • (2024)Burst Image Super-Resolution with Base Frame Selection2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00601(5940-5949)Online publication date: 17-Jun-2024
    • (2024)Neural Spline Fields for Burst Image Fusion and Layer Separation2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02434(25763-25773)Online publication date: 16-Jun-2024
    • Show More Cited By

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media