Abstract
Imaging in low light is difficult because the number of photons arriving at the sensor is low. Imaging dynamic scenes in low-light environments is even more difficult because as the scene moves, pixels in adjacent frames need to be aligned before they can be denoised. Conventional CMOS image sensors (CIS) are at a particular disadvantage in dynamic low-light settings because the exposure cannot be too short lest the read noise overwhelms the signal. We propose a solution using Quanta Image Sensors (QIS) and present a new image reconstruction algorithm. QIS are single-photon image sensors with photon counting capabilities. Studies over the past decade have confirmed the effectiveness of QIS for low-light imaging but reconstruction algorithms for dynamic scenes in low light remain an open problem. We fill the gap by proposing a student-teacher training protocol that transfers knowledge from a motion teacher and a denoising teacher to a student network. We show that dynamic scenes can be reconstructed from a burst of frames at a photon level of 1 photon per pixel per frame. Experimental results confirm the advantages of the proposed method compared to existing methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Abadi, M., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). https://www.tensorflow.org/
Aittala, M., Durand, F.: Burst image deblurring using permutation invariant convolutional neural networks. In: ECCV (2018)
Buades, A., Coll, B., Morel, J.M.: Denoising image sequences does not require motion estimation. In: IEEE Conference Advanced Video and Signal Based Surveillance, pp. 70–74 (2005)
Buades, A., Coll, B., Morel, J.M.: A review of image denoising algorithms, with a new one. SIAM Multiscale Modeling Simul. 4(2), 490–530 (2005)
Burri, S., Maruyama, Y., Michalet, X., Regazzoni, F., Bruschini, C., Charbon, E.: Architecture and applications of a high resolution gated SPAD image sensor. Optics Express 22(14), 17573–17589 (2014)
Callenberg, C., Lyons, A., den Brok, D., Henderson, R., Hullin, M.B., Faccio, D.: EMCCD-SPAD camera data fusion for high spatial resolution time-of-flight imaging. In: Computational Optical Sensing and Imaging. Optical Society of America (2019)
Chan, S.H., Elgendy, O.A., Wang, X.: Images from bits: non-iterative imagereconstruction for Quanta Image Sensors. Sensors 16(11) (2016)
Chan, S.H., Lu, Y.M.: Efficient image reconstruction for gigapixel Quantum Image Sensors. In: IEEE Global Conference Signal and Information Processing (2014)
Chan, S.H., Wang, X., Elgendy, O.A.: Plug-and-play ADMM for image restoration: fixed-point convergence and applications. IEEE Trans. Comput. Imaging 3(1), 84–98 (2017)
Chandramouli, P., Burri, S., Bruschini, C., Charbon, E., Kolb, A.: A bit too much?. In: ICCP, High Speed Imaging from Sparse Photon Counts (2019)
Chen, C., Chen, Q., Do, M.N., Koltun, V.: Seeing motion in the dark. In: ICCV (2019)
Chen, C., Chen, Q., Xu, J., Koltun, V.: Learning to see in the dark. In: CVPR (2018)
Choi, J.H., Elgendy, O.A., Chan, S.H.: Image reconstruction for Quanta Image Sensors using deep neural networks. In: ICASSP (2018)
Chollet, F., et al.: Keras (2015). https://www.keras.io
Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)
Davy, A., et al.: A non-local CNN for video denoising. In: ICIP (2019)
Dutton, N.A., et al.: A SPAD-based QVGA image sensor for single-photon counting and quanta imaging. IEEE Trans. Electron Devices 63(1), 189–196 (2015)
Dutton, N.A., Parmesan, L., Holmes, A.J., Grant, L.A., Henderson, R.K.: 320\( \times \)240 oversampled digital single photon counting image sensor. In: Symposium on VLSI Circuits Digest of Technical Papers (2014)
Elgendy, O.A., Chan, S.H.: Optimal threshold design for Quanta Image Sensor. IEEE Trans. Comput. Imaging 4(1), 99–111 (2017)
Elgendy, O.A., Chan, S.H.: Color Filter Arrays for Quanta Image Sensors. arXiv preprint arXiv:1903.09823 (2019)
Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)
Fossum, E.R.: Gigapixel digital film Sensor (DFS) proposal. In: Nanospace Manipulation of Photons and Electrons for Nanovision Systems (2005)
Fossum, E.R.: Some thoughts on future digital still cameras. In: Image Sensors and Signal Processing for Digital Still Cameras (2006)
Fossum, E.R.: Modeling the performance of single-bit and multi-bit quanta image sensors. IEEE J. Electron Devices Soc. 1(9), 166–174 (2013)
Fu, Q., Jung, C., Xu, K.: Retinex-based perceptual contrast enhancement in images using luminance adaptation. IEEE Access 6, 61277–61286 (2018)
Gariepy, G., et al.: Single-photon sensitive light-in-fight imaging. Nat. Commun. 6(1), 1–7 (2015)
Gnanasambandam, A., Elgendy, O., Ma, J., Chan, S.H.: Megapixel photon-counting color imaging using Quanta Image Sensor. Optics Express 27(12), 17298–17310 (2019)
Gnanasambandam, A., Ma, J., Chan, S.H.: High dynamic range imaging using Quanta Image Sensors. In: International Image Sensors Workshop (2019)
Godard, C., Matzen, K., Uyttendaele, M.: Deep burst denoising. In: ECCV (2018)
Gould, S., Fulton, R., Koller, D.: Decomposing a scene into geometric and semantically consistent regions. In: ICCV (2009)
Guo, X., Li, Y., Ling, H.: LIME: low-light image enhancement via illumination map estimation. IEEE Trans. Image Process. 26(2), 982–993 (2016)
Gupta, A., Ingle, A., Gupta, M.: Asynchronous single-photon 3D imaging. In: ICCV (2019)
Gyongy, I., Dutton, N., Henderson, R.: Single-photon tracking for high-speed vision. Sensors 18(2), 323 (2018)
Hasinoff, S.W., et al.: Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Trans. Graph. 35(6) (2016)
Horn, B.K., Schunck, B.G.: Determining optical flow. In: Techniques and Applications of Image Understanding, vol. 281. International Society Optics and Photonics (1981)
Hu, Z., Cho, S., Wang, J., Yang, M.H.: Deblurring low-light images with light streaks. In: CVPR (2014)
Ji, H., Liu, C., Shen, Z., Xu, Y.: Robust video denoising using low rank matrix completion. In: CVPR (2010)
Joshi, N., Cohen, M.: Seeing Mt. Rainier: Lucky imaging for multi-image denoising, sharpening, and haze removal. In: ICCP (2010)
Kokkinos, F., Lefkimmiatis, S.: Iterative residual CNNs for burst photography applications. In: CVPR (2019)
Lindell, D.B., O’Toole, M., Wetzstein, G.: Single-photon 3D imaging with deepsensor fusion. ACM Trans. Graph. 37(4) (2018)
Liu, C., Freeman, W.: A high-quality video denoising algorithm based on reliable motion estimation. In: ECCV (2010)
Liu, Z., Yuan, L., Tang, X., Uyttendaele, M., Sun, J.: Fast burst images denoising. ACM Trans. Graph., 33(6) (2014)
Lore, K.G., Akintayo, A., Sarkar, S.: LLNet: a deep autoencoder approach to natural low-light image enhancement. Pattern Recogn. 61, 650–662 (2017)
Ma, J., Fossum, E.: A pump-gate jot device with high conversion gain for a Quanta Image Sensor. IEEE J. Electron Devices Soc. 3(2), 73–77 (2015)
Ma, J., Masoodian, S., Starkey, D., Fossum, E.R.: Photon-number-resolving megapixel image sensor at room temperature without avalanche gain. Optica 4(12), 1474–1481 (2017)
Ma, S., Gupta, S., Ulku, A.C., Brushini, C., Charbon, E., Gupta, M.: Quanta burst photography. ACM Trans. Graph. (TOG), 39(4) (2020)
Maggioni, M., Katkovnik, V., Egiazarian, K., Foi, A.: Nonlocal transform-domain filter for volumetric data denoising and reconstruction. IEEE Trans. Image Process. 22(1), 119–133 (2012)
Makitalo, M., Foi, A.: Optimal inversion of the Anscombe transformation in low-count Poisson image denoising. IEEE Trans. Image Process. 20(1), 99–109 (2010)
Malm, H., Oskarsson, M., Warrant, E., et al.: Adaptive enhancement and noise reduction in very low light-level video. In: ICCV (2007)
Mao, X.J., Shen, C., Yang, Y.B.: Image restoration using convolutional auto-encoders with symmetric skip connections. arXiv preprint arXiv:1606.08921 (2016)
Mildenhall, B., et al.: Burst denoising with kernel prediction networks. In: CVPR (2018)
O’Toole, M., Heide, F., Lindell, D.B., Zang, K., Diamond, S., Wetzstein, G.: Reconstructing transient images from single-photon sensors. In: CVPR (2017)
Plotz, T., Roth, S.: Benchmarking denoising algorithms with real photographs. In: CVPR (2017)
Protter, M., Elad, M.: Image sequence denoising via sparse and redundant representations. IEEE Trans. Image Process. 18(1), 27–35 (2008)
Remez, T., Litany, O., Bronstein, A.: A picture is worth a billion bits: real-time image reconstruction from dense binary threshold pixels. In: ICCP (2016)
Remez, T., Litany, O., Giryes, R., Bronstein, A.: Deep convolutional denoising of low-light images. arXiv preprint arXiv:1701.01687 (2017)
Sutour, C., Deledalle, C.A., Aujol, J.F.: Adaptive regularization of the NL-means: application to image and video denoising. IEEE Trans. Image Process. 23(8), 3506–3521 (2014)
Werlberger, M., Pock, T., Unger, M., Bischof, H.: Optical flow guided TV-L 1 video interpolation and restoration. In: International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition (2011)
Xia, Z., Perazzi, F., Gharbi, M., Sunkavalli, K., Chakrabarti, A.: Basis prediction networks for effective burst denoising with large kernels. arXiv preprint arXiv:1912.04421 (2019)
Xu, J., Li, H., Liang, Z., et al.: Real-world noisy image denoising: a new benchmark. arXiv preprint arXiv:1804.02603 (2018)
Yang, F., Lu, Y.M., Sbaiz, L., Vetterli, M.: An optimal algorithm for reconstructing images from binary measurements. In: Proceedings SPIE, vol. 7533 (2010)
Yang, F., Lu, Y.M., Sbaiz, L., Vetterli, M.: Bits from photons: oversampled image acquisition using binary poisson statistics. IEEE Trans. Image Process. 21(4), 1421–1436 (2011)
Yang, F., Sbaiz, L., Charbon, E., Süsstrunk, S., Vetterli, M.: Image reconstruction in the gigavision camera. In: ICCV Workshops (2009)
Zhang, K., Zuo, W., Chen, Y., et al.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7) (2017)
Zhang, K., Zuo, W., Zhang, L.: FFDNet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process. 27(9) (2018)
Acknowledgement
This work is supported in part by the US National Science Foundation under grant CCF-1718007.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Chi, Y., Gnanasambandam, A., Koltun, V., Chan, S.H. (2020). Dynamic Low-Light Imaging with Quanta Image Sensors. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12366. Springer, Cham. https://doi.org/10.1007/978-3-030-58589-1_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-58589-1_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58588-4
Online ISBN: 978-3-030-58589-1
eBook Packages: Computer ScienceComputer Science (R0)