Abstract
Pixel binning-based Quad sensors (mega-pixel resolution camera sensor) offer a promising solution to address the hardware limitations of compact cameras for low-light imaging. However, the binning process leads to reduced spatial resolution and introduces non-Bayer CFA artifacts. In this paper, we propose a Quad CFA-driven remosaicing model that effectively converts noisy Quad Bayer and standard Bayer patterns compatible to existing Image Signal Processor (ISP) without any loss in resolution. To enhance the practicality of the remosaicing model for real-world images affected by mixed noise, we introduce a novel dual-head joint remosaicing and denoising network (DJRD), which addresses the order of denoising and remosaicing by performing them in parallel. In DJRD, we customize two denoising branches for Quad Bayer and Bayer inputs. These branches model non-local and local dependencies, CFA location, and frequency information using residual convolutional layers, Swin Transformer, and wavelet transform-based CNN. Furthermore, to improve the model’s performance on challenging cases, we fine-tune DJRD to handle difficult scenarios by identifying problematic patches through Moire and zipper detection metrics. This post-training phase allows the model to focus on resolving complex image regions. Extensive experiments conducted on simulated and real images in both Bayer and sRGB domains demonstrate that DJRD outperforms competing models by approximately 3 dB, while maintaining the simplicity of implementation without adding any hardware.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Ahmed, T. U., Amirshahi, S. A., & Pedersen, M. (2023). Image demosaicing: Subjective analysis and evaluation of image quality metrics. Image, 30, 25.
Alleysson, D., Susstrunk, S., & Hérault, J. (2005). Linear demosaicing inspired by the human visual system. IEEE Transactions on Image Processing, 14(4), 439–449.
A Sharif, S., Naqvi, R. A., & Biswas, M. (2021). Beyond joint demosaicking and denoising: An image processing pipeline for a pixel-bin image sensor. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition (pp. 233–242).
Daubechies, I. (1990). The wavelet transform, time-frequency localization and signal analysis. IEEE Transactions on Information Theory, 36(5), 961–1005.
Daubechies, I. (1992). Ten lectures on wavelets. New Delhi: SIAM.
Dewil, V., Courtois, A., Rodríguez, M., Ehret, T., Brandonisio, N., Bujoreanu, D. & Arias, P. (2023). Video joint denoising and demosaicing with recurrent cnns. In Proceedings of the ieee/cvf winter conference on applications of computer vision (pp. 5108–5119).
Dong, C., Loy, C. C., He, K., & Tang, X. (2016). Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2), 295–307.
Dong, X., Xu, W., Miao, Z., Ma, L., Zhang, C., Yang, J. & Shen, J. (2022). Abandoning the bayer-filter to see in the dark. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition (pp. 17431–17440).
Dubois, E. (2005). Frequency-domain methods for demosaicking of bayer-sampled color images. IEEE Signal Processing Letters, 12(12), 847–850.
Ehret, T., Davy, A., Arias, P., & Facciolo, G. (2019). Joint demosaicking and denoising by fine-tuning of bursts of raw images. In Proceedings of the ieee/cvf international conference on computer vision (pp. 8868–8877).
Fan, Z., Wu, X., Meng, F., Wu, Y., & Zhang, F. (2023). Otst: A two-phase framework for joint denoising and remosaicing in rgbw cfa. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition (pp. 2832–2841).
Feng, K., Zhao, Y., Chan, J.C.-W., Kong, S. G., Zhang, X., & Wang, B. (2021). Mosaic convolution-attention network for demosaicing multispectral filter array images. IEEE Transactions on Computational Imaging, 7, 864–878.
Gharbi, M., Chaurasia, G., Paris, S., & Durand, F. (2016). Deep joint demosaicking and denoising. ACM Transactions on Graphics (ToG), 35(6), 1–12.
Guo, C., Li, C., Guo, J., Loy, C. C., Hou, J., Kwong, S., & Cong, R. (2020). Zero-reference deep curve estimation for low-light image enhancement. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition (cvpr).
Guo, S., Liang, Z., & Zhang, L. (2021). Joint denoising and demosaicking with green channel prior for real-world burst images. IEEE Transactions on Image Processing, 30, 6930–6942.
Huang, J.-B., Singh, A., & Ahuja, N. (2015). Single image super-resolution from transformed self-exemplars. In Proceedings of the ieee conference on computer vision and pattern recognition (pp. 5197–5206).
Jang, Y., Kim, H., Kim, K., Kim, S., Lee, S., & Yim, J. (2021). A new pdaf correction method of cmos image sensor with nonacell and super pd to improve image quality in binning mode. Electronic Imaging, 2021(9), 220–221.
Jia, J., Sun, H., Liu, X., Xiao, L., Xu, Q., & Zhai, G. (2022). Learning rich information for quad bayer remosaicing and denoising. In European conference on computer vision (pp. 175–191).
Kim, I., Lim, D., Seo, Y., Lee, J., Choi, Y., & Song, S. (2021). On recent results in demosaicing of samsung 108mp cmos sensor using deep learning. In 2021 ieee region 10 symposium (tensymp) (pp. 1–4).
Kim, I., Song, S., Chang, S., Lim, S., & Guo, K. (2020). Deep image demosaicing for submicron image sensors. Electronic Imaging, 2020(7), 60410–60411.
Kim, Y., & Kim, Y. (2019). High-sensitivity pixels with a quad-wrgb color filter and spatial deep-trench isolation. Sensors, 19(21), 4653.
Kim, Y., Lee, J., Kim, S., Bang, J., Hong, D., Kim, T., & Yim, J. (2021). Camera image quality tradeoff processing of image sensor re-mosaic using deep neural network. Electronic Imaging, 2021(9), 206–1.
Levin, A., Nadler, B., Durand, F., & Freeman, W. T. (2012). Patch complexity, finite pixel correlations and optimal denoising. In European conference on computer vision (pp. 73–86).
Li, Y., Huang, J.-B., Ahuja, N., & Yang, M.-H. (2016). Deep joint image filtering. In European conference on computer vision (pp. 154–169).
Liu, J., Wu, C.-H., Wang, Y., Xu, Q., Zhou, Y., Huang, H. & others (2019). Learning raw image denoising with bayer pattern unification and bayer preserving augmentation. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition workshops.
Liu, L., Jia, X., Liu, J., & Tian, Q. (2020). Joint demosaicing and denoising with self guidance. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition (pp. 2240–2249).
Liu, P., Zhang, H., Zhang, K., Lin, L., & Zuo, W. (2018a). Multi-level wavelet-cnn for image restoration. In Proceedings of the ieee conference on computer vision and pattern recognition workshops (pp. 773–782).
Liu, P., Zhang, H., Zhang, K., Lin, L., & Zuo, W. (2018b). Multi-level wavelet-cnn for image restoration. In Proceedings of the ieee conference on computer vision and pattern recognition workshops (pp. 773–782).
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z. & Guo, B. (2021). Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the ieee/cvf international conference on computer vision (pp. 10012–10022).
Ma, K., Gharbi, M., Adams, A., Kamil, S., Li, T.-M., Barnes, C., & Ragan-Kelley, J. (2022). Searching for fast demosaicking algorithms. ACM Transactions on Graphics (TOG), 41(5), 1–18.
Malvar, H. S., He, L.-w., & Cutler, R. (2004). High-quality linear interpolation for demosaicing of bayer-patterned color images. In 2004 ieee international conference on acoustics, speech, and signal processing (vol. 3, pp. iii–485).
Mantiuk, R., Kim, K. J., Rempel, A. G., & Heidrich, W. (2011). Hdr-vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Transactions on Graphics (TOG), 30(4), 1–14.
Mei, K., Li, J., Zhang, J., Wu, H., Li, J., & Huang, R. (2019). Higher-resolution network for image demosaicing and enhancing. In 2019 ieee/cvf international conference on computer vision workshop (iccvw) (pp. 3441–3448).
Mukherjee, J., Parthasarathi, R., & Goyal, S. (2001). Markov random field processing for color demosaicing. Pattern Recognition Letters, 22(3–4), 339–351.
Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 234–241).
Sharif, S., Naqvi, R. A., & Biswas, M. (2021). Sagan: Adversarial spatial-asymmetric attention for noisy nona-bayer reconstruction. arXiv preprint arXiv:2110.08619.
Shekhar Tripathi, A., Danelljan, M., Shukla, S., Timofte, R., & Van Gool, L. (2022). Transform your smartphone into a dslr camera: Learning the isp in the wild. In European Conference on Computer Vision European Conference on Computer Vision (pp. 625–641).
Stojkovic, A., Shopovska, I., Luong, H., Aelterman, J., Jovanov, L., & Philips, W. (2019). The effect of the color filter array layout choice on state-of-the-art demosaicing. Sensors, 19(14), 3215.
Tan, D. S., Chen, W.-Y., & Hua, K.-L. (2018). Deepdemosaicking: Adaptive image demosaicking via multiple deep fully convolutional networks. IEEE Transactions on Image Processing, 27(5), 2408–2419.
Tan, R., Zhang, K., Zuo, W., & Zhang, L. (2017). Color image demosaicking via deep residual learning. In Proc. ieee int. conf. multimedia expo (icme) (pp. 793–798).
Tsai, C.-Y., & Song, K.-T. (2007). A new edge-adaptive demosaicing algorithm for color filter arrays. Image and Vision Computing, 25(9), 1495–1508.
Wu, X., Fan, Z., Zheng, J., Wu, Y., & Zhang, F. (2022). Learning to joint remosaic and denoise in quad bayer cfa via universal multi-scale channel attention network. In European Conference on Computer Vision (pp. 147–160).
Xing, W., & Egiazarian, K. (2021). End-to-end learning for joint image demosaicing, denoising and super-resolution. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition (pp. 3507–3516).
Yang, Q., Yang, G., Jiang, J., Li, C., Feng, R., Zhou, S. & Gu, J. (2022). Mipi 2022 challenge on quad-bayer re-mosaic: Dataset and report. arXiv preprint arXiv:2209.07060.
Yoo, Y., Im, J., & Paik, J. (2015). Low-light image enhancement using adaptive digital pixel binning. Sensors, 15(7), 14917–14931.
Zhang, C., Li, Y., Wang, J., & Hao, P. (2016). Universal demosaicking of color filter arrays. IEEE Transactions on Image Processing, 25(11), 5173–5186.
Zhang, K., Li, Y., Liang, J., Cao, J., Zhang, Y., Tang, H. & Van Gool, L. (2022). Practical blind denoising via swin-conv-unet and data synthesis. arXiv preprint arXiv:2203.13278.
Zhang, L., & Wu, X. (2005). Color demosaicking via directional linear minimum mean square-error estimation. IEEE Transactions on Image Processing, 14(12), 2167–2178.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 586–595).
Zhang, T., Fu, Y., & Li, C. (2022). Deep spatial adaptive network for real image demosaicing. In Proceedings of the AAAI Conference on Artificial Intelligence (vol. 36, pp. 3326–3334).
Author information
Authors and Affiliations
Corresponding authors
Additional information
Communicated by Yasuyuki Matsushita.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix A Additional Results
Appendix A Additional Results
In addition to the main results presented in the manuscript, Tables 10, 11, 12 and 13 provide a detailed quantitative assessment using PSNR, SSIM, and LPIPS metrics in the sRGB domain. These evaluations are conducted on the MIPI dataset, MIT Moiré, and Urban100. Moreover, Figures 17, 18, 19 and 20 offer supplementary visual comparisons between our proposed DJRD method and state-of-the-art techniques.
Visual comparison on MIT Moiré and Urban100
Visual comparison on MIT Moiré and Urban100
Visual comparison on MIT Moiré and Urban100
Visual comparison on MIT Moiré and Urban100
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zeng, H., Feng, K., Cao, J. et al. Inheriting Bayer’s Legacy: Joint Remosaicing and Denoising for Quad Bayer Image Sensor. Int J Comput Vis 132, 4992–5013 (2024). https://doi.org/10.1007/s11263-024-02114-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-024-02114-7