Context Region Identification Based Quality Assessment of 3D Synthesized Views
Authors: 
Sadbhawna Thakur, Vinit Jakhetiya, Badri N. Subudhi, Sunil P. Jaiswal, 
Leida Li, Weisi LinAuthors Info & Claims
Abstract
Perceptual quality assessment of 3D synthesized views is an open research problem in computer vision. Researchers across the globe have developed several algorithms to identify distortions. At the same time, the existing algorithms cannot quantify the context in which these distortions affect the overall perceptual quality. According to the recently proposed 3D view synthesis algorithm, the choice of context region for the disocclusion plays a vital role in predicting the quality of 3D views. The context region taken from the background of a view produces a perceptually better quality of 3D synthesized views than when the context region is taken from the foreground. With this view, the proposed algorithm aims to identify the context region and incorporate this information for the perceptual quality assessment of 3D synthesized views. We observed that the depth energy maps of the 3D synthesized views vary significantly with the change in the context region and subsequently can identify the context region. Hence, in this work, we propose a new and efficient quality assessment algorithm based upon the variation in the depth of 3D synthesized and reference views, giving two-fold advantages: 1. It can predict the quality based on whether the context region is foreground or not. 2. It is also able to suggest the possible location of distortions. We have proposed two new algorithms for both situations when the context region is foreground or not. The overall predicted score is the direct multiplication of the quality score estimated when the context region is foreground or not. When applied to the established benchmark dataset, the proposed technique performs satisfactorily with the PLCC of 0.7707 and 0.7572 of SRCC. Also, the proposed algorithm can work as a plug-in to improve the performance of the existing algorithms.
References
[1]
A. Q. de Oliveira, M. Walter, and C. R. Jung, “An artifact-type aware DIBR method for view synthesis,” IEEE Signal Process. Lett., vol. 25, no. 11, pp. 1705–1709, Nov. 2018.
[2]
T. Masayuki, M. Tehrani, T. Fujii, and T. Yendo, “Free-viewpoint TV,” IEEE Signal Process. Mag., vol. 28, no. 1, pp. 67–76, Jan. 2011.
[3]
A. M. Andrew, “Virtual reality: Exploring the brave new technologies of artificial experience and interactive worlds from cyberspace to teledildontics,” Robotica, vol. 10, no. 3, pp. 278–279, 1992.
[4]
S. Mahmoudpour and P. Schelkens, “Synthesized view quality assessment using feature matching and superpixel difference,” IEEE Signal Process. Lett., vol. 27, pp. 1650–1654, 2020.
[5]
C. Fehn, “A 3D-TV approach using depth-image-based rendering (DIBR),” in Proc. 3rd IASTED Conf. Visual., Imag., Image Process., 2003.
[6]
E. Bosc et al., “Towards a new quality metric for 3-D synthesized view assessment,” IEEE J. Sel. Topics Signal Process., vol. 5, no. 7, pp. 1332–1343, Nov. 2011.
[7]
S. Tian, L. Zhang, L. Morin, and O. Deforges, “A benchmark of DIBR synthesized view quality assessment metrics on a new database for immersive media applications,” IEEE Trans. Multimedia, vol. 21, no. 5, pp. 1235–1247, May 2019.
[8]
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004.
[9]
A. K. Moorthy and A. C. Bovik, “A two-step framework for constructing blind image quality indices,” IEEE Signal Process. Lett., vol. 17, no. 5, pp. 513–516, May 2010.
[10]
A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” IEEE Trans. Image Process., vol. 21, no. 12, pp. 4695–4708, Dec. 2012.
[11]
A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a “completely blind” image quality analyzer,” IEEE Signal Process. Lett., vol. 20, no. 3, pp. 209–212, Mar. 2013.
[12]
K. Gu et al., “Model-based referenceless quality metric of 3 D synthesized images using local image description,” IEEE Trans. Image Process., vol. 27, no. 1, pp. 394–405, Jan. 2018.
[13]
S. Tian, L. Zhang, L. Morin, and O. Déforges, “NIQSV : A no-reference synthesized view quality assessment metric,” IEEE Trans. Image Process., vol. 27, no. 4, pp. 1652–1664, Apr. 2018.
[14]
L. Li, Y. Zhou, K. Gu, W. Lin, and S. Wang, “Quality assessment of DIBR-synthesized images by measuring local geometric distortions and global sharpness,” IEEE Trans. Multimedia, vol. 20, no. 4, pp. 914–926, Apr. 2018.
[15]
V. Jakhetiya et al., “A highly efficient blind image quality assessment metric of 3-D synthesized images using outlier detection,” IEEE Trans. Ind. Informat., vol. 15, no. 7, pp. 4120–4128, Jul. 2019.
[16]
G. Yue, C. Hou, K. Gu, T. Zhou, and G. Zhai, “Combining local and global measures for DIBR-synthesized image quality evaluation,” IEEE Trans. Image Process., vol. 28, no. 4, pp. 2075–2088, Apr. 2019.
[17]
V. Jakhetiya, K. Gu, S. P. Jaiswal, T. Singhal, and Z. Xia, “Kernel-ridge regression-based quality measure and enhancement of three-dimensional-synthesized images,” IEEE Trans. Ind. Electron., vol. 68, no. 1, pp. 423–433, Jan. 2021.
[18]
S. Sadbhawna, V. Jakhetiya, D. Mumtaz, and S. P. Jaiswal, “Distortion specific contrast based no-reference quality assessment of DIBR-synthesized views,” in Proc. IEEE 22nd Int. Workshop Multimedia Signal Process., 2020, pp. 1–5.
[19]
G. Wang et al., “Blind quality metric of DIBR-synthesized images in the discrete wavelet transform domain,” IEEE Trans. Image Process., vol. 29, pp. 1802–1814, 2020.
[20]
J. Yan, Y. Fang, R. Du, Y. Zeng, and Y. Zuo, “No reference quality assessment for 3 D synthesized views by local structure variation and global naturalness change,” IEEE Trans. Image Process., vol. 29, pp. 7443–7453, 2020.
[21]
L. Li, Y. Zhou, J. Wu, F. Li, and G. Shi, “Quality index for view synthesis by measuring instance degradation and global appearance,” IEEE Trans. Multimedia, vol. 23, pp. 320–332, 2021.
[22]
L. Li, Y. Huang, J. Wu, K. Gu, and Y. Fang, “Predicting the quality of view synthesis with color-depth image fusion,” IEEE Trans. Circuits Syst. Video Technol., vol. 31, no. 7, pp. 2509–2521, Jul. 2021.
[23]
L. Li, X. Chen, Y. Zhou, J. Wu, and G. Shi, “Depth image quality assessment for view synthesis based on weighted edge similarity,” in Proc. CVPR Workshops, 2019, pp. 17–25.
[24]
F. Shao, Q. Yuan, W. Lin, and G. Jiang, “No-reference view synthesis quality prediction for 3-D videos based on color–depth interactions,” IEEE Trans. Multimedia, vol. 20, no. 3, pp. 659–674, Mar. 2018.
[25]
X. Liu et al., “Subjective and objective video quality assessment of 3 D synthesized views with texture/depth compression distortion,” IEEE Trans. Image Process., vol. 24, no. 12, pp. 4847–4861, Dec. 2015.
[26]
G. Wang, Z. Wang, K. Gu, K. Jiang, and Z. He, “Reference-free DIBR-synthesized video quality metric in spatial and temporal domains,” IEEE Trans. Circuits Syst. for Video Technol., vol. 32, no. 3, pp. 1119–1132, Mar. 2022.
[27]
M. L. Shih, S. Y. Su, J. Kopf, and J. B. Huang, “3 D photography using context-aware layered depth inpainting,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020, pp. 8025–8035.
[28]
Z. Li and N. Snavely, “Megadepth: Learning single-view depth prediction from internet photos,” in Proc. Comput. Vis. Pattern Recognit., 2018, pp. 2041–2050.
[29]
V. Jakhetiya, W. Lin, S. Jaiswal, K. Gu, and S. C. Guntuku, “Just noticeable difference for natural images using RMS contrast and feed-back mechanism,” Neurocomputing, vol. 275, pp. 366–376, 2018.
[30]
N. Ahmed, T. Natarajan, and K. R. Rao, “Discrete cosine transform,” IEEE Trans. Comput., vol. C- 23, no. 1, pp. 90–93, Jan. 1974.
[31]
M. Saad, A. Bovik, and C. Charrier, “DCT statistics model-based blind image quality assessment,” in Proc. Int. Conf. Image Process., 2011, pp. 3093–3096.
[32]
T. Brandão and M. Queluz, “No-reference image quality assessment based on DCT domain statistics,” Signal Process., vol. 88, no. 4, pp. 822–833, 2008.
[33]
C. Ji et al., “No-reference quality assessment for 3 D synthesized images based on visual-entropy-guided multi-layer features analysis,” Entropy (Basel), 2021.
[34]
Sadbhawna et al., “Perceptually unimportant information reduction and cosine similarity-based quality assessment of 3D-synthesized images,” IEEE Trans. Image Process., vol. 31, pp. 2027–2039, 2022.
[35]
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proc. Conf. Comput. Vis. Pattern Recognit., 2018, pp. 586–595.
[36]
S. Tian, L. Zhang, L. Morin, and O. Déforges, “SC-IQA: Shift compensation based image quality assessment for DIBR-synthesized views,” in Proc. IEEE Visual Commun. Image Process., 2018, pp. 1–4.
[37]
D. Sandić-Stanković, D. Kukolj, and P. Le Callet, “Multi–scale synthesized view assessment based on morphological pyramids,” J. Elect. Eng., vol. 67, pp. 1–9, 2016.
[38]
D. Sandić-Stanković, D. Kukolj, and P. Le Callet, “Dibr synthesized image quality assessment based on morphological wavelets,” in Proc. Seventh Int. Workshop Qual. Multimedia Experience, 2015, pp. 1–6.
[39]
M. Cheon, S. Yoon, B. Kang, and J. Lee, “Perceptual image quality assessment with transformers,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops, 2021, pp. 433–442.
[40]
S. Lao et al., “Attentions help cnns see better: Attention-based hybrid image quality assessment network,” pp. 1139–1148, 2022, arXiv:2204.10485.
[41]
Sadbhawna, V. Jakhetiya, D. Mumtaz, B. N. Subudhi, and S. C. Guntuku, “Stretching artifacts identification for quality assessment of 3D-synthesized views,” IEEE Trans. Image Process., vol. 31, pp. 1737–1750, 2022.
[42]
S. Ling et al., “Re-visiting discriminator for blind free-viewpoint image quality assessment,” IEEE Trans. Multimedia, vol. 23, pp. 4245–4258, 2021.
[43]
Y. J. Jung, H. Kim, and Y. Ro, “Critical binocular asymmetry measure for perceptual quality assessment of synthesized stereo 3 D images in view synthesis,” IEEE Trans. Circuits Syst. for Video Technol., vol. 26, no. 7, pp. 1201–1214, Jul. 2016.
[44]
A. Criminisi, P. Perez, and K. Toyama, “Region filling and object removal by exemplar-based image inpainting,” IEEE Trans. Image Process., vol. 13, no. 9, pp. 1200–1212, Sep. 2004.
[45]
D. Wang, Y. Zhao, Z. Wang, and H. Chen, “Hole-filling for DIBR based on depth and gradient information,” Int. J. Adv. Robotic Syst., vol. 12, no. 2, p. 12, 2015.
[46]
I. Ahn and C. Kim, “A novel depth-based virtual view synthesis method for free viewpoint video,” IEEE Trans. Broadcast., vol. 59, no. 4, pp. 614–626, Dec. 2013.
[47]
G. Luo, Y. Zhu, Z. Li, and L. Zhang, “A hole filling approach based on background reconstruction for view synthesis in 3 D video,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 1781–1789.
[48]
M. Solh and G. AlRegib, “Hierarchical hole-filling for depth-based view synthesis in FTV and 3 D video,” IEEE J. Sel. Topics Signal Process., vol. 6, no. 5, pp. 495–504, Sep. 2012.
[49]
M. Tanimoto, T. Fujii, K. Suzuki, N. Fukushima, and Y. Mori, “Reference softwares for depth estimation and view synthesis,” ISO/IEC JTC1/SC29/WG11 MPEG 20081, Doc. M15377, 2013.
[50]
C. Zhu and S. Li, “Depth image based view synthesis: New insights and perspectives on hole generation and filling,” IEEE Trans. Broadcast., vol. 62, no. 1, pp. 82–93, Mar. 2016.
[51]
S. S. Yoon, H. Sohn, Y. J. Jung, and Y. M. Ro, “Inter-view consistent hole filling in view extrapolation for multi-view image generation,” in Proc. IEEE Int. Conf. Image Process. (ICIP), 2014, pp. 2883–2887.
[52]
S. Xie and Z. Tu, “Holistically-nested edge detection,” in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 1395–1403.
[53]
R. Zhu, F. Zhou, W. Yang, and J. Xue, “On hypothesis testing for comparing image quality assessment metrics [tips tricks],” IEEE Signal Process. Mag., vol. 35, no. 4, pp. 133–136, Jul. 2018.
Index Terms
- Context Region Identification Based Quality Assessment of 3D Synthesized Views
Index terms have been assigned to the content through auto-classification.
Recommendations
Objective image quality assessment of 3D synthesized views
Depth-Image-Based-Rendering (DIBR) techniques are essential for three-dimensional (3D) video applications such as 3D Television (3DTV) and Free-Viewpoint Video. However, this process is based on 3D warping and can induce serious distortions whose impact ...
Subjective and Objective Video Quality Assessment of 3D Synthesized Views With Texture/Depth Compression Distortion
The quality assessment for synthesized video with texture/depth compression distortion is important for the design, optimization, and evaluation of the multi-view video plus depth (MVD)-based 3D video system. In this paper, the subjective and objective ...
Data quality assessment in context: A cognitive perspective
In organizations today, the risk of poor information quality is becoming increasingly high as larger and more complex information resources are being collected and managed. To mitigate this risk, decision makers assess the quality of the information ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
1520-9210 © 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Press
Publication History
Published: 01 January 2023
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 05 Mar 2025