research-article

Subjective and Objective Video Quality Assessment of 3D Synthesized Views With Texture/Depth Compression Distortion

Authors:

Qiang PengAuthors Info & Claims

IEEE Transactions on Image Processing, Volume 24, Issue 12

Pages 4847 - 4861

https://doi.org/10.1109/TIP.2015.2469140

Published: 01 December 2015 Publication History

Abstract

The quality assessment for synthesized video with texture/depth compression distortion is important for the design, optimization, and evaluation of the multi-view video plus depth (MVD)-based 3D video system. In this paper, the subjective and objective studies for synthesized view assessment are both conducted. First, a synthesized video quality database with texture/depth compression distortion is presented with subjective scores given by 56 subjects. The 140 videos are synthesized from ten MVD sequences with different texture/depth quantization combinations. Second, a full reference objective video quality assessment (VQA) method is proposed concerning about the annoying temporal flicker distortion and the change of spatio-temporal activity in the synthesized video. The proposed VQA algorithm has a good performance evaluated on the entire synthesized video quality database, and is particularly prominent on the subsets which have significant temporal flicker distortion induced by depth compression and view synthesis process.

References

[1]

K. Muller, P. Merkle, and T. Wiegand, “3D video representation using depth maps,” Proc. IEEE, vol. 99, no. 4, pp. 643–656, Apr. 2011.

[2]

M. M. Hannuksela, Y. Chen, T. Suzuki, J. Ohm, and G. Sullivan, 3D-AVC Draft Text 7, Joint Collaborative Team 3D Video Coding Extensions (JCT-3V), document JCT3V-E1002V2, Jan. 2013.

[3]

G. Tech, K. Wegner, Y. Chen, and S. Yea, 3D-HEVC Draft Text 5, Joint Collaborative Team 3D Video Coding Extensions (JCT-3V), document JCT3V-I1001V3, Jul. 2014.

[4]

C. Fehn, R. Barre, and S. Pastoor, “Interactive 3-DTV-concepts and key technologies,” Proc. IEEE, vol. 94, no. 3, pp. 524–538, Mar. 2006.

[5]

Subjective Methods Assessment Stereoscopic 3DTV Systems, document ITU-R Rec. BT.2021, 2012.

[6]

L.-H. Wang, X.-J. Huang, M. Xi, D.-X. Li, and M. Zhang, “An asymmetric edge adaptive filter for depth generation and hole filling in 3DTV,” IEEE Trans. Broadcast., vol. 56, no. 3, pp. 425–431, Sep. 2010.

[7]

Y. Zhao, C. Zhu, Z. Chen, D. Tian, and L. Yu, “Boundary artifact reduction in view synthesis of 3D video: From perspective of texture-depth alignment,” IEEE Trans. Broadcast., vol. 57, no. 2, pp. 510–522, Jun. 2011.

[8]

Y. Zhao, C. Zhu, Z. Chen, and L. Yu, “Depth no-synthesis-error model for view synthesis in 3-D video,” IEEE Trans. Image Process., vol. 20, no. 8, pp. 2221–2228, Aug. 2011.

Digital Library

[9]

Y. Zhang, S. Kwong, L. Xu, S. Hu, G. Jiang, and C.-C. J. Kuo, “Regional bit allocation and rate distortion optimization for multiview depth video coding with view synthesis distortion model,” IEEE Trans. Image Process., vol. 22, no. 9, pp. 3497–3512, Sep. 2013.

Digital Library

[10]

Y. Zhang, S. Kwong, S. Hu, and C.-C. J. Kuo, “Efficient multiview depth coding optimization based on allowable depth distortion in view synthesis,” IEEE Trans. Image Process., vol. 23, no. 11, pp. 4879–4892, Nov. 2014.

[11]

S. Hu, S. Kwong, Y. Zhang, and C.-C. J. Kuo, “Rate-distortion optimized rate control for depth map-based 3D video coding,” IEEE Trans. Image Process., vol. 22, no. 2, pp. 585–594, Feb. 2013.

Digital Library

[12]

H. Yuan, S. Kwong, J. Liu, and J. Sun, “A novel distortion model and lagrangian multiplier for depth maps coding,” IEEE Trans. Circuits Syst. Video Technol., vol. 24, no. 3, pp. 443–451, Mar. 2014.

Digital Library

[13]

F. Shao, G. Jiang, W. Lin, M. Yu, and Q. Dai, “Joint bit allocation and rate control for coding multi-view video plus depth based 3D video,” IEEE Trans. Multimedia, vol. 15, no. 8, pp. 1843–1854, Dec. 2013.

Digital Library

[14]

T.-Y. Chung, J.-Y. Sim, and C.-S. Kim, “Bit allocation algorithm with novel view synthesis distortion model for multiview video plus depth coding,” IEEE Trans. Image Process., vol. 23, no. 8, pp. 3254–3267, Aug. 2014.

[15]

M. M. Hannuksela et al., “Multiview-video-plus-depth coding based on the advanced video coding standard,” IEEE Trans. Image Process., vol. 22, no. 9, pp. 3449–3458, Sep. 2013.

Digital Library

[16]

K. Muller et al., “3D high-efficiency video coding for multi-view video and depth data,” IEEE Trans. Image Process., vol. 22, no. 9, pp. 3366–3378, Sep. 2013.

Digital Library

[17]

Methodology for the Subjective Assessment of the Quality of Television Pictures, document ITU-R Rec. BT.500, 2002.

[18]

Subjective Video Quality Assessment Methods for Multimedia Applications, document ITU-T Rec. P.910, 1999.

[19]

S. Chikkerur, V. Sundaram, M. Reisslein, and L. J. Karam, “Objective video quality assessment methods: A classification, review, and performance comparison,” IEEE Trans. Broadcast., vol. 57, no. 2, pp. 165–182, Jun. 2011.

[20]

VQEG Final Report of FR-TV Phase II Validation. [Online]. Available: http://www.itu.int/ITU-T/studygroups/com09/docs/tutorial_opavc.pdf, accessed May 2005.

[21]

K. Seshadrinathan, R. Soundararajan, A. C. Bovik, and L. K. Cormack, “Study of subjective and objective quality assessment of video,” IEEE Trans. Image Process., vol. 19, no. 6, pp. 1427–1441, Jun. 2010.

Digital Library

[22]

P. Lebreton, A. Raake, M. Barkowsky, and P. Le Callet, “Evaluating depth perception of 3D stereoscopic videos,” IEEE J. Sel. Topics Signal Process., vol. 6, no. 6, pp. 710–720, Oct. 2012.

[23]

T. Kim, J. Kang, S. Lee, and A. C. Bovik, “Multimodal interactive continuous scoring of subjective 3D video quality of experience,” IEEE Trans. Multimedia, vol. 16, no. 2, pp. 387–402, Feb. 2014.

Digital Library

[24]

(2011). IRCCyN/IVC DIBR Videos Database. [Online]. Available: http://ivc.univ-nantes.fr/en/databases/DIBR_Videos/

[25]

E. Bosc et al., “Towards a new quality metric for 3D synthesized view assessment,” IEEE J. Sel. Topics Signal Process., vol. 5, no. 7, pp. 1332–1343, Nov. 2011.

[26]

E. Bosc, P. Hanhart, P. Le Callet, and T. Ebrahimi, “A quality assessment protocol for free-viewpoint video sequences synthesized from decompressed depth data,” in Proc. 5th Int. Workshop Quality Multimedia Exper. (QoMEX), Jul. 2013, pp. 100–105.

[27]

SIAT Synthesized Video Quality Database. [Online]. Available: http://codec.siat.ac.cn/SIAT_Synthesized_Video_Quality_Database/index.html, accessed Aug. 2015.

[28]

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004.

Digital Library

[29]

F. Shao, W. Lin, S. Gu, G. Jiang, and T. Srikanthan, “Perceptual fullreference quality assessment of stereoscopic images by considering binocular visual characteristics,” IEEE Trans. Image Process., vol. 22, no. 5, pp. 1940–1953, May 2013.

Digital Library

[30]

Y.-H. Lin and J.-L. Wu, “Quality assessment of stereoscopic 3D image compression by binocular integration behaviors,” IEEE Trans. Image Process., vol. 23, no. 4, pp. 1527–1542, Apr. 2014.

Digital Library

[31]

K. Seshadrinathan and A. C. Bovik, “Motion tuned spatio-temporal quality assessment of natural videos,” IEEE Trans. Image Process., vol. 19, no. 2, pp. 335–350, Feb. 2010.

Digital Library

[32]

V. De Silva, H. K. Arachchi, E. Ekmekcioglu, and A. Kondoz, “Toward an impairment metric for stereoscopic video: A full-reference video quality metric to assess compressed stereoscopic video,” IEEE Trans. Image Process., vol. 22, no. 9, pp. 3392–3404, Sep. 2013.

Digital Library

[33]

D. Rusanovskyy, F.-C. Chen, L. Zhang, and T. Suzuki, 3D-AVC Test Model 8, Joint Collaborative Team 3D Video Coding Extensions (JCT-3V), document JCT3V-F1003, Nov. 2013.

[34]

Y. Zhao and L. Yu, “A perceptual metric for evaluating quality of synthesized sequences in 3DV system,” Proc. SPIE, vol. 7744, p. 77440X, Aug. 2010.

[35]

E. Ekmekcioglu, S. Worrall, D. De Silva, A. Fernando, and A. Kondoz, “Depth based perceptual quality assessment for synthesised camera viewpoints,” in User Centric Media (Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering), vol. 60. Berlin, Germany: Springer-Verlag, 2012, pp. 76–83.

[36]

E. Bosc, P. Le Callet, L. Morin, and M. Pressigout, “An edge-based structural distortion indicator for the quality assessment of 3D synthesized views,” in Proc. Picture Coding Symp. (PCS), May 2012, pp. 249–252.

[37]

E. Bosc, F. Battisti, M. Carli, and P. Le Callet, “A wavelet-based image quality metric for the assessment of 3D synthesized views,” Proc. SPIE, vol. 8648, p. 86481Z, Mar. 2013.

[38]

F. Battisti, E. Bosc, M. Carli, P. Le Callet, and S. Perugia, “Objective image quality assessment of 3D synthesized views,” Signal Process., Image Commun., vol. 30, pp. 78–88, Jan. 2015.

Digital Library

[39]

C.-T. Tsai and H.-M. Hang, “Quality assessment of 3D synthesized views with depth map distortion,” in Proc. Vis. Commun. Image Process. (VCIP), Nov. 2013, pp. 1–6.

[40]

H.264/AVC Reference Software JM 18. [Online]. Available: http://iphome.hhi.de/suehring/tml/download/, accessed May 2015.

[41]

Y. Jia, W. Lin, and A. A. Kassim, “Estimating just-noticeable distortion for video,” IEEE Trans. Circuits Syst. Video Technol., vol. 16, no. 7, pp. 820–829, Jul. 2006.

Digital Library

[42]

A. Liu, W. Lin, and M. Narwaria, “Image quality assessment based on gradient similarity,” IEEE Trans. Image Process., vol. 21, no. 4, pp. 1500–1512, Apr. 2012.

Digital Library

[43]

Draft Call for Proposals on 3D Video Coding Technology, ISO/IEC JTC1/SC29/WG11, document MPEG2011/N11830, Jan. 2011.

[44]

D. Rusanovskyy, K. Müller, and A. Vetro, Common Test Conditions of 3DV Core Experiments, Joint Collaborative Team 3D Video Coding Extensions (JCT-3V), document JCT3V-E1100, Aug. 2013.

[45]

Reference Software for 3D-AVC: 3DV-ATM V10.0. [Online]. Available: http://mpeg3dv.nokiaresearch.com/svn/mpeg3dv/tags/, accessed Nov. 2013.

[46]

VSRS-1D-Fast. [Online]. Available: https://hevc.hhi.fraunhofer.de/svn/svn_3DVCSoftware, accessed Aug. 2015.

[47]

Anchor Software for 3D-HEVC Experiments: 3DV-HTM V8.0. [Online]. Available: https://hevc.hhi.fraunhofer.de/svn/svn_3DVCSoftware, accessed Aug. 2015.

[48]

MSU Perceptual Video Quality Tool. [Online]. Available: http://www.compression.ru/video/quality_measure/perceptual_video_quality_tool_en.html, accessed Aug. 2015.

[49]

R. Miller, Beyond ANOVA: Basics of Applied Statistics (Texts in Statistical Science Series). London, U.K.: Chapman & Hall, 1997.

[50]

L. Fang, N.-M. Cheung, D. Tian, A. Vetro, H. Sun, and O. C. Au, “An analytical model for synthesis distortion estimation in 3D video,” IEEE Trans. Image Process., vol. 23, no. 1, pp. 185–199, Jan. 2014.

Digital Library

[51]

R. Li, B. Zeng, and M. L. Liou, “A new three-step search algorithm for block motion estimation,” IEEE Trans. Circuits Syst. Video Technol., vol. 4, no. 4, pp. 438–442, Aug. 1994.

Digital Library

[52]

Y. Su, M.-T. Sun, and V. Hsu, “Global motion estimation from coarsely sampled motion vector field and the applications,” IEEE Trans. Circuits Syst. Video Technol., vol. 15, no. 2, pp. 232–242, Feb. 2005.

Digital Library

[53]

A. K. Moorthy and A. C. Bovik, “Visual importance pooling for image quality assessment,” IEEE J. Sel. Topics Signal Process., vol. 3, no. 2, pp. 193–201, Apr. 2009.

[54]

M. H. Pinson and S. Wolf, “A new standardized method for objectively measuring video quality,” IEEE Trans. Broadcast., vol. 50, no. 3, pp. 312–322, Sep. 2004.

[55]

A. Ninassi, O. Le Meur, P. Le Callet, and D. Barba, “Considering temporal variations of spatial visual distortions in video quality assessment,” IEEE J. Sel. Topics Signal Process., vol. 3, no. 2, pp. 253–265, Apr. 2009.

[56]

S. L. Cloherty, M. J. Mustari, M. G. Rosa, and M. R. Ibbotson, “Effects of saccades on visual processing in primate MSTd,” Vis. Res., vol. 50, no. 24, pp. 2683–2691, 2010.

[57]

Z. Chen and C. Guillemot, “Perceptually-friendly H.264/AVC video coding based on foveated just-noticeable-distortion model,” IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 6, pp. 806–819, Jun. 2010.

Digital Library

[58]

A. Liu, W. Lin, M. Paul, C. Deng, and F. Zhang, “Just noticeable difference for images with decomposition model for separating edge and textured regions,” IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 11, pp. 1648–1652, Nov. 2010.

Digital Library

[59]

J. Canny, “A computational approach to edge detection,” IEEE Trans. Pattern Anal. Mach. Intell., vol. PAMI-8, no. 6, pp. 679–698, Nov. 1986.

Digital Library

[60]

N. Damera-Venkata, T. D. Kite, W. S. Geisler, B. L. Evans, and A. C. Bovik, “Image quality assessment based on a degradation model,” IEEE Trans. Image Process., vol. 9, no. 4, pp. 636–650, Apr. 2000.

[61]

Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multiscale structural similarity for image quality assessment,” in Proc. Conf. Rec. 37th Asilomar Conf. Signals, Syst. Comput., vol. 2. Nov. 2003, pp. 1398–1402.

[62]

Video Quality Metric (VQM) Software. [Online]. Available: http://www.its.bldrdoc.gov/resources/video-quality-research/software.aspx, accessed Aug. 2015.

[63]

LIVE Software Releases. [Online]. Available: http://live.ece.utexas.edu/research/quality/, accessed Feb. 2015.

Cited By

Chen LWeng LCheng HCheng AHuang C(2025)Temporal Fusion: Continuous-Time Light Field Video FactorizationIEEE Transactions on Image Processing10.1109/TIP.2025.353320334(955-968)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TIP.2025.3533203
Cui MZhang YFan CHamzaoui RLi Q(2024)Colored Point Cloud Quality Assessment Using Complementary Features in 3D and 2D SpacesIEEE Transactions on Multimedia10.1109/TMM.2024.344363426(11111-11125)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3443634
Tian YChen BWang SKwong S(2024)Towards Thousands to One Reference: Can We Trust the Reference Image for Quality Assessment?IEEE Transactions on Multimedia10.1109/TMM.2023.331026826(3278-3290)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3310268
Show More Cited By

Index Terms

Subjective and Objective Video Quality Assessment of 3D Synthesized Views With Texture/Depth Compression Distortion
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
      2. Image and video acquisition
  2. Computer graphics
    1. Image compression
    2. Image manipulation

Index terms have been assigned to the content through auto-classification.

Recommendations

View synthesis distortion elimination filter for depth video coding in 3D video broadcasting

Depth image based rendering (DIBR), which can generate synthesized images according to the users' demand, is a key technique for achieving 3D television. However, view synthesis by DIBR technique is very sensitive to depth coding distortion. Because ...
Coding and intermediate view synthesis of multiview video plus depth
ICIP'09: Proceedings of the 16th IEEE international conference on Image processing

For advanced 3D Video (3DV) applications, efficient data representations are investigated, which only transmit a subset of the views that are required for 3D visualization. From this subset, all intermediate views are synthesized from sample-dense color ...
Analysis of the depth-shift distortion as an estimator for view synthesis distortion

Aiming for 3D Video encoders with reduced computational complexity, we analyze the performance of depth-shift distortion in depth-image based rendering algorithms, incurred when coding depth maps in 3D Video, as an estimator of the distortion of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Image Processing

IEEE Transactions on Image Processing Volume 24, Issue 12

Dec. 2015

1399 pages

ISSN:1057-7149

Issue’s Table of Contents

Copyright © 2015.

Publisher

IEEE Press

Publication History

Published: 01 December 2015

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen LWeng LCheng HCheng AHuang C(2025)Temporal Fusion: Continuous-Time Light Field Video FactorizationIEEE Transactions on Image Processing10.1109/TIP.2025.353320334(955-968)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TIP.2025.3533203
Cui MZhang YFan CHamzaoui RLi Q(2024)Colored Point Cloud Quality Assessment Using Complementary Features in 3D and 2D SpacesIEEE Transactions on Multimedia10.1109/TMM.2024.344363426(11111-11125)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3443634
Tian YChen BWang SKwong S(2024)Towards Thousands to One Reference: Can We Trust the Reference Image for Quality Assessment?IEEE Transactions on Multimedia10.1109/TMM.2023.331026826(3278-3290)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3310268
Xiang JChen PDang YLiang RJiang G(2024)Pseudo Light Field Image and 4D Wavelet-Transform-Based Reduced-Reference Light Field Image Quality AssessmentIEEE Transactions on Multimedia10.1109/TMM.2023.327385526(929-943)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3273855
Liu XWang MWang SKwong S(2024)Bilateral Context Modeling for Residual Coding in Lossless 3D Medical Image CompressionIEEE Transactions on Image Processing10.1109/TIP.2024.337891033(2502-2513)Online publication date: 25-Mar-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3378910
Shi HHuang YWang LWang L(2024)Quality assessment of view synthesis based on unsupervised quality-aware pre-trainingApplied Soft Computing10.1016/j.asoc.2024.111377154:COnline publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1016/j.asoc.2024.111377
Thakur SJakhetiya VSubudhi BJaiswal SLi LLin W(2023)Context Region Identification Based Quality Assessment of 3D Synthesized ViewsIEEE Transactions on Multimedia10.1109/TMM.2022.320666025(6183-6193)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TMM.2022.3206660
Sun YTang SWang CHsu CLi JGao XLe Callet PLi ZLu WYang JWang J(2022)On Objective and Subjective Quality of 6DoF Synthesized Live Immersive VideosProceedings of the 2nd Workshop on Quality of Experience in Visual Multimedia Applications10.1145/3552469.3555709(49-56)Online publication date: 14-Oct-2022
https://dl.acm.org/doi/10.1145/3552469.3555709
Yan JLi JFang YChe ZXia XLiu Y(2022)Subjective and Objective Quality of Experience of Free Viewpoint VideosIEEE Transactions on Image Processing10.1109/TIP.2022.317712731(3896-3907)Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1109/TIP.2022.3177127
Dziembowski AMieloch DStankowski JGrzelka A(2022)IV-PSNR—The Objective Quality Metric for Immersive Video ApplicationsIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2022.317957532:11(7575-7591)Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1109/TCSVT.2022.3179575
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents