research-article

No-reference mesh visual quality assessment via ensemble of convolutional neural networks and compact multi-linear pooling

Authors:

Ilyass Abouelaziz,

Aladine Chetouani,

Mohammed El Hassouni,

Longin Jan Latecki,

Hocine CherifiAuthors Info & Claims

Volume 100, Issue C

https://doi.org/10.1016/j.patcog.2019.107174

Published: 01 April 2020 Publication History

Highlights

•

The perceived quality of 3D meshes is influenced due to distortions caused by many geometric operations.

•

A no-reference mesh visual quality assessment method is proposed to automatically estimate the perceived quality.

•

Deep convolutional networks and compact multi-linear pooling is adopted in our method.

•

Extensive experiments and comparisons with existing methods are conducted on mesh quality datasets.

Abstract

Blind or No reference quality evaluation is a challenging issue since it is done without access to the original content. In this work, we propose a method based on deep learning for the mesh visual quality assessment without reference. For a given 3D model, we first compute its mesh saliency. Then, we extract views from the 3D mesh and the corresponding mesh saliency. After that, the views are split into small patches that are filtered using a saliency threshold. Only the salient patches are selected and used as input data. After that, three pre-trained deep convolutional neural networks are employed for feature learning: VGG, AlexNet, and ResNet. Each network is fine-tuned and produces a feature vector. The Compact Multi-linear Pooling (CMP) is used afterward to fuse the retrieved vectors into a global feature representation. Finally, fully connected layers followed by a regression module are used to estimate the quality score. Extensive experiments are executed on four mesh quality datasets and comparisons with existing methods demonstrate the effectiveness of our method in terms of correlation with subjective scores.

References

[1]

M. Botsch, L. Kobbelt, M. Pauly, P. Alliez, B. Lévy, Polygon Mesh Processing, CRC press, 2010.

[2]

P. Alliez, C. Gotsman, Recent advances in compression of 3d meshes, Advances in Multiresolution for Geometric Modelling, Springer, 2005, pp. 3–26.

[3]

H. Lee, c Dikici, G. Lavoué, F. Dupont, Joint reversible watermarking and progressive compression of 3d meshes, Visual Comput. 27 (6) (2011) 781–792.

[4]

K. Wang, G. Lavoué, F. Denis, A. Baskurt, A comprehensive survey on three-dimensional mesh watermarking, IEEE Trans. Multimed. 10 (8) (2008) 1513–1527.

[5]

Y.P. Wang, S.M. Hu, A new watermarking method for 3d models based on integral invariants, IEEE Transactions on Visualization and Computer Graphics, vol. 15, 2009, pp. 285–294.

[6]

A. Bulbul, T. Capin, G. Lavoué, M. Preda, Assessing visual quality of 3-d polygonal models, IEEE Signal Process. Mag. 28 (6) (2011) 80–90.

[7]

P. Cignoni, C. Rocchini, R. Scopigno, Metro: Measuring error on simplified surfaces, Computer Graphics Forum 17 (1998) 167–174.

[8]

N. Aspert, D. Santa-Cruz, T. Ebrahimi, MESH: Measuring errors between surfaces using the hausdorff distance, Multimedia and Expo, 2002. ICME’02. Proceedings. 2002 IEEE International Conference on, 1, 2002, pp. 705–708.

[9]

Z. Wang, A.C. Bovik, Mean squared error: love it or leave it? A new look at signal fidelity measures, IEEE Signal Process. Mag. 26 (1) (2009) 98–117.

[10]

G. Lavoué, M. Corsini, A comparison of perceptually-based metrics for objective evaluation of geometry processing, IEEE Trans. Multimed. 12 (7) (2010) 636–649.

[11]

Y. Gao, O. Beijbom, N. Zhang, T. Darrell, Compact bilinear pooling, Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 317–326.

[12]

Z. Karni, C. Gotsman, Spectral compression of mesh geometry, Proceedings of the 27th annual conference on Computer graphics and interactive techniques, ACM Press/Addison-Wesley Publishing Co, 2000, pp. 279–286.

[13]

O. Sorkine, D. Cohen-Or, S. Toledo, High-Pass Quantization for Mesh Encoding, Vol. 42, 2003, June.

[14]

Z. Bian, S.M. Hu, R.R. Martin, Evaluation for small visual difference between conforming meshes on strain field, J. Comput. Sci. Technol. 24 (1) (2009) 65–75.

[15]

Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE Trans. Image Process. 13 (4) (2004) 600–612.

Digital Library

[16]

G. Lavoué, E.D. Gelasca, F. Dupont, A. Baskurt, T. Ebrahimi, Perceptually driven 3d distance metrics with application to watermarking, SPIE Optics+ Photonics, International Society for Optics and Photonics, 2006.

[17]

G. Lavoué, A multiscale metric for 3d mesh visual quality assessment, Comput. Graph. Forum 30 (2011) 1427–1437.

[18]

F. Torkhani, K. Wang, J.M. Chassery, A curvature tensor distance for mesh visual quality assessment, Comput. Vis. Graph. (2012) 253–263.

[19]

Z.C. Yildiz, T. Capin, A perceptual quality metric for dynamic triangle meshes, EURASIP J. Image Video Process. 2017 (1) (2017) 12.

[20]

X. Feng, W. Wan, R.Y.D. Xu, S. Perry, P. Li, S. Zhu, A novel spatial pooling method for 3d mesh quality assessment based on percentile weighting strategy, Comput. Graph. 74 (2018) 12–22.

[21]

A. Chetouani, Three-dimensional mesh quality metric with reference based on a support vector regression model, J. Electron. Imaging 27 (4) (2018) 43048.

[22]

M. Corsini, M.-C. Larabi, G. Lavoué, O. Petřík, L. Váša, K. Wang, Perceptual metrics for static and dynamic triangle meshes, Comput. Graph. Forum 32 (2013) 101–125.

[23]

M. Corsini, E.D. Gelasca, T. Ebrahimi, M. Barni, Watermarked 3d mesh quality assessment, IEEE Trans. Multimed. 9 (2) (2007) 247–256.

[24]

L. Váša, J. Rus, Dihedral angle mesh error: a fast perception correlated distortion measure for fixed connectivity triangle meshes, Comput. Graph. Forum 31 (2012) 1715–1724.

[25]

A.K. Moorthy, A.C. Bovik, A two-step framework for constructing blind image quality indices, IEEE Signal Process. Lett. 17 (5) (2010) 513–516.

[26]

C. Li, A.C. Bovik, X. Wu, Blind image quality assessment using a general regression neural network, IEEE Trans. Neural Netw. 22 (5) (2011) 793–799.

Digital Library

[27]

A. Mittal, A.K. Moorthy, A.C. Bovik, No-reference image quality assessment in the spatial domain, IEEE Trans. Image Process. 21 (12) (2012) 4695–4708.

Digital Library

[28]

J. Gu, G. Meng, S. Xiang, C. Pan, Blind image quality assessment via learnable attention-based pooling, Pattern Recognit. 91 (2019) 332–344.

[29]

W. Zhang, C. Qu, L. Ma, J. Guan, R. Huang, Learning structure of stereoscopic image for no-reference quality assessment with convolutional neural network, Pattern Recognit. 59 (2016) 176–187.

[30]

P. Chen, L. Li, X. Zhang, S. Wang, A. Tan, Blind quality index for tone-mapped images based on luminance partition, Pattern Recognit. 89 (2019) 108–118.

[31]

I. Abouelaziz, M.E. Hassouni, H. Cherifi, A curvature based method for blind mesh visual quality assessment using a general regression neural network, Signal-Image Technology & Internet-Based Systems (SITIS), 2016 12th International Conference on, IEEE, 2016, pp. 793–797.

[32]

A. Nouri, C. Charrier, O. Lézoray, 3D blind mesh quality assessment index, Electron. Imaging 2017 (20) (2017) 9–26.

[33]

I. Abouelaziz, M.E. Hassouni, H. Cherifi, No-reference 3d mesh quality assessment based on dihedral angles model and support vector regression, International Conference on Image and Signal Processing, Springer, 2016, pp. 369–377.

[34]

I. Abouelaziz, M.E. Hassouni, H. Cherifi, Blind 3d mesh visual quality assessment using support vector regression, Multimed. Tools Appl. 77 (18) (2018) 24365–24386.

[35]

I. Abouelaziz, M.E. Hassouni, H. Cherifi, A convolutional neural network framework for blind mesh visual quality assessment, IEEE International Conference on Image Processing (ICIP), IEEE, 2017, pp. 755–759.

[36]

I. Abouelaziz, A. Chetouani, M.E. Hassouni, H. Cherifi, A blind mesh visual quality assessment method based on convolutional neural network, Electronic Imaging 2018 (18) (2018).

[37]

I. Abouelaziz, A. Chetouani, M.E. Hassouni, L.J. Latecki, H. Cherifi, Convolutional neural network for blind mesh visual quality assessment using 3d visual saliency, 2018 25th IEEE International Conference on Image Processing (ICIP), IEEE, 2018, pp. 3533–3537.

[38]

C.H. Lee, A. Varshney, D.W. Jacobs, Mesh saliency, ACM Trans. Graph. (TOG ACM) 24 (2005) 659–666.

[39]

L. Itti, C. Koch, E. Niebur, A model of saliency-based visual attention for rapid scene analysis, IEEE Trans. Pattern Anal. Mach.Intell. 20 (11) (1998) 1254–1259.

Digital Library

[40]

S. Bai, X. Bai, Z. Zhou, Z. Zhang, Q. Tian, L.J. Latecki, Gift: towards scalable 3d shape retrieval, IEEE Trans. Multimed. 19 (6) (2017) 1257–1271.

[41]

Z. Zhu, Cong Rao, Song Bai, L. Longin Jan, Training convolutional neural network from multi-domain contour images for 3D shape retrieval, Pattern Recognition Letters, 2017.

[42]

A. Krizhevsky, One weird trick for parallelizing convolutional neural networks, 2014.

[43]

K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, 2014.

[44]

K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.

[45]

A. Fukui, D.H. Park, D. Yang, A. Rohrbach, T. Darrell, M. Rohrbach, Multimodal compact bilinear pooling for visual question answering and visual grounding, 2016.

[46]

F.M. Algashaam, K. Nguyen, M. Alkanhal, V. Chandran, W. Boles, J. Banks, Multispectral periocular classification with multimodal compact multi-linear pooling, IEEE Access 5 (2017) 14572–14578.

[47]

L. Kang, P. Ye, Y. Li, D. Doermann, Convolutional neural networks for no-reference image quality assessment, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1733–1740.

[48]

G. Lavoué, M.C. Larabi, L. Váša, On the efficiency of image metrics for evaluating the visual quality of 3d models, IEEE Trans. Visualization Comput.Graph. 22 (8) (2016) 1987–1999.

[49]

S. Silva, B.S. Santos, C. Ferreira, J. Madeira, A perceptual data repository for polygonal meshes, Visualisation, 2009. VIZ’09. Second International Conference in, IEEE, 2009, pp. 207–212.

[50]

Z. Wang, A.C. Bovik, Modern image quality assessment, synthesis lectures on image, Video Multimed. Process. 2 (1) (2006) 1–156.

[51]

K. Wang, F. Torkhani, A. Montanvert, A fast roughness-based approach to the assessment of 3d mesh visual quality, Comput. Graph. 36 (7) (2012) 808–818.

[52]

P.G. Engeldrum, Psychometric scaling: avoiding the pitfalls and hazards, PICS (2001) 101–107.

Cited By

Roullier BMcQuade FAnjum ABower CLiu L(2024)Automated visual quality assessment for virtual and augmented reality based digital twinsJournal of Cloud Computing: Advances, Systems and Applications10.1186/s13677-024-00616-w13:1Online publication date: 26-Feb-2024
https://dl.acm.org/doi/10.1186/s13677-024-00616-w
Zhang ZSun WWu HZhou YLi CChen ZMin XZhai GLin W(2024)GMS-3DQA: Projection-Based Grid Mini-patch Sampling for 3D Model Quality AssessmentACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364381720:6(1-19)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1145/3643817
Nehmé YDelanoy JDupont FFarrugia JLe Callet PLavoué G(2023)Textured Mesh Quality Assessment: Large-scale Dataset and Deep Learning-based Quality MetricACM Transactions on Graphics10.1145/359278642:3(1-20)Online publication date: 14-Apr-2023
https://dl.acm.org/doi/10.1145/3592786
Show More Cited By

Index Terms

No-reference mesh visual quality assessment via ensemble of convolutional neural networks and compact multi-linear pooling
1. Computing methodologies
2. Human-centered computing

Index terms have been assigned to the content through auto-classification.

Recommendations

3D visual saliency and convolutional neural network for blind mesh quality assessment
Abstract
A number of full reference and reduced reference methods have been proposed in order to estimate the perceived visual quality of 3D meshes. However, in most practical situations, there is a limited access to the information related to the ...
Blind 3D mesh visual quality assessment using support vector regression

Various visual distortions can inevitably affect the 3D meshes during their transmission and geometrical processing. In most practical cases, blind quality assessment becomes a challenging issue due to the unavailability of reference meshes and ...
Convolutional Neural Networks for No-Reference Image Quality Assessment
CVPR '14: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition

In this work we describe a Convolutional Neural Network (CNN) to accurately predict image quality without a reference image. Taking image patches as input, the CNN works in the spatial domain without using hand-crafted features that are employed by most ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Pattern Recognition

Pattern Recognition Volume 100, Issue C

Apr 2020

778 pages

ISSN:0031-3203

Issue’s Table of Contents

Copyright © 2019.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 01 April 2020

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Roullier BMcQuade FAnjum ABower CLiu L(2024)Automated visual quality assessment for virtual and augmented reality based digital twinsJournal of Cloud Computing: Advances, Systems and Applications10.1186/s13677-024-00616-w13:1Online publication date: 26-Feb-2024
https://dl.acm.org/doi/10.1186/s13677-024-00616-w
Zhang ZSun WWu HZhou YLi CChen ZMin XZhai GLin W(2024)GMS-3DQA: Projection-Based Grid Mini-patch Sampling for 3D Model Quality AssessmentACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364381720:6(1-19)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1145/3643817
Nehmé YDelanoy JDupont FFarrugia JLe Callet PLavoué G(2023)Textured Mesh Quality Assessment: Large-scale Dataset and Deep Learning-based Quality MetricACM Transactions on Graphics10.1145/359278642:3(1-20)Online publication date: 14-Apr-2023
https://dl.acm.org/doi/10.1145/3592786
Muthusamy DS. S(2023)Feature Sampling based on Multilayer Perceptive Neural Network for image quality assessmentEngineering Applications of Artificial Intelligence10.1016/j.engappai.2023.106015121:COnline publication date: 1-May-2023
https://dl.acm.org/doi/10.1016/j.engappai.2023.106015
Kerkouri MTliba MChetouani ABruno A(2022)A domain adaptive deep learning solution for scanpath prediction of paintingsProceedings of the 19th International Conference on Content-based Multimedia Indexing10.1145/3549555.3549597(57-63)Online publication date: 14-Sep-2022
https://dl.acm.org/doi/10.1145/3549555.3549597

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents