research-article

Multi-level photo quality assessment with multi-view features

Authors:

Xinmei TianAuthors Info & Claims

Neurocomputing, Volume 168, Issue C

Pages 308 - 319

https://doi.org/10.1016/j.neucom.2015.05.095

Published: 30 November 2015 Publication History

Abstract

Photo quality assessment using the principles of visual aesthetics has been a hot research topic in recent years. Due to the complexity of human aesthetic activities, this task is very challenging. Most existing works apply binary labels ("good" or "bad") to represent the photo quality, and focus on constructing rule-based features under the guidance of photography knowledge. However, those features strictly imitate photography rules and suffer from low effectiveness and high computational cost. Besides, the binary quality representation is oversimplified and fails to distinguish varying quality degrees. To tackle these problems, we construct new effective and efficient features from different views and further fuse them to predict multi-level, instead of binary, photo quality. More specifically, we design a set of compact rule-based features through careful analyses on photographic rules and aesthetic attributes. We propose using Deep Convolutional Neural Network (DCNN) descriptor, which encodes the photo content thoroughly, to implicitly describe the photo quality. Experiments conducted on two large scale benchmark datasets verify the effectiveness of these two different kinds of features. Furthermore, we propose a method to combine these features for multi-level photo quality prediction. This feature fusion method has proven to be effective on a dataset that is carefully organized to support research on this new multi-level photo quality assessment problem.

References

[1]

R. Datta, D. Joshi, J. Li, J.Z. Wang, Studying aesthetics in photographic images using a computational approach, in: ECCV, Springer-Verlag Berlin, Heidelberg, 2006, pp. 288-301.

[2]

Y. Ke, X.O. Tang, F. Jing, The design of high-level features for photo quality assessment, in: CVPR, vol. 1, IEEE, 2006, pp. 419-426.

[3]

Y.W. Luo, X.O. Tang, Photo and video quality evaluation: focusing on the subject, in: ECCV, Springer, 2008, pp. 386-399.

[4]

L.K. Wong, K.L. Low, Saliency-enhanced image aesthetics class prediction, in: ICIP, IEEE, 2009, pp. 997-1000.

[5]

K.Y. Lo, K.H. Liu, C.S. Chen, Assessment of photo aesthetics with efficiency, in: ICPR, IEEE, 2012, pp. 2186-2189.

[6]

Z. Dong, X.M. Tian, Effective and efficient photo quality assessment, in: International Conference on Systems, Man and Cybernetics, IEEE, 2014.

[7]

S. Bhattacharya, R. Sukthankar, M. Shah, A framework for photo-quality assessment and enhancement based on visual aesthetics, in: Proceedings of the International Conference on Multimedia, ACM, 2010, pp. 271-280.

[8]

J. Yu, Y. Rui, D. Tao, Click prediction for web image reranking using multimodal sparse coding, IEEE Trans. Image Process, 23 (2014) 2019-2032.

[9]

J. Yu, D. Tao, M. Wang, Y. Rui, Learning to rank using user clicks and visual features for image retrieval, IEEE Trans. Cybern., 45 (2015) 767-779.

[10]

B. Geng, L.J. Yang, C. Xu, X.S. Hua, S.P. Li, The role of attractiveness in web image search, in: Proceedings of the 19th ACM International Conference on Multimedia, ACM, 2011, pp. 63-72.

[11]

P. Obrador, X. Anguera, O. Rodrigo, N. Oliver, The role of tags and image aesthetics in social image search, in: Proceedings of the First SIGMM Workshop on Social Media, ACM, 2009, pp. 65-72.

Digital Library

[12]

M. Wang, B. Liu, X.-S. Hua, Accessible image search, in: Proceedings of the 17th ACM International Conference on Multimedia, ACM, 2009, pp. 291-300.

[13]

C.C. Li, A.C. Loui, T. Chen, Towards aesthetics: a photo quality assessment and photo selection system, in: Proceedings of the International Conference on Multimedia, ACM, 2010, pp. 827-830.

[14]

Z. Dong, X. Shen, X.M. Tian, Photo quality assessment with DCNN that understands images well, in: International Conference on Multimedia Modeling, IEEE, 2015.

[15]

L. Marchesotti, F. Perronnin, D. Larlus, G. Csurka, Assessing the aesthetic quality of photographs using generic image descriptors, in: ICCV, IEEE, 2011, pp. 1784-1791.

[16]

A. Krizhevsky, I. Sutskever, G.E. Hinton, ImageNet classification with deep convolutional neural networks, in: Advances in Neural Information Processing Systems, 2012, pp. 1097-1105.

Digital Library

[17]

C. Cortes, M. Mohri, A. Rostamizadeh, Two-stage learning kernel algorithms, in: ICML, 2010, pp. 239-246.

[18]

N. Shawe-Taylor, A. Kandola, On kernel target alignment, Adv. Neural Inf. Process. Syst., 14 (2002) 367.

[19]

L. Zhang, D. Zhang, X. Mou, Fsim, IEEE Trans. Image Process., 20 (2011) 2378-2386.

[20]

A. Mittal, A.K. Moorthy, A.C. Bovik, No-reference image quality assessment in the spatial domain, IEEE Trans. Image Process., 21 (2012) 4695-4708.

Digital Library

[21]

H.H. Tong, M.J. Li, H.J. Zhang, J.R. He, C.S. Zhang, Classification of digital photos taken by photographers or home users, in: Advances in Multimedia Information Processing-PCM 2004, Springer, 2005, pp. 198-205.

[22]

M. Nishiyama, T. Okabe, I. Sato, Y. Sato, Aesthetic quality classification of photographs based on color harmony, in: CVPR, IEEE, 2011, pp. 33-40.

[23]

C. Xu, D.C. Tao, C. Xu, A Survey on Multi-view Learning, Preprint:1304.5634.

[24]

J. Yu, M. Wang, D.C. Tao, Semisupervised multiview distance metric learning for cartoon synthesis, IEEE Trans. Image Process., 21 (2012) 4636-4648.

Digital Library

[25]

J. Yu, Y. Rui, Y.Y. Tang, D. Tao, High-order distance-based multiview stochastic learning in image classification. 44(12), 2014, 2431-2442

[26]

R. Datta, J. Li, J.Z. Wang, Algorithmic inferencing of aesthetics and emotion in natural images: an exposition, in: ICIP, IEEE, 2008, pp. 105-108.

[27]

J. San P., T. Yeh, N. Oliver, Leveraging user comments for aesthetic aware image search reranking, in: Proceedings of the 21st International Conference on World Wide Web, ACM, 2012, pp. 439-448.

[28]

H.J. Yao, Image Retrieval Based on Color, Space and Texture Features, 2007, {http://www.paper.edu.cn}.

[29]

C.L. Hua, L. Wei, L.G. Hui, Research and implementation of an image retrieval algorithm based on multiple dominant colors, J. Comput. Res. Dev., 36 (1999).

[30]

F.-H. Kong, Image retrieval using both color and texture features, in: International Conference on Machine Learning and Cybernetics, vol. 4, IEEE, 2009, pp. 2228-2232.

[31]

W.T. Chu, Y.K. Chen, K.T. Chen, Size does matter: how image size affects aesthetic perception?, in: Proceedings of the 21st ACM International Conference on Multimedia, ACM, 2013, pp. 53-62.

[32]

Y. LeCun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel, Handwritten digit recognition with a back-propagation network, in: Advances in Neural Information Processing Systems, Citeseer, 1990.

[33]

Y. LeCun, Y. Bengio, Convolutional networks for images, speech, and time series, The Handbook of Brain Theory and Neural Networks, vol. 3361.

[34]

G. Lanckriet, N. Cristianini, P. Bartlett, L.E. Ghaoui, M.I. Jordan, Learning the kernel matrix with semidefinite programming, J. Mach. Learn. Res., 5 (2004) 27-72.

Digital Library

[35]

N. Srebro, S. Ben-David, Learning bounds for support vector machines with learned kernels, in: Learning Theory, Springer, 2006, pp. 169-183.

[36]

C. Cortes, M. Mohri, A. Rostamizadeh, L2 regularization for learning kernels, in: Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, AUAI Press, 2009, pp. 109-116.

Digital Library

[37]

V. Sindhwani, A.C. Lozano, Non-parametric group orthogonal matching pursuit for sparse learning with multiple kernels, in: Advances in Neural Information Processing Systems, 2011, pp. 2519-2527.

[38]

J. Yu, D. Tao, Modern Machine Learning Techniques and their Applications in Cartoon Animation Research, vol. 4, John Wiley & Sons, 2013.

[39]

N. Murray, L. Marchesotti, F. Perronnin, AVA: a large-scale database for aesthetic visual analysis, in: CVPR, IEEE, 2012, pp. 2408-2415.

[40]

X.O. Tang, W. Luo, X.G. Wang, Content-based photo quality assessment, IEEE Trans. Multimed., 15 (2013) 1930-1943.

Digital Library

Cited By

Li W(2023)Aesthetic Assessment of Packaging Design Based on Con-TransformerInternational Journal of e-Collaboration10.4018/IJeC.31687319:5(1-11)Online publication date: 27-Jan-2023
https://dl.acm.org/doi/10.4018/IJeC.316873
Tian TNan F(2022)A Multitask Convolutional Neural Network for Artwork AppreciationMobile Information Systems10.1155/2022/88047112022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/8804711
Dai Y(2021)Sample-specific repetitive learning for photo aesthetic auto-assessment and highlight elements analysisMultimedia Tools and Applications10.1007/s11042-020-09426-z80:1(1387-1402)Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1007/s11042-020-09426-z
Show More Cited By

Recommendations

Photo aesthetic quality estimation using visual complexity features

The amount of visual data available on the Web is growing explosively and it is becoming increasingly important to explore methods for automatically estimating the quality of this content in a manner that is consistent with the aesthetic perceptions of ...
Content-based photo quality assessment
ICCV '11: Proceedings of the 2011 International Conference on Computer Vision

Automatically assessing photo quality from the perspective of visual aesthetics is of great interest in high-level vision research and has drawn much attention in recent years. In this paper, we propose content-based photo quality assessment using ...
Cascaded multi-3D-view fusion for 3D-oriented object detection
Abstract
Currently, multi-view fusion methods fuse point- or proposal-level features from different views at the end stage of the backbone. This once-end-fusion method is not conducive to the timely adjustment of spatial misalignment for features from ...
Graphical abstract

Display Omitted
Highlights
- A multi-3D-view fusion method for accurate 3D orientation detection is proposed.
- A cascaded multi-view feature fusion module is proposed.
- Multi-view features are fused in a cascaded bi-directional fusion manner.
- Modulated ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Neurocomputing

Neurocomputing Volume 168, Issue C

November 2015

1211 pages

ISSN:0925-2312

Issue’s Table of Contents

Copyright © Elsevier B.V.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 30 November 2015

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li W(2023)Aesthetic Assessment of Packaging Design Based on Con-TransformerInternational Journal of e-Collaboration10.4018/IJeC.31687319:5(1-11)Online publication date: 27-Jan-2023
https://dl.acm.org/doi/10.4018/IJeC.316873
Tian TNan F(2022)A Multitask Convolutional Neural Network for Artwork AppreciationMobile Information Systems10.1155/2022/88047112022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/8804711
Dai Y(2021)Sample-specific repetitive learning for photo aesthetic auto-assessment and highlight elements analysisMultimedia Tools and Applications10.1007/s11042-020-09426-z80:1(1387-1402)Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1007/s11042-020-09426-z
Santos ICastro LRodriguez-Fernandez NTorrente-Patiño ÁCarballal A(2021)Artificial Neural Networks and Deep Learning in the Visual Arts: a reviewNeural Computing and Applications10.1007/s00521-020-05565-433:1(121-157)Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1007/s00521-020-05565-4
Carballal ACedron FSantos ISantos ARomero J(2021)Minimal neural network topology optimization for aesthetic classificationNeural Computing and Applications10.1007/s00521-020-05550-x33:1(107-119)Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1007/s00521-020-05550-x
Jin XWu LZhao GZhou XZhang XLi X(2020)IDEA: A new dataset for image aesthetic scoringMultimedia Tools and Applications10.1007/s11042-018-6436-079:21-22(14341-14355)Online publication date: 1-Jun-2020
https://dl.acm.org/doi/10.1007/s11042-018-6436-0
Carballal AFernandez-Lozano CHeras JRomero J(2020)Transfer learning features for predicting aesthetics through a novel hybrid machine learning methodNeural Computing and Applications10.1007/s00521-019-04065-432:10(5889-5900)Online publication date: 1-May-2020
https://dl.acm.org/doi/10.1007/s00521-019-04065-4
Wen YShao YZheng D(2019)A Novel Deep Convolutional Neural Network Structure for Off-line Handwritten Digit RecognitionProceedings of the 2nd International Conference on Big Data Technologies10.1145/3358528.3358585(216-220)Online publication date: 28-Aug-2019
https://dl.acm.org/doi/10.1145/3358528.3358585
Jin XWu LZhao GLi XZhang XGe SZou DZhou BZhou XAmsaleg LHuet BLarson MGravier GHung HNgo CTsang Ooi W(2019)Aesthetic Attributes Assessment of ImagesProceedings of the 27th ACM International Conference on Multimedia10.1145/3343031.3350970(311-319)Online publication date: 15-Oct-2019
https://dl.acm.org/doi/10.1145/3343031.3350970
Jin XWu LLi XChen SPeng SChi JGe SSong CZhao GMcIlraith SWeinberger K(2018)Predicting aesthetic score distribution through cumulative jensen-shannon divergenceProceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence10.5555/3504035.3504045(77-84)Online publication date: 2-Feb-2018
https://dl.acm.org/doi/10.5555/3504035.3504045
Show More Cited By

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents