article

A novel image annotation model based on content representation with multi-layer segmentation

Authors:

Yubo YuanAuthors Info & Claims

Neural Computing and Applications, Volume 26, Issue 6

Pages 1407 - 1422

https://doi.org/10.1007/s00521-014-1815-6

Published: 01 August 2015 Publication History

Abstract

Image automatic annotation is an important issue of semantic-based image retrieval, and it is still a challenging problem for the reason of semantic gap. In this paper, a novel model with three parts is proposed. The first one is multi-layer image segmentation, in which saliency analysis and normalized cut are combined to segment images into semantic regions in the first layer. While in the second layer, the semantic regions are segmented into grids further . The second one is image content representation by region-based bag-of-words (RBoW) model, which is the variant of BoW model. Considering the correlations of labels, we adopt second-order CRFs as the third part of our model to ensure the accuracy of automatic image annotation. Experimental results show that our multi-layer segmentation-based image annotation model can achieve promising performance for multi-labeling and outperform the model based on single-layer segmentation and previous algorithm on Corel 5K and Pascal VOC 2007 datasets .

References

[1]

Zhang H, Berg A, Maire M, Malik J (2006) SVM-KNN: discriminative nearest neighbor classification for visual category recognition. In: 2006 IEEE computer society conference on computer vision and pattern recognition, vol 2, pp 2126---2136

[2]

Guillaumin M, Mensink T, Verbeek J, Schmid C (2009) TagProp: discriminative metric learning in nearest neighbor models for image auto-annotation. In: 2009 IEEE 12th international conference on computer vision, pp 309---316

[3]

Mei T, Wang Y, Hua XS, Gong SG, Li SP (2008) Coherent image annotation by learning semantic distance. In: 2008 CVPR 2008 IEEE conference on computer vision and pattern recognition, pp 1---8

[4]

Shi JB, Malik J (2000) Normalized cuts and image segmentation. IEEE Trans Pattern Anal Mach Intell 22(8):888---905

Digital Library

[5]

Cour T, Bénézit F, Shi JB (2005) Spectral segmentation with multiscale graph decomposition. Spectr Segm Multiscale Graph Decompos 2:1124---1131

[6]

Deng YN, Manjunath BS (2001) Unsupervised segmentation of color-texture regions in images and video. IEEE Trans Pattern Anal Mach Intell 23(8):800---810

Digital Library

[7]

Serrano N, Savakis A, Luo JB (2002) A computationally efficient approach to indoor/outdoor scene classification. In: 2002 Proceedings 16th international conference on pattern recognition, vol 4, pp 146---149

[8]

Zhang J, Zhao Y, Li D, Chen Z, Yuan Y (2013) Representation of image content of image content with multi-scale segmentation. In: 2013 ICMLC machine learning and cybernetics Tianjin, China, July 14---17

[9]

Zhang J, Li D, Zhao Y, Chen Z, Yuan Y (2015) Representation of image content based on RoI-BoW. J Vis Commun Image Represent 26(1):37C49

[10]

Li FF, Perona P (2005) A Bayesian hierarchical model for learning natural scene categories. In: IEEE computer society conference on computer vision and pattern recognition, 2005 (CVPR 2005), vol 2, pp 524---531

[11]

Sivic J, Zisserman A (2003) Video google: a text retrieval approach to object matching in videos. In: 2003 proceedings ninth IEEE international conference on computer vision, vol 2, pp 1470---1477

[12]

Wang XS, Liu X, Shi ZP, Shi ZZ, Sui HJ (2010) Voting conditional random fields for multi-label image classification. In: 2010 3rd international congress on image and signal processing (CISP), vol 4, pp 1984---1988

[13]

Varma M, Zisserman A (2005) A statistical approach to texture classification from single images. Int J Comput Vis 62:61---81

Digital Library

[14]

Li T, Kweon I-S (2008) A semantic region descriptor for local feature based image categorization. In: 2008 IEEE international conference on acoustics, speech and signal processing (2008 ICASSP), pp 1333---1336

[15]

Zhang JG, Marszalek M, Lazebnik S, Schmid C (2006) Local features and kernels for classification of texture and object categories: a comprehensive study. In: Conference on computer vision and pattern recognition workshop, 2006 (CVPRW'06), p 13

[16]

Wu X, Zhao WL, Ngo CW (2007) Near-duplicate keyframe retrieval with visual keywords and semantic context. In: CIVR '07 proceedings of the 6th ACM international conference on image and video retrieval, pp 162---169

[17]

Alvarez S, Vanrell M (2012) Texton theory revisited: a bag-of-words approach to combine textons. Pattern Recognit 45:4312---4325

Digital Library

[18]

Chen T, Yap KH, Chau LP (2011) From universal bag-of-words to adaptive bag-of-phrases for mobile scene recognition. 2011 18th IEEE international conference on image processing (ICIP), pp 825---828

[19]

Albatal R, Mulhem P, Chiaramella Y (2010) Visual phrases for automatic images annotation. In: 2010 International workshop on content-based multimedia indexing (CBMI), pp 1---6

[20]

Albatal R, Mulhem P, Chiaramella Y (2011) A new ROI grouping schema for automatic image annotation. In: 2011 IEEE international conference on multimedia and expo (ICME), pp 1---6

Digital Library

[21]

Zhang YM, Jia ZY, Chen T (2011) Image retrieval with geometry-preserving visual phrases. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR), pp 809---816

[22]

Wang FY, Zhang SW, Li HP, Zhang NG (2012) Image retrieval using multiple orders of geometry-preserving visual phrases. In: 2012 International conference on image analysis and signal processing (IASP), pp 1---5

[23]

Zhang SL, Tian Q, Hua G (2009) Descriptive visual words and visual phrases for image applications. In: MM '09 proceedings of the 17th ACM international conference on Multimedia, pp 75---84

[24]

Yang C, Dong M, Fotouhi F (2005) Region based image annotation through multiple instance learning. In: Multimedia'05 proceedings of the 13th annual ACM international conference on multimedia, pp 435---438

[25]

Veksler O, Boykov Y, Mehrani P (2010) Superpixels and supervoxels in an energy optimization framework. In: European conference on computer vision (ECCV), pp 211---224

[26]

Huang QX, Han M, Wu B, Ioffe S (2011) A hierarchical conditional random field model for labeling and segmenting images of street scenes. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR), pp 1953---1960

[27]

Zhang DS, Islam MM, Lu GJ (2012) A review on automatic image annotation techniques. Pattern Recognit 45(1):346---362

Digital Library

[28]

Goh KS, Chang EY, Li BT (2005) Using one-class and two-class SVMs for multiclass image annotation. IEEE Trans Knowl Data Eng 17(10):1333---1346

Digital Library

[29]

Qi XJ, Han YT (2007) Incorporating multiple SVMs for automatic image annotation. Pattern Recognit 40(2):728---741

Digital Library

[30]

Shi R, Feng H, Chua TS, Lee CH (2004) An adaptive image content representation and segmentation approach to automatic image annotation. In: International conference on image and video retrieval, pp 545---554

[31]

Tao DP, Jin LW, Liu WF, Li XL (2013) Hessian regularized support vector machines for mobile image annotation on the cloud. IEEE Trans Multimed 15(4):833---844

Digital Library

[32]

Park SB, Lee JW, Kim SK (2004) Content-based image classification using a neural network. Pattern Recognit Lett 25(3):287---300

Digital Library

[33]

Kim S, Park S, Kim M (2004) Image classification into object/non-object classes. In: International conference on image and video retrieval, Dublin, Ireland, pp 393---400

[34]

Frate FD, Pacifici F, Schiavon G, Solimini C (2007) Use of neural networks for automatic classification from high-resolution images. IEEE Trans Geosci Remote Sens 45(4):800---809

[35]

Su JH, Chou CL, Lin CY, Tseng VS (2011) Effective semantic annotation by image-to-concept distribution model. IEEE Trans Multimed 13(3):530---538

Digital Library

[36]

Bao BK, Li T, Yan SC (2012) Hidden-concept driven multilabel image annotation and label ranking, multimedia. IEEE Trans Multimed 14(1):199---210

Digital Library

[37]

Wang Y, Mei T, Gong SG, Hua XS (2009) Combining global, regional and contextual features for automatic image annotation. Pattern Recognit 42(2):259---266

Digital Library

[38]

Hu JW, Lam KM (2013) An efficient two-stage framework for image annotation. Pattern Recognit 46(3):936---947

Digital Library

[39]

Blanchart P, Datcu M (2010) A semi-supervised algorithm for auto-annotation and unknown structures discovery in satellite image databases. IEEE J Sel Top Appl Earth Obs Remote Sens 3(4):698---717

[40]

Liu J, Li MJ, Liu QS, Lu HQ, Ma SD (2009) Image annotation via graph learning. Pattern Recognit 42(2):218---228

Digital Library

[41]

Chen ZH, Fu H, Chi ZR, Feng DD (2012) An adaptive recognition model for image annotation. IEEE Trans Syst Man Cybern Part C Appl Rev 42(6):1120---1127

Digital Library

[42]

Lafferty J, McCallum A, Pereira F (2001) Conditional random fields: probabilistic models for segmenting and labelling sequence data. In: The 18th international conference on machine learning 2001 (ICML 2001), pp 282---289

[43]

He X, Zemel RS, Carreira-Perpiñán MÁ (2004) Multiscale conditional random fields for image labeling. In: Proceedings of the 2004 IEEE computer society conference on computer Vision and pattern recognition, 2004 (CVPR 2004), vol 2, pp 695---702

[44]

Liu T, Sun J, Zheng NN, Tang XO, Shum HY (2007) Learning to detect a salient object. In: IEEE conference on computer vision and pattern recognition (2007 CVPR'07), pp 1---8

[45]

Mensink T, Verbeek J, Csurka G (2012) Tree-structured CRF models for interactive image labeling. IEEE Trans Pattern Anal Mach Intell 35(2):476---489

[46]

Zhong P, Wang RS (2010) Learning conditional random fields for classification of hyperspectral images. IEEE Trans Image Process 19(7):1890---1907

Digital Library

[47]

Zhang J, Hu WW (2013) Multi-label image annotation based on multi-model. In: ACM international conference on ubiguitous information management and communication (ACM ICUIMC 2013), Kota Kinabalu, Malaysia, pp 17---19

Cited By

Zhang JFeng SLi DGao YChen ZYuan Y(2017)Image retrieval using the extended salient regionInformation Sciences: an International Journal10.1016/j.ins.2017.03.005399:C(154-182)Online publication date: 1-Aug-2017
https://dl.acm.org/doi/10.1016/j.ins.2017.03.005
Zhang JGao YFeng SYuan YLee C(undefined)Automatic image region annotation through segmentation based visual semantic analysis and discriminative classification2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2016.7472018(1956-1960)
https://dl.acm.org/doi/10.1109/ICASSP.2016.7472018

A novel image annotation model based on content representation with multi-layer segmentation
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning

Recommendations

A Novel Brain Tumor Segmentation from Multi-Modality MRI via A Level-Set-Based Model

Segmentation of brain tumor from magnetic resonance imaging is a challenging and time-consuming task due to the unpredictable appearance of tumor tissue in practical applications. In this paper we propose a novel level-set-based model for tumor ...
A novel multiphase active contour model for inhomogeneous image segmentation

The problem of image segmentation has been investigated with a focus on inhomogeneous multiphase image segmentation. Intensity inhomogeneity is an undesired phenomenon that represents the main obstacle for magnetic resonance (MR) and natural images ...
Segmentation of brain tissues using a 3-D multi-layer Hidden Markov Model

To compensate for bias field inhomogeneity and reduce noise, we incorporate domain-based knowledge and spatial information into a brain segmentation algorithm by proposing a new multi-layer Hidden Markov model. Brain tissues include Gray Matter (GM), ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Neural Computing and Applications

Neural Computing and Applications Volume 26, Issue 6

August 2015

242 pages

ISSN:0941-0643

EISSN:1433-3058

Issue’s Table of Contents

Copyright © Copyright © 2015 The Natural Computing Applications Forum.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 01 August 2015

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang JFeng SLi DGao YChen ZYuan Y(2017)Image retrieval using the extended salient regionInformation Sciences: an International Journal10.1016/j.ins.2017.03.005399:C(154-182)Online publication date: 1-Aug-2017
https://dl.acm.org/doi/10.1016/j.ins.2017.03.005
Zhang JGao YFeng SYuan YLee C(undefined)Automatic image region annotation through segmentation based visual semantic analysis and discriminative classification2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2016.7472018(1956-1960)
https://dl.acm.org/doi/10.1109/ICASSP.2016.7472018

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents