article

Attention-driven image interpretation with application to image retrieval

Authors:

Dagan FengAuthors Info & Claims

Pattern Recognition, Volume 39, Issue 9

Pages 1604 - 1621

https://doi.org/10.1016/j.patcog.2005.12.015

Published: 01 September 2006 Publication History

Abstract

Visual attention, a selective procedure of human's early vision, plays a very important role for humans to understand a scene by intuitively emphasizing some focused regions/objects. Being aware of this, we propose an attention-driven image interpretation method that pops out visual attentive objects from an image iteratively by maximizing a global attention function. In this method, an image can be interpreted as containing several perceptually attended objects as well as a background, where each object has an attention value. The attention values of attentive objectives are then mapped to importance factors so as to facilitate the subsequent image retrieval. An attention-driven matching algorithm is proposed in this paper based on a retrieval strategy emphasizing attended objects. Experiments on 7376 Hemera color images annotated by keywords show that the retrieval results from our attention-driven approach compare favorably with conventional methods, especially when the important objects are seriously concealed by the irrelevant background.

References

[1]

Long, F., Zhang, H. and Feng, D., Fundamentals of content-based image retrieval. In: Feng, D., Stu, W.C., Zhang, H. (Eds.), Multimedia Information Retrieval and Management, Springer, Berlin. pp. 1-26.

[2]

Banerjee, M. and Kundu, M., Edge based features for content based image retrieval. Pattern Recognition. v36. 2649-2661.

[3]

Lima, J. and Jinb, J., Combining intra-image and inter-class semantics for consumer image retrieval. Pattern Recognition. v38. 847-864.

[4]

H. Feng, T. Chua, A learning-based approach for annotating large on-line image collection, Proceedings of the 10th International Conference on Multimedia Modeling, 2004, 5-7 January 2004, pp. 249-256.

[5]

Fan, J., Gao, Y., Luo, H. and Xu, G., Statistical modeling and conceptualization of natural images. Pattern Recognition. v38. 865-885.

[6]

Li, J. and Wang, J., Automatic linguistic indexing of pictures by a statistical modeling approach. IEEE Trans. Pattern Anal. Mach. Intell. v25 i10. 1-14.

[7]

Deng, Y. and Manjunath, B., Unsupervised segmentation of color-texture regions in images and video. IEEE Trans. Pattern Anal. Mach. Intell. v23 i8. 800-810.

[8]

Carson, C., Belongie, S., Greenspan, H. and Malik, J., Blobworld: image segmentation using expectation-maximization and its application to image querying. IEEE Trans. Pattern Anal. Mach. Intell. v24 i8. 1026-1038.

[9]

Wang, J., Li, J. and Wiederhold, G., Simplicity: semantics-sensitive integrated matching for picture libraries. IEEE Trans. Pattern Anal. Mach. Intell. v23 i9. 947-963.

[10]

Hansen, M. and Carstensen, J., Density-based retrieval from high-similarity image databases. Pattern Recognition. v37. 2155-2164.

[11]

Ko, B. and Byun, H., FRIP: a region-based image retrieval tool using automatic image segmentation and stepwise boolean and matching. IEEE Trans. Multimedia. v7 i1. 105-113.

[12]

Jing, F., Li, M., Zhang, H. and Zhang, B., An efficient and effective region-based image retrieval framework. IEEE Trans. Image Process. v13 i5. 699-709.

[13]

Y. Rubner, C. Tomasi, L. Guibas, A metric for distributions with applications to image databases, Sixth International Conference on Computer Vision, 1998, 4-7 January 1998, pp. 59-66.

[14]

C.S. Lee, W.M., H. Zhang, Information embedding based on user's relevance feedback for image retrieval, Proceedings of the SPIE International Conference on Multimedia Storage and Archiving Systems, vol. IV, Boston, 19-22 September 1999.

[15]

Giacinto, G. and Roli, F., Bayesian relevance feedback for content-based image retrieval. Pattern Recognition. v37. 1499-1508.

[16]

Y. Rui, T.S. Huang, A novel relevance feedback technique in image retrieval, ACM Multimedia '99(Part 2), 10/99, Orlando, FL, USA, pp. 67-70.

[17]

Hennig, C. and Lateckic, L., The choice of vantage objects for image retrieval. Pattern Recognition. v36. 2187-2196.

[18]

Lu, Y., Zhang, H., Liu, W. and Hu, C., Joint semantics and feature based image retrieval using relevance feedback. IEEE Trans. Multimedia. v5 i3. 339-347.

[19]

Theeuwes, J., Visual selective attention: a theoretical analysis. Acta Psychologica. v83. 93-154.

[20]

S. Steinman, B. Steinman, Chapter 14: computational models of visual attention, in: G.K. Hung, K.J. Ciuffreda (Eds.), Models of the Visual System, 2002, pp. 521-563.

[21]

J. Wolfe, 9: The level of attention: mediating between the stimulus and perception, in: L. Harris, M. Jenkin (Eds.), Levels of Perception, 2002, pp. 169-191.

[22]

Joseph, J., Chun, M. and Nakayama, K., Attentional requirements in a 'preattentive' feature search task. Nature. v387. 805-807.

[23]

Niebur, E. and Koch, C., Computational architectures for attention. In: Parasuraman, R. (Ed.), The Attention Brain, MIT Press, Cambridge, MA.

[24]

Wolfe, J. and Horowitz, T., What attributes guide the deployment of visual attention and how do they do it?. Nature Rev. Neurosci. v5. 1-7.

[25]

Itti, L., Koch, C. and Niebur, E., A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. v20 i11. 1254-1259.

[26]

Itti, L. and Koch, C., Computational modeling of visual attention. Nature Rev. Neurosci. v2. 194-203.

[27]

U. Rutishauser, D. Walther, C. Koch, P. Perona, Is bottom-up attention useful for object recognition?, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition(CVPR'04), vol. 2, July 2004, pp. 37-44.

[28]

Dickinson, S., Christensen, H., Tsotsos, J. and Olofsson, G., Active object recognition integrating attention and viewpoint control. Comput. Vision Image Understand. v67 i3. 239-260.

[29]

Tsotsos, J., Culhane, S., Wai, W., Lai, Y., Davis, N. and Nuflo, F., Modeling visual attention via selective tuning. Artif. Intell. v78. 507-545.

[30]

Deco, G. and Schurmann, B., A hierarchical neural system with attentional top-down enhancement of the spatial resolution for object recognition. Vision Res. v40. 2845-2859.

[31]

J. Han, M. Li, H. Zhang, L. Guo, Automatic attention object extraction from images, International Conference on Image Processing, vol. 2, 14-17 September 2003, pp. II-403-406.

[32]

J. Hare, P. Lewis, Salient regions for query by image content, Third International Conference on Image and Video Retrieval, Dublin, Ireland, July 21-23, 2004, pp. 317-325.

[33]

Paschos, G., Perceptually uniform color spaces for color texture analysis: an empirical evaluation. IEEE Trans. Image Process. v10 i6.

[34]

Castleman, K., Digital Image Processing. Prentice-Hall, Englewood Cliffs, NJ.

Cited By

Alcaín EMuñoz ASchiavi EMontemayor A(2021)A non-smooth non-local variational approach to saliency detection in real timeJournal of Real-Time Image Processing10.1007/s11554-020-01016-418:3(739-750)Online publication date: 1-Jun-2021
https://dl.acm.org/doi/10.1007/s11554-020-01016-4
K. SP. S(2021)Dynamic Mode Decomposition based salient edge/region features for content based image retrievalMultimedia Tools and Applications10.1007/s11042-020-10315-880:10(15937-15958)Online publication date: 1-Apr-2021
https://dl.acm.org/doi/10.1007/s11042-020-10315-8
Wu YLiu HYuan JZhang Q(2018)Is visual saliency useful for content-based image retrieval?Multimedia Tools and Applications10.1007/s11042-017-5001-677:11(13983-14006)Online publication date: 1-Jun-2018
https://dl.acm.org/doi/10.1007/s11042-017-5001-6
Show More Cited By

Recommendations

Attention-driven image interpretation, annotation and retrieval
An efficient algorithm for attention-driven image interpretation from segments

In the attention-driven image interpretation process, an image is interpreted as containing several perceptually attended objects as well as the background. The process benefits greatly a content-based image retrieval task with attentively important ...
Image quality assessment metrics combining structural similarity and image fidelity with visual attention

Image quality assessment has a great importance in several image processing applications. Recently, various objective image quality metrics have been proposed in order to predict human visual perception. In this paper, novel image quality metrics, S-...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Pattern Recognition

Pattern Recognition Volume 39, Issue 9

September, 2006

259 pages

ISSN:0031-3203

Issue’s Table of Contents

Copyright © Pattern Recognition Society © 2006.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 01 September 2006

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 20 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Alcaín EMuñoz ASchiavi EMontemayor A(2021)A non-smooth non-local variational approach to saliency detection in real timeJournal of Real-Time Image Processing10.1007/s11554-020-01016-418:3(739-750)Online publication date: 1-Jun-2021
https://dl.acm.org/doi/10.1007/s11554-020-01016-4
K. SP. S(2021)Dynamic Mode Decomposition based salient edge/region features for content based image retrievalMultimedia Tools and Applications10.1007/s11042-020-10315-880:10(15937-15958)Online publication date: 1-Apr-2021
https://dl.acm.org/doi/10.1007/s11042-020-10315-8
Wu YLiu HYuan JZhang Q(2018)Is visual saliency useful for content-based image retrieval?Multimedia Tools and Applications10.1007/s11042-017-5001-677:11(13983-14006)Online publication date: 1-Jun-2018
https://dl.acm.org/doi/10.1007/s11042-017-5001-6
Tan WYan B(2017)Salient object detection via multiple saliency weightsMultimedia Tools and Applications10.1007/s11042-017-4725-776:23(25091-25107)Online publication date: 1-Dec-2017
https://dl.acm.org/doi/10.1007/s11042-017-4725-7
Filali IAllili MBenblidia N(2016)Multi-scale salient object detection using graph ranking and global-local saliency refinementImage Communication10.1016/j.image.2016.07.00747:C(380-401)Online publication date: 1-Sep-2016
https://dl.acm.org/doi/10.1016/j.image.2016.07.007
Zhong MZhao XZou XWang JWang W(2014)Markov chain based computational visual attention model that learns from eye tracking dataPattern Recognition Letters10.5555/2943796.294391449:C(1-10)Online publication date: 1-Nov-2014
https://dl.acm.org/doi/10.5555/2943796.2943914
Wang ZGuan GQiu YZhuo LFeng D(2013)Semantic context based refinement for news video annotationMultimedia Tools and Applications10.1007/s11042-012-1060-x67:3(607-627)Online publication date: 1-Dec-2013
https://dl.acm.org/doi/10.1007/s11042-012-1060-x
Hou GZhang KZhang XKong JZhang M(2013)A novel image retrieval method based on mutual information descriptorsProceedings of the 9th international conference on Intelligent Computing Theories and Technology10.1007/978-3-642-39482-9_50(435-442)Online publication date: 28-Jul-2013
https://dl.acm.org/doi/10.1007/978-3-642-39482-9_50
Hu YRen ZRajan DChia L(2010)Salient region detection by jointly modeling distinctness and redundancy of image contentProceedings of the 10th Asian conference on Computer vision - Volume Part II10.5555/1965992.1966033(515-526)Online publication date: 8-Nov-2010
https://dl.acm.org/doi/10.5555/1965992.1966033
Zhao XXu CChi ZFeng D(2010)A novel shape-based image classification method by featuring radius histogram of dilating discs filled into regular and irregular shapesProceedings of the 17th international conference on Neural information processing: theory and algorithms - Volume Part I10.5555/1939659.1939691(239-246)Online publication date: 22-Nov-2010
https://dl.acm.org/doi/10.5555/1939659.1939691
Show More Cited By

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents