research-article

Harvesting visual concepts for image search with complex queries

Authors:

Tat-Seng ChuaAuthors Info & Claims

MM '12: Proceedings of the 20th ACM international conference on Multimedia

Pages 59 - 68

https://doi.org/10.1145/2393347.2393363

Published: 29 October 2012 Publication History

Abstract

The use of image reranking to boost retrieval performance has been found to be successful for simple queries. It is, however, less effective for complex queries due to the widened semantic gap. This paper presents a scheme to enhance web image reranking for complex queries by fully exploring the information from simple visual concepts. Given a complex query, our scheme first detects the noun-phrase based visual concepts and crawls their top ranked images from popular image search engines. Next, it constructs a heterogeneous probabilistic network to model the relatedness between the complex query and each of its crawled images. The network seamlessly integrates three layers of relationships, i.e., the semantic-level, cross-modality level as well as visual-level. These mutually reinforced layers are established among the complex query and its involved visual concepts, by harnessing the contents of images and their associated textual cues. Based on the derived relevance scores, a new ranking list is generated. Extensive evaluations on a real-world dataset demonstrate that our model is able to characterize the complex queries well and achieve promising performance as compared to the state-of-the-art methods. Based on the proposed scheme, we introduce two applications: photo-based question answering and textual news visualization. Comprehensive experiments well validate the proposed scheme.

References

[1]

Hitwise. see http://weblogs.hitwise.com/alan-long/2009/11/searches_getting_longer.html.

[2]

R. Aly, D. Hiemstra, and R. Ordelman. Building detectors to support searches on combined semantic concepts. In MIR Workshop, 2007.

[3]

J. Bai, D. Song, P. Bruza, J.-Y. Nie, and G. Cao. Query expansion using term relationships in language models for information retrieval. In CIKM, 2005.

Digital Library

[4]

N. Balasubramanian, G. Kumaran, and V. Carvalho. Exploring reductions for long web queries. In SIGIR, 2010.

Digital Library

[5]

M. Bendersky and W. B. Croft. Discovering key concepts in verbose queries. In SIGIR, 2008.

Digital Library

[6]

M. Bendersky, D. Metzler, and W. B. Croft. Learning concept importance using a weighted dependence model. In WSDM, 2010.

Digital Library

[7]

M. Bendersky, D. Metzler, and W. B. Croft. Parameterized concept weighting in verbose queries. In SIGIR, 2011.

Digital Library

[8]

X. Chen, J. Yuan, L. Nie, Z.-j. Zha, S. Yan, and T.-s. Chua. Known-item search by nus. In NIST TRECVID, 2010.

[9]

T.-S. Chua, R. Hong, G. Li, and J. Tang. From text question-answering to multimedia qa on web-scale media resources. In LS-MMRM, 2009.

Digital Library

[10]

R. L. Cilibrasi and P. M. B. Vitanyi. The google similarity distance. TKDE, 2007.

Digital Library

[11]

D. Delgado, J. Magalhaes, and N. Correia. Assisted news reading with automated illustration. In MM, 2010.

Digital Library

[12]

W. H. Hsu, L. S. Kennedy, and S.-F. Chang. Video search reranking through random walk over document-level context graph. In MM, 2007.

Digital Library

[13]

G. Kumaran and J. Allan. A case for shorter queries, and helping users create them. In NAACL-HLT, 2007.

[14]

G. Kumaran and J. Allan. Effective and efficient user interaction for long queries. In SIGIR, 2008.

Digital Library

[15]

M. Lease, J. Allan, and W. B. Croft. Regression rank: Learning to meet the opportunity of descriptive queries. In ECIR, 2009.

Digital Library

[16]

X. Li, C. G. M. Snoek, M. Worring, and A. W. M. Smeulders. Harvesting social images for bi-concept search. TMM, 2012.

Digital Library

[17]

Z. Li, M. Wang, J. Liu, C. Xu, and H. Lu. News contextualization with geographic and visual information. In MM, 2011.

Digital Library

[18]

D. Liu, X.-S. Hua, M. Wang, and H.-J. Zhang. Image retagging. In MM, 2010.

Digital Library

[19]

D. Liu, X.-S. Hua, L. Yang, M. Wang, and H.-J. Zhang. Tag ranking. In WWW, 2009.

Digital Library

[20]

Y. Liu, T. Mei, X.-S. Hua, J. Tang, X. Wu, and S. Li. Learning to video search rerank via pseudo preference feedback. In ICME, 2008.

[21]

M. C. Mihai Surdeanu and H. Zaragoza. Learning to rank answers on large online qa collections. In ACL, 2008.

[22]

N. Morioka and J. Wang. Robust visual reranking via sparsity and ranking constraints. In MM, 2011.

Digital Library

[23]

A. P. Natsev, M. R. Naphade, and J. Te\vsiĆ. Learning the semantics of multimedia queries and concepts from a small number of examples. In MM, 2005.

Digital Library

[24]

L. Nie, M. Wang, Z.-j. Zha, G. Li, and T.-S. Chua. Multimedia answering: enriching text qa with media information. In SIGIR, 2011.

Digital Library

[25]

L. N. Nie, M. Wang, Z. Zha, and T.-S. Chua. Oracle in image search: A content-based approach to performance prediction. TOIS, 2012.

Digital Library

[26]

J. H. Park and W. B. Croft. Query term ranking based on dependency parsing of verbose queries. In SIGIR, 2010.

Digital Library

[27]

S. U. Pillai, T. Suel, and S. Cha. The Perron-Frobenius theorem: some of its applications. Signal Processing Magazine, IEEE, 2005.

[28]

H.-T. Pu. An analysis of failed queries for web image retrieval. Journal of Information Science, 2008.

Digital Library

[29]

M. Szummer and T. Jaakkola. Partially labeled classification with markov random walks. NIPS, 2002.

[30]

X. Tian, L. Yang, J. Wang, Y. Yang, X. Wu, and X.-S. Hua. Bayesian video search reranking. In MM, 2008.

Digital Library

[31]

A. Torralba, R. Fergus, and W. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. TPAMI, 2008.

Digital Library

[32]

M. Wang, K. Yang, X.-S. Hua, and H.-J. Zhang. Towards a relevant and diverse search of social images. TMM, 2010.

Digital Library

[33]

H. Xu, J. Wang, X.-S. Hua, and S. Li. Image search by concept map. In SIGIR, 2010.

Digital Library

[34]

R. Yan, E. Hauptmann, and R. Jin. Multimedia search with pseudo-relevance feedback. In ICVR, 2003.

Digital Library

[35]

J. Yuan, Z.-J. Zha, Y.-T. Zheng, W. Meng, X. Zhou, and T.-S. Chua. Utilizing related samples to enhance interactive concept-based video search. TMM, 2011.

Digital Library

[36]

J. Yuan, Z.-J. Zha, Y.-T. Zheng, M. Wang, X. Zhou, and T.-S. Chua. Learning concept bundles for video search with complex queries. In MM, 2011.

Digital Library

Cited By

Liu YWu ZLo SChen ZKe GYue C(2024)Data reweighting net for web fine-grained image classificationMultimedia Tools and Applications10.1007/s11042-024-18598-x83:33(79985-80005)Online publication date: 2-Mar-2024
https://doi.org/10.1007/s11042-024-18598-x
Zhu HWei YZhao YZhang CHuang S(2023)AMC: Adaptive Multi-expert Collaborative Network for Text-guided Image RetrievalACM Transactions on Multimedia Computing, Communications, and Applications10.1145/358470319:6(1-22)Online publication date: 20-Feb-2023
https://dl.acm.org/doi/10.1145/3584703
Zhang CLin GWang QShen FYao YTang Z(2023)Guided by Meta-Set: A Data-Driven Method for Fine-Grained Visual RecognitionIEEE Transactions on Multimedia10.1109/TMM.2022.318143925(4691-4703)Online publication date: 2023
https://doi.org/10.1109/TMM.2022.3181439
Show More Cited By

Index Terms

Harvesting visual concepts for image search with complex queries
1. Information systems
  1. Information retrieval
    1. Information retrieval query processing
    2. Retrieval models and ranking

Recommendations

Oracle in Image Search: A Content-Based Approach to Performance Prediction

This article studies a novel problem in image search. Given a text query and the image ranking list returned by an image search system, we propose an approach to automatically predict the search performance. We demonstrate that, in order to estimate the ...
Learning to rank images for complex queries in concept-based search
Abstract
Concept-based image search is an emerging search paradigm that utilizes a set of concepts as intermediate semantic descriptors of images to bridge the semantic gap. Typically, a user query is rather complex and cannot be well described ...
Rescue Tail Queries: Learning to Image Search Re-rank via Click-wise Multimodal Fusion
MM '14: Proceedings of the 22nd ACM international conference on Multimedia

Image search engines have achieved good performance for head (popular) queries by leveraging text information and user click data. However, there still remain a large number of tail (rare) queries with relatively unsatisfying search results, which are ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '12: Proceedings of the 20th ACM international conference on Multimedia

October 2012

1584 pages

ISBN:9781450310895

DOI:10.1145/2393347

General Chairs:
Noboru Babaguchi
Osaka University, Japan
,
Kiyoharu Aizawa
The University of Tokyo, Japan
,
John Smith
IBM, USA
,
Program Chairs:
Shin'ichi Satoh
National Institute of Informatics, Japan
,
Thomas Plagemann
University of Oslo, Norway
,
Xian-Sheng Hua
Microsoft, USA
,
Rong Yan
Facebook, USA

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 October 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '12

Sponsor:

SIGMM

MM '12: ACM Multimedia Conference

October 29 - November 2, 2012

Nara, Japan

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

123
Total Citations
View Citations
576
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)1

Reflects downloads up to 14 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu YWu ZLo SChen ZKe GYue C(2024)Data reweighting net for web fine-grained image classificationMultimedia Tools and Applications10.1007/s11042-024-18598-x83:33(79985-80005)Online publication date: 2-Mar-2024
https://doi.org/10.1007/s11042-024-18598-x
Zhu HWei YZhao YZhang CHuang S(2023)AMC: Adaptive Multi-expert Collaborative Network for Text-guided Image RetrievalACM Transactions on Multimedia Computing, Communications, and Applications10.1145/358470319:6(1-22)Online publication date: 20-Feb-2023
https://dl.acm.org/doi/10.1145/3584703
Zhang CLin GWang QShen FYao YTang Z(2023)Guided by Meta-Set: A Data-Driven Method for Fine-Grained Visual RecognitionIEEE Transactions on Multimedia10.1109/TMM.2022.318143925(4691-4703)Online publication date: 2023
https://doi.org/10.1109/TMM.2022.3181439
Sato RSelcuk Candan KLiu HAkoglu LLuna Dong XTang J(2022)Retrieving Black-box Optimal Images from External DatabasesProceedings of the Fifteenth ACM International Conference on Web Search and Data Mining10.1145/3488560.3498462(879-887)Online publication date: 11-Feb-2022
https://dl.acm.org/doi/10.1145/3488560.3498462
Zhang PLi YHuang ZXu X(2022)Aggregation-Based Graph Convolutional Hashing for Unsupervised Cross-Modal RetrievalIEEE Transactions on Multimedia10.1109/TMM.2021.305376624(466-479)Online publication date: 2022
https://doi.org/10.1109/TMM.2021.3053766
Yang XWang SDong JDong JWang MChua T(2022)Video Moment Retrieval With Cross-Modal Neural Architecture SearchIEEE Transactions on Image Processing10.1109/TIP.2022.314061131(1204-1216)Online publication date: 2022
https://doi.org/10.1109/TIP.2022.3140611
Wu YLiu XQin HXia KHu SMa YWang M(2021)Boosting Temporal Binary Coding for Large-Scale Video SearchIEEE Transactions on Multimedia10.1109/TMM.2020.297859323(353-364)Online publication date: 2021
https://doi.org/10.1109/TMM.2020.2978593
Liu XNie XZhou QNie LYin Y(2020)Model Optimization Boosting Framework for Linear Model Hash LearningIEEE Transactions on Image Processing10.1109/TIP.2020.297057729(4254-4268)Online publication date: 2020
https://doi.org/10.1109/TIP.2020.2970577
Gao JZhang WChen ZZhong F(2020)HDMFH: Hypergraph Based Discrete Matrix Factorization Hashing for Multimodal RetrievalICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP40776.2020.9054516(1923-1927)Online publication date: May-2020
https://doi.org/10.1109/ICASSP40776.2020.9054516
Ghose SDas ABhunia ARoy P(2020)Fractional Local Neighborhood Intensity Pattern for Image Retrieval using Genetic AlgorithmMultimedia Tools and Applications10.1007/s11042-020-08752-6Online publication date: 5-Mar-2020
https://doi.org/10.1007/s11042-020-08752-6
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents