Abstract
In this paper, we propose a multimodal query suggestion method for video search which can leverage multimodal processing to improve the quality of search results. When users type general or ambiguous textual queries, our system MQSS provides keyword suggestions and representative image examples in an easy-to-use dropdown manner which can help users specify their search intent more precisely and effortlessly. It is a powerful complement to initial queries. After the queries are formulated as multimodal query (i.e., text, image), the new queries are input to individual search models, such as text-based, concept-based and visual example-based search model. Then we apply multimodal fusion method to aggregate the above-mentioned several search results. The effectiveness of MQSS is demonstrated by evaluations over a web video data set.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Amir A, Argillandery J, Campbell M et al (2005) IBM research TRECVID-2005 video retrieval system. In: TRECVID workshop, Washington DC
Baeza-Yates R, Ribeiro-Neto B (1999) Modern information retrieval. Addison Wesley, Boston
Baeza-Yates R, Hurtado C, Mendoza M (2004) Query recommendation using query logs in search engines. In: EDBT, 2004
Beeferman D, Berger A (2000) Agglomerative clustering of a search engine query log. In: Proceedings of ACM SIGKDD ’00, pp 407–416
Campbell M, Haubold A et al (2006) IBM research TRECVID-2006 video retrieval system. In: TRECVID 2006
Carpineto C, de Mori R, Romano G, Bigi B (2001) An information-theoretic approach to automatic query expansion. ACM TOIS 19(1):1–27
Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas influence, and trends of the new age. In: ACM computing surveys
Devijver PA, Kittler J (1982) Pattern recognition: a statistical approach. Prentice-Hall, London
Donald KM, Smeaton AF (2005) A comparison of score, rank and probability-based fusion methods for video shot retrieval. In: Proceedings of CIVR ’05
Flickr APIs. http://www.flickr.com/services/api/
Frey B, Dueck D (2007) Clustering by passing messages between data points. Science 315 (5814):972
Gao W, Niu C, Nie J-Y, Zhou M, Hu J, Wong K-F, Hon H-W (2007) Cross-lingual query suggestion using query logs of different languages. In: Proceedings of ACM SIGIR’07, pp 463–470
Google Search. http://www.google.com/
Hauptmann AG, Chen M-Y, Christel M, Das D, Lin W-H, Yan R, Yang J, Backfried G, Wu X (2006) Multi-lingual broadcast news retrieval. In: TRECVID 2006
Hauptmann AG, Lin W-H, Yan R et al (2006) Extreme video retrieval: joint maximization of human and computer performance. In: Proceedings of ACM multimedia ’06, Santa Barbara, pp 385–394
Hsu WH, Kennedy LS, Chang SF (2006) Video search reranking via information bottleneck principle. In: Proceedings of ACM multimedia ’06, Santa Barbara, pp 35–44
Hua X-S, Mei T, Lai W, Wang M, Tang J, Qi G-J, Li L, Gu Z (2006) Microsoft research Asia TRECVID 2006: high-level feature extraction and rushes exploitation. In: TREC video retrieval evaluation online proceedings (TRECVID)
Iyengar G, Duygulu P, Feng S et al (2005) Joint visual-text modeling for automatic retrieval of multimedia documents. In: Proceedings of ACM multimedia ’05, Singapore
Kennedy L, Natsev A, Chang S-F (2005) Automatic discovery of query class dependent models for multimodal search. In: Proceedings of ACM Multimedia ’05, Singapore, pp 882–891
Kennedy L, Chang S-F, Natsev A (2008) Query-adaptive fusion for multimodal search. In: Proceedings of the IEEE, vol 96, no 4, pp 567–588
Lam-Adesina AM, Jones GJF (2001) Applying summarization techniques for term selection in relevance feedback. In: Proceedings of ACM SIGIR’01
Liu J, Lai W, Hua X-S, Huang Y, Li S (2007) Video search re-ranking via multi-graph propagation. In: Proceedings of ACM multimedia ’07, Augsburg, pp 208–217
Ribeiro-Neto B, Cristo M, Golgher P, de Moura E (2005) Impedance coupling in content-targeted advertising, In: Proceedings of ACM SIGIR ’05, Salvador, Brazil, pp 496–503
Tian X, Yang L, Wang J et al (2008) Bayesian video search reranking. In: Proceedings of ACM multimedia ’08, Vancouver
Vapnik VN (1995) Statistical learning theory. Wiley, New York
Wang M, Hua X-S, Song Y et al (2006) Automatic video annotation by semi-supervised learning with Kernel density estimation. In: Proceedings of ACM multimedia ’06, Santa Barbara, pp 967–976
Wen J-R, Nie J-Y, Zhang H-J (2003) Clustering user queries of a search engine. In: Proceedings of WWW ’03, pp 162–168
Wu L, Hua X-S, Yu N, Ma W-Y, Li S (2007) Flickr distance, In: Proceedings of ACM multimedia ’07, Augsburg, pp 31–40
Xu J, Croft WB (1996) Query expansion using local and global document analysis. In: Proceedings of ACM SIGIR’96
Yahoo! Search. http://search.yahoo.com/
Yu S, Cai D, Wen J-R, Ma W-Y (2003) Improving pseudo-relevance feedback in web information retrieval using web page segmentation. In: Proceedings of WWW ’03, pp 11–18
Acknowledgements
The authors would like to gratefully acknowledge Dr. Xian-Sheng Hua, Dr. Tao Mei, Dr. Zheng-jun Zha, Dr. Wei Lai, Dr. Meng Wang and Linjun Yang for their thoughtful brainstorming and constructive suggestions on this work. The research was supported by National 973 Program of China (Grant No.2005CB321901).
Author information
Authors and Affiliations
Corresponding author
Additional information
Jing Li and Lusong Li contributed equally to this work.
Rights and permissions
About this article
Cite this article
Li, L., Li, J. MQSS: multimodal query suggestion and searching for video search. Multimed Tools Appl 54, 55–68 (2011). https://doi.org/10.1007/s11042-010-0540-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-010-0540-0