research-article

Online reranking via ordinal informative concepts for context fusion in concept detection and video search

Authors:

Winston H. Hsu,

Homer H. ChenAuthors Info & Claims

IEEE Transactions on Circuits and Systems for Video Technology, Volume 19, Issue 12

Pages 1880 - 1890

https://doi.org/10.1109/TCSVT.2009.2026978

Published: 01 December 2009 Publication History

Abstract

To exploit the co-occurrence patterns of semantic concepts while keeping the simplicity of context fusion, a novel reranking approach is proposed in this paper. The approach, called ordinal reranking, adjusts the ranking of an initial search (or detection) list based on the co-occurrence patterns obtained by using ranking functions such as ListNet. Ranking functions are by nature more effective than classification-based reranking methods in mining ordinal relationships. In addition, the ordinal reranking is free of the ad hoc thresholding for noisy binary labels and requires no extra offline learning or training data. To select informative concepts for reranking, we also propose a new concept selection measurement, wc-tf-idf, which considers the underlying ordinal information of ranking lists and is thus more effective than the feature selection algorithms for classification. Being largely unsupervised, the reranking approach to context fusion can be applied equally well to concept detection and video search. While being extremely efficient, ordinal reranking outperforms existing methods by up to 40% in mean average precision (MAP) for the baseline text-based search and 12% for the baseline concept detection over TRECVID 2005 video search and concept detection benchmark.

References

[1]

National Institute of Standards and Technology. Text Retrieval Conference Video Retrieval Evaluation {Online}. Available: http://www-nlpir.nist.gov/projects/trecvid

[2]

Y.-H. Yang, P.-T. Wu, C.-W. Lee, K.-H. Lin, W. H. Hsu, and H.-H. Chen, "ContextSeer: Context search and recommendation at query time for shared consumer photos," in Proc. Assoc. Comput. Mach. Multimedia, 2008, pp. 199-208.

[3]

M. Naphade et al., "Large-scale concept ontology for multimedia," IEEE Multimedia Mag., vol. 13, no. 3, pp. 86-91, Jul.-Sep. 2006.

Digital Library

[4]

A. Yanagawa et al., "Columbia University's baseline detectors for 374 LSCOM semantic visual concepts," Columbia Univ., New York, NY, ADVENT Tech. Rep. #222-2006-8, 2007.

[5]

W. H. Hsu, L. Kennedy, and S.-F. Chang "Video search reranking through random walk over document-level context graph," in Proc. Assoc. Comput. Mach. Multimedia, 2007, pp. 971-980.

[6]

W. H. Hsu, L. Kennedy, and S.-F. Chang, "Video search reranking via information bottleneck principle," in Proc. Assoc. Comput. Mach. Multimedia, 2006, pp. 35-44.

[7]

L. Kennedy and S.-F. Chang, "A reranking approach for context-based concept fusion in video indexing and retrieval," in Proc. Assoc. Comput. Mach. Int. Conf. Image Video Retrieval, 2007, pp. 333-340.

[8]

R. Yan, R. Jing, and A. Hauptmann, "Multimedia search with pseudorelevance feedback," in Proc. Assoc. Comput. Mach. Int. Conf. Image Video Retrieval, 2003, pp. 238-247.

[9]

J. Battelle, The Search: How Google and Its Rivals Rewrote the Rules of Business and Transformed Our Culture. New York: Portfolio Trade, 2005.

[10]

X. Li, D. Wang, J. Li, and B. Zhang, "Video search in concept subspace: A text like paradigm," in Proc. Assoc. Comput. Mach. Int. Conf. Image Video Retrieval, 2007, pp. 603-610.

Digital Library

[11]

R. Herbrich, T. Graepel, and K. Obermayer, "Support vector learning for ordinal regression," in Proc. Int. Conf. Artif. Neural Netw., 1999, pp. 97-102.

[12]

T. Joachims, "Optimizing search engines using clickthrough data," in Proc. Assoc. Comput. Mach. Special Interest Group Knowl. Discovery Data Mining, 2002, pp. 133-142.

[13]

Y. Cao, J. Xu, T.-Y. Liu, H. Li, Y. Huang, and H.-W. Hon, "Adapting ranking SVM to document retrieval," in Proc. Assoc. Comput. Mach. Special Interest Group Inform. Retrieval, 2006, pp. 186-193.

[14]

A. Aizawa, "An information-theoretic perspective of tf-idf measures," Inform. Process. Manage., vol. 39, pp. 45-65, 2003.

Digital Library

[15]

W. Jiang, S.-F. Chang, and A. C. Loui, "Context-based concept fusion with boosted conditional random fields," in Proc. IEEE Int. Conf. Acoustics Speech Signal Process., vol. 1. 2007, pp. 949-952.

[16]

C. G. Snoek et al., "The MediaMill TRECVID 2006 semantic video search engine," in Proc. Natl. Inst. Standards Technol. Text REtrieval Conf. Video Retrieval Eval. Workshop, 2006.

[17]

S.-F. Chang et al., "Columbia University TRECVID 2005 video search and high-level feature extraction," in Proc. Natl. Inst. Standards Technol. Text REtrieval Conf. Video Retrieval Eval. Workshop, 2005.

[18]

M. Campbell et al., "IBM Research TRECVID 2006 video retrieval system," in Proc. Natl. Inst. Standards Technol. Text REtrieval Conf. Video Retrieval Eval. Workshop, 2006.

[19]

T.-S. Chua et al., "TRECVID 2004 search and feature extraction task by NUS PRIS," in Proc. Natl. Inst. Standards Technol. Text REtrieval Conf. Video Retrieval Eval. Workshop, 2004.

[20]

Z. Cao, T. Qin, T.-Y. Liu, M.-F. Tsai, and H. Li, "Learning to rank: From pairwise approach to listwise approach," in Proc. IEEE Int. Conf. Machine Learning, 2007, pp. 129-136.

[21]

J. Smith, M. Naphade, and A. Natsev, "Multimedia semantic indexing using model vectors," in Proc. IEEE Int. Conf. Multimedia Expo, vol. 2. Jul. 2003, pp. 445-448.

[22]

C. Snoek, M. Worring, D. Koelma, and A. Smeulders, "Learned lexicon-driven interactive video retrieval," in Proc. Assoc. Comput. Mach. Int. Conf. Image Video Retrieval, 2006, pp. 11-20.

Digital Library

[23]

W. Jiang, S.-F. Chang, and A. C. Loui, "Active context-based concept fusion with partial user labels," in Proc. IEEE Int. Conf. Image Process., Oct. 2006, pp. 2917-2920.

[24]

A. G. Hauptmann et al., "Multilingual broadcast news retrieval," in Proc. Natl. Inst. Standards Technol. Text REtrieval Conf. Video Retrieval Eval. Workshop, 2006, pp. 1-12.

[25]

A. Haubold, A. Natsev, and M. Naphade, "Semantic multimedia retrieval using lexical query expansion and model-based reranking," in Proc. IEEE Int. Conf. Multimedia Expo, Jul. 2006, pp. 1761-1764.

[26]

S.-Y. Neo, J. Zhao, M.-Y. Kan, and T.-S. Chua, "Video retrieval using high-level features: Exploiting query matching and confidence-based weighting," in Proc. Assoc. Comput. Mach. Int. Conf. Image Video Retrieval, 2006, pp. 143-152.

Digital Library

[27]

A. Natsev, A. Haubold, J. Tesic, L. Xie, and R. Yan, "Semantic concept-based query expansion and reranking for multimedia retrieval," in Proc. Assoc. Comput. Mach. Multimedia, 2007, pp. 991-1000.

[28]

Y. Yang, J. Carbonell, R. D. Brown, and R. E. Frederking, "Translingual information retrieval: A comparative evaluation," in Proc. Int. Joint Conf. Artif. Intell., 1997, pp. 708-715.

[29]

Y. Freund, R. Iyer, R. E. Schapire, and Y. Singer, "An efficient boosting algorithm for combining preferences," in Proc. IEEE Int. Conf. Machine Learning, 1998, pp. 170-178.

[30]

C. Burges, T. Shaked, E. Renshaw, A. Laizer, M. Deeds, N. Hamilton, and G. Hullender, "Learning to rank using gradient descent," in Proc. IEEE Int. Conf. Machine Learning, 2005, pp. 89-96.

Digital Library

[31]

T.-Y. Liu, J. Xu, T. Qin, W. Xing, and H. Li, "LETOR: Benchmark dataset for research on learning to rank for information retrieval," in Proc. Assoc. Comput. Mach. Special Interest Group Inform. Retrieval Workshop Learning Rank Inform. Retrieval, 2007, pp. 3-10.

[32]

X. Geng, T.-Y. Liu, T. Qin, and H. Li, "Feature selection for ranking," in Proc. Assoc. Comput. Mach. Special Interest Group Inform. Retrieval, 2007, pp. 407-414.

[33]

T. Joachims, "Making large-scale SVM learning practical," Advances in Kernel Methods: Support Vector Learning, B. Schölkopf, C. Burges, and A. Smola, Eds. Cambridge, MA: MIT Press, 1999, pp. 169-184.

[34]

A. K. Dey, "Understanding and using context," Personal Ubiquitous Comput., vol. 5, no. 1, pp. 4-7, 2001.

Digital Library

[35]

J. Sivic and A. Zisserman, "Video Google: A text retrieval approach to object matching in videos," in Proc. IEEE Int. Conf. Comput. Vision, vol. 2. Oct. 2003, pp. 1470-1477.

Cited By

Ji ZPang YYuan YPan J(2016)Relevance and irrelevance graph based marginal Fisher analysis for image search rerankingSignal Processing10.1016/j.sigpro.2015.11.010121:C(139-152)Online publication date: 1-Apr-2016
https://dl.acm.org/doi/10.1016/j.sigpro.2015.11.010
Jing PJi ZYu YZhang Z(2016)Visual search reranking with RElevant Local Discriminant AnalysisNeurocomputing10.1016/j.neucom.2014.12.118173:P2(172-180)Online publication date: 15-Jan-2016
https://dl.acm.org/doi/10.1016/j.neucom.2014.12.118
Lei Pang Shiai Zhu Chong-Wah Ngo (2015)Deep Multimodal Learning for Affective Analysis and RetrievalIEEE Transactions on Multimedia10.1109/TMM.2015.248222817:11(2008-2020)Online publication date: 1-Nov-2015
https://dl.acm.org/doi/10.1109/TMM.2015.2482228
Show More Cited By

Recommendations

Multimodal Fusion for Video Search Reranking

Analysis on click-through data from a very large search engine log shows that users are usually interested in the top-ranked portion of returned search results. Therefore, it is crucial for search engines to achieve high accuracy on the top-ranked ...
Optimizing Visual Search Reranking via Pairwise Learning

Visual search reranking is defined as reordering visual documents (images or video clips) based on the initial search results or some auxiliary knowledge to improve the search precision. Conventional approaches to visual search reranking empirically ...
Optimizing video search reranking via minimum incremental information loss
MIR '08: Proceedings of the 1st ACM international conference on Multimedia information retrieval

This paper is concerned with video search reranking - the task of reordering the initial ranked documents (video shots) to improve the search performance - in an optimization framework. Conventional supervised reranking approaches empirically convert ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Circuits and Systems for Video Technology

IEEE Transactions on Circuits and Systems for Video Technology Volume 19, Issue 12

December 2009

216 pages

ISSN:1051-8215

Issue’s Table of Contents

Copyright © 2009.

Publisher

IEEE Press

Publication History

Published: 01 December 2009

Revised: 21 March 2009

Received: 15 September 2008

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ji ZPang YYuan YPan J(2016)Relevance and irrelevance graph based marginal Fisher analysis for image search rerankingSignal Processing10.1016/j.sigpro.2015.11.010121:C(139-152)Online publication date: 1-Apr-2016
https://dl.acm.org/doi/10.1016/j.sigpro.2015.11.010
Jing PJi ZYu YZhang Z(2016)Visual search reranking with RElevant Local Discriminant AnalysisNeurocomputing10.1016/j.neucom.2014.12.118173:P2(172-180)Online publication date: 15-Jan-2016
https://dl.acm.org/doi/10.1016/j.neucom.2014.12.118
Lei Pang Shiai Zhu Chong-Wah Ngo (2015)Deep Multimodal Learning for Affective Analysis and RetrievalIEEE Transactions on Multimedia10.1109/TMM.2015.248222817:11(2008-2020)Online publication date: 1-Nov-2015
https://dl.acm.org/doi/10.1109/TMM.2015.2482228
Jie Geng Zhenjiang Miao Xiao-Ping Zhang (2015)Efficient Heuristic Methods for Multimodal Fusion and Concept Fusion in Video Concept DetectionIEEE Transactions on Multimedia10.1109/TMM.2015.239819517:4(498-511)Online publication date: 1-Apr-2015
https://dl.acm.org/doi/10.1109/TMM.2015.2398195
Zhong Ji Yanwei Pang Xuelong Li (2015)Relevance Preserving Projection and Ranking for Web Image Search RerankingIEEE Transactions on Image Processing10.1109/TIP.2015.243719824:11(4137-4147)Online publication date: 1-Nov-2015
https://dl.acm.org/doi/10.1109/TIP.2015.2437198
Ji ZPang YHe YZhang H(2015)Semi-supervised LPP algorithms for learning-to-rank-based visual search rerankingInformation Sciences: an International Journal10.1016/j.ins.2014.10.037302:C(83-93)Online publication date: 1-May-2015
https://dl.acm.org/doi/10.1016/j.ins.2014.10.037
Weng MChuang Y(2012)Collaborative video reindexing via matrix factorizationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/2168996.21690038:2(1-20)Online publication date: 22-May-2012
https://dl.acm.org/doi/10.1145/2168996.2169003

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents