research-article

Exquisitor at the Lifelog Search Challenge 2019

Authors:

Omar Shahbaz Khan,

Björn Þór Jónsson,

Stevan Rudinac,

Marcel WorringAuthors Info & Claims

LSC '19: Proceedings of the ACM Workshop on Lifelog Search Challenge

Pages 7 - 11

https://doi.org/10.1145/3326460.3329156

Published: 05 June 2019 Publication History

Abstract

Interactive learning is an umbrella term for methods that attempt to understand the information need of the user and formulate queries that satisfy that information need. We propose to apply the state of the art in interactive multimodal learning to visual lifelog exploration and search, using the Exquisitor system. Exquisitor is a highly scalable interactive learning system, which uses semantic features extracted from visual content and text to suggest relevant media items to the user, based on user relevance feedback on previously suggested items. Findings from our initial experiments indicate that interactive multimodal learning will likely work well for some LSC tasks, but also suggest some potential enhancements.

References

[1]

Vannevar Bush. 1945. As We May Think. The Atlantic Monthly, Vol. 176, 1 (1945), 101--108.

[2]

David A. Cohn, Zoubin Ghahramani, and Michael I. Jordan. 1996. Active Learning with Statistical Models. J. Artificial Intelligence Research, Vol. 4, 1 (1996), 129--145.

Digital Library

[3]

Aaron Duane, Cathal Gurrin, and Wolfgang Huerst. 2018. Virtual Reality Lifelog Explorer: Lifelog Search Challenge at ACM ICMR 2018. In Proceedings of the 2018 ACM Workshop on The Lifelog Search Challenge. ACM, Yokohama, Japan, 20--23.

Digital Library

[4]

Myron Flickner, Harpreet S. Sawhney, Jonathan Ashley, Qian Huang, Byron Dom, Monika Gorkani, Jim Hafner, Denis Lee, Dragutin Petkovic, David Steele, and Peter Yanker. 1995. Query by Image and Video Content: The QBIC System. IEEE Computer, Vol. 28, 9 (1995), 23--32.

Digital Library

[5]

Jim Gemmell, Gordon Bell, Roger Lueder, Steven Drucker, and Curtis Wong. 2002. MyLifeBits: Fulfilling the Memex Vision. In Proc. ACM Multimedia. ACM, Juan les Pins, France, 235--238.

Digital Library

[6]

Gylfi Þór Guðmundsson, Laurent Amsaleg, and Björn Þór Jónsson. 2012. Impact of Storage Technology on the Efficiency of Cluster-based High-dimensional Index Creation. In Proc. International Conference on Database Systems for Advanced Applications (DASFAA). Springer, Busan, South Korea, 53--64.

Digital Library

[7]

Gylfi Þór Guðmundsson, Björn Þór Jónsson, and Laurent Amsaleg. 2010. A Large-scale Performance Study of Cluster-based High-dimensional Indexing. In Proc. International Workshop on Very-large-scale Multimedia Corpus, Mining and Retrieval (VLS-MCMR). ACM, Firenze, Italy, 31--36.

Digital Library

[8]

Cathal Gurrin, Klaus Schoeffmann, Hideo Joho, Andreas Leibetseder, Liting Zhou, Aaron Duane, Duc-Tien Dang-Nguyen, Michael Riegler, Luca Piras, Minh-Triet Tran, et al. 2019. Comparing Approaches to Interactive Lifelog Search at the Lifelog Search Challenge (LSC2018). ITE Transactions on Media Technology and Applications, Vol. 7, 2 (2019), 46--59.

[9]

Cathal Gurrin, Klaus Schoeffmann, Hideo Joho, Bernd Mü nzer, Rami Albatal, Frank Hopfgartner, Liting Zhou, and Duc-Tien Dang-Nguyen. 2019. A Test Collection for Interactive Lifelog Retrieval. In Proc. International Conference on MultiMedia Modeling (MMM). Springer, Thessaloniki, Greece, 312--324.

[10]

Cathal Gurrin, Alan F. Smeaton, and Aiden R. Doherty. 2014. LifeLogging: Personal Big Data. Foundations and Trends in Information Retrieval, Vol. 8, 1 (2014), 1--125.

Digital Library

[11]

Thomas S. Huang, Charlie K. Dagli, Shyamsundar Rajaram, Edward Y. Chang, Michael I. Mandel, Graham E. Poliner, and Daniel P. W. Ellis. 2008. Active Learning for Interactive Multimedia Retrieval. Proc. IEEE, Vol. 96, 4 (2008), 648--667.

[12]

Miriam W. Huijser and Jan C. van Gemert. 2017. Active Decision Boundary Annotation with Deep Generative Models. In Proc. IEEE International Conference on Computer Vision (ICCV). IEEE Computer Society, Venice, Italy, 5296--5305.

[13]

Björn Þór Jónsson, Omar Shahbaz Khan, Hanna Ragnarsdóttir, Þórhildur Þorleiksdóttir, Jan Zahálka, Stevan Rudinac, Gylfi Þór Guðmundsson, Laurent Amsaleg, and Marcel Worring. 2019. Exquisitor: Interactive Learning at Large. arXiv:1904.08689.

[14]

Jakub Lokovc, Gregor Kovalvcik, Tomávs Souvcek, Jaroslav Moravec, and Pvremysl Cech. 2019. VIRET: A Video Retrieval Tool for Interactive Known-Item Search. In Proc. ACM International Conference on Multimedia Retrieval (ICMR). ACM, Ottawa, ON, Canada.

Digital Library

[15]

Jakub Lokovc, Tomávs Souvcek, and Gregor Kovalvcik. 2018. Using an interactive video retrieval tool for lifelog data. In Proceedings of the 2018 ACM Workshop on The Lifelog Search Challenge. ACM, Yokohama, Japan, 15--19.

Digital Library

[16]

Bernd Münzer, Andreas Leibetseder, Sabrina Kletz, Manfred Jürgen Primus, and Klaus Schoeffmann. 2018. lifeXplore at the Lifelog Search Challenge 2018. In Proceedings of the 2018 ACM Workshop on The Lifelog Search Challenge. ACM, Yokohama, Japan, 3--8.

Digital Library

[17]

Chris North. 2006. Toward Measuring Visualization Insight. IEEE Computer Graphics and Applications, Vol. 26, 3 (2006), 6--9.

Digital Library

[18]

Radim Rehruvrek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. ELRA, Valletta, Malta, 45--50.

[19]

Yong Rui, Thomas S. Huang, and Sharad Mehrotra. 1997. Content-Based Image Retrieval with Relevance Feedback in MARS. In Proc. International Conference on Image Processing (ICIP). IEEE Computer Society, Santa Barbara, CA, USA, 815--818.

[20]

Klaus Schoeffmann. 2014. A User-Centric Media Retrieval Competition: The Video Browser Showdown 2012--2014. IEEE MultiMedia, Vol. 21, 4 (2014), 8--13.

[21]

Klaus Schoeffmann, Werner Bailer, Cathal Gurrin, George Awad, and Jakub Lokovc. 2018. Interactive Video Search: Where is the User in the Age of Deep Learning?. In Proc. ACM Multimedia. ACM, Seoul, Republic of Korea, 2101--2103.

Digital Library

[22]

Cees G. M. Snoek, Marcel Worring, Ork de Rooij, Koen E. A. van de Sande, Rong Yan, and Alexander G. Hauptmann. 2008. VideOlympics: Real-Time Evaluation of Multimedia Retrieval Systems. IEEE MultiMedia, Vol. 15, 1 (2008), 86--91.

Digital Library

[23]

Jan Zahálka and Marcel Worring. 2014. Towards Interactive, Intelligent, and Integrated Multimedia Analytics. In Proc. of the IEEE Conference on Visual Analytics Science and Technology (VAST). IEEE Computer Society, Paris, France, 3--12.

[24]

Jan Zahálka, Stevan Rudinac, Björn Þór Jónsson, Dennis C. Koelma, and Marcel Worring. 2018. Blackthorn: Large-Scale Interactive Multimodal Learning. IEEE Transactions on Multimedia, Vol. 20, 3 (2018), 687--698.

Digital Library

Cited By

Khan OSharma UZhu HRudinac SJónsson B(2024)Exquisitor at the Lifelog Search Challenge 2024: Blending Conversational Search with User Relevance FeedbackProceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661132(117-121)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661132
Rossetto LKyriakou ALange SRuosch FWang RWardatzky KBernstein A(2024)LifeGraph 4 - Lifelog Retrieval using Multimodal Knowledge Graphs and Vision-Language ModelsProceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661127(88-92)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661127
Khan OJónsson B(2023)User Relevance Feedback and Novices: Anecdotes from Exquisitor's Participation in Interactive Retrieval CompetitionsProceedings of the 20th International Conference on Content-based Multimedia Indexing10.1145/3617233.3617275(173-177)Online publication date: 20-Sep-2023
https://dl.acm.org/doi/10.1145/3617233.3617275
Show More Cited By

Index Terms

Exquisitor at the Lifelog Search Challenge 2019
1. Information systems
  1. Information retrieval
    1. Specialized information retrieval
      1. Multimedia and multimodal retrieval
  2. Information systems applications
    1. Multimedia information systems
      1. Multimedia databases

Recommendations

lifeXplore at the Lifelog Search Challenge 2018
LSC '18: Proceedings of the 2018 ACM Workshop on The Lifelog Search Challenge

With the growing hype for wearable devices recording biometric data comes the readiness to capture and combine even more personal information as a form of digital diary - lifelogging today is practiced ever more and can be categorized anywhere between ...
Exquisitor at the Lifelog Search Challenge 2020
LSC '20: Proceedings of the Third Annual Workshop on Lifelog Search Challenge

We present an enhanced version of Exquisitor, our interactive and scalable media exploration system. At its core, Exquisitor is an interactive learning system using relevance feedback on media items to build a model of the users' information need. ...
lifeXplore at the Lifelog Search Challenge 2019
LSC '19: Proceedings of the ACM Workshop on Lifelog Search Challenge

First held in Yokohama, Japan in 2018, the Lifelogging Search Challenge (LSC) is an interactive lifelogging content retrieval competition, where international teams are faced with the task of finding specific moments within a large collection of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

LSC '19: Proceedings of the ACM Workshop on Lifelog Search Challenge

June 2019

51 pages

ISBN:9781450367813

DOI:10.1145/3326460

Program Chairs:
Cathal Gurrin
Dublin City University, Ireland
,
Klaus Schoeffmann
Klagenfurt University, Austria
,
Hideo Joho
University of Tsukuba, Japan
,
Duc-Tien Dang-Nguyen
Dublin City University, Ireland
,
Michael Riegler
Oslo Metropolitan University, Norway
,
Luca Piras
University of Cagliari, Italy

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 June 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMR '19

Sponsor:

SIGMM

ICMR '19: International Conference on Multimedia Retrieval

June 10 - 13, 2019

Ottawa ON, Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
174
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)0

Reflects downloads up to 28 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Khan OSharma UZhu HRudinac SJónsson B(2024)Exquisitor at the Lifelog Search Challenge 2024: Blending Conversational Search with User Relevance FeedbackProceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661132(117-121)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661132
Rossetto LKyriakou ALange SRuosch FWang RWardatzky KBernstein A(2024)LifeGraph 4 - Lifelog Retrieval using Multimodal Knowledge Graphs and Vision-Language ModelsProceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661127(88-92)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661127
Khan OJónsson B(2023)User Relevance Feedback and Novices: Anecdotes from Exquisitor's Participation in Interactive Retrieval CompetitionsProceedings of the 20th International Conference on Content-based Multimedia Indexing10.1145/3617233.3617275(173-177)Online publication date: 20-Sep-2023
https://dl.acm.org/doi/10.1145/3617233.3617275
Khan ODuane AJónsson BZahálka JRudinac SWorring MGurrin CSchoeffmann KJónsson BDang-Nguyen DLokoč JTran MHürst WRossetto LHealy G(2021)Exquisitor at the Lifelog Search Challenge 2021: Relationships Between Semantic ClassifiersProceedings of the 4th Annual on Lifelog Search Challenge10.1145/3463948.3469255(3-6)Online publication date: 21-Aug-2021
https://dl.acm.org/doi/10.1145/3463948.3469255
Knudsen EQvortrup TKhan OJónsson BGurrin CSchoeffmann KJónsson BDang-Nguyen DLokoč JTran MHürst WRossetto LHealy G(2021)XQC at the Lifelog Search Challenge 2021: Interactive Learning on a Mobile DeviceProceedings of the 4th Annual on Lifelog Search Challenge10.1145/3463948.3469063(89-93)Online publication date: 21-Aug-2021
https://dl.acm.org/doi/10.1145/3463948.3469063
Mejzlík FVeselý PKratochvíl MSouček TLokoč JGurrin CSchoeffmann KÞór Jónsson BDang-Nguyen DLokoč JTran MHürst W(2020)SOMHunter for Lifelog SearchProceedings of the Third Annual Workshop on Lifelog Search Challenge10.1145/3379172.3391727(73-75)Online publication date: 9-Jun-2020
https://dl.acm.org/doi/10.1145/3379172.3391727
Mai-Nguyen APhan TVo ATran VDao MZettsu KGurrin CSchoeffmann KÞór Jónsson BDang-Nguyen DLokoč JTran MHürst W(2020)BIDAL-HCMUS@LSC2020: An Interactive Multimodal Lifelog Retrieval with Query-to-Sample Attention-based Search EngineProceedings of the Third Annual Workshop on Lifelog Search Challenge10.1145/3379172.3391722(43-49)Online publication date: 9-Jun-2020
https://dl.acm.org/doi/10.1145/3379172.3391722
Tran LNguyen MBinh NLee HGurrin CGurrin CSchoeffmann KÞór Jónsson BDang-Nguyen DLokoč JTran MHürst W(2020)Myscéal: An Experimental Interactive Lifelog Retrieval System for LSC'20Proceedings of the Third Annual Workshop on Lifelog Search Challenge10.1145/3379172.3391719(23-28)Online publication date: 9-Jun-2020
https://dl.acm.org/doi/10.1145/3379172.3391719
Rossetto LBaumgartner MAshena NRuosch FPernischová RBernstein AGurrin CSchoeffmann KÞór Jónsson BDang-Nguyen DLokoč JTran MHürst W(2020)LifeGraph: A Knowledge Graph for LifelogsProceedings of the Third Annual Workshop on Lifelog Search Challenge10.1145/3379172.3391717(13-17)Online publication date: 9-Jun-2020
https://dl.acm.org/doi/10.1145/3379172.3391717
Khan OGurrin CÞór Jónsson BKando NSchoeffmann KChen PO'Connor N(2020)An Interactive Learning System for Large-Scale Multimedia AnalyticsProceedings of the 2020 International Conference on Multimedia Retrieval10.1145/3372278.3391935(368-372)Online publication date: 8-Jun-2020
https://dl.acm.org/doi/10.1145/3372278.3391935
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents