short-paper

GeoWINE: Geolocation based Wiki, Image, News and Event Retrieval

Authors:

Ralph EwerthAuthors Info & Claims

SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 2565 - 2569

https://doi.org/10.1145/3404835.3462786

Published: 11 July 2021 Publication History

Get Access

Abstract

In the context of social media, geolocation inference on news or events has become a very important task. In this paper, we present the GeoWINE (Geolocation-based Wiki-Image-News-Event retrieval) demonstrator, an effective modular system for multimodal retrieval which expects only a single image as input. The GeoWINE system consists of five modules in order to retrieve related information from various sources. The first module is a state-of-the-art model for geolocation estimation of images. The second module performs a geospatial-based query for entity retrieval using the Wikidata knowledge graph. The third module exploits four different image embedding representations, which are used to retrieve most similar entities compared to the input image. The last two modules perform news and event retrieval from EventRegistry and the Open Event Knowledge Graph (OEKG). GeoWINE provides an intuitive interface for end-users and is insightful for experts for reconfiguration to individual setups. The GeoWINE achieves promising results in entity label prediction for images on Google Landmarks dataset. The demonstrator is publicly available at http://cleopatra.ijs.si/geowine/.

Supplementary Material

MP4 File (SIGIR21_de1898.mp4)

In the context of social media, geolocation inference on news or events has become a very important task. GeoWINE (Geolocation-based Wiki-Image-News-Event retrieval) demonstrator, an effective modular system for multimodal retrieval which expects only a single image as input. It consists of five modules to retrieve related information from various sources. The first module is a state-of-the-art model for geolocation estimation of images. The second module performs a geospatial-based query for entity retrieval using the Wikidata knowledge graph. The third module exploits four different image embedding representations, which are used to retrieve most similar entities compared to the input image. The last two modules perform news and event retrieval from EventRegistry and the Open Event Knowledge Graph (OEKG). The GeoWINE achieves promising results in entity label prediction for images on Google Landmarks dataset. The demonstrator is available at http://cleopatra.ijs.si/geowine/.

Download
760.63 MB

References

[1]

Hanaa Al-Lohibi, Tahani Alkhamisi, Maha Assagran, Amal Aljohani, and Asia Othaman Aljahdali. 2020. Awjedni: A Reverse-Image-Search Application. ADCAIJ: Advances in Distributed Computing and Artificial Intelligence Journal, Vol. 9, 3 (2020), 49--68.

Google Scholar

[2]

Jason Armitage, Endri Kacupaj, Golsa Tahmasebzadeh, Swati, Maria Maleshkova, Ralph Ewerth, and Jens Lehmann. 2020. MLM: A Benchmark Dataset for Multitask Learning with Multiple Languages and Modalities. In CIKM '20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19--23, 2020. ACM, 2967--2974. https://doi.org/10.1145/3340531.3412783

Digital Library

Google Scholar

[3]

Jiaxin Cheng, Yue Wu, Wael AbdAlmageed, and Premkumar Natarajan. 2019. QATM: Quality-Aware Template Matching for Deep Learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16--20, 2019. Computer Vision Foundation / IEEE, 11553--11562. https://doi.org/10.1109/CVPR.2019.01182

Crossref

Google Scholar

[4]

Simon Gottschalk and Elena Demidova. 2019. EventKG - the hub of event knowledge on the web - and biographical timeline generation. Semantic Web, Vol. 10, 6 (2019), 1039--1070. https://doi.org/10.3233/SW-190355

Digital Library

Google Scholar

[5]

Simon Gottschalk, Endri Kacupaj, Sara Abdollahi, Diego Alves, Gabriel Amaral, Elisavet Koutsiana, Tin Kuculo, Daniela Major, Caio Mello, Gullal S. Cheema, Abdul Sittar, Swati, Golsa Tahmasebzadeh, and Gaurish Thakkar. 2021. OEKG: The Open Event Knowledge Graph. In Proceedings of the 2nd International Workshop on Cross-lingual Event-centric Open Analytics co-located with the 30th The Web Conference (WWW 2021), Ljubljana, Slovenia, April 12, 2021 (online event due to COVID-19 outbreak) (CEUR Workshop Proceedings, Vol. 2829). CEUR-WS.org, 61--75. http://ceur-ws.org/Vol-2829/paper5.pdf

Google Scholar

[6]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016a. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27--30, 2016. IEEE Computer Society, 770--778. https://doi.org/10.1109/CVPR.2016.90

Crossref

Google Scholar

[7]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016b. Identity Mappings in Deep Residual Networks. In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part IV (Lecture Notes in Computer Science, Vol. 9908). Springer, 630--645. https://doi.org/10.1007/978--3--319--46493-0_38

Crossref

Google Scholar

[8]

Eric Mü ller-Budack, Kader Pustu-Iren, and Ralph Ewerth. 2018. Geolocation Estimation of Photos Using a Hierarchical Model and Scene Classification. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8--14, 2018, Proceedings, Part XII (Lecture Notes in Computer Science, Vol. 11216). Springer, 575--592. https://doi.org/10.1007/978--3-030-01258--8_35

Crossref

Google Scholar

[9]

Eric Mü ller-Budack, Jonas Theiner, Sebastian Diering, Maximilian Idahl, and Ralph Ewerth. 2020. Multimodal Analytics for Real-world News using Measures of Cross-modal Entity Consistency. In Proceedings of the 2020 on International Conference on Multimedia Retrieval, ICMR 2020, Dublin, Ireland, June 8--11, 2020. ACM, 16--25. https://doi.org/10.1145/3372278.3390670

Digital Library

Google Scholar

[10]

Eric Mü ller-Budack, Jonas Theiner, Sebastian Diering, Maximilian Idahl, Sherzod Hakimov, and Ralph Ewerth. 2021. Multimodal News Analytics using Measures of Cross-modal Entity and Context Consistency. Int. J. Multim. Inf. Retr. (2021). https://doi.org/10.1007/s13735-021-00207--4

Crossref

Google Scholar

[11]

Kevin D. Tang, Manohar Paluri, Fei-Fei Li, Robert Fergus, and Lubomir D. Bourdev. 2015. Improving Image Classification with Location Context. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7--13, 2015. IEEE Computer Society, 1008--1016. https://doi.org/10.1109/ICCV.2015.121

Crossref

Google Scholar

[12]

Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 2016. YFCC100M: the new data in multimedia research. Commun. ACM, Vol. 59, 2 (2016), 64--73. https://doi.org/10.1145/2812802

Digital Library

Google Scholar

[13]

Denny Vrandecic and Markus Krö tzsch. 2014. Wikidata: a free collaborative knowledgebase. Commun. ACM, Vol. 57, 10 (2014), 78--85. https://doi.org/10.1145/2629489

Digital Library

Google Scholar

[14]

Tobias Weyand, André Araujo, Bingyi Cao, and Jack Sim. 2020. Google Landmarks Dataset v2 - A Large-Scale Benchmark for Instance-Level Recognition and Retrieval. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13--19, 2020. IEEE, 2572--2581. https://doi.org/10.1109/CVPR42600.2020.00265

Crossref

Google Scholar

Cited By

View all

Cheema GHakimov SMüller-Budack EOtto CBateman JEwerth R(2023)Understanding image-text relations and news values for multimodal news analysisFrontiers in Artificial Intelligence10.3389/frai.2023.11255336Online publication date: 2-May-2023
https://doi.org/10.3389/frai.2023.1125533
Lee ZGanesan VLik T(2023)Image Geolocation from Alternative CuesIRC-SET 202210.1007/978-981-19-7222-5_19(245-256)Online publication date: 1-Jun-2023
https://doi.org/10.1007/978-981-19-7222-5_19
Ali DBlyau TVan de Weghe NVerstockt S(2022)Context-Aware Querying, Geolocalization, and Rephotography of Historical Newspaper ImagesApplied Sciences10.3390/app12211106312:21(11063)Online publication date: 1-Nov-2022
https://doi.org/10.3390/app122111063

Index Terms

GeoWINE: Geolocation based Wiki, Image, News and Event Retrieval
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Information systems
  1. Information retrieval

Recommendations

Leveraging Selective Prediction for Reliable Image Geolocation
MultiMedia Modeling
Abstract
Reliable image geolocation is crucial for several applications, ranging from social media geo-tagging to media verification. State-of-the-art geolocation methods surpass human performance on the task of geolocation estimation from images. However, ...
Performance evaluation and optimization for content-based image retrieval

Performance evaluation of content-based image retrieval (CBIR) systems is an important but still unsolved problem. The reason for its importance is that only performance evaluation allows for comparison and integration of different CBIR systems. We ...
Localized Content-Based Image Retrieval

We define localized content-based image retrieval as a CBIR task where the user is only interested in a portion of the image, and the rest of the image is irrelevant. In this paper we present a localized CBIR system, Accio, that uses labeled images in ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2021

2998 pages

ISBN:9781450380379

DOI:10.1145/3404835

General Chairs:
Fernando Diaz
(Google)
,
Chirag Shah
University of Washington
,
Torsten Suel
New York University
,
Program Chairs:
Pablo Castells
Universidad Autónoma de Madrid, Amazon
,
Rosie Jones
Spotify
,
Tetsuya Sakai
Waseda University

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Horizon 2020 Framework Programme

Conference

SIGIR '21

Sponsor:

SIGIR

SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 11 - 15, 2021

Virtual Event, Canada

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
229
Total Downloads

Downloads (Last 12 months)45
Downloads (Last 6 weeks)3

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Cheema GHakimov SMüller-Budack EOtto CBateman JEwerth R(2023)Understanding image-text relations and news values for multimodal news analysisFrontiers in Artificial Intelligence10.3389/frai.2023.11255336Online publication date: 2-May-2023
https://doi.org/10.3389/frai.2023.1125533
Lee ZGanesan VLik T(2023)Image Geolocation from Alternative CuesIRC-SET 202210.1007/978-981-19-7222-5_19(245-256)Online publication date: 1-Jun-2023
https://doi.org/10.1007/978-981-19-7222-5_19
Ali DBlyau TVan de Weghe NVerstockt S(2022)Context-Aware Querying, Geolocalization, and Rephotography of Historical Newspaper ImagesApplied Sciences10.3390/app12211106312:21(11063)Online publication date: 1-Nov-2022
https://doi.org/10.3390/app122111063

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Leveraging Selective Prediction for Reliable Image Geolocation

Performance evaluation and optimization for content-based image retrieval

Localized Content-Based Image Retrieval