research-article

Relation Understanding in Videos: A Grand Challenge Overview

Authors:

Tat-Seng ChuaAuthors Info & Claims

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

Pages 2652 - 2656

https://doi.org/10.1145/3343031.3356082

Published: 15 October 2019 Publication History

Abstract

ACM Multimedia 2019 Video Relation Understanding Challenge is the first grand challenge aiming at pushing video content analysis at the relational and structural level. This year, the challenge asks the participants to explore and develop innovative algorithms to detect object entities and their relations based on a large-scale user-generated video dataset. The tasks will advance the foundation of future visual systems that are able to perform complex inferences. This paper presents an overview of the grand challenge, including background, detailed descriptions of the three proposed tasks, the corresponding datasets for training, validation and testing, and the evaluation process.

References

[1]

Jingjing Chen and Chong-Wah Ngo. 2016. Deep-based ingredient recognition for cooking recipe retrieval. In Proceedings of the 24th ACM international conference on Multimedia. ACM, 32--41.

Digital Library

[2]

Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et almbox. 2018. AVA: A video dataset of spatio-temporally localized atomic visual actions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6047--6056.

[3]

Jingjia Huang, Nannan Li, Jiaxing Zhong, Thomas H Li, and Ge Li. 2018. Online Action Tube Detection via Resolving the Spatio-temporal Context Pattern. In 2018 ACM Multimedia Conference on Multimedia Conference. ACM, 993--1001.

Digital Library

[4]

Yu-Gang Jiang, Minjun Li, Xi Wang, Wei Liu, and Xian-Sheng Hua. 2018. DeepProduct: Mobile product search with portable deep features. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Vol. 14, 2 (2018), 50.

Digital Library

[5]

Dong Li, Zhaofan Qiu, Qi Dai, Ting Yao, and Tao Mei. 2018. Recurrent tubelet proposal and recognition networks for action detection. In Proceedings of the European conference on computer vision (ECCV). 303--318.

Digital Library

[6]

Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. 2016. Visual relationship detection with language priors. In European Conference on Computer Vision. Springer, 852--869.

[7]

Cong Ma, Yuan Li, Fan Yang, Ziwei Zhang, Yueqing Zhuang, Huizhu Jia, and Xiaodong Xie. 2019. Deep Association: End-to-end Graph-Based Learning for Multiple Object Tracking with Conv-Graph Neural Network. In Proceedings of the 2019 on International Conference on Multimedia Retrieval. ACM, 253--261.

Digital Library

[8]

Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan, and Vincent Vanhoucke. 2017. Youtube-bounding boxes: A large high-precision human-annotated data set for object detection in video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5296--5305.

[9]

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et almbox. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision, Vol. 115, 3 (2015), 211--252.

[10]

Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. 2019. Annotating Objects and Relations in User-Generated Videos. In Proceedings of the 2019 on International Conference on Multimedia Retrieval. ACM, 279--287.

Digital Library

[11]

Xindi Shang, Tongwei Ren, Jingfan Guo, Hanwang Zhang, and Tat-Seng Chua. 2017a. Video visual relation detection. In Proceedings of the 25th ACM international conference on Multimedia. ACM, 1300--1308.

Digital Library

[12]

Xindi Shang, Tongwei Ren, Hanwang Zhang, Gangshan Wu, and Tat-Seng Chua. 2017b. Object trajectory proposal. In 2017 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 331--336.

[13]

Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. [n. d.]. YFCC100M: The New Data in Multimedia Research. Commun. ACM, Vol. 59, 2 ( [n. d.]), 64--73.

[14]

Yunlu Xu, Chengwei Zhang, Zhanzhan Cheng, Jianwen Xie, Yi Niu, Shiliang Pu, and Fei Wu. 2019. Segregated Temporal Assembly Recurrent Networks for Weakly Supervised Multiple Action Detection., Vol. 33 (2019), 9070--9078.

[15]

Xun Yang, Yunshan Ma, Lizi Liao, Meng Wang, and Tat-Seng Chua. 2019. TransNFCM: translation-based neural fashion compatibility modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 403--410.

Digital Library

[16]

Xun Yang, Meng Wang, Luming Zhang, Fuming Sun, Richang Hong, and Meibin Qi. 2016. An efficient tracking system by orthogonalized templates. IEEE Transactions on Industrial Electronics, Vol. 63, 5 (2016), 3187--3197.

Cited By

He WRen JBai RJiang XCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Hierarchical Perceptual and Predictive Analogy-Inference Network for Abstract Visual ReasoningProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681246(4841-4850)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681246
Li HZhu GZhang LJiang YDang YHou HShen PZhao XShah SBennamoun M(2024)Scene Graph Generation: A comprehensive surveyNeurocomputing10.1016/j.neucom.2023.127052566(127052)Online publication date: Jan-2024
https://doi.org/10.1016/j.neucom.2023.127052
Xiao JZhou PYao ALi YHong RYan SChua T(2023)Contrastive Video Question Answering via Video Graph TransformerIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.3292266(1-14)Online publication date: 2023
https://doi.org/10.1109/TPAMI.2023.3292266
Show More Cited By

Index Terms

Relation Understanding in Videos: A Grand Challenge Overview
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Object recognition
      2. Computer vision tasks
        Activity recognition and understanding
        Scene understanding
2. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Information extraction

Recommendations

Annotating Objects and Relations in User-Generated Videos
ICMR '19: Proceedings of the 2019 on International Conference on Multimedia Retrieval

Understanding the objects and relations between them is indispensable to fine-grained video content analysis, which is widely studied in recent research works in multimedia and computer vision. However, existing works are limited to evaluating with ...
Relation Understanding in Videos
MM '19: Proceedings of the 27th ACM International Conference on Multimedia

In this paper, we present our solutions to the grand challenge task "Relation Understanding in Videos" in ACM Multimedia 2019. The challenge task aims to detect instances of target visual relations in a video, where a visual relation instance is ...
Pose-Enhanced Relation Feature for Action Recognition in Still Images
MultiMedia Modeling
Abstract
Due to the lack of motion information, action recognition in still images is considered a challenging task. Previous works focused on contextual information in the image, including human pose, surrounding objects, etc. But they rarely consider the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

October 2019

2794 pages

ISBN:9781450368896

DOI:10.1145/3343031

General Chairs:
Laurent Amsaleg
CNRS-IRISA, France
,
Benoit Huet
EURECOM, France
,
Martha Larson
Radboud University and TU Delft (Netherlands)
,
Program Chairs:
Guillaume Gravier
CNRS-IRISA, France
,
Hayley Hung
Delft University of Technology Netherlands
,
Chong-Wah Ngo
City University of Hong Kong Hong Kong
,
Wei Tsang Ooi
National University of Singapore Singapore

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Research Foundation Singapore

Conference

MM '19

Sponsor:

SIGMM

MM '19: The 27th ACM International Conference on Multimedia

October 21 - 25, 2019

Nice, France

Acceptance Rates

MM '19 Paper Acceptance Rate 252 of 936 submissions, 27%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
393
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)3

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

He WRen JBai RJiang XCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Hierarchical Perceptual and Predictive Analogy-Inference Network for Abstract Visual ReasoningProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681246(4841-4850)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681246
Li HZhu GZhang LJiang YDang YHou HShen PZhao XShah SBennamoun M(2024)Scene Graph Generation: A comprehensive surveyNeurocomputing10.1016/j.neucom.2023.127052566(127052)Online publication date: Jan-2024
https://doi.org/10.1016/j.neucom.2023.127052
Xiao JZhou PYao ALi YHong RYan SChua T(2023)Contrastive Video Question Answering via Video Graph TransformerIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.3292266(1-14)Online publication date: 2023
https://doi.org/10.1109/TPAMI.2023.3292266
Cao QHuang H(2023)Video Visual Relation Detection With Contextual Knowledge EmbeddingIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.327032835:12(13083-13095)Online publication date: 1-Dec-2023
https://doi.org/10.1109/TKDE.2023.3270328
Ji XChen JWu X(2023)Counterfactual Inference for Visual Relationship Detection in Videos2023 IEEE International Conference on Multimedia and Expo (ICME)10.1109/ICME55011.2023.00036(162-167)Online publication date: Jul-2023
https://doi.org/10.1109/ICME55011.2023.00036
Cao QHuang H(2022)Attention Guided Relation Detection Approach for Video Visual Relation DetectionIEEE Transactions on Multimedia10.1109/TMM.2021.310943024(3896-3907)Online publication date: 2022
https://doi.org/10.1109/TMM.2021.3109430
Xiao JZhou PChua TYan S(2022)Video Graph Transformer for Video Question AnsweringComputer Vision – ECCV 202210.1007/978-3-031-20059-5_3(39-58)Online publication date: 29-Oct-2022
https://doi.org/10.1007/978-3-031-20059-5_3
Ji WLi YWei MShang XXiao JRen TChua TShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)VidVRD 2021Proceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3479232(4779-4783)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3479232
Li YYang XShang XChua TShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)Interventional Video Relation DetectionProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475540(4091-4099)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475540
Shang XLi YXiao JJi WChua TShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)Video Visual Relation Detection via Iterative InferenceProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475263(3654-3663)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475263
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents