research-article

Video Visual Relation Detection via Multi-modal Feature Fusion

Authors:

Gangshan WuAuthors Info & Claims

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

Pages 2657 - 2661

https://doi.org/10.1145/3343031.3356076

Published: 15 October 2019 Publication History

Abstract

Video visual relation detection is a meaningful research problem, which aims to build a bridge between dynamic vision and language. In this paper, we propose a novel video visual relation detection method with multi-model feature fusion. First, we detect objects on each frame densely with the state-of-the-art video object detection model, flow-guided feature aggregation (FGFA), and generate object trajectories by linking the temporally independent objects with Seq-NMS and KCF tracker. Next, we break the relation candidates, i.e., co-occurrent object trajectory pairs, into short-term segments and predict relations with spatial-temporal feature and language context feature. Finally, we greedily associate the short-term relation segments into complete relation instances. The experiment results show that our proposed method outperforms other methods by a large margin, which also earned us the first place in visual relation detection task of Video Relation Understanding Challenge (VRU), ACMMM 2019.

References

[1]

Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. 2016. R-fcn: Object detection via region-based fully convolutional networks. In Advances in Neural Information Processing Systems. 379--387.

Digital Library

[2]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition. 248--255.

[3]

Wei Han, Pooya Khorrami, Tom Le Paine, Prajit Ramachandran, Mohammad Babaeizadeh, Honghui Shi, Jianan Li, Shuicheng Yan, and Thomas S Huang. 2016. Seq-nms for video object detection. arXiv preprint arXiv:1602.08465 (2016).

[4]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 770--778.

[5]

Jo ao F Henriques, Rui Caseiro, Pedro Martins, and Jorge Batista. 2014. High-speed tracking with kernelized correlation filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 37, 3 (2014), 583--596.

Digital Library

[6]

Yao-Hung Hubert Tsai, Santosh Divvala, Louis-Philippe Morency, Ruslan Salakhutdinov, and Ali Farhadi. 2019. Video relationship reasoning using gated spatio-temporal energy graph. In IEEE Conference on Computer Vision and Pattern Recognition. 10424--10433.

[7]

Yu-Gang Jiang, Chong-Wah Ngo, and Jun Yang. 2007. Towards optimal bag-of-features for object categorization and semantic video retrieval. In ACM International Conference on Image and Video Retrieval. 494--501.

Digital Library

[8]

Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. 2016. Visual relationship detection with language priors. In European Conference on Computer Vision. 852--869.

[9]

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013).

[10]

Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. 2019. Annotating objects and relations in user-generated videos. In ACM International Conference on Multimedia Retrieval. 279--287.

Digital Library

[11]

Xindi Shang, Tongwei Ren, Jingfan Guo, Hanwang Zhang, and Tat-Seng Chua. 2017a. Video visual relation detection. In ACM International Conference on Multimedia . 1300--1308.

Digital Library

[12]

Xindi Shang, Tongwei Ren, Hanwang Zhang, Gangshan Wu, and Tat-Seng Chua. 2017b. Object trajectory proposal. In IEEE International Conference on Multimedia and Expo. 331--336.

[13]

Xu Sun, Yuantian Wang, Tongwei Ren, Zhi Liu, Zheng-Jun Zha, and Gangshan Wu. 2018. Object trajectory proposal via hierarchical volume grouping. In ACM International Conference on Multimedia Retrieval. 344--352.

Digital Library

[14]

Philippe Weinzaepfel, Jerome Revaud, Zaid Harchaoui, and Cordelia Schmid. 2013. DeepFlow: Large displacement optical flow with deep matching. In IEEE International Conference on Computer Vision. 1385--1392.

Digital Library

[15]

Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. 2015. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853 (2015).

[16]

Haonan Yu, Wang Jiang, Zhiheng Huang, Yang Yi, and Xu Wei. 2016. Video paragraph captioning using hierarchical recurrent neural networks. In IEEE Conference on Computer Vision Pattern Recognition. 4584--4593.

[17]

Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, and Tat-Seng Chua. 2017. Visual translation embedding network for visual relation detection. In IEEE Conference on Computer Vision and Pattern Recognition. 5532--5540.

[18]

Ke Zhang, Wei-Lun Chao, Fei Sha, and Kristen Grauman. 2016. Video summarization with long short-term memory. In European Conference on Computer Vision . 766--782.

[19]

Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen Wei. 2017. Flow-guided feature aggregation for video object detection. In IEEE International Conference on Computer Vision. 408--417.

[20]

Bohan Zhuang, Lingqiao Liu, Chunhua Shen, and Ian Reid. 2017. Towards context-aware interaction recognition for visual relationship detection. In IEEE International Conference on Computer Vision. 589--598.

Cited By

Herzig RAbramovich OBen Avraham EArbelle AKarlinsky LShamir ADarrell TGloberson A(2024)PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Data2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00666(6789-6801)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00666
Wu LLiu PZhao YWang PZhang Y(2024)Human Cognition-Based Consistency Inference Networks for Multi-Modal Fake News DetectionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.328055536:1(211-225)Online publication date: Jan-2024
https://doi.org/10.1109/TKDE.2023.3280555
Qian RFu ZLiu XZhang KLv ZLan X(2024)Video Visual Relation Detection Based on Trajectory Fusion2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650663(1-9)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10650663
Show More Cited By

Index Terms

Video Visual Relation Detection via Multi-modal Feature Fusion
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision

Recommendations

Video Visual Relation Detection
MM '17: Proceedings of the 25th ACM international conference on Multimedia

As a bridge to connect vision and language, visual relations between objects in the form of relation triplet $łangle subject,predicate,object\rangle$, such as "person-touch-dog'' and "cat-above-sofa'', provide a more comprehensive visual content ...
Video Relation Detection via Multiple Hypothesis Association
MM '20: Proceedings of the 28th ACM International Conference on Multimedia

Video visual relation detection (VidVRD) aims at obtaining not only the trajectories of objects but also the dynamic visual relations between them. It provides abundant information for video understanding and can serve as a bridge between vision and ...
Video Relation Detection with Trajectory-aware Multi-modal Features
MM '20: Proceedings of the 28th ACM International Conference on Multimedia

Video relation detection problem refers to the detection of the relationship between different objects in videos, such as spatial relationship and action relationship. In this paper, we present video relation detection with trajectory-aware multi-modal ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

October 2019

2794 pages

ISBN:9781450368896

DOI:10.1145/3343031

General Chairs:
Laurent Amsaleg
CNRS-IRISA, France
,
Benoit Huet
EURECOM, France
,
Martha Larson
Radboud University and TU Delft (Netherlands)
,
Program Chairs:
Guillaume Gravier
CNRS-IRISA, France
,
Hayley Hung
Delft University of Technology Netherlands
,
Chong-Wah Ngo
City University of Hong Kong Hong Kong
,
Wei Tsang Ooi
National University of Singapore Singapore

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Science, Technology and Innovation Commission of Shenzhen Municipality
National Science Foundation of China
Collaborative Innovation Center of Novel Software Technology and Industrialization

Conference

MM '19

Sponsor:

SIGMM

MM '19: The 27th ACM International Conference on Multimedia

October 21 - 25, 2019

Nice, France

Acceptance Rates

MM '19 Paper Acceptance Rate 252 of 936 submissions, 27%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

38
Total Citations
View Citations
836
Total Downloads

Downloads (Last 12 months)47
Downloads (Last 6 weeks)5

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Herzig RAbramovich OBen Avraham EArbelle AKarlinsky LShamir ADarrell TGloberson A(2024)PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Data2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00666(6789-6801)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00666
Wu LLiu PZhao YWang PZhang Y(2024)Human Cognition-Based Consistency Inference Networks for Multi-Modal Fake News DetectionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.328055536:1(211-225)Online publication date: Jan-2024
https://doi.org/10.1109/TKDE.2023.3280555
Qian RFu ZLiu XZhang KLv ZLan X(2024)Video Visual Relation Detection Based on Trajectory Fusion2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650663(1-9)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10650663
Li HZhu GZhang LJiang YDang YHou HShen PZhao XShah SBennamoun M(2024)Scene Graph Generation: A comprehensive surveyNeurocomputing10.1016/j.neucom.2023.127052566(127052)Online publication date: Jan-2024
https://doi.org/10.1016/j.neucom.2023.127052
He YGan MMa Q(2024)Online video visual relation detection with hierarchical multi-modal fusionMultimedia Tools and Applications10.1007/s11042-023-15310-383:24(65707-65727)Online publication date: 18-Jan-2024
https://doi.org/10.1007/s11042-023-15310-3
Xu XQiu JYu BWang Z(2024)Visual Relationship TransformationComputer Vision – ECCV 202410.1007/978-3-031-73650-6_15(251-272)Online publication date: 21-Nov-2024
https://doi.org/10.1007/978-3-031-73650-6_15
Cao QHuang H(2023)Video Visual Relation Detection With Contextual Knowledge EmbeddingIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.327032835:12(13083-13095)Online publication date: 1-Dec-2023
https://doi.org/10.1109/TKDE.2023.3270328
Cao QHuang HRen MYuan C(2023)Concept-Enhanced Relation Network for Video Visual Relation InferenceIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2022.322042633:5(2233-2244)Online publication date: May-2023
https://doi.org/10.1109/TCSVT.2022.3220426
Ji XChen JWu X(2023)Counterfactual Inference for Visual Relationship Detection in Videos2023 IEEE International Conference on Multimedia and Expo (ICME)10.1109/ICME55011.2023.00036(162-167)Online publication date: Jul-2023
https://doi.org/10.1109/ICME55011.2023.00036
Li HWei PMa ZZheng N(2023)Inverse Compositional Learning for Weakly-supervised Relation Grounding2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01419(15431-15441)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.01419
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents