research-article

A Large-scale RGB-D Database for Arbitrary-view Human Action Recognition

Authors:

Wei-Shi ZhengAuthors Info & Claims

MM '18: Proceedings of the 26th ACM international conference on Multimedia

Pages 1510 - 1518

https://doi.org/10.1145/3240508.3240675

Published: 15 October 2018 Publication History

Abstract

Current researches mainly focus on single-view and multiview human action recognition, which can hardly satisfy the requirements of human-robot interaction (HRI) applications to recognize actions from arbitrary views. The lack of databases also sets up barriers. In this paper, we newly collect a large-scale RGB-D action database for arbitrary-view action analysis, including RGB videos, depth and skeleton sequences. The database includes action samples captured in 8 fixed viewpoints and varying-view sequences which covers the entire 360 view angles. In total, 118 persons are invited to act 40 action categories, and 25,600 video samples are collected. Our database involves more articipants, more viewpoints and a large number of samples. More importantly, it is the first database containing the entire 360? varying-view sequences. The database provides sufficient data for cross-view and arbitrary-view action analysis. Besides, we propose a View-guided Skeleton CNN (VS-CNN) to tackle the problem of arbitrary-view action recognition. Experiment results show that the VS-CNN achieves superior performance.

References

[1]

Yi Bin, Yang Yang, Fumin Shen, Ning Xie, Heng Tao Shen, and Xuelong Li. 2018. Describing Video with Attention based Bidirectional LSTM. IEEE Transactions on Cybernetics (2018).

[2]

Z. Cai, L. Wang, X. Peng, and Y. Qiao. 2014. Multi-View Super Vector for Action Recognition. In CVPR.

Digital Library

[3]

Z. Gao, S. Li, Y. Zhu, C. Wang, and H. Zhang. 2017. Collaborative sparse representation leaning model for RGBD action recognition. Journal of Visual Communication and Image Representation 48 (2017), 442--452.

Digital Library

[4]

A. Gupta, J. Martinez, J. J. Little, and R. J. Woodham. 2014. 3D Pose from Motion for Cross-View Action Recognition via Non-linear Circulant Temporal Encoding. In CVPR.

Digital Library

[5]

K. Hara, H. Kataoka, Y. Satoh, and Satoh. 2018. Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?. In CVPR.

[6]

J. Hu, W. Zheng, J. Lai, S. Gong, and T. Xiang. 2016. Exemplarbased Recognition of Human-Object Interactions. IEEE Transactions on Circuits and Systems for Video Technology 26(4) (2016), 647--660.

[7]

J. Hu, W. Zheng, L. Ma, and et al. 2016. Real-time RGB-D Activity Prediction by Soft Regression. In Proc. of ECCV.

[8]

J. Hu, W. S. Zheng, J. Lai, and J. Zhang. 2017. Jointly Learning Heterogeneous Features for RGB-D Activity Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39 (2017), 2186--2200.

Digital Library

[9]

J. Hu, W. S. Zheng, J. H. Lai, and J. Zhang. 2015. Jointly learning heterogeneous features for RGB-D activity recognition. In CVPR.

[10]

M. Hu, Y. Yang, F. Shen, L. Zhang, H. T. Shen, and X. Li. 2017. Robust Web Image Annotation via Exploring Multi-facet and Structural Knowledge. IEEE Transactions on Image Processing 26, 10 (2017), 4871--4884.

[11]

Y. Ji, H. Cheng, Y. Zheng, and H. Li. 2015. Learning Contrastive Feature Distribution Model for Interaction Recognition. Journal of Visual Communication and Image Representation, 33 (Nov. 2015), 340--349.

Digital Library

[12]

Y. Ji, Y. Ko, A. Shimada, H. Nagahara, and R. Taniguchi. 2012. Cooking Gesture Recognition using Local Feature and Depth Image. In Proc. of ACMMM in workshop CEA.

Digital Library

[13]

Y. Ji, Y. Yang, X. Xu, and H. T. Shen. 2017. One-shot Learning based Pattern Transition Map for Action Early Recognition. Signal Processing (2017).

Digital Library

[14]

Y.-G. Jiang, Q. Dai, W. Liu, X. Xue, and C.-W. Ngo. 2015. Human Action Recognition in Unconstrained Videos by Explicit Motion Modeling. IEEE Transactions on Image Processing 24, 11 (2015), 3781--3795.

Digital Library

[15]

Y. G. Jiang, Z. Wu, J. Wang, X. Xue, and S. F. Chang. 2018. Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 2 (2018), 352--364.

Digital Library

[16]

T. Kim and A. Reiter. 2017. Interpretable 3D Human Action Analysis with Temporal Convolutional Networks. In CVPRW.

[17]

M. Wang L. Xie, Q. Tian and B. Zhang. 2014. Spatial Pooling of Heterogeneous Features for Image Classification. IEEE Transactions on Image Processing 23, 5 (2014), 1994--2008.

[18]

C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager. 2017. Temporal convolutional networks for action segmentation and detection. In CVPR.

[19]

C. Li, Z. Huang, Y. Yang, J. Cao, X. Sun, and H. T. Shen. 2017. Hierarchical Latent Concept Discovery for Video Event Detection. IEEE Transactions on Image Processing 26, 5 (2017), 2149--2162.

Digital Library

[20]

C. Li, P. Wang, S. Wang, Y. Hou, and W. Li. 2017. Skeleton-based Action Recognition Using LSTM and CNN. CoRR abs/1707.02356 (2017).

[21]

R. Li and T. Zickler. 2012. Discriminative virtual views for crossview action recognition. In CVPR.

Digital Library

[22]

A. Liu, N. Xu, W. Nie, Y. Su, Y. Wong, and M. Kankanhalli. 2017. Benchmarking a Multimodal and Multiview and Interactive Dataset for Human Action Recognition. IEEE Trans. Cybernetics 47 (2017), 1781--1794.

[23]

J. Liu, M. Shah, B. Kuipers, and S. Savarese. 2011. Cross-view action recognition via view knowledge transfer. In CVPR.

[24]

M. Liu, H. Liu, and C. Chen. 2017. Enhanced skeleton visualization for view invariant human action recognition. Pattern Recognition 68 (2017), 346--362.

Digital Library

[25]

H. Rahmani, A. Mahmood, D. Huynh, and A. Mian. 2014. Histogram of Oriented Principal Components for Cross-View Action Recognition. In ECCV.

[26]

H. Rahmani, A. Mahmood, D. Huynh, and A. Mian. 2016. Histogram of Oriented Principal Components for Cross-View Action Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 38 (2016), 2430--2443.

Digital Library

[27]

H. Rahmani and A. Mian. 2015. Learning a non-linear knowledge transfer model for cross-view action recognition. In CVPR.

[28]

H. Rahmani and A. Mian. 2016. 3D Action Recognition from Novel Viewpoints. In CVPR.

[29]

L. Rybok, S. Friedberger, U. D. Hanebeck, and R. Stiefelhagen. 2011. The KIT Robo-kitchen data set for the evaluation of view-based activity recognition systems. In 11th IEEE-RAS International Conference on Humanoid Robots (Humanoids 2011).

[30]

A. Shahroudy, J. Liu, T. T. Ng, and G. Wang. 2016. NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis. In CVPR.

[31]

Y. Shen, R. Ji, S. Zhang, W. Zuo, Y Wang, and F. Huang. 2018. Generative Adversarial Learning Towards Fast Weakly Supervised Detection. In CVPR.

[32]

J. Wang, Z. Liu, Y. Wu, and J. Yuan. 2014. Learning Actionlet Ensemble for 3D Human Action Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 36 (2014), 914-- 927.

[33]

J. Wang, X. Nie, Y. Xia, Y. Wu, and S.C. Zhu. 2014. Cross-View Action Modeling, Learning, and Recognition. In CVPR.

Digital Library

[34]

P. Wei, Y. Zhao, N. Zheng, and S.C. Zhu. 2013. Modeling 4D Human-Object Interactions for Event and Object Recognition. In ICCV.

Digital Library

[35]

D. Weinland, E. Boyer, and R. Ronfard. 2007. Action Recognition from Arbitrary Views using 3D Exemplars. In ICCV.

[36]

D. Weinland, R. Ronfard, and E. Boyer. 2006. Free viewpoint action recognition using motion history volumes. Computer Vision and Image Understanding 104 (2006), 249--257.

Digital Library

[37]

P. Yan, S. M. Khan, and M. Shah. 2008. Learning 4D action feature models for arbitrary view action recognition. In CVPR.

[38]

S. Yan, Y. Xiong, and D. Lin. 2018. Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition. In AAAI.

[39]

L. Yu, Y. Yang, Z. Huang, P. Wang, J. Song, and H. T. Shen. 2016. Web Video Event Recognition by Semantic Analysis from Ubiquitous Documents. IEEE Transactions on Image Processing 25, 12 (2016), 5689--5701.

Digital Library

[40]

Z.Cheng, L. Qin, Y. Ye, Q. Huang, and Q. Tian. 2012. Human Daily Action Analysis with Multi-view and Color-Depth Data. In ECCV.

Digital Library

[41]

C. Zhang and W. Zheng. 2017. Semi-supervised Multi-view Discrete Hashing for Fast Image Search. IEEE Transactions on Image Processing 26(6) (2017), 2604--2617.

Digital Library

[42]

L. Zhang, Y. Yang, M. Wang, R. Hong, L. Nie, and X. Li. 2016. Detecting Densely Distributed Graph Patterns for Fine-grained Image Categorization. IEEE Transactions on Image Processing 25, 2 (2016), 553--565.

[43]

Z. Zhang, C. Wang, B. Xiao, W. Zhou, S. Liu, and C. Shi. 2013. Cross-View Action Recognition via a Continuous Virtual Path. In CVPR.

Digital Library

[44]

F. Zhu, L. Shao, and M. Lin. 2013. Multi-view action recognition using local similarity random forests and sensor fusion. Pattern Recognition Letters 34 (2013), 20--24.

Digital Library

Cited By

Gao YLu JLi SLi YDu S(2024)Hypergraph-Based Multi-View Action Recognition Using Event CamerasIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.338211746:10(6610-6622)Online publication date: Oct-2024
https://doi.org/10.1109/TPAMI.2024.3382117
Zhang MCai ZPan LHong FGuo XYang LLiu Z(2024)MotionDiffuse: Text-Driven Human Motion Generation With Diffusion ModelIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.335541446:6(4115-4128)Online publication date: Jun-2024
https://doi.org/10.1109/TPAMI.2024.3355414
Zhu WMa XRo DCi HZhang JShi JGao FTian QWang Y(2024)Human Motion Generation: A SurveyIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.333093546:4(2430-2449)Online publication date: Apr-2024
https://doi.org/10.1109/TPAMI.2023.3330935
Show More Cited By

Index Terms

A Large-scale RGB-D Database for Arbitrary-view Human Action Recognition
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Activity recognition and understanding
        Vision for robotics
      2. Image and video acquisition
        Motion capture
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

ReadingAct RGB-D action dataset and human action recognition from local features

New action dataset captured using two Kinect sensors, containing 2340 videos.Applied a spatio-temporal local feature approach to depth action videos.Developed an action recognition framework using a dynamic time alignment approach.Experiment on three ...
Multi-modal & Multi-view & Interactive Benchmark Dataset for Human Action Recognition
MM '15: Proceedings of the 23rd ACM international conference on Multimedia

Human action recognition is one of the most active research areas in both computer vision and machine learning communities. Several methods for human action recognition have been proposed in the literature and promising results have been achieved on the ...
Arbitrary-View Human Action Recognition: A Varying-View RGB-D Action Dataset
Current researches of action recognition which focus on single-view and multi-view recognition can hardly satisfy the requirements of human-robot interaction (HRI) applications for recognizing human actions from arbitrary views. Arbitrary-view recognition ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '18: Proceedings of the 26th ACM international conference on Multimedia

October 2018

2167 pages

ISBN:9781450356657

DOI:10.1145/3240508

General Chairs:
Susanne Boll
University of Oldenburg, Germany
,
Kyoung Mu Lee
Seoul National University, Korea
,
Jiebo Luo
University of Rochester, USA
,
Wenwu Zhu
Tsinghua University, China
,
Program Chairs:
Hyeran Byun
Yonsei University, Korea
,
Chang Wen Chen
State Univ. Of New York at Buffalo, USA
,
Rainer Lienhart
University of Augsburg, Germany
,
Tao Mei
JD AI, China

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Natural Science Foundation of China (NSFC)

Conference

MM '18

Sponsor:

SIGMM

MM '18: ACM Multimedia Conference

October 22 - 26, 2018

Seoul, Republic of Korea

Acceptance Rates

MM '18 Paper Acceptance Rate 209 of 757 submissions, 28%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

45
Total Citations
View Citations
556
Total Downloads

Downloads (Last 12 months)45
Downloads (Last 6 weeks)8

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Gao YLu JLi SLi YDu S(2024)Hypergraph-Based Multi-View Action Recognition Using Event CamerasIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.338211746:10(6610-6622)Online publication date: Oct-2024
https://doi.org/10.1109/TPAMI.2024.3382117
Zhang MCai ZPan LHong FGuo XYang LLiu Z(2024)MotionDiffuse: Text-Driven Human Motion Generation With Diffusion ModelIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.335541446:6(4115-4128)Online publication date: Jun-2024
https://doi.org/10.1109/TPAMI.2024.3355414
Zhu WMa XRo DCi HZhang JShi JGao FTian QWang Y(2024)Human Motion Generation: A SurveyIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.333093546:4(2430-2449)Online publication date: Apr-2024
https://doi.org/10.1109/TPAMI.2023.3330935
Cheng BFeng WWu QChen JCai ZZhang YWang SChe B(2024)3D Human Animation Synthesis based on a Temporal Diffusion Generative Model2024 2nd International Conference on Pattern Recognition, Machine Vision and Intelligent Algorithms (PRMVIA)10.1109/PRMVIA63497.2024.00028(108-116)Online publication date: 24-May-2024
https://doi.org/10.1109/PRMVIA63497.2024.00028
Liu XChen GTang YWang GZhang XLim S(2024)Language-Free Compositional Action Generation via Decoupling RefinementICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10448207(2910-2914)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10448207
Chen HHe SChen Z(2024)RE-STNet: relational enhancement spatio-temporal networks based on skeleton action recognitionMultimedia Tools and Applications10.1007/s11042-024-18864-yOnline publication date: 15-Mar-2024
https://doi.org/10.1007/s11042-024-18864-y
Liu JDai WWang CCheng YTang YTong X(2024)Plan, Posture and Go: Towards Open-Vocabulary Text-to-Motion GenerationComputer Vision – ECCV 202410.1007/978-3-031-73383-3_26(445-463)Online publication date: 3-Nov-2024
https://doi.org/10.1007/978-3-031-73383-3_26
Liu XLi YZeng AZhou ZYou YLu C(2024)Bridging the Gap Between Human Motion and Action Semantics via Kinematic PhrasesComputer Vision – ECCV 202410.1007/978-3-031-73242-3_13(223-240)Online publication date: 29-Oct-2024
https://doi.org/10.1007/978-3-031-73242-3_13
Liu XHou HYang YLi YLu C(2024)Revisit Human-Scene Interaction via Space OccupancyComputer Vision – ECCV 202410.1007/978-3-031-72973-7_1(1-19)Online publication date: 1-Nov-2024
https://doi.org/10.1007/978-3-031-72973-7_1
Dai WChen LWang JLiu JDai BTang Y(2024)MotionLCM: Real-Time Controllable Motion Generation via Latent Consistency ModelComputer Vision – ECCV 202410.1007/978-3-031-72640-8_22(390-408)Online publication date: 29-Oct-2024
https://doi.org/10.1007/978-3-031-72640-8_22
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents