research-article

Video representation and suspicious event detection using semantic technologies

Editor: Armin Haller Authors: Ashish Singh Patel, Giovanni Merlino, Dario Bruneo, Antonio Puliafito, O.P. Vyas, Muneendra OjhaAuthors Info & Claims

Semantic Web, Volume 12, Issue 3

Pages 467 - 491

https://doi.org/10.3233/SW-200393

Published: 01 January 2021 Publication History

Abstract

Storage and analysis of video surveillance data is a significant challenge, requiring video interpretation and event detection in the relevant context. To perform this task, the low-level features including shape, texture, and color information are extracted and represented in symbolic forms. In this work, a methodology is proposed, which extracts the salient features and properties using machine learning techniques and represent this information as Linked Data using a domain ontology that is explicitly tailored for detection of certain activities. An ontology is also developed to include concepts and properties which may be applicable in the domain of surveillance and its applications. The proposed approach is validated with actual implementation and is thus evaluated by recognizing suspicious activity in an open parking space. The suspicious activity detection is formalized through inference rules and SPARQL queries. Eventually, Semantic Web Technology has proven to be a remarkable toolchain to interpret videos, thus opening novel possibilities for video scene representation, and detection of complex events, without any human involvement. The proposed novel approach can thus have representation of frame-level information of a video in structured representation and perform event detection while reducing storage and enhancing semantically-aided retrieval of video data.

References

[1]

S. Arivazhagan and R. Newlin Shebiah, Versatile loitering detection based on non-verbal cues using dense trajectory descriptors, Multimedia Tools and Applications 78(8) (2019), 10933–10963.

Digital Library

[2]

S. Auer, C. Bizer, G. Kobilarov, J. Lehmann and Z. Ives, DBpedia: A nucleus for a web of open data, in: 6th Int’l Semantic Web Conference, Springer, Busan, Korea, 2007, pp. 11–15.

[3]

F. Baradel, C. Wolf, J. Mille and G.W. Taylor, Glimpse clouds: Human activity recognition from unstructured feature points, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2018, pp. 469–478. ISBN 9781538664209.

[4]

A. Ben Mabrouk and E. Zagrouba, Abnormal behavior recognition for intelligent video surveillance systems: A review, Expert Systems with Applications 91 (2018), 480–491.

Digital Library

[5]

A.J. Bermejo, J. Villadangos, J.J. Astrain, A. Córdoba, L. Azpilicueta, U. Gárate and F. Falcone, Ontology based road traffic management in emergency situations, Ad-Hoc and Sensor Wireless Networks 20(1–2) (2013), 47–69.

[6]

C. Bizer, T. Heath and T. Berners-Lee, Linked data-the story so far, International journal on Semantic Web and Information Systems 5(3) (2009), 1–22, http://eprints.soton.ac.uk/271285/.

[7]

L. Caruccio, G. Polese, G. Tortora and D. Iannone, EDCAR: A knowledge representation framework to enhance automatic video surveillance, in: Expert Systems with Applications, 2019.

Digital Library

[8]

S. Chen, Q. Jin, J. Chen and A. Hauptmann, Generating video descriptions with latent topic guidance, IEEE Transactions on Multimedia 21(9) (2019), 2407–2418.

[9]

Z. Cheng, X. Li, J. Shen and A.G. Hauptmann, Which information sources are more effective and reliable in video search, in: SIGIR 2016 – Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2016, pp. 1069–1072.

Digital Library

[10]

Cisco Visual Networking Index: Forecast and Methodology 2016–2021, 2017, http://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/complete-white-paper-c11-481360.html.

[11]

A. Dehghan, S.M. Assari and M. Shah, GMMCP tracker: Globally optimal generalized maximum multi clique problem for multiple object tracking, in: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 4091–4099, ISSN.

[12]

T.H. Duong, N.T. Nguyen, H.B. Truong and V.H. Nguyen, A collaborative algorithm for semantic video annotation using a consensus-based social network analysis, Expert Systems with Applications 42(1) (2015), 246–258.

Digital Library

[13]

N. Elleuch, M. Zarka, A. Ben Ammar and M.A. Alimi, A fuzzy ontology: Based framework for reasoning in visual video content analysis and indexing, in: Proceedings of the Eleventh International Workshop on Multimedia Data Mining, 2011, p. 1. ISBN 978-1-4503-0841-0.

Digital Library

[14]

J. Fan, H. Luo, Y. Gao and R. Jain, Incorporating concept ontology for hierarchical video classification, annotation, and visualization, IEEE Transactions on Multimedia 9(5) (2007), 939–957.

Digital Library

[15]

J. Ferryman, PETS 2006 Benchmark Data, 2006, http://www.cvg.reading.ac.uk/PETS2006/data.html.

[16]

J. Ferryman, PETS 2007 Benchmark Data, 2007, http://www.cvg.reading.ac.uk/PETS2007/data.html.

[17]

G.L. Foresti, C. Micheloni and L. Snidaro, Event classification for automatic visual-based surveillance of parking lots, in: Proceedings – International Conference on Pattern Recognition 3(Dimi), 2004, pp. 314–317. ISBN 0769521282.

[18]

A.R.J. François, R. Nevatia, J. Hobbs and R.C. Bolles, VERL: An ontology framework for representing and annotating video events, IEEE Multimedia 12(4) (2005), 76–86.

Digital Library

[19]

C. Gan, N. Wang, Y. Yang, D.Y. Yeung and A.G. Hauptmann, DevNet: A deep event network for multimedia event detection and evidence recounting, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 07-12-June, 2015, pp. 2568–2577. ISBN 9781467369640.

[20]

J. Gómez-Romero, M.A. Patricio, J. García and J.M. Molina, Ontology-based context representation and reasoning for object tracking and scene interpretation in video, Expert Systems with Applications 38(6) (2011), 7494–7510.

Digital Library

[21]

M. Grassi, C. Morbidoni and M. Nucci, A collaborative video annotation system based on semantic web technologies, Cognitive Computation 4(4) (2012), 497–514.

[22]

R. Hamid, S. Maddi, A. Bobick and I. Essa, Structure from statistics – unsupervised activity analysis using suffix trees, in: Proceedings of the IEEE International Conference on Computer Vision, 2007. ISBN 978-1-4244-1630-1.

[23]

A. Hauptmann, R. Yan, W.H. Lin, M. Christel and H. Wactlar, Can high-level concepts fill the semantic gap in video retrieval? A case study with broadcast news, IEEE Transactions on Multimedia 9(5) (2007), 958–966.

Digital Library

[24]

D. He, F. Li, Q. Zhao, X. Long, Y. Fu and S. Wen, Exploiting Spatial-Temporal Modelling and Multi-Modal Fusion for Human Action Recognition, 2018, http://arxiv.org/abs/1806.10319.

[25]

I. Horrocks, P.F. Patel-Schneider, H. Boley, S. Tabet, B. Grosof, M. Dean et al., SWRL: A semantic web rule language combining OWL and RuleML, W3C Member Submission 21(79) (2004), 1–31.

[26]

M. Jangid, V.K. Verma and V.G. Shankar, Counting and classification of vehicle through virtual region for private parking solution, in: Proceedings of First International Conference on Smart System, Innovations and Computing, Springer, Singapore, Singapore, 2018, pp. 761–770.

[27]

W. Liao, C. Yang, M. Ying Yang and B. Rosenhahn, Security event recognition for visual surveillance, in: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences 4(1W1), 2017, pp. 19–26.

[28]

K. Mahmood, Cloud based sports analytics using semantic web tools and technologies, 2015, pp. 431–433.

[29]

M.A. Musen, The protégé project: A look back and a look forward, AI Matters 1(4) (2015), 4–12.

Digital Library

[30]

M. Naphade, J.R. Smith, J. Tesic, S.F. Chang, W. Hsu, L. Kennedy, A. Hauptmann and J. Curtis, Large-scale concept ontology for multimedia, IEEE Multimedia 13(3) (2006), 86–91.

Digital Library

[31]

R. Nevatia, J. Hobbs and B. Bolles, An ontology for video event representation, in: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2004.

[32]

A.S. Patel, M. Ojha, M. Rani, A. Khare, O.P. Vyas and R. Vyas, Ontology-based multi-agent Smart Bike Sharing System (SBSS), in: 2018 IEEE International Conference on Smart Computing (SMARTCOMP), 2018, pp. 417–422.

[33]

L. Patino, T. Cane, A. Vallee and J. Ferryman, PETS 2016: Dataset and challenge, in: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2016, pp. 1240–1247. ISBN 9781467388504.

[34]

L. Patino and J. Ferryman, PETS 2014: Dataset and challenge, in: 11th IEEE International Conference on Advanced Video and Signal-Based Surveillance, AVSS 2014, 2014. ISBN 9781479948710.

[35]

Z. Qiu, T. Yao and T. Mei, Learning spatio-temporal representation with pseudo-3D residual networks, in: 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 5534–5542.

[36]

J. Redmon, S. Divvala, R. Girshick and A. Farhadi, You only look once: Unified, real-time object detection, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779–788.

[37]

J. Shen, D. Tao and X. Li, Modality mixture projections for semantic video event detection, IEEE Transactions on Circuits and Systems for Video Technology 18(11) (2008), 1587–1596.

Digital Library

[38]

Z. Si, M. Pei, B. Yao and S.C. Zhu, Unsupervised learning of event AND-OR grammar and semantics from video, in: Proceedings of the IEEE International Conference on Computer Vision, 2011, pp. 41–48.

Digital Library

[39]

L.F. Sikos, A novel approach to multimedia ontology engineering for automated reasoning over audiovisual LOD datasets, in: ACIIDS, 2016.

[40]

L.F. Sikos, Description Logics in Multimedia Reasoning, Springer, 2017, pp. 1–205. ISBN 978-3-319-54066-5.

[41]

L.F. Sikos, VidOnt: A core reference ontology for reasoning over video scenes scenes *, Journal of Information and Telecommunication 2(2) (2018), 1–13.

[42]

L.F. Sikos and D.M.W. Powers, Knowledge-driven video information retrieval with LOD: From semi-structured to structured video metadata, in: Proceedings of the Eighth Workshop on Exploiting Semantic Annotations in Information Retrieval, 2015, pp. 35–37.

Digital Library

[43]

C.G. Snoek and M. Worring, Concept-based video retrieval, Foundations and Trends in Information Retrieval 2(4) (2008), 215–322.

Digital Library

[44]

M.Y.K. Tani, A. Lablack, A. Ghomari and I.M. Bilasco, Events detection using a video-surveillance ontology and a rule-based approach, Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 8926, 2015, pp. 299–308. ISBN 9783319161808.

[45]

M. Tschentscher, B. Pruß and D. Horn, A simulated car-park environment for the evaluation of video-based on-site parking guidance systems, in: 2017 IEEE Intelligent Vehicles Symposium (IV), 2017, pp. 1571–1576.

Digital Library

[46]

D. Vallet, P. Castells, M. Fernández, P. Mylonas and Y. Avrithis, Personalized content retrieval in context using ontological knowledge, IEEE Transactions on Circuits and Systems for Video Technology 17(3) (2007), 336–345.

Digital Library

[47]

X. Wang and Q. Ji, Video event recognition with deep hierarchical context model, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 07-12-June, 2015, pp. 4418–4427.

[48]

L. Xie, H. Sundaram and M. Campbell, Event mining in multimedia streams, Proceedings of the IEEE 96(4) (2008), 623–647.

[49]

Z. Xu, Y. Liu, L. Mei, C. Hu and L. Chen, Semantic based representing and organizing surveillance big data using video structural description technology, Journal of Systems and Software 102 (2015), 217–225.

Digital Library

[50]

J. You, G. Liu and A. Perkis, A semantic framework for video genre classification and event analysis, Signal Processing: Image Communication 25(4) (2010), 287–302.

Digital Library

[51]

B. Zhang, V. Appia, I. Pekkucuksen, Y. Liu, A.U. Batur, P. Shastry, S. Liu, S. Sivasankaran and K. Chitnis, A surround view camera solution for embedded systems, in: 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2014, pp. 676–681, ISSN.

Digital Library

[52]

X. Zhu, X. Wu, A.K. Elmagarmid, Z. Feng and L. Wu, Video data mining: Semantic indexing and event detection from the association perspective, IEEE Transactions on Knowledge and Data Engineering 17(5) (2005), 665–677.

Digital Library

Cited By

Liang HSong HZhang SBu Y(2024)Highway spillage detection using an improved STPM anomaly detection network from a surveillance perspectiveApplied Intelligence10.1007/s10489-024-06066-w55:1Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1007/s10489-024-06066-w
Patel AMerlino GPuliafito AVyas RVyas OOjha MTiwari V(2023)An NLP-guided ontology development and refinement approach to represent and query visual informationExpert Systems with Applications: An International Journal10.1016/j.eswa.2022.118998213:PBOnline publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1016/j.eswa.2022.118998
Patel AVyas RVyas OOjha MTiwari V(2022)Motion-compensated online object tracking for activity detection and crowd behavior analysisThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-022-02469-339:5(2127-2147)Online publication date: 13-Apr-2022
https://dl.acm.org/doi/10.1007/s00371-022-02469-3

Index Terms

Video representation and suspicious event detection using semantic technologies
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
    2. Knowledge representation and reasoning
2. Information systems

Index terms have been assigned to the content through auto-classification.

Recommendations

Video semantic concept detection using ontology
ICIMCS '11: Proceedings of the Third International Conference on Internet Multimedia Computing and Service

Semantic concept detection in video is a challenge for video semantic content analysis. The performance of semantic concept detection methods depends on representing the video semantic content exactly. In this paper, perception concept and semantic ...
A semantic representation model for design rationale of products

Design rationale (DR) is crucial information in product design decision support, design analysis and design reuse. In this paper, based on the Issue-based Information System (IBIS) model, a new ontology-based semantic representation model for DR ...
Semantic enrichment for medical ontologies

The Unified Medical Language System (UMLS) contains two separate but interconnected knowledge structures, the Semantic Network (upper level) and the Metathesaurus (lower level). In this paper, we have attempted to work out better how the use of such a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Semantic Web

Semantic Web Volume 12, Issue 3

Open Science Data and the Semantic Web Journal

2021

136 pages

ISSN:1570-0844

EISSN:2210-4968

Issue’s Table of Contents

© 2021 – IOS Press. All rights reserved.

Publisher

IOS Press

Netherlands

Publication History

Published: 01 January 2021

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liang HSong HZhang SBu Y(2024)Highway spillage detection using an improved STPM anomaly detection network from a surveillance perspectiveApplied Intelligence10.1007/s10489-024-06066-w55:1Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1007/s10489-024-06066-w
Patel AMerlino GPuliafito AVyas RVyas OOjha MTiwari V(2023)An NLP-guided ontology development and refinement approach to represent and query visual informationExpert Systems with Applications: An International Journal10.1016/j.eswa.2022.118998213:PBOnline publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1016/j.eswa.2022.118998
Patel AVyas RVyas OOjha MTiwari V(2022)Motion-compensated online object tracking for activity detection and crowd behavior analysisThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-022-02469-339:5(2127-2147)Online publication date: 13-Apr-2022
https://dl.acm.org/doi/10.1007/s00371-022-02469-3

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents