Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3611643.3613879acmconferencesArticle/Chapter ViewAbstractPublication PagesfseConference Proceedingsconference-collections
research-article

KDDT: Knowledge Distillation-Empowered Digital Twin for Anomaly Detection

Published: 30 November 2023 Publication History

Abstract

Cyber-physical systems (CPSs), like train control and management systems (TCMS), are becoming ubiquitous in critical infrastructures. As safety-critical systems, ensuring their dependability during operation is crucial. Digital twins (DTs) have been increasingly studied for this purpose owing to their capability of runtime monitoring and warning, prediction and detection of anomalies, etc. However, constructing a DT for anomaly detection in TCMS necessitates sufficient training data and extracting both chronological and context features with high quality. Hence, in this paper, we propose a novel method named KDDT for TCMS anomaly detection. KDDT harnesses a language model (LM) and a long short-term memory (LSTM) network to extract contexts and chronological features, respectively. To enrich data volume, KDDT benefits from out-of-domain data with knowledge distillation (KD). We evaluated KDDT with two datasets from our industry partner Alstom and obtained the F1 scores of 0.931 and 0.915, respectively, demonstrating the effectiveness of KDDT. We also explored individual contributions of the DT model, LM, and KD to the overall performance of KDDT, via a comprehensive empirical study, and observed average F1 score improvements of 12.4%, 3%, and 6.05%, respectively.

References

[1]
Mohiuddin Ahmed, Abdun Naser Mahmood, and Jiankun Hu. 2016. A survey of network anomaly detection techniques. Journal of Network and Computer Applications, 60 (2016), Jan., 19–31. issn:10848045 https://doi.org/10.1016/j.jnca.2015.11.016
[2]
R Anushiya and V S Lavanya. 2021. A COMPARATIVE STUDY ON INTRUSION DETECTION SYSTEMS FOR SECURED COMMUNICATION IN INTERNET OF THINGS. ICTACT JOURNAL ON COMMUNICATION TECHNOLOGY, 12, 03 (2021).
[3]
Andrea Arcuri and Lionel Briand. 2011. A practical guide for using statistical tests to assess randomized algorithms in software engineering. Proceedings - International Conference on Software Engineering, 1–10. isbn:9781450304450 issn:02705257 https://doi.org/10.1145/1985793.1985795
[4]
Michael W Browne. 2000. Cross-validation methods. Journal of mathematical psychology, 44, 1 (2000), 108–132.
[5]
KR1442 Chowdhary and KR Chowdhary. 2020. Natural language processing. Fundamentals of artificial intelligence, 603–649.
[6]
Kevin Clark, Minh-Thang Luong, Christopher D Manning, and Quoc V Le. 2018. Semi-supervised sequence modeling with cross-view training. arXiv preprint arXiv:1809.08370.
[7]
Violeta Damjanovic-Behrendt. 2018. A Digital Twin-based Privacy Enhancement Mechanism for the Automotive Industry. https://doi.org/10.1109/IS.2018.8710526 Pages: 279
[8]
Avisha Das and Rakesh M Verma. 2020. Can Machines Tell Stories? A Comparative Study of Deep Neural Language Models and Metrics. IEEE Access, 8 (2020), 181258–181292.
[9]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arxiv:1810.04805 arXiv:1810.04805 [cs]
[10]
Shi Dong, Ping Wang, and Khushnood Abbas. 2021. A survey on deep learning and its applications. Computer Science Review, 40 (2021), 100379. issn:1574-0137 https://doi.org/10.1016/j.cosrev.2021.100379
[11]
Matthias Eckhart and Andreas Ekelhart. 2019. Digital Twins for Cyber-Physical Systems Security: State of the Art and Outlook. In Security and Quality in Cyber-Physical Systems Engineering: With Forewords by Robert M. Lee and Tom Gilb, Stefan Biffl, Matthias Eckhart, Arndt Lüder, and Edgar Weippl (Eds.). Springer International Publishing, Cham. 383–412. isbn:978-3-030-25312-7 https://doi.org/10.1007/978-3-030-25312-7_14
[12]
Eleazar Eskin. 2000. Anomaly Detection over Noisy Data Using Learned Probability Distributions. In Proceedings of the Seventeenth International Conference on Machine Learning (ICML ’00). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. 255–262. isbn:1558607072
[13]
Soumadip Ghosh, Arindrajit Pal, Amitava Nag, Shayak Sadhu, and Ramsekher Pati. 2017. Network anomaly detection using a fuzzy rule-based classifier. In Computer, Communication and Electrical Technology (1 ed.), Debatosh Guha, Badal Chakraborty, and Himadri Sekhar Dutta (Eds.). CRC Press, 61–65. isbn:978-1-315-40062-4 https://doi.org/10.1201/9781315400624-12
[14]
Eric L. Goodman, Chase Zimmerman, and Corey Hudson. 2020. Packet2Vec: Utilizing Word2Vec for Feature Extraction in Packet Data. arxiv:2004.14477 arXiv:2004.14477 [cs]
[15]
Daniela Narezo Guzman, Edin Hadzic, Robert Schuil, Eric Baars, and Jörn Christoffer Groos. 2018. Data-driven condition now- and forecasting of railway switches for improvement in the quality of railway transportation.
[16]
H. Zhao, H. Chen, W. Dong, X. Sun, and Y. Ji. 2017. Fault diagnosis of rail turnout system based on case-based reasoning with compound distance methods. In 2017 29th Chinese Control And Decision Conference (CCDC). 4205–4210. isbn:1948-9447 https://doi.org/10.1109/CCDC.2017.7979237 Journal Abbreviation: 2017 29th Chinese Control And Decision Conference (CCDC)
[17]
Liping Han, Shaukat Ali, Tao Yue, Aitor Arrieta, and Maite Arratibel. 2022. Uncertainty-aware Robustness Assessment of Industrial Elevator Systems.
[18]
Liping Han, Tao Yue, Shaukat Ali, Aitor Arrieta, and Maite Arratibel. 2022. Are Elevator Software Robust against Uncertainties? Results and Experiences from an Industrial Case Study. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA. 1331–1342. isbn:9781450394130 https://doi.org/10.1145/3540250.3558955
[19]
Mohammad Kazim Hooshmand and Doreswamy Hosahalli. 2022. Network anomaly detection using deep learning techniques. CAAI Transactions on Intelligence Technology, 7, 2 (2022), June, 228–243. issn:2468-2322, 2468-2322 https://doi.org/10.1049/cit2.12078
[20]
Umar Islam, Rami Qays Malik, Amnah S. Al-Johani, Muhammad. Riaz Khan, Yousef Ibrahim Daradkeh, Ijaz Ahmad, Khalid A. Alissa, Zulkiflee Abdul-Samad, and Elsayed M. Tag-Eldin. 2022. A Novel Anomaly Detection System on the Internet of Railways Using Extended Neural Networks. Electronics, 11, 18 (2022), Sept., 2813. issn:2079-9292 https://doi.org/10.3390/electronics11182813
[21]
Donghwoon Kwon, Kathiravan Natarajan, Sang C. Suh, Hyunjoo Kim, and Jinoh Kim. 2018. An Empirical Study on Network Anomaly Detection Using Convolutional Neural Networks. In 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS). 1595–1598. https://doi.org/10.1109/ICDCS.2018.00178
[22]
Ulf Lamping and Ed Warnicke. 2004. Wireshark user’s guide. Interface, 4, 6 (2004), 1.
[23]
Christina Latsou, Maryam Farsi, and John Ahmet Erkoyuncu. 2023. Digital twin-enabled automated anomaly detection and bottleneck identification in complex manufacturing systems using a multi-agent approach. Journal of Manufacturing Systems, 67 (2023), April, 242–264. issn:0278-6125 https://doi.org/10.1016/j.jmsy.2023.02.008
[24]
Zhipeng Li, Zheng Qin, Kai Huang, Xiao Yang, and Shuxiong Ye. 2017. Intrusion Detection Using Convolutional Neural Networks for Representation Learning. In Neural Information Processing, Derong Liu, Shengli Xie, Yuanqing Li, Dongbin Zhao, and El-Sayed M. El-Alfy (Eds.). Springer International Publishing, Cham. 858–866. isbn:978-3-319-70139-4
[25]
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
[26]
Chengjie Lu, Qinghua Xu, Tao Yue, Shaukat Ali, Thomas Schwitalla, and Jan F. Nygård. 2023. EvoCLINICAL: Evolving Cyber-cyber Digital Twin with Active Transfer Learning for Automated Cancer Registry System. In Proceedings of the 31th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2023). Association for Computing Machinery, New York, NY, USA. 11 pages. isbn:979-8-4007-0327-0/23/12 https://doi.org/10.1145/3611643.3613897
[27]
Yuan Luo, Ya Xiao, Long Cheng, Guojun Peng, and Danfeng Yao. 2021. Deep learning-based anomaly detection in cyber-physical systems: Progress and opportunities. ACM Computing Surveys (CSUR), 54, 5 (2021), 1–36.
[28]
Mamoru Mimura and Hidema Tanaka. 2018. Reading network packets as a natural language for intrusion detection. In Information Security and Cryptology–ICISC 2017: 20th International Conference, Seoul, South Korea, November 29-December 1, 2017, Revised Selected Papers 20. 339–350.
[29]
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch.
[30]
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arxiv:1802.05365 arXiv:1802.05365 [cs]
[31]
Lucas Pinheiro Cinelli, Matheus Araújo Marins, Eduardo Antúnio Barros da Silva, and Sérgio Lima Netto. 2021. Variational Autoencoder. In Variational Methods for Machine Learning with Applications to Deep Networks, Lucas Pinheiro Cinelli, Matheus Araújo Marins, Eduardo Antônio Barros da Silva, and Sérgio Lima Netto (Eds.). Springer International Publishing, Cham. 111–149. isbn:978-3-030-70679-1 https://doi.org/10.1007/978-3-030-70679-1_5
[32]
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. [n. d.]. Improving Language Understanding by Generative Pre-Training.
[33]
Roland Rosen, Georg von Wichert, George Lo, and Kurt Dirk Bettenhausen. 2015. About The Importance of Autonomy and Digital Twins for the Future of Manufacturing. IFAC-PapersOnLine, 48 (2015), 567–572.
[34]
Maulshree Singh, Evert Fuenmayor, Eoin P Hinchy, Yuansong Qiao, Niall Murray, and Declan Devine. 2021. Digital twin: Origin to future. Applied System Innovation, 4, 2 (2021), 36.
[35]
Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, and Eftychios Protopapadakis. 2018. Deep learning for computer vision: A brief review. Computational intelligence and neuroscience, 2018 (2018).
[36]
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent Abilities of Large Language Models. arxiv:2206.07682 arXiv:2206.07682 [cs]
[37]
Qinghua Xu, Shaukat Ali, and Tao Yue. 2021. Digital twin-based anomaly detection in cyber-physical systems. IEEE, 205–216. isbn:1-72816-836-8
[38]
Qinghua Xu, Shaukat Ali, and Tao Yue. 2023. Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems. ACM Transactions on Software Engineering and Methodology, Feb., 3582571. issn:1049-331X, 1557-7392 https://doi.org/10.1145/3582571
[39]
Qinghua Xu, Shaukat Ali, Tao Yue, and Maite Arratibel. 2022. Uncertainty-aware transfer learning to evolve digital twins for industrial elevators. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, Singapore Singapore. 1257–1268. isbn:978-1-4503-9413-0 https://doi.org/10.1145/3540250.3558957
[40]
Qinghua Xu, Shaukat Ali, Tao Yue, Nedim Zaimovic, and Singh Inderjeet. 2023. Uncertainty-Aware Transfer Learning to Evolve Digital Twins for Industrial Elevators. In Proceedings of the 31th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2023). Association for Computing Machinery, New York, NY, USA. 11 pages. isbn:979-8-4007-0327-0/23/12 https://doi.org/10.1145/3611643.3613879
[41]
Yong Yu, Xiaosheng Si, Changhua Hu, and Jianxun Zhang. 2019. A review of recurrent neural networks: LSTM cells and network architectures. Neural computation, 31, 7 (2019), 1235–1270.
[42]
Tao Yue, Paolo Arcaini, and Shaukat Ali. 2021. Understanding Digital Twins for Cyber-Physical Systems: A Conceptual Model. In Leveraging Applications of Formal Methods, Verification and Validation: Tools and Trends, Tiziana Margaria and Bernhard Steffen (Eds.). Springer International Publishing, Cham. 54–71. isbn:978-3-030-83723-5
[43]
Muhammad Nouman Zafar, Wasif Afzal, Eduard Enoiu, Athanasios Stratis, Aitor Arrieta, and Goiuria Sagardui. 2021. Model-Based Testing in Practice: An Industrial Case Study using GraphWalker. In 14th Innovations in Software Engineering Conference (formerly known as India Software Engineering Conference). ACM, Bhubaneswar, Odisha India. 1–11. isbn:978-1-4503-9046-0 https://doi.org/10.1145/3452383.3452388

Cited By

View all
  • (2024)AgraBOT: Accelerating Third-Party Security Risk Management in Enterprise Setting through Generative AICompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663829(74-79)Online publication date: 10-Jul-2024
  • (2024)Pretrain, Prompt, and Transfer: Evolving Digital Twins for Time-to-Event Analysis in Cyber-Physical SystemsIEEE Transactions on Software Engineering10.1109/TSE.2024.338857250:6(1464-1477)Online publication date: 15-Apr-2024
  • (2024)Foundation Models for the Digital Twins Creation of Cyber-Physical SystemsLeveraging Applications of Formal Methods, Verification and Validation. Application Areas10.1007/978-3-031-75390-9_2(9-26)Online publication date: 27-Oct-2024

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
ESEC/FSE 2023: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
November 2023
2215 pages
ISBN:9798400703270
DOI:10.1145/3611643
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 November 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Train Control and Management System
  2. anomaly detection
  3. digital twin
  4. knowledge distillation

Qualifiers

  • Research-article

Funding Sources

Conference

ESEC/FSE '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)216
  • Downloads (Last 6 weeks)17
Reflects downloads up to 19 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2024)AgraBOT: Accelerating Third-Party Security Risk Management in Enterprise Setting through Generative AICompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663829(74-79)Online publication date: 10-Jul-2024
  • (2024)Pretrain, Prompt, and Transfer: Evolving Digital Twins for Time-to-Event Analysis in Cyber-Physical SystemsIEEE Transactions on Software Engineering10.1109/TSE.2024.338857250:6(1464-1477)Online publication date: 15-Apr-2024
  • (2024)Foundation Models for the Digital Twins Creation of Cyber-Physical SystemsLeveraging Applications of Formal Methods, Verification and Validation. Application Areas10.1007/978-3-031-75390-9_2(9-26)Online publication date: 27-Oct-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media