research-article

Using Machine Learning on Testing IoT Applications: a systematic mapping

Authors:

Lavínia Freitas,

Valéria LelliAuthors Info & Claims

WebMedia '22: Proceedings of the Brazilian Symposium on Multimedia and the Web

Pages 348 - 358

https://doi.org/10.1145/3539637.3558049

Published: 07 November 2022 Publication History

Abstract

Internet of Things (IoT) devices are increasingly present in people’s daily lives. Thus has increased research interest in investigating strategies that can ensure that these applications work as expected considering specific and vital characteristics of IoT, for example, security, performance and interoperability. In a testing point of view, there is a need to optimize and define an efficient strategy, from its planning to its execution. Considering all the steps that can be taken to test an IoT application, this process, if performed manually, can demand great effort and time. Machine learning (ML) algorithms have been applied in several areas of computing in order to optimize and automate processes that involve large volumes of data. In this paper, we present a systematic mapping resulting in 40 studies that highlights techniques or approaches that use machine learning algorithms for the most diverse goals within the IoT application testing process, such as the use of neural networks for predicting the cost of time in the preparation and execution of tests; identification of security attacks; and automatic generation of test cases from textual language. We also identified that the vast majority of testing techniques are focused on a specific IoT characteristic (e.g., security, performance), specially security, and apply the machine learning algorithm in two ways: directly in the algorithm, called predictive maintenance, or during the execution of planned tests, both of them bring difficulties related to extracting and defining data to train ML algorithms.

References

[1]

Bernhard K Aichernig, Franz Pernkopf, Richard Schumi, and Andreas Wurm. 2019. Predicting and testing latencies with deep learning: An iot case study. In International Conference on Tests and Proofs. Springer, 93–111. https://doi.org/10.1007/978-3-030-31157-5_7

Digital Library

[2]

Irina Alam and Puneet Gupta. 2020. SAME-Infer: Software Assisted Memory Resilience for Efficient Inference at the Edge. In The International Symposium on Memory Systems. 10–22. https://doi.org/10.1145/3422575.3422774

Digital Library

[3]

Domenico Amalfitano, Nicola Amatucci, Vincenzo De Simone, Vincenzo Riccio, and Fasolino Anna Rita. 2017. Towards a Thing-In-the-Loop Approach for the Verification and Validation of IoT Systems. In Proceedings of the 1st ACM Workshop on the Internet of Safe Things (Delft, Netherlands) (SafeThings’17). Association for Computing Machinery, New York, NY, USA, 57–63. https://doi.org/10.1145/3137003.3137007

Digital Library

[4]

Rossana MC Andrade, Rainara M Carvalho, Italo Linhares de Araújo, Káthia M Oliveira, and Marcio EF Maia. 2017. What changes from ubiquitous computing to internet of things in interaction evaluation?. In International Conference on Distributed, Ambient, and Pervasive Interactions. Springer. https://doi.org/10.1007/978-3-319-58697-7_1

Digital Library

[5]

Simon D Duque Anton, Anna Pia Lohfink, Christoph Garth, and Hans Dieter Schotten. 2019. Security in process: Detecting attacks in industrial process data. In Proceedings of the Third Central European Cybersecurity Conference. 1–6. https://doi.org/10.1145/3360664.3360669

Digital Library

[6]

Francisco Arellano-Espitia, Miguel Delgado-Prieto, Victor Martínez-Viol, Ángel Fernández-Sobrino, and Roque Alfredo Osornio-Rios. 2021. Anomaly Detection in Electromechanical Systems by means of Deep-Autoencoder. In 2021 26th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA). IEEE, 01–06. https://doi.org/10.1109/ETFA45728.2021.9613529

Digital Library

[7]

Kevin ASHTON. 2009. A Coisa da Internet das Coisas. Artigo publicado em (2009).

[8]

Serkan Ayvaz and Koray Alpay. 2021. Predictive maintenance system for production lines in manufacturing: A machine learning approach using IoT data in real-time. Expert Systems with Applications 173 (2021), 114598. https://doi.org/10.1016/j.eswa.2021.114598

Digital Library

[9]

Antonia Bertolino, Antonello Calabró, Guglielmo De Angelis, Micael Gallego, Boni García, and Francisco Gortázar. 2018. When the testing gets tough, the tough get ElasTest. In 2018 IEEE/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion). IEEE, 17–20. https://doi.org/10.1145/3183440.3183497

Digital Library

[10]

Gerd Brewka. 1996. Artificial intelligence—a modern approach by Stuart Russell and Peter Norvig, Prentice Hall. Series in Artificial Intelligence, Englewood Cliffs, NJ.The Knowledge Engineering Review 11, 1 (1996), 78–79. https://doi.org/10.1017/S0269888900007724

[11]

Miroslav Bures, Tomas Cerny, and Bestoun S. Ahmed. 2019. Internet of Things: Current Challenges in the Quality Assurance and Testing Methods. In Information Science and Applications 2018, Kuinam J. Kim and Nakhoon Baek (Eds.). Springer Singapore, Singapore, 625–634. https://doi.org/10.1007/978-981-13-1056-0_61

[12]

Liana Carvalho, Valéria Lelli, and Rossana Andrade. 2022. Performance Testing Guide for IoT Applications. In Proceedings of the 24th International Conference on Enterprise Information Systems - Volume 1: ICEIS,. INSTICC, SciTePress, 667–678. https://doi.org/10.5220/0011090800003179

[13]

Mariela Cortés, Raphael Saraiva, Marcia Souza, Patricia Mello, and Pamella Soares. 2019. Adoption of software testing in internet of things: A systematic literature mapping. In Proceedings of the IV Brazilian Symposium on Systematic and Automated Software Testing. 3–11. https://doi.org/10.1145/3356317.3356326

Digital Library

[14]

Peter Danielis, Moritz Beckmann, and Jan Skodzik. 2020. An ISO-Compliant Test Procedure for Technical Risk Analyses of IoT Systems Based on STRIDE. In 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE, 499–504. https://doi.org/10.1109/COMPSAC48688.2020.0-203

[15]

Fabrizio De Vita, Dario Bruneo, and Sajal K Das. 2020. On the use of a full stack hardware/software infrastructure for sensor data fusion and fault prediction in industry 4.0. Pattern Recognition Letters 138 (2020), 30–37. https://doi.org/10.1016/j.patrec.2020.06.028

[16]

Vasily Desnitsky, Andrey Chechulin, and Igor Kotenko. 2022. Multi-Aspect Based Approach to Attack Detection in IoT Clouds. Sensors 22, 5 (2022), 1831. https://doi.org/10.3390/s22051831

[17]

Vinicius HS Durelli, Rafael S Durelli, Simone S Borges, Andre T Endo, Marcelo M Eler, Diego RC Dias, and Marcelo P Guimarães. 2019. Machine learning applied to software testing: A systematic mapping study. IEEE Transactions on Reliability 68, 3 (2019), 1189–1212. https://doi.org/10.1109/TR.2019.2892517

[18]

Thomas Fehlmann and Eberhard Kranich. 2017. Autonomous real-time software & systems testing. In Proceedings of the 27th International Workshop on Software Measurement and 12th International Conference on Software Process and Product Measurement. 54–63. https://doi.org/10.1145/3143434.3143444

Digital Library

[19]

Peter Flach. 2012. Machine learning: the art and science of algorithms that make sense of data. Cambridge University Press.

Digital Library

[20]

Anna Katrina Gomez and SimiKamini Bajaj. 2019. Challenges of testing complex Internet of Things (IoT) devices and systems. In 2019 11th International Conference on Knowledge and Systems Engineering (KSE). IEEE, 1–4. https://doi.org/10.1109/KSE.2019.8919324

[21]

VVF Grubisic, JPF Aguiar, and Z Simeu-Abazi. 2020. A Review on Intelligent Predictive Maintenance: Bibliometric analysis and new research directions. In 2020 International Conference on Control, Automation and Diagnosis (ICCAD). IEEE, 1–6. https://doi.org/10.1109/ICCAD49821.2020.9260504

[22]

Eman Hammad, Abhijit Kumar Nag, Anitha Chennamaneni, Mohsen Aghashahi, and Erdogan Dogdu. 2021. A Deep-Defense Approach for Next-Gen Cyber-Resilient Inter-Dependent Critical Infrastructure Systems. In 2021 Resilience Week (RWS). IEEE, 1–7. https://doi.org/10.1109/RWS52686.2021.9611790

[23]

Sohaib Hanif, Tuba Ilyas, and Muhammad Zeeshan. 2019. Intrusion detection in IoT using artificial neural networks on UNSW-15 dataset. In 2019 IEEE 16th International Conference on Smart Cities: Improving Quality of Life Using ICT & IoT and AI (HONET-ICT). IEEE, 152–156. https://doi.org/10.1109/HONET.2019.8908122

[24]

Kuo-Kai Hsieh, Li-C Wang, Wen Chen, and Jayanta Bhadra. 2017. Learning to produce direct tests for security verification using constrained process discovery. In 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC). IEEE, 1–6. https://doi.org/10.1145/3061639.3062271

Digital Library

[25]

Faisal Jamil, Dohyeun Kim, 2021. An Ensemble of a Prediction and Learning Mechanism for Improving Accuracy of Anomaly Detection in Network Intrusion Environments. Sustainability 13, 18 (2021), 10057. https://doi.org/10.3390/su131810057

[26]

Kyeong-Joong Jeong, Jin-Duk Park, Kyusoon Hwang, Seong-Lyun Kim, and Won-Yong Shin. 2022. Two-Stage Deep Anomaly Detection With Heterogeneous Time Series Data. IEEE Access 10(2022), 13704–13714. https://doi.org/10.1109/ACCESS.2022.3147188

[27]

Maxim Kalinin and Peter Zegzhda. 2020. AI-based Security for the Smart Networks. In 13th International Conference on Security of Information and Networks. 1–4. https://doi.org/10.1145/3433174.3433593

Digital Library

[28]

Kavin Kamaraj, Behnam Dezfouli, and Yuhong Liu. 2019. Edge mining on iot devices using anomaly detection. In 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE, 33–40. https://doi.org/10.1109/APSIPAASC47483.2019.9023207

[29]

Jalindar Karande and Sarang Joshi. 2021. DEDA: An algorithm for early detection of topology attacks in the internet of things. International Journal of Electrical and Computer Engineering 11, 2(2021), 1761. https://doi.org/10.11591/ijece.v11i2.pp1761-1770

[30]

Barbara Kitchenham and Stuart Charters. 2007. Guidelines for performing systematic literature reviews in software engineering. (2007).

[31]

Vasiliy Krundyshev. 2020. Neural network approach to assessing cybersecurity risks in large-scale dynamic networks. In 13th International Conference on Security of Information and Networks. 1–8. https://doi.org/10.1145/3433174.3433603

Digital Library

[32]

Chu Luo, Jorge Goncalves, Eduardo Velloso, and Vassilis Kostakos. 2020. A survey of context simulation for testing mobile context-aware applications. ACM Computing Surveys (CSUR) 53, 1 (2020), 1–39. https://doi.org/10.1145/3372788

Digital Library

[33]

S Manimurugan. 2021. IoT-Fog-Cloud model for anomaly detection using improved Naïve Bayes and principal component analysis. Journal of Ambient Intelligence and Humanized Computing (2021), 1–10. https://doi.org/10.1007/s12652-020-02723-3

[34]

Noha Medhat, Sherin M Moussa, Nagwa Lotfy Badr, and Mohamed F Tolba. 2020. A Framework for Continuous Regression and Integration Testing in IoT Systems Based on Deep Learning and Search-Based Techniques. IEEE Access 8(2020), 215716–215726. https://doi.org/10.1109/ACCESS.2020.3039931

[35]

Noha Medhat Mohamed, Sherin Moussa, Nagwa Badr, and Mohamed Tolba. 2021. Enhancing Test Cases Prioritization for Internet of Things based systems using Search-based Technique. International Journal of Intelligent Computing and Information Sciences 21, 1 (2021), 84–94. https://doi.org/10.21608/ijicis.2021.69462.1076

[36]

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. 2018. Foundations of machine learning. MIT press.

Digital Library

[37]

W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. 2019. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences 116, 44(2019), 22071–22080. https://doi.org/10.1073/pnas.1900654116

[38]

TD Ovasapyan, VD Danilov, and Dmitry A Moskvin. 2021. Application of synthetic data generation methods to the detection of network attacks on internet of things devices. Automatic Control and Computer Sciences 55, 8 (2021), 991–998. https://doi.org/10.3103/S0146411621080241

Digital Library

[39]

Keyur K Patel, Sunil M Patel, 2016. Internet of things-IOT: definition, characteristics, architecture, enabling technologies, application & future challenges. International journal of engineering science and computing 6, 5(2016). https://doi.org/10.4010/2016.1482

[40]

Ivan Porres, Tanwir Ahmad, Hergys Rexha, Sébastien Lafond, and Dragos Truscan. 2020. Automatic exploratory performance testing using a discriminator neural network. In 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 105–113. https://doi.org/10.1109/ICSTW50294.2020.00030

[41]

P Shanmuga Prabha and S Magesh Kumar. 2022. A Novel Cyber-attack Leads Prediction System using Cascaded R2CNN Model. International Journal of Advanced Computer Science and Applications 13, 2(2022). https://doi.org/10.14569/IJACSA.2022.0130260

[42]

Sumit Pundir, Mohammad S Obaidat, Mohammad Wazid, Ashok Kumar Das, Devesh Pratap Singh, and Joel JPC Rodrigues. 2021. MADP-IIME: malware attack detection protocol in IoT-enabled industrial multimedia environment using machine learning approach. Multimedia Systems (2021), 1–13. https://doi.org/10.1007/s00530-020-00743-9

Digital Library

[43]

Mamunur Rashid, Joarder Kamruzzaman, Tasadduq Imam, Santoso Wibowo, and Steven Gordon. 2022. A tree-based stacking ensemble technique with feature selection for network intrusion detection. Applied Intelligence(2022), 1–14. https://doi.org/10.1007/s10489-021-02968-1

Digital Library

[44]

Anas Saci, Arafat Al-Dweik, and Abdallah Shami. 2021. Autocorrelation integrated gaussian based anomaly detection using sensory data in industrial manufacturing. IEEE Sensors Journal 21, 7 (2021), 9231–9241. https://doi.org/10.1109/JSEN.2021.3053039

[45]

Ahmed Saeed, Ali Ahmadinia, Abbas Javed, and Hadi Larijani. 2016. Intelligent intrusion detection in low-power IoTs. ACM Transactions on Internet Technology (TOIT) 16, 4 (2016), 1–25. https://doi.org/10.1145/2990499

Digital Library

[46]

Safdar Aqeel Safdar, Tao Yue, and Shaukat Ali. 2021. Recommending Faulty Configurations for Interacting Systems Under Test Using Multi-objective Search. ACM Transactions on Software Engineering and Methodology (TOSEM) 30, 4(2021), 1–36. https://doi.org/10.1145/3464939

Digital Library

[47]

Firooz B Saghezchi, Georgios Mantas, Manuel A Violas, A Manuel de Oliveira Duarte, and Jonathan Rodriguez. 2022. Machine learning for DDoS attack detection in industry 4.0 CPPSs. Electronics 11, 4 (2022), 602. https://doi.org/10.3390/electronics11040602

[48]

Iqbal H Sarker. 2021. Cyberlearning: effectiveness analysis of machine learning security modeling to detect cyber-anomalies and multi-attacks. Internet of Things 14(2021), 100393. https://doi.org/10.1016/j.iot.2021.100393

[49]

Iqbal H Sarker, Yoosef B Abushark, Fawaz Alsolami, and Asif Irshad Khan. 2020. Intrudtree: a machine learning based cyber security intrusion detection model. Symmetry 12, 5 (2020), 754. https://doi.org/10.3390/sym12050754

[50]

Iqbal H Sarker, Asif Irshad Khan, Yoosef B Abushark, and Fawaz Alsolami. 2022. Internet of things (iot) security intelligence: a comprehensive overview, machine learning solutions and research directions. Mobile Networks and Applications(2022), 1–17. https://doi.org/10.1007/s11036-022-01937-3

Digital Library

[51]

V Sathyavathy and D Shanmuga Priyaa. 2018. Software Testing Techniques with Artificial Intelligence in IoT Applications. International Journal of Recent Technology and Engineering (IJRTE) 7 (2018).

[52]

Le-Tian Sha, Fu Xiao, Hai-Ping Huang, Yu Chen, and Ru-Chuan Wang. 2019. Catching Escapers: A Detection Method for Advanced Persistent Escapers in Industry Internet of Things Based on Identity-based Broadcast Encryption (IBBE). ACM Transactions on Embedded Computing Systems (TECS) 18, 3(2019), 1–25. https://doi.org/10.1145/3319615

Digital Library

[53]

Vishal Sharma, Ilsun You, Kangbin Yim, Ray Chen, and Jin-Hee Cho. 2019. BRIoT: Behavior rule specification-based misbehavior detection for IoT-embedded cyber-physical systems. IEEE Access 7(2019), 118556–118580. https://doi.org/10.1109/ACCESS.2019.2917135

[54]

Dr Ovidiu Vermesan SINTEF and Dr Norway. 2014. Peter FriessEU, Belgium,“Internet of Things–From Research and Innovation to Market Deployment”.

[55]

Ian Sommerville. 2010. Software Engineering (9ed.). Addison-Wesley, Harlow, England.

[56]

Ovidiu Vermesan and Peter Friess. 2013. Internet of things: converging technologies for smart environments and integrated ecosystems. River publishers.

[57]

Claes Wohlin. 2014. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th international conference on evaluation and assessment in software engineering. 1–10. https://doi.org/10.1145/2601248.2601268

Digital Library

[58]

Tianwei Xing, Luis Garcia, Federico Cerutti, Lance Kaplan, Alun Preece, and Mani Srivastava. 2021. DeepSQA: Understanding Sensor Data via Question Answering. In Proceedings of the International Conference on Internet-of-Things Design and Implementation. 106–118. https://doi.org/10.1145/3450268.3453529

Digital Library

[59]

Wenjin Yu, Tharam Dillon, Fahed Mostafa, Wenny Rahayu, and Yuehua Liu. 2019. A global manufacturing big data ecosystem for fault detection in predictive maintenance. IEEE Transactions on Industrial Informatics 16, 1 (2019), 183–192. https://doi.org/10.1109/TII.2019.2915846

[60]

Wenjin Yu, Yuehua Liu, Tharam Dillon, Wenny Rahayu, and Fahed Mostafa. 2021. An integrated framework for health state monitoring in a smart factory employing IoT and big data techniques. IEEE Internet of Things Journal(2021). https://doi.org/10.1109/JIOT.2021.3096637

[61]

Ying Zhao, Lei Wang, Shijie Li, Fangfang Zhou, Xiaoru Lin, Qiang Lu, and Lei Ren. 2019. A visual analysis approach for understanding durability test data of automotive products. ACM Transactions on Intelligent Systems and Technology (TIST) 10, 6(2019), 1–23. https://doi.org/10.1145/3345640

Digital Library

Cited By

Minani JSabir FMoha NGuéhéneuc Y(2024)A Systematic Review of IoT Systems Testing: Objectives, Approaches, Tools, and ChallengesIEEE Transactions on Software Engineering10.1109/TSE.2024.336361150:4(785-815)Online publication date: 12-Feb-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3363611
Alencar RFernandes BLima Pda Silva C(2024)AI techniques for automated penetration testing in MQTT networks: a literature investigationInternational Journal of Computers and Applications10.1080/1206212X.2024.244350447:1(106-121)Online publication date: 23-Dec-2024
https://doi.org/10.1080/1206212X.2024.2443504

Index Terms

Using Machine Learning on Testing IoT Applications: a systematic mapping

Recommendations

A top-down survey on securing IoT with machine learning: goals, recent advances and challenges

The Internet of Things (IoT) has seen it all from being just another innovation to a leading technology; it is now a binding force that interconnects various aspects of our lives. The IoT's tremendous growth is driven by emerging applications and evolving ...
Security and Privacy in IoT Using Machine Learning and Blockchain: Threats and Countermeasures
Invited Tutorial and Regular Papers

Security and privacy of users have become significant concerns due to the involvement of the Internet of Things (IoT) devices in numerous applications. Cyber threats are growing at an explosive pace making the existing security and privacy measures ...
Machine learning and the Internet of Things security: Solutions and open challenges
Highlights
- Emphasizing security challenges and requirements of IoT-based systems.
- ...
Abstract
Internet of Things (IoT) is a pervasively-used technology for the last few years. IoT technologies are also responsible for intensifying various everyday smart applications improving the standard of living. However, the inter-crossing ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

WebMedia '22: Proceedings of the Brazilian Symposium on Multimedia and the Web

November 2022

389 pages

ISBN:9781450394093

DOI:10.1145/3539637

Copyright © 2022 ACM.

© 2022 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 November 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

CNPQ

Conference

WebMedia '22

WebMedia '22: Brazilian Symposium on Multimedia and Web

November 7 - 11, 2022

Curitiba, Brazil

Acceptance Rates

Overall Acceptance Rate 270 of 873 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
173
Total Downloads

Downloads (Last 12 months)43
Downloads (Last 6 weeks)3

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Minani JSabir FMoha NGuéhéneuc Y(2024)A Systematic Review of IoT Systems Testing: Objectives, Approaches, Tools, and ChallengesIEEE Transactions on Software Engineering10.1109/TSE.2024.336361150:4(785-815)Online publication date: 12-Feb-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3363611
Alencar RFernandes BLima Pda Silva C(2024)AI techniques for automated penetration testing in MQTT networks: a literature investigationInternational Journal of Computers and Applications10.1080/1206212X.2024.244350447:1(106-121)Online publication date: 23-Dec-2024
https://doi.org/10.1080/1206212X.2024.2443504

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten