research-article

Towards the development of realistic botnet dataset in the Internet of Things for network forensic analytics: : Bot-IoT dataset

Authors:

Nickolaos Koroniotis,

Elena Sitnikova,

Benjamin TurnbullAuthors Info & Claims

Volume 100, Issue C

Pages 779 - 796

https://doi.org/10.1016/j.future.2019.05.041

Published: 01 November 2019 Publication History

Abstract

The proliferation of IoT systems, has seen them targeted by malicious third parties. To address this challenge, realistic protection and investigation countermeasures, such as network intrusion detection and network forensic systems, need to be effectively developed. For this purpose, a well-structured and representative dataset is paramount for training and validating the credibility of the systems. Although there are several network datasets, in most cases, not much information is given about the Botnet scenarios that were used. This paper proposes a new dataset, so-called Bot-IoT, which incorporates legitimate and simulated IoT network traffic, along with various types of attacks. We also present a realistic testbed environment for addressing the existing dataset drawbacks of capturing complete network information, accurate labeling, as well as recent and complex attack diversity. Finally, we evaluate the reliability of the BoT-IoT dataset using different statistical and machine learning methods for forensics purposes compared with the benchmark datasets. This work provides the baseline for allowing botnet identification across IoT-specific networks. The Bot-IoT dataset can be accessed at Bot-iot (2018) [1].

Highlights

•

Designing a new realistic Bot-IoT dataset and give a detailed description of designing the testbed configuration and simulated IoT sensors.

•

Analyzing the proposed features of the dataset using Correlation Coefficient and Joint Entropy techniques.

•

Evaluating the performance of network forensic methods, based on machine and deep learning algorithms using the botnet-IoT dataset compared with popular datasets.

References

[1]

Bot-iot, 2018, URL https://www.unsw.adfa.edu.au/unsw-canberra-cyber/cybersecurity/ADFA-NB15-Datasets/bot_iot.php.

[2]

Moustafa N., Turnbull B., Choo K.-K.R., Towards automation of vulnerability and exploitation identification in iiot networks, in: 2018 IEEE International Conference on Industrial Internet, ICII, IEEE, 2018, pp. 139–145.

[3]

Moustafa N., Turnbull B., Choo K.-K.R., An ensemble intrusion detection technique based on proposed statistical flow features for protecting network traffic of internet of things, IEEE Internet Things J. (2018),.

[4]

Kolias C., Kambourakis G., Stavrou A., Voas J., Ddos in the iot: Mirai and other botnets, Computer 50 (7) (2017) 80–84.

Digital Library

[5]

Pimenta Rodrigues G.A., de Oliveira Albuquerque R., Gomes de Deus F.E., et al., Cybersecurity and network forensics: Analysis of malicious traffic towards a honeynet with deep packet inspection, Appl. Sci. 7 (10) (2017) 1082.

[6]

Liu C., Yang C., Zhang X., Chen J., External integrity verification for outsourced big data in cloud and iot: A big picture, Future Gener. Comput. Syst. 49 (2015) 58–67.

[7]

Grajeda C., Breitinger F., Baggili I., Availability of datasets for digital forensics–and what is missing, Digit. Investig. 22 (2017) S94–S105.

[8]

KDDcup99 dataset, URL http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html.

[9]

I. Sharafaldin, A. Lashkari, A.A. Ghorbani, Toward generating a new intrusion detection dataset and intrusion traffic characterization, in: Proceedings of Fourth International Conference on Information Systems Security and Privacy, ICISSP, 2018.

[10]

Moustafa N., Slay J., UNSW-Nb15: a comprehensive data set for network intrusion detection systems (UNSW-nb15 network data set), in: Military Communications and Information Systems Conference (MilCIS), 2015, IEEE, 2015, pp. 1–6.

[11]

1998 DARPA intrusion detection evaluation data set, URL https://www.ll.mit.edu/ideval/data/1998data.html.

[12]

Koroniotis N., Moustafa N., Sitnikova E., Slay J., Towards developing network forensic mechanism for botnet activities in the iot based on machine learning techniques, in: International Conference on Mobile Networks and Management, Springer, 2017, pp. 30–44.

[13]

Gubbi J., Buyya R., Marusic S., Palaniswami M., Internet of things (iot): A vision, architectural elements, and future directions, Future Gener. Comput. Syst. 29 (7) (2013) 1645–1660.

Digital Library

[14]

Silva S.S., Silva R.M., Pinto R.C., Salles R.M., Botnets: A survey, Comput. Netw. 57 (2) (2013) 378–403.

Digital Library

[15]

Khattak S., Ramay N.R., Khan K.R., Syed A.A., Khayam S.A., A taxonomy of botnet behavior, detection, and defense, IEEE Commun. Surv. Tutor. 16 (2) (2014) 898–924.

[16]

Amini P., Araghizadeh M.A., Azmi R., A survey on botnet: classification, detection and defense, in: Electronics Symposium (IES), 2015 International, IEEE, 2015, pp. 233–238.

[17]

Palmer G., A road map for digital forensic research: Report from the first digital forensic workshop, 7–8 August 2001, 2001.

[18]

Moustafa N., Slay J., A network forensic scheme using correntropy-variation for attack detection, in: IFIP International Conference on Digital Forensics, Springer, 2018, pp. 225–239.

[19]

Alomari E., Manickam S., Gupta B., Singh P., Anbar M., Design, deployment and use of HTTP-based botnet (HBB) testbed, in: Advanced Communication Technology, ICACT, 2014 16th International Conference on, IEEE, 2014, pp. 1265–1269.

[20]

Carl L., et al., Using machine learning technliques to identify botnet traffic, in: Local Computer Networks, Proceedings 2006 31st IEEE Conference on, IEEE, 2006.

[21]

Bhatia S., Schmidt D., Mohay G., Tickle A., A framework for generating realistic traffic for distributed denial-of-service attacks and flash events, Comput. Secur. 40 (2014) 95–107.

Digital Library

[22]

Behal S., Kumar K., Detection of ddos attacks and flash events using information theory metrics–an empirical investigation, Comput. Commun. 103 (2017) 18–28.

Digital Library

[23]

Doshi R., Apthorpe N., Feamster N., Machine learning ddos detection for consumer internet of things devices, 2018, arXiv preprint arXiv:1804.04159.

[24]

Hodo E., Bellekens X., Hamilton A., Dubouilh P.-L., Iorkyase E., Tachtatzis C., Atkinson R., Threat analysis of iot networks using artificial neural network intrusion detection system, in: Networks, Computers and Communications, ISNCC, 2016 International Symposium on, IEEE, 2016, pp. 1–6.

[25]

Garcia-Teodoro P., Diaz-Verdejo J., Maciá-Fernández G., Vázquez E., Anomaly-based network intrusion detection: Techniques, systems and challenges, Comput. Secur. 28 (1–2) (2009) 18–28.

Digital Library

[26]

Moustafa N., Creech G., Sitnikova E., Keshk M., Collaborative anomaly detection framework for handling big data of cloud computing, in: 2017 Military Communications and Information Systems Conference, MilCIS, IEEE, 2017, pp. 1–6.

[27]

Moustafa N., Choo K.-K.R., Radwan I., Camtepe S., Outlier dirichlet mixture mechanism: Adversarial statistical learning for anomaly detection in the fog, IEEE Trans. Inf. Forensics Secur. (2019),.

Digital Library

[28]

Wang K., Du M., Sun Y., Vinel A., Zhang Y., Attack detection and distributed forensics in machine-to-machine networks, IEEE Netw. 30 (6) (2016) 49–55.

[29]

Rieck K., Holz T., Willems C., Düssel P., Laskov P., Learning and classification of malware behavior, in: International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, Springer, 2008, pp. 108–125.

[30]

Nguyen T.T., Armitage G., A survey of techniques for internet traffic classification using machine learning, IEEE commun. Surv. Tutor. 10 (4) (2008) 56–76.

Digital Library

[31]

Moustafa N., Creech G., Slay J., Flow aggregator module for analysing network traffic, in: Progress in Computing, Analytics and Networking, Springer, 2018, pp. 19–29.

[32]

De Vel O., Anderson A., Corney M., Mohay G., Mining e-mail content for author identification forensics, ACM SIGMOD Rec. 30 (4) (2001) 55–64.

[33]

Ostinato, URL https://ostinato.org/.

[34]

Soni D., Makwana A., A survey on MQTT: a protocol of internet of things (iot), in: Proceeding of the International Conference on Telecommunication, Power Analysis and Computing Techniques, IN, Chennai, 2017.

[35]

Brugger S., Chow J., An assessment of the DARPA IDS evaluation dataset using snort, UCDAVIS Dep. Comput. Sci. 1 (2007) (2007) 22.

[36]

G.M. Fernández, J. Camacho, R. Magán-Carrión, P. Garcıa-Teodoro, R. Theron, UGR’16: A new dataset for the evaluation of cyclostationarity-based network IDSs.

[37]

Tavallaee M., Bagheri E., Lu W., Ghorbani A.A., A detailed analysis of the KDD cup 99 data set, in: Computational Intelligence for Security and Defense Applications, 2009. CISDA 2009. IEEE Symposium on, IEEE, 2009, pp. 1–6.

[38]

Unibs, university of brescia dataset, 2009, URL http://www.ing.unibs.it/ntw/tools/traces/.

[39]

Bhuyan M.H., Bhattacharyya D.K., Kalita J.K., Towards generating real-life datasets for network intrusion detection, IJ Netw. Secur. 17 (6) (2015) 683–701.

[40]

Center of Applied Internet Data Analysis, URL https://www.caida.org/data/.

[41]

Lawrence berkley national laboratory (LBNL), icsi, LBNL/icsi enterprise tracing project, 2005, URL http://www.icir.org/enterprise-tracing/.

[42]

Canadian Institute of Cybersecurity, University of new Brunswick, ISCX dataset, URL http://www.unb.ca/cic/datasets/index.html.

[43]

Ammar A., A decision tree classifier for intrusion detection priority tagging, J. Comput. Commun. 3 (04) (2015) 52.

[44]

Gogoi P., Bhuyan M.H., Bhattacharyya D., Kalita J.K., Packet and flow based network intrusion dataset, in: International Conference on Contemporary Computing, Springer, 2012, pp. 322–334.

[45]

Node-Red tool, URL https://nodered.org/.

[46]

Argus tool, URL https://qosient.com/argus/index.shtml.

[47]

ESXi hypervisor, URL https://www.vmware.com/au/products/esxi-and-esx.html.

[48]

vSphere client, URL https://www.vmware.com/au/products/vsphere.html.

[49]

IoT hub AWS, URL https://aws.amazon.com/iot-core/features/.

[50]

Mosquitto MQTT broker, URL https://mosquitto.org/.

[51]

Emerson R.W., Causation and pearson’s correlation coefficient, J. Visual Impair. Blind. 109 (3) (2015) 242–244.

[52]

Lesne A., Shannon Entropy: a rigorous notion at the crossroads between probability, information theory, dynamical systems and statistical physics, Math. Struct. Comput. Sci. 24 (3) (2014).

[53]

Cron scheduling package, URL https://packages.ubuntu.com/search?keywords=cron.

[54]

Tshark network analysis tool, URL https://www.wireshark.org/.

[55]

Argus (audit record generation and utilization system), URL https://qosient.com/argus/.

[56]

Paliwal S., Gupta R., Denial-of-service, probing & remote to user (r2l) attack detection using genetic algorithm, Int. J. Comput. Appl. 60 (19) (2012) 57–62.

[57]

Bartlett G., Heidemann J., Papadopoulos C., Understanding passive and active service discovery (extended), Technical Report ISI-TR-2007-642, USC/Information Sciences Institute, 2007.

[58]

Hoque N., Bhuyan M.H., Baishya R.C., Bhattacharyya D.K., Kalita J.K., Network attacks: Taxonomy, tools and systems, J. Netw. Comput. Appl. 40 (2014) 307–324.

[59]

Lyon G.F., Nmap network scanning: The official Nmap project guide to network discovery and security scanning, Insecure, 2009.

[60]

hping, URL http://www.hping.org.

[61]

Xprobe2, URL https://www.aldeid.com/wiki/Xprobe2.

[62]

Zargar S.T., Joshi J., Tipper D., A survey of defense mechanisms against distributed denial of service (ddos) flooding attacks, IEEE commun. Surv. Tutor. 15 (4) (2013) 2046–2069.

[63]

Tankard C., Advanced persistent threats and how to monitor and deter them, Netw. Secur. 2011 (8) (2011) 16–19.

[64]

Jesudoss A., Subramaniam N., A survey on authentication attacks and countermeasures in a distributed environment, Indian J. Comput. Sci. Eng. 5 (2014) 71–77.

[65]

Metasploit framework, URL https://www.metasploit.com/.

[66]

Logkeys software, URL http://manpages.ubuntu.com/manpages/xenial/man8/logkeys.8.html.

[67]

Hydra software, URL https://packages.ubuntu.com/trusty/net/hydra.

[68]

Zheng Y., Kwoh C.K., A feature subset selection method based on high-dimensional mutual information, Entropy 13 (4) (2011) 860–901.

[69]

Meyer D., Wien F., Support vector machines, R News 1 (3) (2001) 23–26.

[70]

Grossberg S., Recurrent neural networks, Scholarpedia 8 (2) (2013) 1888.

[71]

Greff K., Srivastava R.K., Koutník J., Steunebrink B.R., Schmidhuber J., LSTM: A search space odyssey, IEEE Trans. Neural Netw. Learn. Syst. 28 (10) (2017) 2222–2232.

Cited By

Mahjoub CHamdi MAlkanhel RMohamed SEjbali R(2024)An adversarial environment reinforcement learning-driven intrusion detection algorithm for Internet of ThingsEURASIP Journal on Wireless Communications and Networking10.1186/s13638-024-02348-62024:1Online publication date: 4-May-2024
https://dl.acm.org/doi/10.1186/s13638-024-02348-6
Plazas Olaya MVergara Tejada JAedo Cobo J(2024)Securing Microservices-Based IoT NetworksJournal of Computer Networks and Communications10.1155/2024/92815292024Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1155/2024/9281529
Touahria Miliani MSadat SHaddad MSeba HAmrouche K(2024)Comparing Hyperbolic Graph Embedding models on Anomaly Detection for CybersecurityProceedings of the 19th International Conference on Availability, Reliability and Security10.1145/3664476.3670445(1-11)Online publication date: 30-Jul-2024
https://dl.acm.org/doi/10.1145/3664476.3670445
Show More Cited By

Index Terms

Towards the development of realistic botnet dataset in the Internet of Things for network forensic analytics: Bot-IoT dataset

Index terms have been assigned to the content through auto-classification.

Recommendations

Towards the Development of Realistic DoS Dataset for Intelligent Transportation Systems
Abstract
Vehicular ad-hoc networks (VANETs) present security vulnerabilities, which make them prone to diverse cyberattacks. Denial of Service (DoS) is one of the most prevalent and severe cyberattack that targets VANETs. To tackle this cyberattack and ...
Mobile botnet detection using network forensics
FIS'10: Proceedings of the Third future internet conference on Future internet

Malicious software (malware) infects large numbers of computers around the world. This malware can be used to promote unwanted products, disseminate offensive content, or provide unauthorized access to personal and financial information. Until recently ...
Botnet Detection via mining of network traffic flow
Abstract
During the past decade, botnet has emerged as a very serious threat to cyber security by proving it’s capability of compromising billions of computers and making them does the illegal work. There are a number of existing ways by which botnet can ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Future Generation Computer Systems

Future Generation Computer Systems Volume 100, Issue C

Nov 2019

1103 pages

ISSN:0167-739X

Issue’s Table of Contents

Elsevier B.V.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 November 2019

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

203
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Mahjoub CHamdi MAlkanhel RMohamed SEjbali R(2024)An adversarial environment reinforcement learning-driven intrusion detection algorithm for Internet of ThingsEURASIP Journal on Wireless Communications and Networking10.1186/s13638-024-02348-62024:1Online publication date: 4-May-2024
https://dl.acm.org/doi/10.1186/s13638-024-02348-6
Plazas Olaya MVergara Tejada JAedo Cobo J(2024)Securing Microservices-Based IoT NetworksJournal of Computer Networks and Communications10.1155/2024/92815292024Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1155/2024/9281529
Touahria Miliani MSadat SHaddad MSeba HAmrouche K(2024)Comparing Hyperbolic Graph Embedding models on Anomaly Detection for CybersecurityProceedings of the 19th International Conference on Availability, Reliability and Security10.1145/3664476.3670445(1-11)Online publication date: 30-Jul-2024
https://dl.acm.org/doi/10.1145/3664476.3670445
Jaber MBoutry NParrend P(2024)Graph-Based Spectral Analysis for Detecting Cyber AttacksProceedings of the 19th International Conference on Availability, Reliability and Security10.1145/3664476.3664498(1-14)Online publication date: 30-Jul-2024
https://dl.acm.org/doi/10.1145/3664476.3664498
Everson DCheng L(2024)A Survey on Network Attack Surface MappingDigital Threats: Research and Practice10.1145/36400195:2(1-25)Online publication date: 10-Jan-2024
https://dl.acm.org/doi/10.1145/3640019
Wu JDai HKent KYen JXu CWang Y(2024)Open Set Dandelion Network for IoT Intrusion DetectionACM Transactions on Internet Technology10.1145/363982224:1(1-26)Online publication date: 9-Jan-2024
https://dl.acm.org/doi/10.1145/3639822
Djaidja TBrik BMohammed Senouci SBoualouache AGhamri-Doudane Y(2024)Early Network Intrusion Detection Enabled by Attention Mechanisms and RNNsIEEE Transactions on Information Forensics and Security10.1109/TIFS.2024.344186219(7783-7793)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1109/TIFS.2024.3441862
Li RLi QHuang YZou QZhao DZhang ZJiang YZhu FVasilakos A(2024)SeIoT: Detecting Anomalous Semantics in Smart Homes via Knowledge GraphIEEE Transactions on Information Forensics and Security10.1109/TIFS.2024.342885619(7005-7018)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIFS.2024.3428856
Nakıp MGelenbe E(2024)Online Self-Supervised Deep Learning for Intrusion Detection SystemsIEEE Transactions on Information Forensics and Security10.1109/TIFS.2024.340214819(5668-5683)Online publication date: 16-May-2024
https://dl.acm.org/doi/10.1109/TIFS.2024.3402148
Zhang XWang QQin MWang YOhtsuki TAdebisi BSari HGui G(2024)Enhanced Few-Shot Malware Traffic Classification via Integrating Knowledge Transfer With Neural Architecture SearchIEEE Transactions on Information Forensics and Security10.1109/TIFS.2024.339662419(5245-5256)Online publication date: 3-May-2024
https://dl.acm.org/doi/10.1109/TIFS.2024.3396624
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents