research-article

Network Traffic Monitoring System Based on Big Data Technology

Authors:

Zhixin ShiAuthors Info & Claims

ICBDC '18: Proceedings of the 3rd International Conference on Big Data and Computing

Pages 27 - 32

https://doi.org/10.1145/3220199.3220221

Published: 28 April 2018 Publication History

Abstract

With the rapid growth of network traffic and the increasing rich methods of network attacks, traditional network traffic monitoring system cannot meet the requirements of data storage and query in real time. Therefore, how to monitor the large scale network traffic effectively has become an important challenge for network security management. Aiming at it, we propose a new network monitoring system where Netflow as the monitoring object based on big data technology, which has four main functions: it can use Filebeat to collect Netflow in real time; it transfers the data reliably based on Logstash; it stores the data in ElasticSearch, it analyzes and displays the data in real time through Kabana. The experimental results show that our system is capable of meeting millisecond responses to 100 million of Netflows. It can meet the requirements of real-time monitoring for large-scale network traffic, and provide the basis for network security control.

References

[1]

Ling Gao-yuan and Zhu Lin, 2016. Design and Implementation of the Internet Traffic Monitoring and Control Software. COMPUTER ENGINEERING&SOFTWARE. 37, 7 (Jan. 2016), 48--52.

[2]

Zhang Kaifei, 2014. The importance of network traffic monitoring to network security governance. Journal of Henan Science and Technology. 12 (Dec. 2014), 4.

[3]

https://en.wikipedia.org/wiki/WannaCry_ransomware_attack

[4]

Zhang Jianyu. 2010. Research on the Key Technologies of Network Traffic Processing for Internet Security Monitoring. Peking University.

[5]

Liu B, Lin C, Jian Q, et al. A NetFlow based flow analysis and monitoring system in enterprise networks{J}. Computer Networks the International Journal of Computer & Telecommunications Networking, 2008, 52(5):1074--1092.

Digital Library

[6]

Zhang W, Gong J, Gu W, et al. NetFlow-based network traffic monitoring{C}// Network Operations and Management Symposium. IEEE, 2011:1--4.

[7]

He Rongyi. Research and Implementation of Network Traffic Monitoring Management System{J}. Silicon Valley, 2008(9):20--20.

[8]

Tang Haina,Li Jun. RRD-based Network Traffic Monitoring Method{J}. Microelectronics and Computers, 2003, 20(7):52--55.

[9]

Xu F, Deng M. Design and Analysis of Campus Network Traffic Detetction System. {J}. Journal of Xiaogan University, 2009.

[10]

Xiong Q, Huang M. Network traffic monitor system based on Netflow and asynchronous service{J}. Computer Engineering, 2006, 32(13):144--146.

[11]

Zhang Long. 2013. Implementation of Abnormal Traffic Detection System Based on Cloud Computing. Tianjin University of Technology.

[12]

Zhang Zhe. 2010. Network Security Event Acquisition Technology Based on NetFlow. Harbin Engineering University.

[13]

Chen Xin. 2013. Research on anomaly Detection Method Based on NetFlow and sFlow Network Flow Fusion.Harbin Engineering University.

[14]

aabc. 2017. Netflow iptables module for Linux kernel (official) https://github.com/aabc/ipt-netflow

[15]

Mark Fullmer.2018.{EB/OL}.https://linux.die.net/man/1/flow-tools.

[16]

Sachdeva G S. The ELK Stack in Production{M}// Practical ELK Stack. Apress, 2017.

[17]

Chen He. 2015. Using Logstash and Elastic Search to Achieve real-time Statistical Analysisi of DSpace Logs. Data Analysis and Knowledge Discovery. 5, (May. 2015), 88--93

[18]

Gormley C, Tong Z. Elasticsearch the definitive guide{J}. Oreilly Media, 2015.

Digital Library

[19]

Khan R. "Deeper Data Dimensions with Kibana"{J}. 2014.

[20]

Huang weiqiang. 2007. Research on Network Traffic Behavior Monitoring Based on sFlow. South China Normal University.

Cited By

Gupta MRai AFarooq MP. S(2023)Network Security and Protection Strategies for Big Data: Challenges and Innovations2023 6th International Conference on Contemporary Computing and Informatics (IC3I)10.1109/IC3I59117.2023.10398011(705-709)Online publication date: 14-Sep-2023
https://doi.org/10.1109/IC3I59117.2023.10398011
El Alaoui IGahi Y(2020)Network Security Strategies in Big Data ContextProcedia Computer Science10.1016/j.procs.2020.07.108175(730-736)Online publication date: 2020
https://doi.org/10.1016/j.procs.2020.07.108
Fu PLiu CYang QLi ZGou GXiong GLi Z(2020)NSA-Net: A NetFlow Sequence Attention Network for Virtual Private Network Traffic DetectionWeb Information Systems Engineering – WISE 202010.1007/978-3-030-62005-9_31(430-444)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1007/978-3-030-62005-9_31

Index Terms

Network Traffic Monitoring System Based on Big Data Technology

Recommendations

On construction of a network log management system using ELK Stack with Ceph
Abstract
A log management system is essential for the networks administrator. With a log management tool, we can collect, store, analyze, archive, and finally dispose of the log information. In this paper, we propose the architecture model of a log ...
The Study and Design of Network Traffic Monitoring Based on Socket
ICCIS '12: Proceedings of the 2012 Fourth International Conference on Computational and Information Sciences

The rapid development of Internet technology and the expansion of application fields about network are making the network traffic monitoring as an important part of network management. In this paper, a network traffic monitoring system is studied and ...
Big Data Analytics

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICBDC '18: Proceedings of the 3rd International Conference on Big Data and Computing

April 2018

155 pages

ISBN:9781450364263

DOI:10.1145/3220199

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

Shenzhen University: Shenzhen University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 April 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICBDC '18

ICBDC '18: 2018 International Conference on Big Data and Computing

April 28 - 30, 2018

Shenzhen, China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
405
Total Downloads

Downloads (Last 12 months)24
Downloads (Last 6 weeks)1

Reflects downloads up to 30 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Gupta MRai AFarooq MP. S(2023)Network Security and Protection Strategies for Big Data: Challenges and Innovations2023 6th International Conference on Contemporary Computing and Informatics (IC3I)10.1109/IC3I59117.2023.10398011(705-709)Online publication date: 14-Sep-2023
https://doi.org/10.1109/IC3I59117.2023.10398011
El Alaoui IGahi Y(2020)Network Security Strategies in Big Data ContextProcedia Computer Science10.1016/j.procs.2020.07.108175(730-736)Online publication date: 2020
https://doi.org/10.1016/j.procs.2020.07.108
Fu PLiu CYang QLi ZGou GXiong GLi Z(2020)NSA-Net: A NetFlow Sequence Attention Network for Virtual Private Network Traffic DetectionWeb Information Systems Engineering – WISE 202010.1007/978-3-030-62005-9_31(430-444)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1007/978-3-030-62005-9_31

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents