CN115766079B - Traffic data processing method and device, electronic equipment and readable storage medium - Google Patents
Traffic data processing method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN115766079B CN115766079B CN202211237558.2A CN202211237558A CN115766079B CN 115766079 B CN115766079 B CN 115766079B CN 202211237558 A CN202211237558 A CN 202211237558A CN 115766079 B CN115766079 B CN 115766079B
- Authority
- CN
- China
- Prior art keywords
- flow
- rule
- alarm
- detection engine
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 15
- 238000001514 detection method Methods 0.000 claims abstract description 121
- 238000000034 method Methods 0.000 claims abstract description 34
- 238000012544 monitoring process Methods 0.000 claims abstract description 33
- 238000012545 processing Methods 0.000 claims description 50
- 238000004590 computer program Methods 0.000 claims description 3
- 238000012986 modification Methods 0.000 abstract description 8
- 230000004048 modification Effects 0.000 abstract description 8
- 230000008569 process Effects 0.000 abstract description 8
- 238000005111 flow chemistry technique Methods 0.000 description 18
- 238000010586 diagram Methods 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 241001501944 Suricata Species 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000003306 harvesting Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The embodiment of the invention provides a flow data processing method, a device, electronic equipment and a readable storage medium, which are used for acquiring flow data to be detected in a network; based on the flow alarm rule in the rule configuration file of the detection engine in the monitoring server, detecting flow data by using the detection engine according to the flow alarm rule so as to generate alarm information; the flow alarm rules in the rule configuration file are obtained by periodically polling alarm rules configured by a user; and outputting alarm information to the information acquisition end. Therefore, the user can autonomously configure the alarm rule and poll the alarm rule to obtain the flow alarm rule so as to enable the detection engine to detect the flow data, and the alarm rule configured and managed by the user can be utilized to detect the flow data, so that the rule is easier to configure and manage, the simplicity of configuring the flow alarm rule is improved, the modification process is simplified, and the convenience of managing the flow alarm rule is further improved to a certain extent.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and apparatus for processing flow data, an electronic device, and a readable storage medium.
Background
With the application and popularization of networks, a great number of network attacks, malicious traffic and network intrusion behaviors also occur, and network security becomes a problem which cannot be ignored.
In order to know the real running condition of the network and discover the problems existing in the running process in time, the network traffic needs to be comprehensively detected. In the prior art, a surica engine is often arranged and deployed in a server in a single mode, and flow data is detected based on a flow alarming rule in a command line of the server so as to perform flow data alarming work. If the flow alarm rule is to be modified, the command line defining the flow alarm rule in the server is required to be modified to modify the flow alarm rule of the surica engine, but the modification mode is not easy to operate by a user, and the modification process is complex.
Disclosure of Invention
The invention provides a flow data processing method, a device, equipment and a storage medium, so as to solve the problem of complex modification process.
In order to solve the technical problems, the invention is realized as follows:
in a first aspect, the present invention provides a flow data processing method applied to a monitoring server in a flow processing system, the method comprising:
acquiring flow data to be detected in a network;
based on the flow alarm rule in the rule configuration file of the detection engine in the monitoring server, detecting the flow data by using the detection engine according to the flow alarm rule so as to generate alarm information; the flow alarm rules in the rule configuration file are obtained by periodically polling alarm rules configured by a user;
and outputting the alarm information to an information acquisition end.
Optionally, the acquiring the traffic data to be detected in the network includes:
mirroring the flow data in the network to a designated network card based on a mirroring port, and reading a flow acquisition mode defined in a configuration file;
and polling the appointed network card according to the flow acquisition mode based on the acquisition service corresponding to the detection engine so as to acquire the flow data as the flow data to be detected.
Optionally, before the detecting engine is used to detect the flow data according to the flow alarm rule based on the flow alarm rule in the rule configuration file of the detecting engine in the monitoring server to generate alarm information, the method further includes:
Periodically polling data stored in a specified location to pull alert rules configured by the user from the specified location; the alarm rule configured by the user is configured by the user based on a preset rule configuration page;
writing the alarm rule configured by the user into the rule configuration file;
the detecting the flow data by the detecting engine according to the flow alarming rule to generate alarming information comprises the following steps:
and for any flow data, matching the flow data with the flow alarm rule by utilizing the detection engine, and generating corresponding alarm information for the flow data under the condition that the flow data is matched with the flow alarm rule.
Optionally, the method further comprises:
periodically polling the running state of the detection engine through a designated data processing tool in the monitoring server to acquire state information of the detection engine;
and sending state information of the detection engine to a user at regular time based on the specified data processing tool.
Optionally, the method further comprises:
periodically pulling a control instruction through a designated data processing tool in the monitoring server;
Under the condition that a control instruction for representing stopping acquisition is pulled, sending an acquisition stopping command indicated by the control instruction to the detection engine so as to stop acquisition services corresponding to the detection engine;
and under the condition that a control instruction for representing starting acquisition is pulled, sending an acquisition starting command indicated by the control instruction to the detection engine so as to start acquisition service corresponding to the detection engine.
Optionally, the flow alert rule includes an IP address whitelist, and for any one of the flow data, matching the flow data with the flow alert rule by using the detection engine, and generating corresponding alert information for the flow data when the flow data is matched with the flow alert rule, where the method includes:
acquiring an IP address corresponding to the flow data to be detected;
and under the condition that the IP address does not belong to the IP address white list, generating corresponding alarm information for the flow data by utilizing the detection engine.
In a second aspect, the present invention provides a flow data processing apparatus for use in a monitoring server in a flow processing system, the apparatus comprising:
The first acquisition module is used for acquiring flow data to be detected in the network;
the first detection module is used for detecting the flow data by utilizing the detection engine according to the flow alarm rule based on the flow alarm rule in the rule configuration file of the detection engine in the monitoring server so as to generate alarm information; the flow alarm rules in the rule configuration file are obtained by periodically polling alarm rules configured by a user;
the first output module is used for outputting the alarm information to the information acquisition end.
Optionally, the first obtaining module includes:
the first reading module is used for mirroring the flow data in the network to a designated network card based on the mirroring port and reading a flow acquisition mode defined in the configuration file;
the first acquisition module is used for polling the appointed network card according to the flow acquisition mode based on the acquisition service corresponding to the detection engine so as to acquire the flow data as the flow data to be detected.
In a third aspect, the present invention provides an electronic device comprising: a processor, a memory and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the above-mentioned flow data processing method when executing the program.
In a fourth aspect, the present invention provides a readable storage medium, which when executed by a processor of an electronic device, enables the electronic device to perform the above-described traffic data processing method.
In the embodiment of the invention, the flow data to be detected in the network is obtained; based on the flow alarm rule in the rule configuration file of the detection engine in the monitoring server, detecting flow data by using the detection engine according to the flow alarm rule so as to generate alarm information; the flow alarm rules in the rule configuration file are obtained by periodically polling alarm rules configured by a user; and outputting alarm information to the information acquisition end. In this way, the user autonomously configures the alarm rule, polls the alarm rule to obtain the flow alarm rule for the detection engine to detect the flow data, the user can realize the configuration management of the flow alarm rule in the detection engine only by configuring and managing the alarm rule in the flow processing system, the management of the flow alarm rule by adding or modifying a command line in a server can be avoided, the alarm rule configured and managed by the user in the system is utilized to detect the flow data, so that the rule is easier to configure and manage, the user operation is easier, the simplicity of configuring the flow alarm rule is improved, the modification process is simplified, and the complexity of the modification process is reduced.
Meanwhile, under the condition that the flow alarm rule needs to be modified, the alarm rule is only required to be configured and modified at the user side, so that the convenience in managing the flow alarm rule is improved to a certain extent.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of steps of a flow data processing method according to an embodiment of the present invention;
FIG. 2 is a system diagram of a flow data processing method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a specific example provided by an embodiment of the present invention;
FIG. 4 is a block diagram of a flow data processing apparatus according to an embodiment of the present invention;
fig. 5 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Fig. 1 is a flow chart of steps of a flow data processing method according to an embodiment of the present invention, which is applied to a monitoring server in a flow processing system.
As shown in fig. 1, the method may include:
and 101, acquiring flow data to be detected in a network.
In the embodiment of the invention, a user accesses an internet application through a network, and the generated flow data is processed by a flow processing system, wherein the flow data generated by the user accessing the internet application through the network is the flow data in the network. Specifically, the traffic data in the network can be obtained in real time through related protocols such as HyperText transfer protocol (HTTP, hyperText Transfer Protocol), simple Network Management Protocol (SNMP) and the like, the number of network traffic data is counted, and the collected traffic data is stored in a corresponding storage server. The flow data may be an access record of the user, and one flow data may correspond to one access request.
And 102, detecting the flow data by using the detection engine according to the flow alarm rule based on the flow alarm rule in the rule configuration file of the detection engine in the monitoring server so as to generate alarm information.
The flow alarm rules in the rule configuration file are obtained by periodically polling alarm rules configured by a user. The user can configure the alarm rule according to the requirement in the flow processing system, and the flow alarm rule according to which the detection engine detects the flow data is determined by periodically polling the alarm rule.
Network security personnel may use network traffic analysis to identify any malicious or suspicious packets in the traffic. Likewise, network operators may also use network traffic analysis to monitor download/upload speed, throughput, network transmission performance, various protocol transmission quality, applications and traffic, etc., to learn about network operation and perspective network activity.
In the embodiment of the invention, based on the flow alarm rule, the flow data is detected by using the detection engine, and the obtained detection result is recorded, namely, alarm information corresponding to the flow data is generated. The detection engine may be a cyber threat detection engine deployed in a monitoring server for traffic analysis, and by way of example, the detection engine may be a surica engine or other engine having similar functionality, as the embodiments of the invention are not limited in this regard.
Step 103, outputting the alarm information to an information acquisition end.
In the embodiment of the present invention, the information obtaining end may be a specified file or a specified message system in the flow processing system, where the specified message system is as follows: the kafka system. The output file format may be JSON format, which is not limited by the embodiments of the present invention. In one possible implementation, the surica engine may output traffic and alert information to a designated file or directly into kafka for use by the traffic processing system.
In summary, in the embodiment of the present invention, the flow data to be detected in the network is obtained; based on the flow alarm rule in the rule configuration file of the detection engine in the monitoring server, detecting flow data by using the detection engine according to the flow alarm rule so as to generate alarm information; the flow alarm rules in the rule configuration file are obtained by periodically polling alarm rules configured by a user; and outputting alarm information to the information acquisition end. In this way, the user autonomously configures the alarm rule and polls the alarm rule to obtain the flow alarm rule for the detection engine to detect the flow data, the user can realize the configuration management of the flow alarm rule in the detection engine only by configuring and managing the alarm rule in the flow processing system, the management of the flow alarm rule by adding or modifying a command line in a server can be avoided, the flow data is detected by utilizing the alarm rule configured and managed by the user in the system, so that the rule is easier to configure and manage, the user operation is easier, the simplicity of configuring the flow alarm rule is improved, and meanwhile, the convenience of managing the flow alarm rule is improved to a certain extent only by configuring and modifying the alarm rule at the user side under the condition that the flow alarm rule needs to be modified.
Optionally, step 101 may include the steps of:
and 1011, mirroring the flow data in the network to a designated network card based on the mirror port, and reading a flow acquisition mode defined in the configuration file.
In the embodiment of the invention, a user accesses the Internet application, the generated flow data is interacted through the switch, and the flow data is mirrored to the appointed network card through the mirror image port so as to be used for flow collection by the flow collection tool. And reading a flow collection mode defined in a configuration file of the monitoring server, wherein the flow collection mode is a mode of collecting flow by using a specified flow collection tool, and the flow collection tool is a tool deployed in the monitoring server and used for collecting flow.
In one possible implementation, the flow harvesting tool may be a data plane development kit (DPDK, data Plane Development Kit). Specifically, a mirror port is opened on the network device, all original traffic in the physical interface to be monitored is copied to the mirror port, the mirror port is connected to a designated network card, the designated network card is bound to the DPDK in an execution script file to collect traffic, and meanwhile, the necessary configuration of DPDK operation such as driving and large page memory is configured in the execution script file.
Step 1012, polling the designated network card according to the flow collection mode based on the collection service corresponding to the detection engine, so as to collect the flow data as the flow data to be detected.
In the embodiment of the invention, the content in the configuration file in the detection engine is configured as the flow collection mode, and the detection engine collects the flow based on the flow collection tool corresponding to the flow collection mode so as to provide collection service. And polling the flow data of the appointed network card according to the flow acquisition mode, namely polling the flow data of the appointed network card by utilizing a flow acquisition tool corresponding to the flow acquisition mode, wherein all acquired flow data are flow data to be detected by the detection engine.
In a possible implementation manner, the DPDK is configured as a flow collection tool corresponding to the detection engine in a configuration file of the detection engine, so that the detection engine is started in a manner of collecting flow by the DPDK, that is, after the DPDK is used for collecting flow data, the detection engine is used for detecting the flow data, so that the efficiency of flow collection can be improved.
In the embodiment of the invention, the flow data in the network is mirrored to the designated network card, and then the designated network card is polled for flow collection based on the flow collection mode defined in the configuration file.
Optionally, before step 102, an embodiment of the present invention may include the following steps:
step 201, periodically polling data stored in a designated location to pull the alarm rule configured by the user from the designated location; the alarm rules configured by the user are configured by the user based on preset rule configuration pages.
In the embodiment of the invention, the user can perform configuration management on the alarm rule through the rule configuration page in the system. The designated location may be a memory space in the flow processing system that may include associated configuration files therein by accessing data stored at the designated location and retrieving data characterizing the alert rules. Specifically, the log collection tool log kit can access at each interval for a preset time, and the alarm rules stored in the relevant configuration files are obtained, wherein the preset time can be set according to the needs, and the embodiment of the invention is not limited to the preset time.
Step 202, writing the alarm rule configured by the user into the rule configuration file.
In the embodiment of the invention, after the alarm rule configured by the user is acquired, the alarm rule is written into the rule configuration file in the detection engine. In this way, the alarm rules in the flow processing system are periodically polled and acquired, and the acquired alarm rules are written into the rule configuration file, so that the purpose of changing the flow alarm rules in the detection engine can be realized by only modifying the alarm rules stored in the relevant configuration file in the flow processing system and periodically polling the alarm rules under the condition that the alarm rules need to be changed.
Accordingly, the step 102 may include the steps of:
step 203, for any one of the flow data, matching the flow data with the flow alarm rule by using the detection engine, and generating corresponding alarm information for the flow data when the flow data is matched with the flow alarm rule.
In the embodiment of the invention, the detection engine matches the flow data with the flow alarm rule, which can be the field matching of binary protocol, mainly the comparison of numerical values, or the content matching of transmission layer protocol, mainly the matching search of character strings. And under the condition that the flow data is matched with the flow alarm rule, recording the matching result to generate alarm information corresponding to the flow data.
Optionally, in the case that the flow alert data matches the flow alert rule, the alert information may be processed according to a processing rule defined in the detection engine, where the processing rule includes stopping scanning, blocking the flow data, releasing the flow data, and recording the corresponding alert information.
In the embodiment of the invention, the rule configuration page is used for configuring the alarm rule by a user, and the rule configuration file of the detection engine is written into the rule configuration page as the flow alarm rule, so that the rule configuration can be avoided through commands in the server, and the rule configuration page is only required to be modified under the condition that the flow alarm rule needs to be modified, thereby not only improving the simplicity and convenience of rule modification, but also reducing the maintenance cost of a user side. And by periodically polling the alarm rules at the designated positions, the configuration files of the detection engine can be timely polled and written when the alarm rules change, so that the requirement of changing the flow alarm rules according to the needs in daily use is met.
Optionally, the embodiment of the invention further comprises the following steps:
step 301, periodically polling the running state of the detection engine by a designated data processing tool in the monitoring server to obtain the state information of the detection engine.
In the embodiment of the present invention, the designated data processing tool may be a log kit tool, or may be another processing tool with a similar function, which is not limited in this embodiment of the present invention. For example, in the case where the designated data processing tool is a logkit, the surica and DPDK integrated logkit may be installed to the monitoring server to enable traffic collection and traffic monitoring. Periodic polling may be accomplished by the designated data processing tool sending heartbeat information to the detection engine at intervals, the detection engine returning a designated message to the designated data processing tool after receiving the heartbeat information indicating its own operational status. After receiving the specified message, the specified data processing tool may use the specified message as the state information of the detection engine, or may generate the state information of the corresponding detection engine according to the specified message.
The time interval can be set according to the needs, and the specified message can be a normal operation message or an abnormal operation message. It will be appreciated that the normal operation message indicates that the detection engine itself is operating normally and the abnormal operation message indicates that the detection engine itself is operating abnormally. And within the preset time after the data processing tool sends the heartbeat information, if the data processing tool does not receive the designated message returned by the detection engine, indicating that the detection engine is abnormal in operation.
Step 302, sending state information of the detection engine to a user based on the specified data processing tool.
In the embodiment of the invention, after the specified data processing tool acquires the state information of the detection engine, the state information is sent to the user at regular time, wherein the specified data processing tool can upload the state information to the flow processing system, and the user can acquire the state information by accessing a specified interface of the flow processing system so as to monitor the state of the detection engine. Specifically, when the data processing tool is designated as a logkit, the logkit can be utilized to call an interface of the flow processing system at regular time through a heartbeat mechanism, and a user can access the flow processing system through a browser to monitor the running state of the detection engine. The state information may be sent to the user at fixed time or may be sent at preset time intervals, which is not limited in the embodiment of the present invention.
In the embodiment of the invention, the processing state of the detection engine is polled by the data processing tool and the state information is sent to the user, so that the user can monitor the running state of the detection engine in real time and can acquire the related information of the flow acquisition and the alarm state in time by sending the state information to the user.
Optionally, the embodiment of the invention further comprises the following steps:
step 401, periodically pulling a control instruction by a designated data processing tool in the monitoring server.
In the embodiment of the invention, the control instruction is pulled every preset time by the appointed data processing tool, wherein the control instruction can be a configuration interface of a user in a flow processing system, and the generation of the corresponding control instruction is triggered by clicking a button capable of controlling the start and stop of the acquisition service. It can be understood that when the state of the acquisition service corresponding to the current detection engine is an on state, that is, the detection engine is executing the operations of flow acquisition and flow alarm, clicking a button for controlling to start and stop the acquisition service once triggers to generate the first control instruction. The control instructions may include a first control instruction and a second control instruction, where the first control instruction characterizes stopping acquisition and the second control instruction characterizes starting acquisition. When the state of the acquisition service corresponding to the current detection engine is in a closed state, namely the detection engine does not execute the operations of flow acquisition and flow alarming, clicking a button for controlling the start and stop of the acquisition service once triggers the generation of a second control instruction.
And step 402, under the condition that a control instruction for representing stopping acquisition is pulled, sending an acquisition stopping command indicated by the control instruction to the detection engine so as to stop acquisition services corresponding to the detection engine.
In the embodiment of the invention, when the designated data processing tool pulls the control instruction (the first control instruction) representing stopping acquisition, the designated data processing tool sends the acquisition stopping command indicated by the control instruction to the detection engine, and the detection engine stops corresponding acquisition service when receiving the acquisition stopping command.
Step 403, under the condition that a control instruction for representing starting acquisition is pulled, sending a starting acquisition command indicated by the control instruction to the detection engine so as to start acquisition service corresponding to the detection engine.
In the embodiment of the invention, when the designated data processing tool pulls the control instruction (the second control instruction) representing the start acquisition, the designated data processing tool sends the start acquisition command indicated by the control instruction to the detection engine, and the detection engine starts the corresponding acquisition service when receiving the start acquisition command.
According to the embodiment of the invention, the control instruction triggered by the user is regularly pulled by the appointed data processing tool, so that the acquisition service of the detection engine can be started and stopped conveniently, the acquisition service is stopped or started without modifying the command in the server, the convenience of starting and stopping the detection engine is improved, and the operation and maintenance cost is reduced to a certain extent.
Optionally, the traffic alert rule includes an internet protocol (IP, internet Protocol) address whitelist.
In the embodiment of the invention, the IP address white list can comprise one or more IP addresses, and the IP address white list is used for representing that traffic alarm processing is not performed on traffic data corresponding to the IP addresses in the list.
Step 203 may comprise the steps of:
step 2031, obtaining an IP address corresponding to the flow data to be detected.
In the embodiment of the invention, the IP address may be a source port address of the traffic data to be detected, and one IP address may correspond to a plurality of traffic data. When the flow data is acquired, the flow data carries corresponding data information, and the data information can contain a source port IP address and a target port IP address corresponding to the flow data.
Step 2032, generating, by using the detection engine, corresponding alarm information for the traffic data, in a case where the IP address does not belong to the IP address whitelist.
In the embodiment of the invention, the acquired IP address is matched with all IP addresses contained in the IP address white list, so that the alarm information of the flow data corresponding to the IP address is characterized to be recorded under the condition that the IP address cannot be matched with any IP address contained in the IP address white list, namely, under the condition that the IP address corresponding to the flow data to be detected does not belong to the IP address white list. The corresponding alarm information is generated for the flow data by using the detection engine, and the relevant information of the flow data can be recorded according to the appointed data format to generate the corresponding alarm information under the condition that the flow data is matched with the flow alarm rule. The specified data format may be json, log, pcap format, and the alarm information may include a timestamp corresponding to the traffic data, quintuple information (generally referred to as a source IP address, a source port, a destination IP address, a destination port, and a transport layer protocol), corresponding signature information, and so on.
Similarly, when the IP address belongs to the IP address white list, no alarm information corresponding to the traffic data is generated. And under the condition that the IP address can be matched with any one IP address contained in the IP address white list, namely under the condition that the IP address corresponding to the flow data to be detected belongs to the IP address white list, the alarm information corresponding to the IP address is characterized without recording.
In the embodiment of the invention, the IP address white list is arranged in the flow alarm rule, so that the IP address which does not need to carry out flow alarm can be flexibly released according to the need, the alarm information is not recorded, the processing energy consumption of the server is further reduced to a certain extent, and the data processing efficiency is improved.
Fig. 2 is a system diagram of a flow data processing method according to an embodiment of the present invention. As shown in fig. 2, the surica and DPDK integrated log kit are installed to the monitoring server, the user uses the internet service at the client to access the internet application, the traffic data is interacted through the switch, the traffic data is mirrored to Suricata, DPDK and the log kit integrated service through the mirror port by directly connecting the network cable, and the traffic analysis and alarm processing is performed in combination with the rules in the surica rule configuration file, and the obtained alarm information is output to the designated file or the message system, wherein the message system may be: the kafka system.
Fig. 3 is a schematic diagram of a specific example provided by the embodiment of the present invention, where, as shown in fig. 3, a detection engine adopts a surica engine, a designated data processing tool adopts a log kit, and a traffic collection mode adopts DPDK to collect traffic. Specifically, the traffic data is mirrored to the physical machine network card installed in a DPDK binding mode through a mirror port in a network cable direct connection mode, a configuration file of a logkit starting Surica is modified, and the Surica is configured to start in a manner of collecting traffic through the DPDK. The logkit pulls Intrusion Detection System (IDS)/Intrusion Prevention System (IPS) alarm rule data configured in a rule configuration page by means of timing polling, and writes the alarm rule data into a rule configuration file of surica. And after receiving the flow data acquired by the DPDK, the Surica obtains alarm information by matching IDS/IPS alarm rules, and outputs the alarm information to a message system or a designated file for other applications in the flow processing system.
Fig. 4 is a block diagram of a flow data processing apparatus according to an embodiment of the present invention, which is applied to a monitoring server in a flow processing system, where the apparatus 50 may include:
a first obtaining module 501, configured to obtain traffic data to be detected in a network.
The first detection module 502 is configured to detect, based on a flow alert rule in a rule configuration file of a detection engine in the monitoring server, the flow data according to the flow alert rule by using the detection engine, so as to generate alert information; the flow alarm rules in the rule configuration file are obtained by periodically polling alarm rules configured by a user.
A first output module 503, configured to output the alarm information to an information acquisition end.
The first obtaining module 501 includes:
the first reading module is used for mirroring the flow data in the network to a designated network card based on the mirroring port and reading a flow acquisition mode defined in the configuration file;
the first acquisition module is used for polling the appointed network card according to the flow acquisition mode based on the acquisition service corresponding to the detection engine so as to acquire the flow data as the flow data to be detected.
Optionally, the apparatus 50 may further include:
a first pulling module for periodically polling data stored in a specified location to pull the alert rule configured by the user from the specified location; the alarm rules configured by the user are configured by the user based on preset rule configuration pages.
And the first writing module is used for writing the alarm rule configured by the user into the rule configuration file.
Accordingly, the first detection module 502 may further include:
and the first matching module is used for matching the flow data with the flow alarm rules by utilizing the detection engine for any flow data, and generating corresponding alarm information for the flow data under the condition that the flow data is matched with the flow alarm rules.
Optionally, the first matching module may include:
and the second acquisition module is used for acquiring the IP address corresponding to the flow data to be detected.
And the first generation module is used for generating corresponding alarm information for the flow data by utilizing the detection engine under the condition that the IP address does not belong to the IP address white list.
Optionally, the apparatus 50 may further include:
And the third acquisition module is used for periodically polling the running state of the detection engine through a designated data processing tool in the monitoring server so as to acquire the state information of the detection engine.
And the first sending module is used for sending the state information of the detection engine to a user at regular time based on the designated data processing tool.
Optionally, the apparatus 50 may further include:
and the second pulling module is used for periodically pulling the control instruction through a designated data processing tool in the monitoring server.
And the third sending module is used for sending a collection stopping command indicated by the control command to the detection engine under the condition that the control command for representing collection stopping is pulled, so as to stop the collection service corresponding to the detection engine.
And the fourth sending module is used for sending a collection starting command indicated by the control command to the detection engine under the condition that the control command for representing the collection starting is pulled, so as to start the collection service corresponding to the detection engine.
The present invention also provides an electronic device, see fig. 5, comprising: a processor 601, a memory 602 and a computer program 6021 stored on the memory and executable on the processor, which when executed implements the flow data processing method of the foregoing embodiment.
The present invention also provides a readable storage medium which, when executed by a processor of an electronic device, enables the electronic device to perform the traffic data processing method of the foregoing embodiment.
For the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments for relevant points.
The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general-purpose systems may also be used with the teachings herein. The required structure for a construction of such a system is apparent from the description above. In addition, the present invention is not directed to any particular programming language. It will be appreciated that the teachings of the present invention described herein may be implemented in a variety of programming languages, and the above description of specific languages is provided for disclosure of enablement and best mode of the present invention.
In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be construed as reflecting the intention that: i.e., the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules in the apparatus of the embodiments may be adaptively changed and disposed in one or more apparatuses different from the embodiments. The modules or units or components of the embodiments may be combined into one module or unit or component and, furthermore, they may be divided into a plurality of sub-modules or sub-units or sub-components. Any combination of all features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or units of any method or apparatus so disclosed, may be used in combination, except insofar as at least some of such features and/or processes or units are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings), may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that some or all of the functions of some or all of the components in a sorting device according to the present invention may be implemented in practice using a microprocessor or Digital Signal Processor (DSP). The present invention may also be implemented as an apparatus or device program for performing part or all of the methods described herein. Such a program embodying the present invention may be stored on a computer readable medium, or may have the form of one or more signals. Such signals may be downloaded from an internet website, provided on a carrier signal, or provided in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third, etc. do not denote any order. These words may be interpreted as names.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.
The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present invention. Therefore, the protection scope of the invention is subject to the protection scope of the claims.
Claims (9)
1. A method of traffic data processing, characterized by being applied to a monitoring server in a traffic processing system, the method comprising:
acquiring flow data to be detected in a network;
based on the flow alarm rule in the rule configuration file of the detection engine in the monitoring server, detecting the flow data by using the detection engine according to the flow alarm rule so as to generate alarm information; the flow alarm rules in the rule configuration file are obtained by periodically polling alarm rules configured by a user;
Outputting the alarm information to an information acquisition end;
periodically polling the running state of the detection engine through a designated data processing tool in the monitoring server to acquire state information of the detection engine;
sending state information of the detection engine to a user at regular time based on the specified data processing tool;
wherein, before the detecting engine is utilized to detect the flow data according to the flow alarming rule based on the flow alarming rule in the rule configuration file of the detecting engine in the monitoring server to generate alarming information, the method further comprises:
periodically polling data stored in a specified location to pull alert rules configured by the user from the specified location; the alarm rules configured by the user are configured by the user based on preset rule configuration pages.
2. The method according to claim 1, wherein the obtaining traffic data to be detected in the network comprises:
mirroring the flow data in the network to a designated network card based on a mirroring port, and reading a flow acquisition mode defined in a configuration file;
and polling the appointed network card according to the flow acquisition mode based on the acquisition service corresponding to the detection engine so as to acquire the flow data as the flow data to be detected.
3. The method of claim 1, wherein the method further comprises, prior to detecting the flow data with the detection engine according to the flow alert rules based on the flow alert rules in the rule profile of the detection engine in the monitoring server to generate alert information:
writing the alarm rule configured by the user into the rule configuration file;
the detecting the flow data by the detecting engine according to the flow alarming rule to generate alarming information comprises the following steps:
and for any flow data, matching the flow data with the flow alarm rule by utilizing the detection engine, and generating corresponding alarm information for the flow data under the condition that the flow data is matched with the flow alarm rule.
4. A method according to any one of claims 1-3, wherein the method further comprises:
periodically pulling a control instruction through a designated data processing tool in the monitoring server;
under the condition that a control instruction for representing stopping acquisition is pulled, sending an acquisition stopping command indicated by the control instruction to the detection engine so as to stop acquisition services corresponding to the detection engine;
And under the condition that a control instruction for representing starting acquisition is pulled, sending an acquisition starting command indicated by the control instruction to the detection engine so as to start acquisition service corresponding to the detection engine.
5. The method of claim 3, wherein the traffic alert rule comprises a whitelist of IP addresses, and wherein for any of the traffic data, matching the traffic data with the traffic alert rule using the detection engine, and generating corresponding alert information for the traffic data if the traffic data matches the traffic alert rule, comprises:
acquiring an IP address corresponding to the flow data to be detected;
and under the condition that the IP address does not belong to the IP address white list, generating corresponding alarm information for the flow data by utilizing the detection engine.
6. A traffic data processing apparatus for use with a monitoring server in a traffic processing system, the apparatus comprising:
the first acquisition module is used for acquiring flow data to be detected in the network;
the first detection module is used for detecting the flow data by utilizing the detection engine according to the flow alarm rule based on the flow alarm rule in the rule configuration file of the detection engine in the monitoring server so as to generate alarm information; the flow alarm rules in the rule configuration file are obtained by periodically polling alarm rules configured by a user;
The first output module is used for outputting the alarm information to the information acquisition end;
a third acquisition module, configured to periodically poll, by a designated data processing tool in the monitoring server, an operation state of the detection engine, so as to acquire state information of the detection engine;
a first sending module, configured to send status information of the detection engine to a user at regular time based on the specified data processing tool;
a first pulling module for periodically polling data stored in a specified location to pull the alert rule configured by the user from the specified location; the alarm rules configured by the user are configured by the user based on preset rule configuration pages.
7. The apparatus of claim 6, wherein the first acquisition module comprises:
the first reading module is used for mirroring the flow data in the network to a designated network card based on the mirroring port and reading a flow acquisition mode defined in the configuration file;
the first acquisition module is used for polling the appointed network card according to the flow acquisition mode based on the acquisition service corresponding to the detection engine so as to acquire the flow data as the flow data to be detected.
8. An electronic device, comprising:
a processor, a memory and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the flow data processing method according to any of claims 1-5 when executing the program.
9. A readable storage medium, characterized in that instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the flow data processing method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211237558.2A CN115766079B (en) | 2022-10-10 | 2022-10-10 | Traffic data processing method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211237558.2A CN115766079B (en) | 2022-10-10 | 2022-10-10 | Traffic data processing method and device, electronic equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115766079A CN115766079A (en) | 2023-03-07 |
| CN115766079B true CN115766079B (en) | 2023-12-05 |
Family
ID=85351172
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211237558.2A Active CN115766079B (en) | 2022-10-10 | 2022-10-10 | Traffic data processing method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115766079B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107483448A (en) * | 2017-08-24 | 2017-12-15 | 中国科学院信息工程研究所 | A kind of network security detection method and detecting system |
| CN108683681A (en) * | 2018-06-01 | 2018-10-19 | 杭州安恒信息技术股份有限公司 | A kind of smart home intrusion detection method and device based on traffic policy |
| CN110430225A (en) * | 2019-09-16 | 2019-11-08 | 杭州安恒信息技术股份有限公司 | A kind of industrial equipment monitoring and managing method, device, equipment and readable storage medium storing program for executing |
| CN111917740A (en) * | 2020-07-15 | 2020-11-10 | 杭州安恒信息技术股份有限公司 | Abnormal flow alarm log detection method, device, equipment and medium |
| CN111935074A (en) * | 2020-06-22 | 2020-11-13 | 国网电力科学研究院有限公司 | Integrated network security detection method and device |
| CN112910842A (en) * | 2021-01-14 | 2021-06-04 | 中国电子科技集团公司第十五研究所 | Network attack event evidence obtaining method and device based on flow reduction |
| CN113542253A (en) * | 2021-07-12 | 2021-10-22 | 杭州安恒信息技术股份有限公司 | Network flow detection method, device, equipment and medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11223643B2 (en) * | 2019-11-07 | 2022-01-11 | Illumio, Inc. | Managing a segmentation policy based on attack pattern detection |
-
2022
- 2022-10-10 CN CN202211237558.2A patent/CN115766079B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107483448A (en) * | 2017-08-24 | 2017-12-15 | 中国科学院信息工程研究所 | A kind of network security detection method and detecting system |
| CN108683681A (en) * | 2018-06-01 | 2018-10-19 | 杭州安恒信息技术股份有限公司 | A kind of smart home intrusion detection method and device based on traffic policy |
| CN110430225A (en) * | 2019-09-16 | 2019-11-08 | 杭州安恒信息技术股份有限公司 | A kind of industrial equipment monitoring and managing method, device, equipment and readable storage medium storing program for executing |
| CN111935074A (en) * | 2020-06-22 | 2020-11-13 | 国网电力科学研究院有限公司 | Integrated network security detection method and device |
| CN111917740A (en) * | 2020-07-15 | 2020-11-10 | 杭州安恒信息技术股份有限公司 | Abnormal flow alarm log detection method, device, equipment and medium |
| CN112910842A (en) * | 2021-01-14 | 2021-06-04 | 中国电子科技集团公司第十五研究所 | Network attack event evidence obtaining method and device based on flow reduction |
| CN113542253A (en) * | 2021-07-12 | 2021-10-22 | 杭州安恒信息技术股份有限公司 | Network flow detection method, device, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115766079A (en) | 2023-03-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7544738B2 (en) | Detecting Sensitive Data Exposure Through Logging | |
| US9825973B2 (en) | Website security | |
| Kumar et al. | Signature based intrusion detection system using SNORT | |
| WO2022083226A1 (en) | Anomaly identification method and system, storage medium and electronic device | |
| US11361073B2 (en) | Analysis apparatus, analysis method, and analysis program | |
| US11100046B2 (en) | Intelligent security context aware elastic storage | |
| US20030084328A1 (en) | Method and computer-readable medium for integrating a decode engine with an intrusion detection system | |
| Roschke et al. | An extensible and virtualization-compatible IDS management architecture | |
| US12039048B2 (en) | System and method for automatic generation of malware detection traps | |
| CN104144142B (en) | A kind of Web bug excavation methods and system | |
| US20160234238A1 (en) | System and method for web-based log analysis | |
| CN104038466B (en) | Intruding detection system, method and apparatus for cloud computing environment | |
| US8661456B2 (en) | Extendable event processing through services | |
| JP2007334536A (en) | Behavior analysis system for malware | |
| CN114077525A (en) | Abnormal log processing method and device, terminal equipment, cloud server and system | |
| CN105550593A (en) | Cloud disk file monitoring method and device based on local area network | |
| CN111212035A (en) | Host computer defect confirming and automatic repairing method and system based on same | |
| CN113810408A (en) | Network attack organization detection method, device, equipment and readable storage medium | |
| Deeter et al. | APHIDS: A mobile agent-based programmable hybrid intrusion detection system | |
| US20200067981A1 (en) | Deception server deployment | |
| CN116208415A (en) | Method, device and equipment for managing API (application program interface) assets | |
| CN110138780B (en) | Method for realizing Internet of things terminal threat detection based on probe technology | |
| CN115766079B (en) | Traffic data processing method and device, electronic equipment and readable storage medium | |
| JP2022067092A (en) | Cyber security protection system and related proactive suspicious domain alert system | |
| CN105933186A (en) | Security detection method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |