CN111708681B - Log processing method, device, equipment and storage medium - Google Patents
Log processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111708681B CN111708681B CN202010542073.9A CN202010542073A CN111708681B CN 111708681 B CN111708681 B CN 111708681B CN 202010542073 A CN202010542073 A CN 202010542073A CN 111708681 B CN111708681 B CN 111708681B
- Authority
- CN
- China
- Prior art keywords
- log
- output
- information entropy
- level
- path
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 26
- 238000000034 method Methods 0.000 claims abstract description 155
- 238000004458 analytical method Methods 0.000 claims abstract description 93
- 238000012545 processing Methods 0.000 claims abstract description 34
- 238000004590 computer program Methods 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 9
- 230000003287 optical effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 239000013307 optical fiber Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 230000002401 inhibitory effect Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000011017 operating method Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
- G06F11/3093—Configuration details thereof, e.g. installation, enabling, spatial arrangement of the probes
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention discloses a log processing method, a log processing device, log processing equipment and a log processing storage medium. The method comprises the following steps: performing semantic analysis on all logs to be output in a preset log output cache pool to obtain a semantic analysis result of each log to be output, and processing the set log level of each log to be output according to the matching result of the set log level of each log to be output and the semantic analysis result; acquiring a method calling path of each log to be output, and calculating the information entropy of the method calling path of each log to be output; determining an output mode corresponding to each log to be output according to the information entropy of the method call path of each log to be output; and controlling the output of each log to be output according to the output mode corresponding to each log to be output. The embodiment of the invention can ensure that the actual output log level of the log is correct and the output position of the log is proper.
Description
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to a log processing method, a log processing device, log processing equipment and a log processing storage medium.
Background
Computer systems have become a part of everyday life today. The computer may faithfully record all the operating procedures and generate the information needed for the decision. With the continuous improvement of informatization construction of various industries, the operation of services is more and more concentrated on an information system or an information platform. The log information can record operation information and flow completely and really, the log is user behavior fact data, and all service data can be obtained from the log.
In the related art, the Log output framework Log4j is usually used to complete the output of the Log. Log4j log print statements are placed where errors may occur in the system. The log4j log printing statement is placed at the log output position. When the system runs, log printing statements are logged through log4j, and output of the log through the level is achieved. Log4j provides Log levels with five levels of DEBUG (DEBUG), Information (INFO), Warning (WARN), ERROR (ERROR), and FATAL exception (FATAL) in order of importance. The log format of output can also be set by log4j log print statements, which can specify log output to a console, and output to a file. However, the general Log output framework Log4j in the related art cannot guarantee that the actual Log output level of the Log is correct and the Log output position is proper.
Disclosure of Invention
Embodiments of the present invention provide a log processing method, an apparatus, a device, and a storage medium, so as to optimize an existing log processing manner, ensure that a log level actually output by a log is correct, and a log output position is appropriate.
In a first aspect, an embodiment of the present invention provides a log processing method, including:
performing semantic analysis on all logs to be output in a preset log output cache pool to obtain a semantic analysis result of each log to be output, and processing the set log level of each log to be output according to the matching result of the set log level of each log to be output and the semantic analysis result;
acquiring a method calling path of each log to be output, and calculating the information entropy of the method calling path of each log to be output;
determining an output mode corresponding to each log to be output according to the information entropy of the method call path of each log to be output;
and controlling the output of each log to be output according to the output mode corresponding to each log to be output.
In a second aspect, an embodiment of the present invention further provides a log processing apparatus, including:
the level matching module is used for performing semantic analysis on all logs to be output in a preset log output cache pool to obtain a semantic analysis result of each log to be output, and processing the set log level of each log to be output according to the matching result of the set log level of each log to be output and the semantic analysis result;
the information entropy calculation module is used for acquiring the method calling path of each log to be output and calculating the information entropy of the method calling path of each log to be output;
the output mode determining module is used for determining the output mode corresponding to each log to be output according to the information entropy of the method calling path of each log to be output;
and the output control module is used for controlling the output of each log to be output according to the output mode corresponding to each log to be output.
In a third aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the log processing method according to the embodiment of the present invention.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the log processing method according to the embodiment of the present invention.
The technical scheme of the embodiment of the invention obtains the semantic analysis result of each log to be output by performing semantic analysis on all the logs to be output in a preset log output cache pool, processes the set log level of each log to be output according to the matching result of the set log level of each log to be output and the semantic analysis result, obtains the method calling path of each log to be output, calculates the information entropy of the method calling path of each log to be output, determines the output mode corresponding to each log to be output according to the information entropy of the method calling path of each log to be output, controls the output of each log to be output according to the output mode corresponding to each log to be output, determines whether the log level of the log to be output is correct or not by matching the set log level of the log to be output and the semantic analysis result, and can determine the information entropy of the method calling path, and determining whether the log output position corresponding to the log to be output is proper or not, so that the log level of the actual output of the log can be ensured to be correct, and the log output position is proper.
Drawings
Fig. 1 is a flowchart of a log processing method according to an embodiment of the present invention.
Fig. 2 is a flowchart of a log processing method according to a second embodiment of the present invention.
Fig. 3 is a schematic structural diagram of a log processing apparatus according to a third embodiment of the present invention.
Fig. 4 is a schematic structural diagram of a computer device according to a fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention.
It should be further noted that, for the convenience of description, only some but not all of the relevant aspects of the present invention are shown in the drawings. Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
Example one
Fig. 1 is a flowchart of a log processing method according to an embodiment of the present invention. The embodiment of the invention can be suitable for processing the log to be output, and can ensure that the actual output log level of the log is correct and the output position of the log is proper. As shown in fig. 1, the method of the embodiment of the present invention specifically includes:
step 101, performing semantic analysis on all logs to be output in a preset log output cache pool to obtain a semantic analysis result of each log to be output, and processing the set log level of each log to be output according to a matching result of the set log level of each log to be output and the semantic analysis result.
In this embodiment, the system includes a plurality of code blocks, and each code block is used to implement a different service function. A log output framework is deployed in the system in advance, and log printing statements are placed in at least one code block where errors can occur in the system. When the system runs, each log printing statement records the execution process information of the code block, generates a log of the code block, and then sends the generated log of the code block to a preset log output cache pool as a to-be-output log corresponding to the code block. The preset log output cache pool is used for storing the log to be output.
Optionally, performing semantic analysis on all logs to be output in a preset log output cache pool to obtain a semantic analysis result of each log to be output, where the semantic analysis result includes: and performing semantic analysis on all logs to be output in a preset log output cache pool through a preset semantic analysis model to obtain a semantic analysis result of each log to be output.
Specifically, the log texts of a plurality of logs and the correct log categories of the logs are used as training samples, and the machine learning model is trained to obtain a preset semantic analysis model. The input of the preset semantic analysis model is a log text, and the output of the preset semantic analysis model is a log level. And sequentially inputting the log text of each log to be output in a preset log output cache pool to a preset semantic analysis model so that the preset semantic analysis model performs semantic analysis on the log text of each log to be output and outputs the log grade of each log to be output. And the semantic analysis result of each log to be output is the log level of the log to be output, which is determined according to the semantics of the log text.
Optionally, the processing the set log level of each log to be output according to the matching result between the set log level of each log to be output and the semantic analysis result may include: judging whether the set log level of each log to be output is matched with the semantic analysis result; if the set log level of each log to be output is determined to be matched with the semantic analysis result, keeping the set log level of each log to be output; and if the set log level of the log to be output is determined to be not matched with the semantic analysis result, updating the set log level of the log to be output into the semantic analysis result of the log to be output.
In this embodiment, the log level set for each log to be output is the log level set for the log to be output by the log print statement. And judging whether the set log level of each log to be output is matched with the semantic analysis result, namely judging whether the log level set for the log to be output by the log printing statement is matched with the log level of the log to be output determined according to the semantics of the log text.
And if the set log level of each log to be output is matched with the semantic analysis result, the log level set for the log to be output by the log printing statement is matched with the log level of the log to be output determined according to the semantics of the log text, and the set log level of each log to be output is correct, the set log level of each log to be output is maintained.
And if the set log level of the log to be output is determined to be not matched with the semantic analysis result, indicating that the log level set for some logs to be output by the log printing statement is not matched with the log level of some logs to be output determined according to the semantics of the log text, and the set log level of the log to be output is incorrect, updating the set log level of the log to be output into the semantic analysis result of the log to be output.
Optionally, a Log output framework Log4j is deployed in the system in advance, and a Log4j Log print statement is placed in at least one code block where an error may occur in the system. The log level has five levels of Debugging (DEBUG), Information (INFO), Warning (WARN), ERROR (ERROR), and FATAL exception (FATAL) in order of the degree of importance.
And 102, acquiring a method calling path of each log to be output, and calculating the information entropy of the method calling path of each log to be output.
Optionally, the obtaining the method call path of each log to be output may include: and acquiring a method calling path of each log to be output through a preset path acquisition statement.
The path acquisition statement is a statement for acquiring a method call path. The method is a set of statements that accomplish a certain function. During execution of the code block, different methods may be called in sequence. The method call path is a path in which the code block calls a method. Illustratively, method a () calls method b () and method b () calls method c () during execution of the code block. The method call path of the code block is "a → b → c".
Specifically, a line of path acquisition statement log () is added to the last of the code block where each log printing statement is located. And acquiring a method calling path of each code block as a corresponding method calling path of the log to be output in the execution process of each code block through a preset path acquisition statement.
Optionally, calculating the information entropy of each method call path of the log to be output may include: and calculating the information entropy of the method call path of each log to be output according to the following information entropy calculation formula:
H(xi)＝-p(xi)·log2 p(xi)，
wherein, H (x)i) Information entropy, p (x), of method call path to log for target to be outputi) Probability of calling a path for executing a method of a target to-be-output Log, LiFrequency of calling paths for methods of a target to-be-output log, L0The total number of paths for the method call.
Specifically, the target log to be output is a currently processed log to be output. Total number of method call paths L0Is the total number of the method call paths in the preset log output cache pool. Frequency L of method call path of target to-be-output logiIs the number of occurrences of the method call path of the target log to be output.
Illustratively, the preset log output cache pool stores 100 logs to be output. One party corresponding to each log to be outputA method call path. The total number of the method call paths in the preset log output cache pool is 100. I.e. the total number of method call paths L0Is 100. In all the method call paths in the preset log output cache pool, the method call path of the target log to be output appears for 2 times. I.e. frequency L of method call paths of the target to-be-output logiIs 2. The probability of executing the method call path of the target log to be output is 0.02.
Therefore, the information entropy of the method call path of each log to be output is calculated according to the information entropy calculation formula.
And 103, determining an output mode corresponding to each log to be output according to the information entropy of the method call path of each log to be output.
Optionally, the determining, according to the information entropy of the method call path of each log to be output, an output mode corresponding to each log to be output may include: sequentially acquiring one log to be output in each log to be output as a target log to be output; judging whether the information entropy of the method calling path of the target log to be output is larger than a preset information entropy threshold value or not; if the information entropy of the method calling path of the target log to be output is larger than a preset information entropy threshold, determining an output mode corresponding to the target log to be output as reserved output; if the information entropy of the method calling path of the target log to be output is less than or equal to a preset information entropy threshold, determining that the output mode corresponding to the target log to be output is not output; and returning to execute the operation of sequentially acquiring one log to be output in the logs to be output as a target log to be output until the processing of all the logs to be output is completed.
The information entropy of the method call path represents the information amount of the method call path and also represents the information amount of the corresponding log to be output. The log information to be output with larger information entropy of the method calling path is important information for fault positioning of the system, and shows that the corresponding position for placing the log printing statement is appropriate, namely the position for outputting the log is appropriate, so that important log data can be obtained. The log information to be output with smaller information entropy of the method calling path is smaller, is important information for fault positioning of the system, and indicates that the corresponding position for placing the log printing statement is not appropriate, namely the corresponding position for outputting the log is not appropriate, so that important log data cannot be acquired.
Specifically, the output mode corresponding to each log to be output is determined according to the information entropy of the method call path of each log to be output in the preset log output cache pool: and determining an output mode corresponding to the log to be output as reserved output aiming at the log to be output with the information entropy of the method calling path larger than a preset information entropy threshold value, namely the corresponding log output position is the appropriate log to be output. And determining an output mode corresponding to the log to be output as non-output aiming at the log to be output with the information entropy of the method calling path smaller than a preset information entropy threshold value, namely the log to be output with the output position unsuitable. The preset information entropy threshold value can be set according to the service requirement. Therefore, the output position of the log corresponding to the log output subsequently can be ensured to be appropriate.
And 104, controlling the output of each log to be output according to the output mode corresponding to each log to be output.
Optionally, the controlling the output of each log to be output according to the output mode corresponding to each log to be output may include: if the output mode corresponding to the target log to be output is reserved output, outputting the target log to be output to a corresponding target position; and if the output mode corresponding to the target log to be output is non-output, determining not to output the target log to be output.
Specifically, the output of each log to be output is controlled in sequence according to the output mode corresponding to each log to be output in a preset log output cache pool: outputting the log to be output, which is output in a reserved manner, to a corresponding target position, wherein the target position can be a console or a file; and the corresponding output mode is not output, namely the output-reserved log to be output is reserved. Therefore, the effect that only the output position of the corresponding log is the proper log to be output is achieved, and the output positions of the logs corresponding to the logs to be output subsequently are all ensured to be proper.
Optionally, the log processing method of this embodiment may be executed by a proxy plug-in. The log processing method of the embodiment is executed by the proxy plug-in, and comprises the following steps: performing semantic analysis on all logs to be output in a preset log output cache pool to obtain a semantic analysis result of each log to be output, and processing the set log level of each log to be output according to the matching result of the set log level of each log to be output and the semantic analysis result; acquiring a method calling path of each log to be output, and calculating the information entropy of the method calling path of each log to be output; determining an output mode corresponding to each log to be output according to the information entropy of the method call path of each log to be output; and controlling the output of each log to be output according to the output mode corresponding to each log to be output.
Optionally, the log processing method of this embodiment further includes: counting the frequency of each method calling path according to each method calling path of the log to be output; and adjusting the output position corresponding to each log to be output according to the frequency of each method calling path and the set log level of the log to be output.
Specifically, merging method call paths of each log to be output in a preset log output cache pool, and counting the frequency of each method call path. And for each log to be output, if the frequency of the method calling path of the log to be output exceeds a preset upper frequency limit or the set log level of the log to be output is not more than a Warning (WARN) level, determining not to output the log to be output, inhibiting the subsequent output of the log output position corresponding to the log to be output, and not outputting the log from the position. Or, for each log to be output, if the frequency of the method call path of the log to be output exceeds a preset upper frequency limit or the set log level of the log to be output is not more than a Warning (WARN) level, determining not to output the log to be output, and setting a subsequent output mode of a position of log output corresponding to the log to be output as sampling output, namely, when the position detects a log acquisition request, outputting the log again. The preset upper frequency limit can be set according to the service requirement.
The logs to be output with too high frequency of the method calling path and the logs to be output with the set log level not higher than the Warning (WARN) level are not important information for fault positioning of the system generally, and indicate that the corresponding position for placing the log printing statement is not appropriate, namely the corresponding position for outputting the logs is not appropriate, so that important log data cannot be acquired. The output of the log to be output with too high frequency of the method calling path and the log to be output with lower log level is limited, so that the log output position corresponding to the subsequently output log can be further ensured to be appropriate.
The embodiment of the invention provides a log processing method, which comprises the steps of performing semantic analysis on all logs to be output in a preset log output cache pool to obtain a semantic analysis result of each log to be output, processing the set log level of each log to be output according to the matching result of the set log level of each log to be output and the semantic analysis result to obtain a method calling path of each log to be output, calculating the information entropy of the method calling path of each log to be output, determining the output mode corresponding to each log to be output according to the information entropy of the method calling path of each log to be output, controlling the output of each log to be output according to the output mode corresponding to each log to be output, and determining whether the log level of each log to be output is correct or not by matching the set log level of each log to be output with the semantic analysis result, whether the log output position corresponding to the log to be output is appropriate can be determined according to the information entropy of the method calling path, and the log output position is appropriate and the log output level is correct.
Example two
Fig. 2 is a flowchart of a log processing method according to a second embodiment of the present invention. In this embodiment of the present invention, the processing of the set log level of each log to be output according to the matching result between the set log level of each log to be output and the semantic analysis result includes: judging whether the set log level of each log to be output is matched with the semantic analysis result; if the set log level of the log to be output is determined to be not matched with the semantic analysis result, updating the set log level of the log to be output into the semantic analysis result of the log to be output; and if the set log level of each log to be output is determined to be matched with the semantic analysis result, maintaining the set log level of each log to be output.
As shown in fig. 2, the method of the embodiment of the present invention specifically includes:
Non-exhaustive descriptions of the present embodiments may refer to the foregoing embodiments.
And 203, updating the set log level of the log to be output into a semantic analysis result of the log to be output.
In this embodiment, if it is determined that the set log level of the log to be output does not match the semantic analysis result, the set log level of the log to be output whose set log level does not match the semantic analysis result is updated to the semantic analysis result of the log to be output, that is, the log level of the log to be output determined according to the semantics of the log text is used to replace the unmatched set log level, so that the incorrect log level is corrected to the correct log level.
And step 204, keeping the set log level of each log to be output.
And step 206, determining an output mode corresponding to each log to be output according to the information entropy of the method call path of each log to be output.
And step 207, controlling the output of each log to be output according to the output mode corresponding to each log to be output.
Optionally, the log processing method of this embodiment may be executed by a proxy plug-in. The log processing method of the embodiment is executed by the proxy plug-in, and comprises the following steps: performing semantic analysis on all logs to be output in a preset log output cache pool to obtain a semantic analysis result of each log to be output; judging whether the set log level of each log to be output is matched with the semantic analysis result; if not, updating the set log level of the log to be output into the semantic analysis result of the log to be output; if yes, keeping the set log level of each log to be output; acquiring a method calling path of each log to be output, and calculating the information entropy of the method calling path of each log to be output; determining an output mode corresponding to each log to be output according to the information entropy of the method call path of each log to be output; and controlling the output of each log to be output according to the output mode corresponding to each log to be output.
The embodiment of the invention provides a log processing method, which updates the set log level of a log to be output into the semantic analysis result of the log to be output when the set log level of the log to be output is determined to be not matched with the semantic analysis result, and can update the set log level of the log to be output into the semantic analysis result of the log to be output, wherein the set log level of the log to be output is not matched with the semantic analysis result, so that the incorrect log level is corrected into the correct log level, and the log level actually output by the log is ensured to be correct.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a log processing apparatus according to a third embodiment of the present invention, and as shown in fig. 3, the apparatus includes: a level matching module 301, an information entropy calculation module 302, an output mode determination module 303, and an output control module 304.
The level matching module 301 is configured to perform semantic analysis on all logs to be output in a preset log output cache pool to obtain a semantic analysis result of each log to be output, and process the set log level of each log to be output according to a matching result between the set log level of each log to be output and the semantic analysis result; an information entropy calculation module 302, configured to obtain a method call path of each log to be output, and calculate an information entropy of the method call path of each log to be output; an output mode determining module 303, configured to determine, according to the information entropy of the method call path of each log to be output, an output mode corresponding to each log to be output; and the output control module 304 is configured to control output of each log to be output according to an output mode corresponding to each log to be output.
The embodiment of the invention provides a log processing device, which obtains a semantic analysis result of each log to be output by performing semantic analysis on all logs to be output in a preset log output cache pool, processes the set log level of each log to be output according to the matching result of the set log level of each log to be output and the semantic analysis result, obtains a method calling path of each log to be output, calculates the information entropy of the method calling path of each log to be output, determines an output mode corresponding to each log to be output according to the information entropy of the method calling path of each log to be output, controls the output of each log to be output according to the output mode corresponding to each log to be output, can determine whether the log level of the log to be output is correct or not by matching the set log level of the log to be output and the semantic analysis result, and can determine the information entropy of the method calling path, and determining whether the log output position corresponding to the log to be output is proper or not, so that the log level of the actual output of the log can be ensured to be correct, and the log output position is proper.
In an optional implementation manner of the embodiment of the present invention, optionally, the level matching module 301 may include: the level matching unit is used for judging whether the set log level of each log to be output is matched with the semantic analysis result or not; the level updating unit is used for updating the set log level of the log to be output into the semantic analysis result of the log to be output if the set log level of the log to be output is determined to be not matched with the semantic analysis result; and the level holding unit is used for holding the set log level of each log to be output if the set log level of each log to be output is determined to be matched with the semantic analysis result.
In an optional implementation manner of the embodiment of the present invention, optionally, the log processing method is executed by a proxy plug-in.
In an optional implementation manner of the embodiment of the present invention, optionally, the information entropy calculation module 302 may include: and the path acquisition unit is used for acquiring the method calling path of each log to be output through a preset path acquisition statement.
In an optional implementation manner of the embodiment of the present invention, optionally, the output manner determining module 303 may include: the log obtaining unit is used for sequentially obtaining one log to be output in the logs to be output as a target log to be output; the threshold value judging unit is used for judging whether the information entropy of the method calling path of the target log to be output is larger than a preset information entropy threshold value or not; a first mode determining unit, configured to determine, if the information entropy of the method call path of the target log to be output is greater than a preset information entropy threshold, an output mode corresponding to the target log to be output as reserved output; a second mode determining unit, configured to determine, if the information entropy of the method call path of the target log to be output is less than or equal to a preset information entropy threshold, an output mode corresponding to the target log to be output as no output; and the operation returning unit is used for returning and executing the operation of sequentially acquiring one log to be output in the logs to be output as the target log to be output until the processing of all the logs to be output is completed.
In an optional implementation manner of the embodiment of the present invention, optionally, the output control module 304 may include: the first output control unit is used for outputting the target log to be output to a corresponding target position if the output mode corresponding to the target log to be output is reserved output; and the second output control unit is used for determining not to output the target log to be output if the output mode corresponding to the target log to be output is not output.
In an optional implementation manner of the embodiment of the present invention, optionally, the log processing apparatus may further include: the path frequency counting module is used for counting the frequency of each method calling path according to each method calling path of the log to be output; and the output position adjusting module is used for adjusting the output position corresponding to each log to be output according to the frequency of each method calling path and the set log grade of the log to be output.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
The log processing device can execute the log processing method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of executing the log processing method.
Example four
Fig. 4 is a schematic structural diagram of a computer device according to a fourth embodiment of the present invention. FIG. 4 illustrates a block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention. The computer device 12 shown in FIG. 4 is only one example and should not bring any limitations to the functionality or scope of use of embodiments of the present invention.
As shown in FIG. 4, computer device 12 is in the form of a general purpose computing device. The components of computer device 12 may include, but are not limited to: one or more processors 16, a memory 28, and a bus 18 that connects the various system components (including the memory 28 and the processors 16).
Computer device 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer device 12 and includes both volatile and nonvolatile media, removable and non-removable media.
The memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. Computer device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 4, and commonly referred to as a "hard drive"). Although not shown in FIG. 4, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.
Computer device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), with one or more devices that enable a user to interact with computer device 12, and/or with any devices (e.g., network card, modem, etc.) that enable computer device 12 to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 22. Also, computer device 12 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with the other modules of computer device 12 via bus 18. It should be appreciated that although not shown in FIG. 4, other hardware and/or software modules may be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
The processor 16 executes various functional applications and data processing by running the program stored in the memory 28, thereby implementing the log processing method provided by the embodiment of the present invention: performing semantic analysis on all logs to be output in a preset log output cache pool to obtain a semantic analysis result of each log to be output, and processing the set log level of each log to be output according to the matching result of the set log level of each log to be output and the semantic analysis result; acquiring a method calling path of each log to be output, and calculating the information entropy of the method calling path of each log to be output; determining an output mode corresponding to each log to be output according to the information entropy of the method call path of each log to be output; and controlling the output of each log to be output according to the output mode corresponding to each log to be output.
EXAMPLE five
Fifth embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the method for processing a log provided in the fifth embodiment of the present invention is implemented: performing semantic analysis on all logs to be output in a preset log output cache pool to obtain a semantic analysis result of each log to be output, and processing the set log level of each log to be output according to the matching result of the set log level of each log to be output and the semantic analysis result; acquiring a method calling path of each log to be output, and calculating the information entropy of the method calling path of each log to be output; determining an output mode corresponding to each log to be output according to the information entropy of the method call path of each log to be output; and controlling the output of each log to be output according to the output mode corresponding to each log to be output.
Any combination of one or more computer-readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or computer device. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A log processing method, comprising:
performing semantic analysis on all logs to be output in a preset log output cache pool to obtain a semantic analysis result of each log to be output, and processing the set log level of each log to be output according to the matching result of the set log level of each log to be output and the semantic analysis result;
acquiring a method calling path of each log to be output, and calculating the information entropy of the method calling path of each log to be output;
the calculating the information entropy of the method call path of each log to be output comprises the following steps: and calculating the information entropy of the method calling path of each log to be output according to the following information entropy calculation formula:
wherein,
the information entropy of the method call path for the target to-be-output log,
the probability of invoking a path in order to execute a method of the target pending output log,
the frequency of the paths is called for the method of the target to-be-output log,
the total number of call paths for the method;
determining an output mode corresponding to each log to be output according to the information entropy of the method calling path of each log to be output, wherein the information entropy of the method calling path is the information quantity of the corresponding log to be output;
and controlling the output of each log to be output according to the output mode corresponding to each log to be output.
2. The method according to claim 1, wherein the processing the set log level of each log to be output according to the matching result between the set log level of each log to be output and the semantic analysis result includes:
judging whether the set log level of each log to be output is matched with the semantic analysis result; if the set log level of the log to be output is determined to be not matched with the semantic analysis result, updating the set log level of the log to be output into the semantic analysis result of the log to be output;
and if the set log level of each log to be output is determined to be matched with the semantic analysis result, maintaining the set log level of each log to be output.
3. The method according to claim 1 or 2, characterized in that the method is performed by a proxy plugin.
4. The method according to claim 1, wherein the obtaining of the method call path of each log to be output comprises:
and acquiring a method calling path of each log to be output through a preset path acquisition statement.
5. The method according to claim 1, wherein the determining, according to the information entropy of the method call path of each log to be output, an output mode corresponding to each log to be output includes:
sequentially acquiring one log to be output in each log to be output as a target log to be output;
judging whether the information entropy of the method calling path of the target log to be output is larger than a preset information entropy threshold value or not;
if the information entropy of the method calling path of the target log to be output is larger than a preset information entropy threshold, determining an output mode corresponding to the target log to be output as reserved output;
if the information entropy of the method calling path of the target log to be output is less than or equal to a preset information entropy threshold, determining that the output mode corresponding to the target log to be output is not output;
and returning to execute the operation of sequentially acquiring one log to be output in the logs to be output as a target log to be output until the processing of all the logs to be output is completed.
6. The method according to claim 5, wherein the controlling the output of each log to be output according to the output mode corresponding to each log to be output comprises:
if the output mode corresponding to the target log to be output is reserved output, outputting the target log to be output to a corresponding target position;
and if the output mode corresponding to the target log to be output is non-output, determining not to output the target log to be output.
7. The method of claim 1, further comprising:
counting the frequency of each method calling path according to each method calling path of the log to be output;
and adjusting the output position corresponding to each log to be output according to the frequency of each method calling path and the set log level of the log to be output.
8. A log processing apparatus, comprising:
the level matching module is used for performing semantic analysis on all logs to be output in a preset log output cache pool to obtain a semantic analysis result of each log to be output, and processing the set log level of each log to be output according to the matching result of the set log level of each log to be output and the semantic analysis result;
the information entropy calculation module is used for acquiring the method calling path of each log to be output and calculating the information entropy of the method calling path of each log to be output;
the calculating the information entropy of the method call path of each log to be output comprises the following steps: and calculating the information entropy of the method calling path of each log to be output according to the following information entropy calculation formula:
wherein,
the information entropy of the method call path for the target to-be-output log,
the probability of invoking a path in order to execute a method of the target pending output log,
the frequency of the paths is called for the method of the target to-be-output log,
the total number of call paths for the method;
the output mode determining module is used for determining the output mode corresponding to each log to be output according to the information entropy of the method calling path of each log to be output, wherein the information entropy of the method calling path is the information quantity of the corresponding log to be output;
and the output control module is used for controlling the output of each log to be output according to the output mode corresponding to each log to be output.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the log processing method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the log processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010542073.9A CN111708681B (en) | 2020-06-15 | 2020-06-15 | Log processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010542073.9A CN111708681B (en) | 2020-06-15 | 2020-06-15 | Log processing method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111708681A CN111708681A (en) | 2020-09-25 |
| CN111708681B true CN111708681B (en) | 2021-05-07 |
Family
ID=72540635
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010542073.9A Active CN111708681B (en) | 2020-06-15 | 2020-06-15 | Log processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111708681B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113448818B (en) * | 2021-07-01 | 2023-01-24 | 山东浪潮通软信息科技有限公司 | Log tracking method, device and medium based on probability |
| CN113656246B (en) * | 2021-08-13 | 2024-03-19 | 杭州安恒信息技术股份有限公司 | Log analysis method, system, electronic device, computer equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106547470A (en) * | 2015-09-16 | 2017-03-29 | 伊姆西公司 | Daily record storage optimization method and equipment |
| CN109597757A (en) * | 2018-11-07 | 2019-04-09 | 北京航空航天大学 | The measure of similarity between a kind of software network based on multidimensional time-series entropy |
| CN110377244A (en) * | 2019-07-12 | 2019-10-25 | 苏州浪潮智能科技有限公司 | A kind of log Method of printing, device, equipment and storage medium |
| CN110868382A (en) * | 2018-12-21 | 2020-03-06 | 哈尔滨安天科技集团股份有限公司 | Decision tree-based network threat assessment method, device and storage medium |
Family Cites Families (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6889219B2 (en) * | 2002-01-22 | 2005-05-03 | International Business Machines Corporation | Method of tuning a decision network and a decision tree model |
| CN101751288A (en) * | 2008-12-12 | 2010-06-23 | 中国移动通信集团山东有限公司 | Method, device and system applying process scheduler |
| CN103257852B (en) * | 2012-02-17 | 2017-08-04 | 阿里巴巴集团控股有限公司 | The method and apparatus that a kind of development environment of distribution application system is built |
| CN104850780A (en) * | 2015-04-27 | 2015-08-19 | 北京北信源软件股份有限公司 | Discrimination method for advanced persistent threat attack |
| CN105893225A (en) * | 2015-08-25 | 2016-08-24 | 乐视网信息技术（北京）股份有限公司 | Automatic error processing method and device |
| KR102149866B1 (en) * | 2015-09-29 | 2020-09-01 | 에이에스엠엘 네델란즈 비.브이. | Methods of modeling systems such as lithographic systems or performing predictive maintenance of systems, and associated lithographic systems. |
| CN105740121B (en) * | 2016-01-26 | 2018-08-28 | 中国银行股份有限公司 | A kind of monitoring of daily record text and method for early warning, device |
| CN106685742B (en) * | 2017-03-02 | 2020-04-10 | 北京邮电大学 | Network fault diagnosis method and device |
| US10440037B2 (en) * | 2017-03-31 | 2019-10-08 | Mcafee, Llc | Identifying malware-suspect end points through entropy changes in consolidated logs |
| CN107360152A (en) * | 2017-07-07 | 2017-11-17 | 四川大学 | A kind of Web based on semantic analysis threatens sensory perceptual system |
| CN108205486A (en) * | 2017-12-26 | 2018-06-26 | 上海中畅数据技术有限公司 | A kind of intelligent distributed call chain tracking based on machine learning |
| CN108280021A (en) * | 2018-01-25 | 2018-07-13 | 郑州云海信息技术有限公司 | A kind of logging level analysis method based on machine learning |
| CN110708339B (en) * | 2019-11-06 | 2021-06-22 | 四川长虹电器股份有限公司 | Correlation analysis method based on WEB log |
| CN110806962B (en) * | 2019-11-06 | 2021-04-16 | 星环信息科技(上海)股份有限公司 | Log level prediction method, device and storage medium |
| CN111124859A (en) * | 2019-12-13 | 2020-05-08 | 北京浪潮数据技术有限公司 | Log processing method, device, equipment and storage medium |
| CN111143303B (en) * | 2019-12-31 | 2023-06-02 | 海南电网有限责任公司信息通信分公司 | Log classification method based on information gain and improved KNN algorithm |
-
2020
- 2020-06-15 CN CN202010542073.9A patent/CN111708681B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106547470A (en) * | 2015-09-16 | 2017-03-29 | 伊姆西公司 | Daily record storage optimization method and equipment |
| CN109597757A (en) * | 2018-11-07 | 2019-04-09 | 北京航空航天大学 | The measure of similarity between a kind of software network based on multidimensional time-series entropy |
| CN110868382A (en) * | 2018-12-21 | 2020-03-06 | 哈尔滨安天科技集团股份有限公司 | Decision tree-based network threat assessment method, device and storage medium |
| CN110377244A (en) * | 2019-07-12 | 2019-10-25 | 苏州浪潮智能科技有限公司 | A kind of log Method of printing, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111708681A (en) | 2020-09-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107506300B (en) | User interface testing method, device, server and storage medium | |
| CN108334439B (en) | Pressure testing method, device, equipment and storage medium | |
| CN108319575B (en) | Page component checking method, device, server and storage medium | |
| CN111708681B (en) | Log processing method, device, equipment and storage medium | |
| CN111061740B (en) | Data synchronization method, device and storage medium | |
| WO2022089267A1 (en) | Sample data acquisition method, image segmentation method and apparatus, device and medium | |
| CN110659210A (en) | Information acquisition method and device, electronic equipment and storage medium | |
| CN115310460A (en) | Machine translation quality evaluation method, device, equipment and storage medium | |
| CN110716843B (en) | System fault analysis processing method and device, storage medium and electronic equipment | |
| CN111104400A (en) | Data normalization method and device, electronic equipment and storage medium | |
| CN110865829B (en) | Database upgrading method, system, equipment and storage medium | |
| CN113408070A (en) | Method, device and equipment for determining engine parameters and storage medium | |
| CN113553309A (en) | Log template determination method and device, electronic equipment and storage medium | |
| CN111367813B (en) | Automatic testing method and device for decision engine, server and storage medium | |
| CN112069158A (en) | Data restoration method, device, equipment and storage medium | |
| CN115909386A (en) | Method, equipment and storage medium for completing and correcting pipeline instrument flow chart | |
| US11392371B2 (en) | Identification of a partial code to be refactored within a source code | |
| CN110908818B (en) | Verification method, device, equipment and storage medium | |
| CN109918231B (en) | Data reforming abnormity repairing method, device, equipment and storage medium | |
| CN113032469B (en) | Text structured model training and medical text structuring method and device | |
| CN110991147B (en) | Font detection method and device, electronic equipment and storage medium | |
| CN111741046B (en) | Data reporting method, data acquisition method, device, equipment and medium | |
| CN113010114A (en) | Data processing method and device, computer equipment and storage medium | |
| CN110888665B (en) | Method, device, equipment and storage medium for submitting changed content | |
| CN111176718A (en) | Script online method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |