CN111045782A - Log processing method and device, electronic equipment and computer readable storage medium - Google Patents
Log processing method and device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN111045782A CN111045782A CN201911142773.2A CN201911142773A CN111045782A CN 111045782 A CN111045782 A CN 111045782A CN 201911142773 A CN201911142773 A CN 201911142773A CN 111045782 A CN111045782 A CN 111045782A
- Authority
- CN
- China
- Prior art keywords
- log
- data
- processing thread
- log data
- service processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 17
- 238000012545 processing Methods 0.000 claims abstract description 235
- 238000000034 method Methods 0.000 claims abstract description 63
- 230000008569 process Effects 0.000 claims abstract description 35
- 238000004891 communication Methods 0.000 claims description 19
- 230000002159 abnormal effect Effects 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 8
- 230000006870 function Effects 0.000 claims description 7
- 238000011161 development Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000006872 improvement Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000007639 printing Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention discloses a log processing method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: creating a service processing thread and a log processing thread; writing the log data generated in the running process of the application program into a queue of a preset type through the service processing thread; and acquiring the log data from the queue through the log processing thread, and outputting the log data to a preset log file. The invention introduces the log processing thread, adopts an asynchronous processing mode, obtains the log data generated in the running process of the application program by the service processing thread, sends the log data to the log processing thread through the queue, outputs the log data to the log file by the log processing thread, effectively isolates the large-overhead log processing task from the data processing task, reduces the log processing pressure of the service processing thread, improves the data processing capability and improves the data processing efficiency of the service processing thread.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a log processing method and apparatus, an electronic device, and a computer-readable storage medium.
Background
With the rapid development of the internet and the continuous improvement of the business volume, the data access flow based on the network is rapidly increased, and particularly the access flow to a data center, a large-scale enterprise, a portal website and the like even reaches the level of 10 Gb/s. Compared with the development of network technology, the increase of network bandwidth and application service is far higher than the increase of server processing speed and memory access speed, and the number of users is continuously increased while the network bandwidth is increased, which causes the resource consumption of the server to be serious, so that the server becomes a network bottleneck and often becomes a network failure point. Therefore, establishing a server cluster and balancing the traffic among the server clusters by using a load balancing technology become a main solution for solving the problems.
Currently, many new techniques are applied to the load balancer in order to improve the load balancing performance. For example: the DPVS (Data Plane Development Kit-Virtual Server) load balancer is a Virtual Server (Virtual Server) based on a DPDK (Data Plane Development Kit), so that a high-performance four-layer load balancer is realized using the DPDK, and Data processing performance and throughput can be improved and the work efficiency of a Data Plane application program can be improved by avoiding interruption, no locking, CPU (central processing unit) affinity, large-page memory and other technologies.
However, when the DPVS load balancer is abnormal, for example: when the memory resource is insufficient or other resources are insufficient, the DPVS load balancer may cause low data processing efficiency due to processing of log data corresponding to the abnormal information. Specifically, the DPVS load balancer prints log data in a multi-core multi-thread polling mode. That is, in the DPVS load balancer, the data cores and the threads are in one-to-one correspondence, and when polling is performed among a plurality of data cores, the thread corresponding to the polled data core prints log data generated by the polled data core through an IO (Input/Output) interface operation, but because the IO interface operation time is long and it is necessary to wait for the IO interface operation to complete before polling a next data core, the polling process time is too long. When an application program is abnormal in the running process, the data core can print a large amount of log data of the abnormal information, and the data processing efficiency is reduced.
Disclosure of Invention
Embodiments of the present invention provide a log processing method, an apparatus, an electronic device, and a computer-readable storage medium, so as to solve the problem that when a DPVS load balancer is abnormal, the DPVS load balancer may have low data processing efficiency due to processing log data corresponding to abnormal information. The specific technical scheme is as follows:
in a first aspect of the present invention, there is provided a log processing method, including: creating a service processing thread and a log processing thread; writing the log data generated in the running process of the application program into a queue of a preset type through the service processing thread; and acquiring the log data from the queue through the log processing thread, and outputting the log data to a preset log file.
Wherein, the creating of the service processing thread and the log processing thread includes: creating a plurality of service processing threads and a log processing thread in a DPVS load balancer; and each service processing thread is correspondingly bound with one service processing data core, and the log processing thread is correspondingly bound with one log processing data core.
Wherein the outputting the log data to a preset log file comprises: determining a service processing data core for generating the log data according to the field information of the log data; and outputting the log data to a log file which is preset for the service processing thread corresponding to the service processing data core according to the service processing thread corresponding to the service processing data core.
The method includes that in the process of writing log data generated in the running process of an application program into a preset type queue through the service processing thread, the method further includes: and if the queue of the preset type is blocked, suspending the writing of the log data generated in the running process of the application program into the queue of the preset type through the service processing thread, or writing the log data only retaining preset field information into the queue of the preset type through the service processing thread.
Wherein the preset field information at least comprises one of the following information: and function name and line number corresponding to the abnormal code.
Before outputting the log data to a preset log file, the method further comprises: if the number of the log data is multiple, performing hash operation on each log data through the log processing thread to obtain a hash value corresponding to each log data; according to the hash values respectively corresponding to the plurality of log data, performing deduplication processing on the plurality of log data to obtain a plurality of non-duplicated log data; the outputting the log data to a preset log file includes: and outputting the non-repeated log data to a preset log file through input/output (IO) interface operation.
Wherein the type of the queue comprises: lock-free queues and/or message queues.
In a second aspect of the present invention, there is also provided a log processing apparatus, including: the creating module is used for creating a service processing thread and a log processing thread; the writing module is used for writing the log data generated in the running process of the application program into a queue of a preset type through the service processing thread; and the output module is used for acquiring the log data from the queue through the log processing thread and outputting the log data to a preset log file.
In another aspect of the present invention, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus; a memory for storing a computer program; a processor for implementing any of the above method steps when executing a program stored in the memory.
In yet another aspect of the present invention, there is also provided a computer-readable storage medium having stored therein instructions, which when run on a computer, cause the computer to perform any of the methods described above.
In yet another aspect of the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform any of the methods described above.
The log processing method, the log processing device, the electronic equipment and the computer readable storage medium introduce a log processing thread, adopt an asynchronous processing mode, obtain log data generated in the running process of an application program by a service processing thread, send the log data to the log processing thread through a queue, and output the log data to a log file by the log processing thread, thereby effectively isolating a large-overhead log processing task from a data processing task, reducing the log processing pressure of the service processing thread, improving the data processing capacity and improving the data processing efficiency of the service processing thread.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
FIG. 1 is a flow diagram of a log processing method according to an embodiment of the invention;
FIG. 2 is a flowchart illustrating a method for processing logs according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating a log processing method according to an embodiment of the invention;
FIG. 4 is a block diagram of a log processing apparatus according to an embodiment of the present invention;
fig. 5 is a block diagram of an electronic device according to an embodiment of the invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention.
The embodiment of the invention provides a log processing method. The execution subject of this embodiment is a device that needs to process log data. For example: the execution subject is a server, a load balancer and the like. The load balancer may be a DPVS load balancer.
Fig. 1 is a flowchart illustrating a log processing method according to an embodiment of the present invention.
Step S110, a service processing thread and a log processing thread are created.
And the service processing thread is used for transmitting the log data generated in the running process of the application program to the log processing thread. The application program runs in the execution main body of the present embodiment.
And the log processing thread is used for outputting the log data transmitted by the service processing thread to a log file.
In this embodiment, with the CPU affinity of the DPVS load balancer, multiple service processing threads and one log processing thread can be created at the DPVS load balancer. Each service processing thread is correspondingly bound with one service processing data core, and each log processing thread is correspondingly bound with one log processing data core.
And step S120, writing the log data generated in the running process of the application program into a preset type queue through the service processing thread.
In this embodiment, the service processing thread may transmit the log data to the log processing thread through a queue of a preset type. The type of the queue includes: lock-free queues and/or message queues.
In the log data, at least the following field information is included: and function name and line number corresponding to the abnormal code. Of course, in the log data, the following field information may also be included: and generating the information of the data core of the log data and the custom field information. The information of the data core is, for example, the ID of the data core. The custom field information can be set according to requirements, for example: and the name of the code module where the abnormal code is located.
Specifically, in the running process of the application program, the data core generates log data, and the service processing thread acquires the log data and writes the log data into a queue of a preset type. The data cores and the service processing threads have corresponding relations, and the service processing threads acquire the log data generated by the corresponding data cores, so that the service processing threads acquiring the log data can be determined according to the information of the data cores in the log data.
And if the queue of the preset type is blocked, the log data generated in the running process of the application program is suspended from being written into the queue of the preset type through the service processing thread, or the log data only retaining preset field information is written into the queue of the preset type through the service processing thread. The preset field information may be important information in log data, such as: the preset field information at least comprises one of the following information: and function name and line number corresponding to the abnormal code. Only important field information is reserved in the log data, so that the length of the log data can be shortened, and the processing pressure of log processing threads is reduced.
Step S130, obtaining the log data from the queue through the log processing thread, and outputting the log data to a preset log file.
And acquiring and caching the log data from the queue through the log processing thread, and outputting the cached log data to a preset log file by adopting the same or different output modes according to different quantities of the cached log data at preset time intervals. For example:
and if the number of the log data is one, directly outputting the log data to a preset log file through IO interface operation.
If the number of the log data is multiple, performing hash operation on each log data through the log processing thread to obtain a hash value corresponding to each log data; according to the hash values respectively corresponding to the plurality of log data, performing deduplication processing on the plurality of log data to obtain a plurality of non-duplicated log data; and outputting the non-repeated log data to a preset log file through IO (Input/Output) interface operation. Further, the deduplication processing includes: and inquiring a plurality of log data with the same hash value, and reserving one log data in the plurality of log data with the same hash value.
The above example adopts different output modes to output the log data to a preset log file. When the log data are more, especially for abnormal log data with strong repeatability, a large amount of repeated log data can be filtered out by carrying out deduplication processing on the log data, and a large amount of meaningless expenses are reduced.
In this embodiment, the number of the service processing threads may be multiple, and before the log data is output to a preset log file, a log file may be correspondingly set for each service processing thread, or a common log file may be set for multiple service processing threads. If a log file is correspondingly set for each service processing thread, determining a service processing data core for generating the log data according to the field information of the log data; and outputting the log data to a log file which is preset for the service processing thread corresponding to the service processing data core according to the service processing thread corresponding to the service processing data core. For example: the field information of the log data may be an ID of the data core.
In the embodiment, a log processing thread is introduced, an asynchronous processing mode is adopted, log data generated in the running process of an application program is obtained by a service processing thread and is sent to the log processing thread through a queue, the log data is output to a log file by the log processing thread, a large-overhead log processing task and a data processing task are effectively isolated, the log processing pressure of the service processing thread is reduced, the data processing capacity is improved, and the data processing efficiency of the service processing thread is improved.
The log processing method of the present invention will be further described below, taking the example of performing log processing at a DPVS load balancer.
Fig. 2 is a detailed flowchart of a log processing method according to an embodiment of the invention.
Step S210, creating a plurality of service processing threads and a log processing thread, and binding the plurality of service processing threads and the log processing thread to corresponding data cores respectively.
The DPVS load balancer comprises a plurality of data cores, and the created threads correspond to the data cores one by one.
And the log processing thread is correspondingly bound with a log processing data core. The log processing data core is used for providing operation resources and storage resources for the log processing thread.
And each service processing thread is correspondingly bound with one service processing data core. Each business processing data core is used for providing operation resources and storage resources for the business processing thread, and each business processing data core runs part of codes of the application program and generates log data in the running process of the part of codes.
In step S220, the plurality of service processing threads respectively write the log data generated by the corresponding data core (service processing data core) into a preset lock-free queue.
A lock-free queue may be correspondingly set for each service processing thread, or a shared lock-free queue may be correspondingly set for a plurality of service processing threads. And the business processing thread writes the log data generated by the corresponding business processing data core into the lock-free queue corresponding to the business processing thread.
Fig. 3 is a schematic diagram illustrating a log processing method according to an embodiment of the invention. The DPVS load balancer comprises n data cores (n is more than 1), each data core is correspondingly bound with a service processing thread, each service processing thread corresponds to a lock-free queue, thus, n lock-free queues are set, and the service processing thread writes log data generated by the corresponding data core into the corresponding lock-free queue, such as: and the business processing thread 1 writes the log generated by the data core 1 into the lock-free queue 1.
In the process of writing log data into the lock-free queue, if the lock-free queue is full (blocked), the log data can be suspended from being written into the lock-free queue, and the log data needing to be written into the lock-free queue is directly discarded; the method can also select important field information in the log data, shorten the length of the log data and write the log data with the shortened length into the lock-free queue, so that the log data is processed without influencing the data processing performance, and the balance between the log processing efficiency and the data processing performance is achieved.
In step S230, the log processing thread obtains log data from the lock-free queue, and performs deduplication processing on the obtained log data at intervals of a preset time period.
After acquiring the log data, the log processing thread does not directly output the log data to the log file, but caches the log data. As shown in fig. 3, the log processing thread acquires log data from the lock-free queue 1 to the lock-free queue n, and caches the acquired log data in the cache of the corresponding log processing data core, so as to integrate the acquired log data to obtain non-duplicated log data, and then outputs the non-duplicated log data to the log file.
In step S240, the log processing thread outputs the log data obtained after the deduplication processing to a preset log file.
In this embodiment, in an asynchronous processing manner, the plurality of service processing threads are respectively responsible for acquiring the log data generated by the corresponding service processing data core, and the log data generated by the plurality of service processing data cores is uniformly output to the log file by the log processing threads, so that the large-overhead log processing task and the data processing task are effectively isolated, the log printing pressure of the service processing data cores is relieved, and the data processing task can be executed efficiently even when each service processing data core is abnormal. The problem that the data core processing capacity is reduced under the abnormal condition is solved through the embodiment, and the stability of the DPVS load balancer is effectively improved, so that a stable network environment is provided for high-concurrency and high-flow services.
The embodiment of the invention also provides a log processing device. Fig. 4 is a block diagram of a log processing apparatus according to an embodiment of the present invention.
The log processing device comprises: a creation module 410, a write module 420, and an output module 430.
And a creating module 410 for creating a service processing thread and a log processing thread.
A writing module 420, configured to write, by the service processing thread, log data generated in an application program running process into a preset type of queue.
And an output module 430, configured to obtain the log data from the queue through the log processing thread, and output the log data to a preset log file.
Wherein the creating module 410 is configured to: creating a plurality of service processing threads and a log processing thread in a DPVS load balancer; and each service processing thread is correspondingly bound with one service processing data core, and the log processing thread is correspondingly bound with one log processing data core.
Wherein the output module 430 is further configured to: determining a service processing data core for generating the log data according to the field information of the log data; and outputting the log data to a log file which is preset for the service processing thread corresponding to the service processing data core according to the service processing thread corresponding to the service processing data core.
The writing module 420 is further configured to, in the process of writing the log data generated in the running process of the application into the queue of the preset type through the service processing thread, suspend writing the log data generated in the running process of the application into the queue of the preset type through the service processing thread if the queue of the preset type is blocked, or write the log data only retaining the preset field information into the queue of the preset type through the service processing thread.
Wherein the preset field information at least comprises one of the following information: and function name and line number corresponding to the abnormal code.
Wherein the output module 430 is further configured to: before outputting the log data to a preset log file, if the number of the log data is multiple, performing hash operation on each log data through the log processing thread to obtain a hash value corresponding to each log data; according to the hash values respectively corresponding to the plurality of log data, performing deduplication processing on the plurality of log data to obtain a plurality of non-duplicated log data; and outputting the non-repeated log data to a preset log file through input/output (IO) interface operation.
Wherein the type of the queue comprises: lock-free queues and/or message queues.
An embodiment of the present invention further provides an electronic device, as shown in fig. 5, including a processor 510, a communication interface 520, a memory 530 and a communication bus 540, where the processor 510, the communication interface 520, and the memory 530 complete mutual communication through the communication bus 540.
A memory 530 for storing a computer program.
The processor 510, when executing the program stored in the memory 530, implements the following steps: creating a service processing thread and a log processing thread; writing the log data generated in the running process of the application program into a queue of a preset type through the service processing thread; and acquiring the log data from the queue through the log processing thread, and outputting the log data to a preset log file.
Wherein, the creating of the service processing thread and the log processing thread includes: creating a plurality of service processing threads and a log processing thread in a DPVS load balancer; and each service processing thread is correspondingly bound with one service processing data core, and the log processing thread is correspondingly bound with one log processing data core.
Wherein the outputting the log data to a preset log file comprises: determining a service processing data core for generating the log data according to the field information of the log data; and outputting the log data to a log file which is preset for the service processing thread corresponding to the service processing data core according to the service processing thread corresponding to the service processing data core.
The method includes that in the process of writing log data generated in the running process of an application program into a preset type queue through the service processing thread, the method further includes: and if the queue of the preset type is blocked, suspending the writing of the log data generated in the running process of the application program into the queue of the preset type through the service processing thread, or writing the log data only retaining preset field information into the queue of the preset type through the service processing thread.
Wherein the preset field information at least comprises one of the following information: and function name and line number corresponding to the abnormal code.
Before outputting the log data to a preset log file, the method further comprises: if the number of the log data is multiple, performing hash operation on each log data through the log processing thread to obtain a hash value corresponding to each log data; according to the hash values respectively corresponding to the plurality of log data, performing deduplication processing on the plurality of log data to obtain a plurality of non-duplicated log data; the outputting the log data to a preset log file includes: and outputting the non-repeated log data to a preset log file through input/output (IO) interface operation.
Wherein the type of the queue comprises: lock-free queues and/or message queues.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
In another embodiment of the present invention, a computer-readable storage medium is further provided, which stores instructions that, when executed on a computer, cause the computer to execute the log processing method described in any one of the above embodiments.
In yet another embodiment of the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the log processing method described in any of the above embodiments.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (10)
1. A log processing method, comprising:
creating a service processing thread and a log processing thread;
writing the log data generated in the running process of the application program into a queue of a preset type through the service processing thread;
and acquiring the log data from the queue through the log processing thread, and outputting the log data to a preset log file.
2. The method of claim 1, wherein creating a business processing thread and a log processing thread comprises:
creating a plurality of service processing threads and a log processing thread on a DPVS load balancer of a data plane development suite; and each service processing thread is correspondingly bound with one service processing data core, and the log processing thread is correspondingly bound with one log processing data core.
3. The method of claim 2, wherein outputting the log data to a preset log file comprises:
determining a service processing data core for generating the log data according to the field information of the log data;
and outputting the log data to a log file which is preset for the service processing thread corresponding to the service processing data core according to the service processing thread corresponding to the service processing data core.
4. The method according to claim 1, wherein in the process of writing log data generated in the process of running the application program into a queue of a preset type through the business processing thread, the method further comprises:
and if the queue of the preset type is blocked, suspending the writing of the log data generated in the running process of the application program into the queue of the preset type through the service processing thread, or writing the log data only retaining preset field information into the queue of the preset type through the service processing thread.
5. The method of claim 4, wherein the preset field information comprises at least one of: and function name and line number corresponding to the abnormal code.
6. The method of claim 1,
before outputting the log data to a preset log file, the method further comprises:
if the number of the log data is multiple, performing hash operation on each log data through the log processing thread to obtain a hash value corresponding to each log data;
according to the hash values respectively corresponding to the plurality of log data, performing deduplication processing on the plurality of log data to obtain a plurality of non-duplicated log data;
the outputting the log data to a preset log file includes:
and outputting the non-repeated log data to a preset log file through input/output (IO) interface operation.
7. The method of any of claims 1-6, wherein the type of the queue comprises: lock-free queues and/or message queues.
8. A log processing apparatus, comprising:
the creating module is used for creating a service processing thread and a log processing thread;
the writing module is used for writing the log data generated in the running process of the application program into a queue of a preset type through the service processing thread;
and the output module is used for acquiring the log data from the queue through the log processing thread and outputting the log data to a preset log file.
9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 7 when executing a program stored in the memory.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method steps of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911142773.2A CN111045782B (en) | 2019-11-20 | 2019-11-20 | Log processing method, device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911142773.2A CN111045782B (en) | 2019-11-20 | 2019-11-20 | Log processing method, device, electronic equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111045782A true CN111045782A (en) | 2020-04-21 |
| CN111045782B CN111045782B (en) | 2024-01-12 |
Family
ID=70232504
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911142773.2A Active CN111045782B (en) | 2019-11-20 | 2019-11-20 | Log processing method, device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111045782B (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111858077A (en) * | 2020-07-15 | 2020-10-30 | 济南浪潮数据技术有限公司 | Recording method, device and equipment for IO request log in storage system |
| CN112306980A (en) * | 2020-10-30 | 2021-02-02 | 深圳市元征科技股份有限公司 | Log processing method and terminal equipment |
| CN112380001A (en) * | 2020-10-30 | 2021-02-19 | 网宿科技股份有限公司 | Log output method, load balancing device and computer readable storage medium |
| CN112948136A (en) * | 2021-02-02 | 2021-06-11 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Method for implementing asynchronous log record of embedded operating system |
| CN113342632A (en) * | 2021-07-07 | 2021-09-03 | 北京物芯科技有限责任公司 | Simulation data automatic processing method and device, electronic equipment and storage medium |
| CN113420032A (en) * | 2021-07-20 | 2021-09-21 | 奇安信科技集团股份有限公司 | Classification storage method and device for logs |
| CN113810337A (en) * | 2020-06-12 | 2021-12-17 | 中国电信股份有限公司 | Method, device and storage medium for network message duplicate removal |
| CN114116642A (en) * | 2021-11-23 | 2022-03-01 | 中能融合智慧科技有限公司 | Log duplicate removal and generation method and system based on WRR queue |
| CN114265637A (en) * | 2021-12-24 | 2022-04-01 | 中电信数智科技有限公司 | Archive management system configuration method, system, electronic device and storage medium |
| CN117891695A (en) * | 2024-03-14 | 2024-04-16 | 新华三技术有限公司 | Log recording method and device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140280394A1 (en) * | 2013-03-14 | 2014-09-18 | Quantum Corporation | Multi-Threaded Message Passing Journal |
| CN104252405A (en) * | 2013-06-26 | 2014-12-31 | 腾讯科技（深圳）有限公司 | Log information output method and device |
| CN106708578A (en) * | 2016-12-23 | 2017-05-24 | 北京五八信息技术有限公司 | Dual-thread-based journal output method and device |
| CN106844143A (en) * | 2016-12-27 | 2017-06-13 | 微梦创科网络科技（中国）有限公司 | A kind of daily record duplicate removal treatment method and device |
| CN108052675A (en) * | 2017-12-28 | 2018-05-18 | 惠州Tcl家电集团有限公司 | Blog management method, system and computer readable storage medium |
| CN108132955A (en) * | 2016-12-01 | 2018-06-08 | 中移(苏州)软件技术有限公司 | A kind of log storing method and device |
-
2019
- 2019-11-20 CN CN201911142773.2A patent/CN111045782B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140280394A1 (en) * | 2013-03-14 | 2014-09-18 | Quantum Corporation | Multi-Threaded Message Passing Journal |
| CN104252405A (en) * | 2013-06-26 | 2014-12-31 | 腾讯科技（深圳）有限公司 | Log information output method and device |
| CN108132955A (en) * | 2016-12-01 | 2018-06-08 | 中移(苏州)软件技术有限公司 | A kind of log storing method and device |
| CN106708578A (en) * | 2016-12-23 | 2017-05-24 | 北京五八信息技术有限公司 | Dual-thread-based journal output method and device |
| CN106844143A (en) * | 2016-12-27 | 2017-06-13 | 微梦创科网络科技（中国）有限公司 | A kind of daily record duplicate removal treatment method and device |
| CN108052675A (en) * | 2017-12-28 | 2018-05-18 | 惠州Tcl家电集团有限公司 | Blog management method, system and computer readable storage medium |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113810337A (en) * | 2020-06-12 | 2021-12-17 | 中国电信股份有限公司 | Method, device and storage medium for network message duplicate removal |
| CN113810337B (en) * | 2020-06-12 | 2023-08-08 | 天翼云科技有限公司 | Method, device and storage medium for network message deduplication |
| CN111858077A (en) * | 2020-07-15 | 2020-10-30 | 济南浪潮数据技术有限公司 | Recording method, device and equipment for IO request log in storage system |
| CN112306980A (en) * | 2020-10-30 | 2021-02-02 | 深圳市元征科技股份有限公司 | Log processing method and terminal equipment |
| CN112380001A (en) * | 2020-10-30 | 2021-02-19 | 网宿科技股份有限公司 | Log output method, load balancing device and computer readable storage medium |
| CN112380001B (en) * | 2020-10-30 | 2024-11-08 | 网宿科技股份有限公司 | Log output method, load balancing device and computer readable storage medium |
| CN112948136A (en) * | 2021-02-02 | 2021-06-11 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Method for implementing asynchronous log record of embedded operating system |
| CN113342632A (en) * | 2021-07-07 | 2021-09-03 | 北京物芯科技有限责任公司 | Simulation data automatic processing method and device, electronic equipment and storage medium |
| CN113420032A (en) * | 2021-07-20 | 2021-09-21 | 奇安信科技集团股份有限公司 | Classification storage method and device for logs |
| CN114116642A (en) * | 2021-11-23 | 2022-03-01 | 中能融合智慧科技有限公司 | Log duplicate removal and generation method and system based on WRR queue |
| CN114265637A (en) * | 2021-12-24 | 2022-04-01 | 中电信数智科技有限公司 | Archive management system configuration method, system, electronic device and storage medium |
| CN117891695A (en) * | 2024-03-14 | 2024-04-16 | 新华三技术有限公司 | Log recording method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111045782B (en) | 2024-01-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111045782B (en) | Log processing method, device, electronic equipment and computer readable storage medium | |
| CN109729106B (en) | Method, system and computer program product for processing computing tasks | |
| WO2015101091A1 (en) | Distributed resource scheduling method and device | |
| US10613992B2 (en) | Systems and methods for remote procedure call | |
| CN103064960A (en) | Method and equipment for database query | |
| CN112445857A (en) | Resource quota management method and device based on database | |
| CN110069217B (en) | Data storage method and device | |
| CN114595043A (en) | IO (input/output) scheduling method and device | |
| CN114090623A (en) | Method and device for creating cache resources, electronic equipment and storage medium | |
| CN113067875A (en) | Access method, device and equipment based on dynamic flow control of micro-service gateway | |
| EP2829972B1 (en) | Method and apparatus for allocating stream processing unit | |
| CN110851276A (en) | Service request processing method, device, server and storage medium | |
| CN111385255A (en) | Asynchronous call implementation method and device, server and server cluster | |
| CN110209466B (en) | Resource allocation method, resource allocation device and storage medium | |
| US10616317B2 (en) | Method and system for affinity load balancing | |
| CN111694670A (en) | Resource allocation method, device, equipment and computer readable medium | |
| CN117370046A (en) | Inter-process communication method, system, device and storage medium | |
| CN110990148A (en) | Method, device and medium for optimizing storage performance | |
| CN114374657B (en) | Data processing method and device | |
| CN110933122A (en) | Method, apparatus, and computer storage medium for managing server | |
| CN117093335A (en) | Task scheduling method and device for distributed storage system | |
| CN112181641A (en) | Thread processing method, device, equipment and storage medium | |
| CN110968370B (en) | Data processing method and device | |
| CN115080657B (en) | Operation token distribution method, system and equipment applied to distributed storage | |
| CN114185682B (en) | Log output method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |