Nothing Special   »   [go: up one dir, main page]

CN111045782A - Log processing method and device, electronic equipment and computer readable storage medium - Google Patents

Log processing method and device, electronic equipment and computer readable storage medium Download PDF

Info

Publication number
CN111045782A
CN111045782A CN201911142773.2A CN201911142773A CN111045782A CN 111045782 A CN111045782 A CN 111045782A CN 201911142773 A CN201911142773 A CN 201911142773A CN 111045782 A CN111045782 A CN 111045782A
Authority
CN
China
Prior art keywords
log
data
processing thread
log data
service processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201911142773.2A
Other languages
Chinese (zh)
Other versions
CN111045782B (en
Inventor
王业通
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing QIYI Century Science and Technology Co Ltd
Original Assignee
Beijing QIYI Century Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing QIYI Century Science and Technology Co Ltd filed Critical Beijing QIYI Century Science and Technology Co Ltd
Priority to CN201911142773.2A priority Critical patent/CN111045782B/en
Publication of CN111045782A publication Critical patent/CN111045782A/en
Application granted granted Critical
Publication of CN111045782B publication Critical patent/CN111045782B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44Arrangements for executing specific programs
    • G06F9/455Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533Hypervisors; Virtual machine monitors
    • G06F9/45558Hypervisor-specific management and integration aspects
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/1805Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815Journaling file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/54Interprogram communication
    • G06F9/546Message passing systems or structures, e.g. queues
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44Arrangements for executing specific programs
    • G06F9/455Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533Hypervisors; Virtual machine monitors
    • G06F9/45558Hypervisor-specific management and integration aspects
    • G06F2009/4557Distribution of virtual machine instances; Migration and load balancing
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00Indexing scheme relating to G06F9/00
    • G06F2209/54Indexing scheme relating to G06F9/54
    • G06F2209/548Queue
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The embodiment of the invention discloses a log processing method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: creating a service processing thread and a log processing thread; writing the log data generated in the running process of the application program into a queue of a preset type through the service processing thread; and acquiring the log data from the queue through the log processing thread, and outputting the log data to a preset log file. The invention introduces the log processing thread, adopts an asynchronous processing mode, obtains the log data generated in the running process of the application program by the service processing thread, sends the log data to the log processing thread through the queue, outputs the log data to the log file by the log processing thread, effectively isolates the large-overhead log processing task from the data processing task, reduces the log processing pressure of the service processing thread, improves the data processing capability and improves the data processing efficiency of the service processing thread.

Description

Log processing method and device, electronic equipment and computer readable storage medium
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a log processing method and apparatus, an electronic device, and a computer-readable storage medium.
Background
With the rapid development of the internet and the continuous improvement of the business volume, the data access flow based on the network is rapidly increased, and particularly the access flow to a data center, a large-scale enterprise, a portal website and the like even reaches the level of 10 Gb/s. Compared with the development of network technology, the increase of network bandwidth and application service is far higher than the increase of server processing speed and memory access speed, and the number of users is continuously increased while the network bandwidth is increased, which causes the resource consumption of the server to be serious, so that the server becomes a network bottleneck and often becomes a network failure point. Therefore, establishing a server cluster and balancing the traffic among the server clusters by using a load balancing technology become a main solution for solving the problems.
Currently, many new techniques are applied to the load balancer in order to improve the load balancing performance. For example: the DPVS (Data Plane Development Kit-Virtual Server) load balancer is a Virtual Server (Virtual Server) based on a DPDK (Data Plane Development Kit), so that a high-performance four-layer load balancer is realized using the DPDK, and Data processing performance and throughput can be improved and the work efficiency of a Data Plane application program can be improved by avoiding interruption, no locking, CPU (central processing unit) affinity, large-page memory and other technologies.
However, when the DPVS load balancer is abnormal, for example: when the memory resource is insufficient or other resources are insufficient, the DPVS load balancer may cause low data processing efficiency due to processing of log data corresponding to the abnormal information. Specifically, the DPVS load balancer prints log data in a multi-core multi-thread polling mode. That is, in the DPVS load balancer, the data cores and the threads are in one-to-one correspondence, and when polling is performed among a plurality of data cores, the thread corresponding to the polled data core prints log data generated by the polled data core through an IO (Input/Output) interface operation, but because the IO interface operation time is long and it is necessary to wait for the IO interface operation to complete before polling a next data core, the polling process time is too long. When an application program is abnormal in the running process, the data core can print a large amount of log data of the abnormal information, and the data processing efficiency is reduced.
Disclosure of Invention
Embodiments of the present invention provide a log processing method, an apparatus, an electronic device, and a computer-readable storage medium, so as to solve the problem that when a DPVS load balancer is abnormal, the DPVS load balancer may have low data processing efficiency due to processing log data corresponding to abnormal information. The specific technical scheme is as follows:
in a first aspect of the present invention, there is provided a log processing method, including: creating a service processing thread and a log processing thread; writing the log data generated in the running process of the application program into a queue of a preset type through the service processing thread; and acquiring the log data from the queue through the log processing thread, and outputting the log data to a preset log file.
Wherein, the creating of the service processing thread and the log processing thread includes: creating a plurality of service processing threads and a log processing thread in a DPVS load balancer; and each service processing thread is correspondingly bound with one service processing data core, and the log processing thread is correspondingly bound with one log processing data core.
Wherein the outputting the log data to a preset log file comprises: determining a service processing data core for generating the log data according to the field information of the log data; and outputting the log data to a log file which is preset for the service processing thread corresponding to the service processing data core according to the service processing thread corresponding to the service processing data core.
The method includes that in the process of writing log data generated in the running process of an application program into a preset type queue through the service processing thread, the method further includes: and if the queue of the preset type is blocked, suspending the writing of the log data generated in the running process of the application program into the queue of the preset type through the service processing thread, or writing the log data only retaining preset field information into the queue of the preset type through the service processing thread.
Wherein the preset field information at least comprises one of the following information: and function name and line number corresponding to the abnormal code.
Before outputting the log data to a preset log file, the method further comprises: if the number of the log data is multiple, performing hash operation on each log data through the log processing thread to obtain a hash value corresponding to each log data; according to the hash values respectively corresponding to the plurality of log data, performing deduplication processing on the plurality of log data to obtain a plurality of non-duplicated log data; the outputting the log data to a preset log file includes: and outputting the non-repeated log data to a preset log file through input/output (IO) interface operation.
Wherein the type of the queue comprises: lock-free queues and/or message queues.
In a second aspect of the present invention, there is also provided a log processing apparatus, including: the creating module is used for creating a service processing thread and a log processing thread; the writing module is used for writing the log data generated in the running process of the application program into a queue of a preset type through the service processing thread; and the output module is used for acquiring the log data from the queue through the log processing thread and outputting the log data to a preset log file.
In another aspect of the present invention, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus; a memory for storing a computer program; a processor for implementing any of the above method steps when executing a program stored in the memory.
In yet another aspect of the present invention, there is also provided a computer-readable storage medium having stored therein instructions, which when run on a computer, cause the computer to perform any of the methods described above.
In yet another aspect of the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform any of the methods described above.
The log processing method, the log processing device, the electronic equipment and the computer readable storage medium introduce a log processing thread, adopt an asynchronous processing mode, obtain log data generated in the running process of an application program by a service processing thread, send the log data to the log processing thread through a queue, and output the log data to a log file by the log processing thread, thereby effectively isolating a large-overhead log processing task from a data processing task, reducing the log processing pressure of the service processing thread, improving the data processing capacity and improving the data processing efficiency of the service processing thread.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
FIG. 1 is a flow diagram of a log processing method according to an embodiment of the invention;
FIG. 2 is a flowchart illustrating a method for processing logs according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating a log processing method according to an embodiment of the invention;
FIG. 4 is a block diagram of a log processing apparatus according to an embodiment of the present invention;
fig. 5 is a block diagram of an electronic device according to an embodiment of the invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention.
The embodiment of the invention provides a log processing method. The execution subject of this embodiment is a device that needs to process log data. For example: the execution subject is a server, a load balancer and the like. The load balancer may be a DPVS load balancer.
Fig. 1 is a flowchart illustrating a log processing method according to an embodiment of the present invention.
Step S110, a service processing thread and a log processing thread are created.
And the service processing thread is used for transmitting the log data generated in the running process of the application program to the log processing thread. The application program runs in the execution main body of the present embodiment.
And the log processing thread is used for outputting the log data transmitted by the service processing thread to a log file.
In this embodiment, with the CPU affinity of the DPVS load balancer, multiple service processing threads and one log processing thread can be created at the DPVS load balancer. Each service processing thread is correspondingly bound with one service processing data core, and each log processing thread is correspondingly bound with one log processing data core.
And step S120, writing the log data generated in the running process of the application program into a preset type queue through the service processing thread.
In this embodiment, the service processing thread may transmit the log data to the log processing thread through a queue of a preset type. The type of the queue includes: lock-free queues and/or message queues.
In the log data, at least the following field information is included: and function name and line number corresponding to the abnormal code. Of course, in the log data, the following field information may also be included: and generating the information of the data core of the log data and the custom field information. The information of the data core is, for example, the ID of the data core. The custom field information can be set according to requirements, for example: and the name of the code module where the abnormal code is located.
Specifically, in the running process of the application program, the data core generates log data, and the service processing thread acquires the log data and writes the log data into a queue of a preset type. The data cores and the service processing threads have corresponding relations, and the service processing threads acquire the log data generated by the corresponding data cores, so that the service processing threads acquiring the log data can be determined according to the information of the data cores in the log data.
And if the queue of the preset type is blocked, the log data generated in the running process of the application program is suspended from being written into the queue of the preset type through the service processing thread, or the log data only retaining preset field information is written into the queue of the preset type through the service processing thread. The preset field information may be important information in log data, such as: the preset field information at least comprises one of the following information: and function name and line number corresponding to the abnormal code. Only important field information is reserved in the log data, so that the length of the log data can be shortened, and the processing pressure of log processing threads is reduced.
Step S130, obtaining the log data from the queue through the log processing thread, and outputting the log data to a preset log file.
And acquiring and caching the log data from the queue through the log processing thread, and outputting the cached log data to a preset log file by adopting the same or different output modes according to different quantities of the cached log data at preset time intervals. For example:
and if the number of the log data is one, directly outputting the log data to a preset log file through IO interface operation.
If the number of the log data is multiple, performing hash operation on each log data through the log processing thread to obtain a hash value corresponding to each log data; according to the hash values respectively corresponding to the plurality of log data, performing deduplication processing on the plurality of log data to obtain a plurality of non-duplicated log data; and outputting the non-repeated log data to a preset log file through IO (Input/Output) interface operation. Further, the deduplication processing includes: and inquiring a plurality of log data with the same hash value, and reserving one log data in the plurality of log data with the same hash value.
The above example adopts different output modes to output the log data to a preset log file. When the log data are more, especially for abnormal log data with strong repeatability, a large amount of repeated log data can be filtered out by carrying out deduplication processing on the log data, and a large amount of meaningless expenses are reduced.
In this embodiment, the number of the service processing threads may be multiple, and before the log data is output to a preset log file, a log file may be correspondingly set for each service processing thread, or a common log file may be set for multiple service processing threads. If a log file is correspondingly set for each service processing thread, determining a service processing data core for generating the log data according to the field information of the log data; and outputting the log data to a log file which is preset for the service processing thread corresponding to the service processing data core according to the service processing thread corresponding to the service processing data core. For example: the field information of the log data may be an ID of the data core.
In the embodiment, a log processing thread is introduced, an asynchronous processing mode is adopted, log data generated in the running process of an application program is obtained by a service processing thread and is sent to the log processing thread through a queue, the log data is output to a log file by the log processing thread, a large-overhead log processing task and a data processing task are effectively isolated, the log processing pressure of the service processing thread is reduced, the data processing capacity is improved, and the data processing efficiency of the service processing thread is improved.
The log processing method of the present invention will be further described below, taking the example of performing log processing at a DPVS load balancer.
Fig. 2 is a detailed flowchart of a log processing method according to an embodiment of the invention.
Step S210, creating a plurality of service processing threads and a log processing thread, and binding the plurality of service processing threads and the log processing thread to corresponding data cores respectively.
The DPVS load balancer comprises a plurality of data cores, and the created threads correspond to the data cores one by one.
And the log processing thread is correspondingly bound with a log processing data core. The log processing data core is used for providing operation resources and storage resources for the log processing thread.
And each service processing thread is correspondingly bound with one service processing data core. Each business processing data core is used for providing operation resources and storage resources for the business processing thread, and each business processing data core runs part of codes of the application program and generates log data in the running process of the part of codes.
In step S220, the plurality of service processing threads respectively write the log data generated by the corresponding data core (service processing data core) into a preset lock-free queue.
A lock-free queue may be correspondingly set for each service processing thread, or a shared lock-free queue may be correspondingly set for a plurality of service processing threads. And the business processing thread writes the log data generated by the corresponding business processing data core into the lock-free queue corresponding to the business processing thread.
Fig. 3 is a schematic diagram illustrating a log processing method according to an embodiment of the invention. The DPVS load balancer comprises n data cores (n is more than 1), each data core is correspondingly bound with a service processing thread, each service processing thread corresponds to a lock-free queue, thus, n lock-free queues are set, and the service processing thread writes log data generated by the corresponding data core into the corresponding lock-free queue, such as: and the business processing thread 1 writes the log generated by the data core 1 into the lock-free queue 1.
In the process of writing log data into the lock-free queue, if the lock-free queue is full (blocked), the log data can be suspended from being written into the lock-free queue, and the log data needing to be written into the lock-free queue is directly discarded; the method can also select important field information in the log data, shorten the length of the log data and write the log data with the shortened length into the lock-free queue, so that the log data is processed without influencing the data processing performance, and the balance between the log processing efficiency and the data processing performance is achieved.
In step S230, the log processing thread obtains log data from the lock-free queue, and performs deduplication processing on the obtained log data at intervals of a preset time period.
After acquiring the log data, the log processing thread does not directly output the log data to the log file, but caches the log data. As shown in fig. 3, the log processing thread acquires log data from the lock-free queue 1 to the lock-free queue n, and caches the acquired log data in the cache of the corresponding log processing data core, so as to integrate the acquired log data to obtain non-duplicated log data, and then outputs the non-duplicated log data to the log file.
In step S240, the log processing thread outputs the log data obtained after the deduplication processing to a preset log file.
In this embodiment, in an asynchronous processing manner, the plurality of service processing threads are respectively responsible for acquiring the log data generated by the corresponding service processing data core, and the log data generated by the plurality of service processing data cores is uniformly output to the log file by the log processing threads, so that the large-overhead log processing task and the data processing task are effectively isolated, the log printing pressure of the service processing data cores is relieved, and the data processing task can be executed efficiently even when each service processing data core is abnormal. The problem that the data core processing capacity is reduced under the abnormal condition is solved through the embodiment, and the stability of the DPVS load balancer is effectively improved, so that a stable network environment is provided for high-concurrency and high-flow services.
The embodiment of the invention also provides a log processing device. Fig. 4 is a block diagram of a log processing apparatus according to an embodiment of the present invention.
The log processing device comprises: a creation module 410, a write module 420, and an output module 430.
And a creating module 410 for creating a service processing thread and a log processing thread.
A writing module 420, configured to write, by the service processing thread, log data generated in an application program running process into a preset type of queue.
And an output module 430, configured to obtain the log data from the queue through the log processing thread, and output the log data to a preset log file.
Wherein the creating module 410 is configured to: creating a plurality of service processing threads and a log processing thread in a DPVS load balancer; and each service processing thread is correspondingly bound with one service processing data core, and the log processing thread is correspondingly bound with one log processing data core.
Wherein the output module 430 is further configured to: determining a service processing data core for generating the log data according to the field information of the log data; and outputting the log data to a log file which is preset for the service processing thread corresponding to the service processing data core according to the service processing thread corresponding to the service processing data core.
The writing module 420 is further configured to, in the process of writing the log data generated in the running process of the application into the queue of the preset type through the service processing thread, suspend writing the log data generated in the running process of the application into the queue of the preset type through the service processing thread if the queue of the preset type is blocked, or write the log data only retaining the preset field information into the queue of the preset type through the service processing thread.
Wherein the preset field information at least comprises one of the following information: and function name and line number corresponding to the abnormal code.
Wherein the output module 430 is further configured to: before outputting the log data to a preset log file, if the number of the log data is multiple, performing hash operation on each log data through the log processing thread to obtain a hash value corresponding to each log data; according to the hash values respectively corresponding to the plurality of log data, performing deduplication processing on the plurality of log data to obtain a plurality of non-duplicated log data; and outputting the non-repeated log data to a preset log file through input/output (IO) interface operation.
Wherein the type of the queue comprises: lock-free queues and/or message queues.
An embodiment of the present invention further provides an electronic device, as shown in fig. 5, including a processor 510, a communication interface 520, a memory 530 and a communication bus 540, where the processor 510, the communication interface 520, and the memory 530 complete mutual communication through the communication bus 540.
A memory 530 for storing a computer program.
The processor 510, when executing the program stored in the memory 530, implements the following steps: creating a service processing thread and a log processing thread; writing the log data generated in the running process of the application program into a queue of a preset type through the service processing thread; and acquiring the log data from the queue through the log processing thread, and outputting the log data to a preset log file.
Wherein, the creating of the service processing thread and the log processing thread includes: creating a plurality of service processing threads and a log processing thread in a DPVS load balancer; and each service processing thread is correspondingly bound with one service processing data core, and the log processing thread is correspondingly bound with one log processing data core.
Wherein the outputting the log data to a preset log file comprises: determining a service processing data core for generating the log data according to the field information of the log data; and outputting the log data to a log file which is preset for the service processing thread corresponding to the service processing data core according to the service processing thread corresponding to the service processing data core.
The method includes that in the process of writing log data generated in the running process of an application program into a preset type queue through the service processing thread, the method further includes: and if the queue of the preset type is blocked, suspending the writing of the log data generated in the running process of the application program into the queue of the preset type through the service processing thread, or writing the log data only retaining preset field information into the queue of the preset type through the service processing thread.
Wherein the preset field information at least comprises one of the following information: and function name and line number corresponding to the abnormal code.
Before outputting the log data to a preset log file, the method further comprises: if the number of the log data is multiple, performing hash operation on each log data through the log processing thread to obtain a hash value corresponding to each log data; according to the hash values respectively corresponding to the plurality of log data, performing deduplication processing on the plurality of log data to obtain a plurality of non-duplicated log data; the outputting the log data to a preset log file includes: and outputting the non-repeated log data to a preset log file through input/output (IO) interface operation.
Wherein the type of the queue comprises: lock-free queues and/or message queues.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
In another embodiment of the present invention, a computer-readable storage medium is further provided, which stores instructions that, when executed on a computer, cause the computer to execute the log processing method described in any one of the above embodiments.
In yet another embodiment of the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the log processing method described in any of the above embodiments.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims (10)

1. A log processing method, comprising:
creating a service processing thread and a log processing thread;
writing the log data generated in the running process of the application program into a queue of a preset type through the service processing thread;
and acquiring the log data from the queue through the log processing thread, and outputting the log data to a preset log file.
2. The method of claim 1, wherein creating a business processing thread and a log processing thread comprises:
creating a plurality of service processing threads and a log processing thread on a DPVS load balancer of a data plane development suite; and each service processing thread is correspondingly bound with one service processing data core, and the log processing thread is correspondingly bound with one log processing data core.
3. The method of claim 2, wherein outputting the log data to a preset log file comprises:
determining a service processing data core for generating the log data according to the field information of the log data;
and outputting the log data to a log file which is preset for the service processing thread corresponding to the service processing data core according to the service processing thread corresponding to the service processing data core.
4. The method according to claim 1, wherein in the process of writing log data generated in the process of running the application program into a queue of a preset type through the business processing thread, the method further comprises:
and if the queue of the preset type is blocked, suspending the writing of the log data generated in the running process of the application program into the queue of the preset type through the service processing thread, or writing the log data only retaining preset field information into the queue of the preset type through the service processing thread.
5. The method of claim 4, wherein the preset field information comprises at least one of: and function name and line number corresponding to the abnormal code.
6. The method of claim 1,
before outputting the log data to a preset log file, the method further comprises:
if the number of the log data is multiple, performing hash operation on each log data through the log processing thread to obtain a hash value corresponding to each log data;
according to the hash values respectively corresponding to the plurality of log data, performing deduplication processing on the plurality of log data to obtain a plurality of non-duplicated log data;
the outputting the log data to a preset log file includes:
and outputting the non-repeated log data to a preset log file through input/output (IO) interface operation.
7. The method of any of claims 1-6, wherein the type of the queue comprises: lock-free queues and/or message queues.
8. A log processing apparatus, comprising:
the creating module is used for creating a service processing thread and a log processing thread;
the writing module is used for writing the log data generated in the running process of the application program into a queue of a preset type through the service processing thread;
and the output module is used for acquiring the log data from the queue through the log processing thread and outputting the log data to a preset log file.
9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 7 when executing a program stored in the memory.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method steps of any one of claims 1 to 7.
CN201911142773.2A 2019-11-20 2019-11-20 Log processing method, device, electronic equipment and computer readable storage medium Active CN111045782B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911142773.2A CN111045782B (en) 2019-11-20 2019-11-20 Log processing method, device, electronic equipment and computer readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911142773.2A CN111045782B (en) 2019-11-20 2019-11-20 Log processing method, device, electronic equipment and computer readable storage medium

Publications (2)

Publication Number Publication Date
CN111045782A true CN111045782A (en) 2020-04-21
CN111045782B CN111045782B (en) 2024-01-12

Family

ID=70232504

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911142773.2A Active CN111045782B (en) 2019-11-20 2019-11-20 Log processing method, device, electronic equipment and computer readable storage medium

Country Status (1)

Country Link
CN (1) CN111045782B (en)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111858077A (en) * 2020-07-15 2020-10-30 济南浪潮数据技术有限公司 Recording method, device and equipment for IO request log in storage system
CN112306980A (en) * 2020-10-30 2021-02-02 深圳市元征科技股份有限公司 Log processing method and terminal equipment
CN112380001A (en) * 2020-10-30 2021-02-19 网宿科技股份有限公司 Log output method, load balancing device and computer readable storage medium
CN112948136A (en) * 2021-02-02 2021-06-11 华东计算技术研究所(中国电子科技集团公司第三十二研究所) Method for implementing asynchronous log record of embedded operating system
CN113342632A (en) * 2021-07-07 2021-09-03 北京物芯科技有限责任公司 Simulation data automatic processing method and device, electronic equipment and storage medium
CN113420032A (en) * 2021-07-20 2021-09-21 奇安信科技集团股份有限公司 Classification storage method and device for logs
CN113810337A (en) * 2020-06-12 2021-12-17 中国电信股份有限公司 Method, device and storage medium for network message duplicate removal
CN114116642A (en) * 2021-11-23 2022-03-01 中能融合智慧科技有限公司 Log duplicate removal and generation method and system based on WRR queue
CN114265637A (en) * 2021-12-24 2022-04-01 中电信数智科技有限公司 Archive management system configuration method, system, electronic device and storage medium
CN117891695A (en) * 2024-03-14 2024-04-16 新华三技术有限公司 Log recording method and device, electronic equipment and storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140280394A1 (en) * 2013-03-14 2014-09-18 Quantum Corporation Multi-Threaded Message Passing Journal
CN104252405A (en) * 2013-06-26 2014-12-31 腾讯科技(深圳)有限公司 Log information output method and device
CN106708578A (en) * 2016-12-23 2017-05-24 北京五八信息技术有限公司 Dual-thread-based journal output method and device
CN106844143A (en) * 2016-12-27 2017-06-13 微梦创科网络科技(中国)有限公司 A kind of daily record duplicate removal treatment method and device
CN108052675A (en) * 2017-12-28 2018-05-18 惠州Tcl家电集团有限公司 Blog management method, system and computer readable storage medium
CN108132955A (en) * 2016-12-01 2018-06-08 中移(苏州)软件技术有限公司 A kind of log storing method and device

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140280394A1 (en) * 2013-03-14 2014-09-18 Quantum Corporation Multi-Threaded Message Passing Journal
CN104252405A (en) * 2013-06-26 2014-12-31 腾讯科技(深圳)有限公司 Log information output method and device
CN108132955A (en) * 2016-12-01 2018-06-08 中移(苏州)软件技术有限公司 A kind of log storing method and device
CN106708578A (en) * 2016-12-23 2017-05-24 北京五八信息技术有限公司 Dual-thread-based journal output method and device
CN106844143A (en) * 2016-12-27 2017-06-13 微梦创科网络科技(中国)有限公司 A kind of daily record duplicate removal treatment method and device
CN108052675A (en) * 2017-12-28 2018-05-18 惠州Tcl家电集团有限公司 Blog management method, system and computer readable storage medium

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113810337A (en) * 2020-06-12 2021-12-17 中国电信股份有限公司 Method, device and storage medium for network message duplicate removal
CN113810337B (en) * 2020-06-12 2023-08-08 天翼云科技有限公司 Method, device and storage medium for network message deduplication
CN111858077A (en) * 2020-07-15 2020-10-30 济南浪潮数据技术有限公司 Recording method, device and equipment for IO request log in storage system
CN112306980A (en) * 2020-10-30 2021-02-02 深圳市元征科技股份有限公司 Log processing method and terminal equipment
CN112380001A (en) * 2020-10-30 2021-02-19 网宿科技股份有限公司 Log output method, load balancing device and computer readable storage medium
CN112380001B (en) * 2020-10-30 2024-11-08 网宿科技股份有限公司 Log output method, load balancing device and computer readable storage medium
CN112948136A (en) * 2021-02-02 2021-06-11 华东计算技术研究所(中国电子科技集团公司第三十二研究所) Method for implementing asynchronous log record of embedded operating system
CN113342632A (en) * 2021-07-07 2021-09-03 北京物芯科技有限责任公司 Simulation data automatic processing method and device, electronic equipment and storage medium
CN113420032A (en) * 2021-07-20 2021-09-21 奇安信科技集团股份有限公司 Classification storage method and device for logs
CN114116642A (en) * 2021-11-23 2022-03-01 中能融合智慧科技有限公司 Log duplicate removal and generation method and system based on WRR queue
CN114265637A (en) * 2021-12-24 2022-04-01 中电信数智科技有限公司 Archive management system configuration method, system, electronic device and storage medium
CN117891695A (en) * 2024-03-14 2024-04-16 新华三技术有限公司 Log recording method and device, electronic equipment and storage medium

Also Published As

Publication number Publication date
CN111045782B (en) 2024-01-12

Similar Documents

Publication Publication Date Title
CN111045782B (en) Log processing method, device, electronic equipment and computer readable storage medium
CN109729106B (en) Method, system and computer program product for processing computing tasks
WO2015101091A1 (en) Distributed resource scheduling method and device
US10613992B2 (en) Systems and methods for remote procedure call
CN103064960A (en) Method and equipment for database query
CN112445857A (en) Resource quota management method and device based on database
CN110069217B (en) Data storage method and device
CN114595043A (en) IO (input/output) scheduling method and device
CN114090623A (en) Method and device for creating cache resources, electronic equipment and storage medium
CN113067875A (en) Access method, device and equipment based on dynamic flow control of micro-service gateway
EP2829972B1 (en) Method and apparatus for allocating stream processing unit
CN110851276A (en) Service request processing method, device, server and storage medium
CN111385255A (en) Asynchronous call implementation method and device, server and server cluster
CN110209466B (en) Resource allocation method, resource allocation device and storage medium
US10616317B2 (en) Method and system for affinity load balancing
CN111694670A (en) Resource allocation method, device, equipment and computer readable medium
CN117370046A (en) Inter-process communication method, system, device and storage medium
CN110990148A (en) Method, device and medium for optimizing storage performance
CN114374657B (en) Data processing method and device
CN110933122A (en) Method, apparatus, and computer storage medium for managing server
CN117093335A (en) Task scheduling method and device for distributed storage system
CN112181641A (en) Thread processing method, device, equipment and storage medium
CN110968370B (en) Data processing method and device
CN115080657B (en) Operation token distribution method, system and equipment applied to distributed storage
CN114185682B (en) Log output method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant