Nothing Special   »   [go: up one dir, main page]

CN109885545A - It stores, the method, apparatus of inquiry log information - Google Patents

It stores, the method, apparatus of inquiry log information Download PDF

Info

Publication number
CN109885545A
CN109885545A CN201910106607.0A CN201910106607A CN109885545A CN 109885545 A CN109885545 A CN 109885545A CN 201910106607 A CN201910106607 A CN 201910106607A CN 109885545 A CN109885545 A CN 109885545A
Authority
CN
China
Prior art keywords
log information
template
log
database
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910106607.0A
Other languages
Chinese (zh)
Inventor
李同军
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to CN201910106607.0A priority Critical patent/CN109885545A/en
Publication of CN109885545A publication Critical patent/CN109885545A/en
Priority to PCT/CN2019/107127 priority patent/WO2020155651A1/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/13File access structures, e.g. distributed indices
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

This application provides a kind of methods for storing log information, this method comprises: identifying first part and the second part of a plurality of log information from a plurality of log information, the first part of a plurality of log information is to be less than or equal to identical part in first threshold log information in a plurality of log information, and the second part of a plurality of log information is to be less than or equal to different part in second threshold log information in a plurality of log information;The second part that a plurality of log information is replaced with occupy-place identifier forms the corresponding log information template of a plurality of log information;The second part of the corresponding log information template of a plurality of log information and a plurality of log information is stored respectively to database.Technical solution provided by the present application can reduce the identical occupied memory space in part in a plurality of log information, save storage resource.

Description

It stores, the method, apparatus of inquiry log information
Technical field
This application involves field of storage, and believe more particularly, to a kind of method for storing log information, inquiry log The method, apparatus and computer readable storage medium of breath.
Background technique
Computer log is in computer system operational process by operating system, middleware, platform itself generate or It is generated by the program assembly of User Exploitation, has recorded the operating status and service condition of equipment and system itself, these descriptions Information essentially describes the key operation that system is carried out and the mistake that system is occurred in the process of running and exception etc..With Family can be by the analysis to system log, and understanding system often occurs in operation or the accidental problem, so as to Targetedly to improve the operation maintenance to system, and then improve the safety and efficiency of system operation.
In traditional log information storage scheme, original log information is stored directly in database.On the one hand, it passes There are a large amount of duplicate information in original log in the log information storage scheme of system, is accounted for so as to cause a large amount of duplicate information With most of storage resource, the waste in resource is caused.On the other hand, it is retrieved in log information of the user to storage During, a large amount of duplicate information can make recall precision low in original log, and user is a large amount of to what is retrieved Original log when carrying out data analysis, a large amount of duplicate parts can make the real-time of customer analysis process poor, be unfavorable for It is quickly found the problem by log.
Summary of the invention
The application provides a kind of method of method and inquiry log information for storing log information, can reduce a plurality of log The duplicate occupied memory space in part in information saves storage resource, and shortens user to the retrieval time of log information, Improve the use value of log information.
In a first aspect, a kind of method for storing log information is provided, this method comprises: identifying from a plurality of log information The first part of a plurality of log information and second part out replace the second of a plurality of log information with occupy-place identifier Part forms the corresponding log information template of a plurality of log information, and the log information template includes a plurality of log The first part of information, by second of a plurality of corresponding log information template of log information and a plurality of log information Divide and is stored respectively to database.
Wherein, the first part of a plurality of log information is to be less than or equal to the first threshold in a plurality of log information It is worth identical part in a log information, the second part of a plurality of log information is to be less than in a plurality of log information Or it is equal to different part in second threshold log information.
It should be understood that first threshold can be less than or equal to a plurality of log information total quantity, second threshold can be less than or Equal to the total quantity of a plurality of log information, first threshold and second threshold can be equal or unequal.
In above-mentioned technical proposal, original log information can be handled, in a plurality of log information, from being less than or wait Identical part and different part are identified in a plurality of log information of a plurality of log information quantity.A plurality of log is believed Identical part only stores one in breath, so that the log information after simplifying does not have to carry in log information template one Long string character string, reduces memory space shared by log information, saves the storage resource consumption in log service system.
In one possible implementation, the method also includes: it is corresponding described to generate a plurality of log information The mark of log information template.
In above-mentioned technical proposal, unique mark corresponding to each log information template can be calculated, so as to To be gone back by log information template identification to log information during the user in later period retrieves log information Original, so that user is available to arrive original log information.Not only section can be understood by the analysis to original log information Point often occurs in operation or the accidental problem, can also pass through mark corresponding to analysis log information template The regularity of distribution excavates potential system exception, and issues warning reminding in advance.
In alternatively possible implementation, user examines by log information of the log services API to storage During rope, the search of keyword can be carried out in log information template.
In above-mentioned technical proposal, the log information that user needs is carried out by passing through keyword in log information template Search, since log information template will not be repeated largely, the log information of user's needs can be quickly found, is reduced The time of search improves the real-time of customer analysis process, is conducive to quickly find the problem by log.
In alternatively possible implementation, by the second part of a plurality of log information and the log information mould The mark of plate is stored in the first space of the database;By the mark of the log information template and the log information template It is stored in the second space of the database.
Second aspect provides a kind of method of inquiry log information, and the method is applied to log information inquiry system, The log information inquiry system includes database, and the database purchase has the second part of a plurality of log information and described more The corresponding log information template of log information, the log information template includes the first part of a plurality of log information, The first part of a plurality of log information is to be less than or equal to first threshold log information in a plurality of log information In identical part, the second part of a plurality of log information is to be less than or equal to the second threshold in a plurality of log information It is worth different part in a log information, which comprises
Inquiry request is received, the inquiry request is for inquiring the log information stored in database;According to the inquiry Request obtains the second part of a plurality of log information of storage from the database and a plurality of log information corresponds to Log information template;Bring the second part of a plurality of log information into a plurality of log information corresponding log information Template obtains corresponding log information, returns to the corresponding log information to the client for issuing the inquiry request,
In above-mentioned technical proposal into, the second part of a plurality of log information can be brought to the corresponding log of a plurality of log information The original log information of formation is back to user by information model.User can be made to the original log information got It is analyzed, understanding node often occurs in operation or the accidental problem.
In one possible implementation, before the reception inquiry request, the method also includes: from a plurality of day Identified in will information a plurality of log information the first part and the second part;
The second part of a plurality of log information is replaced with occupy-place identifier, and it is corresponding to form a plurality of log information Log information template;
The second part of a plurality of corresponding log information template of log information and a plurality of log information is distinguished It stores to the database.
In alternatively possible implementation, the method also includes: generate the corresponding institute of a plurality of log information State the mark of log information template.
In alternatively possible implementation, the method also includes: by the second part of a plurality of log information The first space of the database is stored in the mark of the log information template;By the log information template and the day The mark of will information model is stored in the second space of the database.
The third aspect, provides a kind of device for storing log information, which includes:
Identification module, for identified from a plurality of log information a plurality of log information first part and second Point, wherein the first part of a plurality of log information is to be less than or equal to first threshold item in a plurality of log information Identical part in log information, the second part of a plurality of log information are to be less than or wait in a plurality of log information The different part in second threshold log information;
Processing module is formed described more for replacing the second part of a plurality of log information by occupy-place identifier The corresponding log information template of log information, the log information template includes the first part of a plurality of log information;
Memory module, for by a plurality of corresponding log information template of log information and a plurality of log information Second part is stored respectively to database.
In one possible implementation, described device further include: generation module, for generating a plurality of log letter Cease the mark of the corresponding log information template.
In alternatively possible implementation, the memory module is specifically used for: by the of a plurality of log information The mark of two parts and the log information template is stored in the first space of the database;By the log information template and The mark of the log information template is stored in the second space of the database.
Fourth aspect provides a kind of device of inquiry log information, and described device is applied to log information inquiry system, The log information inquiry system includes database, and the database purchase has the second part of a plurality of log information and described more The corresponding log information template of log information, the log information template includes the first part of a plurality of log information, The first part of a plurality of log information is to be less than or equal to first threshold log information in a plurality of log information In identical part, the second part of a plurality of log information is to be less than or equal to the second threshold in a plurality of log information It is worth different part in a log information, described device includes:
Receiving module, for receiving inquiry request, the inquiry request is for inquiring the log information stored in database;
Module is obtained, for obtaining a plurality of log information of storage from the database according to the inquiry request Second part and the corresponding log information template of a plurality of log information;It is also used to the second of a plurality of log information Part brings the corresponding log information template of a plurality of log information into and obtains corresponding log information.
Optionally, the device of the inquiry log information can also include return module, for asking to the sending inquiry The client asked returns to the corresponding log information
In one possible implementation, described device further include: identification module, for knowing from a plurality of log information Not Chu a plurality of log information the first part and the second part;
Processing module is formed described more for replacing the second part of a plurality of log information by occupy-place identifier The corresponding log information template of log information;
Memory module, for by a plurality of corresponding log information template of log information and a plurality of log information Second part is stored respectively to database.
In alternatively possible implementation, described device further include: generation module, for generating a plurality of log The mark of the corresponding log information template of information.
In alternatively possible implementation, memory module is specifically used for: by second of a plurality of log information Divide the first space that the database is stored in the mark of the log information template;By the log information template and described The mark of log information template is stored in the second space of the database.
5th aspect provides a kind of device for storing log information.The device for storing log information includes at least one Calculate node, each calculate node include processor and memory.The processor of this at least one calculate node is deposited for executing The program code stored in reservoir, to execute the method in above-mentioned first aspect or the possible implementation of first aspect.
6th aspect, provides a kind of device of inquiry log information.The device of inquiry log information includes at least one Calculate node, each calculate node include processor and memory.The processor of this at least one calculate node is deposited for executing The program code stored in reservoir, to execute the method in above-mentioned second aspect or the possible implementation of second aspect.
7th aspect, the application provide a kind of non-transient non-volatile computer readable storage medium storing program for executing, which can It reads to be stored with instruction in storage medium, when it runs at least one calculate node, so that at least one calculate node is held Method in the above-mentioned first aspect of row or the possible implementation of first aspect.
Eighth aspect, the application provide a kind of non-transient non-volatile computer readable storage medium storing program for executing, which can It reads to be stored with instruction in storage medium, when it runs at least one calculate node, so that at least one calculate node is held Method in the above-mentioned second aspect of row or the possible implementation of second aspect.
9th aspect, this application provides a kind of computer program products comprising instruction, when it is at least one calculating When being run on node, so that at least one calculate node executes in above-mentioned first aspect or the possible implementation of first aspect Method.
Tenth aspect, this application provides a kind of computer program products comprising instruction, when it is at least one calculating When being run on node, so that at least one calculate node executes in above-mentioned first aspect or the possible implementation of first aspect Method.
The application can also be further combined on the basis of the implementation that above-mentioned various aspects provide to provide more More implementations.
Detailed description of the invention
Fig. 1 is a kind of schematic block diagram of Distributed Log Server System 100 applied to the embodiment of the present application.
Fig. 2 is a kind of schematic flow chart of method for storing log information provided by the embodiments of the present application.
Fig. 3 is a kind of structural schematic figure of device 300 for storing log information provided by the embodiments of the present application.
Fig. 4 is a kind of structural schematic figure of device 400 for storing log information provided by the embodiments of the present application.
Fig. 5 is a kind of structural schematic figure of the device 500 of inquiry log information provided by the embodiments of the present application.
Fig. 6 is a kind of structural schematic figure of the device 600 of inquiry log information provided by the embodiments of the present application.
Specific embodiment
Below in conjunction with attached drawing, the technical solution in the application is described.
Computer log is in computer system operational process by operating system, middleware, platform itself generate or It is generated by the program assembly of User Exploitation, has recorded the operating status and service condition of equipment and system itself, these descriptions Information essentially describes the key operation that system is carried out and the mistake that system is occurred in the process of running and exception etc..With Family can be by the analysis to system log, and understanding system often occurs in operation or the accidental problem, so as to Targetedly to improve the operation maintenance to system, and then improve the safety and efficiency of system operation.
Under large-scale distributed scene, needs special log system or log services are responsible for collecting, store and From the log information of a large amount of Managed nodes, and provide the log number that relevant Retrieval Interface analyzes their care for user filtering According to.Distributed log system 100 is described in detail below.
Fig. 1 is a kind of schematic block diagram of Distributed Log Server System 100 applied to the embodiment of the present application.Such as Fig. 1 Shown, system 100 may include server-side 110, at least one node.
The embodiment of the present application is not specifically limited the quantity of at least one node, and for ease of description, the application is implemented Example is described by taking node 120, node 130 as an example.
It should be understood that node 120, node 130 can be the different virtual machine (virtual installed on a physical host Machine, VM), it is also possible to the VM installed on different physical hosts.Node 120, node 130 are also possible to physical host.
At least one node in log service system 100 in the process of running, operating system, platform itself or by The program assembly of User Exploitation can produce corresponding log information, which describes what at least one node was carried out Key operation and the mistake occurred in the process of running and exception etc..At least one node can be by the log information of generation It is stored in corresponding journal file.User is by the analysis to system log, not only it will be seen that at least one node is being transported In row often occur or the accidental problem, to targetedly change to the operation maintenance of at least one node It is kind.Potential system exception can also be excavated, and in advance by the regularity of distribution of mark corresponding to analysis log information template Issue warning reminding.
Log collection agency 121, log collection agency 131 are deployed on node 120 and node 130 respectively.In node 120 Log collection act on behalf of the log information saved in the journal file in 121 available nodes 120 under specified directory, and pass through The log information for the node 120 that message-oriented middleware 114 will acquire reports to server-side 110.Likewise, the day in node 130 The log information saved in journal file in the available node 130 of will Collection agent 131 under specified directory, and pass through message The log information for the node 130 that middleware 114 will acquire reports to server-side 110.
It should be noted that log collection proxy module 121, log collection proxy module 131 can be software program reality Existing module.
Data preprocessing module 113 in log service system 110 can receive node 120 and/or node 130 divides After the log information not reported, the log information reported can be pre-processed.And the log information after processing is deposited Storage is in the distributed data base 112 in server-side 110.
Specifically, the data preprocessing module 113 in server-side 110 can receive what message-oriented middleware 114 reported After the log information of at least one node, log information is stored into distributed data base 112 later plus label information. In order to which user is when retrieving relevant log information, relevant log information can be scanned for by label information.
It should be understood that label information can be day self-contained by log information, to be also possible at least one node Will Collection agent is got according to the attribute of the node where log information and the application service operated above.The application is implemented Example is not specifically limited the content of label information, can include but is not limited to: the preservation of the journal file where log information Path, address Internet protocol (internet protocol, IP) of node where log information, the affiliated service of log information Region (region) information of the public cloud at place, information on services belonging to log information is (for example, service identifiers (identifier, ID)), module information (for example, component ID) belonging to log information, tenant ID etc..
Log services application programming interfaces (application programinterface, API) in server-side 110 111 can provide user search function, and user can service API 111 by browser or command-line tool, usage log The interface of offer is interacted with log service system 100.
Specifically, user can scan for the log information that it is needed by log services API 111, for example, can With the various label informations and log keyword to log information, the log letter being stored in distributed data base 112 is obtained Breath.In order to which according to the log information got, understanding node often occurs in operation or the accidental problem, Ke Yiyou Targetedly the operation of node is safeguarded, to improve the running efficiency of system and efficiency of node.
In traditional log storage scheme, due to being that directly original log information is stored directly in database, It is middle to store a large amount of duplicate information.On the one hand, after log information quantity is more than to a certain degree, a large amount of log informations are adopted Collection, the transmission of cross-node, the occupancy of the resource in mass data library will all face huge challenge.On the other hand, original log is believed The direct storage of breath can also reduce the service efficiency to the log information.For example, in the case where searching for scene generally, in flood tide original day It is very long that time-consuming is searched in will information, when usage log information does abnormality detection and other relevant data mining analysis, is needed It concentrates and a large amount of log informations is analyzed, real-time is poor, is unfavorable for quickly finding the problem.
The embodiment of the present application provides a kind of method of storing data, can handle original log information, leads to A large amount of duplicate data in elimination initial data are crossed, memory space shared by original data is reduced.
It should be noted that Fig. 1 is a kind of possible scene applied to the embodiment of the present application, the embodiment of the present application is provided Method be also applied in the scene that a large amount of duplicate non-structured data are stored.
Fig. 2 is a kind of schematic flow chart of method for storing log information provided by the embodiments of the present application.Such as Fig. 1 institute Show, this method may include step 210-230, and step 210-230 is described in detail separately below.
Step 210: first part and the second part of a plurality of log information are identified from a plurality of log information.
The first part and second of a plurality of log information can be identified in the embodiment of the present application from a plurality of log information Part, wherein first part, which can be, is less than or equal to identical portion in first threshold log information in a plurality of log information Point, second part is to be less than or equal to different part in second threshold log information in a plurality of log information.
It should be understood that first threshold can be less than or equal to a plurality of log information total quantity, second threshold can be less than or Equal to the total quantity of a plurality of log information.
It should be noted that first threshold and second threshold can be equal or unequal, the embodiment of the present application is to this It is not especially limited.
Referring to Fig. 1, the log collection that can be at least one node in the embodiment of the present application is acted on behalf of to a plurality of log information In be less than or equal to first threshold log information compare identification, the data prediction mould being also possible in server-side 110 Block 113 compares identification less than or equal to first threshold log information in a plurality of log information.As an example, exist The processor and/or memory of log collection agency 121 in node 120 feelings more sufficient for the data volume of its processing Under condition, after log collection agency 121 can obtain the original log information of storage in the journal file from specified directory, It is identified to first threshold log information is less than or equal in a plurality of log information, identifies above-mentioned first part and second Part.As another example, the processor and/or memory of the log collection agency 121 in node 120 is relative to its processing Data volume for it is relatively limited in the case where, the data preprocessing module 113 in server-side 110 can receive at least one After multiple original log informations for reporting of log collection agency in node, to being less than or equal to the in a plurality of log information One threshold value log information compares identification, is less than or equal to first threshold log letter in a plurality of log information by comparing Breath, identifies above-mentioned first part and second part.
Step 220: replacing the second part of a plurality of log information with occupy-place identifier, form a plurality of log information pair The log information template answered.
The embodiment of the present application can be handled from the second part of the above-mentioned a plurality of log information identified, example It such as, can be by replacing the second part in a plurality of log information with occupy-place identifier, so that it is corresponding to form an a plurality of log One log information template, so that being the corresponding relationship of 1:M between log information template and M original log information, to disappear Except duplicate parts a large amount of in a plurality of log information.Wherein, M is the positive integer greater than 1.
The type in the second part of a plurality of log information can be distinguished in the embodiment of the present application by occupy-place identifier, The type for distinguishing changing unit in the second part of a plurality of log information can not be had to, the application is not especially limited this.Make For an example, when types of variables in the second part for distinguishing a plurality of log information by occupy-place identifier, %d can be used As the occupy-place identifier for indicating numeric type variable, use %s as the variable occupy-place identifier for indicating character string type.
Step 230: by the second of a plurality of corresponding log information template of log information and a plurality of log information Part is stored respectively to database.
Comparison can will distinguished to multiple log informations in the embodiment of the present application, identify multiple log informations First part and second part.The corresponding log information template of multiple log informations is stored in distributed data base 112 An entity in, and the part changed in each log information is stored in another entity of distributed data base 112.
In above-mentioned technical proposal, original log information can be handled, be changed in identifying multiple log informations Part and constant part after, part constant in each log information is only stored one, and by each log information It is condensed into part and log information template identification, the relevant label information of the log information etc. including its variation.To make Log information after must simplifying does not have to carry a lot of character string in template, and it is empty to reduce storage shared by log information Between, save the storage resource consumption in log service system.
Optionally, in some embodiments, template identification corresponding to each log information template can be generated, and will Each log information simplifies to include: log information template identification, variate-value, timestamp, label information etc..In order to user It, can be by day for including in each log information during being retrieved to log information and being reduced into raw information Will information model mark, gets log information template.So as to include according in log information template and log information Timestamp, variate-value log information is reverted to original state.
It should be understood that the timestamp for including in log information after simplifying can be what log information itself recorded.Label Information can be the log collection being also possible at least one node by log information itself note and act on behalf of according to log information What the node at place and the attribute of the application service operated above were got, it specifically please refers to and above label information is retouched It states, details are not described herein again.
The embodiment of the present application can extract the feature of an information by information fingerprint extraction algorithm, and according to this information Feature convert thereof into one group of code.The code can be used as unique template mark corresponding to each log information module Know, identical log information template mark having the same under different nodes can be made in this way.As an example rather than It limits, can be generated corresponding to each log information module only by summary info (message-digest, MD5) algorithm One template identification.
It should be understood that MD5 algorithm is a kind of Cryptographic Hash Function being widely used, it can produce out 128 (16 words Section) hashed value (hash value), for ensuring that information transmission is complete consistent.Below in conjunction with specific embodiment to passing through The realization process that MD5 algorithm generates unique template identification corresponding to each log information module is described in detail, Details are not described herein again.
In the embodiment of the present application, generated by using information fingerprint technique unique corresponding to each log information template Template identification, identical log information template mark having the same under different nodes can be made, so as to reduce Be unified for overhead brought by the communication that each log information template is distributed between the cross-node of its corresponding mark and Complexity.
It is specific to being simplified described in the above process to log information below with reference to specific log information content Realization process is described in detail.For ease of description, below to extract a log information from 3 original log information It is illustrated for template.
It should be understood that following example is used for the purpose of helping skilled in the art to understand the embodiment of the present application, and have to By the embodiment of the present application be limited to illustrated by specific value or concrete scene.Those skilled in the art are according to below given Example, it is clear that the modification or variation of various equivalences can be carried out, such modification or variation also fall into the model of the embodiment of the present application In enclosing.
Original log information 1:[2018-11-09 13:39:14.696] [10.0.26.102] [INF0] [bulk_ thread-2][BulkHandlerRunable:submits 138]get data from queue timeout!
Original log information 2:[2018-11-09 13:39:15.698] [10.0.26.102] [INF0] [bulk_ thread-5][BulkHandlerRunable:submits 138]get data from queue timeout!
Original log information 3:[2018-11-09 13:39:16.066] [10.0.26.102] [INF0] [bulk_ thread-3][BulkHandlerRunable:submits 138]get data from queue timeout!
Step 1: identifying the part and constant portion changed in each original log information in 3 original log information Point.
It can be identified by schema extraction algorithm in the embodiment of the present application and extract each original in 3 original log information The part and constant part changed in beginning log information.It is non-limiting as example, iteration subregion Web log mining can be passed through (iterative partitioning log mining, IPLOM) algorithm is to identifying each original in 3 original log information The part and constant part changed in beginning log information, and can be replaced with occupy-place identifier every in 3 original log information The part of a original log information change.
Schema extraction algorithm by step 1, the recognition result after comparing to above-mentioned 3 original log information is such as Shown in lower:
Original log information 1 is as follows relative to the comparison result of original log information 2,3:
Constant part: [10.0.26.102] [INF0] [bulk_thread-] [BulkHandlerRunable: submits 138]get data from queue timeout!
The part of variation: 2, [2018-11-09 13:39:14.696]
With the numerical part changed in occupy-place identifier %d replacement original log information 1, original log is indicated with tm field The field of logging time in information 1, log original log information 1 can indicate are as follows:
Tm [10.0.26.102] [INF0] [bulk_thread-%d] [BulkHandler Runable:submits 138]get data from queue timeout!
Wherein, %d=2, tm=[2018-11-09 13:39:14.696]
Original log information 2 is as follows relative to the comparison result of original log information 1,3:
Constant part: [10.0.26.102] [INF0] [bulk_thread-] [BulkHandlerRunable: submits 138]get data from queue timeout!
The part of variation: 5, [2018-11-09 13:39:15.698]
With the part changed in occupy-place identifier %d replacement original log information 2, original log information is indicated with tm field The field of logging time in 2, original log information 2 can indicate are as follows:
Tm [10.0.26.102] [INF0] [bulk_thread-%d] [BulkHandler Runable:submits 138]get data from queue timeout!
Wherein, %d=5, tm=[2018-11-09 13:39:15.698].
Original log information 3 is as follows relative to the comparison result of original log information 1,2:
Constant part: [10.0.26.102] [INF0] [bulk_thread-] [BulkHandlerRunable: submits 138]get data from queue timeout!
The part of variation: 3, [2018-11-09 13:39:16.066]
With the part changed in occupy-place identifier %d replacement original log information 3, original log information is indicated with tm field The field of logging time in 3, original log information 3 can indicate are as follows:
Tm [10.0.26.102] [INF0] [bulk_thread-%d] [BulkHandler Runable:submits 138]get data from queue timeout!
Wherein, %d=3, tm=[2018-11-09 13:39:16.066].
In conclusion the log information template identified in 3 original log information are as follows: tm [INF0] [bulk_ Thread-%d] [BulkHandlerRunable:submits 138] get data from queue timeout!
Step 2: generating the corresponding unique mark of log information template.
Under the distributed scene including at least one node, there may be identical log information moulds for different nodes Plate, so that the mark of log information template and the template corresponds.Digital Fingerprinting Technology can be used in the embodiment of the present application Realize that the mark of log information template and the template corresponds, for example, it is corresponding to generate log information template using MD5 algorithm Unique mark.
Specifically, the character string of the above-mentioned log information template of MD5 algorithm is calculated, generation 128 (16 byte) dissipates Train value, the hashed value can be used as the corresponding unique mark of above-mentioned log information template.
It can also be selected from the hashed value for generating one 128 (16 byte) by MD5 algorithm in the embodiment of the present application 64 out (8 byte) hashed values, and using 64 hashed values as corresponding unique mark of actual log information template. There are many implementations that 64 hashed values are selected from 128 hashed values, as an example, selects from 128 hashed values 64 odd number bit positions are selected out as 64 hashed values.As another example, 64 idols are selected from 128 hashed values The part of numerical digit is as 64 hashed values.As another example, intermediate 64 are selected from 128 hashed values as 64 Position hashed value.As another example, it is added after 128 hashed values can also being folded and obtains 64 hashed values.
For example, can indicate the mark of above-mentioned log information template with tid field.
{ tid=5085657271133051000
Tm [INF0] [bulk_thread-%d] [BulkHandlerRunable:submits 138] get data from queue timeout!}
Step 3: the field of logging time in conversion original log information.
It should be understood that step 3 is optional.
In order to reduce the occupied storage resource of character string in the field for indicating logging time, original log can be believed Logging time in breath is converted, and logging time is converted into the millisecond offset numbers of fiducial time.It as an example, can be with Using January 1 as fiducial time in 1970, the time recorded in each primitive beginning log information was calculated relative to fiducial time Number of seconds deviant.
The tm=for example, tm=1541770754696 in calculated original log information 1, in original log information 2 1541770755698, the tm=1541770756066 in original log information 3.
Step 4: indicating the part of the variation in each original log information.
The field for storing variate-value in each original log information can be increased in the embodiment of the present application, shown as one Example, indicates variate-value list character string with mg field.
Variate-value for example, the variable value list character string mg=(2) in original log information 1, in original log information 2 List character string mg=(5), the variable value list character string mg=(3) in original log information 3.
The field for storing the number of variate-value in each original log information can also be increased in the embodiment of the present application, made For an example, the number of variate-value is indicated with VL field.
For example, the number VL=1 of the variate-value in original log information 1, the number of the variate-value in original log information 2 VL=1, the number VL=1 of the variate-value in original log information 3.
Step 5: adding the label information of each original log information.
Relevant label information can be added to each original log information in the embodiment of the present application, in order to by each Kind label information gets the log information of needs.
For ease of description, the embodiment of the present application is using label information as the path of the journal file where log information, day It is illustrated for the IP address of node where will information.
IP=by taking label information is the IP address of log information place node as an example, in original log information 1 10.0.26.102, the IP=10.0.26.102 in original log information 2, the IP=in original log information 3 10.0.26.102。
By taking the path of the journal file where label information is log information as an example, the embodiment of the present application can be to each The path of journal file carries out information fingerprint extraction, forms the corresponding unique mark in path of each journal file, from And storage resource can be saved.Specific information fingerprint extracts and generates the process of mark and the calculating of log information template identification Process is similar, specifically please refers to the generating process of log information template identification, details are not described herein again.
As an example, the path of the journal file where indicating log information with pid field.For example, original log Pid=864995200973638000 in information 1, the pid=864995200973638000 in original log information 2, it is former Pid=864995200973638000 in beginning log information 3.
By above-mentioned steps 1- step 5, by above-mentioned 3 original log information reductions at: timestamp+template identification+variable+ Associated tag information.And template identification and template can be respectively stored in the different tables of distributed data base 112.
As an example, original log information can be split into 3 parts to store respectively.For example, table 1 is for storing Log information after simplifying, table 2 are used to store the path of the journal file where log information, and table 3 is original for storing 3 The template of log information.
The content stored in table 1 is as follows:
Log information 1 after simplifying can indicate are as follows:
Log information 2 after simplifying can indicate are as follows:
Log information 3 after simplifying can indicate are as follows:
The content stored in table 2 is as follows:
The content stored in table 3 is as follows:
It should be understood that in the embodiment of the present application, step some or all of can be executed in the embodiment of the present application, these Step or operation are only examples, and the deformation of other operations or various operations can also be performed in the embodiment of the present application.In addition, each Step can be executed according to the different sequences that the embodiment of the present application is presented, and it is possible to not really want to execute the application implementation All operationss in example.
Optionally, in some embodiments, user is passing through 111 pairs of log services API storages in the embodiment of the present application During log information is retrieved, the search of keyword can be carried out, in log information template so as to quickly search The log information needed to user, reduces the time of search.
Optionally, in some embodiments, the embodiment of the present application can also be at 111 end log services API to the essence of storage Log information after letter is restored, and arrives original log information in order to which user is available, and can be by original Log information analysis, understand node in operation often occur or the accidental problem.
Specifically, as shown in Figure 1, user can be defeated by 111 end log services API in distributed data base 112 Enter various label informations and log keyword to get the log information of needs.Distributed data base 112 can be according to pass The log information template identification tid for including in key word obtains associated log information template and variable value list character string Mg, and according to the log information template and variable value list character string mg got, variable value list character string mg is automatic The position of the occupy-place identifier of log information template is brought into, to restore original log information.
It should be noted that the distributed data base 112 in the embodiment of the present application can be distributed relational data Library is also possible to the non-relational database with join function.Distributed relevant database or with join function Non-relational database associated log information template can be obtained by log information template identification tid and variate-value arranges Table character string mg.
Optionally, in some embodiments, user can service the interface that API 111 is provided by browser usage log It is interacted with log service system 100.
Specifically, user can be requested by browser input inquiry, it may include for needing to look into the inquiry request The keyword of the log information of inquiry and/or relevant label information.Database can be according to the log information in inquiry request Keyword and/or relevant label information obtain the log information of storage from the table of database, and can be by the log of storage Information feeds back to user.User not only can by the analysis to log information, understand node in operation often occur or The accidental problem can also excavate potential system by the regularity of distribution of mark corresponding to analysis log information template System is abnormal, and issues warning reminding in advance.
It should be understood that the keyword for including in inquiry request can be some character string in log information.
As an example, the inquiry request of user's input is to inquire the log information in " on January 1st, 2018 ", database The value of tm can be determined according to the time " on January 1st, 2018 " of inquiry after receiving inquiry request.And it can be with Satisfactory log information is obtained from the log information stored in the table 1 of database according to the value of tm.Meanwhile database Join function log information template corresponding to the log information template identification in the log information can be obtained from table 3, The log information stored in table 1 is brought into log information template, generates original log information, and original log can be believed Breath feeds back to user.As another example, user can also a word in inquiry request in input journal information model Symbol string, database can obtain log information template corresponding to the log information template identification in the log information in table 3. The log with the log information template identification is obtained in the log information that the join function of database can also be stored from table 1 The log information stored in table 1 is brought into log information template by information, generates original log information, and can will be original Log information feeds back to user.
It is below that the log information for inquiring IP address to generate on " 10.0.26.102 " is made with the inquiry request that user inputs For example, detailed explanation is carried out to the process of user query log information
It include: IP=10.0.26.102 in the inquiry request that user is inputted by log services API 111.Database 112 According to IP=10.0.26.102, the log letter of IP=10.0.26.102 can be got from the log information stored in table 1 Breath, as follows:
The join function of database 112 can also be tid=according in above-mentioned log information 5085657271133051000, the corresponding log information template of tid=5085657271133051000 is obtained from table 3, such as Shown in lower:
Database 112 can restore the timestamp information of original carrying in log information according to the tm in log information. For example, tm=1541770754696 is reduced into [2018-11-09 13:39:14.696], also by tm=1541770755698 Original be reduced at [2018-11-09 13:39:15.698], by tm=1541770756066 [2018-11-09 13:39: 16.066]。
Database 112 can also by the timestamp information of original carrying and it is above-mentioned got from table 1 meet IP= 10.0.26.102 log information is brought into log information template, forms original log information.Original log information is provided To the front-end interface of log services API 111, to realize the reduction to original log information.
The original log information for being supplied to the front-end interface of log services API 111 is as follows:
Log information 1:
[2018-11-09 13:39:14.696][10.0.26.102][INF0][bulk_thread-2] [BulkHandlerRunable:submits 138]get data from queue timeout!
Log information 2:
[2018-11-09 13:39:15.698][10.0.26.102][INF0][bulk_thread-5] [BulkHandlerRunable:submits 138]get data from queue timeout!
Log information 3:
[2018-11-09 13:39:16.066][10.0.26.102][INF0][bulk_thread-3] [BulkHandlerRunable:submits 138]get data from queue timeout!
Above in association with Fig. 1 to Fig. 2, the embodiment of the present application is described in detail, the method and inquiry of storage log information are provided The Installation practice of the application is described in detail below in conjunction with Fig. 3-Fig. 4 in the method for log information.It should be understood that embodiment of the method Description corresponded to each other with the description of Installation practice, therefore, the part being not described in detail may refer to previous methods embodiment.
Fig. 3 is a kind of structural schematic figure of device 300 for storing log information provided by the embodiments of the present application.The dress Setting 300 includes: identification module 310, processing module 320, memory module 330.
Identification module 310, for identifying the first part and of a plurality of log information from a plurality of log information Two parts, wherein the first part of a plurality of log information is to be less than or equal to the first threshold in a plurality of log information It is worth identical part in a log information, the second part of a plurality of log information is to be less than in a plurality of log information Or it is equal to different part in second threshold log information;
Processing module 320, for replacing the second part of a plurality of log information by occupy-place identifier, described in formation The corresponding log information template of a plurality of log information;
Memory module 330, for believing a plurality of corresponding log information template of log information and a plurality of log The second part of breath is stored respectively to database.
Optionally, in some embodiments, described device 300 further include:
Generation module 340, for generating the mark of the corresponding log information template of a plurality of log information.
Optionally, in some embodiments, the memory module 330 is specifically used for: by the of a plurality of log information The mark of two parts and the log information template is stored in the first space of the database;By the log information template and The mark of the log information template is stored in the second space of the database.
It should be understood that the device 300 of the storage log information of the embodiment of the present application can pass through specific integrated circuit (application-specific integrated circuit, ASIC) is realized or programmable logic device (programmable logic device, PLD) is realized, above-mentioned PLD can be complicated process logical device (complex Programmable logical device, CPLD), field programmable gate array (field-programmable gate Array, FPGA), Universal Array Logic (generic array logic, GAL) or any combination thereof.Software can also be passed through When realizing the method for storage log information shown in Fig. 2, the device 300 and its modules for storing log information may be soft Part module.
Fig. 4 is a kind of structural schematic figure of device 400 for storing log information provided by the embodiments of the present application.Storage The device 400 of log information includes at least one calculate node 410, and calculate node 410 may include processing unit 411 and communication Interface 412, processing unit 411 is for executing function defined in various software programs, for example, for realizing storage log information Function.Communication interface 412 is used to carry out communication interaction with other calculate nodes, and other equipment can be other physical services Device, specifically, communication interface 412 can be adapter.
Optionally, which can also include input/output interface 413, and input/output interface 413 is connected with Input-output apparatus, information for receiving input export operating result.Input/output interface 413 can be mouse, key Disk, display or CD-ROM drive etc..Optionally, which can also include additional storage 414, also commonly referred to as outer It deposits, the storage medium of additional storage 414 can be magnetic medium (for example, floppy disk, hard disk, tape), optical medium (such as light Disk) or semiconductor medium (such as solid state hard disk) etc..
Optionally, calculate node 410 can also include bus 415.Wherein, processing unit 411, communication interface 412, defeated Enter/output interface 413, additional storage 414 can connect by bus 415.Bus 415 can be Peripheral Component Interconnect standard (peripheral component interconnect, PCI) bus or expanding the industrial standard structure (extended Industry standard architecture, EISA) bus etc..Bus 415 can be divided into address bus, data/address bus, Control bus etc..Only to be indicated with a line in Fig. 4, it is not intended that an only bus or a type of convenient for indicating Bus.
Processing unit 411 can there are many specific implementation forms, such as processing unit 411 may include 4112 He of processor Memory 4111, processor 4112 execute the correlation of embodiment shown in Fig. 2 according to the program instruction stored in memory 4111 Operation.Processor 4112 can be central processing unit (central processing unit, CPU).The processor can be with It is other general processors, digital signal processor (digital signal processor, DSP), specific integrated circuit (application specific integrated circuit, ASIC), ready-made programmable gate array (field Programmable gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic, Discrete hardware components etc..General processor can be microprocessor or the processor is also possible to any conventional processor Deng.Or the processor 410 uses one or more integrated circuits, for executing relative program, to realize the embodiment of the present application Provided technical solution.
It should be understood that the processor 4112 of calculate node 410 can by the program instruction that is stored in memory 4111 come Run at least one of identification module 310 as shown in Figure 3, processing module 320, memory module 330.
It should also be understood that the memory 4111 or additional storage 414 of calculate node 410 can also be stored with described in Fig. 2 Database.
Above and other operation and/or function of each unit in the device 400 of log information is stored respectively for reality The corresponding process of method in existing Fig. 2, for sake of simplicity, details are not described herein.
Fig. 5 is a kind of structural schematic figure of the device 500 of inquiry log information provided by the embodiments of the present application.This is looked into Asking the 500 of log information includes: receiving module 510, acquisition module 520, return module 530.
Receiving module 510, for receiving inquiry request, the inquiry request is used to inquire the log letter stored in database Breath;
Module 520 is obtained, for obtaining a plurality of log of storage from the database according to the inquiry request The second part of information and the corresponding log information template of a plurality of log information;By second of a plurality of log information Point bringing the corresponding log information template of a plurality of log information into obtains the corresponding log information of the inquiry request;
It optionally, can also include return module 530, for returning to the correspondence to the client for issuing the inquiry request Log information.
Return module 530 and receiving module 510 can also be realized by same module.
Optionally, in some embodiments, described device 500 further include:
Identification module 540, for identifying the first part of a plurality of log information from a plurality of log information With the second part;
Processing module 550 is formed described more for replacing the second part of a plurality of log information with occupy-place identifier The corresponding log information template of log information;
Memory module 560, for believing a plurality of corresponding log information template of log information and a plurality of log The second part of breath is stored respectively to database.
Optionally, in some embodiments, described device 500 further include:
Generation module 570, for generating the mark of the corresponding log information template of a plurality of log information.
Optionally, in some embodiments, memory module 560 is specifically used for: by second of a plurality of log information Divide the first space that the database is stored in the mark of the log information template;By the log information template and described The mark of log information template is stored in the second space of the database.
It should be understood that the device 500 of the inquiry log information of the embodiment of the present application can pass through specific integrated circuit (application-specific integrated circuit, ASIC) is realized or programmable logic device (programmable logic device, PLD) is realized, above-mentioned PLD can be complicated process logical device (complex Programmable logical device, CPLD), field programmable gate array (field-programmable gate Array, FPGA), Universal Array Logic (generic array logic, GAL) or any combination thereof.Software can also be passed through When realizing the method for inquiry log information, the device 500 and its modules of inquiry log information may be software module.
Fig. 6 is a kind of structural schematic figure of the device 600 of inquiry log information provided by the embodiments of the present application.Inquiry The device 600 of log information includes at least one calculate node 610, and calculate node 610 may include processing unit 611 and communication Interface 612, processing unit 611 is for executing function defined in various software programs, for example, for realizing storage log information Function.Communication interface 412 is used to carry out communication interaction with other calculate nodes, and other equipment can be other physical services Device, specifically, communication interface 612 can be adapter.
Optionally, which can also include input/output interface 613, and input/output interface 413 is connected with Input-output apparatus, information for receiving input export operating result.Input/output interface 613 can be mouse, key Disk, display or CD-ROM drive etc..Optionally, which can also include additional storage 614, also commonly referred to as outer It deposits, the storage medium of additional storage 614 can be magnetic medium (for example, floppy disk, hard disk, tape), optical medium (such as light Disk) or semiconductor medium (such as solid state hard disk) etc..
Optionally, calculate node 610 can also include bus 615.Wherein, processing unit 611, communication interface 612, defeated Enter/output interface 613, additional storage 614 can connect by bus 615.The function and realization of bus 615 and bus 415 Mode is similar.
Processing unit 611 can there are many specific implementation forms, such as processing unit 611 may include 6112 He of processor Memory 6111, processor 6112 execute the relevant operation of above-described embodiment according to the program instruction stored in memory 6111. Processor 6112 can be central processing unit (central processing unit, CPU).The processor can also be it Its general processor, digital signal processor (digital signal processor, DSP), specific integrated circuit (application specific integrated circuit, ASIC), ready-made programmable gate array (field Programmable gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic, Discrete hardware components etc..General processor can be microprocessor or the processor is also possible to any conventional processor Deng.Or the processor 610 uses one or more integrated circuits, for executing relative program, to realize the embodiment of the present application Provided technical solution.
It should be understood that the processor 6112 of calculate node 610 can by the program instruction that is stored in memory 6111 come At least one of run receiving module 510 as shown in Figure 5, obtain module 520, return module 530.
It should also be understood that the memory 6111 or additional storage 614 of calculate node 610 can also be stored with described in Fig. 2 Database.
Above and other operation and/or function of each unit in the device 600 of inquiry log information is respectively for reality The corresponding process of the method for existing above-mentioned inquiry log information, for sake of simplicity, details are not described herein.
Above-described embodiment can be realized wholly or partly by software, hardware, firmware or any other combination.When When using software realization, above-described embodiment can be realized entirely or partly in the form of a computer program product.The calculating Machine program product includes one or more computer instructions.When loading on computers or executing the computer program instructions, It entirely or partly generates according to process or function described in the embodiment of the present application.The computer can for general purpose computer, Special purpose computer, computer network or other programmable devices.The computer instruction can store to be deposited computer-readable In storage media, or from a computer readable storage medium to the transmission of another computer readable storage medium, for example, described Computer instruction can pass through wired (such as coaxial cable, light from a web-site, computer, server or data center Fine, Digital Subscriber Line (DSL)) or wireless (such as infrared, wireless, microwave etc.) mode to another web-site, computer, clothes Business device or data center are transmitted.The computer readable storage medium can be any available Jie that computer can access Matter includes either the data storage devices such as server, the data center of one or more usable medium set.Described use is situated between Matter can be magnetic medium (for example, floppy disk, hard disk, tape), optical medium (for example, DVD) or semiconductor medium.Semiconductor Medium can be solid state hard disk (solid state drive, SSD).
Those of ordinary skill in the art may be aware that list described in conjunction with the examples disclosed in the embodiments of the present disclosure Member and algorithm steps can be realized with the combination of electronic hardware or computer software and electronic hardware.These functions are actually It is implemented in hardware or software, the specific application and design constraint depending on technical solution.Professional technician Each specific application can be used different methods to achieve the described function.
It is apparent to those skilled in the art that for convenience and simplicity of description, the system of foregoing description, The specific work process of device and unit, can refer to corresponding processes in the foregoing method embodiment, and details are not described herein.
In several embodiments provided herein, it should be understood that disclosed systems, devices and methods, it can be with It realizes by another way.For example, the apparatus embodiments described above are merely exemplary, for example, the unit It divides, only a kind of logical function partition, there may be another division manner in actual implementation, such as multiple units or components It can be combined or can be integrated into another system, or some features can be ignored or not executed.Another point, it is shown or The mutual coupling, direct-coupling or communication connection discussed can be through some interfaces, the indirect coupling of device or unit It closes or communicates to connect, can be electrical property, mechanical or other forms.
The unit as illustrated by the separation member may or may not be physically separated, aobvious as unit The component shown may or may not be physical unit, it can and it is in one place, or may be distributed over multiple In network unit.It can select some or all of unit therein according to the actual needs to realize the mesh of this embodiment scheme 's.

Claims (18)

1. a kind of method for storing log information, which is characterized in that the described method includes:
First part and the second part of a plurality of log information are identified from a plurality of log information, wherein described a plurality of The first part of log information is identical less than or equal in first threshold log information in a plurality of log information Part, the second part of a plurality of log information are to be less than or equal to second threshold log in a plurality of log information Different part in information;
The second part of a plurality of log information is replaced with occupy-place identifier, forms the corresponding log of a plurality of log information Information model, the log information template include the first part of a plurality of log information;
The second part of a plurality of corresponding log information template of log information and a plurality of log information is stored respectively To database.
2. the method according to claim 1, wherein the method also includes:
Generate the mark of the corresponding log information template of a plurality of log information.
3. according to the method described in claim 2, it is characterized in that, described by the corresponding log information of a plurality of log information The second part of template and a plurality of log information is stored respectively to database, comprising:
The mark of the second part of a plurality of log information and the log information template is stored in the of the database One space;
The mark of the log information template and the log information template is stored in the second space of the database.
4. a kind of method of inquiry log information, which is characterized in that the method is applied to log information inquiry system, the day Will information query system includes database, the database purchase have a plurality of log information second part and a plurality of log The corresponding log information template of information, the log information template includes the first part of a plurality of log information, described more The first part of log information be less than or equal in a plurality of log information it is identical in first threshold log information Part, the second part of a plurality of log information is to be less than or equal to second threshold item day in a plurality of log information Different part in will information, which comprises
Inquiry request is received, the inquiry request is for inquiring the log information stored in database;
The second part of a plurality of log information of storage and described is obtained from the database according to the inquiry request The corresponding log information template of a plurality of log information;
Bring the second part of a plurality of log information into a plurality of log information corresponding log information template acquisition pair The log information answered.
5. according to the method described in claim 4, it is characterized in that, the method is also wrapped before the reception inquiry request It includes:
Identified from a plurality of log information a plurality of log information the first part and a plurality of log information The second part;
The second part of a plurality of log information is replaced with occupy-place identifier, and it is corresponding to form a plurality of log information Log information template;
The second part of a plurality of corresponding log information template of log information and a plurality of log information is stored respectively To the database.
6. according to the method described in claim 5, it is characterized in that, the method also includes:
Generate the mark of the corresponding log information template of a plurality of log information.
7. according to the method described in claim 6, it is characterized in that, the method also includes:
The mark of the second part of a plurality of log information and the log information template is stored in the of the database One space;
The mark of the log information template and the log information template is stored in the second space of the database.
8. a kind of device for storing log information, which is characterized in that described device includes:
Identification module, for identifying first part and the second part of a plurality of log information from a plurality of log information, Wherein, the first part of a plurality of log information is to be less than or equal to first threshold log in a plurality of log information Identical part in information, the second part of a plurality of log information are to be less than or equal to the in a plurality of log information Different part in two threshold value log informations;
Processing module forms a plurality of day for replacing the second part of a plurality of log information by occupy-place identifier The corresponding log information template of will information, the log information template include the first part of a plurality of log information;
Memory module, for by the second of a plurality of corresponding log information template of log information and a plurality of log information Part is stored respectively to database.
9. device according to claim 8, which is characterized in that described device further include:
Generation module, for generating the mark of the corresponding log information template of a plurality of log information.
10. device according to claim 9, which is characterized in that the memory module is specifically used for:
The mark of the second part of a plurality of log information and the log information template is stored in the of the database One space;
The mark of the log information template and the log information template is stored in the second space of the database.
11. a kind of device of inquiry log information, which is characterized in that described device is applied to log information inquiry system, described Log information inquiry system includes database, the database purchase have a plurality of log information second part and a plurality of day The corresponding log information template of will information, the log information template includes the first part of a plurality of log information, described The first part of a plurality of log information is to be less than or equal to phase in first threshold log information in a plurality of log information Same part, the second part of a plurality of log information are to be less than or equal to second threshold item in a plurality of log information Different part, described device include: in log information
Receiving module, for receiving inquiry request, the inquiry request is for inquiring the log information stored in database;
Module is obtained, the of a plurality of log information for obtaining storage from the database according to the inquiry request Two parts and the corresponding log information template of a plurality of log information;Bring the second part of a plurality of log information into institute It states the corresponding log information template of a plurality of log information and obtains corresponding log information.
12. device according to claim 11, which is characterized in that described device further include:
Identification module, for identifying the first part and described of a plurality of log information from a plurality of log information Two parts;
Processing module forms a plurality of log for replacing the second part of a plurality of log information with occupy-place identifier The corresponding log information template of information;
Memory module, for by the second of a plurality of corresponding log information template of log information and a plurality of log information Part is stored respectively to database.
13. device according to claim 12, which is characterized in that described device further include:
Generation module, for generating the mark of the corresponding log information template of a plurality of log information.
14. device according to claim 13, which is characterized in that memory module is specifically used for:
The mark of the second part of a plurality of log information and the log information template is stored in the of the database One space;
The mark of the log information template and the log information template is stored in the second space of the database.
15. a kind of device for storing log information, which is characterized in that including at least one calculate node, each calculate node packet Processor and memory are included, when the device of the storage log information is run, the processor runs the meter in the memory Calculation machine is executed instruction in the method as described in any one of claims 1 to 3 of execution.
16. a kind of device of inquiry log information, which is characterized in that including at least one calculate node, each calculate node packet Processor and memory are included, when the device of the storage log information is run, the processor runs the meter in the memory Calculation machine is executed instruction in the method as described in any one of claim 4 to 7 of execution.
17. a kind of non-volatile computer readable storage medium storing program for executing that computer is non-transient, which is characterized in that including computer journey Sequence, when the computer program is run at least one calculate node, so that at least one calculate node executes such as right It is required that method described in any one of 1 to 3.
18. a kind of non-volatile computer readable storage medium storing program for executing that computer is non-transient, which is characterized in that including computer journey Sequence, when the computer program is run at least one calculate node, so that at least one calculate node executes such as right It is required that method described in any one of 4 to 7.
CN201910106607.0A 2019-02-02 2019-02-02 It stores, the method, apparatus of inquiry log information Pending CN109885545A (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201910106607.0A CN109885545A (en) 2019-02-02 2019-02-02 It stores, the method, apparatus of inquiry log information
PCT/CN2019/107127 WO2020155651A1 (en) 2019-02-02 2019-09-21 Method and device for storing and querying log information

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910106607.0A CN109885545A (en) 2019-02-02 2019-02-02 It stores, the method, apparatus of inquiry log information

Publications (1)

Publication Number Publication Date
CN109885545A true CN109885545A (en) 2019-06-14

Family

ID=66927813

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910106607.0A Pending CN109885545A (en) 2019-02-02 2019-02-02 It stores, the method, apparatus of inquiry log information

Country Status (2)

Country Link
CN (1) CN109885545A (en)
WO (1) WO2020155651A1 (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020155651A1 (en) * 2019-02-02 2020-08-06 华为技术有限公司 Method and device for storing and querying log information
CN112152823A (en) * 2019-06-26 2020-12-29 北京易真学思教育科技有限公司 Website operation error monitoring method and device and computer storage medium
CN113420003A (en) * 2021-06-30 2021-09-21 中国航空油料有限责任公司 Method, device, equipment and medium for processing data interaction log
CN113472555A (en) * 2020-03-30 2021-10-01 华为技术有限公司 Fault detection method, system, device, server and storage medium
CN116894021A (en) * 2023-05-24 2023-10-17 北京优特捷信息技术有限公司 Log data storage method, query method, device, equipment and medium

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116644039B (en) * 2023-05-25 2023-12-19 安徽继远软件有限公司 Automatic acquisition and analysis method for online capacity operation log based on big data

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080072140A1 (en) * 2006-07-05 2008-03-20 Vydiswaran V G V Techniques for inducing high quality structural templates for electronic documents
US20110219046A1 (en) * 2010-03-05 2011-09-08 Igor Nesmyanovich System, method and computer program product for managing data storage and rule-driven communications for a plurality of tenants
CN103544176A (en) * 2012-07-13 2014-01-29 百度在线网络技术(北京)有限公司 Method and device for generating page structure template corresponding to multiple pages
CN104331487A (en) * 2014-11-13 2015-02-04 携程计算机技术(上海)有限公司 Method and device for processing logs
CN108241658A (en) * 2016-12-24 2018-07-03 北京亿阳信通科技有限公司 A kind of logging mode finds method and system
CN108694206A (en) * 2017-04-11 2018-10-23 富士通株式会社 Information processing method and device

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8533193B2 (en) * 2010-11-17 2013-09-10 Hewlett-Packard Development Company, L.P. Managing log entries
US8620928B1 (en) * 2012-07-16 2013-12-31 International Business Machines Corporation Automatically generating a log parser given a sample log
US9166866B2 (en) * 2013-04-30 2015-10-20 Microsoft Technology Licensing, Llc Hydration and dehydration with placeholders
CN109885545A (en) * 2019-02-02 2019-06-14 华为技术有限公司 It stores, the method, apparatus of inquiry log information

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080072140A1 (en) * 2006-07-05 2008-03-20 Vydiswaran V G V Techniques for inducing high quality structural templates for electronic documents
US20110219046A1 (en) * 2010-03-05 2011-09-08 Igor Nesmyanovich System, method and computer program product for managing data storage and rule-driven communications for a plurality of tenants
CN103544176A (en) * 2012-07-13 2014-01-29 百度在线网络技术(北京)有限公司 Method and device for generating page structure template corresponding to multiple pages
CN104331487A (en) * 2014-11-13 2015-02-04 携程计算机技术(上海)有限公司 Method and device for processing logs
CN108241658A (en) * 2016-12-24 2018-07-03 北京亿阳信通科技有限公司 A kind of logging mode finds method and system
CN108694206A (en) * 2017-04-11 2018-10-23 富士通株式会社 Information processing method and device

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020155651A1 (en) * 2019-02-02 2020-08-06 华为技术有限公司 Method and device for storing and querying log information
CN112152823A (en) * 2019-06-26 2020-12-29 北京易真学思教育科技有限公司 Website operation error monitoring method and device and computer storage medium
CN113472555A (en) * 2020-03-30 2021-10-01 华为技术有限公司 Fault detection method, system, device, server and storage medium
CN113472555B (en) * 2020-03-30 2022-09-23 华为技术有限公司 Fault detection method, system, device, server and storage medium
CN113420003A (en) * 2021-06-30 2021-09-21 中国航空油料有限责任公司 Method, device, equipment and medium for processing data interaction log
CN116894021A (en) * 2023-05-24 2023-10-17 北京优特捷信息技术有限公司 Log data storage method, query method, device, equipment and medium

Also Published As

Publication number Publication date
WO2020155651A1 (en) 2020-08-06

Similar Documents

Publication Publication Date Title
CN109885545A (en) It stores, the method, apparatus of inquiry log information
US11860874B2 (en) Multi-partitioning data for combination operations
US11341131B2 (en) Query scheduling based on a query-resource allocation and resource availability
US11599541B2 (en) Determining records generated by a processing task of a query
US11663227B2 (en) Generating a subquery for a distinct data intake and query system
US20200364223A1 (en) Search time estimate in a data intake and query system
US11615104B2 (en) Subquery generation based on a data ingest estimate of an external data system
US10447772B2 (en) Managed function execution for processing data streams in real time
US10798148B2 (en) Configuring nodes of distributed systems
US20210117425A1 (en) Management of distributed computing framework components in a data fabric service system
US20190095493A1 (en) Multi-partition operation in combination operations
EP2182448A1 (en) Federated configuration data management
CN107111527A (en) Data Stream Processing language for analytical instrument software
US11829381B2 (en) Data source metric visualizations
WO2020087082A1 (en) Trace and span sampling and analysis for instrumented software
CN108573029B (en) Method, device and storage medium for acquiring network access relation data
US11604789B1 (en) Bi-directional query updates in a user interface
JP2013140510A (en) Data processing method, decentralized processing system, and program
US11681707B1 (en) Analytics query response transmission
US11663172B2 (en) Cascading payload replication
US20180176095A1 (en) Data analytics rendering for triage efficiency
US12130829B2 (en) Generation of modified queries using a field value for different fields
CN117971606A (en) Log management system and method based on elastic search
US11899670B1 (en) Generation of queries for execution at a separate system
US10324818B2 (en) Data analytics correlation for heterogeneous monitoring systems

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20190614

RJ01 Rejection of invention patent application after publication