CN109408479B - Log data adding method, system, computer device and storage medium - Google Patents
Log data adding method, system, computer device and storage medium Download PDFInfo
- Publication number
- CN109408479B CN109408479B CN201811093596.9A CN201811093596A CN109408479B CN 109408479 B CN109408479 B CN 109408479B CN 201811093596 A CN201811093596 A CN 201811093596A CN 109408479 B CN109408479 B CN 109408479B
- Authority
- CN
- China
- Prior art keywords
- information
- log
- external data
- lookup table
- data source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The present invention relates to the field of data processing technologies, and in particular, to a log data adding method, a log data adding system, a computer device, and a storage medium. The method comprises the following steps: performing field analysis on the original log to extract multiple effective field information; obtaining a lookup table converter, wherein the lookup table converter is a field mapping information table containing mapping relations between each item of effective field information and external data sources; and sequentially matching the effective field information with the lookup table converter, and adding the data information in the external data source mapped by the effective field information into the original log when the lookup table converter contains the effective field information. The method enriches the data in the original log by adding the mapping information into the original log from the external data source through the lookup table converter, and provides valuable log data for log event analysis.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a log data adding method, a log data adding system, a computer device, and a storage medium.
Background
The current log management platform stores the collected original log directly or carries out simple preprocessing on the log information in a mode of regular analysis and separator, but related information cannot be added to the original log from an external data source. If the occurrence of a log event depends on external data source information, such a log event may appear to have some fly over when the cause is analyzed. If it is desired to sort logs by entries of the subsystems, it may be difficult to sort the subsystems by log, since the original log typically contains only the user names of the distinguishable login servers, although the user names may be mapped to different subsystems, storing only the original log.
Disclosure of Invention
In view of this, it is necessary to provide a log data adding method, system, computer device, and storage medium for the problem that when the occurrence of a log event depends on external data source information, it is impossible to add related information from an external data source to an original log.
A log data adding method, comprising:
performing field analysis on an original log stored in a distributed log processing platform, and extracting multiple effective field information;
obtaining a lookup table converter, wherein the lookup table converter is a field mapping information table containing mapping relations between each item of effective field information and external data sources, and the external data sources are stored in a database of the distributed log processing platform;
and sequentially matching the effective field information with the lookup table converter, adding the data information in the external data source mapped by the effective field information to the original log when the lookup table converter contains the effective field information, and storing the original log added with the data information in a database of the distributed log processing platform.
In one embodiment, the field parsing of the original log stored in the distributed log processing platform includes:
when the original log is of a structured log type, carrying out field analysis on the original log by adopting a regular expression, a separator mode or a json extractor;
and when the original log is of an unstructured log type, carrying out field analysis on the original log by adopting a hook expression.
In one embodiment, the valid field information includes at least one of an IP address, hostname information, a user name of a login server, an employee number, API data interface information, URL address information, or a return status code;
the external data source is a plurality of search information tables in csv format, and comprises at least one search information table of an IP geographic position information table, an IP corresponding company information table, an asset information table, an employee information table, interface user information, website information or a state code lookup table;
the IP address in the effective field information has a mapping relation with the IP geographic position information table and the IP corresponding company information table in the external data source, the host name information in the effective field information has a mapping relation with the asset information table in the external data source, the user name and employee number of a login server in the effective field information have a mapping relation with the employee information table in the external data source, the API data interface information in the effective field information has a mapping relation with the interface user information in the external data source, the URL website information in the effective field information has a mapping relation with the website information in the external data source, and the return status code in the effective field information has a mapping relation with the status code query table in the external data source.
In one embodiment, before the obtaining the lookup table converter, the method further includes establishing a mapping relationship between the lookup table converter and the external data source:
acquiring configuration information through an input field on a configuration interface, wherein the configuration interface is preset on the distributed log processing platform, the input field is preset on the configuration interface, the configuration information comprises effective field information, lookup table ID information and field information to be added, and the acquired configuration information is stored in the field mapping information table;
and acquiring an external data source through an uploading interface on the configuration interface and storing the external data source in the database, wherein the uploading interface is preset on the configuration interface, a storage path of the external data source is stored in the lookup table storage path information of the field mapping information table, and the storage path corresponds to the effective field information, so that a mapping relation between the lookup table converter and the external data source is established.
In one embodiment, the mapping relationship between the pair of the lookup table converter and the external data source further includes:
after establishing a mapping relation between the lookup table converter and the external data source, releasing the mapping relation to other servers in the form of automatic events, wherein the other servers are servers with processing mechanisms of the events registered in the distributed log processing platform;
and acquiring the external data sources synchronously updated by the other servers, and updating the updated external data sources to the corresponding databases.
In one embodiment, before the matching the valid field information with the lookup table converter in sequence, the method includes:
sequentially reading the lookup table storage paths in the lookup table converter, and reading all external data sources stored in the database through the lookup table storage paths;
and loading all the external data sources into a memory for caching.
In one embodiment, the matching the valid field information with the lookup table converter sequentially includes:
sequentially matching the extracted effective field information with the lookup table converter, and acquiring lookup table ID information and field information to be added corresponding to the effective field information when the lookup table converter contains the effective field information;
searching an external data source which is the same as the ID information of the lookup table in a memory, reading field information to be added in the external data source, adding the field information to be added into the original log, and storing the original log in a database.
A log data adding system comprising:
the analysis unit is used for carrying out field analysis on the original log stored in the distributed log processing platform and extracting a plurality of effective field information;
the acquisition unit is used for acquiring a lookup table converter, wherein the lookup table converter is a field mapping information table containing mapping relations between each item of effective field information and external data sources, and the external data sources are stored in a database of the distributed log processing platform;
and the matching and adding unit is used for sequentially matching the effective field information with the lookup table converter, adding the data information in the external data source mapped by the effective field information to the original log when the lookup table converter contains the effective field information, and storing the original log added with the data information in a database of the distributed log processing platform.
A computer device comprising a memory and a processor, the memory having stored therein computer readable instructions which, when executed by the processor, cause the processor to perform the steps of the log data adding method described above.
A storage medium storing computer readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the log data adding method described above.
The method, the device, the computer equipment and the storage medium for adding the log data comprise the steps of carrying out field analysis on an original log stored in a distributed log processing platform, and extracting a plurality of effective field information; obtaining a lookup table converter, wherein the lookup table converter is a field mapping information table containing mapping relations between each item of effective field information and external data sources, and the external data sources are stored in a database of the distributed log processing platform; and sequentially matching the effective field information with the lookup table converter, adding the data information in the external data source mapped by the effective field information to the original log when the lookup table converter contains the effective field information, and storing the original log added with the data information in a database of the distributed log processing platform. The method enriches the data in the original log by adding the mapping information into the original log from the external data source through the lookup table converter, and provides valuable log data for log event analysis.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention.
FIG. 1 is a flow chart of a log data addition method in one embodiment of the invention;
FIG. 2 is a flow diagram before a lookup table converter is acquired in one embodiment;
FIG. 3 is a flow chart of step S3 in one embodiment;
fig. 4 is a block diagram of a log data adding system according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
Fig. 1 is a flowchart of a log data adding method according to an embodiment of the present invention, as shown in fig. 1, the log data adding method includes the following steps:
step S1, field analysis: and carrying out field analysis on the original log stored in the distributed log processing platform, and extracting a plurality of effective field information.
The log mainly comprises a system log, an application program log and a safety log, each log records related descriptions such as a time stamp, a host name, a user, operation behaviors and the like, and system operation and development personnel can know software and hardware information of a server through the log, check errors in the configuration process, cause of the errors and the like. The load of the server, performance safety and timely analysis of problems and error root-cause correction errors can be known by frequently analyzing the logs. In the big data age, the number of logs is huge, the variety is diversified, the difficulty of statistics and retrieval of the logs is increased, and the Linux commands such as grep, awk, wc and the like are generally used for realizing the retrieval and the statistics, but the method is unavoidable and unconscious for the requirements of higher inquiry, sequencing, statistics and the like and huge machine quantity. In the past, logs were stored in a decentralized manner on each server, and were not centrally managed, and were difficult to perform and even deleted. Centralized log management is used, and logs on all servers are collected and summarized, namely, the logs on other servers are collected and processed by adopting a distributed log processing platform, so that the logs can be more conveniently and comprehensively analyzed.
In one embodiment, field parsing of an original log stored in a distributed log processing platform includes: when the original log is of a structured log type, carrying out field analysis on the original log by adopting a regular expression, a separator mode or a json extractor; when the original log is of an unstructured log type, field analysis is carried out on the original log by adopting a hook expression.
According to different collected servers, the original log structures generated by the servers are different and generally comprise structured log types and unstructured log types, the original log generally has log file names or log IDs, a log type lookup table can be preset, and the log type lookup table comprises the log file names or the log IDs and the corresponding log types. Before field analysis is carried out on the original log, the log type corresponding to the original log is found in a log type lookup table, and the field analysis is carried out on the original log by adopting a regular expression, a separator mode, a json extractor or a hook expression according to the log type.
According to the embodiment, according to different log types of the original log, different modes are adopted for field analysis, so that more accurate and complete effective field information can be analyzed.
Step S2, obtaining a lookup table converter: the lookup table converter is a field mapping information table containing a mapping relationship between each item of valid field information and an external data source stored in a database of the distributed log processing platform.
In one embodiment, the extracted plurality of valid field information includes at least one of an IP address, hostname information, a user name of a login server, an employee number, API data interface information, URL address information, or a return status code; the external data source is a plurality of search information tables in csv format, and comprises at least one search information table of an IP geographic position information table, an IP corresponding company information table, an asset information table, an employee information table, interface user information, website information or a state code query table; the IP address in the effective field information has a mapping relation with an IP geographic position information table in the external data source and an IP corresponding company information table, the host name information in the effective field information has a mapping relation with an asset information table in the external data source, the user name and employee number of a login server in the effective field information have a mapping relation with an employee information table in the external data source, the API data interface information in the effective field information has a mapping relation with interface user information in the external data source, the URL website information in the effective field information has a mapping relation with website information in the external data source, and the return status code in the effective field information has a mapping relation with a status code lookup table in the external data source. The mapping relationship between the specific valid field information and the external data source is shown in the following table 1:
TABLE 1
The embodiment exemplifies the valid field information which is usually resolved in the original log, and the valid field information needs to be mapped with the mapping relation of the external data source, so that the mapping relation of the lookup table converter can be easily matched with the external data source corresponding to the valid field information.
In one embodiment, as shown in fig. 2, before obtaining the lookup table converter, the method further includes establishing a mapping relationship between the lookup table converter and an external data source:
step S201, acquiring configuration information and storing: acquiring configuration information through an input field on a configuration interface, presetting the configuration interface on a distributed log processing platform, presetting the input field on the configuration interface, wherein the configuration information comprises effective field information, lookup table ID information and field information to be added, and storing the acquired configuration information in a field mapping information table.
The look-up table converter in this step, i.e. the field mapping information table, is shown in table 2:
| valid field information | Lookup table ID information | To be added with field information | The lookup table stores path information |
| XXXX | XXXX | XXXX | XXXX |
TABLE 2
As shown in table 2, the configuration information acquired through the configuration interface is stored in the valid field information, the lookup table ID information, and the field information to be added of the field mapping information table, respectively, and the lookup table storage path is acquired and stored through step S202.
Step S202, obtaining an external data source: the method comprises the steps of obtaining an external data source through an uploading interface on a configuration interface and storing the external data source in a database, presetting the uploading interface on the configuration interface, storing a storage path of the external data source in lookup table storage path information of a field mapping information table, and enabling the storage path to correspond to effective field information so as to establish a mapping relation between a lookup table converter and the external data source.
Because the distributed log processing platform does not store an external data source, when the lookup table converter is configured, the external data source is uploaded and stored into the local distributed log processing platform through an uploading interface on the configuration interface, and after the storage is completed, the storage path is automatically stored in a lookup table storage path corresponding to effective field information in the field mapping information table.
Specifically, the configuration information acquired through the configuration interface acquires an external data source through the uploading interface, stores the external data source in the database, automatically stores the storage path in the storage path information of the lookup table, and obtains a field mapping information table as follows:
valid field information: an IP address;
look-up table ID information: geo_location_src.csv;
field information needs to be added:
src_location_country,src_location_province,src_location_city,src_location_lat,src_location_lon；
the lookup table stores path information: XX/XX/Externaldate.
The effective field information of the IP address needs to be mapped with an external data source with the lookup table ID information of geo-location-src.csv, the external data source is an IP geographic position information table, and according to the IP address, the field information needing to be added is country, province, city, latitude and longitude and is used for distributed denial of service DDOS attack analysis.
In addition, the IP address can also correspond to an external data source of an IP corresponding company information table, and the field information is a company ID and a company name to be added at the moment and used for analyzing the access condition of the user.
When the effective field information is host name information, the effective field information corresponds to an external data source of an asset information table, and the field information is required to be added at the moment to be an IP address.
When the effective field information is the user name or employee number of the login server, the external data source corresponding to one employee information table is needed to be added, and the field information is at least one piece of information such as employee name, company ID, company name or employee contact information and the like and is used for analyzing the user access record.
In the embodiment, configuration information input by a user and an uploaded external data source are acquired by presetting a configuration interface, a field mapping information table is stored, and a mapping relationship between a lookup table converter and the external data source is automatically established by corresponding relationships between effective field information and lookup table ID information in the field mapping information table and between field information to be added and lookup table storage path information.
In one embodiment, as shown in fig. 2, further comprising:
step S203, event release: after the mapping relation between the lookup table converter and the external data source is established, the mapping relation is released to other servers in the form of automatic events, and the other servers are servers with processing mechanisms of the events registered in the distributed log processing platform.
Since the external data source of the present invention is typically maintained and updated on other servers, once the external data source on the other servers is updated, the effort is very great if the external data source is uploaded again manually by the user. In the step, an event for automatic update, such as an updateDatasetEvent event, is preset in a distributed log processing platform, and other servers register a processing mechanism of the event. After the mapping relation between the lookup table converter and the external data source is established, an updatedatatEvent event is automatically triggered, the storage position of the external data source is issued to other corresponding servers through the updatedatatEvent event, namely other servers are informed, the external data source is mapped on the distributed log processing platform, and when the other servers update the external data source, the other servers are triggered to automatically update the updated external data source to the distributed log processing platform.
Step S204, synchronously updating the external data source: and acquiring external data sources synchronously updated by other servers, and updating the updated external data sources to corresponding databases.
The other servers establish a synchronous update processing mechanism with the local distributed log processing platform through the updatedDatasetEvent event, so that after the external data sources of the other servers update the external data sources, the updated external data sources are automatically sent to the distributed log processing platform, and the distributed log processing platform acquires the updated external data sources and updates the updated external data sources to corresponding storage paths.
According to the embodiment, the external data source is automatically updated in a synchronous mode, so that time and labor are saved in an automatic updating mode, and the fact that the external data source stored by the distributed log processing platform is the latest and most reliable data information is guaranteed.
Step S3, matching and adding data information: and sequentially matching the effective field information with the lookup table converter, adding the data information in the external data source mapped by the effective field information to the original log when the lookup table converter contains the effective field information, and storing the original log added with the data information in a database of the distributed log processing platform.
In one embodiment, as shown in fig. 3, before the valid field information is matched with the lookup table converter in sequence, the method includes:
step S301, caching the external data source: sequentially reading the lookup table storage paths in the lookup table converter, and reading all external data sources stored in the database through the lookup table storage paths; all external data sources are loaded into the memory for caching.
When caching the external data source, the following data structure is adopted and caching is carried out:
Map<String,List<Map<String,Object>>>
the data structure of the embodiment is optimized, the key of the Map does not store redundant information, 64 threads can be used for simultaneously processing the original log, all the processing threads share one part of cached data, the memory space is saved, and the function of distributed multithreading is realized.
In one embodiment, as shown in fig. 3, when matching the valid field information with the lookup table converter in turn, it includes:
step S302, matching valid field information: and sequentially matching the extracted effective field information with a lookup table converter, and acquiring the lookup table ID information corresponding to the effective field information and the field information to be added when the lookup table converter contains the effective field information.
The lookup table converter is a field mapping information table, the content of which is shown in table 2, when the effective field information extracted in step S1 is matched with the lookup table converter, the lookup table converter contains the effective field information, which indicates that the effective field information has an external data source with a mapping relationship, and the data information needs to be obtained from the external data source and added into the original log. In order to obtain the data information, the step needs to obtain the lookup table ID information corresponding to the valid field information from the lookup table converter, so as to find the corresponding external data source, and obtain the field information to be added, so as to add the corresponding data information.
Specifically, when the effective field information extracted in step S1 contains an IP address, the lookup table converter contains the effective field information of the IP address, and this step obtains:
look-up table ID information: geo_location_src.csv;
field information needs to be added:
src_location_country,src_location_province,src_location_city,
src_location_lat,src_location_lon；
step S303, adding data: and searching an external data source which is the same as the ID information of the lookup table in the memory, reading field information to be added in the external data source, adding the field information to be added into an original log, and storing the original log in a database.
The external data source contains all data information corresponding to the field information to be added, and after searching the external data source in the memory according to the lookup table ID information obtained in step S302, the field information to be added corresponding to the external data source can be sequentially read, and the field information to be added is added to the original log. In addition, the valid field information and the field information to be added may be added together at the forefront of the original log, or the field information to be added may be added after the valid field information. After the field information to be added is added into the original log, when the original log is stored in a database, an index table is established, and the original log is stored in an index storage mode so as to facilitate the subsequent user to search the original log.
Specifically, the external data source is found according to the ID information of the lookup table obtained in step S302, where the external data source is an IP geographic location information table, and the IP geographic location information table contains information such as country, province, city, latitude, longitude, etc. corresponding to an IP address. After the information is read in the step, the information is added into an original log and stored, and the subsequent analysis of the distributed denial of service DDOS attack can be conveniently performed.
According to the method, the external data source can be shared in the memory when the original log is processed by the multithread in the mode of searching the external data source in the memory, so that memory space is saved, and the efficiency of data addition can be greatly improved by the mode of processing the multithread simultaneously.
In one embodiment, a log data adding system is provided, as shown in fig. 4, including the following units:
the analysis unit is used for carrying out field analysis on the original log stored in the distributed log processing platform and extracting a plurality of effective field information;
the acquisition unit is used for acquiring a lookup table converter, wherein the lookup table converter is a field mapping information table containing mapping relations between each item of effective field information and external data sources, and the external data sources are stored in a database of the distributed log processing platform;
and the matching and adding unit is used for sequentially matching the effective field information with the lookup table converter, adding the data information in the external data source mapped by the effective field information to the original log when the lookup table converter contains the effective field information, and storing the original log added with the data information in a database of the distributed log processing platform.
In one embodiment, a computer device is provided, including a memory and a processor, where the memory stores computer readable instructions that, when executed by the processor, cause the processor to implement the steps in the log data adding method of each embodiment described above when executing the computer readable instructions.
In one embodiment, a storage medium storing computer readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps in the log data adding method of each of the above embodiments is presented.
Those of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be implemented by a program to instruct related hardware, the program may be stored in a computer readable storage medium, and the storage medium may include: read Only Memory (ROM), random access Memory (RAM, random Access Memory), magnetic or optical disk, and the like.
The technical features of the above-described embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above-described embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above-described embodiments represent only some exemplary embodiments of the invention, which are described in more detail and are not to be construed as limiting the scope of the invention. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the invention, which are all within the scope of the invention. Accordingly, the scope of protection of the present invention is to be determined by the appended claims.
Claims (10)
1. A log data adding method, comprising:
performing field analysis on an original log stored in a distributed log processing platform, and extracting multiple effective field information;
obtaining a lookup table converter, wherein the lookup table converter is a field mapping information table containing mapping relations between each item of effective field information and external data sources, and the external data sources are stored in a database of the distributed log processing platform;
and sequentially matching the effective field information with the lookup table converter, adding the data information in the external data source mapped by the effective field information to the original log when the lookup table converter contains the effective field information, and storing the original log added with the data information in a database of the distributed log processing platform.
2. The method for adding log data according to claim 1, wherein the field parsing of the original log stored in the distributed log processing platform comprises:
when the original log is of a structured log type, carrying out field analysis on the original log by adopting a regular expression, a separator mode or a json extractor;
and when the original log is of an unstructured log type, carrying out field analysis on the original log by adopting a hook expression.
3. The log data adding method as claimed in claim 1, wherein the valid field information includes at least one of an IP address, hostname information, a user name of a login server, an employee number, API data interface information, URL address information, or a return status code;
the external data source is a plurality of search information tables in csv format, and comprises at least one search information table of an IP geographic position information table, an IP corresponding company information table, an asset information table, an employee information table, interface user information, website information or a state code lookup table;
the IP address in the effective field information has a mapping relation with the IP geographic position information table and the IP corresponding company information table in the external data source, the host name information in the effective field information has a mapping relation with the asset information table in the external data source, the user name and employee number of a login server in the effective field information have a mapping relation with the employee information table in the external data source, the API data interface information in the effective field information has a mapping relation with the interface user information in the external data source, the URL website information in the effective field information has a mapping relation with the website information in the external data source, and the return status code in the effective field information has a mapping relation with the status code query table in the external data source.
4. The method of claim 1, further comprising, prior to the obtaining the lookup table converter, establishing a mapping relationship between the lookup table converter and the external data source:
acquiring configuration information through an input field on a configuration interface, wherein the configuration interface is preset on the distributed log processing platform, the input field is preset on the configuration interface, the configuration information comprises effective field information, lookup table ID information and field information to be added, and the acquired configuration information is stored in the field mapping information table;
and acquiring an external data source through an uploading interface on the configuration interface and storing the external data source in the database, wherein the uploading interface is preset on the configuration interface, a storage path of the external data source is stored in the lookup table storage path information of the field mapping information table, and the storage path corresponds to the effective field information, so that a mapping relation between the lookup table converter and the external data source is established.
5. The log data adding method as set forth in claim 4, wherein said establishing a mapping relationship between said lookup table converter and said external data source further comprises:
after establishing a mapping relation between the lookup table converter and the external data source, releasing the mapping relation to other servers in the form of automatic events, wherein the other servers are servers with processing mechanisms of the events registered in the distributed log processing platform;
and acquiring the external data sources synchronously updated by the other servers, and updating the updated external data sources to the corresponding databases.
6. The log data adding method as set forth in claim 1, wherein before said sequentially matching said valid field information with said look-up table converter, comprising:
sequentially reading the lookup table storage paths in the lookup table converter, and reading all external data sources stored in the database through the lookup table storage paths;
and loading all the external data sources into a memory for caching.
7. The log data adding method as set forth in claim 6, wherein said sequentially matching said valid field information with said lookup table converter comprises:
sequentially matching the extracted effective field information with the lookup table converter, and acquiring lookup table ID information and field information to be added corresponding to the effective field information when the lookup table converter contains the effective field information;
searching an external data source which is the same as the ID information of the lookup table in a memory, reading field information to be added in the external data source, adding the field information to be added into the original log, and storing the original log in a database.
8. A log data adding system, comprising:
the analysis unit is used for carrying out field analysis on the original log stored in the distributed log processing platform and extracting a plurality of effective field information;
the acquisition unit is used for acquiring a lookup table converter, wherein the lookup table converter is a field mapping information table containing mapping relations between each item of effective field information and external data sources, and the external data sources are stored in a database of the distributed log processing platform;
and the matching and adding unit is used for sequentially matching the effective field information with the lookup table converter, adding the data information in the external data source mapped by the effective field information to the original log when the lookup table converter contains the effective field information, and storing the original log added with the data information in a database of the distributed log processing platform.
9. A computer device comprising a memory and a processor, the memory having stored therein computer readable instructions which, when executed by the processor, cause the processor to perform the steps of the log data adding method of any of claims 1 to 7.
10. A storage medium storing computer readable instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of the log data adding method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811093596.9A CN109408479B (en) | 2018-09-19 | 2018-09-19 | Log data adding method, system, computer device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811093596.9A CN109408479B (en) | 2018-09-19 | 2018-09-19 | Log data adding method, system, computer device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109408479A CN109408479A (en) | 2019-03-01 |
| CN109408479B true CN109408479B (en) | 2023-05-30 |
Family
ID=65465128
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811093596.9A Active CN109408479B (en) | 2018-09-19 | 2018-09-19 | Log data adding method, system, computer device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109408479B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110209562A (en) * | 2019-05-13 | 2019-09-06 | 深圳前海微众银行股份有限公司 | A kind of log analysis method and Analysis server |
| CN112347165B (en) * | 2019-08-08 | 2023-11-03 | 腾讯科技（深圳）有限公司 | Log processing method and device, server and computer readable storage medium |
| CN111881094B (en) * | 2020-07-28 | 2023-07-18 | 平安科技（深圳）有限公司 | Method, device, terminal and storage medium for extracting key information in log |
| CN112231325A (en) * | 2020-08-31 | 2021-01-15 | 中国电力科学研究院有限公司 | Data mapping method and system for offline and online equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101931562A (en) * | 2010-09-29 | 2010-12-29 | 杭州华三通信技术有限公司 | Web log processing method and device |
| CN104572689A (en) * | 2013-10-17 | 2015-04-29 | 腾讯科技（深圳）有限公司 | Data synchronizing method, device and system |
| CN106021554A (en) * | 2016-05-30 | 2016-10-12 | 北京奇艺世纪科技有限公司 | Log analysis method and device |
| CN106385331A (en) * | 2016-09-08 | 2017-02-08 | 努比亚技术有限公司 | Method and system for monitoring alarm based on log |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7653633B2 (en) * | 2005-11-12 | 2010-01-26 | Logrhythm, Inc. | Log collection, structuring and processing |
-
2018
- 2018-09-19 CN CN201811093596.9A patent/CN109408479B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101931562A (en) * | 2010-09-29 | 2010-12-29 | 杭州华三通信技术有限公司 | Web log processing method and device |
| CN104572689A (en) * | 2013-10-17 | 2015-04-29 | 腾讯科技（深圳）有限公司 | Data synchronizing method, device and system |
| CN106021554A (en) * | 2016-05-30 | 2016-10-12 | 北京奇艺世纪科技有限公司 | Log analysis method and device |
| CN106385331A (en) * | 2016-09-08 | 2017-02-08 | 努比亚技术有限公司 | Method and system for monitoring alarm based on log |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109408479A (en) | 2019-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109408479B (en) | Log data adding method, system, computer device and storage medium | |
| CN111459985B (en) | Identification information processing method and device | |
| CN110099059B (en) | Domain name identification method and device and storage medium | |
| CN109726202B (en) | Block chain data storage method and computer storage medium | |
| US20200042510A1 (en) | Method and device for correlating multiple tables in a database environment | |
| CN106951557B (en) | Log association method and device and computer system applying log association method and device | |
| CN103795811B (en) | Information storage and data statistical management method based on meta data storage | |
| US10885036B2 (en) | Obtaining incremental updates from a database using a partial query | |
| US10776345B2 (en) | Efficiently updating a secondary index associated with a log-structured merge-tree database | |
| EP3964976A1 (en) | Cloud inference system | |
| US11151112B2 (en) | Correlating multiple tables in a non-relational database environment | |
| CN111740868B (en) | Alarm data processing method and device and storage medium | |
| CN111859132A (en) | Data processing method and device, intelligent equipment and storage medium | |
| JP2013191211A (en) | Method and system for storing and acquiring data | |
| US8396877B2 (en) | Method and apparatus for generating a fused view of one or more people | |
| JP2009217426A (en) | Information processor, resource identification program, and resource identification method | |
| JP5436309B2 (en) | Data processing apparatus, data processing method, and program | |
| US9529855B2 (en) | Systems and methods for point of interest data ingestion | |
| CN110008243B (en) | Data table processing method and device | |
| CN116820874A (en) | Enterprise-level big data component and method for monitoring and alarming application | |
| CN116701337B (en) | Log data processing method and device, electronic equipment and storage medium | |
| US8433699B1 (en) | Object identity and addressability | |
| CN111274350B (en) | Data processing method, device, computer equipment and storage medium | |
| CN114153830B (en) | Data verification method and device, computer storage medium and electronic equipment | |
| CN111723146B (en) | Method for monitoring database, management system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |