CN111984505B

CN111984505B - Fortune dimension data acquisition device and acquisition method

Info

Publication number: CN111984505B
Application number: CN202010849458.XA
Authority: CN
Inventors: 王龙; 汪兆伟
Original assignee: Haoyue Technology Co ltd
Current assignee: Haoyue Technology Co ltd
Priority date: 2020-08-21
Filing date: 2020-08-21
Publication date: 2023-06-16
Anticipated expiration: 2040-08-21
Also published as: CN111984505A

Abstract

The invention discloses a device and a method for collecting operation and maintenance data. The data acquisition engine and the data acquisition method comprise data acquisition, arrangement, calculation and output, wherein the output comprises calculation of a ranking list, provision of real-time data of objects to middleware, writing of a database and transmission of the database to a data analysis platform, so that operation and maintenance service of a large-scale complex environment is realized.

Description

Fortune dimension data acquisition device and acquisition method

Technical Field

The invention relates to a data collection device and a data collection method for operation and maintenance.

Background

As the machine room is larger and larger in scale, the service system carried by the machine room is more and more complex, the operation and maintenance of the machine room is more and more complex, the required supervision content is not limited to the operation condition of network equipment, the operation condition of a server is required to be supervised, and various operation indexes including application programs on the server, physical environment and dynamic environment indexes of the machine room are required to be supervised, and even the operation data of some application systems operated in the machine room are required to be supervised.

The existing operation and maintenance mode can not meet the complex operation and maintenance requirements of a large-scale machine room, for example, icinga is an open source monitoring system used for monitoring the operation conditions of a network host and service, the operation and maintenance mode can monitor the operation indexes of the host and the service under the condition of setting a threshold value, and after the operation indexes of the monitored objects exceed the threshold value range, alarm information is given. In addition, icinga also has a web page of the Icinga, and can check the running index condition of the configured monitored equipment. Similar open source software is also Ganglia, zabbix, NMS, and the like.

Existing Icinga has the defects that:

1. the calculation of the monitoring data mainly depends on script realization, index comparison calculation is carried out through simple threshold configuration, and the threshold configuration comparison is unified and indifferently treated;

2. the configuration of the acquisition script is troublesome;

3. monitoring objects or acquisition script configuration changes, restarting service is needed, monitoring is interrupted, and an environment requiring uninterrupted monitoring for 7x24 hours cannot be met;

4. the monitoring mechanism is simple to realize, is difficult to deal with the monitoring of the machine room with more than 1000 monitoring points, the problem of displaying the running condition of the large-scale machine room needs a more complex mechanism, and the large-scale machine room needs intellectualization for alarm processing, so that the problem of alarm storm is avoided;

5. complex machine room monitoring requires a more complex data processing system, one assumption is that a data processing system is additionally built, and then data collected by Icinga is processed, and whether monitored data of Icinga is relatively difficult to dock to other systems or is difficult to dock to other systems, a restul form API is required, the restul is based on HTTP protocol, performance of the restul is bottleneck, and in addition, the data format and content of Icinga spitting are not necessarily very suitable for the additional data processing system.

Therefore, how to reduce the workload of the data analysis platform is of great significance to the operation and maintenance of the machine room at present.

Disclosure of Invention

The invention aims at providing a running data acquisition device which is mainly responsible for scheduling and executing acquisition tasks and collecting and processing acquisition results.

To achieve the object of the present invention, there is provided a motion-dimension data acquisition device including:

the data center unit is used for establishing and maintaining a hash table of the monitoring object;

the acquisition task unit is used for assembling an acquisition task hash table according to the established monitoring object hash table and the acquisition task configuration file, scheduling a monitoring object script according to the assembled acquisition task hash table and acquiring output content of the monitoring object script, wherein the output content is used as an acquisition result to be assembled into an acquisition result data queue;

the data processing unit is used for processing the data in the data queue of the acquisition result, and the processed data is assembled into a target data queue and can be used for updating a monitored object hash table established by the data center unit;

the working contents or the working states of the data center unit, the acquisition task unit and the data processing unit are adjusted through configuration files.

The data acquisition engine provided by the invention belongs to data acquisition in a complex operation and maintenance scheme, and adopts the configuration file to configure the working states or the working states of the data center unit, the acquisition task unit and the data processing unit, so that the data acquisition task, the acquisition frequency and the acquisition load can be flexibly configured, and the acquired data can be processed. After the data processing unit processes the collected data, the hash table of the monitoring object is updated through the processed data, so that the problem that in the prior art, when the data of the monitoring object is changed, service is required to be restarted, and monitoring is interrupted is solved.

Further, the acquisition engine provided by the invention further comprises a data output unit, wherein the data output unit is used for reading the monitored object data in the monitored object hash table and outputting the read monitored object data to the outside of the engine, and the data output unit is used for processing in one or more modes of:

1) Calculating target indexes of the monitoring object data into top ranking list data according to the requirements of the configuration file, and writing the top ranking list data into middleware for calling and displaying by a page display module;

2) And sending the target index data of the read monitoring object data to a data analysis platform according to the configuration file requirement.

The data output unit is used for realizing the calculation of the ranking list of each target index of the monitored object data; and the target index data of the monitored object data can be sent to the data analysis platform, so that the data analysis platform only needs to define a required data format, does not need to process a collection process and a differentiated script data format, and realizes a data providing task.

The data output unit is used for carrying out top ranking list calculation and data transmission according to the requirements of the configuration file, and can flexibly configure the allowable storage time and the data required to be stored through the configuration file, so that the key index data in the monitored object data can be stored Xu Hua, and various target index ranking list data can be flexibly calculated in real time; and the data transmission is effectively managed through the realization of the configuration file, so that the accuracy of the data transmission is ensured.

Further, the acquisition task unit includes:

the task set subunit traverses the monitoring object hash table, and matches all monitoring objects in the monitoring object hash table one by using the acquisition task configuration file, and the acquisition task hash table is assembled after the acquisition task configuration file is matched;

The task distributor reads task information in the acquisition task hash table and distributes acquisition tasks to different task acquisition thread pools according to task attributes;

and the task acquisition thread pool is used for calling the monitoring object script according to the allocation of the task allocator, collecting the output content returned by the monitoring object script and writing the output content into an acquisition result data queue.

Further, the task collection thread pool comprises a rapid task collection thread pool for processing rapid tasks, a general task collection thread pool for processing general tasks and a failure task collection thread pool for processing failure tasks.

Further, the data formats in the monitored object hash table, the acquisition result data queue and the target data queue are JSON formats. The JSON format is easy for people to read and write, is easy for machines to analyze and generate, and effectively improves network transmission efficiency.

Another object of the present invention is to provide a method for acquiring operation data, comprising the steps of:

step S01: establishing a monitoring object hash table according to monitoring object data, wherein the monitoring object data comprises basic information, script parameters and real-time index data of a monitoring object;

Step S02: data acquisition, namely assembling an acquisition task hash table according to the monitoring object hash table and the acquisition task configuration file established in the step S01, scheduling a monitoring object script according to the assembled acquisition task hash table, and acquiring output content of the monitoring object script, wherein the output content is used as an acquisition result to establish an acquisition result data queue;

step S03: and (3) data processing, namely continuously extracting the acquisition result from the acquisition result data queue established in the step (S02) to process, wherein the processed data is also used for establishing a target data queue and can be used for updating a monitoring object hash table established by the data center unit.

Further, the data acquisition method provided by the invention further comprises a data output step, wherein the step of reading the monitored object data in the monitored object hash table and outputting the read monitored object data to the outside of the engine comprises one or more of the following modes:

1) Storing Xu Hua, namely periodically reading the monitoring object data to be stored in the monitoring object hash table according to the requirement of the storage configuration file, and writing the monitoring object data into a time sequence database;

2) Calculating target indexes of the monitoring object data into top ranking list data according to the requirements of the configuration file, and writing the top ranking list data into middleware for calling and displaying by a page display module;

3) And sending the target index data of the read monitoring object data to a data analysis platform according to the configuration file requirement.

Further, the step S02 of data acquisition specifically includes the following steps:

step S021: reading an acquisition task configuration file;

step S022: traversing the monitoring object hash table established in the step S01, and matching all monitoring objects in the monitoring object hash table with the acquisition task configuration file read in the step S021 one by one to assemble an acquisition task hash table;

step S023: extracting the acquisition task of which the time arrives, distributing the task according to the acquisition task attribute, calling the monitoring object script, and collecting the output content returned by the monitoring object script.

Further, the collection task configuration file read in the step S021 is used for setting one or more collection tasks, and the elements of each collection task include a task name, a collection monitoring object type, a command template of a monitoring object script to be called, a parameter name required by the collection monitoring object script, an interval time of each collection of each monitoring object, a collection task attribute and a processing configuration file template for processing the collection result; and the specific parameter values in the parameter names required by the acquisition monitoring object script are acquired from the object hash table established in the step S01.

Further, the collection task attributes include a general task attribute, a quick task attribute and a failure task attribute, and each collection task can be assigned a quick task attribute or a general task attribute when initially configured.

Further, before executing step S023, the following processing is further performed before calling the monitoring object script according to the acquisition task attribute and collecting the output content returned by the monitoring object script:

step S0231: judging whether the acquisition task with the extraction time reaching is executing or not, if not, executing the step S0232; otherwise, executing step S0235;

step S0232: judging whether the last execution of the acquisition task with the arrival of the extraction time fails, if so, inserting the acquisition task with the arrival of the extraction time into a failed task queue; otherwise, executing step S0233;

step S0233: judging whether the task configuration of the acquisition task reached by the extraction time belongs to a quick task, if so, inserting the acquisition task reached by the extraction time into a quick task queue; otherwise, executing step S0234;

step S0234: judging whether the monitored object in the acquisition task with the reaching extraction time has a general task which is being executed, if so, not executing the acquisition task with the reaching extraction time, waiting for the next extraction, otherwise, inserting the acquisition task with the reaching extraction time into a general task queue;

Step S0235: judging whether the last execution of the acquisition task with the extraction time reached is overtime, if yes, killing the task being executed, and waiting for the next extraction time; if not, the next extraction is waited.

Further, the assembling process of the acquisition task hash table in step S02 further includes a determination of whether the acquisition task configuration file and the monitoring object data are updated, if there is update, updating and reading the acquisition task configuration, if there is no update, determining whether to end the acquisition task, if yes, ending, if not, continuing to determine whether to update the acquisition task configuration and the monitoring object data, and sequentially cycling.

Further, the step S03 processes the data in the collected result data queue by the following steps:

step SA031: extracting data to be processed from the data queue of the acquisition result;

step SA032: and respectively searching a source template, a processing template and a target template required by the data processing according to the object type, the acquisition task type and the target template name of the extracted data to be processed, then calculating, and inserting the calculation result into a target data queue.

Further, the source template, the processing template and the target template are searched through a source data module configuration file, a processing template configuration file and a target data template configuration file respectively.

The beneficial effects of the invention include:

1) The data acquisition engine and the data acquisition method comprise data acquisition, arrangement, calculation and output, wherein the output comprises calculation of a ranking list, provision of real-time data of objects to middleware, writing of a database and transmission of the database to a data analysis platform, so that operation and maintenance service of a large-scale complex environment is realized.

2) The multi-thread pool mode is adopted, so that the performance and monitoring requirements of the acquisition host are balanced.

3) The quick task attribute or the general task attribute is endowed when each acquisition task is initially configured, so that the maximum performance of acquisition is exerted, the condition of overlarge burden of monitored equipment is avoided, and the problem of execution conflict of the monitoring script is solved.

4) The configuration file is adopted to configure and update without restarting the service, and the configuration of the acquisition and monitoring objects is simple.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention. It is evident that the drawings in the following description are only some embodiments of the present invention and that other drawings may be obtained from these drawings without inventive effort for a person of ordinary skill in the art. In the drawings:

FIG. 1 is a diagram of a data acquisition engine according to the present invention;

FIG. 2 is a flow chart of a data acquisition process provided by the present invention;

FIG. 3 is a flow chart of a data processing process provided by the present invention;

in fig. 1-3, the solid lines represent flow paths, the dashed lines represent data operations,

in the drawing, hashMap represents a hash table, for example, a monitored object HashMap represents a monitored object hash table, and an acquisition task HashMap represents an acquisition task hash table; the Queue represents a Queue, for example, the acquisition result data Queue represents an acquisition result data Queue, and the data output task Queue represents a data output task Queue; influxdb refers to a time series database; redis is a middleware product.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art.

The invention provides a fortune dimension data acquisition device which comprises a data center unit, an acquisition task unit, a data processing unit and a data output unit, wherein the functions of the four units are as follows:

The data center unit is used for establishing and maintaining a monitoring object hash table, wherein the monitoring object hash table comprises basic information, script parameters and real-time index data of a monitoring object; the basic information and script parameters of the monitoring object can be from a mysql database, the format of the real-time index data is defined by a template configuration file, and the content of the real-time index data is filled and updated after the data processing unit processes the acquired data.

The acquisition task unit is in charge of executing an acquisition task, scheduling a monitoring script and acquiring output content of the monitoring script; specifically, a monitoring object script is scheduled according to a monitoring object hash table, the output content of the monitoring object script is obtained, and the output content is used as a collection result to construct a collection result data queue.

The data processing unit is used for carrying out integration processing on the data in the data queue of the acquisition result, and the data after the integration processing is assembled into a target data queue and can be used for updating the hash table of the monitoring object established by the data center unit.

A data output unit, configured to output the data in the target data queue to the outside of the engine, where the data output unit is configured to process one or more of the following ways:

1) Calculating target indexes of the monitoring objects stored in the data output data queue into top ranking list data according to configuration requirements, writing the top ranking list data into middleware, and calling and displaying the top ranking list data by a page display module, for example, a real-time refreshed server network card outflow speed ranking list;

2) According to the configuration file requirement, real-time index data of the monitoring object in the data output task queue is sent to a data analysis platform for more complex service analysis, alarm processing and the like; this may be sent to the data analysis platform via a tcp long connection or otherwise.

The data center unit in the text periodically writes the monitoring object data to be stored in the monitoring object hash table into the time sequence database under the configuration of the configuration file.

The acquisition engine provided by the invention can realize data acquisition and processing of the monitoring object and updating of the data of the monitoring object through the data center unit, the acquisition task unit and the data processing unit, and the data output unit is arranged according to the needs and is used for reading the data of the monitoring object and outputting the data to the outside of the engine.

The working contents or working states of the data center unit, the acquisition task unit, the data processing unit and the data output unit are adjusted by the configuration files, and the method specifically comprises the following steps:

The main working content of the data center unit is to create and maintain all monitored objects and real-time index data of the monitored objects (the data description format is JSON format), and store the real-time index data of the monitored objects to a time sequence database at regular intervals; first, when the data center unit creates the monitored object, the type of the object (such as a network device or a server), the data format of the object (JSON hierarchy), and the data content of the object (such as cpu usage, memory usage, disk capacity, etc. of the server) are all defined according to a configuration file, where the configuration file is a data object template mentioned herein. Secondly, the behavior of the data center unit when writing the monitoring object index data into the time series database is also constrained by the configuration file, for example, the network device needs to write cpu occupancy rate, port uplink traffic, port downlink traffic, and the like, and needs to write once for 1 minute, for example, the http application needs to write connection time, response time, download speed, and the like, and needs to write once for 30 seconds. The configuration file will constrain what devices need to write what metrics to the timing database every interval.

The collecting task unit mainly works for collecting data, and comprises the steps of firstly assembling the collecting task, then executing the collecting action according to the attribute of the task, wherein the process of assembling the collecting task is completed according to a collecting task configuration file, and the object type (such as a MySql database or a Linux server) to which the task belongs, a collecting program or script required to be called by the collecting task, parameters required by the collecting program or script, the calling interval time of the collecting task and whether the collecting task is a fast task or a slow task are defined in the task configuration file. After the acquisition task unit is assembled with the acquisition task, the acquisition task unit starts to execute acquisition, calls an acquisition script or program, then collects acquisition result data returned by the acquisition script or the acquisition program, and inserts the acquisition result data into an acquisition result queue.

The data processing unit processes the data acquired by the acquisition task unit, updates the processed data to the monitoring object of the data center unit, and defines and constrains the processing process by the configuration file. The data processing process mainly comprises three steps, namely 1, taking an index to be processed from an acquisition result (the acquisition result is also a data set in a JSON format), 2, calculating or copying the taken index, and 3, filling the index into the processing result (the processing result is also the data set in the JSON format). In the three steps, the index is fetched in the first step, and the index is fetched from which acquisition task of which equipment type to which node of the acquisition result is constrained in the configuration file; the second step of processing, defining what kind of calculation (speed, difference, summation and ratio) is needed for the index in the configuration file; third, the configuration file defines where this processed data fills the result set.

The collection engine provided herein provides data primarily for a data analysis platform in a complex operation and maintenance system, although other data collection may be used.

Herein, the acquisition task unit includes:

the task acquisition thread pool is used for calling the monitoring object script according to the allocation of the task allocator, collecting the output content returned by the monitoring object script and writing the output content into an acquisition result data queue; the task collection thread pool comprises a rapid task collection thread pool for processing rapid tasks, a general task collection thread pool for processing general tasks and a failure task collection thread pool for processing failure tasks.

The acquisition task file is configured to set one or more acquisition tasks, and the elements of each acquisition task comprise a task name, an acquisition monitoring object type, a command template of a monitoring object script to be called, a parameter name required by the acquisition monitoring object script, the interval time of each acquisition of each monitoring object, an acquisition task attribute and a processing configuration file template for processing the acquisition result; and acquiring the specific parameter values in the parameter names required by the acquisition of the monitoring object script from the monitoring object hash table.

Referring to fig. 1, after the acquisition engine provided herein is started, 4 threads are created to respectively complete functions of a data center unit, an acquisition task unit, a data processing unit and a data output unit, and a specific acquisition process is as follows:

the method comprises the steps of establishing a monitoring object hash table, starting a data center thread, loading the basic information and script parameters of a monitoring object obtained from a MySql database by a loading object module, loading the monitoring object data, and comparing the data with a memory monitoring object data set to obtain the monitoring object hash table; the process of reading the monitoring object data from the MySql database and comparing the data is as follows: after the monitored object is read from MySql, if the monitored object already exists in the monitored object hash table, it is required to compare whether the basic attribute (such as account number, password) of the monitored object has been modified, because the object addition, modification and deletion is completed on the web page, if the object is modified by the page, it is required to reestablish the memory monitored object, and reassemble the acquisition task of the monitored object.

Judging whether the monitoring object hash table is updated after the monitoring object hash table is built, if so, updating the basic information and the data structure of the monitoring object by using the data processed by the data processing unit; if not, not updating; and judging whether the time allowed time is reached, if so, calculating the time allowed data, writing the time sequence database, and if not, waiting and judging whether the whole acquisition program is exited, and if so, ending.

After the monitoring object hash table is established, the acquisition task thread loads the acquisition task configuration, traverses the monitoring object hash table, reads the monitoring object data, and matches all monitoring objects in the monitoring object hash table one by using the acquisition task configuration file, and the acquisition task hash table is assembled after the acquisition task configuration file is matched.

When the extraction time of the acquired task arrives, the acquired task is distributed to different task acquisition thread pools through a task distributor according to the acquisition task attribute, the task acquisition thread pools call the monitoring object script, collect the output content returned by the monitoring object script, and write the output content into an acquisition result data queue.

And loading processing configuration after the data processing thread is started, reading queue data from the acquired result data queue as data to be processed, and performing processing calculation such as difference value, speed, peak value and the like on the current value and the last frequency value of the data to be processed to obtain target data, wherein the target data is used for constructing a target data queue and updating monitoring object data in a monitoring object hash table.

After the data output thread is started, loading output configuration, adding a data output task according to the configuration, and reading data from a target data queue according to the data output task to build a data output task queue; the data output unit reads data from the data output task queue according to the data output task and processes the data in one or more modes as follows:

1) And Xu Hua, periodically extracting indexes to be stored in the data output task queue according to the storage configuration, and writing the indexes into a time sequence database. The time Xu Hua is to store data periodically in time series.

2) Calculating target indexes of the monitoring objects stored in the data output data queue into top ranking list data according to configuration requirements, writing the top ranking list data into middleware, and calling and displaying the top ranking list data by a page display module, for example, a real-time refreshed server network card outflow speed ranking list;

top leaderboard data: is a data set, and the central idea is to express the ranking condition of a certain index of a certain class of monitoring objects, such as the first 10 machines occupied by a server cpu in the current monitoring range.

The implementation process of the Top ranking list comprises the following steps: firstly, adding a ranking list setting in a configuration file, configuring the name of the ranking list, the type of a monitoring object, a field to be displayed, a field used for calculating ranking and a ranking list height (the height represents that the collection has a plurality of objects, for example, 100 servers, and only the first 10 machines need to be calculated and displayed; then, after the program collects the corresponding indexes of the corresponding objects, the ranking list is filled, and the objects and the object index data of the ranking list are continuously refreshed according to the real-time data condition. Multiple and various leaderboards can be set in the configuration file according to the monitoring service requirement.

3) According to the configuration file requirement, real-time index data of the monitoring object in the data output task queue is sent to a data analysis platform for more complex service analysis, alarm processing and the like; this may be sent to the data analysis platform via a tcp long connection or otherwise.

The collection task configuration file is used for setting one or more collection tasks, and the elements of each collection task comprise a task name, a collection monitoring object type, a command template of a monitoring object script to be called, a parameter name required by the collection monitoring object script, the interval time of each collection of each monitoring object, collection task attributes and a processing configuration file template used for processing the collection result; the specific parameter values in the parameter names required by the acquisition of the monitoring object script are acquired from the object hash table established in the step S01. The acquisition task attribute can be set as a general task attribute, a quick task attribute and a failure task attribute according to task conditions, and each acquisition task can be endowed with the quick task attribute or the general task attribute when being initially configured, for example, an acquisition task which needs to be quickly checked, can be quickly responded by a monitored object, has small requirements on the performance of an acquisition engine and the monitored object, can be endowed with a quick attribute, and other acquisition tasks which are relatively slow in interaction and do not need to be frequently acquired can be endowed with a general attribute. If a certain acquisition task is overtime or fails in the data acquisition process, the acquisition task is temporarily endowed with a 'failure' attribute until the acquisition task is successfully acquired, and the original attribute is recovered.

Each acquisition task can define acquisition interval time according to the condition of a host and the condition of a site, the acquisition interval time of a quick task is generally shorter, and the interval time of a general task can be longer. The collection actions are completed by a task collection thread pool, which comprises a rapid task collection thread pool for executing rapid tasks, a general task collection thread pool for executing general tasks and a failure task collection thread pool for executing failure tasks. Each thread pool may be sized according to task conditions and host conditions to balance acquisition host performance and monitoring requirements.

The attribute configuration is performed on each acquisition task at the beginning, but the acquisition task is configured into a fast task, and the acquisition task can be arranged into a slow task queue when actually executed, the reference factor is whether the last execution of the next task fails, and if the last execution fails, even if the attribute in the task configuration file is the fast task, the execution is arranged into the slow task queue. As shown in fig. 2, the specific steps are as follows:

Before the general task and the failed task are executed, judging whether the acquisition task with the extraction time reaching the acquisition time has the general task or the failed task which is being executed, if so, waiting to be acquired next time, wherein the general task can be designed to be more complex, occupies more resources of an acquisition host and a monitored machine, does not allow the same object to face a plurality of complex tasks, and in some cases, a plurality of general tasks are possibly mutually conflicted; most of failed tasks are failed due to downtime or disconnection of the monitored machine, so that the situation that a plurality of tasks run together to occupy thread resources, cpu resources and the like of the acquisition host is avoided.

After the acquisition task attribute is determined, an acquisition command is executed through the fast task acquisition thread pool, the general task acquisition thread pool and the failure task acquisition thread pool, a return result is obtained, the return result is inserted into an acquisition result data queue, and the result in the acquisition result data queue can be used for updating the acquisition task state in the acquisition task hash table.

And in the assembling process of the acquisition task hash table, the acquisition task configuration and the judgment of whether the monitoring object data are updated are further included, if the updating exists, the acquisition task configuration is updated and read, if the updating does not exist, whether the acquisition task is ended is judged, if the updating does not exist, the acquisition task is ended, if the updating does not exist, the acquisition task configuration and the monitoring object data are continuously judged, and the circulation is sequentially carried out.

Referring to fig. 3, the basic process of data processing herein is: and extracting data to be processed, respectively searching a source template, a processing template and a target template required by the data processing according to the information such as the object type, the acquisition task type and the target template name of the data to be processed, then calculating, and inserting a calculation result into a target data queue.

The source data format, the calculation basis and the target data format are specified through the configuration file, so that the purposes of flexible assembly and flexible output after acquisition are achieved.

The data processed by the data processing unit can update the real-time index data of the monitored object data in the monitored object hash table in a mapping mode, the real-time index data of one monitored object can be collected and refreshed by a plurality of scripts in different time, each of the plurality of scripts has a data format, the real-time data expressing one monitored object is a unified data format defined for analysis and display by an engine, and various sub-objects, sub-combinations and the like can exist, so that the mapping assembly rule has flexible configuration items.

Herein, the following is the case:

the data source refers to the collected data to be processed.

Object types such as network devices, windows servers, liux servers, mysql databases, aphach services, etc.

The type of acquisition task: whether the task is directed to an object that is a network device or a server, etc.

Target template name: the target template is a data processing template corresponding to the acquisition task, and the acquired data is processed into target data according to the template.

The source template, the processing template and the target template are all things configured in the configuration file, the acquired data are data sources which are organized according to the source template, the data source processing is processed according to the processing template, and the processing result is organized according to the target template.

The data exchange format is JSON, because the JSON format is easy to read and write by people, and is easy to analyze and generate by machines, and the network transmission efficiency is effectively improved; XML, ASN.1, etc. may also be used.

The acquisition engine is a program in the form of a multithreaded pool and can also be realized in the form of multiple processes. The hash table, the queue and other data objects used by the acquisition engine can also be replaced by a middleware form.

Herein, the following is the case:

JSON: (JavaScript Object Notation, JS object profile) is a lightweight data exchange format. It stores and presents data in a text format that is completely independent of the programming language, based on a subset of ECMAScript (js specification formulated by the european computer institute). The compact and clear hierarchical structure makes JSON an ideal data exchange language. Is easy to read and write by people, is easy to analyze and generate by machines, and effectively improves the network transmission efficiency.

XML: an extensible markup language, a subset of the standard generic markup language, is a markup language used to mark electronic files to be structured.

Asn.1: ASN.1 abstract syntax notation (Abstract Syntax Notation One) ASN.1 is an ISO/ITU-T standard describing a data format for representing, encoding, transmitting and decoding data. It provides a complete set of formal formats for describing the structure of objects, regardless of how they are executed in language and the specific reference to these data, and without going to what application is at all.

And (3) middleware: the middleware is a type of software between the application system and the system software, uses basic services (functions) provided by the system software to connect various parts of the application system or different applications on the network, and can achieve the purposes of resource sharing and function sharing. At present, it is not very strictly defined, but the definition of IDC is generally accepted: middleware is a separate system software service by which distributed application software shares resources among different technologies, and is located on top of the client server's operating system, managing computing resources and network communications. In this sense, middleware can be represented by an equation: middleware = platform + communication, which also defines that middleware can only be called for use in a distributed system, while also distinguishing it from supporting software and utility software.

The process comprises the following steps: is a running activity of a program with independent functions with respect to a certain data set. It can apply for and own system resources, is a dynamic concept, and is an active entity. It includes not only the code of the program, but also the current activity, represented by the value of the program counter and the contents of the processing registers.

Thread: (English: thread) is the minimum unit that the operating system can perform operation scheduling. It is included in the process and is the actual unit of operation in the process. One thread refers to a single sequential control flow in a process, and multiple threads can be concurrent in a process, each thread executing different tasks in parallel. Lightweight processes (lightweight processes) are also called in Unix System V and SunOS, but lightweight processes refer more to kernel threads (kernel threads) and user threads (user threads) are called threads. Threads are the basic units of independent scheduling and dispatch. The thread may be a kernel thread, such as a Win32 thread, scheduled by an operating system kernel; user threads which are self-scheduled by user processes, such as POSIX Thread of a Linux platform; or by the kernel mixed scheduling with the threads of user processes, such as Windows 7. Multiple threads in the same process will share all system resources in the process, such as virtual address space, file descriptors, signal processing, and so forth. However, a plurality of threads in the same process have their own call stack (call stack), their own register context (register context), and their own thread local store (thread-local store). A process may have many threads, each thread executing different tasks in parallel.

The collection engine and method provided herein can be used for collection of any operation and maintenance data, and the specific collection content is configured according to the actual operation and maintenance requirements of the project, for example, the collection is commonly performed in the industry: the network equipment operation state, the port number, each port state, each port uplink flow downlink flow uplink packet loss rate downlink packet loss rate log. And (3) a server: cpu utilization, memory utilization, hard disk utilization, running status of various databases, number of user connections, corresponding time, tablespace conditions, hit rate conditions.

The printer running state, whether paper is out of order, the state of each ink box, or other functions such as virtual machine, cloud server, camera, machine room power supply, machine room air conditioner, smoke sensor, temperature and humidity, etc. can be configured to monitor and display in a centralized manner.

The present disclosure has been described with respect to the above-described embodiments, however, the above-described embodiments are merely examples of implementation of the present disclosure. It must be noted that the disclosed embodiments do not limit the scope of the present disclosure. Rather, the foregoing is considered to be illustrative, and it is to be understood that the invention is not limited to the specific details disclosed herein.

Claims

1. A dimension data acquisition device, the device comprising:

2. The operation and data acquisition device according to claim 1, wherein: the system also comprises a data output unit, wherein the data output unit is used for reading the monitored object data in the monitored object hash table and outputting the read monitored object data to the outside of the engine, and the data output unit is used for processing in one or more modes of:

3. A motion and data acquisition device according to claim 1 or 2, characterized in that: and the data center unit writes the monitoring object data to be stored in the monitoring object hash table into a time sequence database at regular intervals under the configuration of the configuration file.

4. A motion data acquisition device according to claim 1 or 2, characterized in that the acquisition task unit comprises:

5. The operation and data acquisition device according to claim 4, wherein: the acquisition task file is configured to set one or more acquisition tasks, and the elements of each acquisition task comprise a task name, an acquisition monitoring object type, a command template of a monitoring object script to be called, a parameter name required by the acquisition monitoring object script, the interval time of each acquisition of each monitoring object, an acquisition task attribute and a processing configuration file template for processing the acquisition result; and acquiring the specific parameter values in the parameter names required by the acquisition of the monitoring object script from the monitoring object hash table.

6. The operation and data acquisition device according to claim 4, wherein: the task collection thread pool comprises a rapid task collection thread pool for processing rapid tasks, a general task collection thread pool for processing general tasks and a failure task collection thread pool for processing failure tasks.

7. A motion and data acquisition device according to claim 1 or 2, characterized in that: the data formats in the monitored object hash table, the acquisition result data queue and the target data queue are JSON formats.

8. A method of operation and data acquisition, the method comprising the steps of:

step S03: and (3) data processing, namely continuously extracting the acquisition result from the acquisition result data queue established in the step (S02) to process, wherein the processed data is also used for establishing a target data queue and can be used for updating the hash table of the monitoring object established in the step (S01).

9. The operational dimension data collection method of claim 8, wherein: the method also comprises a data output step, wherein the step reads the monitored object data in the monitored object hash table and outputs the read monitored object data to the outside of the engine, and the method comprises one or more of the following modes:

10. The operational dimension data collection method of claim 8, wherein: the step S02 of data acquisition specifically comprises the following steps:

step S021: reading an acquisition task configuration file;

11. The operational dimension data collection method of claim 10, wherein: the collection task configuration file read in the step S021 is used for setting one or more collection tasks, and the elements of each collection task comprise a task name, a collection monitoring object type, a command template of a monitoring object script to be called, a parameter name required by the collection monitoring object script, the interval time of each collection of each monitoring object, collection task attributes and a processing configuration file template for processing the collection result; and the specific parameter values in the parameter names required by the acquisition monitoring object script are acquired from the object hash table established in the step S01.

12. The operational dimension data collection method of claim 11, wherein: the acquisition task attributes comprise a general task attribute, a quick task attribute and a failure task attribute, and each acquisition task can be endowed with the quick task attribute or the general task attribute when being initially configured.

13. The operational dimension data collection method of claim 10, wherein: before the step S023 is executed, the following processing is further performed, before the monitoring object script is called according to the acquisition task attribute and the output content returned by the monitoring object script is collected:

14. The operational dimension data collection method of claim 8, wherein: and step S02, judging whether the acquisition task configuration file and the monitoring object data are updated or not, if yes, updating and reading the acquisition task configuration, if no, judging whether the acquisition task is ended or not, if not, continuing to judge whether the acquisition task configuration and the monitoring object data are updated or not, and sequentially circulating.

15. The operational dimension data collection method of claim 8, wherein: the step S03 processes the data in the collected result data queue by the following steps:

16. The operational dimension data collection method of claim 15, wherein: and searching the source template, the processing template and the target template through a source data module configuration file, a processing template configuration file and a target data template configuration file respectively.