CN111026621A - Monitoring alarm method, device, equipment and medium for Elasticissearch cluster - Google Patents
Monitoring alarm method, device, equipment and medium for Elasticissearch cluster Download PDFInfo
- Publication number
- CN111026621A CN111026621A CN201911342583.5A CN201911342583A CN111026621A CN 111026621 A CN111026621 A CN 111026621A CN 201911342583 A CN201911342583 A CN 201911342583A CN 111026621 A CN111026621 A CN 111026621A
- Authority
- CN
- China
- Prior art keywords
- cluster
- operation data
- target
- data
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000012544 monitoring process Methods 0.000 title claims abstract description 37
- 230000002159 abnormal effect Effects 0.000 claims abstract description 65
- 238000013480 data collection Methods 0.000 claims abstract description 33
- 238000010219 correlation analysis Methods 0.000 claims abstract description 26
- 238000004590 computer program Methods 0.000 claims description 19
- 230000000903 blocking effect Effects 0.000 claims description 12
- 238000004140 cleaning Methods 0.000 claims description 4
- 238000007405 data analysis Methods 0.000 claims description 3
- 230000000875 corresponding effect Effects 0.000 claims 4
- 230000005856 abnormality Effects 0.000 claims 2
- 230000002596 correlated effect Effects 0.000 claims 1
- 239000003795 chemical substances by application Substances 0.000 description 25
- 230000000007 visual effect Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 239000012634 fragment Substances 0.000 description 4
- 238000012423 maintenance Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 2
- 238000012098 association analyses Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3055—Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/875—Monitoring of systems including the internet
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Computer Hardware Design (AREA)
- Debugging And Monitoring (AREA)
Abstract
本申请公开了一种面向Elasticsearch集群的监控报警方法、装置、设备、介质，该方法包括：通过Elasticsearch集群中的主节点上的Restful API获取目标集群运行数据；通过Elasticsearch集群中的工作节点上的数据收集代理获取目标节点运行数据，其中，所述数据收集代理为部署在所述工作节点中的可执行文件，用于收集所述工作节点的所述目标节点运行数据；将所述目标集群运行数据与所述目标节点运行数据进行关联分析，以判断所述Elasticsearch集群的运行状态是否出现异常；如果所述Elasticsearch集群的运行状态出现异常，则通过预设报警方式进行相应的报警。这样能够监控Elasticsearch集群的运行情况，并对异常运行情况产生相应的报警，且误报率低，可靠性高，成本低。
The present application discloses an Elasticsearch cluster-oriented monitoring and alarming method, device, equipment, and medium. The method includes: obtaining target cluster operation data through a Restful API on a master node in an Elasticsearch cluster; The data collection agent obtains target node operation data, wherein the data collection agent is an executable file deployed in the worker node, and is used to collect the target node operation data of the worker node; run the target cluster Correlation analysis is performed between the data and the operating data of the target node to determine whether the running state of the Elasticsearch cluster is abnormal; if the running state of the Elasticsearch cluster is abnormal, a corresponding alarm is issued by a preset alarm method. In this way, the operation of the Elasticsearch cluster can be monitored, and corresponding alarms can be generated for abnormal operation, and the false alarm rate is low, the reliability is high, and the cost is low.
Description
Technical Field
The application relates to the technical field of an elastic search cluster, in particular to a monitoring alarm method, device, equipment and medium for the elastic search cluster.
Background
With the advent of the big data age, ElasitcSearch is gaining more and more favor as a distributed full-text search engine. How to effectively and inexpensively monitor and manage the Elasticsearch cluster has been a big problem. At present, the monitoring method of the Elasticsearch cluster mainly collects cluster operation data on a main node in the Elasticsearch cluster, judges whether the cluster operation data is greater than or equal to a preset alarm threshold value, and gives a corresponding alarm when the cluster operation data is greater than or equal to the preset alarm threshold value, so that a large number of false alarms exist, the false alarm rate is high, and after the alarm, operation and maintenance personnel are required to investigate the reason of the alarm, the cost is high, and the reliability is low.
Disclosure of Invention
In view of this, an object of the present application is to provide a monitoring alarm method, apparatus, device, and medium for an Elasticsearch cluster, which can monitor an operating condition of the Elasticsearch cluster and generate a corresponding alarm for an abnormal operating condition, and has a low false alarm rate, high reliability, and low cost. The specific scheme is as follows:
in a first aspect, the application discloses an Elasticsearch cluster-oriented monitoring alarm method, which includes:
acquiring target cluster operation data through a Restful API on a main node in an Elasticissearch cluster;
acquiring target node operation data through a data collection agent on a working node in an Elasticissearch cluster, wherein the data collection agent is an executable file deployed in the working node and is used for collecting the target node operation data of the working node;
performing correlation analysis on the target cluster operation data and the target node operation data to judge whether the operation state of the Elasticissearch cluster is abnormal or not;
and if the running state of the Elasticissearch cluster is abnormal, giving corresponding alarm in a preset alarm mode.
Optionally, the obtaining the target cluster operation data through a Restful API on the master node in the Elasticsearch cluster includes:
acquiring target cluster running data comprising any one or a combination of several of a child node IP, a child node name, an index number, index health, an index state, a merge thread number, a task name, task running time, a segment number, a segment size, query delay, JVM (java virtual machine) usage, GC time, GC times, storage space occupied by the index and a fragment volume through a Restful API (application programming interface) on a main node in an elastic segment cluster.
Optionally, the obtaining, by the data collection agent on the working node in the Elasticsearch cluster, the target node operation data includes:
the method comprises the steps of obtaining target node operation data comprising any one or combination of CPU utilization rate, hard disk utilization rate, memory utilization rate, hard disk reading rate, hard disk writing rate and hard disk io blocking rate through a data collection agent on a working node in an Elasticissearch cluster.
Optionally, before performing the association analysis on the target cluster operation data and the target node operation data, the method further includes:
and cleaning the target cluster operation data and the target node operation data according to a preset association rule, and storing the target cluster operation data and the target node operation data into a corresponding database according to dates.
Optionally, after the target cluster operation data and the target node operation data are cleaned according to a preset association rule, the method further includes:
and performing visual display on the cleaned target cluster operation data and the cleaned target node operation data through ECharts drawing.
Optionally, after the corresponding alarm is performed in the preset alarm manner, the method further includes:
and analyzing the correlation analysis result to obtain an abnormal generation reason, and carrying out visual display on the abnormal generation reason.
Optionally, the performing, by performing correlation analysis on the target cluster operation data and the target node operation data to determine whether the operation state of the Elasticsearch cluster is abnormal includes:
judging whether the index running state of the Elasticissearch cluster is abnormal or not by analyzing the index name, the index number, the index health degree, the index state, the merge thread number, the segment size, the query delay and the fragment volume;
judging whether the operating efficiency of the Elasticissearch cluster is abnormal or not by analyzing the hard disk reading rate, the hard disk writing rate and the hard disk io blocking rate;
judging whether the operation of the related tasks of the Elasticissearch cluster is abnormal or not by analyzing the task operation time, the hard disk reading rate, the hard disk writing rate and the hard disk io blocking rate;
and judging whether the system operation load of the Elasticise cluster is abnormal or not by analyzing the JVM usage amount, the GC time, the GC times, the storage space occupied by the index, the CPU usage rate and the hard disk usage rate.
In a second aspect, the present application discloses an Elasticsearch cluster-oriented monitoring alarm device, including:
the system comprises a first data acquisition module, a second data acquisition module and a first data processing module, wherein the first data acquisition module is used for acquiring target cluster operation data through a Restful API on a main node in an Elasticissearch cluster;
a second data acquisition module, configured to acquire target node operation data through a data collection agent on a working node in an Elasticsearch cluster, where the data collection agent is an executable file deployed in the working node and is configured to collect the target node operation data of the working node;
the data analysis module is used for performing correlation analysis on the target cluster operation data and the target node operation data so as to judge whether the operation state of the Elasticise cluster is abnormal or not;
and the alarm module is used for giving corresponding alarm in a preset alarm mode when the running state of the Elasticissearch cluster is abnormal.
In a third aspect, the present application discloses an Elasticsearch cluster-oriented monitoring alarm device, including:
a memory and a processor;
wherein the memory is used for storing a computer program;
the processor is configured to execute the computer program to implement the aforementioned disclosed monitoring alarm method for the Elasticsearch cluster.
In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned disclosed monitoring alarm method for an Elasticsearch cluster.
Therefore, the method comprises the steps that firstly, the Restful API on the main node in the Elasticissearch cluster is used for obtaining the running data of the target cluster; acquiring target node operation data through a data collection agent on a working node in an Elasticissearch cluster, wherein the data collection agent is an executable file deployed in the working node and is used for collecting the target node operation data of the working node; then, performing correlation analysis on the target cluster operation data and the target node operation data to judge whether the operation state of the Elasticissearch cluster is abnormal or not; and if the running state of the Elasticissearch cluster is abnormal, giving corresponding alarm in a preset alarm mode. Therefore, after the target cluster operation data on the main node and the target node operation data on the working node in the Elasticissearch cluster are obtained, the target cluster operation data and the target node operation data are subjected to correlation analysis to judge whether the operation state of the Elasticissearch cluster is abnormal or not, if yes, corresponding alarm is given, so that the operation condition of the Elasticissearch cluster can be monitored, corresponding alarm is given to the abnormal operation condition, and the Elasticissearch cluster alarm system is low in false alarm rate, high in reliability and low in cost.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a flowchart of an Elasticissearch cluster-oriented monitoring alarm method disclosed in the present application;
FIG. 2 is a flowchart of a specific monitoring alarm method for an Elasticsearch cluster disclosed in the present application;
FIG. 3 is a schematic structural diagram of an elastic search cluster-oriented monitoring alarm device disclosed in the present application;
FIG. 4 is a structural diagram of a monitoring alarm device facing to an elastic search cluster disclosed in the present application;
fig. 5 is a diagram of a server structure disclosed in the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
At present, the monitoring method of the Elasticsearch cluster mainly collects cluster operation data on a main node in the Elasticsearch cluster, judges whether the cluster operation data is greater than or equal to a preset alarm threshold value, and gives a corresponding alarm when the cluster operation data is greater than or equal to the preset alarm threshold value, so that a large number of false alarms exist, the false alarm rate is high, and after the alarm, operation and maintenance personnel are required to investigate the reason of the alarm, the cost is high, and the reliability is low. In view of the above, the application provides an Elasticsearch cluster-oriented monitoring alarm method, which can monitor the operation condition of an Elasticsearch cluster and generate a corresponding alarm for an abnormal operation condition, and has the advantages of low false alarm rate, high reliability and low cost.
Referring to fig. 1, an embodiment of the present application discloses an Elasticsearch cluster-oriented monitoring alarm method, including:
step S11: and acquiring the target cluster operation data through a Restful API on the main node in the Elasticissearch cluster.
In this embodiment, the target cluster operating data on the main node in the Elasticsearch cluster needs to be acquired first, and specifically, the target cluster operating data may be acquired through an Application Programming Interface (Restful API) on the main node. The target cluster operation data may also include other cluster operation data, wherein the target cluster operation data includes any one or a combination of several of an IP child node, a child node name, an index number, an index health, an index state, a merge thread number, a task name, a task operation time, a segment number, a segment size, an inquiry delay, a usage amount of a JVM (Java Virtual Machine), a GC (Garbage Collection) time, a GC number, an index occupied storage space, and a segment volume.
Step S12: and acquiring target node operation data through a data collection agent on a working node in the Elasticissearch cluster, wherein the data collection agent is an executable file deployed in the working node and is used for collecting the target node operation data of the working node.
In this embodiment, it is further required to obtain target node operation data on a working node in the Elasticsearch cluster, specifically, the target node operation data is obtained through a data collection agent on the working node, where the data collection agent is an executable file deployed in the working node and is used for the target node operation data of the working node, the target node operation data includes any one or a combination of a CPU usage rate, a hard disk usage rate, a memory usage rate, a hard disk read rate, a hard disk write rate, and a hard disk io blocking rate, and the target node operation data may further include other node operation data.
Step S13: and performing correlation analysis on the target cluster operation data and the target node operation data to judge whether the operation state of the Elasticissearch cluster is abnormal or not.
After the target cluster operation data and the target node operation data are obtained, correlation analysis needs to be performed on the target cluster operation data and the target node operation data to judge whether the operation state of the Elasticise cluster is abnormal or not. The target cluster operation data on the main node in the Elasticissearch cluster and the target node operation data on the working node have relevance, and whether the operation data of the working node and the operation data of the main node are corresponding and consistent or not can be determined by analyzing the operation data with relevance so as to judge whether the operation state of the Elasticissearch cluster is abnormal or not.
Step S14: and if the running state of the Elasticissearch cluster is abnormal, giving corresponding alarm in a preset alarm mode.
In a specific implementation process, after the target cluster operation data and the target node operation data are subjected to correlation analysis, if the operation state of the Elasticsearch cluster is abnormal, corresponding alarm is given in a preset alarm mode. Specifically, the alarm mode includes, but is not limited to, a visual information prompting mode and a voice prompting mode. For example by way of mail.
Therefore, the method comprises the steps that firstly, the Restful API on the main node in the Elasticissearch cluster is used for obtaining the running data of the target cluster; acquiring target node operation data through a data collection agent on a working node in an Elasticissearch cluster, wherein the data collection agent is an executable file deployed in the working node and is used for collecting the target node operation data of the working node; then, performing correlation analysis on the target cluster operation data and the target node operation data to judge whether the operation state of the Elasticissearch cluster is abnormal or not; and if the running state of the Elasticissearch cluster is abnormal, giving corresponding alarm in a preset alarm mode. Therefore, after the target cluster operation data on the main node and the target node operation data on the working node in the Elasticissearch cluster are obtained, the target cluster operation data and the target node operation data are subjected to correlation analysis to judge whether the operation state of the Elasticissearch cluster is abnormal or not, if yes, corresponding alarm is given, so that the operation condition of the Elasticissearch cluster can be monitored, corresponding alarm is given to the abnormal operation condition, and the Elasticissearch cluster alarm system is low in false alarm rate, high in reliability and low in cost.
Referring to fig. 2, an embodiment of the present application discloses a specific monitoring alarm method for an Elasticsearch cluster, where the method includes:
step S21: and acquiring the target cluster operation data through a Restful API on the main node in the Elasticissearch cluster.
Step S22: the method comprises the steps of obtaining target node operation data through a data collection agent on a working node in an Elasticissearch cluster, wherein the data collection agent is an executable file deployed in the working node and used for collecting the target node operation data of the working node.
Step S23: and cleaning the target cluster operation data and the target node operation data according to a preset association rule, and storing the target cluster operation data and the target node operation data into a corresponding database according to dates.
In a specific implementation process, after the target cluster operation data and the target node operation data are obtained, the target cluster operation data and the target node operation data need to be cleaned according to a preset association rule, and the cleaned target cluster operation data and the cleaned target node operation data are stored in corresponding databases. Specifically, the target cluster operation data and the target node operation data are cleaned according to a preset rule by taking a time line as a basis, so that the target cluster operation data and the target node operation data are associated.
Step S24: and performing visual display on the cleaned target cluster operation data and the cleaned target node operation data through ECharts drawing.
It can be understood that after the cleaned target cluster operation data and target node operation data are stored in the corresponding databases, the target cluster operation data and the target node operation data in the databases need to be visually displayed through an ECharts drawing, so that operation and maintenance personnel can know the current operation condition of the whole cluster.
Step S25: and performing correlation analysis on the target cluster operation data and the target node operation data to judge whether the operation state of the Elasticissearch cluster is abnormal or not.
It can be understood that the target cluster operation data and the target node operation data need to be subjected to correlation analysis to determine whether the operation state of the Elasticsearch cluster is abnormal. Specifically, whether the operating efficiency of the Elasticsearch cluster is abnormal is judged by analyzing the hard disk reading rate, the hard disk writing rate and the hard disk io blocking rate; judging whether the operation of the related tasks of the Elasticissearch cluster is abnormal or not by analyzing the task operation time, the hard disk reading rate, the hard disk writing rate and the hard disk io blocking rate; and judging whether the system operation load of the Elasticise cluster is abnormal or not by analyzing the JVM usage amount, the GC time, the GC times, the storage space occupied by the index, the CPU usage rate and the hard disk usage rate. And when the index running state, the running effect, the related task running or the system running load of the Elasticissearch cluster are abnormal, judging that the running state of the Elasticissearch cluster is abnormal.
Step S26: and if the running state of the Elasticissearch cluster is abnormal, giving corresponding alarm in a preset alarm mode.
Step S27: and analyzing the correlation analysis result to obtain an abnormal generation reason, and carrying out visual display on the abnormal generation reason.
Referring to fig. 3, an embodiment of the present application discloses an Elasticsearch cluster-oriented monitoring alarm device, including:
a first data obtaining module 11, configured to obtain target cluster operation data through a Restful API on a host node in an Elasticsearch cluster;
a second data obtaining module 12, configured to obtain target node operation data through a data collection agent on a working node in an Elasticsearch cluster, where the data collection agent is an executable file deployed in the working node and is configured to collect the target node operation data of the working node;
the data analysis module 13 is configured to perform correlation analysis on the target cluster operation data and the target node operation data to determine whether an operation state of the Elasticsearch cluster is abnormal;
and the alarm module 14 is configured to perform corresponding alarm in a preset alarm manner when the operating state of the Elasticsearch cluster is abnormal.
Therefore, the method comprises the steps that firstly, the Restful API on the main node in the Elasticissearch cluster is used for obtaining the running data of the target cluster; acquiring target node operation data through a data collection agent on a working node in an Elasticissearch cluster, wherein the data collection agent is an executable file deployed in the working node and is used for collecting the target node operation data of the working node; then, performing correlation analysis on the target cluster operation data and the target node operation data to judge whether the operation state of the Elasticissearch cluster is abnormal or not; and if the running state of the Elasticissearch cluster is abnormal, giving corresponding alarm in a preset alarm mode. Therefore, after the target cluster operation data on the main node and the target node operation data on the working node in the Elasticissearch cluster are obtained, the target cluster operation data and the target node operation data are subjected to correlation analysis to judge whether the operation state of the Elasticissearch cluster is abnormal or not, if yes, corresponding alarm is given, so that the operation condition of the Elasticissearch cluster can be monitored, corresponding alarm is given to the abnormal operation condition, and the Elasticissearch cluster alarm system is low in false alarm rate, high in reliability and low in cost.
Further, referring to fig. 4, an embodiment of the present application further discloses an Elasticsearch cluster-oriented monitoring alarm device, including: a processor 21 and a memory 22.
Wherein the memory 22 is used for storing a computer program; the processor 21 is configured to execute the computer program to implement the monitoring alarm method facing the Elasticsearch cluster disclosed in the foregoing embodiment.
For a specific process of the monitoring alarm method for the Elasticsearch cluster, reference may be made to corresponding contents disclosed in the foregoing embodiments, and details are not repeated herein.
Further, as shown in fig. 5, a schematic diagram of a server structure provided in the embodiment of the present application is shown. The server 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, sensors 25, and a communication bus 26. The memory 42 is configured to store a computer program, and the computer program is loaded and executed by the processor 21 to implement relevant steps in the monitoring alarm method for an Elasticsearch cluster disclosed in any of the foregoing embodiments.
In this embodiment, the power supply 23 is configured to provide a working voltage for each hardware on the internet of things device; the communication interface 24 creates a data transmission channel between the server 20 and an external device, and the communication protocol followed by the communication interface is any communication protocol applicable to the technical solution of the present application, and is not specifically limited again; sensor 25 for acquiring sensor data, specific sensor types including, but not limited to, speed sensor, temperature sensor, infrared sensor, light sensor, sound sensor, image sensor, and the like.
In addition, the storage 22 is used as a carrier for storing resources, and may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., the resources stored thereon include an operating system 221, a computer program 222, data 223, etc., and the storage may be a transient storage or a permanent storage.
The operating system 221 is used to manage and control hardware and computer programs 222 on the internet of things device 20, so as to implement operations and processing on the mass databases 223 in the processor 21 and the memory 22, and may be Windows, Unix, Linux, or the like. The computer program 222 may further include a computer program that can be used to perform other specific tasks in addition to the computer program that can be used to perform the Elasticsearch cluster-oriented monitoring alarm method disclosed in any of the foregoing embodiments. The data 223 may include data received by the server and transmitted from an external device, or may include data collected by the sensor 25 itself.
Further, an embodiment of the present application also discloses a computer readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the following steps:
acquiring target cluster operation data through a Restful API on a main node in an Elasticissearch cluster; acquiring target node operation data through a data collection agent on a working node in an Elasticissearch cluster, wherein the data collection agent is an executable file deployed in the working node and is used for collecting the target node operation data of the working node; performing correlation analysis on the target cluster operation data and the target node operation data to judge whether the operation state of the Elasticissearch cluster is abnormal or not; and if the running state of the Elasticissearch cluster is abnormal, giving corresponding alarm in a preset alarm mode.
Therefore, the method comprises the steps that firstly, the Restful API on the main node in the Elasticissearch cluster is used for obtaining the running data of the target cluster; acquiring target node operation data through a data collection agent on a working node in an Elasticissearch cluster, wherein the data collection agent is an executable file deployed in the working node and is used for collecting the target node operation data of the working node; then, performing correlation analysis on the target cluster operation data and the target node operation data to judge whether the operation state of the Elasticissearch cluster is abnormal or not; and if the running state of the Elasticissearch cluster is abnormal, giving corresponding alarm in a preset alarm mode. Therefore, after the target cluster operation data on the main node and the target node operation data on the working node in the Elasticissearch cluster are obtained, the target cluster operation data and the target node operation data are subjected to correlation analysis to judge whether the operation state of the Elasticissearch cluster is abnormal or not, if yes, corresponding alarm is given, so that the operation condition of the Elasticissearch cluster can be monitored, corresponding alarm is given to the abnormal operation condition, and the Elasticissearch cluster alarm system is low in false alarm rate, high in reliability and low in cost.
In this embodiment, when the computer subprogram stored in the computer-readable storage medium is executed by the processor, the following steps may be specifically implemented: acquiring target cluster running data comprising any one or a combination of several of a child node IP, a child node name, an index number, index health, an index state, a merge thread number, a task name, task running time, a segment number, a segment size, query delay, JVM (java virtual machine) usage, GC time, GC times, storage space occupied by the index and a fragment volume through a Restful API (application programming interface) on a main node in an elastic segment cluster.
In this embodiment, when the computer subprogram stored in the computer-readable storage medium is executed by the processor, the following steps may be specifically implemented: the method comprises the steps of obtaining target node operation data comprising any one or combination of CPU utilization rate, hard disk utilization rate, memory utilization rate, hard disk reading rate, hard disk writing rate and hard disk io blocking rate through a data collection agent on a working node in an Elasticissearch cluster.
In this embodiment, when the computer subprogram stored in the computer-readable storage medium is executed by the processor, the following steps may be specifically implemented: and cleaning the target cluster operation data and the target node operation data according to a preset association rule, and storing the target cluster operation data and the target node operation data into a corresponding database according to dates.
In this embodiment, when the computer subprogram stored in the computer-readable storage medium is executed by the processor, the following steps may be specifically implemented: and performing visual display on the cleaned target cluster operation data and the cleaned target node operation data through ECharts drawing.
In this embodiment, when the computer subprogram stored in the computer-readable storage medium is executed by the processor, the following steps may be specifically implemented: and analyzing the correlation analysis result to obtain an abnormal generation reason, and carrying out visual display on the abnormal generation reason.
In this embodiment, when the computer subprogram stored in the computer-readable storage medium is executed by the processor, the following steps may be specifically implemented: judging whether the index running state of the Elasticissearch cluster is abnormal or not by analyzing the index name, the index number, the index health degree, the index state, the merge thread number, the segment size, the query delay and the fragment volume; judging whether the operating efficiency of the Elasticissearch cluster is abnormal or not by analyzing the hard disk reading rate, the hard disk writing rate and the hard disk io blocking rate; judging whether the operation of the related tasks of the Elasticissearch cluster is abnormal or not by analyzing the task operation time, the hard disk reading rate, the hard disk writing rate and the hard disk io blocking rate; and judging whether the system operation load of the Elasticise cluster is abnormal or not by analyzing the JVM usage amount, the GC time, the GC times, the storage space occupied by the index, the CPU usage rate and the hard disk usage rate.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Finally, it is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of other elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The detailed description is given above to the monitoring alarm method, device, equipment and medium for the Elasticsearch cluster provided by the present application, a specific example is applied in the present application to explain the principle and the implementation manner of the present application, and the description of the above embodiment is only used to help understanding the method and the core idea of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911342583.5A CN111026621B (en) | 2019-12-23 | 2019-12-23 | Monitoring alarm method, device, equipment and medium for Elasticissearch cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911342583.5A CN111026621B (en) | 2019-12-23 | 2019-12-23 | Monitoring alarm method, device, equipment and medium for Elasticissearch cluster |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111026621A true CN111026621A (en) | 2020-04-17 |
| CN111026621B CN111026621B (en) | 2023-04-07 |
Family
ID=70211821
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911342583.5A Active CN111026621B (en) | 2019-12-23 | 2019-12-23 | Monitoring alarm method, device, equipment and medium for Elasticissearch cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111026621B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112100160A (en) * | 2020-11-05 | 2020-12-18 | 四川新网银行股份有限公司 | Elastic Search based double-activity real-time data warehouse construction method |
| CN112380107A (en) * | 2020-12-08 | 2021-02-19 | 无锡无边网络技术有限公司 | Operation and maintenance system data acquisition system and method based on management information system |
| CN112532435A (en) * | 2020-11-20 | 2021-03-19 | 深信服科技股份有限公司 | Operation and maintenance method, operation and maintenance management platform, equipment and medium |
| CN113192228A (en) * | 2021-04-30 | 2021-07-30 | 中国工商银行股份有限公司 | Cluster automation inspection method and device |
| CN113608964A (en) * | 2021-08-09 | 2021-11-05 | 宁畅信息产业(北京)有限公司 | A cluster automatic monitoring method, device, electronic device and storage medium |
| CN115686381A (en) * | 2022-12-29 | 2023-02-03 | 苏州浪潮智能科技有限公司 | Prediction method and device for storage cluster running state |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090193436A1 (en) * | 2008-01-30 | 2009-07-30 | Inventec Corporation | Alarm display system of cluster storage system and method thereof |
| CN105933175A (en) * | 2016-07-14 | 2016-09-07 | 微额速达（上海）金融信息服务有限公司 | Real-time monitoring early-warning system |
| CN106383776A (en) * | 2016-08-30 | 2017-02-08 | 北京北信源软件股份有限公司 | Monitoring and self-healing method and apparatus for distributed search cluster system |
| WO2017071563A1 (en) * | 2015-10-31 | 2017-05-04 | 华为技术有限公司 | Data storage method and cluster management node |
| CN107734035A (en) * | 2017-10-17 | 2018-02-23 | 华南理工大学 | A kind of Virtual Cluster automatic telescopic method under cloud computing environment |
| CN109101397A (en) * | 2018-08-01 | 2018-12-28 | 武汉索雅信息技术有限公司 | High-Performance Computing Cluster monitoring method, unit and storage medium |
| CN110309030A (en) * | 2019-07-05 | 2019-10-08 | 亿玛创新网络(天津)有限公司 | Log analysis monitoring system and method based on ELK and Zabbix |
-
2019
- 2019-12-23 CN CN201911342583.5A patent/CN111026621B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090193436A1 (en) * | 2008-01-30 | 2009-07-30 | Inventec Corporation | Alarm display system of cluster storage system and method thereof |
| WO2017071563A1 (en) * | 2015-10-31 | 2017-05-04 | 华为技术有限公司 | Data storage method and cluster management node |
| CN105933175A (en) * | 2016-07-14 | 2016-09-07 | 微额速达（上海）金融信息服务有限公司 | Real-time monitoring early-warning system |
| CN106383776A (en) * | 2016-08-30 | 2017-02-08 | 北京北信源软件股份有限公司 | Monitoring and self-healing method and apparatus for distributed search cluster system |
| CN107734035A (en) * | 2017-10-17 | 2018-02-23 | 华南理工大学 | A kind of Virtual Cluster automatic telescopic method under cloud computing environment |
| CN109101397A (en) * | 2018-08-01 | 2018-12-28 | 武汉索雅信息技术有限公司 | High-Performance Computing Cluster monitoring method, unit and storage medium |
| CN110309030A (en) * | 2019-07-05 | 2019-10-08 | 亿玛创新网络(天津)有限公司 | Log analysis monitoring system and method based on ELK and Zabbix |
Non-Patent Citations (3)
| Title |
|---|
| 姚攀;马玉鹏;徐春香;: "基于ELK的日志分析系统研究及应用", 计算机工程与设计 * |
| 王景德;李建宇;付喜春;刘洪海;李家俊;: "基于多架构集群一体化动态监控软件的实现", 信息技术 * |
| 胡庆宝;姜晓巍;石京燕;程耀东;梁翠萍;: "基于Elasticsearch的实时集群日志采集和分析系统实现", 科研信息化技术与应用 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112100160A (en) * | 2020-11-05 | 2020-12-18 | 四川新网银行股份有限公司 | Elastic Search based double-activity real-time data warehouse construction method |
| CN112532435A (en) * | 2020-11-20 | 2021-03-19 | 深信服科技股份有限公司 | Operation and maintenance method, operation and maintenance management platform, equipment and medium |
| CN112532435B (en) * | 2020-11-20 | 2023-09-08 | 深信服科技股份有限公司 | Operation and maintenance method, operation and maintenance management platform, equipment and medium |
| CN112380107A (en) * | 2020-12-08 | 2021-02-19 | 无锡无边网络技术有限公司 | Operation and maintenance system data acquisition system and method based on management information system |
| CN113192228A (en) * | 2021-04-30 | 2021-07-30 | 中国工商银行股份有限公司 | Cluster automation inspection method and device |
| CN113608964A (en) * | 2021-08-09 | 2021-11-05 | 宁畅信息产业(北京)有限公司 | A cluster automatic monitoring method, device, electronic device and storage medium |
| CN115686381A (en) * | 2022-12-29 | 2023-02-03 | 苏州浪潮智能科技有限公司 | Prediction method and device for storage cluster running state |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111026621B (en) | 2023-04-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111026621A (en) | Monitoring alarm method, device, equipment and medium for Elasticissearch cluster | |
| JP5385982B2 (en) | A management system that outputs information indicating the recovery method corresponding to the root cause of the failure | |
| US8286139B2 (en) | Call stack sampling for threads having latencies exceeding a threshold | |
| US7310590B1 (en) | Time series anomaly detection using multiple statistical models | |
| CN107924360B (en) | Diagnostic framework in a computing system | |
| US20100153431A1 (en) | Alert triggered statistics collections | |
| CN110888783A (en) | Monitoring method and device of micro-service system and electronic equipment | |
| US10657099B1 (en) | Systems and methods for transformation and analysis of logfile data | |
| US9692654B2 (en) | Systems and methods for correlating derived metrics for system activity | |
| JPWO2010061735A1 (en) | System for supporting action execution according to detection event, method for supporting action execution according to detection event, support apparatus, and computer program | |
| CN114006727B (en) | Alarm association analysis method, device, equipment and storage medium | |
| US9043652B2 (en) | User-coordinated resource recovery | |
| US10114731B2 (en) | Including kernel object information in a user dump | |
| US11650966B2 (en) | Assisted problem identification in a computing system | |
| US10187264B1 (en) | Gateway path variable detection for metric collection | |
| CN112800061B (en) | Data storage method, device, server and storage medium | |
| CA2759365A1 (en) | Identification of thread progress information | |
| JP2018165857A (en) | Analysis device, analysis system, analysis method, and analysis program | |
| US9397921B2 (en) | Method and system for signal categorization for monitoring and detecting health changes in a database system | |
| CN109586989A (en) | A kind of state detection method, device and group system | |
| JP2004348640A (en) | Method and system for managing network | |
| CN113901441A (en) | User abnormal request detection method, device, equipment and storage medium | |
| JP2014137635A (en) | Response time monitoring program, method and device | |
| CN115580528A (en) | Fault root cause positioning method, device, equipment and readable storage medium | |
| CN113760856A (en) | Database management method and device, computer readable storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20200417 Assignee: Hangzhou Anheng Information Security Technology Co.,Ltd. Assignor: Dbappsecurity Co.,Ltd. Contract record no.: X2024980043364 Denomination of invention: Monitoring and alarm methods, devices, equipment, and media for Elasticsearch clusters Granted publication date: 20230407 License type: Common License Record date: 20241231 |
|
| EE01 | Entry into force of recordation of patent licensing contract |