CN111143438A - Workshop field data real-time monitoring and anomaly detection method based on stream processing - Google Patents
Workshop field data real-time monitoring and anomaly detection method based on stream processing Download PDFInfo
- Publication number
- CN111143438A CN111143438A CN201911387816.3A CN201911387816A CN111143438A CN 111143438 A CN111143438 A CN 111143438A CN 201911387816 A CN201911387816 A CN 201911387816A CN 111143438 A CN111143438 A CN 111143438A
- Authority
- CN
- China
- Prior art keywords
- data
- real
- time
- anomaly detection
- monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012545 processing Methods 0.000 title claims abstract description 70
- 238000012544 monitoring process Methods 0.000 title claims abstract description 46
- 238000001514 detection method Methods 0.000 title claims abstract description 45
- 238000004519 manufacturing process Methods 0.000 claims abstract description 32
- 230000002159 abnormal effect Effects 0.000 claims abstract description 23
- 238000007781 pre-processing Methods 0.000 claims abstract description 15
- 238000000034 method Methods 0.000 claims description 30
- 238000004364 calculation method Methods 0.000 claims description 27
- 238000005111 flow chemistry technique Methods 0.000 claims description 11
- 238000004458 analytical method Methods 0.000 claims description 7
- 238000004140 cleaning Methods 0.000 claims description 7
- 238000004220 aggregation Methods 0.000 claims description 6
- 230000002776 aggregation Effects 0.000 claims description 6
- 238000007726 management method Methods 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000012217 deletion Methods 0.000 claims description 3
- 230000037430 deletion Effects 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 3
- 238000011160 research Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 abstract description 16
- 238000013480 data collection Methods 0.000 abstract 1
- 238000012360 testing method Methods 0.000 description 16
- 238000010586 diagram Methods 0.000 description 7
- 238000003860 storage Methods 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 238000007689 inspection Methods 0.000 description 3
- 239000000463 material Substances 0.000 description 3
- 238000003466 welding Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000010223 real-time analysis Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000009413 insulation Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 239000000178 monomer Substances 0.000 description 1
- 238000011056 performance test Methods 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000012797 qualification Methods 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 238000011895 specific detection Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2477—Temporal data queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/04—Manufacturing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Probability & Statistics with Applications (AREA)
- Business, Economics & Management (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Computing Systems (AREA)
- Manufacturing & Machinery (AREA)
- Quality & Reliability (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- General Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- General Factory Administration (AREA)
Abstract
The invention relates to the technical field of workshop field data monitoring, in particular to a workshop field data real-time monitoring and anomaly detection method based on stream processing, which comprises the following steps: firstly, data are collected by workshop production operation data collection equipment and data flow is generated, then data caching is carried out through a Kafka distributed message queue, the data are transmitted to a next-level Bolt for independent data preprocessing, operation data statistics and abnormal detection based on a sliding time window are carried out, and then according to the data corresponding to the judged abnormal state, the data are identified and compared with the corresponding abnormal state data in a database; according to the invention, through data preprocessing is carried out on the data stream, obvious errors such as null values, outliers and the like of production field operation data are eliminated, and an operation data statistics and anomaly detection technology based on a sliding time window is designed, so that the timeliness of operation data processing is ensured, and the relevance of connected data is also ensured to a certain extent.
Description
Technical Field
The invention relates to the technical field of workshop field data monitoring, in particular to a real-time workshop field data monitoring and anomaly detection method based on stream processing.
Background
The industrial manufacturing industry rises rapidly along with the development of Chinese economy, and the development of the manufacturing capability of the industrial manufacturing industry directly influences the national economic development and social progress. Due to the complexity of the site environment of the production workshop and the diversity of product information, the resource allocation of enterprises is unreasonable, and material waste and economic loss are caused.
In recent years, with the development of sensing detection technology and the popularization of internet of things equipment, workshop production site operation data shows exponential rise, for a workshop site, the real-time performance of a monitoring system is low, the expandability is not high, delay and even running are easy to occur, the total data amount and the value are rapidly increased and lost along with the time lapse, the mining value exceeds the computing capacity of traditional data processing, the real-time monitoring and management requirements on workshop production cannot be met, and the challenges of real-time processing of mass operation data streams, mass data storage, real-time visual analysis of data and the like are brought to the existing abnormal state monitoring system based on the workshop site operation data.
Based on the above, the invention designs a real-time monitoring and anomaly detection method for workshop field data based on stream processing, so as to solve the problems.
Disclosure of Invention
The invention aims to provide a workshop field data real-time monitoring and anomaly detection method based on stream processing, so as to solve the problems in the background technology.
In order to achieve the purpose, the invention provides the following technical scheme: a real-time monitoring and anomaly detection method for workshop field data based on stream processing is characterized by comprising the following steps:
s1: the workshop production operation data acquisition equipment acquires data and generates a data stream;
s2: respectively registering nodes of Kafka and Storm on a distributed server management system Zookeeper, and uniformly managing the server nodes of Kafka and Storm;
s3: performing data caching through a Kafka distributed message queue, taking a real-time big data computing platform with a Storm deployed as a data consumption end of the Kafka, adopting an integrated Kafka Spout of the Kafka and the Storm as a data source connected with the message queue and the big data platform, and transmitting the data source to the next-level Bolt in the form of a data carrier for analysis and processing;
s4: after the data source Kafkaspout receives the operation data, firstly splitting Bolt through the data to obtain the data of different operation data on different acquisition channels;
s5: performing data preprocessing in Bolt alone;
s6: then, performing running data statistics and anomaly detection based on the sliding time window, setting the sliding window to be 0.5min, setting the basic time window to be 5s, performing independent calculation tasks under each basic time window, sliding the calculation tasks within 0.5min by using the sliding window of 5s at the moment, finally merging the basic time windows, and performing statistics on the calculation result of the whole calculation window;
s7: after the spit and the Bolt are finished, setting the data flow direction and the grouping mode of each component in Topology of Storm;
s8: and realizing on-line judgment of the workshop production field state in a Storm real-time processing frame through a real-time flow clustering algorithm, identifying and comparing the corresponding data in the judged abnormal state with the corresponding abnormal state data in the database, outputting an abnormal record, and storing the abnormal record in the database.
Preferably, the step of preprocessing the data in step S5 includes data cleaning, data formatting and determining whether the data needs to be stored.
Preferably, the running data statistics in step S6 refers to statistics of the maximum, minimum, average, frequency of occurrence, and energy utilization of the running data, and the anomaly detection mainly includes real-time critical detection and anomaly monitoring based on a sliding time window.
Preferably, the specific steps and units of the operation data statistics and the anomaly detection based on the sliding time window in step S6 include:
the method comprises the following steps: a sliding window processing unit: firstly, setting time and parameters of data transmission required by a sliding window, wherein the parameters mainly comprise unit window length and sliding window length;
step two: a field splitting processing unit: splitting the received operation data according to the type of the monitoring signal, and sending the operation data of the same type to a next-stage data processing unit;
step three: a data statistics processing unit: the data indexes of the maximum value and the average value of the data are calculated in the unit time window, and the calculation result is sent to the next-stage data processing unit to carry out data aggregation of the whole time window;
step four: frequency count calculation processing unit: the method mainly aims to realize running data service needing frequency statistics;
step five: a threshold value judgment processing unit: according to the type of each monitoring data, combining the data characteristics and related research, and making a strategy for judging the threshold value;
step six: a summary calculation processing unit: and combining and counting the data of all the basic time windows on the processing unit, and using a global summary data aggregation mode, namely sending all the data to the same processing unit for final calculation.
Preferably, the specific way of data cleaning is as follows: firstly, checking whether the data is qualified, if so, continuing to process the subsequent data, if not, executing data cleaning and filtering operation, and deleting the data with null value, outlier and obvious error information from the monitoring data.
Preferably, the data formatting includes data calibration and redundancy value deletion, and the specific method includes: the data is calibrated and formatted to convert analog values to real values while eliminating redundant information such as data headers, headers and other fields from the data stored by the sensor and acquisition system.
Preferably, the judgment basis for judging whether the data needs to be stored is that whether the data needs to be processed first and then stored in the database.
Compared with the prior art, the invention has the beneficial effects that: according to the invention, through data preprocessing on the data stream, obvious errors such as null values, outliers, redundant data, format errors and the like of production field operation data are eliminated; the running data statistics and anomaly detection technology based on the sliding time window is designed, so that the timeliness of running data processing is guaranteed, the relevance of connected data is guaranteed to a certain extent, the state information in the field process of a production workshop is fully, accurately and timely obtained, the data in the production process is effectively controlled, analyzed and managed, and production error prevention is carried out in real time.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a process flow diagram of the plant site operational data flow of the present invention;
FIG. 2 is a flow chart of data preprocessing according to the present invention;
FIG. 3 is a schematic diagram of data statistics and detection based on a sliding time window according to the present invention;
FIG. 4 is a flow chart of the implementation of data statistics and detection based on sliding time window according to the present invention;
FIG. 5 is a business flow diagram of the condition monitoring platform software system for production data of the present invention;
FIG. 6 is a schematic diagram of the model and detection index of the present invention;
FIG. 7 is a diagram of a front-end cell open circuit voltage test page according to the present invention;
FIG. 8 is an interface diagram of the inspection record of the workshop production process of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1-5, the present invention provides a technical solution: a real-time monitoring and anomaly detection method for workshop field data based on stream processing comprises the following steps:
s1: the workshop production operation data acquisition equipment acquires data and generates a data stream;
s2: the user has different operation authorities according to the authority of the account, logs in the system after the identity is successfully verified, initializes the system, respectively registers the nodes of Kafka and Storm on a distributed server management system Zookeeper, and uniformly manages the server nodes of Kafka and Storm;
s3: performing data caching through a Kafka distributed message queue, taking a real-time big data computing platform with a Storm deployed as a data consumption end of the Kafka, adopting an integrated Kafka Spout of the Kafka and the Storm as a data source connected with the message queue and the big data platform, and transmitting the data source to the next-level Bolt in the form of a data carrier for analysis and processing;
s4: after the data source Kafkaspout receives the operation data, firstly splitting Bolt through the data to obtain the data of different operation data on different acquisition channels;
s5: data preprocessing is carried out in Bolt alone, and the data preprocessing can also be realized by programming in Bolt alone in combination with specific operation data monitoring requirements; in the actual production and operation process, the sensor and the data acquisition equipment are influenced by external electromagnetic and noise interference or abnormal conditions, and the data preprocessing is used for eliminating error and redundant data generated under the influence;
the first step of data preprocessing is to clean the data, firstly check whether the data is qualified, and if so, continue to process the subsequent data. If the data is not qualified, performing a data cleaning and filtering operation to remove data with null values, outliers and significant error information from the monitored data;
and a second step of data formatting, including data calibration and redundant value deletion, specifically, calibrating and formatting the data, converting the analog value into a real value, and simultaneously, storing data in the sensor and the acquisition system, wherein the data contains a plurality of redundant information, such as data titles, headers and other fields, which are useless data for the monitoring system and need to be deleted.
And thirdly, judging whether the data need to be stored or not, and distinguishing the data by taking whether the data need to be directly stored in a database or need to be processed after storage as a judgment basis.
S6: then, performing operation data statistics and anomaly detection based on the sliding time window, setting the sliding window to be 0.5min, setting the basic time window to be 5s, performing independent calculation tasks under each basic time window, wherein the calculation tasks slide within 0.5min by using the sliding window of 5s, finally merging the basic time windows, and performing statistics on the calculation result of the whole calculation window (0.5min), wherein the operation data statistics refers to the statistics on the indexes of the maximum value, the minimum value, the average value, the frequency of occurrence, the energy utilization rate and the like of the operation data, the anomaly detection mainly comprises real-time critical detection and anomaly monitoring based on the sliding time window, and the specific steps and units of the operation data statistics and the anomaly detection based on the sliding time window comprise:
the method comprises the following steps: a sliding window processing unit: firstly, setting time and parameters of data transmission required by a sliding window, wherein the parameters mainly comprise unit window length and sliding window length;
step two: a field splitting processing unit: splitting the received operation data according to the type of the monitoring signal, and sending the operation data of the same type to a next-stage data processing unit;
step three: a data statistics processing unit: the calculation of data indexes such as the maximum value, the average value and the like of the data is realized in the unit time window, and the calculation result is sent to the next-stage data processing unit for data aggregation of the whole time window;
step four: frequency count calculation processing unit: the method mainly aims to realize running data service needing frequency statistics;
step five: a threshold value judgment processing unit: according to the type of each monitoring data, combining the data characteristics and related research, and making a strategy for judging the threshold value;
step six: a summary calculation processing unit: and combining and counting the data of all the basic time windows on the processing unit, and using a global summary data aggregation mode, namely sending all the data to the same processing unit for final calculation.
The flow processing is essentially to consider the operation data as a continuous data flow, the calculation for the data is to calculate a single data point, namely, to process the data once when the data is new, although the method can ensure the timeliness of the operation data processing, the relevance of the connected data is cut off to a certain extent, and the relevance of the operation data is very important for the prediction and analysis of the machine tool state, so that the processing method of introducing the sliding time window is necessary for processing the operation data
S7: after the spit and the Bolt are finished, setting the data flow direction and the grouping mode of each component in Topology of Storm;
s8: and the on-line judgment of the workshop production field state is realized in a Storm real-time processing frame through a real-time flow clustering algorithm, the corresponding data in the abnormal state is identified and compared with the corresponding abnormal state data in the database according to the judged data in the abnormal state, then an abnormal record is output and stored in the database, and the result is displayed in a front-end interface.
Taking an assembly test of the power battery as an example, the specific implementation mode is as follows:
firstly, a user has different operation authorities according to the authority of an account, after the identity is successfully verified, the user logs in a system, the system is initialized, operation data are accessed to a big data platform through a Kafka message queue according to the processing requirement of the data, data preprocessing operation is carried out, then data stream processing is carried out, the online judgment of the workshop production field state is realized through a real-time stream clustering algorithm in a Storm real-time processing frame, the corresponding data in the judged abnormal state are identified and compared with the corresponding abnormal state data in a database, then an abnormal record is output and stored in the database.
The specific detection and processing procedures after logging in the system are as follows:
the first step is as follows: cell input; the second step is that: mapping code combination; the third step: laser welding; the fourth step: testing internal resistance/insulation withstand voltage; the fifth step: and (4) functional testing.
The monitoring method is characterized in that each stage of a product process is collected, tested and analyzed based on a sensing technology and an automatic identification technology, and the effectiveness of monitoring the abnormal state of the product of the production line based on the workshop field operation data flow is verified. The data acquisition system adopts sensors such as a daily position sensor, an Agilent sensor and a Mitsubishi sensor, after acquired data are preprocessed, a data processing result is accessed into a Kafka message queue, and the models and detection indexes of part of data acquisition devices are shown in FIG. 6.
And then, carrying out operation data acquisition and storage technology test. In different procedures of a workshop production line, a plurality of types of sensors are configured, wherein a daily BT3563 judges whether a battery cell is qualified or not by detecting indexes such as open-circuit voltage, internal resistance and the like, and in each procedure of battery processing production, the monitored and collected indexes comprise product mass operation data such as low-impedance detected resistance, total voltage, total internal resistance, monomer temperature, onboard temperature and the like of a function test product. By building a distributed cluster environment, the collected data is calculated under the condition that the distributed cluster has no other tasks, and the operation result is analyzed. Wherein kafka is used as a message queue; and Storm is used as a distributed operation data flow processing framework, so that real-time and stable monitoring on the limited production data of the workshop is realized. By comparing the front-end query result with the running data of the background database, it can be found that: the designed database can finish the storage of the running data, and the problems of database crash, buffer zone fullness and the like do not occur in the storage process; the data query result is not lost or repeated during operation, the query speed is high, and the data query requirement can be basically met.
And finally, carrying out real-time stream processing technology test and analysis. In the stream processing technology and the application case of the field production monitoring system in the test workshop, the distributed database is selected for real-time storage, the Ajax technology is used for real-time refreshing of operation data, the interface content of a webpage end is updated in real time, and real-time analysis, processing and uploading of the data are realized. The case tests the application effect and performance of the abnormal state based on the flow processing technology. And taking the function completion degree and the product qualification rate of the system as test standards. The specific tests and analyses were as follows:
firstly, the abnormal state monitoring test process based on the stream processing technology is realized by respectively registering nodes of Kafka and Storm on a distributed server management system Zookeeper and uniformly managing the server nodes of Kafka and Storm. In order to ensure that monitoring data can be processed in real time, data caching is carried out through a Kafka distributed message queue, a real-time big data computing platform with a Storm deployed serves as a data consuming end of the Kafka, an integrated Kafka Spout of the Kafka and the Storm serves as a data source connected with the message queue and the big data platform, and the data source is transmitted to a next-level Bolt in a data carrier mode to be analyzed and processed. After the data source Kafkaspout receives the operation data, the Bolt is firstly split through the data to obtain the data of different operation data on different acquisition channels, and the data is independently programmed in the Bolt by combining with the specific operation data monitoring requirement, so that the pretreatment of the operation data, the data statistics based on the sliding time window, the data abnormity monitoring and the like are realized. After logic writing of each Spout and Bolt is completed, setting data flow direction and grouping mode of each component in Storm's Topology;
then, performing a flow processing technology function test, wherein in the actual production process of a workshop, the first procedure is Cell investment, the procedure is mainly to judge whether the test index range is met or not by detecting the attenuation index of the open-circuit voltage and measuring the actual voltage, and at the moment, if the electric core is not qualified, the electric core needs to be returned to a supplier; real-time, efficient and stable detection of the index is very important, so that a storm-based stream processing technology is introduced to process and analyze the acquired data in real time, and the processing delay is obviously reduced.
The real-time processing technology of machine tool running data based on stream processing mainly realizes data preprocessing, data statistics, data anomaly detection and real-time running state judgment in a monitoring system. The data preprocessing and the data statistics are mainly used for completing data processing in a background, mainly displaying data abnormity detection and running state judgment results in a front-end interface, and displaying the results in an abnormity management module. And respectively setting the minimum value and the maximum value of the open-circuit voltage to 3.294V and 3.299V according to the cell parameters and the product requirements. Directly carrying out threshold value monitoring to easily generate a false alarm phenomenon due to the reasons that a measured voltage attenuation signal is easily interfered by the outside, the instantaneous change characteristic is large, the measured value is accurate and the like, counting the measured mean value and the maximum minimum value of the latest 5 seconds per second according to a data critical anomaly detection strategy based on a sliding time window, and outputting NG (specified number) if the probability that the actual value of the open-circuit voltage of the battery cell exceeds the alarm threshold value is more than 80 percent, wherein the NG is unqualified and the result is displayed in a front-end interface; if the value calculated and analyzed in real time by stream processing calculation is within the range specified by the process, the value is judged to be qualified, and 'OK' is fed back on the front-end interface. In the process, by introducing the sliding time window concept based on the flow processing, the detection stability and efficiency are improved, the quality false detection rate of the battery cell is prevented, and the detection and production efficiency of a workshop factory is improved. FIG. 7 is a diagram of real-time inspection information for the front-end inspection interface.
Real-time refreshing of running data and database access are achieved through the Ajax technology, and writing of a front-end page is completed through the Html5 and the CSS style. In the test case, the cell open-circuit voltage and the welding current for monitoring the welding quality are used as test objects. The detection processing delay meets the requirements of a production field, does not influence actual processing and production, can load performance test results and detection indexes of all link processes by using a web browser, and can complete dynamic adjustment and modification of the detection indexes at a web end. In the production process of each process, the monitoring record item interface of each processing process is shown in fig. 8.
In the description herein, references to the description of "one embodiment," "an example," "a specific example" or the like are intended to mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The preferred embodiments of the invention disclosed above are intended to be illustrative only. The preferred embodiments are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, to thereby enable others skilled in the art to best utilize the invention. The invention is limited only by the claims and their full scope and equivalents.
Claims (7)
1. A real-time monitoring and anomaly detection method for workshop field data based on stream processing is characterized by comprising the following steps:
s1: the workshop production operation data acquisition equipment acquires data and generates a data stream;
s2: respectively registering nodes of Kafka and Storm on a distributed server management system Zookeeper, and uniformly managing the server nodes of Kafka and Storm;
s3: performing data caching through a Kafka distributed message queue, taking a real-time big data computing platform with a Storm deployed as a data consumption end of the Kafka, adopting an integrated Kafka Spout of the Kafka and the Storm as a data source connected with the message queue and the big data platform, and transmitting the data source to the next-level Bolt in the form of a data carrier for analysis and processing;
s4: after the data source Kafkaspout receives the operation data, firstly splitting Bolt through the data to obtain the data of different operation data on different acquisition channels;
s5: performing data preprocessing in Bolt alone;
s6: then, performing running data statistics and anomaly detection based on the sliding time window, setting the sliding window to be 0.5min, setting the basic time window to be 5s, performing independent calculation tasks under each basic time window, sliding the calculation tasks within 0.5min by using the sliding window of 5s at the moment, finally merging the basic time windows, and performing statistics on the calculation result of the whole calculation window;
s7: after the spit and the Bolt are finished, setting the data flow direction and the grouping mode of each component in Topology of Storm;
s8: and realizing on-line judgment of the workshop production field state in a Storm real-time processing frame through a real-time flow clustering algorithm, identifying and comparing the corresponding data in the judged abnormal state with the corresponding abnormal state data in the database, outputting an abnormal record, and storing the abnormal record in the database.
2. The flow processing-based workshop field data real-time monitoring and anomaly detection method according to claim 1, characterized in that: the data preprocessing step in step S5 includes data cleaning, data formatting, and determining whether data needs to be stored.
3. The flow processing-based workshop field data real-time monitoring and anomaly detection method according to claim 1, characterized in that: the operation data statistics in step S6 refers to statistics of the maximum, minimum, average, frequency of occurrence, and energy utilization of the operation data, and the anomaly detection mainly includes real-time critical detection and anomaly monitoring based on a sliding time window.
4. The flow processing-based workshop field data real-time monitoring and anomaly detection method according to claim 1, characterized in that: the specific steps and units of the running data statistics and anomaly detection based on the sliding time window in step S6 include:
the method comprises the following steps: a sliding window processing unit: firstly, setting time and parameters of data transmission required by a sliding window, wherein the parameters mainly comprise unit window length and sliding window length;
step two: a field splitting processing unit: splitting the received operation data according to the type of the monitoring signal, and sending the operation data of the same type to a next-stage data processing unit;
step three: a data statistics processing unit: the data indexes of the maximum value and the average value of the data are calculated in the unit time window, and the calculation result is sent to the next-stage data processing unit to carry out data aggregation of the whole time window;
step four: frequency count calculation processing unit: the method mainly aims to realize running data service needing frequency statistics;
step five: a threshold value judgment processing unit: according to the type of each monitoring data, combining the data characteristics and related research, and making a strategy for judging the threshold value;
step six: a summary calculation processing unit: and combining and counting the data of all the basic time windows on the processing unit, and using a global summary data aggregation mode, namely sending all the data to the same processing unit for final calculation.
5. The flow processing-based workshop field data real-time monitoring and anomaly detection method according to claim 2, characterized in that: the specific mode of data cleaning is as follows: firstly, checking whether the data is qualified, if so, continuing to process the subsequent data, if not, executing data cleaning and filtering operation, and deleting the data with null value, outlier and obvious error information from the monitoring data.
6. The flow processing-based workshop field data real-time monitoring and anomaly detection method according to claim 2, characterized in that: the data formatting comprises data calibration and redundant value deletion, and the specific mode is as follows: the data is calibrated and formatted to convert analog values to real values while eliminating redundant information such as data headers, headers and other fields from the data stored by the sensor and acquisition system.
7. The flow processing-based workshop field data real-time monitoring and anomaly detection method according to claim 2, characterized in that: the judgment basis for judging whether the data needs to be stored is that whether the data needs to be processed first and then stored in the database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911387816.3A CN111143438B (en) | 2019-12-30 | 2019-12-30 | Workshop field data real-time monitoring and anomaly detection method based on stream processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911387816.3A CN111143438B (en) | 2019-12-30 | 2019-12-30 | Workshop field data real-time monitoring and anomaly detection method based on stream processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111143438A true CN111143438A (en) | 2020-05-12 |
| CN111143438B CN111143438B (en) | 2023-09-12 |
Family
ID=70521564
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911387816.3A Active CN111143438B (en) | 2019-12-30 | 2019-12-30 | Workshop field data real-time monitoring and anomaly detection method based on stream processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111143438B (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112000858A (en) * | 2020-08-17 | 2020-11-27 | 浙江鸿城科技有限责任公司 | Workshop data anomaly identification method |
| CN112699113A (en) * | 2021-01-12 | 2021-04-23 | 上海交通大学 | Industrial manufacturing process operation monitoring system driven by time sequence data stream |
| CN113325824A (en) * | 2021-06-02 | 2021-08-31 | 三门核电有限公司 | Regulating valve abnormity identification method and system based on threshold monitoring |
| CN113486063A (en) * | 2021-07-05 | 2021-10-08 | 国网河北省电力有限公司信息通信分公司 | Method and device for processing flow data in power internet of things and terminal equipment |
| CN113762163A (en) * | 2021-09-09 | 2021-12-07 | 杭州澳亚生物技术股份有限公司 | GMP workshop intelligent monitoring management method and system |
| CN113760989A (en) * | 2021-02-04 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Method, device and equipment for processing unbounded stream data and storage medium |
| CN113993001A (en) * | 2021-09-08 | 2022-01-28 | 四创电子股份有限公司 | Real-time streaming analysis alarm method based on sliding data window |
| WO2023050229A1 (en) * | 2021-09-29 | 2023-04-06 | 西门子股份公司 | Industrial data processing method and apparatus, electronic device, and storage medium |
| CN117331921A (en) * | 2023-09-28 | 2024-01-02 | 石家庄铁道大学 | Bearing monitoring multisource data processing method |
| CN117520011A (en) * | 2023-11-09 | 2024-02-06 | 中国三峡建工(集团)有限公司 | Sharing management method and system for real-time data memory |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107968840A (en) * | 2017-12-15 | 2018-04-27 | 华北电力大学（保定） | A kind of extensive power equipment monitoring, alarming Real-time Data Processing Method and system |
| US10129118B1 (en) * | 2016-03-29 | 2018-11-13 | Amazon Technologies, Inc. | Real time anomaly detection for data streams |
| CN110347116A (en) * | 2019-07-17 | 2019-10-18 | 重庆大学 | A kind of conditions of machine tool monitoring system and monitoring method based on operation data stream |
-
2019
- 2019-12-30 CN CN201911387816.3A patent/CN111143438B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10129118B1 (en) * | 2016-03-29 | 2018-11-13 | Amazon Technologies, Inc. | Real time anomaly detection for data streams |
| CN107968840A (en) * | 2017-12-15 | 2018-04-27 | 华北电力大学（保定） | A kind of extensive power equipment monitoring, alarming Real-time Data Processing Method and system |
| CN110347116A (en) * | 2019-07-17 | 2019-10-18 | 重庆大学 | A kind of conditions of machine tool monitoring system and monitoring method based on operation data stream |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112000858A (en) * | 2020-08-17 | 2020-11-27 | 浙江鸿城科技有限责任公司 | Workshop data anomaly identification method |
| CN112000858B (en) * | 2020-08-17 | 2024-04-05 | 浙江鸿城科技有限责任公司 | Workshop data anomaly identification method |
| CN112699113A (en) * | 2021-01-12 | 2021-04-23 | 上海交通大学 | Industrial manufacturing process operation monitoring system driven by time sequence data stream |
| CN113760989A (en) * | 2021-02-04 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Method, device and equipment for processing unbounded stream data and storage medium |
| CN113325824A (en) * | 2021-06-02 | 2021-08-31 | 三门核电有限公司 | Regulating valve abnormity identification method and system based on threshold monitoring |
| CN113486063A (en) * | 2021-07-05 | 2021-10-08 | 国网河北省电力有限公司信息通信分公司 | Method and device for processing flow data in power internet of things and terminal equipment |
| CN113993001A (en) * | 2021-09-08 | 2022-01-28 | 四创电子股份有限公司 | Real-time streaming analysis alarm method based on sliding data window |
| CN113993001B (en) * | 2021-09-08 | 2024-04-12 | 四创电子股份有限公司 | Real-time stream analysis alarm method based on sliding data window |
| CN113762163A (en) * | 2021-09-09 | 2021-12-07 | 杭州澳亚生物技术股份有限公司 | GMP workshop intelligent monitoring management method and system |
| WO2023050229A1 (en) * | 2021-09-29 | 2023-04-06 | 西门子股份公司 | Industrial data processing method and apparatus, electronic device, and storage medium |
| CN117331921A (en) * | 2023-09-28 | 2024-01-02 | 石家庄铁道大学 | Bearing monitoring multisource data processing method |
| CN117520011A (en) * | 2023-11-09 | 2024-02-06 | 中国三峡建工(集团)有限公司 | Sharing management method and system for real-time data memory |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111143438B (en) | 2023-09-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111143438B (en) | Workshop field data real-time monitoring and anomaly detection method based on stream processing | |
| CN110347116B (en) | Machine tool state monitoring system and monitoring method based on operation data flow | |
| CN104765322B (en) | Production line monitoring system | |
| WO2022174679A1 (en) | Method and apparatus for predicting voltage inconsistency fault of battery cells, and server | |
| CN115186883A (en) | Industrial equipment health state monitoring system and method based on Bian Yun collaborative computing | |
| CN113177646B (en) | Power distribution equipment online monitoring method and system based on self-adaptive edge proxy | |
| Qiu et al. | Combine harvester remote monitoring system based on multi-source information fusion | |
| CN112987696A (en) | Regional power distribution network equipment management platform and operation method thereof | |
| CN111627199A (en) | Hydropower station dam safety monitoring system and monitoring method | |
| CN112580858A (en) | Equipment parameter prediction analysis method and system | |
| CN116108402A (en) | Method, equipment and storage medium based on electric power multi-source heterogeneous data fusion analysis | |
| CN111123873B (en) | Production data acquisition method and system based on stream processing technology | |
| CN117560300A (en) | Intelligent internet of things flow prediction and optimization system | |
| CN117572837B (en) | Intelligent power plant AI active operation and maintenance method and system | |
| CN207992717U (en) | A kind of gate of hydropower station on-line condition monitoring system | |
| CN117614124A (en) | Electric power information acquisition system based on dual-mode communication | |
| CN111314110B (en) | Fault early warning method for distributed system | |
| CN115114124A (en) | Host risk assessment method and device | |
| Li et al. | Meteorological radar fault diagnosis based on deep learning | |
| CN116862109A (en) | Regional carbon emission situation awareness early warning method | |
| CN115469585B (en) | Hydropower unit state monitoring method and system based on big data | |
| CN109687584B (en) | Power transmission internet of things communication network access optimization method | |
| CN114165430B (en) | Edge-computing-based computer pump health monitoring method, system, equipment and medium | |
| CN203825457U (en) | Cigarette making and tipping machine set control system based on K-BUS bus | |
| CN112820090A (en) | Alarm system based on big data and implementation method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |