CN111221807A - Cloud service-oriented industrial equipment big data quality testing method and architecture - Google Patents
Cloud service-oriented industrial equipment big data quality testing method and architecture Download PDFInfo
- Publication number
- CN111221807A CN111221807A CN201911359368.6A CN201911359368A CN111221807A CN 111221807 A CN111221807 A CN 111221807A CN 201911359368 A CN201911359368 A CN 201911359368A CN 111221807 A CN111221807 A CN 111221807A
- Authority
- CN
- China
- Prior art keywords
- data
- monitoring terminal
- detection monitoring
- comparison
- cloud platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Computer Security & Cryptography (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Testing Or Calibration Of Command Recording Devices (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a cloud service-oriented industrial equipment big data quality testing method and a cloud service-oriented industrial equipment big data quality testing framework. The cloud platform acquires industrial data, and performs consistency comparison on the industrial data to acquire a consistency comparison result. The cloud platform acquires the data stored in the database and the cache, and performs integrity comparison on the data in the cache and the data in the database to acquire an integrity comparison result. The method is characterized in that the flow direction of industrial data is used as a main line, according to the data flow direction of generation, transmission and storage of industrial equipment detection big data, the data detected and monitored by the industrial equipment are subjected to quality test by respectively adopting accuracy comparison, consistency comparison and integrity comparison, the data quality of the industrial equipment detection and monitoring big data on the cloud platform is ensured, and effective data support is further provided for the diagnosis and prediction function of the industrial equipment cloud service platform.
Description
Technical Field
The invention relates to the field of industrial equipment, in particular to a cloud service-oriented method and a cloud service-oriented architecture for testing the quality of large data of industrial equipment.
Background
The detection monitoring data based on the intelligent sensor is an important data source for industrial big data collection and is also a foundation for industrial big data analysis. The data quality may be problematic due to subjective or objective reasons such as detecting multi-source heterogeneity of monitoring data, acquiring terminal abnormality, communication protocols, link problems, design defects, and the like. The high-quality effective data is a precondition for the large industrial data to play a role, and only if the data is high-quality effective, potential and useful information can be mined from the data, so that a diagnosis and prediction function is provided for in-service industrial equipment, and intelligent evaluation service can be provided for enterprises better.
Hadoop is the most popular big data processing architecture at present, and the most core design of the architecture is as follows: HDFS and MapReduce. HDFS provides storage for massive data, while MapReduce provides computation for massive data. The major big data tests at present are also functional tests and non-functional tests that are spread around the Hadoop framework.
A data script or a segment of data calculation logic operates correctly under big data on the premise that the function is correct, so the work of the functionality test is to ensure the correctness of the data and the correctness of the service logic. The big data script also has input and output, which is similar to background logic test in function test, and has no interface, all the data is processed by a background server, a tester must know the whole processing flow, the circulation of each data, the input and the output of each step, and then can judge whether the final output result is correct, and the same is true for big data test, and we need to know the function of each script, the input and the output of each script, and the whole data circulation process, so as to judge whether the function realized by the big data is correct.
Due to the application of big data to specific industries, besides functional tests, non-functional tests need to be performed under the whole big data processing framework, and the following are performed:
a. and (3) performance testing: the performance is the most critical dimension for evaluating a big data analysis system, and the big data analysis system performance mainly comprises a plurality of indexes such as throughput, task completion time and memory utilization rate, and can reflect the performances such as processing capacity and resource utilization capacity of a big data analysis platform.
b. Testing fault tolerance: it can recover automatically from partial failure without verifying that it affects overall performance, and in particular, when a failure occurs, the big data analysis system should continue to operate in an acceptable manner while recovering, and to some extent, when an error occurs.
c. And (3) usability testing: high availability of big data has been one of the indispensable characteristics of big data analytics to guarantee the continuity of data application traffic.
d. And (3) testing expansibility: the elastic expansion capability is particularly important for the file system in the big data era, and the file system expansibility test mainly comprises testing the elastic expansion capability (expansion/retraction) of the system and performance influence brought by the expansion system, verifying whether the system has linear expansion capability or not, and mainly taking manual test as a main test.
e. And (3) stability testing: the big data analysis system usually runs continuously for a long time, and the importance of stability is self-evident, and the stability test mainly verifies whether the system can still run normally and functions normally under the permission of a long time (7/30/180/365 × 24).
f. Testing a deployment mode: the big data has the characteristic of scale-out, and can construct a large-scale and high-performance file system cluster. For different applications and solutions, file system deployment modes are significantly different, and the solutions and specific deployments are generally designed according to application scenarios and then manually tested.
h. And (3) pressure testing: the pressure test is to verify whether the system can still normally run, whether the function is normal and the system resource consumption condition under the large pressure of the system construction, including data multi-client, high OPS pressure, high IOPS/throughput pressure, thereby providing a basis for the large data operation.
The above-described common functional and non-functional tests are basically tests on the system side without problems with default data. Whereas for big data analysis, the stored data is first valid. In the face of complex information flow, data are often incomplete and even have quality problems, and especially, the diagnosis and evaluation of industrial equipment mainly carries out analysis and processing according to detection and monitoring data, so that more valuable information is obtained, the usability and diversity of data sources are improved, the data quality of an acquisition end and a data warehouse becomes a very important factor, and the data flow becomes non-negligible.
Disclosure of Invention
Technical problem to be solved
The invention provides a cloud service-oriented method and a cloud service-oriented architecture for testing the quality of large data of industrial equipment, and aims to solve the problems that the accuracy of data acquired by a detection and monitoring terminal is low and the quality of the data is not tested in the production, transmission and storage processes of industrial data.
(II) technical scheme
The invention provides a cloud service-oriented method for testing the big data quality of industrial equipment, which is characterized by comprising the following steps:
s1, carrying out accuracy test on the detection monitoring terminal to obtain accuracy, adjusting parameters of the detection monitoring terminal according to the value of the accuracy to obtain a qualified detection monitoring terminal, and obtaining industrial data by using the qualified detection monitoring terminal;
s2, connecting the qualified detection monitoring terminal with a cloud platform, uploading the acquired industrial data to the cloud platform by the qualified detection monitoring terminal, and carrying out consistency comparison on the acquired industrial data by the cloud platform to acquire a consistency comparison result;
and S3, connecting the cloud platform to the database and the cache, wherein the cloud platform acquires the data stored in the database and the cache, and performs integrity comparison between the data stored in the cache and the data stored in the database to acquire an integrity comparison result.
Preferably, the performing accuracy test on the detection monitoring terminal to obtain accuracy, adjusting parameters of the detection monitoring terminal according to the value of the accuracy, and obtaining the qualified detection monitoring terminal includes:
s01, setting a corresponding reference piece according to a detection monitoring terminal to be tested, wherein the reference piece generates standard data in real time;
s02, the detection monitoring terminal acquires the standard data by using a sensor arranged on the reference piece, the standard data acquired by the detection monitoring terminal is used as test data, and the test data is compared with the standard data in accuracy to acquire accuracy;
s03, the detection monitoring terminal judges whether the accuracy value reaches a set value; if so, taking the detection monitoring terminal as the qualified detection monitoring terminal; if not, the parameters of the detection monitoring terminal are corrected by the correction function, and the steps S02 and S03 are repeated.
Preferably, the accuracy comparison specifically includes:
s11, respectively reading the data in the standard data and the test data at the same time interval into a memory space, and performing denoising pretreatment on the two groups of data in the memory space;
s12, respectively calculating peak positions of the two groups of data subjected to denoising preprocessing, and shifting the peak positions of the test data to the peak positions of the standard data;
s13, respectively stretching windows leftwards and rightwards by adopting a mean interpolation method to the test data with the peak position of the standard data as a middle point, and performing window difference processing on the stretched test data and the standard data;
s14, calculating the accuracy value of the test data by calculating the percentage of the window difference in the area of the standard data.
Preferably, the consistency comparison comprises a data consistency comparison and a protocol consistency comparison.
Preferably, the data consistency comparison comprises:
s211, the qualified detection monitoring terminal is in communication connection with a cloud platform according to a set protocol, and the cloud platform sends a test request to the qualified detection monitoring terminal according to set time;
s212, the qualified detection monitoring terminal acquires and encapsulates the industrial data to obtain encapsulated data, and the industrial data before encapsulation is backed up to a set file;
s213, the cloud platform obtains the encapsulation data, analyzes the encapsulation data to obtain analysis data, and compares the analysis data with the consistency of the data content of the industrial data in the setting file to obtain a data consistency comparison value.
Preferably, the comparing the consistency of the analytical data and the data content of the industrial data in the setting file comprises:
s2131, comparing the data content of the data items of the analyzed data with the data items of data stored in a database in advance;
s2132, identifying and recording the number of data items with different data contents in the analysis data and the data stored in the database to obtain the wrong data volume of the data contents, and obtaining the correct data volume of the data contents according to the wrong data volume of the data contents;
and S2133, comparing the data quantity with correct data content with the data quantity of the analyzed data to obtain a data consistency comparison value.
Preferably, the protocol consistency comparison comprises:
s221, when the qualified detection monitoring terminal receives a test request, comparing the data item of the analysis data with the data item of the data stored in advance in a database in a data format;
s222, identifying and recording the number of data items with different data formats in the analysis data and the data stored in the database to obtain data quantity with wrong data formats, and obtaining data quantity with correct data formats according to the data quantity with wrong data formats;
and S223, comparing the data quantity with the correct data format with the data quantity of the analysis data to obtain a protocol consistency comparison value.
Preferably, the integrity comparison comprises:
s31, the cloud platform stores the analysis data into a database according to a storage strategy, and simultaneously stores the analysis data into a cache;
s32, the cloud platform acquires a starting instruction requiring integrity test, acquires corresponding data in the database according to the data identification of the analysis data stored in the cache, and then performs integrity comparison on the analysis data in the cache and the data items of the data acquired from the database to obtain an integrity comparison value, wherein the data identification comprises a device number and a time stamp.
Further, the invention also discloses a cloud service-oriented industrial equipment big data quality testing architecture, which comprises:
the detection monitoring terminal is used for acquiring the industrial data and packaging and sending the industrial data;
the cloud platform is used for receiving and analyzing the packaging data, obtaining the analysis data and storing the analysis data into a database;
and the data comparison module is used for comparing the content of the data and the format of the data to obtain a comparison result.
Preferably, the data comparison module comprises:
the accuracy comparison module is used for carrying out accuracy test on the detection monitoring terminal to obtain accuracy, adjusting parameters of the detection monitoring terminal according to the numerical value of the accuracy to obtain a qualified detection monitoring terminal, and acquiring industrial data by using the qualified detection monitoring terminal;
the consistency comparison module is used for connecting the qualified detection monitoring terminal with a cloud platform, the qualified detection monitoring terminal uploads the obtained industrial data to the cloud platform, and the cloud platform performs consistency comparison on the obtained industrial data to obtain a consistency comparison result;
and the integrity comparison module is used for connecting the cloud platform to the database and the cache, acquiring the data stored in the database and the cache by the cloud platform, and performing integrity comparison on the data stored in the cache and the data stored in the database to acquire an integrity comparison result.
(III) advantageous effects
The invention has the beneficial effects that: the method has the advantages that the flow direction of industrial data is taken as a main line, the quality of the data detected and monitored by the industrial equipment is tested by respectively adopting accuracy comparison, consistency comparison and integrity comparison according to the data flow direction of the generation, transmission and storage of the large data detected and monitored by the industrial equipment, an improved and perfect accurate target is provided for a detection monitoring terminal and a cloud platform, the data quality of the large data detected and monitored by the industrial equipment on the cloud platform is improved, and effective data support is further provided for the diagnosis and prediction function of the cloud service platform of the industrial equipment.
Drawings
Fig. 1 is a flowchart of a method for testing big data quality of industrial equipment facing cloud services according to the present invention;
FIG. 2 is a flowchart of a method for obtaining a qualified detection monitoring terminal in the cloud service-oriented industrial equipment big data quality testing method of the present invention;
FIG. 3 is a flowchart of a method for comparing accuracy in the method for testing the big data quality of the cloud service-oriented industrial equipment;
FIG. 4 is a flowchart of a method for comparing data consistency in the cloud service-oriented industrial equipment big data quality testing method of the present invention;
FIG. 5 is a flowchart of a specific method for data consistency comparison in the cloud service-oriented method for testing the big data quality of the industrial equipment;
FIG. 6 is a flowchart of a protocol consistency comparison method in the cloud service-oriented industrial equipment big data quality testing method of the present invention;
FIG. 7 is a flowchart of an integrity comparison method in the cloud service-oriented industrial equipment big data quality testing method of the present invention;
FIG. 8 is a connection block diagram of a cloud service oriented industrial equipment big data quality testing architecture of the present invention;
FIG. 9 is a connection block diagram of an accuracy comparison module in the cloud service-oriented industrial equipment big data quality test architecture of the present invention;
FIG. 10 is a connection block diagram of a consistency comparison module in the test architecture of the cloud service-oriented industrial equipment big data quality;
fig. 11 is a connection block diagram of an integrity comparison module in the cloud service-oriented industrial equipment big data quality test architecture of the present invention.
Detailed Description
For the purpose of better explaining the present disclosure, and to facilitate understanding thereof, the present disclosure will be described in detail below by way of specific embodiments with reference to the accompanying drawings.
All technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein in the description of the disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.
Fig. 1 is a flowchart of a method for testing big data quality of industrial equipment facing cloud services according to an embodiment of the present disclosure, and as shown in fig. 1, fig. 1 is a flowchart of a method for testing big data quality of industrial equipment facing cloud services according to the present disclosure, and the method includes the following steps:
in step S1, performing accuracy test on the detection monitoring terminal to obtain accuracy, adjusting parameters of the detection monitoring terminal according to the value of the accuracy to obtain a qualified detection monitoring terminal, and acquiring industrial data by using the qualified detection monitoring terminal;
in step S2, connecting the qualified detection monitoring terminal with the cloud platform, uploading the acquired industrial data to the cloud platform by the qualified detection monitoring terminal, and performing consistency comparison on the acquired industrial data by the cloud platform to acquire a consistency comparison result;
in step S3, the cloud platform is connected to the database and the cache, and the cloud platform obtains the data stored in the database and the cache, and performs integrity comparison between the data stored in the cache and the data stored in the database, so as to obtain an integrity comparison result.
Before step S1, a qualified detection monitoring terminal is further obtained, as shown in fig. 2, fig. 2 is a flowchart of a method for obtaining a qualified detection monitoring terminal in the cloud service-oriented industrial equipment big data quality testing method of the present invention.
In step S01, a corresponding reference is set according to the detection monitoring terminal to be tested, and the reference generates standard data in real time;
in step S02, the detection monitoring terminal acquires standard data by using a sensor disposed on the reference member, uses the standard data acquired by the detection monitoring terminal as test data, and compares the test data with the standard data to acquire accuracy;
in step S03, the detection monitoring terminal determines whether the numerical value of the accuracy reaches a set value, and the set value is preferably 97% or more; if so, taking the detection monitoring terminal as a qualified detection monitoring terminal; if not, the parameters of the detection monitoring terminal are corrected by the correction function, and the steps S02 and S03 are repeated.
The specific implementation of the steps of the embodiment shown in fig. 1 is described in detail below:
as shown in fig. 3, fig. 3 is a flowchart illustrating an accuracy comparison of the cloud service-oriented industrial equipment big data quality testing method according to the present invention, and the accuracy comparison in step S1 specifically includes the following steps:
step S11, respectively reading the data in the same time period in the standard data and the test data into a memory space, and performing denoising pretreatment on the two groups of data in the memory space;
step S12, respectively calculating peak positions of the two groups of data after denoising pretreatment, and shifting the peak positions of the test data to the peak positions of the standard data;
step S13, adopting a mean interpolation method to respectively stretch windows of the test data leftwards and rightwards by taking the peak position of the standard data as a middle point, and carrying out window difference processing on the stretched test data and the standard data;
and step S14, calculating the accuracy value of the test data by calculating the percentage of the window difference in the area of the standard data.
Further, the consistency comparison includes not only the comparison of data consistency but also the comparison of protocol consistency. When the qualification testing monitoring terminal accesses the cloud platform, besides connection configuration, protocol agreement is required, and a data format is stored in a database in a data item mode. The cloud platform compares the received data with the data format of the data stored in the database each time the data is received, for example: the third item of data identified in the database is boolean data (0/1), and the third item of data analyzed is T, which is inconsistent.
Fig. 4 is a flowchart of a method for comparing data consistency in the cloud service-oriented industrial equipment big data quality test method of the present invention, and the method includes the following steps:
s211, performing communication connection between the qualified detection monitoring terminal and a cloud platform according to a set protocol, and sending a test request to the qualified detection monitoring terminal by the cloud platform according to set time;
s212, the qualified detection monitoring terminal acquires industrial data and encapsulates the industrial data to obtain encapsulated data, the industrial data before encapsulation is backed up into a set file, and the file is sent to a cloud platform through an ftp (file transfer protocol);
step S213, the cloud platform obtains the encapsulation data and analyzes the encapsulation data to obtain analysis data, and compares the consistency of the data content of the analysis data and the industrial data in the setting file to obtain a data consistency comparison value.
It should be noted that each piece of data has a unique device number and a timestamp as a unique identifier, and the data in the transmission file and the parsed data are associated with each other by the timestamp.
As shown in fig. 5, fig. 5 is a flowchart of a specific method for comparing data consistency in the cloud service-oriented industrial device big data quality testing method of the present invention, and further supplementary description is made on the method for comparing consistency of data content in step S213, which specifically includes the following steps:
step 2131, comparing the data content of the data items of the analyzed data with the data items of the data pre-stored in the database;
step 2132, identifying and recording the number of data items with different analytical data and data contents in the data stored in the database to obtain the data volume with wrong data contents, and obtaining the data volume with correct data contents according to the wrong data volume of the data contents;
step 2133, comparing the data quantity with correct data content with the data quantity of the analyzed data to obtain a data consistency comparison value.
As shown in fig. 6, fig. 6 is a flowchart of a method for comparing protocol consistency in the cloud service-oriented industrial equipment big data quality testing method of the present invention, and the method includes the following steps:
step S221, when the qualification testing monitoring terminal receives the testing request, comparing the data format of the data item of the analyzed data with the data item of the data pre-stored in the database;
step S222, identifying and recording the number of data items with different data formats in the analysis data and the data stored in the database to obtain the data volume with wrong data formats, and obtaining the data volume with correct data formats according to the data volume with wrong data formats;
and step S223, comparing the data quantity with the correct data format with the data quantity of the analysis data to obtain a protocol consistency comparison value.
When the platform is built, the requirement of integrity test is considered, corresponding functions are built for data storage operation and query operation of each type of detection and monitoring terminal, the functions are maintained in a database, and data of a certain device with a certain time stamp in the database can be obtained through the query functions. The cloud platform receives the analyzed data, directly stores the analyzed data in the cache, and circularly accumulates and presses the analyzed data, so that the earliest data can be flushed by the latest data.
The integrity test is an event actively initiated by an interface button, and after clicking, integrity comparison is performed. As shown in fig. 7, fig. 7 is a flowchart of a method for comparing integrity in the method for testing the quality of the cloud service-oriented industrial device big data, and the method specifically includes the following steps:
step S31, the cloud platform stores the analysis data into a database according to a storage strategy, and simultaneously stores the analysis data into a cache;
step S32, the cloud platform acquires a starting instruction requiring integrity test, acquires corresponding data in the database according to the data identification of the analysis data stored in the cache, and then performs integrity comparison on the analysis data in the cache and the data items of the data acquired from the database to obtain an integrity comparison value, wherein the data identification comprises a device number and a time stamp.
Integrity comparison firstly verifies the integrity of the data item, secondly verifies the integrity of the data itself, and finally calculates the value giving the integrity comparison.
According to the method, the data quality is comprehensively and effectively managed by comparing the three types of data, the accuracy of detecting the large data quality by the industrial equipment is ensured, and effective data support is further provided for the diagnosis and prediction function of the cloud service platform of the industrial equipment.
Further, the present invention also provides a cloud service-oriented industrial device big data quality test architecture, as shown in fig. 8, fig. 8 is a connection block diagram of the cloud service-oriented industrial device big data quality test architecture of the present invention, and the test architecture includes a detection monitoring terminal, a cloud platform, and a data comparison module. The detection monitoring terminal is used for acquiring industrial data, and packaging and sending the industrial data; the cloud platform is used for receiving and analyzing the packaging data to obtain analysis data and storing the analysis data into the database; the data comparison module is used for comparing the content of the data and the format of the data to obtain a comparison result.
Preferably, the data comparison module comprises an accuracy comparison module, a consistency comparison module and an integrity comparison module.
As shown in fig. 9, fig. 9 is a connection block diagram of an accuracy comparison module in the cloud service-oriented industrial equipment large data quality test architecture of the present invention. The accuracy comparison module is used for carrying out accuracy test on the detection monitoring terminal, obtaining accuracy, adjusting parameters of the detection monitoring terminal according to the numerical value of the accuracy to obtain a qualified detection monitoring terminal, and obtaining industrial data by using the qualified detection monitoring terminal.
As shown in fig. 10, fig. 10 is a connection block diagram of a consistency comparison module in the test architecture of the cloud service-oriented industrial equipment with large data quality according to the present invention. The consistency comparison module is used for connecting the qualified detection monitoring terminal with the cloud platform, the qualified detection monitoring terminal uploads the acquired industrial data to the cloud platform, and the cloud platform performs consistency comparison on the acquired industrial data to acquire a consistency comparison result.
As shown in fig. 11, fig. 11 is a connection block diagram of an integrity comparison module in the cloud service oriented industrial equipment large data quality test architecture of the present invention. The integrity comparison module is used for connecting the cloud platform to the database and the cache, the cloud platform obtains data stored in the database and the cache, integrity comparison is carried out on the data stored in the cache and the data stored in the database, and an integrity comparison result is obtained.
The data detection module is added in the data acquisition, transmission and storage processes, and through data comparison and transmission protocol comparison, the data quality detection module is used for providing values of accuracy, consistency and integrity of data quality, effectively improving the quality of industrial data, providing improved and perfect accurate targets for detecting and monitoring terminals and cloud platforms, improving the data quality of large data detected and monitored by industrial equipment on the cloud platform, and further providing effective data support for the diagnosis and prediction functions of the cloud service platform of the industrial equipment.
It should be understood that the above description of specific embodiments of the present invention is only for the purpose of illustrating the technical lines and features of the present invention, and is intended to enable those skilled in the art to understand the contents of the present invention and to implement the present invention, but the present invention is not limited to the above specific embodiments. It is intended that all such changes and modifications as fall within the scope of the appended claims be embraced therein.
Claims (10)
1. A cloud service-oriented industrial equipment big data quality testing method is characterized by comprising the following steps:
s1, carrying out accuracy test on the detection monitoring terminal to obtain accuracy, adjusting parameters of the detection monitoring terminal according to the value of the accuracy to obtain a qualified detection monitoring terminal, and obtaining industrial data by using the qualified detection monitoring terminal;
s2, connecting the qualified detection monitoring terminal with a cloud platform, uploading the acquired industrial data to the cloud platform by the qualified detection monitoring terminal, and carrying out consistency comparison on the acquired industrial data by the cloud platform to acquire a consistency comparison result;
and S3, connecting the cloud platform to the database and the cache, wherein the cloud platform acquires the data stored in the database and the cache, and performs integrity comparison between the data stored in the cache and the data stored in the database to acquire an integrity comparison result.
2. The method as claimed in claim 1, wherein the performing an accuracy test on the monitoring terminal to obtain an accuracy, and adjusting the parameters of the monitoring terminal according to the value of the accuracy to obtain a qualified monitoring terminal comprises:
s01, setting a corresponding reference piece according to a detection monitoring terminal to be tested, wherein the reference piece generates standard data in real time;
s02, the detection monitoring terminal acquires the standard data by using a sensor arranged on the reference piece, the standard data acquired by the detection monitoring terminal is used as test data, and the test data is compared with the standard data in accuracy to acquire accuracy;
s03, the detection monitoring terminal judges whether the accuracy value reaches a set value; if so, taking the detection monitoring terminal as the qualified detection monitoring terminal; if not, the parameters of the detection monitoring terminal are corrected by the correction function, and the steps S02 and S03 are repeated.
3. The test method of claim 2, wherein the accuracy comparison specifically comprises:
s11, respectively reading the data in the standard data and the test data at the same time interval into a memory space, and performing denoising pretreatment on the two groups of data in the memory space;
s12, respectively calculating peak positions of the two groups of data subjected to denoising preprocessing, and shifting the peak positions of the test data to the peak positions of the standard data;
s13, respectively stretching windows leftwards and rightwards by adopting a mean interpolation method to the test data with the peak position of the standard data as a middle point, and performing window difference processing on the stretched test data and the standard data;
s14, calculating the accuracy value of the test data by calculating the percentage of the window difference in the area of the standard data.
4. The test method of claim 1, wherein the consistency comparison comprises a data consistency comparison and a protocol consistency comparison.
5. The testing method of claim 4, wherein the data consistency comparison comprises:
s211, the qualified detection monitoring terminal is in communication connection with a cloud platform according to a set protocol, and the cloud platform sends a test request to the qualified detection monitoring terminal according to set time;
s212, the qualified detection monitoring terminal acquires and encapsulates the industrial data to obtain encapsulated data, and the industrial data before encapsulation is backed up to a set file;
s213, the cloud platform obtains the encapsulation data, analyzes the encapsulation data to obtain analysis data, and compares the analysis data with the consistency of the data content of the industrial data in the setting file to obtain a data consistency comparison value.
6. The testing method of claim 5, wherein the comparing the analytical data to the data content of the industrial data in the profile comprises:
s2131, comparing the data content of the data items of the analyzed data with the data items of data stored in a database in advance;
s2132, identifying and recording the number of data items with different data contents in the analysis data and the data stored in the database to obtain the wrong data volume of the data contents, and obtaining the correct data volume of the data contents according to the wrong data volume of the data contents;
and S2133, comparing the data quantity with correct data content with the data quantity of the analyzed data to obtain a data consistency comparison value.
7. The testing method of claim 6, wherein the protocol consistency comparison comprises:
s221, when the qualified detection monitoring terminal receives a test request, comparing the data item of the analysis data with the data item of the data stored in advance in a database in a data format;
s222, identifying and recording the number of data items with different data formats in the analysis data and the data stored in the database to obtain data quantity with wrong data formats, and obtaining data quantity with correct data formats according to the data quantity with wrong data formats;
and S223, comparing the data quantity with the correct data format with the data quantity of the analysis data to obtain a protocol consistency comparison value.
8. The test method of claim 6, wherein the integrity comparison comprises:
s31, the cloud platform stores the analysis data into a database according to a storage strategy, and simultaneously stores the analysis data into a cache;
s32, the cloud platform acquires a starting instruction requiring integrity test, acquires corresponding data in the database according to the data identification of the analysis data stored in the cache, and then performs integrity comparison on the analysis data in the cache and the data items of the data acquired from the database to obtain an integrity comparison value, wherein the data identification comprises a device number and a time stamp.
9. A cloud service-oriented industrial equipment big data quality testing architecture is characterized by comprising:
the detection monitoring terminal is used for acquiring the industrial data and packaging and sending the industrial data;
the cloud platform is used for receiving and analyzing the packaging data, obtaining the analysis data and storing the analysis data into a database;
and the data comparison module is used for comparing the content of the data and the format of the data to obtain a comparison result.
10. The test architecture of claim 9, wherein the data comparison module comprises:
the accuracy comparison module is used for carrying out accuracy test on the detection monitoring terminal to obtain accuracy, adjusting parameters of the detection monitoring terminal according to the numerical value of the accuracy to obtain a qualified detection monitoring terminal, and acquiring industrial data by using the qualified detection monitoring terminal;
the consistency comparison module is used for connecting the qualified detection monitoring terminal with a cloud platform, the qualified detection monitoring terminal uploads the obtained industrial data to the cloud platform, and the cloud platform performs consistency comparison on the obtained industrial data to obtain a consistency comparison result;
and the integrity comparison module is used for connecting the cloud platform to the database and the cache, acquiring the data stored in the database and the cache by the cloud platform, and performing integrity comparison on the data stored in the cache and the data stored in the database to acquire an integrity comparison result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911359368.6A CN111221807B (en) | 2019-12-25 | 2019-12-25 | Cloud service-oriented industrial equipment big data quality testing method and framework |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911359368.6A CN111221807B (en) | 2019-12-25 | 2019-12-25 | Cloud service-oriented industrial equipment big data quality testing method and framework |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111221807A true CN111221807A (en) | 2020-06-02 |
| CN111221807B CN111221807B (en) | 2023-07-18 |
Family
ID=70829198
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911359368.6A Active CN111221807B (en) | 2019-12-25 | 2019-12-25 | Cloud service-oriented industrial equipment big data quality testing method and framework |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111221807B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112016828A (en) * | 2020-08-26 | 2020-12-01 | 中国特种设备检测研究院 | Industrial equipment health management cloud platform architecture based on streaming big data |
| CN112632048A (en) * | 2020-12-18 | 2021-04-09 | 恩亿科(北京)数据科技有限公司 | Data quality detection method, system, electronic equipment and storage medium |
| CN112835334A (en) * | 2020-12-31 | 2021-05-25 | 广州明珞装备股份有限公司 | Industrial data platform testing method and device, computer equipment and storage medium |
| CN115077748A (en) * | 2022-06-28 | 2022-09-20 | 北京市热力集团有限责任公司 | Two-in-one detection system for heat meter |
| CN116527555A (en) * | 2023-06-20 | 2023-08-01 | 中国标准化研究院 | Cross-platform data intercommunication consistency test method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015161616A1 (en) * | 2014-04-21 | 2015-10-29 | 中兴通讯股份有限公司 | Method and device for collecting data, and method and device for testing capacity of system |
| CN106899691A (en) * | 2017-03-16 | 2017-06-27 | 广州大学 | A kind of Intelligent internet of things monitoring system and method based on cloud platform |
| CN108322548A (en) * | 2018-03-07 | 2018-07-24 | 浙江大学 | A kind of industrial process data analyzing platform based on cloud computing |
| CN109933581A (en) * | 2019-03-01 | 2019-06-25 | 武汉达梦数据库有限公司 | A kind of data quality checking method and system |
| CN110174568A (en) * | 2019-05-15 | 2019-08-27 | 广西电网有限责任公司电力科学研究院 | A kind of multiple services calibration system of equipment for monitoring power quality and its calibration method |
| CN110336845A (en) * | 2019-04-02 | 2019-10-15 | 中国联合网络通信集团有限公司 | Industrial product quality method of real-time, equipment and system |
-
2019
- 2019-12-25 CN CN201911359368.6A patent/CN111221807B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015161616A1 (en) * | 2014-04-21 | 2015-10-29 | 中兴通讯股份有限公司 | Method and device for collecting data, and method and device for testing capacity of system |
| CN106899691A (en) * | 2017-03-16 | 2017-06-27 | 广州大学 | A kind of Intelligent internet of things monitoring system and method based on cloud platform |
| CN108322548A (en) * | 2018-03-07 | 2018-07-24 | 浙江大学 | A kind of industrial process data analyzing platform based on cloud computing |
| CN109933581A (en) * | 2019-03-01 | 2019-06-25 | 武汉达梦数据库有限公司 | A kind of data quality checking method and system |
| CN110336845A (en) * | 2019-04-02 | 2019-10-15 | 中国联合网络通信集团有限公司 | Industrial product quality method of real-time, equipment and system |
| CN110174568A (en) * | 2019-05-15 | 2019-08-27 | 广西电网有限责任公司电力科学研究院 | A kind of multiple services calibration system of equipment for monitoring power quality and its calibration method |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112016828A (en) * | 2020-08-26 | 2020-12-01 | 中国特种设备检测研究院 | Industrial equipment health management cloud platform architecture based on streaming big data |
| CN112016828B (en) * | 2020-08-26 | 2024-03-12 | 中国特种设备检测研究院 | Industrial equipment health management cloud platform architecture based on streaming big data |
| CN112632048A (en) * | 2020-12-18 | 2021-04-09 | 恩亿科(北京)数据科技有限公司 | Data quality detection method, system, electronic equipment and storage medium |
| CN112835334A (en) * | 2020-12-31 | 2021-05-25 | 广州明珞装备股份有限公司 | Industrial data platform testing method and device, computer equipment and storage medium |
| CN112835334B (en) * | 2020-12-31 | 2022-05-27 | 广州明珞装备股份有限公司 | Industrial data platform testing method and device, computer equipment and storage medium |
| CN115077748A (en) * | 2022-06-28 | 2022-09-20 | 北京市热力集团有限责任公司 | Two-in-one detection system for heat meter |
| CN116527555A (en) * | 2023-06-20 | 2023-08-01 | 中国标准化研究院 | Cross-platform data intercommunication consistency test method |
| CN116527555B (en) * | 2023-06-20 | 2023-09-12 | 中国标准化研究院 | Cross-platform data intercommunication consistency test method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111221807B (en) | 2023-07-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111221807B (en) | Cloud service-oriented industrial equipment big data quality testing method and framework | |
| CN108683562B (en) | Anomaly detection positioning method and device, computer equipment and storage medium | |
| WO2017000424A1 (en) | Protocol detection method and apparatus | |
| US11038587B2 (en) | Method and apparatus for locating fault cause, and storage medium | |
| CN110750458A (en) | Big data platform testing method and device, readable storage medium and electronic equipment | |
| CN108900319B (en) | Fault detection method and device | |
| CN102096410B (en) | Dynamic function test method of high-speed train operation control system | |
| KR20190060547A (en) | Method of Deriving and Visualizing the Causes of Process Malfunctions Through Machine Learning Model In Data Imbalance Environment | |
| CN114460439A (en) | Digital integrated circuit test system | |
| CN115277261A (en) | Abnormal machine intelligent identification method, device and equipment based on industrial control network virus | |
| CN111309584B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN113114987B (en) | Hadoop-based power system inspection method and system | |
| CN112034820A (en) | Cloud-based hardware-in-loop system testing method and system and storage medium | |
| CN109783263B (en) | Method and system for processing aging test fault of server | |
| CN113300912A (en) | Equipment testing method and device and electronic equipment | |
| CN115065612B (en) | Testing method and device for full-link pressure measurement transformation | |
| CN109460356B (en) | Data fusion method for software fault prediction | |
| CN113904954B (en) | System for testing wireless communication module | |
| CN114020500A (en) | Method, server and equipment for acquiring equipment fault information | |
| CN109688198B (en) | Distributed system and fault detection method | |
| CN117610998A (en) | Material defect detection method, device, equipment and storage medium | |
| CN114185538A (en) | Data testing method and device, computer equipment and storage medium | |
| CN118470601A (en) | Intelligent driving perception algorithm automatic evaluation method and device based on video injection | |
| CN115686982A (en) | Vehicle-mounted system host test method, device, equipment and readable storage medium | |
| CN115543712A (en) | Automatic detection method and device for equipment configuration information, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |