Nothing Special   »   [go: up one dir, main page]

CN117390008A - Method and device for processing measurement data of multi-type observation instrument - Google Patents

Method and device for processing measurement data of multi-type observation instrument Download PDF

Info

Publication number
CN117390008A
CN117390008A CN202311685187.9A CN202311685187A CN117390008A CN 117390008 A CN117390008 A CN 117390008A CN 202311685187 A CN202311685187 A CN 202311685187A CN 117390008 A CN117390008 A CN 117390008A
Authority
CN
China
Prior art keywords
target
measurement data
data
verification result
processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202311685187.9A
Other languages
Chinese (zh)
Other versions
CN117390008B (en
Inventor
李卡平
宋月辉
王颖
楼子辰
陈磊
吴宝征
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Planet Aerospace Information Technology Co ltd
Original Assignee
Beijing Planet Aerospace Information Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Planet Aerospace Information Technology Co ltd filed Critical Beijing Planet Aerospace Information Technology Co ltd
Priority to CN202311685187.9A priority Critical patent/CN117390008B/en
Publication of CN117390008A publication Critical patent/CN117390008A/en
Application granted granted Critical
Publication of CN117390008B publication Critical patent/CN117390008B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21Design, administration or maintenance of databases
    • G06F16/215Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Testing Or Calibration Of Command Recording Devices (AREA)

Abstract

The application provides a measurement data processing method and device of a multi-type observation instrument, which belong to the technical field of environmental monitoring, and the method comprises the following steps: determining a target parameter and corresponding target measurement data in response to the current data request; the target measurement data is measurement data of which the measurement area and the measurement time are matched with the current data request in the total measurement data corresponding to the target parameters; the total measurement data corresponding to the target parameters are obtained by measuring various different types of observation instruments; acquiring target measurement data from an original database, and processing the target measurement data based on a primary verification rule base to obtain a primary verification result base; processing the measurement data in the initial verification result library based on the accurate verification rule library to obtain an accurate verification result library; and determining target distribution data based on the quality scores of the measurement data in the fine verification result library, and sending the target distribution data to a target user, so that the acquisition and processing efficiency of the measurement data of the multi-type observation instrument can be improved.

Description

Method and device for processing measurement data of multi-type observation instrument
Technical Field
The present disclosure relates to the field of environmental monitoring technologies, and in particular, to a method and an apparatus for processing measurement data of a multi-type observation instrument.
Background
With the deterioration of ecological environment, environmental protection is becoming a widespread concern. To accurately and timely take environmental protection measures, accurate monitoring of relevant environmental information (including atmospheric information, water resource information, vegetation information, etc.) is required.
Existing environmental monitoring means generally adopt various observation instruments for comprehensive monitoring, including observation instruments arranged on the ground and observation instruments arranged in the air, such as remote sensing satellites and the like. However, because the measurement data structures of different types of observation instruments are different, each observation instrument independently performs data acquisition, processing and storage, and then manually derives measurement data from each observation instrument and performs total analysis. And because the arrangement of different observation instruments is asynchronous and the positions are scattered, the installation progress of the different observation instruments is asynchronous, and the acquisition period of instrument measurement data is prolonged. Meanwhile, the data of different observation instruments are required to be processed by using different software, and fusion can be carried out after all data are processed, so that the processing period of the measurement data of the observation instruments is too long, and the processing efficiency of the measurement data is reduced.
Disclosure of Invention
The application provides a method and a device for processing measurement data of a multi-type observation instrument, so as to improve the acquisition and processing efficiency of the measurement data of the multi-type observation instrument.
The application provides a measurement data processing method of a multi-type observation instrument, which comprises the following steps:
determining a target parameter and corresponding target measurement data in response to the current data request; the target parameter is a parameter matched with the current data request; the target measurement data are measurement data of which the measurement area and the measurement time are matched with the current data request in the total measurement data corresponding to the target parameters; the total measurement data corresponding to the target parameters are obtained by measuring a plurality of different types of observation instruments;
acquiring the target measurement data from an original database, and processing the target measurement data based on a primary verification rule base to obtain a primary verification result base;
processing the measurement data in the initial verification result library based on a fine verification rule library to obtain a fine verification result library;
and determining target distribution data based on the quality scores of the measurement data in the fine verification result library, and transmitting the target distribution data to a target user corresponding to the current data request.
According to the measurement data processing method of the multi-type observation instrument provided by the application, the primary verification rule base comprises fusion rules of similar heterogeneous data and screening rules of invalid data, and correspondingly, the primary verification rule base is used for processing the target measurement data to obtain a primary verification result base, and the method specifically comprises the following steps:
preprocessing target measurement data corresponding to each target parameter based on the fusion rule of the similar heterogeneous data to obtain target measurement data sets corresponding to each target parameter respectively;
screening the invalid data in the target measurement data sets corresponding to the target parameters respectively based on the screening rule of the invalid data to obtain effective target measurement data sets corresponding to the target parameters respectively;
and generating a primary verification result library based on the effective target measurement data sets respectively corresponding to the target parameters.
According to the measurement data processing method of the multi-type observation instrument provided by the application, the filtering rule based on the invalid data filters the invalid data in the target measurement data set corresponding to each target parameter respectively, and the method specifically comprises the following steps:
screening first invalid data in target measurement data sets corresponding to all target parameters respectively based on a preset parameter value standard fluctuation range to obtain potential valid target measurement data sets corresponding to all target parameters respectively;
And screening second invalid data in the potential valid target measurement data sets corresponding to the target parameters respectively based on the parameter value prediction curve updated in real time to obtain valid target measurement data sets corresponding to the target parameters respectively.
According to the measurement data processing method of the multi-type observation instrument provided by the application, the precision verification rule base is used for processing the measurement data in the initial verification result base to obtain the precision verification result base, and the method specifically comprises the following steps:
correcting the measured data in the effective target measured data sets corresponding to the target parameters respectively based on a preset precision lifting algorithm to obtain corrected target measured data sets corresponding to the target parameters respectively;
and generating a fine verification result library based on the corrected target measurement data sets respectively corresponding to the target parameters.
According to the measurement data processing method of the multi-type observation instrument provided by the application, the target distribution data is determined based on the quality scores of the measurement data in the fine verification result library, and the method specifically comprises the following steps:
dividing correction target measurement data sets corresponding to all target parameters in a fine verification result library into a plurality of correction target measurement data subsets based on a preset minimum statistical time length;
And for any correction target measurement data set, determining quality scores of a plurality of correction target measurement data subsets corresponding to the correction target measurement data set, and taking the correction target measurement data subset with the quality scores not lower than a preset threshold value as target distribution data.
According to the measurement data processing method of the multi-type observation instrument, the quality scores of all the correction target measurement data subsets are determined based on the effective data proportion and the missing measurement rate of the correction target measurement data subsets.
According to the measurement data processing method of the multi-type observation instrument provided by the application, the current data request comprises target parameters, a measurement area, measurement time and indication information of a target user.
The present application also provides a measurement data processing apparatus of a multi-type scope, the apparatus comprising:
the target measurement data determining unit is used for determining target parameters and corresponding target measurement data in response to the current data request; the target parameter is a parameter matched with the current data request; the target measurement data are measurement data of which the measurement area and the measurement time are matched with the current data request in the total measurement data corresponding to the target parameters; the total measurement data corresponding to the target parameters are obtained by measuring a plurality of different types of observation instruments;
The primary verification unit is used for acquiring the target measurement data from the original database and processing the target measurement data based on a primary verification rule base to obtain a primary verification result base;
the fine verification unit is used for processing the measurement data in the initial verification result library based on the fine verification rule library to obtain a fine verification result library;
and the distribution unit is used for determining target distribution data based on the quality scores of the measured data in the fine verification result library and transmitting the target distribution data to a target user corresponding to the current data request.
The present application also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of a method of processing measurement data of a multi-type scope as described in any of the above.
The present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of a method of processing measurement data of a multi-type scope as described in any of the above.
The application provides a method and a device for processing measurement data of a multi-type observation instrument, wherein the method comprises the following steps: determining a target parameter and corresponding target measurement data in response to the current data request; the target parameter is a parameter matched with the current data request; the target measurement data are measurement data of which the measurement area and the measurement time are matched with the current data request in the total measurement data corresponding to the target parameters; the total measurement data corresponding to the target parameters are obtained by measuring a plurality of different types of observation instruments; acquiring the target measurement data from an original database, and processing the target measurement data based on a primary verification rule base to obtain a primary verification result base; processing the measurement data in the initial verification result library based on a fine verification rule library to obtain a fine verification result library; and determining target distribution data based on the quality scores of the measurement data in the fine verification result library, and sending the target distribution data to a target user corresponding to the current data request, so that the acquisition and processing efficiency of the measurement data of the multi-type observation instrument can be improved, and the quality of the data is ensured.
Drawings
For a clearer description of the present application or of the prior art, the drawings that are used in the description of the embodiments or of the prior art will be briefly described, it being apparent that the drawings in the description below are some embodiments of the present application, and that other drawings may be obtained from these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for processing measurement data of a multi-type scope provided by the present application;
FIG. 2 is a schematic diagram of a generation flow of the initial verification result library provided by the application;
FIG. 3 is a schematic diagram of a generation flow of the proof-of-date library provided by the present application;
FIG. 4 is a schematic diagram of the structure of a measurement data processing device of the multi-type scope provided by the present application;
fig. 5 is a schematic structural diagram of an electronic device provided in the present application.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions in the present application will be clearly and completely described below with reference to the drawings in the present application, and it is apparent that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
Each observation instrument is an observation instrument
Fig. 1 is a flow chart of a measurement data processing method of a multi-type observation instrument provided in the present application, as shown in fig. 1, the method includes:
step 101, determining target parameters and corresponding target measurement data in response to a current data request; the target parameter is a parameter matched with the current data request; the target measurement data are measurement data of which the measurement area and the measurement time are matched with the current data request in the total measurement data corresponding to the target parameters; the full measurement data corresponding to the target parameters are obtained through measurement of a plurality of different types of observation instruments.
And 102, acquiring the target measurement data from an original database, and processing the target measurement data based on a primary verification rule base to obtain a primary verification result base.
And step 103, processing the measurement data in the initial verification result library based on the fine verification rule library to obtain a fine verification result library.
And 104, determining target distribution data based on the quality scores of the measurement data in the fine verification result library, and transmitting the target distribution data to a target user corresponding to the current data request.
Specifically, the measurement data processing method of the multi-type observation instrument according to the embodiment of the present application may be set in a server in the form of software, where the server is communicatively connected to each observation instrument to collect measurement data of each observation instrument and store the collected measurement data in an original database set in the server, and then perform centralized processing and distribution of measurement data based on the measurement data processing method of the multi-type observation instrument according to the embodiment of the present application.
Based on the foregoing, since the measurement data structures of different types of observation apparatuses are different, and the observation apparatuses for measuring the same parameter may also have different measurement data structures due to different manufacturers and/or models, in the prior art, each observation apparatus independently performs data acquisition, processing and storage, and then manually derives measurement data from each observation apparatus and performs aggregate analysis, which results in low data acquisition efficiency. In order to solve the problem, the embodiment of the application receives the measurement data of the corresponding observation instrument through a receiving plug-in matched with each observation instrument and arranged in a server. The receiving plug-in can be added, deleted or updated in a dynamic hot plug mode based on different parameter types, so that the problems that the installation period of an early-stage observation instrument is long, different observation instruments are not synchronous in arrangement, and data are needed to be manually derived from the observation instruments are effectively solved, and the efficiency of measuring data access is improved. As for the communication mode of the observation instrument and the receiving plug-in unit, wired communication or wireless communication can be adopted, and the communication mode can be specifically selected according to the model and the actual requirement of the observation instrument, and the embodiment of the application is not particularly limited.
Further, the server also comprises a data processing module and a verification module. The data processing module is used for acquiring target measurement data corresponding to the target parameters in response to a data request sent by a target user, determining target distribution data based on the target measurement data and feeding back the target distribution data to the target user. Notably, the target parameters are parameters that match the current data request, and based on the needs of the user, the target parameters may be any environmental metrics such as the aforementioned soil moisture content, surface reflectivity and leaf area index, NDVI (Normalized Difference Vegetation Index, normalized vegetation index), and the like. Correspondingly, the application of the finally obtained target distribution data can be used for analyzing the environment state according to the requirement of the user, can also be used for realizing the verification of satellite remote sensing data, and can be used as standard data for comparison with data measured by the user by oneself, and the embodiment of the application is not particularly limited.
The target measurement data are measurement data of which the measurement area and the measurement time in the total measurement data (namely all acquired measurement data) corresponding to the target parameters are matched with the current data request. It can be understood that the user can add the indication information of the parameter type, the measurement area and the measurement time into the data request according to the own requirement, and meanwhile, the user identification information can be added into the data request by himself. Based on the method, the server can accurately acquire the target measurement data and accurately feed back the target measurement data to the target user after determining the target distribution data. It can be further understood that the measurement data fed back by each observation instrument includes equipment identity information (such as a number), observation position information and observation time information, so as to accurately identify the observation instrument, the measurement area and the monitoring time corresponding to the measurement data. Based on the above, the embodiment of the application can quickly determine and acquire the target measurement data corresponding to the target parameter based on the parameter type, the measurement area and the indication information of the measurement time set in the current data request.
Meanwhile, in order to ensure accuracy of final distribution data, the embodiment of the application processes the target measurement data based on a preset primary verification rule base to obtain a primary verification result base, processes the measurement data in the primary verification result base based on a preset fine verification rule base to obtain a fine verification result base, and determines the target distribution data based on quality scores of the measurement data in the fine verification result base so as to ensure accuracy of the target distribution data to the greatest extent.
In addition, the server further comprises: the plug-in management module is used for carrying out configuration and updating operation of receiving plug-ins, and the check rule management module is used for carrying out configuration and updating operation on check rules in the initial check rule library and the fine check rule library. Based on the foregoing, it can be known that, because the layout of different observation apparatuses is performed asynchronously and the positions are dispersed, the installation progress of the different observation apparatuses is not synchronous, and for this problem, the plug-in management module provided in the embodiment of the present application can perform the configuration and update operation of the receiving plug-ins based on the installation and use conditions of the observation apparatuses, so that the configuration and update of the receiving plug-ins can be performed in time no matter how the states of the observation apparatuses change, and further the acquisition efficiency of the measurement data is ensured. Meanwhile, the verification rules in the initial verification rule base and the fine verification rule base are configured and updated through the verification rule management module, so that accuracy of measurement data and flexibility of rule configuration can be further ensured.
More specifically, because the measurement data structures of different types of observation apparatuses are different, and different observation apparatuses for measuring the same parameter may also have different measurement data structures due to different manufacturers and/or models, to implement efficient processing of data, heterogeneous data of the same type (for example, soil moisture content data measured by soil moisture analyzers of different models and vegetation index data measured by vegetation index measuring devices of different types) need to be fused at first, and meanwhile, in order to ensure accuracy of the measurement data, invalid data in the measurement data need to be screened (i.e., removed). Based on the above, in the embodiment of the application, the fusion rule of the similar heterogeneous data and the screening rule of the invalid data are set in the initial verification rule base so as to perform efficient fusion on the similar heterogeneous data in the target measurement data, and meanwhile, the invalid data are removed so as to obtain the initial verification result base. Fig. 2 is a schematic diagram of a generation flow of the initial verification result library provided in the present application, as shown in fig. 2, where the initial verification rule library is based on processing the target measurement data to obtain the initial verification result library, and specifically includes:
step 1021, preprocessing the target measurement data corresponding to each target parameter based on the fusion rule of the heterogeneous data of the same kind to obtain a target measurement data set corresponding to each target parameter respectively;
Step 1022, screening the invalid data in the target measurement data sets corresponding to the target parameters respectively based on the screening rule of the invalid data to obtain effective target measurement data sets corresponding to the target parameters respectively;
step 1023, generating a primary verification result library based on the effective target measurement data sets respectively corresponding to the target parameters.
It may be understood that, in the embodiment of the present application, corresponding parameter recording templates may be preset as fusion rules of heterogeneous data of the same type based on characteristics of different parameters, where the parameter recording templates include data items to be recorded and recording formats of the data items, and on this basis, for target measurement data corresponding to each target parameter, preprocessing may be performed based on the corresponding parameter recording templates to obtain target measurement data sets corresponding to each target parameter respectively. For any target parameter, the target parameters may correspond to observation instruments of different models/manufacturers, based on the target parameters, the measurement data of the same observation instrument can be clustered based on the equipment number in the corresponding target measurement data set, and then the measurement data is preprocessed based on the corresponding parameter recording template. The preprocessing mode comprises at least one of format conversion and parameter extraction, specifically, for part of observation instruments, the measurement parameters are target parameters, for example, the target parameters are soil moisture content, the soil moisture analyzer can directly measure the soil moisture content, and only the data formats of the soil moisture content measured by the soil moisture analyzers of different models are different, so that the data can be obtained only through format conversion. For some observers, the measured parameter is not a target parameter, but an associated parameter of the target parameter, for example, the target parameter is NDVI, and the vegetation index measuring apparatus can only measure the illumination intensities of the near infrared band and the Red band (i.e., the associated parameter of the target parameter) based on the Nir sensor and the Red sensor, but the NDVI needs to be calculated based on the illumination intensities of the near infrared band and the Red band, and for this type of data, parameter extraction needs to be performed (i.e., the target parameter is calculated based on the associated parameter). While for measurement data of a portion of the scope, format conversion and parameter extraction may need to be performed simultaneously, embodiments of the present application are not intended to be exhaustive.
After the target measurement data sets corresponding to the target parameters are obtained, invalid data in the target measurement data sets are required to be removed to ensure the validity of the measurement data, so that errors in the processes of verification, comparison or analysis and the like by using the measurement data of a subsequent user are avoided. Specifically, the filtering rule based on the invalid data filters the invalid data in the target measurement data set corresponding to each target parameter, and specifically includes:
screening first invalid data in target measurement data sets corresponding to all target parameters respectively based on a preset parameter value standard fluctuation range to obtain potential valid target measurement data sets corresponding to all target parameters respectively;
and screening second invalid data in the potential valid target measurement data sets corresponding to the target parameters respectively based on the parameter value prediction curve updated in real time to obtain valid target measurement data sets corresponding to the target parameters respectively.
It should be noted that conventional invalid data filtering is only based on experience to reject data with obvious numerical value abnormality (for example, the numerical value is null or messy code), but some data with not obvious numerical value abnormality cannot be screened, so that errors exist in the final result. Based on this, the embodiment of the present application performs filtering of invalid data from two layers:
Firstly, historical data of target parameters of a current area are analyzed, a parameter value standard fluctuation range corresponding to the target parameters is determined, and first invalid data in target measurement data sets respectively corresponding to the target parameters are screened based on the parameter value standard fluctuation range, so that potential valid target measurement data sets respectively corresponding to the target parameters are obtained. It can be understood that the first invalid data, namely, the data of which the numerical value is not in the standard fluctuation range of the parameter value (including null value and messy code), can be based on the first invalid data, and besides the data of which the prior art can screen the null value, the messy code and other obvious anomalies, the embodiment of the application can also provide the data of which the numerical value is obviously not in accordance with the theoretical fluctuation range, so that the accuracy of the measured data is improved. Based on this, the application further researches find that, because the target parameter is interfered by the environment and the working state of the observation instrument, the measurement data partially conforming to the fluctuation range of the parameter value standard may have larger error, and thus becomes invalid data (namely second invalid data) losing the reference value. In order to solve the problem, the embodiment of the application updates the parameter value prediction curve of each observation instrument corresponding to the potential effective target measurement data set in real time through the pre-trained parameter value prediction model, and determines second invalid data based on the comparison result of the parameter value prediction curve and the corresponding measurement data in the potential effective target measurement data set. The comparison result may be a difference between the predicted value and the actual measured value, and if the difference exceeds a preset difference threshold, the corresponding measured data is judged to be the second invalid data, and the preset difference threshold may be adjusted according to the actual measurement accuracy, which is not particularly limited in the embodiment of the present application.
It should be noted that, in order to ensure the accuracy of the parameter value prediction result and thus the accuracy of the measured data, the input of the parameter value prediction model at the current moment is preferably the parameter value measured data of the previous period, the state parameters (including electric quantity, use duration and the like) of the observation instrument and the environmental parameters (including weather, air temperature and the like), based on which the accuracy of the parameter value prediction result can be ensured to the maximum extent. Any existing practical training method can be adopted in the training process of the model, and the embodiment of the application is not described herein.
By combining the two invalid data screening layers, the accuracy of the effective target measurement data set corresponding to each target parameter can be guaranteed to the greatest extent. On the basis, for the effective target measurement data sets respectively corresponding to each target parameter, the application finds that after part of invalid data is removed, the data of a specific time point of a specific coordinate is lost, and then the problem of data mismatch in scenes such as satellite remote sensing data verification by a subsequent user through measurement data is caused. Aiming at the problem, the embodiment of the application further carries out precision improvement operation on the measurement data in the initial verification result library based on the accurate verification rule library. Specifically, fig. 3 is a schematic diagram of a generation flow of the proof-of-date result library provided in the present application, as shown in fig. 3, where the processing, based on the proof-of-date rule library, of measurement data in the initial proof-of-date result library to obtain the proof-of-date result library specifically includes:
Step 1031, correcting the measurement data in the effective target measurement data sets corresponding to the target parameters respectively based on a preset precision lifting algorithm to obtain corrected target measurement data sets corresponding to the target parameters respectively;
step 1032, generating a fine verification result library based on the corrected target measurement data sets corresponding to the target parameters.
It can be understood that the correction of the measurement data in the effective target measurement data set includes two dimensions, the first dimension is to the rejected invalid data, the corresponding correction value is determined and interpolated, preferably, the correction value of the invalid data can be determined by using the parameter value prediction model, and based on this, the maximization of the efficiency of the primary verification and the fine verification can be ensured. Of course, missing value prediction and filling may be performed in other feasible manners, which are not specifically limited in the embodiments of the present application; the second dimension is to take a single observation instrument as a unit, carry out integral correction on corresponding measurement data in an effective target measurement data set, specifically, can regularly calibrate the observation instrument and determine corresponding calibration coefficients, and correct the measurement data of each observation instrument based on the calibration coefficients. Based on the method, the comprehensiveness and the accuracy of the measured data in the corrected target measured data set corresponding to each target parameter can be guaranteed to the greatest extent.
After obtaining the proof effort library, the target distribution data can be determined based on the quality score of the measurement data in the proof effort library, specifically, the determining the target distribution data based on the quality score of the measurement data in the proof effort library specifically includes:
dividing correction target measurement data sets corresponding to all target parameters in a fine verification result library into a plurality of correction target measurement data subsets based on a preset minimum statistical time length;
and for any correction target measurement data set, determining quality scores of a plurality of correction target measurement data subsets corresponding to the correction target measurement data set, and taking the correction target measurement data subset with the quality scores not lower than a preset threshold value as target distribution data.
Wherein the quality score for each modified target measurement data subset is determined based on the effective data duty cycle and the missing measurement rate of the modified target measurement data subset.
Specifically, based on the foregoing, it is known that, in the initial verification stage, the first invalid data and the second invalid data in the target measurement data set are screened and removed, so as to obtain valid target measurement data sets corresponding to each target parameter respectively, and although the correction value of the removed invalid data is determined and interpolated in the fine verification stage, the accuracy of the correction value cannot be completely ensured. Meanwhile, there may be a missing measurement (i.e., measurement data is not obtained in a part of the period due to failure of the scope or the like) in the target measurement data set corresponding to each target parameter. Both of the above reasons may lead to a decrease in accuracy of the measurement data in the corrected target measurement data set, thereby affecting the application of the user. Based on this, the embodiment of the application further divides the corrected target measurement data set corresponding to each target parameter in the fine verification result library into a plurality of corrected target measurement data subsets based on a preset minimum statistical duration, determines quality scores of the plurality of corrected target measurement data subsets corresponding to any corrected target measurement data set, and uses the corrected target measurement data subsets with the quality scores not lower than a preset threshold as target distribution data. Based on the above, the accuracy of the target distribution data can be ensured to the maximum extent. It can be understood that the length of the measurement time interval corresponding to the measurement data in each correction target measurement data subset is the minimum statistical duration. The minimum statistical duration may be set according to actual needs, for example, 1 day, 1 week or 1 month, and the size of the preset threshold may also be adjusted according to the requirement of the application scenario on data precision, which is not specifically limited in the embodiment of the present application. It may be further understood that the effective data duty ratio of the correction target measurement data subset may be calculated according to the interpolation number (corresponding to the invalid data number) and the total data amount in the correction target measurement data subset, the missing measurement rate of the correction target measurement data subset may be calculated according to the total data amount in the correction target measurement data subset and the theoretical data amount corresponding to the minimum statistical duration, and the theoretical data amount corresponding to the minimum statistical duration may be determined according to the sampling frequency of the observation instrument.
The embodiment of the application can determine and set the scoring weight (marked as A) of the effective data duty ratio (marked as X) and the scoring weight (marked as B) of the lack-of-measurement ratio (marked as Y) based on the historical measurement data, and then the quality scoring calculation formula of the modified target measurement data subset is as follows: s=a×x+b×y. It will be appreciated that, given the example of percentile formulation, a+b=100.
According to the method provided by the embodiment of the application, the target parameters and the corresponding target measurement data are determined by responding to the current data request; the target parameter is a parameter matched with the current data request; the target measurement data are measurement data of which the measurement area and the measurement time are matched with the current data request in the total measurement data corresponding to the target parameters; the total measurement data corresponding to the target parameters are obtained by measuring a plurality of different types of observation instruments; acquiring the target measurement data from an original database, and processing the target measurement data based on a primary verification rule base to obtain a primary verification result base; processing the measurement data in the initial verification result library based on a fine verification rule library to obtain a fine verification result library; and determining target distribution data based on the quality scores of the measurement data in the fine verification result library, and sending the target distribution data to a target user corresponding to the current data request, so that the acquisition and processing efficiency of the measurement data of the multi-type observation instrument can be improved, and the quality of the data is ensured.
The measurement data processing device of the multi-type observation instrument provided by the application is described below, and the measurement data processing device of the multi-type observation instrument described below and the measurement data processing method of the multi-type observation instrument described above can be referred to correspondingly with each other.
Based on any of the above embodiments, fig. 4 is a schematic structural diagram of a measurement data processing apparatus of a multi-type scope provided in the present application, as shown in fig. 4, the apparatus includes:
a target measurement data determining unit 201, configured to determine a target parameter and corresponding target measurement data in response to a current data request; the target parameter is a parameter matched with the current data request; the target measurement data are measurement data of which the measurement area and the measurement time are matched with the current data request in the total measurement data corresponding to the target parameters; the total measurement data corresponding to the target parameters are obtained by measuring a plurality of different types of observation instruments;
the primary verification unit 202 is configured to obtain the target measurement data from an original database, and process the target measurement data based on a primary verification rule base to obtain a primary verification result base;
the fine verification unit 203 is configured to process the measurement data in the initial verification result library based on a fine verification rule library to obtain a fine verification result library;
And the distribution unit 204 is used for determining target distribution data based on the quality scores of the measurement data in the fine verification result library and sending the target distribution data to a target user corresponding to the current data request.
According to the device provided by the embodiment of the application, the target measurement data determining unit 201 responds to the current data request to determine the target parameters and the corresponding target measurement data; the target parameter is a parameter matched with the current data request; the target measurement data are measurement data of which the measurement area and the measurement time are matched with the current data request in the total measurement data corresponding to the target parameters; the total measurement data corresponding to the target parameters are obtained by measuring a plurality of different types of observation instruments; the primary verification unit 202 acquires the target measurement data from the original database, and processes the target measurement data based on a primary verification rule base to obtain a primary verification result base; the fine verification unit 203 processes the measurement data in the initial verification result library based on a fine verification rule library to obtain a fine verification result library; the distribution unit 204 determines target distribution data based on the quality scores of the measurement data in the fine verification result library, and sends the target distribution data to the target user corresponding to the current data request, so that the acquisition and processing efficiency of the measurement data of the multi-type observation instrument can be improved, and meanwhile, the quality of the data is ensured.
Based on the above embodiment, the primary verification rule base includes fusion rules of similar heterogeneous data and screening rules of invalid data, and correspondingly, the primary verification rule base is used for processing the target measurement data to obtain a primary verification result base, and specifically includes:
preprocessing target measurement data corresponding to each target parameter based on the fusion rule of the similar heterogeneous data to obtain target measurement data sets corresponding to each target parameter respectively;
screening the invalid data in the target measurement data sets corresponding to the target parameters respectively based on the screening rule of the invalid data to obtain effective target measurement data sets corresponding to the target parameters respectively;
and generating a primary verification result library based on the effective target measurement data sets respectively corresponding to the target parameters.
Based on any one of the foregoing embodiments, the filtering rule based on the invalid data filters the invalid data in the target measurement data set corresponding to each target parameter, which specifically includes:
screening first invalid data in target measurement data sets corresponding to all target parameters respectively based on a preset parameter value standard fluctuation range to obtain potential valid target measurement data sets corresponding to all target parameters respectively;
And screening second invalid data in the potential valid target measurement data sets corresponding to the target parameters respectively based on the parameter value prediction curve updated in real time to obtain valid target measurement data sets corresponding to the target parameters respectively.
Based on any one of the above embodiments, the processing, based on the fine verification rule base, the measurement data in the initial verification result base to obtain the fine verification result base specifically includes:
correcting the measured data in the effective target measured data sets corresponding to the target parameters respectively based on a preset precision lifting algorithm to obtain corrected target measured data sets corresponding to the target parameters respectively;
and generating a fine verification result library based on the corrected target measurement data sets respectively corresponding to the target parameters.
Based on any of the above embodiments, the determining the target distribution data based on the quality scores of the measurement data in the proof-result library specifically includes:
dividing correction target measurement data sets corresponding to all target parameters in a fine verification result library into a plurality of correction target measurement data subsets based on a preset minimum statistical time length;
and for any correction target measurement data set, determining quality scores of a plurality of correction target measurement data subsets corresponding to the correction target measurement data set, and taking the correction target measurement data subset with the quality scores not lower than a preset threshold value as target distribution data.
Based on any of the above embodiments, the quality score for each of the modified target measurement data subsets is determined based on the effective data duty cycle and the rate of absence of measurement for the modified target measurement data subset.
Based on any of the above embodiments, the current data request includes target parameters, measurement areas, measurement time, and indication information of the target user.
Fig. 5 illustrates a physical schematic diagram of an electronic device, as shown in fig. 5, which may include: processor 301, communication interface 302, memory 303 and communication bus 304, wherein processor 301, communication interface 302, memory 303 accomplish the intercommunication through communication bus 304. The processor 301 may invoke logic instructions in the memory 303 to perform the method of processing measurement data of a multi-type scope provided by the methods described above, the method comprising: determining a target parameter and corresponding target measurement data in response to the current data request; the target parameter is a parameter matched with the current data request; the target measurement data are measurement data of which the measurement area and the measurement time are matched with the current data request in the total measurement data corresponding to the target parameters; the total measurement data corresponding to the target parameters are obtained by measuring a plurality of different types of observation instruments; acquiring the target measurement data from an original database, and processing the target measurement data based on a primary verification rule base to obtain a primary verification result base; processing the measurement data in the initial verification result library based on a fine verification rule library to obtain a fine verification result library; and determining target distribution data based on the quality scores of the measurement data in the fine verification result library, and transmitting the target distribution data to a target user corresponding to the current data request.
Further, the logic instructions in the memory 303 may be implemented in the form of software functional units and stored in a computer readable storage medium when sold or used as a stand alone product. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, an optical disk, or other various media capable of storing program codes.
In another aspect, the present application also provides a computer program product comprising a computer program, the computer program being storable on a non-transitory computer readable storage medium, the computer program, when executed by a processor, being capable of performing a method of processing measurement data of a multi-type scope provided by the methods described above, the method comprising: determining a target parameter and corresponding target measurement data in response to the current data request; the target parameter is a parameter matched with the current data request; the target measurement data are measurement data of which the measurement area and the measurement time are matched with the current data request in the total measurement data corresponding to the target parameters; the total measurement data corresponding to the target parameters are obtained by measuring a plurality of different types of observation instruments; acquiring the target measurement data from an original database, and processing the target measurement data based on a primary verification rule base to obtain a primary verification result base; processing the measurement data in the initial verification result library based on a fine verification rule library to obtain a fine verification result library; and determining target distribution data based on the quality scores of the measurement data in the fine verification result library, and transmitting the target distribution data to a target user corresponding to the current data request.
In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform a method of processing measurement data of a multi-type scope provided by the methods described above, the method comprising: determining a target parameter and corresponding target measurement data in response to the current data request; the target parameter is a parameter matched with the current data request; the target measurement data are measurement data of which the measurement area and the measurement time are matched with the current data request in the total measurement data corresponding to the target parameters; the total measurement data corresponding to the target parameters are obtained by measuring a plurality of different types of observation instruments; acquiring the target measurement data from an original database, and processing the target measurement data based on a primary verification rule base to obtain a primary verification result base; processing the measurement data in the initial verification result library based on a fine verification rule library to obtain a fine verification result library; and determining target distribution data based on the quality scores of the measurement data in the fine verification result library, and transmitting the target distribution data to a target user corresponding to the current data request.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as a ROM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and are not limiting thereof; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the corresponding technical solutions.

Claims (10)

1. A method of processing measurement data for a multi-type scope, the method comprising:
determining a target parameter and corresponding target measurement data in response to the current data request; the target parameter is a parameter matched with the current data request; the target measurement data are measurement data of which the measurement area and the measurement time are matched with the current data request in the total measurement data corresponding to the target parameters; the total measurement data corresponding to the target parameters are obtained by measuring a plurality of different types of observation instruments;
acquiring the target measurement data from an original database, and processing the target measurement data based on a primary verification rule base to obtain a primary verification result base;
Processing the measurement data in the initial verification result library based on a fine verification rule library to obtain a fine verification result library;
and determining target distribution data based on the quality scores of the measurement data in the fine verification result library, and transmitting the target distribution data to a target user corresponding to the current data request.
2. The method for processing measurement data of a multi-type observation instrument according to claim 1, wherein the primary verification rule base includes fusion rules of similar heterogeneous data and screening rules of invalid data, and the processing of the target measurement data based on the primary verification rule base to obtain a primary verification result base specifically includes:
preprocessing target measurement data corresponding to each target parameter based on the fusion rule of the similar heterogeneous data to obtain target measurement data sets corresponding to each target parameter respectively;
screening the invalid data in the target measurement data sets corresponding to the target parameters respectively based on the screening rule of the invalid data to obtain effective target measurement data sets corresponding to the target parameters respectively;
and generating a primary verification result library based on the effective target measurement data sets respectively corresponding to the target parameters.
3. The method for processing measurement data of a multi-type observation instrument according to claim 2, wherein the filtering rule based on the invalid data filters the invalid data in the target measurement data set corresponding to each target parameter, and specifically includes:
screening first invalid data in target measurement data sets corresponding to all target parameters respectively based on a preset parameter value standard fluctuation range to obtain potential valid target measurement data sets corresponding to all target parameters respectively;
and screening second invalid data in the potential valid target measurement data sets corresponding to the target parameters respectively based on the parameter value prediction curve updated in real time to obtain valid target measurement data sets corresponding to the target parameters respectively.
4. The method for processing measurement data of a multi-type observation instrument according to claim 3, wherein the processing of the measurement data in the initial verification result library based on the fine verification rule library to obtain a fine verification result library specifically comprises:
correcting the measured data in the effective target measured data sets corresponding to the target parameters respectively based on a preset precision lifting algorithm to obtain corrected target measured data sets corresponding to the target parameters respectively;
And generating a fine verification result library based on the corrected target measurement data sets respectively corresponding to the target parameters.
5. The method for processing measurement data of a multi-type observation instrument according to claim 4, wherein the determining target distribution data based on the quality scores of the measurement data in the fine verification result library specifically comprises:
dividing correction target measurement data sets corresponding to all target parameters in a fine verification result library into a plurality of correction target measurement data subsets based on a preset minimum statistical time length;
and for any correction target measurement data set, determining quality scores of a plurality of correction target measurement data subsets corresponding to the correction target measurement data set, and taking the correction target measurement data subset with the quality scores not lower than a preset threshold value as target distribution data.
6. The method of claim 5, wherein the quality score for each of the modified target measurement data subsets is determined based on the effective data duty cycle and the absence rate of the modified target measurement data subset.
7. The method according to claim 6, wherein the current data request includes target parameters, measurement areas, measurement time, and indication information of a target user.
8. A measurement data processing apparatus for a multi-type scope, the apparatus comprising:
the target measurement data determining unit is used for determining target parameters and corresponding target measurement data in response to the current data request; the target parameter is a parameter matched with the current data request; the target measurement data are measurement data of which the measurement area and the measurement time are matched with the current data request in the total measurement data corresponding to the target parameters; the total measurement data corresponding to the target parameters are obtained by measuring a plurality of different types of observation instruments;
the primary verification unit is used for acquiring the target measurement data from the original database and processing the target measurement data based on a primary verification rule base to obtain a primary verification result base;
the fine verification unit is used for processing the measurement data in the initial verification result library based on the fine verification rule library to obtain a fine verification result library;
and the distribution unit is used for determining target distribution data based on the quality scores of the measured data in the fine verification result library and transmitting the target distribution data to a target user corresponding to the current data request.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method for processing measurement data of a multi-type scope according to any one of claims 1 to 7 when the program is executed by the processor.
10. A non-transitory computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method for processing measured data of a multi-type scope according to any one of claims 1 to 7.
CN202311685187.9A 2023-12-11 2023-12-11 Method and device for processing measurement data of multi-type observation instrument Active CN117390008B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311685187.9A CN117390008B (en) 2023-12-11 2023-12-11 Method and device for processing measurement data of multi-type observation instrument

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311685187.9A CN117390008B (en) 2023-12-11 2023-12-11 Method and device for processing measurement data of multi-type observation instrument

Publications (2)

Publication Number Publication Date
CN117390008A true CN117390008A (en) 2024-01-12
CN117390008B CN117390008B (en) 2024-04-12

Family

ID=89466923

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311685187.9A Active CN117390008B (en) 2023-12-11 2023-12-11 Method and device for processing measurement data of multi-type observation instrument

Country Status (1)

Country Link
CN (1) CN117390008B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117892206A (en) * 2024-03-15 2024-04-16 深圳市志奋领科技有限公司 Method and device for managing moisture content measurement model, electronic equipment and storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018098721A1 (en) * 2016-11-30 2018-06-07 深圳益强信息科技有限公司 Environment data monitoring method and system
CN111506876A (en) * 2020-04-02 2020-08-07 沈阳数云科技有限公司 Data prediction analysis method, system, equipment and readable storage medium
CN112905560A (en) * 2021-02-02 2021-06-04 中国科学院地理科学与资源研究所 Air pollution prediction method based on multi-source time-space big data deep fusion
CN113869571A (en) * 2021-09-22 2021-12-31 国网天津市电力公司电力科学研究院 Smart energy building comprehensive information physical fusion method based on heterogeneous data
CN115309734A (en) * 2022-08-26 2022-11-08 广东电网有限责任公司东莞供电局 Multi-source heterogeneous data processing method for transformer substation
CN116340437A (en) * 2023-03-24 2023-06-27 南京邮电大学 Multi-clustering method for large-scale multi-source heterogeneous data

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018098721A1 (en) * 2016-11-30 2018-06-07 深圳益强信息科技有限公司 Environment data monitoring method and system
CN111506876A (en) * 2020-04-02 2020-08-07 沈阳数云科技有限公司 Data prediction analysis method, system, equipment and readable storage medium
CN112905560A (en) * 2021-02-02 2021-06-04 中国科学院地理科学与资源研究所 Air pollution prediction method based on multi-source time-space big data deep fusion
CN113869571A (en) * 2021-09-22 2021-12-31 国网天津市电力公司电力科学研究院 Smart energy building comprehensive information physical fusion method based on heterogeneous data
CN115309734A (en) * 2022-08-26 2022-11-08 广东电网有限责任公司东莞供电局 Multi-source heterogeneous data processing method for transformer substation
CN116340437A (en) * 2023-03-24 2023-06-27 南京邮电大学 Multi-clustering method for large-scale multi-source heterogeneous data

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117892206A (en) * 2024-03-15 2024-04-16 深圳市志奋领科技有限公司 Method and device for managing moisture content measurement model, electronic equipment and storage medium

Also Published As

Publication number Publication date
CN117390008B (en) 2024-04-12

Similar Documents

Publication Publication Date Title
CN117390008B (en) Method and device for processing measurement data of multi-type observation instrument
CN102521560B (en) Instrument pointer image identification method of high-robustness rod
WO2019214309A1 (en) Model test method and device
CN112446397B (en) Grass yield estimation method, device and storage medium based on remote sensing and random forest
KR101953558B1 (en) Apparatus and Method for Fault Management of Smart Devices
CN113114530A (en) Network element health state detection method and equipment
CN114219847A (en) Method and system for determining crop planting area based on phenological characteristics and storage medium
CN113065538A (en) Pressure sensor detection method, device and equipment based on image recognition
CN113624694A (en) Inversion method and device for atmospheric methane concentration
CN111541986A (en) Positioning method, positioning device, storage medium and processor
CN117710833B (en) Mapping geographic information data acquisition method and related device based on cloud computing
CN110096335B (en) Service concurrency prediction method for different types of virtual machines
CN117630000B (en) Soil salt content evaluation method and device, electronic equipment and storage medium
CN108830444B (en) Method and device for evaluating and correcting sounding observation data
CN118068887A (en) Pasture planting humidity control method
CN109982368B (en) Method, device, equipment and medium for checking cell azimuth
CN112241981A (en) Method and device for verifying precision of secondary measurement data of crop planting area
CN116994126A (en) Crop leaf area index obtaining method and device based on canopy reflectivity spectrum
CN116379935A (en) Island shallow sea water depth prediction method, device and equipment based on multispectral image
CN111225440A (en) Cooperative positioning method and device and electronic equipment
CN112733906B (en) Global high-resolution daily earth surface momentum roughness length estimation method
CN114723144A (en) Wind measurement data correction method and device and electronic equipment
CN114067307A (en) Transformer substation instrument scale calibration method and system
CN111337139A (en) Temperature measuring device and method for measuring temperature
CN113098971A (en) Electronic blood pressure counting data transmission monitoring system based on internet

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant