CN201993755U - Data filtration, compression and storage system of real-time database - Google Patents
Data filtration, compression and storage system of real-time database Download PDFInfo
- Publication number
- CN201993755U CN201993755U CN 201120031612 CN201120031612U CN201993755U CN 201993755 U CN201993755 U CN 201993755U CN 201120031612 CN201120031612 CN 201120031612 CN 201120031612 U CN201120031612 U CN 201120031612U CN 201993755 U CN201993755 U CN 201993755U
- Authority
- CN
- China
- Prior art keywords
- data
- compression
- filter
- cache device
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Lifetime
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The utility model relates to a data filtration, compression and storage system of a real-time database. The data filtration, compression and storage system has higher compression efficiency and can improve the read-write performance of a hard disk during data persistence. The data filtration, compression and storage system comprises a level-1 cache memory, a compression filter, a level-2 cache memory and a data storage device which are sequentially connected; the level-1 cache memory is used for buffer memory of data from a data source; the compression filter is used for performing data compression on the data in the level-1 cache memory, and important data in the level-2 cache memory is stored by the compression filter; and the important data is stored in the data storage device by the level-2 cache memory when an archiving condition is triggered.
Description
Technical field
The utility model relates to data processing, especially relates to a kind of data filter compression storage system of real time data.
Background technology
Real-time data base (Real Time DataBase, RTDB) be used for the service data of harvester, grasp the operation conditions of device, and the critical data of production run is monitored and analyzed, the problem that occurs is in time handled, historical data is analysed scientifically, make the production run state steady, the material supply balance reduces unit consumption, increase economic efficiency, reduce cost.
Minority u s company is monopolizing industrial real-time data base field at present.Their valuable product, usually only at large-scale huge enterprise, the needed real-time data base products quotation of medium-sized enterprise Renminbi up to a million possibly, the deployment cost of great number becomes process control automation with informationalized threshold, has restricted the development of domestic medium and small sized enterprises.
Data compression method is one of core methed of real-time data base.In data compression process, data behind the buffer memory, through overcompression, store hard disk into then in internal memory after gathering from data source.In this course, whether the design of compressibility is reasonable, will have influence on streaming rate and compression efficiency.
The utility model content
Technical problem to be solved in the utility model provides a kind of data filter compression storage system of real-time data base, and it has higher compression efficiency, and makes disk read-write performance when improving data persistence.
The utility model is to solve the problems of the technologies described above the data filter compression storage system that the technical scheme that adopts is a kind of real-time data base of proposition, comprises the level cache device, compression filter, L2 cache device and the data storage device that connect successively; This level cache device is in order to the data of buffer memory from a data source; This compression filter is in order to carrying out data compression to the data in the level cache device, and the storage significant data is to this L2 cache device; These L2 cache device structured storage data, and when the filing condition is triggered, store this significant data to this data storage device.
In the data filter compression storage system of above-mentioned real-time data base, also comprise an event filter, connect this level cache device, this event filter filters the data from a data source, and trust data is stored to this level cache device, to improve compression algorithm performance and reliability.
In the data filter compression storage system of above-mentioned real-time data base, this L2 cache device comprises an ephemeral data buffer, carries out the request of this compression filter of data compression in order to response.
In the data filter compression storage system of above-mentioned real-time data base, this level cache device is to be made of nonvolatile memory.
In the data filter compression storage system of above-mentioned real-time data base, this L2 cache device is to be made of nonvolatile memory.
In the data filter compression storage system of above-mentioned real-time data base, this data storage device is to be made of hard disk.
The utility model makes it compared with prior art owing to adopt above technical scheme, by the L2 cache device is set, improves data persistence efficient and retrieval performance.And, the interim parameter storage device that the L2 cache device is included, the data used in compression process of store compressed filtrator so that the response compression filter is asked fast, improve compression efficiency temporarily.
Description of drawings
For above-mentioned purpose of the present utility model, feature and advantage can be become apparent, below in conjunction with accompanying drawing embodiment of the present utility model is elaborated, wherein:
Fig. 1 illustrates the data filter compression memory system architecture of the industrial real-time data base of the utility model one embodiment.
Fig. 2 illustrates the data compression flow process of the industrial real-time data base of the utility model one embodiment.
Fig. 3 illustrates the data compression flow process of the industrial real-time data base of another embodiment of the utility model.
Fig. 4 A-4C illustrates the compression algorithm principle of the utility model one embodiment.
Embodiment
Fig. 1 illustrates the data filter compression storage system according to the industrial real-time data base of the utility model one embodiment.With reference to shown in Figure 1, system comprises event filter 20, level cache device 30, compression filter 40, L2 cache device 50 and data storage device 60.Level cache device 30 and L2 cache device 50 can be by volatile memory, and as the internal memory formation of computing machine, 60 of data storage devices can be by persistent storage medium, and for example hard disc of computer constitutes.
Data in the level cache device 30 will further be filtered through compression filter 40, and significant data is deposited in the L2 cache device 50.Compression filter 40 also can empty replacement to level cache device 30.Significant data in the L2 cache device 50 becomes a plurality of historical datas through filing, after merging, is kept at enduringly in the data storage device 60.The actual result who implements proves that L2 cache result 50 setting helps making the data structureization after overcompression, is convenient to follow-up storage and retrieval.In addition, L2 cache device 50 comprises an interim parameter storage, with the data that interim store compressed filtrator 40 uses in compression process, asks fast so that respond compression filter 40.
In the practical application, the data of hardware collection are comprising a large amount of noises, and these factors all affect the waveform of former data, if effectively do not control, can cause efficiency of algorithm to seriously influence.Therefore, before data enter compression process, must carry out suitable filtration.Before data enter compression filter 50, need earlier through event filter 30.
The task of event filter 30 is that according to the attribute of data self and the predefine of data source, whether determination data receives.The following content of main execution: (1) is worth event filtering: the difference of the numerical value that new data and last time receive is greater than the abnormal variation value of regulation, and new data and mistiming of receiving data last time are more than or equal to the minimum time value of regulation, then received, delivered to compression filter.(2) carrying out time-event filters: the difference of the temporal information of the data that the temporal information of new data and last time receive is then received more than or equal to the maximum time value of regulation, delivers to compression filter.If the time interval of new data and last data, new data will not receive less than regulation minimum time value.
It is abnormal variation (being used for the value incident) and minimax time (being used for temporal filtering) that event filter 30 relates to main parameter.
Through unusual filtered data, raw data noise rate reduces (though can not determine eliminate noise fully) greatly relatively, plays crucial effects for the operate as normal of compression filter 40.
In context of the present utility model, the continuous recording amount of a certain needs is called " data ", and the result that " data " are repeatedly write down is called " numerical value " or " data point ".At this, the data compression algorithm in the compression filter 40 is that a plurality of numerical value or the data point of the same data of continuous maintenance are screened, and judges which numerical value needs to preserve, and which numerical value can abandon.Compression filter 40 is based on the theory of " removing the adjacent not obvious data item of variation ", but takes improved data compression method.An advantage of this method is to solve in " revolving door " compression algorithm flow process the problem that temporary data set is excessive.
According to an embodiment of the present utility model, propose " slope relative method " and replace traditional temporary data set method.Fig. 4 A-4C illustrates the compression algorithm principle of the utility model one embodiment.In Fig. 4 A-4C, the A point is last savepoint, and the D point is the latest data point, and the B point is the point of slope maximum between A point and D point, and the C point is the point of slope minimum between A point and D point.
In " the slope relative method " of the utility model one embodiment, the minimum and maximum numerical value of only interim storage slope.If the slope that new numerical value and a last storage numerical value form need not carry out other tests between minimum and maximum slope; If the slope that new numerical value and a last storage numerical value form is outside minimum and maximum slope, only need the numerical value of test maximum slope and minimum slope whether to drop on to have stored in the parallelogram that numerical value and new numerical value forms, thus decision storage current data point (if it is outer to drop on parallelogram) or continue to accept new data point (if it is interior to drop on parallelogram).
For realizing above-mentioned method, make up an interim parameter storage.This interim parameter storage can be structured in the L2 cache device 50 shown in Figure 1.At this, each observed reading is the single data in the individual data source that obtains of single pass.The memory contents of interim parameter storage comprises:
(1) the nearest storage numerical value of each corresponding point of buffer memory and last one is observed numerical value, and these two values are open to user and other assemblies.
(2) the maximum slope Kmax of each data of buffer memory, minimum slope Kmin, and record obtains the observed reading U of Kmax and Kmin
Kmax, U
Kmin
Fig. 2 illustrates the data compression method flow process of the industrial real-time data base of the utility model one embodiment.The flow process of slope relative method is described below in conjunction with Fig. 2:
At first, accept a data point at step S101;
At step S102, carry out event filtering via an event filter 10, if insincere data then abandon data at step S103, otherwise enter step S104.
At step S104, judge whether to receive this point data, if not, show it is to receive this data for the first time, at step S105, this point data exists in the interim parameter storage, returns step S101 then.If, then enter step S106, judge whether maximum slope Kmax and minimum slope Kmin are empty, if be empty,, utilize this point and a last memory point to calculate maximum slope Kmax and minimum slope Kmin then at step S107, return step S101 then.
At step S108, utilize minimum and maximum slope point U
Kmax, U
KminCalculating K 3, K4, computing method are as follows:
K3＝(U
Kmax-(M+ΔC))/Δt2；
K4＝(U
Kmin-(M-ΔC))/Δt3。
Wherein M is a last storage numerical value, and Δ C is the compression deviation, and Δ t2, Δ t3 are two numerical value U to be calculated
KmaxWith Δ C, and U
KminAnd the time interval between the Δ C.
At step S109, calculate the slope K of current point.
K＝(V-M)/Δt1。
Wherein Δ t1 is the time interval between V and the M, and V is current numerical value.
At step S110, if the slope K of current point satisfies: K3≤K≤K4, judge then whether K is contained in [Kmin, Kmax], if Kmin＞K then makes Kmin=K upgrade Ukmin=V simultaneously, if K＞Kmax then makes Kmax=K and upgrades Ukmax=V.
At step S111, current point does not satisfy preservation condition, does not preserve current point.But being that data are observed needs, and current point is write on one in the observed reading register, for other programs or flow process use.
On the contrary, if in the slope K of the current point of step S112 not in [K3, K4] interval, then preserve current data point to L2 cache device 50, upgrade up-to-date storage numerical value simultaneously.
At step S113, judge whether to trigger 50 filings of L2 cache device, if then file at step S114.
In any case flow process is handled next data at step S115, turn back to step S101.
In embodiment of the present utility model, the condition that triggers filing can be: time interval arrives, and L2 cache device 50 is full, system's generation great change etc.
As the variation example of present embodiment, in Fig. 3,, directly current data point is stored in the data storage device at step S112 ', enter step S115 then.
When the benefit of new algorithm is the new data point arrival, by comparing the choice that slope comes the determination number strong point, only keep the value of two minimum and maximum data points of slope, buffer size is constant, can not cause the internal memory waste and overflow.The time series of industry real time data has certain waveform rule (as sine wave) usually, experiment shows that the compressibility of industrial real time data is all higher, such data characteristic has determined temporary data set that certain scale is generally all arranged, and the decision operation efficient compared on the certain scale temporary data set of the compare operation of slope is higher so.Compression experiment for mass data has proved that also the performance of new algorithm has advantage, thus the more successful solution of embodiment of the present utility model the left problem of tradition " revolving door " compression algorithm.
Though the utility model discloses as above with preferred embodiment; right its is not in order to limit the utility model; any those skilled in the art; in not breaking away from spirit and scope of the present utility model; when doing a little modification and perfect, therefore protection domain of the present utility model is worked as with being as the criterion that claims were defined.
Claims (6)
1. the data filter of real-time data base compression storage system is characterized in that, comprises the level cache device, compression filter, L2 cache device and the data storage device that connect successively; This level cache device is in order to the data of buffer memory from a data source; This compression filter is in order to carrying out data compression to the data in the level cache device, and the storage significant data is to this L2 cache device; These L2 cache device structured storage data, and when the filing condition is triggered, store this significant data to this data storage device.
2. the data filter of real-time data base as claimed in claim 1 compresses storage system, it is characterized in that, also comprise an event filter, connect this level cache device, this event filter filters the data from a data source, and trust data is stored to this level cache device.
3. the data filter of real-time data base as claimed in claim 1 compression storage system is characterized in that this L2 cache device comprises an interim parameter storage, carries out the request of this compression filter of data compression in order to response.
4. the data filter of real-time data base as claimed in claim 1 compression storage system is characterized in that this level cache device is to be made of nonvolatile memory.
5. the data filter of real-time data base as claimed in claim 1 compression storage system is characterized in that this L2 cache device is to be made of nonvolatile memory.
6. the filtration of real-time data base as claimed in claim 1 compression storage system is characterized in that this data storage device is to be made of hard disk.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201120031612 CN201993755U (en) | 2011-01-30 | 2011-01-30 | Data filtration, compression and storage system of real-time database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201120031612 CN201993755U (en) | 2011-01-30 | 2011-01-30 | Data filtration, compression and storage system of real-time database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN201993755U true CN201993755U (en) | 2011-09-28 |
Family
ID=44670238
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201120031612 Expired - Lifetime CN201993755U (en) | 2011-01-30 | 2011-01-30 | Data filtration, compression and storage system of real-time database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN201993755U (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102664633A (en) * | 2012-04-16 | 2012-09-12 | 天津市英贝特航天科技有限公司 | Real-time acquisition, compression and memory method for data in system |
| CN102929799A (en) * | 2012-10-17 | 2013-02-13 | 北京西塔网络科技股份有限公司 | Data collection and storage method and system |
| CN103020151A (en) * | 2012-11-22 | 2013-04-03 | 用友软件股份有限公司 | Large data quantity batch processing system and large data quantity batch processing method |
| CN103207887A (en) * | 2013-01-14 | 2013-07-17 | 广州佳都信息技术研发有限公司 | Efficient historical trend data storage method |
| CN103514112A (en) * | 2012-06-29 | 2014-01-15 | 浙江大华技术股份有限公司 | Data storage method and system |
| CN104682962A (en) * | 2015-02-09 | 2015-06-03 | 南京邦耀科技发展有限公司 | Compression method for massive fuel gas data |
| CN105095421A (en) * | 2015-07-14 | 2015-11-25 | 南京国电南自美卓控制系统有限公司 | Distributed storage method for real-time database |
| CN106156038A (en) * | 2015-03-26 | 2016-11-23 | 腾讯科技（深圳）有限公司 | Date storage method and device |
| CN106354774A (en) * | 2016-08-22 | 2017-01-25 | 东北大学 | Real-time industrial process big data compression and storage system and method |
| CN109271206A (en) * | 2018-08-24 | 2019-01-25 | 晶晨半导体(上海)股份有限公司 | A kind of memory compression and store method that exception is live |
| CN112182034A (en) * | 2019-07-03 | 2021-01-05 | 河南许继仪表有限公司 | Data compression method and device |
| CN114676102A (en) * | 2022-03-10 | 2022-06-28 | 中国船舶重工集团公司第七一一研究所 | Database control method and control system |
-
2011
- 2011-01-30 CN CN 201120031612 patent/CN201993755U/en not_active Expired - Lifetime
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102664633A (en) * | 2012-04-16 | 2012-09-12 | 天津市英贝特航天科技有限公司 | Real-time acquisition, compression and memory method for data in system |
| CN102664633B (en) * | 2012-04-16 | 2015-05-20 | 天津市英贝特航天科技有限公司 | Real-time acquisition, compression and memory method for data in system |
| CN103514112B (en) * | 2012-06-29 | 2016-11-02 | 浙江大华技术股份有限公司 | A kind of date storage method and system |
| CN103514112A (en) * | 2012-06-29 | 2014-01-15 | 浙江大华技术股份有限公司 | Data storage method and system |
| CN102929799A (en) * | 2012-10-17 | 2013-02-13 | 北京西塔网络科技股份有限公司 | Data collection and storage method and system |
| CN102929799B (en) * | 2012-10-17 | 2016-04-13 | 北京西塔网络科技股份有限公司 | Data acquisition storage means and system |
| CN103020151B (en) * | 2012-11-22 | 2015-12-02 | 用友网络科技股份有限公司 | Big data quantity batch processing system and big data quantity batch processing method |
| CN103020151A (en) * | 2012-11-22 | 2013-04-03 | 用友软件股份有限公司 | Large data quantity batch processing system and large data quantity batch processing method |
| CN103207887A (en) * | 2013-01-14 | 2013-07-17 | 广州佳都信息技术研发有限公司 | Efficient historical trend data storage method |
| CN104682962A (en) * | 2015-02-09 | 2015-06-03 | 南京邦耀科技发展有限公司 | Compression method for massive fuel gas data |
| CN106156038A (en) * | 2015-03-26 | 2016-11-23 | 腾讯科技（深圳）有限公司 | Date storage method and device |
| CN106156038B (en) * | 2015-03-26 | 2019-11-29 | 腾讯科技（深圳）有限公司 | Date storage method and device |
| CN105095421A (en) * | 2015-07-14 | 2015-11-25 | 南京国电南自美卓控制系统有限公司 | Distributed storage method for real-time database |
| CN106354774A (en) * | 2016-08-22 | 2017-01-25 | 东北大学 | Real-time industrial process big data compression and storage system and method |
| CN109271206A (en) * | 2018-08-24 | 2019-01-25 | 晶晨半导体(上海)股份有限公司 | A kind of memory compression and store method that exception is live |
| CN112182034A (en) * | 2019-07-03 | 2021-01-05 | 河南许继仪表有限公司 | Data compression method and device |
| CN114676102A (en) * | 2022-03-10 | 2022-06-28 | 中国船舶重工集团公司第七一一研究所 | Database control method and control system |
| CN114676102B (en) * | 2022-03-10 | 2024-05-03 | 中国船舶集团有限公司第七一一研究所 | Database control method and control system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN201993755U (en) | Data filtration, compression and storage system of real-time database | |
| CN103595416B (en) | Compress creation data stream and with the system and method for various criterion filtered compressed data | |
| CN104102737B (en) | A kind of historical data storage method and system | |
| CN105630885A (en) | Abnormal power consumption detection method and system | |
| CN112730938A (en) | Electricity stealing user judgment method based on electricity utilization collection big data | |
| CN104699807A (en) | Automatic monitoring and expansion method for ORACLE data table space | |
| CN101894319A (en) | Tobacco enterprise data quality management system and method | |
| CN105278879A (en) | Method and device for processing monitoring data | |
| CN103310375A (en) | Intelligent skynet behavior audit analyzing system | |
| CN102622367B (en) | Method for filtering and compressing process data | |
| CN105955876A (en) | Data monitoring processing method and apparatus | |
| CN102779138A (en) | Hard disk access method of real time data | |
| CN103353883A (en) | Big data stream type cluster processing system and method for on-demand clustering | |
| CN117453137A (en) | Cloud intelligent operation and maintenance system data management system | |
| CN118152378B (en) | Construction method and system of intelligent data center | |
| CN111898068A (en) | Anomaly detection method based on PERT algorithm and instrument usage analysis | |
| CN102332004A (en) | Data processing method and system for managing mass data | |
| CN110688846B (en) | Periodic word mining method, system, electronic equipment and readable storage medium | |
| CN110162513A (en) | Tables of data connection method and computer readable storage medium for the processing of smart grid big data | |
| CN111752727B (en) | Log analysis-based three-layer association recognition method for database | |
| CN116450734B (en) | Distributed storage method for development and construction digital twin data of industrial park | |
| CN104484375A (en) | Method and system for automatically building database in item analysis process | |
| CN109165212A (en) | Big data real-time monitoring and auditing method | |
| CN102779097B (en) | Memory access method of flow data | |
| CN102567471B (en) | Information gain computation based mass data abnormality detecting method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CX01 | Expiry of patent term | ||
| CX01 | Expiry of patent term |
Granted publication date: 20110928 |