CN106202566A - A kind of magnanimity electricity consumption data mixing based on big data storage system and method - Google Patents
A kind of magnanimity electricity consumption data mixing based on big data storage system and method Download PDFInfo
- Publication number
- CN106202566A CN106202566A CN201610629218.2A CN201610629218A CN106202566A CN 106202566 A CN106202566 A CN 106202566A CN 201610629218 A CN201610629218 A CN 201610629218A CN 106202566 A CN106202566 A CN 106202566A
- Authority
- CN
- China
- Prior art keywords
- data
- distributed memory
- power information
- electricity consumption
- acquisition unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
nullA kind of magnanimity electricity consumption data mixing storage system and method giving big data，Including data acquisition unit、Power information acquisition system、Power information processing system，Wherein power information acquisition system includes that collection destroies subsystem and mass data platform，Power information processing system includes data prediction device、Data Post device、Distributed memory and data query server，Wherein power information acquisition system is connected with data acquisition unit and power information processing system respectively，Subsystem copied by collection，Mass data platform，Data prediction device and data after-treatment device are sequentially connected with，Data prediction device and data after-treatment device are bi-directionally connected with distributed memory respectively，Distributed memory is connected with data query server，Can be quick、Efficiently、Process magnanimity real time data in time，Ensure equipment safety simultaneously、Stable、Run efficiently.
Description
Technical field
The present invention relates to electricity consumption data analysis application field, be specifically related to a kind of magnanimity electricity consumption data based on big data
Mixing storage system and method.
Background technology
Along with developing rapidly of computer technology, the data of every profession and trade increase rapidly, data quantitative change increasing, type
Also getting more and more, data structure also tends to complicate, and the most each equipment of traditional data base is independently placed, and needs bigger
Space for its deployment, exists and is difficult to shortcomings such as disposing, relatively costly, it is impossible to meet the general requirement of user.
Time series data is the time series data of band time tag, and its typical feature is that generation frequency is fast, depends critically upon
Acquisition time, measuring point multiple data quantity are big.In power industry, in order to ensure that equipment is safe and stable, run efficiently, it will usually
Monitoring the running status of the various kinds of equipment such as generating, power transformation in real time, gathering the substantial amounts of time series data of acquisition can be as equipment
The basis of the senior application such as running status assessment, equipment operation failure early warning, equipment dependability analysis, thus, the most quickly, high
Effect, in time process one that the weight assets industry such as magnanimity real time data, always electric power, chemical industry, oil, iron and steel faces great
Problem.
In power industry, history service data collection and analysis, real-time or near-realtime data instant analysis are power industries
Content important during middle informatization, it needs complete set, stablizes, agrees with the big data of practical business scene
The solution of analytical equipment, props up reliable and stable bottom datas of analysis classes business scenario offer in real time such as equipment fault early-warnings
Support.
In recent years, along with IT technology fast developments such as cloud computing, big data, machine learning, data minings, distributed deposit
Storage, high-performance calculation all obtain key breakthrough in theoretical research and engineering practice aspect, industry emerged a collection of with
Hadoop is that the big data of representative process and application solution.
Hadoop is a distributed system architecture, including distributed file system HDFS (Hadoop
Distributed File System), distributed memory system HBase, parallel computation programming model MapReduce etc. several
Core, it can greatly simplify the processing procedure of large-scale data, and Spark is science and engineering at a kind of distributed big data
Tool, itself does not provide data storage function, and it may operate on HDFS or other the distributed file system of Hadoop,
The design original intention of Spark is contemplated to solve asking of Hadoop MapReduce reading and writing of files system repeatedly thus inefficiency
Topic, it supports that datarams is resident, it is achieved that In-memory by building elasticity distribution formula data set (RDD) structure
MapReduce framework, makes up the deficiency of MapReduce under application-specific scene.Skill that what Hadoop, Spark etc. were general increase income
Art assembly has some limitations in terms of functional completeness, operation stability, and based on some derivative for Hadoop commercializations
Deviation is there is again in big data platform with the actual demand of power business scene, thus, the business of depth analysis research power industry
Demand, builds a kind of magnanimity electricity consumption data mixing based on big data based on big data technique storage system and method, has
Profound significance and stronger value.
" dividing and rule " is the marrow of big data technique, and it improves the processing speed of data by parallel processing technique,
Design original intention is to be realized the parallel processing of big data by a large amount of servers at a low price, by traditional inquiry, statistics and data analysis
Carry out distributed treatment, process task is assigned to different process nodes, is derived from being substantially improved of process performance.
Introduce the distributed storage of big data, distributed computation ability comprehensively, for monitoring collection copy relevant time series data,
The result data that the user-dependent account data of electricity consumption, statistical analysis are relevant formulates special storage scheme respectively, in conjunction with actual industry
Business scenario building mixing storage system；Distributed Calculation module is transferred to after the preprocessing process of data, last handling process being peeled off
Perform, data processing complexity can not only be reduced, improve time series data and access handling capacity, it is also possible to break through mass memory, i.e.
The performance bottleneck of seat query aspects.
But, the function problem of comprehensive inquiry, short-term history aggregation of data cannot be performed at present for long history data
The performance issue of inquiry, the Cost Problems (cost of Oracle all-in-one is far above Hadoop cluster) of data storage is all thorny
Problem to be solved, builds a kind of magnanimity electricity consumption data mixing based on big data based on big data technique storage system and side
Method is imperative, and the enforcement of system can also provide data supporting for the structure of analysis mining class application.
Summary of the invention
It is an object of the invention to overcome the deficiencies in the prior art, it is provided that one can process sea quickly, efficiently, in time
Amount real time data, the magnanimity electricity consumption data mixing based on big data simultaneously ensure that equipment is safe and stable, running efficiently storage
System and method.
The invention provides a kind of magnanimity electricity consumption data mixing based on big data storage system, including data acquisition packaging
Put, power information acquisition system, power information processing system, wherein power information acquisition system includes that collection copies subsystem and magnanimity
Data platform, power information processing system includes data prediction device, Data Post device, distributed memory and data
Inquiry server, wherein power information acquisition system is connected with data acquisition unit and power information processing system respectively, and collection is copied
Subsystem, mass data platform, data prediction device and data after-treatment device are sequentially connected with, data prediction device sum
Being bi-directionally connected with distributed memory respectively according to after-treatment device, distributed memory is connected with data query server, wherein；
Data acquisition unit, obtains user power utilization Monitoring Data, and the user collected is used in real time or quasi real time
Pyroelectric monitor data are transferred to power information acquisition system；
Electricity consumption gathers information system, copies subsystem and mass data platform including collection, will use for copying subsystem by collection
Electricity consumption data acquisition process in family is also pushed to magnanimity information platform and stores, and in the way of propelling movement or streaming output
Mode, the user power utilization Monitoring Data after processing is pushed to distributed memory or output to data prediction device；
Data prediction device, for receiving, in the way of streaming access, the user sent from power information acquisition system
Electricity consumption monitoring data, or in the way of batch access, automatically obtain power information collection system by predefined operation plan
Conventional equipment account data in system and historical data, and it is stored in distributed memory；
Data Post device, for by the output data of data prediction device, to platform under different operational indicators
Account data and historical data filter and calculate, and enter account data and historical data according to the most programmed process logic
Row processes, and training forms data mining model, draws achievement data, and achievement data is passed back to distributed memory deposits
Storage；
Distributed memory, for by the account data after data prediction and Data Post, historical data and
Achievement data stores；
Data query server, for directly inquiring about data from distributed memory, to user power utilization data long history
Data perform comprehensive inquiry and the inquiry of short-term history aggregation of data.
Preferably, data acquisition unit includes that the power information being installed in monitoring device gathers sensor.
Preferably, data acquisition unit also include monitoring device installation region and/or temperature detector.
Preferably, data prediction device is additionally operable to be directly connected to data collection point and obtains user power utilization Monitoring Data.
Preferably, also include the manual input device being connected with power information acquisition system, for because of safety requirements in fact
Input Monitor Connector device data when having executed quarantine measures or do not supported data access.
Preferably, data prediction device is additionally operable to call and receive user's acquisition system propelling movement in time series data memorizer
The new time series data produced, and new time series data is repeated training process, data mining model is updated.
Preferably, the result after Data Post device processes is that power information predicts the outcome and/or load prediction results.
Preferably, manual input device is notebook computer, panel computer and/or mobile phone.
Present invention also offers a kind of magnanimity electricity consumption data mixing based on big data storage method, include successively walking as follows
Rapid:
(1) initialize, the initial parameter of data acquisition unit is set, control data acquisition according to the initial parameter set
The sampling period of device is 15 times per hour, and the sampling time is 7 days, and average A by the data of sampling in 7 days；
(2) under the conditions of same initial parameter, repeat step (1) 5 time, try to achieve the meansigma methods of 5 times respectively, delete 5 times
Two numerical value that middle meansigma methods is minimum and maximum, remaining meansigma methods of 3 times is designated as B, C, D；
(3) orderMake P' be data acquisition unit measure numerical value in real time, then:
If A.Then data acquisition unit stable performance, enters step (4)；
If B.Then data acquisition unit unstable properties, then enter step (1)；
(4) user power utilization Monitoring Data is obtained in real time or quasi real time, and the user power utilization Monitoring Data transmission that will collect
Store to the magnanimity information platform in power information acquisition system, in the way of propelling movement, be pushed in distributed memory,
Or in the way of streaming output, user power utilization Monitoring Data is exported to data prediction device；
(5) in the way of batch access, the routine in distributed memory is automatically obtained by predefined operation plan
Account data and historical data, carry out the cleaning of data by equipment account data and historical data with preprocessing rule, filter, turn
The pretreatment changed, and the output of pretreated data is stored to distributed memory；
(6) by recent Monitoring Data, conventional equipment account data, and the higher history achievement data of concern rate,
The internal memory being cached in distributed memory in model metadata and preprocessing rule data set；
(7) scheduling engine is driven to call and receive time series data memorizer by computing engines during Data Post
The data of storage, and according to the most programmed process logic, the data called and receive are processed, training forms number
According to mining model, by the data back after computing unit processes to distributed memory；
(8) directly read data from distributed memory and/or receive from the data of Data Post, and carrying out point
Analysis processes, and provides data supporting for excavating class data analysis；
(9) by data query server, traditional inquiry, statistics and data analysis are carried out distributed treatment, at general
Reason task is assigned to different process nodes.
Magnanimity electricity consumption data mixing based on the big data storage system and method for the present invention, it is possible to achieve:
1) with stable, reliable, increase income distributed memory system and parallel computation service efficiently as core, for monitoring collection
Copy relevant time series data, the user-dependent account data of electricity consumption, result data that statistical analysis is relevant are formulated special project respectively and are deposited
Storage scheme, in conjunction with practical business scenario building mixing storage system；Hand over after the preprocessing process of data, last handling process are peeled off
Performed by Distributed Calculation module, data processing complexity can not only be reduced, improve time series data access handling capacity, it is also possible to
Break through mass memory, the performance bottleneck of extemporaneous query aspects；
2) real-time and punctual collection data, ageing height, and optimization devise data acquiring frequency, collecting efficiency
Height, but the low usefulness of efficiency is high, and apparatus function is powerful, it is possible to and the function solving the execution comprehensive inquiry of long history data is asked
Topic, solves the performance issue of short-term history aggregation of data inquiry, solves the Cost Problems (one-tenth of Oracle all-in-one of data storage
This is far above Hadoop cluster), it is also possible to the structure for the application of analysis mining class provides data supporting；
3) for the reliability of system data, devise average data and confirm scheme so that user's Monitoring Data is more
Reliable and stable, alleviate the workload of device, service life is longer, and performance is more stable；
4) the data acquisition unit performances evaluation mode optimized so that data are more reliable.
Accompanying drawing explanation
Fig. 1 is magnanimity electricity consumption data mixing memory system architecture schematic diagrams based on big data
Fig. 2 is that magnanimity electricity consumption data mixing based on big data storage system adopts data enforcement data analysis figure
Detailed description of the invention
Being embodied as the following detailed description of the present invention, it is necessary to it is pointed out here that, below implement to be only intended to this
Bright further illustrates, it is impossible to be interpreted as limiting the scope of the invention, and this art skilled person is according to above-mentioned
Some nonessential improvement and adjustment that the present invention is made by summary of the invention, still fall within protection scope of the present invention.
The invention provides a kind of magnanimity electricity consumption data mixing based on big data storage system and method, such as accompanying drawing 1 institute
Show, including data acquisition unit 1, power information acquisition system 2, power information processing system 3, wherein power information acquisition system
2 include that collection destroies subsystem and mass data platform, and power information processing system 3 includes data prediction device 5, Data Post
Device 4, distributed memory 5 and data query server 7, wherein power information acquisition system 2 respectively with data acquisition unit 1
Connect with power information processing system 3.Collection copies subsystem, mass data platform, data prediction device 5, and Data Post fills
Putting 4 to be sequentially connected with, data prediction device 5 and data after-treatment device 4 are bi-directionally connected with distributed memory 6 respectively, distribution
Formula memorizer 6 is connected with data query server 6.
Data acquisition unit 1, obtains user power utilization Monitoring Data, and the user collected is used in real time or quasi real time
Pyroelectric monitor data are transferred to power information acquisition system 2, and data acquisition unit includes the power information being installed in monitoring device
Gather sensor, it is also possible to include the sensors such as the photographic head of monitoring device installation region, temperature detector, electricity consumption collection simultaneously
Information system can be real-time by user power utilization Monitoring Data store, and by propelling movement in the way of or with streaming output side
Formula, exports user power utilization Monitoring Data to distributed memory 6 or data prediction device 5.
Electricity consumption gathers information system, copies subsystem and mass data platform including collection, for by user power utilization data acquisition
And be pushed to magnanimity information platform (oracle) and store, and in the way of propelling movement or the mode of streaming output, will use
Power utilization information collection system data-pushing is to distributed memory 6 or exports to data prediction device 5.
Data prediction device 5, for receiving transmission in power information acquisition system in the way of streaming access
User power utilization Monitoring Data and in the way of batch access, obtains power information collection system automatically by predefined operation plan
Conventional equipment account data in system, and historical data, and be stored in distributed memory.Data prediction will collect
Data by the batch various ways such as accesss, streaming access, artificial importing acquisition, it is also possible to be directly connected to data collection point and obtain
Take family electricity consumption monitoring data.The data accessed before storing, can carry out the pretreatment of necessity, utilize pre-configured
The operations such as preprocessing rule is carried out, filters, conversion, data are through Data Integration or are directly stored in distributed memory
In, the high data of some access frequencys are waited for some rules, be typically stored at data cached in, for some history service numbers
According to, the data that access frequency is little, after data prediction, it is typically stored in business datum；For system definition one
The data such as a little data prediction are regular, computation rule, model data, are typically stored in configuration data；Either business datum
Still configuring data, there is bigger difference in its visiting frequency, performance requirement in concrete business scenario, for visiting frequency
The data high, performance requirement is high, system concentrated to be cached in Installed System Memory, these business datums being buffered and configuration
Data are referred to as data cached.It is said that in general, the higher history index of recent business datum, concern rate, model metadata, number
The data access frequency such as Data preprocess rule are higher, may be regarded as data cached.
Data Post device 4, for by the output data of data prediction device 5, right under different operational indicators
Account data and historical data filter and calculate, according to the most programmed process logic to account data and historical data
Processing, training forms data mining model, draws achievement data, and achievement data passes back to distributed memory carries out
Storage.Data Post can utilize the managerial experiences of industry specialists to combine Principle of Statistics and research and analyse mining algorithm, with
The historical data that family electricity consumption is relevant is input, and training forms data mining model, and different sample datas can form different numbers
According to mining model (example: monthly distinguish), the new time series data produced for user power utilization can repeat training process, enters
The sustained improvement of row data mining model；The data mining model created may participate in electricity consumption and gathers data, real-time or accurate real
Time mode analyze every evaluation index of power equipment and user power utilization information.The training process of mining model has related to batch
Amount calculates, and calculates operation by batch and realizes；The application process of mining model has related to streaming calculating, is calculated by streaming and makees
Industry realizes；Additionally, two kinds of computation schemas can be also used for realizing appraisal of equipment index, voice semantics recognition, text semantic analysis
Etc. the calculating task unrelated with mining model.
Batch is calculated operation and is driven by scheduling engine, reads in from power information acquisition system or distributed memory 5
Account data, business historical data, calculate according to the most programmed process logic, result of calculation can be written back to data
Memory block；It is also to be driven by scheduling engine that streaming calculates operation, and data are from power information acquisition system 2 or distributed memory
5 access in a streaming manner, calculate according to the most programmed process logic, and result of calculation can be written back to distributed
Memorizer 5.
Calculate operation be used for defining (also referred to as jobs node) topological structure of calculating task and perform logic, be similar to work
Stream (Workflow), its definition procedure can complete in the job design device that system provides, in terms of the visual angle of computing engines, each
Jobs node corresponds to a computing unit (Compute Unit), and the programmed logic that computing unit is corresponding is referred to as operator
(Transformation).System provides visual modeling tool, preset abundant data to process and data display operator, simultaneously
Open operator development specifications, supports the secondary development of practical business scene.
Distributed memory 6, also known as time series data memorizer, for the platform after data prediction and Data Post
The storage of account data, historical data and achievement data.Distributed Storage can provide basic guarantee for data query service.
The data base related in distributed memory 5 mainly has distributed file system HDFS (Hadoop Distributed File
System), columnar database HBase (Hadoop Database), memory database Redis, relational database Oracle etc..
Oracle database is mainly used in storage configuration data and partial service data, and HDFS is as the distribution of big data platform bottom
Formula file system elements, the HBASE for upper strata provides support, it is also possible to the directly non-sequential part in storage service data,
HBASE be a high reliability, high-performance, towards row, telescopic distributed memory system, be mainly used in storage service data
In time preamble section, Redis is that key-value based on internal memory stores system, is here mainly used in depositing caching number
According to.
Data query server 7, data query service directly inquires about data from distributed memory, for user power utilization number
Performing comprehensive inquiry according to long history data, short-term history aggregation of data is inquired about, and Query Result is analysis mining class data analysis
Thering is provided data supporting, its interaction is not related to data and calculates, and typical case's application scenarios is mainly comprehensive inquiry, visual presentation
Deng.
The present invention also provides for a kind of magnanimity electricity consumption data mixing storage method being based on big data, includes as follows successively
Step:
(1) initialize, the initial parameter of data acquisition unit is set, control data acquisition according to the initial parameter set
The sampling period of device is 15 times per hour, and the sampling time is 7 days, and average A by the data of sampling in 7 days；
(2) under the conditions of same initial parameter, repeat step (1) 5 time, try to achieve the meansigma methods of 5 times respectively, delete 5 times
Two numerical value that middle meansigma methods is minimum and maximum, remaining meansigma methods of 3 times is designated as B, C, D；
(3) orderMake P' be data acquisition unit measure numerical value in real time, then:
If A.Then data acquisition unit stable performance, enters step (4)；
If B.Then data acquisition unit unstable properties, then enter step (1)；
(4) user power utilization Monitoring Data is obtained in real time or quasi real time, and the user power utilization Monitoring Data transmission that will collect
Store to the magnanimity information platform in power information acquisition system, in the way of propelling movement, be pushed in distributed memory,
Or in the way of streaming output, user power utilization Monitoring Data is exported to data prediction device；
(5) in the way of batch access, the routine in distributed memory is automatically obtained by predefined operation plan
Account data and historical data, carry out the cleaning of data by equipment account data and historical data with preprocessing rule, filter, turn
The pretreatment changed, and the output of pretreated data is stored to distributed memory；
(6) by recent Monitoring Data, conventional equipment account data, and the higher history achievement data of concern rate,
The internal memory being cached in distributed memory in model metadata and preprocessing rule data set；
(7) scheduling engine is driven to call and receive time series data memorizer by computing engines during Data Post
The data of storage, and according to the most programmed process logic, the data called and receive are processed, training forms number
According to mining model, by the data back after computing unit processes to distributed memory；
(8) directly read data from distributed memory and/or receive from the data of Data Post, and carrying out point
Analysis processes, and provides data supporting for excavating class data analysis；
(9) by data query server, traditional inquiry, statistics and data analysis are carried out distributed treatment, at general
Reason task is assigned to different process nodes.
It is below specific embodiment:
Embodiment 1: calculate and extract the continuous low-voltage customer quantity that low voltage condition occurred for 7 days and detail (user's voltage
Value continues one hour less than 198V)
Embodiment 2: the low-voltage customer quantity of low voltage condition, detailed and involved platform occurred by monthly calculating extraction
District's quantity is with detailed.
Embodiment 3: by monthly calculate extraction occurred the magnitude of voltage of a point less than 198V low-voltage customer quantity, detail and
Involved platform district quantity is with detailed.
Embodiment 4: statistics the whole province's low-voltage customer rate of qualified voltage situation every day
Visible as in figure 2 it is shown, under conventional architectures and big data framework, by with adopting data query performance comparison, with adopting
Efficiency data query has had and has significantly promoted.
Magnanimity electricity consumption data mixing based on the big data storage system and method for the present invention is to be filled by software and hardware
The cooperation put completes, and but be not limited to that this, under certain condition, it is also possible to realize by the way of hardware completely.
Although for illustrative purposes, it has been described that the illustrative embodiments of the present invention, but those skilled in the art
Member, can be in form and details it will be appreciated that in the case of without departing from the scope and spirit of the invention disclosed in claims
On carry out various amendment, add and the change of replacement etc., and all these change all should belong to claims of the present invention
Each step in protection domain, and each department of claimed product and method, can in any combination
Form is combined.Therefore, the description to the embodiment that disclosed in this invention is not intended to limit the scope of the present invention,
But be used for describing the present invention.Correspondingly, the scope of the present invention is not limited by embodiment of above, but by claim or
Its equivalent is defined.
Claims (9)
1. magnanimity electricity consumption data mixing based on a big data storage system, it is characterised in that: include data acquisition unit, use
Power utilization information collection system, power information processing system, wherein power information acquisition system includes that collection copies subsystem and mass data
Platform, power information processing system includes data prediction device, Data Post device, distributed memory and data query
Server, wherein power information acquisition system is connected with data acquisition unit and power information processing system respectively, and subsystem copied by collection
System, mass data platform, data prediction device and data after-treatment device are sequentially connected with, after data prediction device and data
Processing means is bi-directionally connected with distributed memory respectively, and distributed memory is connected with data query server, wherein；
Data acquisition unit, obtains user power utilization Monitoring Data, and the user power utilization prison that will collect in real time or quasi real time
Survey data and be transferred to power information acquisition system；
Electricity consumption gathers information system, copies subsystem and mass data platform including collection, user is used for copying subsystem by collection
Electric data collecting processes and is pushed to magnanimity information platform and stores, and by propelling movement in the way of or streaming output side
Formula, the user power utilization Monitoring Data after processing is pushed to distributed memory or output to data prediction device；
Data prediction device, for receiving, in the way of streaming access, the user power utilization sent from power information acquisition system
Monitoring Data, or in the way of batch access, automatically obtained in power information acquisition system by predefined operation plan
Conventional equipment account data and historical data, and be stored in distributed memory；
Data Post device, for by the output data of data prediction device, to account number under different operational indicators
According to filtering with historical data and calculating, according to the most programmed process logic to account data and historical data at
Reason, training forms data mining model, draws achievement data, and achievement data is passed back to distributed memory stores；
Distributed memory, for by the account data after data prediction and Data Post, historical data and index
Data store；
Data query server, for directly inquiring about data from distributed memory, to user power utilization data long history data
Perform comprehensive inquiry and the inquiry of short-term history aggregation of data.
2. magnanimity electricity consumption data mixing storage system as claimed in claim 1, it is characterised in that: data acquisition unit includes peace
The power information being loaded in monitoring device gathers sensor.
3. magnanimity electricity consumption data mixing storage system as claimed in claim 1, it is characterised in that: data acquisition unit also includes
Monitoring device installation region and/or temperature detector.
4. magnanimity electricity consumption data mixing storage system as claimed in claim 1, it is characterised in that: data prediction device is also used
User power utilization Monitoring Data is obtained in being directly connected to data collection point.
5. magnanimity electricity consumption data mixing storage system as claimed in claim 1, it is characterised in that: also include adopting with power information
The manual input device that collecting system connects, for because safety requirements implements quarantine measures or does not support the situation of data access
Lower Input Monitor Connector device data.
6. magnanimity electricity consumption data mixing storage system as claimed in claim 1, it is characterised in that: data prediction device is also used
The new time series data produced is pushed in calling and receive user's acquisition system in time series data memorizer, and to new time ordinal number
According to repeating training process, data mining model is updated.
7. magnanimity electricity consumption data mixing storage system as claimed in claim 1, it is characterised in that: Data Post device processes
After result be that power information predicts the outcome and/or load prediction results.
8. magnanimity electricity consumption data mixing storage system as claimed in claim 5, it is characterised in that: manual input device is notes
This computer, panel computer and/or mobile phone.
9. the mixing utilizing the magnanimity electricity consumption data mixing storage system as described in the claims 1-8 stores a method,
It is characterized in that, in turn include the following steps:
(1) initialize, the initial parameter of data acquisition unit is set, control data acquisition unit according to the initial parameter set
Sampling period be 15 times per hour, the sampling time is 7 days, and average A by the data of sampling in 7 days；
(2) under the conditions of same initial parameter, repeat step (1) 5 time, try to achieve the meansigma methods of 5 times respectively, delete in 5 times flat
Two numerical value that average is minimum and maximum, remaining meansigma methods of 3 times is designated as B, C, D；
(3) orderMake P' be data acquisition unit measure numerical value in real time, then:
If A.Then data acquisition unit stable performance, enters step (4)；
If B.Then data acquisition unit unstable properties, then enter step (1)；
(4) obtain user power utilization Monitoring Data in real time or quasi real time, and be transferred to the user power utilization Monitoring Data collected use
Magnanimity information platform in power utilization information collection system stores, and in the way of propelling movement, is pushed in distributed memory, or
In the way of streaming output, user power utilization Monitoring Data is exported to data prediction device；
(5) in the way of batch access, the conventional account in distributed memory is automatically obtained by predefined operation plan
Data and historical data, carry out the cleaning of data by equipment account data and historical data with preprocessing rule, filter, change
Pretreatment, and the output of pretreated data is stored to distributed memory；
(6) by history achievement data higher to recent Monitoring Data, conventional equipment account data, and concern rate, model
The internal memory being cached in distributed memory in metadata and preprocessing rule data set；
(7) drive scheduling engine to call and receive time series data memorizer by computing engines during Data Post to store
Data, and the data called and receive are processed according to the most programmed process logic, training forms data and digs
Pick model, by the data back after computing unit processes to distributed memory；
(8) directly read data from distributed memory and/or receive the data from Data Post, and being analyzed place
Reason, provides data supporting for excavating class data analysis；
(9) by data query server, traditional inquiry, statistics and data analysis are carried out distributed treatment, process is appointed
Business is assigned to different process nodes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610629218.2A CN106202566A (en) | 2016-08-02 | 2016-08-02 | A kind of magnanimity electricity consumption data mixing based on big data storage system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610629218.2A CN106202566A (en) | 2016-08-02 | 2016-08-02 | A kind of magnanimity electricity consumption data mixing based on big data storage system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106202566A true CN106202566A (en) | 2016-12-07 |
Family
ID=57498362
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610629218.2A Pending CN106202566A (en) | 2016-08-02 | 2016-08-02 | A kind of magnanimity electricity consumption data mixing based on big data storage system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106202566A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106777395A (en) * | 2017-03-01 | 2017-05-31 | 北京航空航天大学 | A kind of topic based on community's text data finds system |
| CN106874482A (en) * | 2017-02-20 | 2017-06-20 | 山东鲁能软件技术有限公司 | A kind of device and method of the patterned data prediction based on big data technology |
| CN107273524A (en) * | 2017-06-23 | 2017-10-20 | 国网上海市电力公司 | A kind of intelligent power distribution big data application system |
| CN107483858A (en) * | 2017-08-31 | 2017-12-15 | 益和电气集团股份有限公司 | The distributed memory system and its distributed storage method of electricity consumption enterprise supervision video |
| CN107895017A (en) * | 2017-11-14 | 2018-04-10 | 国网江苏省电力公司电力科学研究院 | A kind of electric energy quality monitoring system construction method based on big data technology |
| CN108844572A (en) * | 2018-03-29 | 2018-11-20 | 深圳市晟达机械设计有限公司 | A kind of highway tunnel structure intelligent monitor system based on big data and WSN technology |
| CN110008213A (en) * | 2019-03-13 | 2019-07-12 | 国电南瑞科技股份有限公司 | A kind of regulator control system real time data separate type management method |
| CN111241087A (en) * | 2020-03-10 | 2020-06-05 | 天津市普迅电力信息技术有限公司 | Energy data automatic processing method based on distributed computing |
| CN111414355A (en) * | 2020-03-19 | 2020-07-14 | 中国能源建设集团广东省电力设计研究院有限公司 | Offshore wind farm data monitoring and storing system, method and device |
| CN113849508A (en) * | 2021-09-22 | 2021-12-28 | 广东电网有限责任公司 | Data storage method, device, equipment and medium |
| CN115242870A (en) * | 2022-06-23 | 2022-10-25 | 宁波三星医疗电气股份有限公司 | Electricity consumption data pushing method and device, server and computer readable storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103955509A (en) * | 2014-04-30 | 2014-07-30 | 广西电网公司电力科学研究院 | Quick search method for massive electric power metering data |
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| KR101597045B1 (en) * | 2014-09-03 | 2016-02-23 | (주)솔투로 | Apparatus and method for serial output by multi process distribution of big data |
| CN105427193A (en) * | 2015-12-17 | 2016-03-23 | 山东鲁能软件技术有限公司 | Device and method for big data analysis based on distributed time sequence data service |
-
2016
- 2016-08-02 CN CN201610629218.2A patent/CN106202566A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103955509A (en) * | 2014-04-30 | 2014-07-30 | 广西电网公司电力科学研究院 | Quick search method for massive electric power metering data |
| KR101597045B1 (en) * | 2014-09-03 | 2016-02-23 | (주)솔투로 | Apparatus and method for serial output by multi process distribution of big data |
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| CN105427193A (en) * | 2015-12-17 | 2016-03-23 | 山东鲁能软件技术有限公司 | Device and method for big data analysis based on distributed time sequence data service |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106874482A (en) * | 2017-02-20 | 2017-06-20 | 山东鲁能软件技术有限公司 | A kind of device and method of the patterned data prediction based on big data technology |
| CN106777395A (en) * | 2017-03-01 | 2017-05-31 | 北京航空航天大学 | A kind of topic based on community's text data finds system |
| CN107273524A (en) * | 2017-06-23 | 2017-10-20 | 国网上海市电力公司 | A kind of intelligent power distribution big data application system |
| CN107483858A (en) * | 2017-08-31 | 2017-12-15 | 益和电气集团股份有限公司 | The distributed memory system and its distributed storage method of electricity consumption enterprise supervision video |
| CN107895017A (en) * | 2017-11-14 | 2018-04-10 | 国网江苏省电力公司电力科学研究院 | A kind of electric energy quality monitoring system construction method based on big data technology |
| CN108844572A (en) * | 2018-03-29 | 2018-11-20 | 深圳市晟达机械设计有限公司 | A kind of highway tunnel structure intelligent monitor system based on big data and WSN technology |
| CN110008213A (en) * | 2019-03-13 | 2019-07-12 | 国电南瑞科技股份有限公司 | A kind of regulator control system real time data separate type management method |
| CN111241087A (en) * | 2020-03-10 | 2020-06-05 | 天津市普迅电力信息技术有限公司 | Energy data automatic processing method based on distributed computing |
| CN111414355A (en) * | 2020-03-19 | 2020-07-14 | 中国能源建设集团广东省电力设计研究院有限公司 | Offshore wind farm data monitoring and storing system, method and device |
| CN113849508A (en) * | 2021-09-22 | 2021-12-28 | 广东电网有限责任公司 | Data storage method, device, equipment and medium |
| CN115242870A (en) * | 2022-06-23 | 2022-10-25 | 宁波三星医疗电气股份有限公司 | Electricity consumption data pushing method and device, server and computer readable storage medium |
| CN115242870B (en) * | 2022-06-23 | 2023-10-27 | 宁波三星医疗电气股份有限公司 | Power consumption data pushing method, device, server and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106202566A (en) | A kind of magnanimity electricity consumption data mixing based on big data storage system and method | |
| CN105608144B (en) | A kind of big data analysis stage apparatus and method based on multilayered model iteration | |
| CN105608758B (en) | A kind of big data analysis platform device and method calculated based on algorithm configuration and distributed stream | |
| CN105427193B (en) | A kind of big data analysis device and method based on distributed time series data service | |
| CN106294644B (en) | A kind of magnanimity time series data collection and treatment device and method based on big data technology | |
| CN110334274A (en) | Information-pushing method, device, computer equipment and storage medium | |
| CN109272155A (en) | A kind of corporate behavior analysis system based on big data | |
| EP3035211B1 (en) | Visualizing large data volumes utilizing initial sampling and multi-stage calculations | |
| CN107766402A (en) | A kind of building dictionary cloud source of houses big data platform | |
| CN108039959A (en) | Situation Awareness method, system and the relevant apparatus of a kind of data | |
| CN105139281A (en) | Method and system for processing big data of electric power marketing | |
| CN104205039A (en) | Interest-driven business intelligence systems and methods of data analysis using interest-driven data pipelines | |
| CN110069526A (en) | System and method for distributed networks database query engine | |
| US10133779B2 (en) | Query hint management for a database management system | |
| CN112181960A (en) | Intelligent operation and maintenance framework system based on AIOps | |
| CN114416855A (en) | Visualization platform and method based on electric power big data | |
| CN107832876A (en) | Subregion peak load Forecasting Methodology based on MapReduce frameworks | |
| Kun et al. | Application of big data technology in scientific research data management of military enterprises | |
| CN112817958A (en) | Electric power planning data acquisition method and device and intelligent terminal | |
| CN113254517A (en) | Service providing method based on internet big data | |
| CN109033196A (en) | A kind of distributed data scheduling system and method | |
| CN109165203A (en) | Large public building energy consumption data based on Hadoop framework stores analysis method | |
| CN111260452B (en) | Method and system for constructing tax big data model | |
| CN114969183A (en) | Information management service platform applied to highway construction | |
| Sun et al. | The implementation of air pollution monitoring service using hybrid database converter |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20161207 |