CN106326252B - Data processing method and device for database - Google Patents
Data processing method and device for database Download PDFInfo
- Publication number
- CN106326252B CN106326252B CN201510355202.2A CN201510355202A CN106326252B CN 106326252 B CN106326252 B CN 106326252B CN 201510355202 A CN201510355202 A CN 201510355202A CN 106326252 B CN106326252 B CN 106326252B
- Authority
- CN
- China
- Prior art keywords
- data block
- data
- parameter
- server
- heat
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method of a database, which comprises the following steps: acquiring a first data block server with the largest data flow and a second data block server with the smallest data flow in all the data block servers; acquiring a first data block with the maximum data flow in each data block corresponding to a first data block server and a second data block with the minimum data flow in each data block corresponding to a second data block server; and exchanging the data request in the first data block with the data processing request in the second data block. The invention also discloses a data processing device of the database. The invention ensures that the data flow among the data block servers in the database is more balanced, and improves the performance and the working efficiency of the database.
Description
Technical Field
The present invention relates to the field of data storage technologies, and in particular, to a data processing method and apparatus for a database.
Background
With the continuous emergence of social application platforms and e-commerce type internet applications, people are marked to step into a higher life era. In such a massive data, high concurrent access situation, it is crucial that the database can provide stable and reliable performance. However, the conventional database cannot meet the problem of large data, and in the technical background, the distributed database is produced. HBase is the implementation of NoSQL (Not relational database) on Hadoop, a distributed, column-oriented open source database. It provides strong scalability and data manipulation performance. In the HBase, as the table record data is gradually split and finally formed into data blocks as the table record data is increased, different data blocks are allocated to corresponding data block servers by the host for management.
However, in practical applications, a large number of read and write requests are concentrated in some data blocks of some data block servers because of unbalanced load of the data block servers or unbalanced number of service requests in each data block server, so that the HBase cannot perform service processing in time, and further the performance of the HBase is affected, and the working efficiency of the HBase is low; even if a large number of read and write requests are concentrated on a certain data block server, the data block server may be down, and at this time, the HBase may transfer the read and write requests to other data block servers, which may cause continuous down of other data block servers, and finally cause complete interruption of the service corresponding to the read and write requests.
Disclosure of Invention
The invention provides a data processing method and a data processing device for a database, and aims to solve the technical problem that a large number of read and write requests in HBase are concentrated in certain data blocks of certain data block servers, so that the HBase cannot process services in time.
In order to achieve the above object, the present invention provides a data processing method for a database, which includes the following steps:
acquiring a first data block server with the largest data flow and a second data block server with the smallest data flow in all the data block servers;
acquiring a first data block with the maximum data flow in each data block corresponding to a first data block server and a second data block with the minimum data flow in each data block corresponding to a second data block server;
and exchanging the data request in the first data block with the data processing request in the second data block.
Preferably, the step of acquiring a first data block server with the largest data traffic and a second data block server with the smallest data traffic in each data block server includes:
acquiring flow parameters of a data block in preset time, wherein the flow parameters comprise written data flow and/or read data flow;
acquiring a locality parameter corresponding to the flow parameter of the data block, a historical heat parameter of the data block and a distribution parameter of a data block server;
calculating and obtaining the heat parameter of the data block server based on the flow parameter, the locality parameter, the historical heat parameter and the distribution parameter;
and taking the data block server with the maximum heat parameter as a first data block server, and taking the data block server with the minimum heat parameter as a second data block server.
Preferably, before the step of obtaining a first data block with a maximum data traffic in each data block corresponding to the first data block server and a second data block with a minimum data traffic in each data block corresponding to the second data block server, the method further includes:
acquiring a first heat parameter of the first data block server and a second heat parameter of a second data block server;
determining whether a first difference value between the first heat parameter and the second heat parameter is greater than a preset threshold value;
and when the first difference value between the first heat parameter and the second heat parameter is greater than a preset threshold value, executing the step of acquiring a first data block with the largest data flow in each data block corresponding to the first data block server and a second data block with the smallest data flow in each data block corresponding to the second data block server.
Preferably, between the step of obtaining a first heat parameter of the first data block server and a second heat parameter of the second data block server and the step of determining whether a first difference between the first heat parameter and the second heat parameter is greater than a preset threshold, the method further includes:
determining whether a first heat parameter of the first data block server is greater than a preset heat;
and when the first heat parameter of the first data block server is greater than a preset heat, executing the step of determining whether a first difference value between the first heat parameter and the second heat parameter is greater than a preset threshold value.
Preferably, between the step of obtaining a first data block with the largest data traffic in each data block corresponding to the first data block server and the step of obtaining a second data block with the smallest data traffic in each data block corresponding to the second data block server and the step of exchanging the data request in the first data block with the data processing request in the second data block, the method further includes:
generating an adjustment plan based on the first data block and the second data block, wherein configuration parameters of the database in the adjustment plan are heat parameters of each data block server and flow parameters corresponding to the data blocks after data requests in the first data block and data processing requests in the second data block are exchanged;
acquiring a third data block server with the largest heat parameter and a fourth data block server with the smallest heat parameter in an adjustment plan, and acquiring a third heat parameter of the third data block server and a fourth heat parameter of the fourth data block server;
determining whether a second difference value between the third heat parameter and the fourth heat parameter is greater than the preset threshold value;
when the second difference is larger than the preset threshold, acquiring a third data block with the largest data flow in each data block corresponding to a third data block server and a fourth data block with the smallest data flow in each data block corresponding to a fourth data block server;
after the step of interchanging data requests in the first data block with data processing requests in a second data block, the method further comprises:
and exchanging the data request of the third data block with the data processing request of the fourth data block.
In order to achieve the above object, the present invention also provides a data processing apparatus for a database, including:
the first acquisition module is used for acquiring a first data block server with the largest data flow and a second data block server with the smallest data flow in all the data block servers;
a second obtaining module, configured to obtain a first data block with a largest data flow in each data block corresponding to a first data block server, and a second data block with a smallest data flow in each data block corresponding to a second data block server;
and the first exchange module is used for exchanging the data request in the first data block with the data processing request in the second data block.
Preferably, the first obtaining module includes:
the device comprises a first obtaining unit, a second obtaining unit and a processing unit, wherein the first obtaining unit is used for obtaining flow parameters of a data block in preset time, and the flow parameters comprise written data flow and/or read data flow;
the second acquisition unit is used for acquiring a locality parameter corresponding to the flow parameter of the data block, a historical heat parameter of the data block and a distribution parameter of the data block server;
the calculation unit is used for calculating and obtaining the heat parameter of the data block server based on the flow parameter, the locality parameter, the historical heat parameter and the distribution parameter;
and the setting unit is used for taking the data block server with the maximum heat parameter as a first data block server and taking the data block server with the minimum heat parameter as a second data block server.
Preferably, the data processing apparatus of the database further includes:
the third acquisition module is used for acquiring the first heat parameter of the first data block server and the second heat parameter of the second data block server;
the first determining module is used for determining whether a first difference value of the first heat parameter and the second heat parameter is larger than a preset threshold value;
the second obtaining module is further configured to obtain a first data block with a largest data flow in each data block corresponding to the first data block server and a second data block with a smallest data flow in each data block corresponding to the second data block server when a first difference value between the first heat parameter and the second heat parameter is greater than a preset threshold.
Preferably, the data processing apparatus of the database further includes:
the second determining module is used for determining whether the first heat parameter of the first data block server is greater than the preset heat;
the first determining module is further configured to determine whether a first difference between the first heat parameter and the second heat parameter is greater than a preset threshold value when the first heat parameter of the first data block server is greater than a preset heat.
Preferably, the data processing apparatus of the database further includes:
a generating module, configured to generate an adjustment plan based on the first data block and the second data block, where a configuration parameter of the database in the adjustment plan is a heat parameter of each data block server and a traffic parameter corresponding to the data block after exchanging a data request in the first data block and a data processing request in the second data block;
the fourth obtaining module is used for obtaining a third data block server with the largest heat parameter and a fourth data block server with the smallest heat parameter in the adjustment plan, and obtaining a third heat parameter of the third data block server and a fourth heat parameter of the fourth data block server;
a third determining module, configured to determine whether a second difference between the third heat parameter and the fourth heat parameter is greater than the preset threshold;
a fifth obtaining module, configured to obtain, when the second difference is greater than the preset threshold, a third data block with a largest data flow in each data block corresponding to a third data block server, and a fourth data block with a smallest data flow in each data block corresponding to a fourth data block server;
and the second exchange module is used for exchanging the data request of the third data block with the data processing request of the fourth data block.
In the invention, a first data block with maximum data flow in a first data block server with maximum data flow and a second data block with minimum data flow in a second data block server with minimum data flow are obtained, and then a data request in the first data block and a data request in the second data block are exchanged; the data processing method and the data processing system have the advantages that the data request of the first data block and the data processing request of the second data block are exchanged, so that the data flow between the first data block server and the second data block server is more balanced, the database can timely process the service corresponding to the data request of the first data block, the data block server is effectively prevented from being crashed when the flow parameters of the data blocks in the database are too large, and the performance and the working efficiency of the database are improved.
Drawings
FIG. 1 is a schematic flow chart of a data processing method of a database according to a first embodiment of the present invention;
FIG. 2 is a flowchart illustrating a detailed process of the steps of acquiring a first chunk server and a second chunk server according to the present invention;
FIG. 3 is a flowchart illustrating a second embodiment of a database processing method according to the present invention;
FIG. 4 is a flowchart illustrating a data processing method of a database according to a third embodiment of the present invention;
FIG. 5 is a functional block diagram of a first embodiment of a database processing apparatus according to the present invention;
FIG. 6 is a block diagram illustrating a refinement function of the computing block of FIG. 5;
FIG. 7 is a functional block diagram of a second embodiment of a database processing apparatus according to the present invention;
FIG. 8 is a functional block diagram of a data processing apparatus according to a third embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention provides a data processing method of a database.
Referring to fig. 1, fig. 1 is a schematic flow chart of a data processing method of a database according to a first embodiment of the present invention.
In this embodiment, the data processing method of the database includes:
step S10, obtaining a first data block server with the largest data flow and a second data block server with the smallest data flow in each data block server;
in the embodiment, a set of read-write pressure acquisition service is added to each data block server of a database, the data traffic of each data block server within a preset time is periodically acquired, and then a first data block server with the largest data traffic and a second data block server with the smallest data traffic within the preset time are determined; or adding a set of read-write pressure acquisition service in the data block of each data block server of the database, periodically acquiring the data flow of each data block within preset time, then calculating the data flow of the data block server, and then determining a first data block server with the maximum data flow and a second data block server with the minimum data flow within the preset time.
Step S20, acquiring a first data block with the largest data flow among the data blocks corresponding to the first data block server, and a second data block with the smallest data flow among the data blocks corresponding to the second data block server;
the obtained first data block is the data block with the maximum data flow in the first data block server, and the obtained second data block is the data block with the minimum data flow in the second data block server; it is easy to understand that, when the first data block and the second data block are obtained, if two data blocks with the same data traffic and the largest data traffic exist in the first data block server or two data blocks with the same data traffic and the smallest data traffic exist in the second data block server, the first data block or the second data block is determined according to the row key of the data blocks.
Of course, if the first data block belongs to the meta-meta table or the nameSpace table, the subsequent flow of this embodiment is ended, that is, when the first data block belongs to the meta table or the nameSpace table, the exchange of the data request with the second data block is not required, because the read-write access of the meta table or the nameSpace table is relatively small, that is, the data traffic written in the meta table or the nameSpace table and/or the data traffic read from the meta table or the nameSpace table, and it is possible to effectively avoid frequently moving the two system internal tables when the read-write traffic is small.
Step S30, interchanging the data request in the first data block and the data processing request in the second data block.
The data processing request in the second data block is the processing request of the write data and/or the read data received by the second data block in the preset time; the data processing method comprises the steps of setting a data request of a first data block on a second data block, and setting a data processing request of the second data block on the first data block, so that data traffic written in and/or read data traffic of the first data block is exchanged with data traffic written in and/or read data traffic of the second data block, and further the heat parameter of a first data block server is reduced, the heat parameter of a second data block server is improved, and the data traffic of a data block server of a database is more balanced.
In this embodiment, after the data request of the first data block is exchanged with the data request of the second data block, the database information in the database is updated based on the adjusted database, including the written data traffic and/or the read data traffic information of the updated data block, and the exchange information of the data requests between the data blocks, and the like.
In this embodiment, the database is an HBase, the data block server is a RegionServe in the HBase, and the data block is a Region in the RegionServe.
In this embodiment, a first data block with the largest data flow in a first data block server with the largest data flow and a second data block with the smallest data flow in a second data block server with the smallest data flow are obtained, and then a data request in the first data block and a data request in the second data block are exchanged; the data processing method and the data processing system have the advantages that the data request of the first data block and the data processing request of the second data block are exchanged, so that the data flow between the first data block server and the second data block server is more balanced, the database can timely process the service corresponding to the data request of the first data block, the data block server is effectively prevented from being crashed when the flow parameters of the data blocks in the database are too large, and the performance and the working efficiency of the database are improved.
Referring to fig. 2, fig. 2 is a detailed flowchart of the steps of acquiring the first data block server and the second data block server according to the present invention.
A detailed flow embodiment of step S10 of the present invention is proposed based on the first embodiment, in this embodiment, step S10 includes:
step S11, obtaining flow parameters of the data block within a preset time, wherein the flow parameters comprise written data flow and/or read data flow;
the written data flow refers to the sum of the sizes of the data packets corresponding to all the write requests of the data block and the sum of the sizes of the data packets corresponding to all the read requests of the data block, and therefore, the obtaining of the flow parameter of the data block within the preset time specifically refers to obtaining the sum of the sizes of the data packets corresponding to all the write requests of the data block and/or the sum of the sizes of the data packets corresponding to all the read requests of the data block. Of course, the flow parameter of the data block in the preset time may be obtained periodically, or may be obtained according to a preset time interval. In this embodiment, a set of read-write pressure acquisition service is added to the data block of each data block server of the database, and the read and write flow of each data block in a preset time is periodically acquired.
Step S12, obtaining a locality parameter corresponding to the flow parameter of the data block, a historical heat parameter of the data block and a distribution parameter of a data block server;
the locality parameter localPercent corresponding to the flow parameter of the data block refers to a locality ratio weight of data written in and/or read from the data block, and a data proportion occupied by a data block server to which the data block belongs in the data written in and/or read from the data block, and a value range of the locality parameter localPercent is [0, 1 ].
The historical heat parameter HistoryPercent of the data block refers to the historical heat trend weight of the data block, the value range of the historical heat trend weight is [0, 1], the historical heat parameter HistoryPercent is obtained by calculation according to the curve slope trend of the latest X (default to 3) historical period access requests on the data block, wherein the larger the slope is, the access requests always show a forward increasing trend.
The distribution parameter tableeprenecent of the data block server refers to the equilibrium distribution rate of the Table, that is, the equilibrium distribution rate of the Table corresponding to the data block of the data block server.
Step S13, calculating and obtaining the heat parameter of the data block server based on the flow parameter, the locality parameter, the historical heat parameter and the distribution parameter;
the calculation formula of the heat parameter of the data block server is as follows:
HotScorer＝∑[(1-α)×LastQualityScorer+α×CurQualityScorer]×(1/LocalPercent)×(1+HistorySlope)×(1+TablePerCent)
in the formula, LastQualityScore is data flow and/or read data flow of a data block in last preset time, CurQualityScore is flow parameter of the data block in current preset time, α represents heat weight value of the data block server at last time, the value range is [0, 1], if the value is set to 0, the last heat access is not considered, generally, the value of α is 0.5, and the heat on the data block at this time and the heat of the data block at last time are considered simultaneously.
In step S14, the data chunk server with the highest heat parameter is used as the first data chunk server, and the data chunk server with the lowest heat parameter is used as the second data chunk server.
And setting the data block server with the maximum temperature parameter in the database as a first data block server and setting the data block server with the minimum temperature parameter in the database as a second data block server according to the temperature parameter of the data block server. Of course, in the process of acquiring the first data block server and the second data block server, if two hottest data block servers with the same heat parameter or two coldest data block servers with the same heat parameter exist in the database, the data block servers with the previous selection sequence are used as the first data block server or the second data block server according to the machine name reference sequence of the data block servers.
In this embodiment, the heat parameter of the data block server is obtained through calculation according to the obtained flow parameter, the obtained locality parameter, the obtained historical heat parameter and the obtained distribution parameter, then the data block server with the largest heat parameter is used as a first data block server, and the data block server with the smallest heat parameter is used as a second data block server, so that the first data block server and the second data block server can be respectively the hottest data block server and the coldest data block server of the database, the accuracy of the first data block and the accuracy of the second data block are further improved, and the performance and the working efficiency of the database are further improved.
Referring to fig. 3, fig. 3 is a flowchart illustrating a data processing method of a database according to a second embodiment of the present invention.
A second embodiment of the data processing method of the database of the present invention is proposed based on the second embodiment, and in this embodiment, before step S20, the data processing method of the database further includes:
step S40, acquiring a first heat parameter of the first data block server and a second heat parameter of the second data block server;
and acquiring a first heat parameter of the first data block server and a second heat parameter of the second data block server from the heat parameters of the data block servers.
Step S50, determining whether a first difference between the first heat parameter and the second heat parameter is greater than a preset threshold;
as can be seen from the foregoing embodiment, the first heat parameter is greater than the second heat parameter, a first difference between the first heat parameter and the second heat parameter, that is, a difference between the first heat parameter and the second heat parameter, is greater than zero, the preset threshold is a maximum value of the heat parameter between the hottest data block server and the coldest data block server in the database, and when the first difference between the first heat parameter and the second heat parameter is greater than the preset threshold, the data block of the hottest data block server and the data block of the coldest data block server need to be balanced.
And when the first difference value between the first heat parameter and the second heat parameter is greater than a preset threshold value, executing the step of acquiring a first data block with the largest data flow in each data block corresponding to the first data block server and a second data block with the smallest data flow in each data block corresponding to the second data block server.
Namely, step S20 is replaced with: and when the first difference value between the first heat parameter and the second heat parameter is larger than a preset threshold value, acquiring a first data block with the largest data flow in each data block corresponding to a first data block server and a second data block with the smallest data flow in each data block corresponding to a second data block server.
In this embodiment, by executing the step of acquiring the first data block of the first data block server and the second data block of the second data block server when the first difference between the first heat parameter and the second heat parameter is greater than the preset threshold, it is avoided that the data request of the first data block and the data request of the second data block are performed when the first difference between the first heat parameter and the second heat parameter does not exceed the preset threshold, and thus, unnecessary data traffic of the data block servers in the database is avoided, and performance and work efficiency of the database are further improved.
Referring to fig. 4, fig. 4 is a flowchart illustrating a data processing method of a database according to a third embodiment of the present invention.
A third embodiment of the data processing method of the database of the present invention is proposed based on the third embodiment, and in this embodiment, between step S40 and step S50, the data processing method of the database further includes:
step S60, determining whether a first heat parameter of the first data block server is greater than a preset heat;
in this embodiment, a corresponding preset heat may be set for the data block server in the database, specifically, when the heat parameter of the data block server does not exceed the preset heat, it indicates that the data block server does not exceed the bearing capacity of its own data processing, that is, the data block in the data block server does not need to be balanced and adjusted, which is easy to understand, when it is determined that the first heat parameter of the first data block server is greater than the preset heat, it indicates that the data processing capacity of the first data block server has exceeded its bearing capacity, and it needs to be adjusted by the data block in the first data block server and the data blocks in other data block servers, so as to balance the data traffic of the data block servers in the database. When the first heat parameter of the first data block server does not exceed the preset heat, the first data block server does not exceed the data processing tolerance of the first data block server, and therefore the process of this embodiment may be ended.
And when the first heat parameter of the first data block server is greater than a preset heat, executing the step of determining whether a first difference value between the first heat parameter and the second heat parameter is greater than a preset threshold value.
That is, in step S50, it is determined whether a first difference between the first heat parameter and the second heat parameter is greater than a preset threshold when the first heat parameter of the first data block server is greater than a preset heat.
In this embodiment, whether the first heat parameter of the first data block server is greater than the preset heat is determined, and then when the first heat parameter is greater than the preset heat, the step of determining that the first difference between the first heat parameter and the second heat parameter is greater than the preset threshold is performed, so as to determine whether the first data block server needs to be adjusted by the data block of the first data block server and the data blocks of other data block servers, thereby further avoiding unnecessary data traffic of the data block servers in the balance adjustment database, and improving the efficiency of the database.
A fourth embodiment of the data processing method of the database of the present invention is proposed based on the above-mentioned embodiment, and in this embodiment, between step S20 and step S30, the data processing method of the database further includes:
generating an adjustment plan based on the first data block and the second data block, wherein configuration parameters of the database in the adjustment plan are heat parameters of each data block server and flow parameters corresponding to the data blocks after data requests in the first data block and data processing requests in the second data block are exchanged;
that is, assuming that the data request in the first data block and the data request in the second data block are exchanged completely, that is, the data traffic of the first data block server and the data traffic of the second data block server are exchanged completely, an adjustment plan is generated according to the configuration parameters of the database after the exchange is completed, and of course, the heat parameters of the first data block server and the second data block server in the adjustment plan are recalculated heat parameters, so as to avoid an error occurring when the first data block server and the second data block server obtain the data block server with the maximum or minimum heat parameters in the database after the exchange.
Acquiring a third data block server with the largest heat parameter and a fourth data block server with the smallest heat parameter in an adjustment plan, and acquiring a third heat parameter of the third data block server and a fourth heat parameter of the fourth data block server;
determining whether a second difference value between the third heat parameter and the fourth heat parameter is greater than the preset threshold value;
of course, before determining that the second difference between the third heat parameter of the third data block server and the fourth heat parameter of the fourth data block server is greater than the preset threshold, it may also be determined that the heat parameter of the third data block server is greater than the preset heat, so as to avoid unnecessarily balancing data traffic between data block servers in the database.
When the second difference is larger than the preset threshold, acquiring a third data block with the largest data flow in each data block corresponding to a third data block server and a fourth data block with the smallest data flow in each data block corresponding to a fourth data block server;
after the step of interchanging data requests in the first data block with data processing requests in a second data block, the method further comprises:
and exchanging the data request of the third data block with the data processing request of the fourth data block.
And setting the data request of the third data block on the fourth data block, and setting the data processing request of the fourth data block on the third data block, so that the data traffic written and/or read by the third data block is interchanged with the data traffic written and/or read by the fourth data block.
Of course, before the step of exchanging the data request of the third data block and the data request of the fourth data block, the balancing plan may be updated based on the third data block and the fourth data block, whether a data block server with an absolute value of a difference between two heat parameters greater than a preset threshold exists in the database is continuously and circularly judged, the above steps are continued until a data block server with an absolute value of a difference between two heat parameters greater than a preset threshold does not exist in the database, and then link exchange of the data blocks is sequentially performed according to the balancing plan, so that data traffic between the data block servers of the database is more balanced. It is easy to understand that, because the database has a service pause of a second level when performing link switching of data blocks, the data blocks performing link switching each time cannot be too many, so as to avoid long service pauses from affecting the user experience.
In this embodiment, a third data block server with the largest heat parameter and a fourth data block server with the smallest heat parameter are determined based on the adjustment plan, then when it is determined that the second difference is greater than the preset threshold, a third data block with the largest data flow in the third data block server and a fourth data block with the smallest data flow in the fourth data block server are determined, and finally, after step S30, the data request of the third data block and the data processing request of the fourth data block are interchanged; the data flow written in and/or read out by the third data block and the data flow written in and/or read out by the fourth data block are exchanged, so that the heat of the third data block server is reduced, and the heat of the fourth data block server is improved, so that the data flow between the third data block server and the fourth data block server is more balanced, the database can process the service corresponding to the data request of the third data block in time, the downtime of the third data block server caused by the overlarge flow parameter of the third data block in the database is effectively avoided, and the performance and the working efficiency of the database are further improved.
Of course, the data processing method of the database may also perform load balancing based on the table, and the flow includes: acquiring data flow written in and/or read from a table in a database, and calculating to obtain the hottest table and the coldest table based on the data flow written in and/or read from the table; judging whether the written data flow and/or the read data flow of the hottest table exceeds the load heat threshold of the table; when the written data flow and/or the read data flow of the hottest table exceeds a load heat threshold, obtaining the hottest data block in the table and the coldest data block in the coldest table, and then exchanging a link corresponding to the data request of the hottest data block with a link corresponding to the data request of the coldest data block. Of course, the process of load balancing based on the table may also be executed in a loop manner as in the above embodiment, so as to implement parallel migration of data blocks in multiple tables, and this type of load balancing based on the table is suitable for cases where there are not many "hot tables" and many "cold tables" compared to data block migration in the data block server, and a hot table means that the data traffic of the table exceeds the load hot threshold of the table.
The invention further provides a data processing device of the database.
Referring to fig. 5, fig. 5 is a functional block diagram of a data processing apparatus of a database according to a first embodiment of the present invention.
In this embodiment, the data processing apparatus of the database includes:
a first obtaining module 10, configured to obtain a first data block server with a largest data flow and a second data block server with a smallest data flow in each data block server;
in the embodiment, a set of read-write pressure acquisition service is added to each data block server of a database, the data traffic of each data block server within a preset time is periodically acquired, and then a first data block server with the largest data traffic and a second data block server with the smallest data traffic within the preset time are determined; or adding a set of read-write pressure acquisition service in the data block of each data block server of the database, periodically acquiring the data flow of each data block within preset time, then calculating the data flow of the data block server, and then determining a first data block server with the maximum data flow and a second data block server with the minimum data flow within the preset time.
A second obtaining module 20, configured to obtain a first data block with a largest data flow in each data block corresponding to a first data block server, and a second data block with a smallest data flow in each data block corresponding to a second data block server;
the first data block acquired by the second acquisition module 20 is a data block with the largest data flow in the first data block server, and the second data block acquired is a data block with the smallest data flow in the second data block server; it is easy to understand that, when the first data block and the second data block are obtained, if two data blocks with the same and the largest data traffic exist in the first data block server or two data blocks with the same and the smallest data traffic exist in the second data block server, the first data block or the second data block is determined according to the row key of the data blocks.
A first exchanging module 30, configured to exchange the data request in the first data block with the data processing request in the second data block.
The data request is a request for writing data and/or reading data received by a data block within a preset time, the first exchange module 30 sets the data request of the first data block on the second data block, and sets the data processing request of the second data block on the first data block, so that the data traffic written in and/or read data traffic of the first data block is interchanged with the data traffic written in and/or read data traffic of the second data block, thereby reducing the heat parameter of the first data block server, improving the heat parameter of the second data block server, and making the data traffic of the data block servers of the database more balanced.
In this embodiment, a second obtaining module 20 obtains a first data block with the largest data flow in a first data block server with the largest data flow and a second data block with the smallest data flow in a second data block server with the smallest data flow, and then a first exchanging module 30 exchanges a data request in the first data block and a data processing request in the second data block; the data processing method and the data processing system have the advantages that the data request of the first data block and the data processing request of the second data block are exchanged, so that the data flow between the first data block server and the second data block server is more balanced, the database can timely process the service corresponding to the data request of the first data block, the data block server is effectively prevented from being crashed when the flow parameters of the data blocks in the database are too large, and the performance and the working efficiency of the database are improved.
Referring to fig. 6, fig. 6 is a schematic diagram of a detailed functional module of the computing module in fig. 5.
An embodiment of a detailed function module of the calculating module 20 of the present invention is provided based on a first embodiment, in which the first obtaining module 10 includes:
a first obtaining unit 11, configured to obtain a traffic parameter of a data block in a preset time, where the traffic parameter includes a written data traffic and/or a read data traffic;
the written data flow refers to the sum of the sizes of the data packets corresponding to all the write requests of the data block and the sum of the sizes of the data packets corresponding to all the read requests of the data block, and therefore, the obtaining of the flow parameter of the data block within the preset time specifically refers to obtaining the sum of the sizes of the data packets corresponding to all the write requests of the data block and/or the sum of the sizes of the data packets corresponding to all the read requests of the data block. Of course, the flow parameter of the data block in the preset time may be obtained periodically, or may be obtained according to a preset time interval. In this embodiment, a set of read-write pressure acquisition service is added to the data block of each data block server of the database, and the read and write flow of each data block in a preset time is periodically acquired.
A second obtaining unit 12, configured to obtain a locality parameter corresponding to the traffic parameter of the data block, a historical heat parameter of the data block, and a distribution parameter of the data block server;
the locality parameter localPercent corresponding to the flow parameter of the data block refers to a locality ratio weight of data written in and/or read from the data block, and a data proportion occupied by a data block server to which the data block belongs in the data written in and/or read from the data block, and a value range of the locality parameter localPercent is [0, 1 ].
The historical heat parameter HistoryPercent of the data block refers to the historical heat trend weight of the data block, the value range of the historical heat trend weight is [0, 1], the historical heat parameter HistoryPercent is obtained by calculation according to the curve slope trend of the latest X (default to 3) historical period access requests on the data block, wherein the larger the slope is, the access requests always show a forward increasing trend.
The distribution parameter tableeprenecent of the data block server refers to the equilibrium distribution rate of the Table, that is, the equilibrium distribution rate of the Table corresponding to the data block of the data block server.
A calculating unit 13, configured to calculate and obtain a heat parameter of the data block server based on the traffic parameter, the locality parameter, the historical heat parameter, and the distribution parameter;
the calculation formula of the heat parameter of the data block server is as follows:
HotScorer＝∑[(1-α)×LastQualityScorer+α×CurQualityScorer]×(1/LocalPercent)×(1+HistorySlope)×(1+TablePerCent)
in the formula, LastQualityScore is data flow and/or read data flow of a data block in last preset time, CurQualityScore is flow parameter of the data block in current preset time, α represents heat weight value of the data block server at last time, the value range is [0, 1], if the value is set to 0, the last heat access is not considered, generally, the value of α is 0.5, and the heat on the data block at this time and the heat of the data block at last time are considered simultaneously.
And the setting unit 14 is configured to use the data block server with the largest heat parameter as the first data block server, and use the data block server with the smallest heat parameter as the second data block server.
In this embodiment, the calculation unit 13 calculates and obtains the heat parameter of the data block server according to the obtained flow parameter, the locality parameter, the historical heat parameter, and the distribution parameter, and then the setting unit 14 uses the data block server with the largest heat parameter as the first data block server, and uses the data block server with the smallest heat parameter as the second data block server, so that it can be ensured that the first data block server and the second data block server are respectively the hottest data block server and the coldest data block server of the database, and the accuracy of the first data block and the second data block is further improved, and the performance and the working efficiency of the database are further improved.
Referring to fig. 7, fig. 7 is a functional block diagram of a data processing apparatus of a database according to a second embodiment of the present invention.
A second embodiment of the data processing apparatus of the database according to the present invention is provided based on the second embodiment, and in this embodiment, the data processing apparatus of the database further includes:
a third obtaining module 40, configured to obtain a first heat parameter of the first data block server and a second heat parameter of the second data block server;
a first determining module 50, configured to determine whether a first difference between the first heat parameter and the second heat parameter is greater than a preset threshold;
the second obtaining module 20 is further configured to obtain, when a first difference between the first heat parameter and the second heat parameter is greater than a preset threshold, a first data block with a largest data flow in each data block corresponding to the first data block server, and a second data block with a smallest data flow in each data block corresponding to the second data block server.
In this embodiment, a third obtaining module 40 obtains a first heat parameter of a first data block server and a second heat parameter of a second data block server, and then a second obtaining module 20 obtains the first data block of the first data block server and the second data block of the second data block server when the first determining module 50 determines that a first difference between the first heat parameter and the second heat parameter is greater than a preset threshold; the data request of the first data block and the data request of the second data block are avoided when the first difference value of the first heat parameter and the second heat parameter does not exceed the preset threshold value, so that the data flow of a data block server in the database is avoided being unnecessarily balanced, and the performance and the working efficiency of the database are further improved.
Referring to fig. 8, fig. 8 is a functional block diagram of a data processing apparatus of a database according to a third embodiment of the present invention.
A third embodiment of the data processing apparatus of the database according to the present invention is provided based on the third embodiment, and in this embodiment, the data processing apparatus of the database further includes:
a second determining module 60, configured to determine whether a first heat parameter of the first data block server is greater than a preset heat;
the first determining module is further configured to determine whether a first difference between the first heat parameter and the second heat parameter is greater than a preset threshold value when the first heat parameter of the first data block server is greater than a preset heat.
In this embodiment, the second determining module 60 determines that the first heat parameter of the first data block server is greater than the preset heat, and then the first determining module 60 determines that the first difference between the first heat parameter and the second heat parameter is greater than the preset threshold when the first heat parameter of the first data block server is greater than the preset heat; and further, whether the first data block server needs to be adjusted through the data block of the first data block server and the data blocks in other data block servers is determined, so that the unnecessary data flow of the data block servers in the balance adjustment database is further avoided, and the efficiency of the database is improved.
A fourth embodiment of the data processing apparatus of the database according to the present invention is provided based on the above embodiment, and in this embodiment, the data processing apparatus of the database further includes:
a generating module, configured to generate an adjustment plan based on the first data block and the second data block, where a configuration parameter of the database in the adjustment plan is a heat parameter of each data block server and a traffic parameter corresponding to the data block after exchanging a data request in the first data block and a data request in the second data block;
assuming that the data request in the first data block and the data request in the second data block are exchanged completely, that is, the data traffic of the first data block server and the data traffic of the second data block server are exchanged completely, an adjustment plan is generated according to the configuration parameters of the database after the exchange is completed, and certainly, the heat parameters of the first data block server and the second data block server in the adjustment plan are recalculated heat parameters, so as to avoid an error occurring when the first data block server and the second data block server obtain the data block server with the maximum or minimum heat parameters in the database after the exchange.
The fourth obtaining module is used for obtaining a third data block server with the largest heat parameter and a fourth data block server with the smallest heat parameter in the adjustment plan, and obtaining a third heat parameter of the third data block server and a fourth heat parameter of the fourth data block server;
a third determining module, configured to determine whether a second difference between the third heat parameter and the fourth heat parameter is greater than the preset threshold;
of course, before the third determining module determines that the second difference between the third heat parameter of the third data block server and the fourth heat parameter of the fourth data block server is greater than the preset threshold, it may also be determined that the heat parameter of the third data block server is greater than the preset heat, so as to avoid unnecessary data traffic between data block servers in the balanced database.
A fifth obtaining module, configured to obtain, when the second difference is greater than the preset threshold, a third data block with a largest data flow in each data block corresponding to a third data block server, and a fourth data block with a smallest data flow in each data block corresponding to a fourth data block server;
and the second exchange module is used for exchanging the data request of the third data block with the data processing request of the fourth data block.
The second exchange module sets the data request of the third data block on the fourth data block and sets the data processing request of the fourth data block on the third data block, so that the data traffic written and/or read by the third data block is interchanged with the data traffic written and/or read by the fourth data block.
In this embodiment, a third data block server with the largest heat parameter and a fourth data block server with the smallest heat parameter are determined by a fourth obtaining module based on the adjustment plan, then when the third determining module determines that the second difference is greater than the preset threshold, a fifth obtaining module determines a third data block with the largest data flow in the third data block server and a fourth data block with the smallest data flow in the fourth data block server, and finally a second exchanging module exchanges a data request of the third data block and a data processing request of the fourth data block; the data flow written in and/or read out by the third data block and the data flow written in and/or read out by the fourth data block are exchanged, so that the heat of the third data block server is reduced, and the heat of the fourth data block server is improved, so that the data flow between the third data block server and the fourth data block server is more balanced, the database can process the service corresponding to the data request of the third data block in time, the downtime of the third data block server caused by the overlarge flow parameter of the third data block in the database is effectively avoided, and the performance and the working efficiency of the database are further improved.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (8)
1. A data processing method of a database is characterized by comprising the following steps:
acquiring a first data block server with the largest data flow and a second data block server with the smallest data flow in all the data block servers;
acquiring a first data block with the maximum data flow in each data block corresponding to a first data block server and a second data block with the minimum data flow in each data block corresponding to a second data block server;
interchanging a data request in the first data block with a data processing request in a second data block;
the step of obtaining a first data block server with the largest data flow and a second data block server with the smallest data flow in each data block server includes:
acquiring flow parameters of a data block in preset time, wherein the flow parameters comprise written data flow and/or read data flow;
acquiring a locality parameter corresponding to the flow parameter of the data block, a historical heat parameter of the data block and a distribution parameter of a data block server;
calculating and obtaining the heat parameter of the data block server based on the flow parameter, the locality parameter, the historical heat parameter and the distribution parameter;
and taking the data block server with the maximum heat parameter as a first data block server, and taking the data block server with the minimum heat parameter as a second data block server.
2. The data processing method of the database according to claim 1, wherein before the step of obtaining a first data block with the largest data traffic in each data block corresponding to a first data block server and a second data block with the smallest data traffic in each data block corresponding to a second data block server, the method further comprises:
acquiring a first heat parameter of the first data block server and a second heat parameter of a second data block server;
determining whether a first difference value between the first heat parameter and the second heat parameter is greater than a preset threshold value;
and when the first difference value between the first heat parameter and the second heat parameter is greater than a preset threshold value, executing the step of acquiring a first data block with the largest data flow in each data block corresponding to the first data block server and a second data block with the smallest data flow in each data block corresponding to the second data block server.
3. The data processing method of the database according to claim 2, wherein between the step of obtaining a first heat parameter of the first data block server and a second heat parameter of a second data block server and determining whether a first difference between the first heat parameter and the second heat parameter is greater than a preset threshold, the method further comprises:
determining whether a first heat parameter of the first data block server is greater than a preset heat;
and when the first heat parameter of the first data block server is greater than a preset heat, executing the step of determining whether a first difference value between the first heat parameter and the second heat parameter is greater than a preset threshold value.
4. The data processing method of the database according to claim 2 or 3, wherein between the step of obtaining the first data block with the largest data traffic in each data block corresponding to the first data block server and the second data block with the smallest data traffic in each data block corresponding to the second data block server and the step of exchanging the data request in the first data block with the data processing request in the second data block, the method further comprises:
generating an adjustment plan based on the first data block and the second data block, wherein configuration parameters of the database in the adjustment plan are heat parameters of each data block server and flow parameters corresponding to the data blocks after data requests in the first data block and data processing requests in the second data block are exchanged;
acquiring a third data block server with the largest heat parameter and a fourth data block server with the smallest heat parameter in an adjustment plan, and acquiring a third heat parameter of the third data block server and a fourth heat parameter of the fourth data block server;
determining whether a second difference value between the third heat parameter and the fourth heat parameter is greater than the preset threshold value;
when the second difference is larger than the preset threshold, acquiring a third data block with the largest data flow in each data block corresponding to a third data block server and a fourth data block with the smallest data flow in each data block corresponding to a fourth data block server;
after the step of interchanging data requests in the first data block with data processing requests in a second data block, the method further comprises:
and exchanging the data request of the third data block with the data processing request of the fourth data block.
5. A data processing apparatus of a database, characterized in that the data processing apparatus of the database comprises:
the first acquisition module is used for acquiring a first data block server with the largest data flow and a second data block server with the smallest data flow in all the data block servers;
a second obtaining module, configured to obtain a first data block with a largest data flow in each data block corresponding to a first data block server, and a second data block with a smallest data flow in each data block corresponding to a second data block server;
the first exchange module is used for exchanging the data request in the first data block with the data processing request in the second data block;
the first obtaining module comprises:
the device comprises a first obtaining unit, a second obtaining unit and a processing unit, wherein the first obtaining unit is used for obtaining flow parameters of a data block in preset time, and the flow parameters comprise written data flow and/or read data flow;
the second acquisition unit is used for acquiring a locality parameter corresponding to the flow parameter of the data block, a historical heat parameter of the data block and a distribution parameter of the data block server;
the calculation unit is used for calculating and obtaining the heat parameter of the data block server based on the flow parameter, the locality parameter, the historical heat parameter and the distribution parameter;
and the setting unit is used for taking the data block server with the maximum heat parameter as a first data block server and taking the data block server with the minimum heat parameter as a second data block server.
6. The database data processing apparatus of claim 5, wherein the database data processing apparatus further comprises:
the third acquisition module is used for acquiring the first heat parameter of the first data block server and the second heat parameter of the second data block server;
the first determining module is used for determining whether a first difference value of the first heat parameter and the second heat parameter is larger than a preset threshold value;
the second obtaining module is further configured to obtain a first data block with a largest data flow in each data block corresponding to the first data block server and a second data block with a smallest data flow in each data block corresponding to the second data block server when a first difference value between the first heat parameter and the second heat parameter is greater than a preset threshold.
7. The database data processing apparatus of claim 6, wherein the database data processing apparatus further comprises:
the second determining module is used for determining whether the first heat parameter of the first data block server is greater than the preset heat;
the first determining module is further configured to determine whether a first difference between the first heat parameter and the second heat parameter is greater than a preset threshold value when the first heat parameter of the first data block server is greater than a preset heat.
8. The database data processing apparatus according to claim 6 or 7, wherein the database data processing apparatus further comprises:
a generating module, configured to generate an adjustment plan based on the first data block and the second data block, where a configuration parameter of the database in the adjustment plan is a heat parameter of each data block server and a traffic parameter corresponding to the data block after exchanging a data request in the first data block and a data processing request in the second data block;
the fourth obtaining module is used for obtaining a third data block server with the largest heat parameter and a fourth data block server with the smallest heat parameter in the adjustment plan, and obtaining a third heat parameter of the third data block server and a fourth heat parameter of the fourth data block server;
a third determining module, configured to determine whether a second difference between the third heat parameter and the fourth heat parameter is greater than the preset threshold;
a fifth obtaining module, configured to obtain, when the second difference is greater than the preset threshold, a third data block with a largest data flow in each data block corresponding to a third data block server, and a fourth data block with a smallest data flow in each data block corresponding to a fourth data block server;
and the second exchange module is used for exchanging the data request of the third data block with the data processing request of the fourth data block.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510355202.2A CN106326252B (en) | 2015-06-24 | 2015-06-24 | Data processing method and device for database |
| PCT/CN2015/089383 WO2016206192A1 (en) | 2015-06-24 | 2015-09-10 | Data processing method and device for database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510355202.2A CN106326252B (en) | 2015-06-24 | 2015-06-24 | Data processing method and device for database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106326252A CN106326252A (en) | 2017-01-11 |
| CN106326252B true CN106326252B (en) | 2020-05-22 |
Family
ID=57584359
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510355202.2A Active CN106326252B (en) | 2015-06-24 | 2015-06-24 | Data processing method and device for database |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN106326252B (en) |
| WO (1) | WO2016206192A1 (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107562542B (en) * | 2017-09-06 | 2020-04-07 | 腾讯科技（深圳）有限公司 | Data partitioning method and device for distributed data processing system |
| CN107748649B (en) * | 2017-09-18 | 2021-03-30 | 北京京东尚科信息技术有限公司 | Method and device for caching data |
| CN110324403A (en) * | 2019-05-23 | 2019-10-11 | 平安科技（深圳）有限公司 | Dynamic divides library method for routing, device, server and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101146114A (en) * | 2007-10-12 | 2008-03-19 | 中兴通讯股份有限公司 | A load share method and system of stream media server |
| CN101594371A (en) * | 2008-05-28 | 2009-12-02 | 山东省标准化研究院 | The load balance optimization method of food safety trace back database |
| CN102110014A (en) * | 2011-03-15 | 2011-06-29 | 合肥华云通信技术有限公司 | Method for balancing loads of virtual machine (VM) |
| CN102316045A (en) * | 2011-10-19 | 2012-01-11 | 杭州华三通信技术有限公司 | Method for realizing flow rate balance in multi-service transport platform (MSTP) networking and network equipment |
| CN102916896A (en) * | 2011-08-01 | 2013-02-06 | 赛尔网络有限公司 | Method and device for distributing multi-channel port mirroring mixed data stream |
| CN103220354A (en) * | 2013-04-18 | 2013-07-24 | 广东宜通世纪科技股份有限公司 | Method for achieving load balancing of server cluster |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103873504A (en) * | 2012-12-12 | 2014-06-18 | 鸿富锦精密工业（深圳）有限公司 | System enabling data blocks to be stored in distributed server and method thereof |
| CN103312825B (en) * | 2013-07-10 | 2016-08-10 | 中国人民解放军国防科学技术大学 | A kind of distributed data storage method and apparatus |
| CN104580268A (en) * | 2013-10-09 | 2015-04-29 | 南京中兴新软件有限责任公司 | Method and device for transmitting file data |
| CN104158902B (en) * | 2014-08-27 | 2017-09-19 | 浙江宇视科技有限公司 | A kind of Hbase data blocks distribution method and device based on number of request |
-
2015
- 2015-06-24 CN CN201510355202.2A patent/CN106326252B/en active Active
- 2015-09-10 WO PCT/CN2015/089383 patent/WO2016206192A1/en active Application Filing
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101146114A (en) * | 2007-10-12 | 2008-03-19 | 中兴通讯股份有限公司 | A load share method and system of stream media server |
| CN101594371A (en) * | 2008-05-28 | 2009-12-02 | 山东省标准化研究院 | The load balance optimization method of food safety trace back database |
| CN102110014A (en) * | 2011-03-15 | 2011-06-29 | 合肥华云通信技术有限公司 | Method for balancing loads of virtual machine (VM) |
| CN102916896A (en) * | 2011-08-01 | 2013-02-06 | 赛尔网络有限公司 | Method and device for distributing multi-channel port mirroring mixed data stream |
| CN102316045A (en) * | 2011-10-19 | 2012-01-11 | 杭州华三通信技术有限公司 | Method for realizing flow rate balance in multi-service transport platform (MSTP) networking and network equipment |
| CN103220354A (en) * | 2013-04-18 | 2013-07-24 | 广东宜通世纪科技股份有限公司 | Method for achieving load balancing of server cluster |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106326252A (en) | 2017-01-11 |
| WO2016206192A1 (en) | 2016-12-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP2019504412A (en) | Short link processing method, device, and server | |
| CN102984280B (en) | Data backup system and method for social cloud storage network application | |
| CN102857578B (en) | A kind of file uploading method of network hard disc, system and net dish client | |
| CN106326252B (en) | Data processing method and device for database | |
| CN112579602B (en) | Multi-version data storage method, device, computer equipment and storage medium | |
| WO2018113317A1 (en) | Data migration method, apparatus, and system | |
| CN103548004B (en) | The method and apparatus of dynamic data attemper is realized in file system | |
| CN102523290B (en) | Data processing method, device and system | |
| CN104199784B (en) | A kind of data migration method and device based on classification storage | |
| EP2765522B1 (en) | Method and device for data pre-heating | |
| CN104765749A (en) | Data storage method and device | |
| CN110413978B (en) | Data paging export method, device, computer equipment and storage medium | |
| CN107391761B (en) | Data management method and device based on repeated data deletion technology | |
| CN103631894A (en) | Dynamic copy management method based on HDFS | |
| CN108345643A (en) | A kind of data processing method and device | |
| CN104142953A (en) | Metadata dynamic management method and system | |
| CN109145053A (en) | Data processing method and device, client, server | |
| CN111064776A (en) | Method and device for generating blocks in block chain and storage medium | |
| CN107423425A (en) | A kind of data quick storage and querying method to K/V forms | |
| EP2940587A1 (en) | Computer, control device for computer system, and recording medium | |
| CN110704431A (en) | Hierarchical storage management method for mass data | |
| US10581997B2 (en) | Techniques for storing or accessing a key-value item | |
| CN105207993A (en) | Data access and scheduling method in CDN, and system | |
| CN105022833A (en) | Data processing method, nodes and monitoring system | |
| CN113961536B (en) | Method for dynamically calculating MD5 value in storage device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |