CN107391632B - Database storage processing method and device, computing equipment and computer storage medium - Google Patents
Database storage processing method and device, computing equipment and computer storage medium Download PDFInfo
- Publication number
- CN107391632B CN107391632B CN201710555960.8A CN201710555960A CN107391632B CN 107391632 B CN107391632 B CN 107391632B CN 201710555960 A CN201710555960 A CN 201710555960A CN 107391632 B CN107391632 B CN 107391632B
- Authority
- CN
- China
- Prior art keywords
- data
- stored
- database
- value pair
- key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a database storage processing method, a device, a computing device and a computer storage medium, wherein the database storage processing method comprises the following steps: receiving a data storage request which is sent by a client and is suitable for a first database; the data storage request carries data to be stored; judging whether the data to be stored carried by the data storage request is data in a data key value pair form; if the data to be stored is judged to be not in the data key value pair form, converting the data to be stored into conversion data in at least one data key value pair form according to the data type of the data to be stored; wherein the transformed data in the form of at least one data key-value pair has a data tag; storing the transformed data in the form of at least one data key-value pair in a second database. According to the technical scheme provided by the invention, the data storage cost can be effectively reduced, and the database storage mode is optimized.
Description
Technical Field
The invention relates to the technical field of internet, in particular to a database storage processing method, a database storage processing device, a computing device and a computer storage medium.
Background
With the continuous development of internet technology, data storage technology has also been developed rapidly, and many Key-Value-based storage schemes, such as a Redis storage scheme and a RocksDB storage scheme, have appeared. Taking a Redis storage scheme as an example, Redis can support storage of data in a simple data key value pair form, and can also support storage of data types such as a list type (list), a hash type (hash), a set type (set), an ordered set type (zset), and the like. However, Redis is a database that uses memory as a data carrier, and the cost of data storage is very high. Therefore, the existing Key-Value storage scheme has the problem of high data storage cost.
Disclosure of Invention
In view of the above, the present invention has been made to provide a database storage processing method, apparatus, computing device and computer storage medium that overcome or at least partially solve the above-mentioned problems.
According to an aspect of the present invention, there is provided a database storage processing method, including:
receiving a data storage request which is sent by a client and is suitable for a first database; the data storage request carries data to be stored;
judging whether the data to be stored carried by the data storage request is data in a data key value pair form;
if the data to be stored is judged to be not in the data key value pair form, converting the data to be stored into conversion data in at least one data key value pair form according to the data type of the data to be stored; wherein the transformed data in the form of at least one data key-value pair has a data tag;
storing the transformed data in the form of at least one data key-value pair in a second database.
According to another aspect of the present invention, there is provided a database storage processing apparatus, the apparatus comprising:
the receiving module is suitable for receiving a data storage request which is sent by a client and is suitable for a first database; the data storage request carries data to be stored;
the judging module is suitable for judging whether the data to be stored carried by the data storage request is data in a data key value pair form;
the conversion module is suitable for converting the data to be stored into conversion data in the form of at least one data key value pair according to the data type of the data to be stored if the judgment module judges that the data to be stored is not data in the form of the data key value pair; wherein the transformed data in the form of at least one data key-value pair has a data tag;
a storage module adapted to store the transformed data in the form of at least one data key-value pair in a second database.
According to yet another aspect of the present invention, there is provided a computing device comprising: the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the database storage processing method.
According to still another aspect of the present invention, there is provided a computer storage medium having at least one executable instruction stored therein, where the executable instruction causes a processor to perform operations corresponding to the database storage processing method.
According to the technical scheme provided by the invention, a data storage request which is sent by a client and is suitable for a first database is received, whether data to be stored carried by the data storage request is data in a data key value pair form is judged, under the condition that the data to be stored is judged not to be the data in the data key value pair form, the data to be stored is converted into conversion data in at least one data key value pair form according to the data type of the data to be stored, and then the conversion data in the at least one data key value pair form is stored into a second database. According to the technical scheme provided by the invention, under the condition that the normal access of the client is not influenced, the data can be stored into the second database according to the data storage request applicable to the first database, so that the storage pressure of the first database is reduced, the data storage cost is effectively reduced, and the database storage mode is optimized.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 is a flow diagram illustrating a database storage processing method according to one embodiment of the invention;
FIG. 2 is a flow diagram illustrating a database storage processing method according to another embodiment of the invention;
FIG. 3 is a flow diagram illustrating a data query in a database storage processing method according to yet another embodiment of the invention;
FIG. 4 shows a block diagram of a database storage processing apparatus according to an embodiment of the invention;
FIG. 5 is a block diagram showing the structure of a database storage processing apparatus according to another embodiment of the present invention;
FIG. 6 shows a schematic structural diagram of a computing device according to an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Fig. 1 is a flow chart illustrating a database storage processing method according to an embodiment of the present invention, and as shown in fig. 1, the method includes the following steps:
step S100, receiving a data storage request applicable to the first database sent by the client.
The method is executed by an interaction layer built in the second database, and the interaction layer supports the data access protocol of the first database, so that the normal access of the client is not influenced. Then when the client sends a data storage request applicable to the first database, the data storage request applicable to the first database sent by the client may be received in step S100. Wherein, the data storage request carries the data to be stored.
Step S101, judging whether the data to be stored carried by the data storage request is data in a data key value pair form; if yes, go to step S102; if not, step S103 is executed.
The second database is a database based on key value storage. After receiving the data storage request, in step S101, it is determined whether the data to be stored carried by the data storage request is data in the form of data key value pairs. If the data to be stored carried by the data storage request is judged to be data in a data key value pair form, which indicates that the data to be stored can be directly stored into the second database without data conversion, the step S102 is executed; if the data to be stored carried by the data storage request is not data in the form of the data key value pair, which indicates that the data to be stored needs to be converted and can be stored in the second database after conversion, step S103 is executed.
And step S102, storing the data to be stored into a second database in a data key value pair mode.
If it is determined in step S101 that the data to be stored is in the form of data key value pairs, the data to be stored is directly stored in the second database in the form of data key value pairs.
In a specific embodiment, the first database may be a database using memory as a data carrier, i.e. a memory database, and the second database may be a database using a disk as a data carrier, i.e. a disk database. Because the data storage cost of the disk database is far lower than that of the memory database, the method can store the data to be stored into the disk database under the condition of not influencing the normal access of the client, thereby not only reducing the storage pressure of the memory database, but also effectively reducing the data storage cost.
Step S103, converting the data to be stored into conversion data in the form of at least one data key value pair according to the data type of the data to be stored.
In the case that it is determined in step S101 that the data to be stored is not data in the form of a data key value pair, since the second database is a database based on key value storage, the data to be stored cannot be directly stored in the second database, in step S103, data conversion is performed on the data to be stored according to the data type of the data to be stored, the data to be stored is converted into data that the second database supports storage, and specifically, the data to be stored is converted into conversion data in the form of at least one data key value pair. Wherein the transformed data in the form of at least one data key-value pair has a data tag. The data mark is used for indicating the relation between the conversion data in the form of at least one data key-value pair, and the conversion data in the form of at least one data key-value pair can be restored into the original data of the data type according to the data mark.
Step S104, storing the conversion data in the form of at least one data key value pair in a second database.
After converting the data to be stored into the conversion data in the form of at least one data key-value pair, the conversion data in the form of at least one data key-value pair can be stored into the second database.
According to the database storage processing method provided by the embodiment, a data storage request which is sent by a client and is suitable for a first database is received, whether data to be stored carried by the data storage request is data in a data key value pair form is judged, under the condition that the data to be stored is judged not to be the data in the data key value pair form, the data to be stored is converted into conversion data in at least one data key value pair form according to the data type of the data to be stored, and then the conversion data in the at least one data key value pair form is stored into a second database. According to the technical scheme provided by the invention, under the condition that the normal access of the client is not influenced, the data to be stored can be stored in the second database according to the data storage request applicable to the first database, so that the storage pressure of the first database is reduced, the data storage cost is effectively reduced, and the database storage mode is optimized.
Fig. 2 is a flow chart illustrating a database storage processing method according to another embodiment of the present invention, and as shown in fig. 2, the method includes the following steps:
step S200, receiving a data storage request which is sent by a client and is suitable for a first database.
The method is executed by an interaction layer built in a second database, the interaction layer supporting a data access protocol of a first database, and in particular, the interaction layer built in the second database according to the data access protocol of the first database, so that the interaction layer can support the data access protocol of the first database, and then when a client sends a data storage request applicable to the first database, the data storage request applicable to the first database sent by the client can be received in step S200. Wherein, the data storage request carries the data to be stored.
Step S201, judging whether the data to be stored carried by the data storage request is data in a data key value pair form; if yes, go to step S202; if not, go to step S204.
Taking the first database as a Redis database and the second database as a RocksDB database or a LevelDB database as an example, both the first database and the second database can support data in a simple data key value pair form, in addition, the first database can also support storage of data types such as a list type, a hash type, a set type and an ordered set type, and the second database cannot support storage of the data types. Therefore, in step S201, it is necessary to determine whether the data to be stored carried by the data storage request is data in the form of data key value pairs. If the data to be stored carried by the data storage request is judged to be data in a data key value pair form, which indicates that the data to be stored can be directly stored into the second database without data conversion, the step S202 is executed; if the data to be stored carried by the data storage request is not data in the form of the data key value pair, which indicates that the data to be stored needs to be converted and can be stored in the second database after conversion, step S204 is executed.
And S202, compressing the data to be stored by using a preset compression algorithm.
In the case that the data to be stored is determined to be data in the form of data key value pairs in step S201, it is indicated that the data to be stored can be directly stored in the second database without performing data conversion on the data to be stored, and in order to reduce the storage space occupied by data storage, in step S202, the data to be stored is compressed by using a preset compression algorithm. The preset compression algorithm can be set by those skilled in the art according to actual needs, and is not limited herein.
And step S203, storing the compressed data to be stored into a second database.
After the data to be stored is compressed, in step S203, the compressed data to be stored is stored in the second database. In a specific embodiment, the first database is a Redis database, the second database is a RocksDB database or a LevelDB database, the Redis database is a memory database, and the RocksDB database and the LevelDB database are disk databases.
Step S204, converting the data to be stored into conversion data in the form of at least one data key value pair according to the data type of the data to be stored.
Taking the first database as a Redis database and the second database as a RocksDB database or a LevelDB database as an example, the first database can support the storage of data types such as a list type, a hash type, a set type and an ordered set type, but the second database cannot support the storage of the data types, so the invention further provides a data conversion scheme aiming at the data types, wherein the specific content of the data conversion scheme is as follows:
(1) if the data type of the data to be stored is a list type, generating a data mark corresponding to each element according to the name of the data to be stored and the sequence of the elements in the list aiming at each element in the data to be stored; and taking the name of the element as a data key, taking the data mark corresponding to the element and the element value of the element as data values corresponding to the data key, and constructing conversion data in a data key value pair form.
For example, the data type of a certain data to be stored is a list type, the name (name) of the data to be stored is a, the data to be stored comprises a total of three elements, which are named B, C, D, each element having a corresponding element value, in one particular embodiment, the corresponding data tag generated for the first element B may be a1, wherein A in the data label A1 represents the name of the data to which the element belongs, 1 represents the order of the element in the data to which the element belongs, then A1 represents that the element is the first element in the data to be stored with the name of A, and then, taking the name B of the element as a data key, and taking the data tag A1 corresponding to the element and the element value of the element B as data values corresponding to the data key, so as to construct conversion data in the form of a data key value pair corresponding to the element B. In order to distinguish the data mark corresponding to the element and the element value of the element, a separator may be set between the data mark corresponding to the element and the element value of the element, and then the data mark and the element value may be used as the data value corresponding to the data key.
Similarly, the corresponding data label generated for the second element C is a2, the name C of the element is used as a data key, the data label a2 corresponding to the element and the element value of the element C are used as data values corresponding to the data key, and conversion data in the form of a data key value pair corresponding to the element C is constructed; the corresponding data label generated for the third element D is A3, the name D of the element is used as a data key, the data label A3 corresponding to the element and the element value of the element D are used as data values corresponding to the data key, and the conversion data in the form of the data key value pair corresponding to the element D is constructed. In this way, the data to be stored is converted into converted data in the form of three key value pairs of data.
(2) If the data type of the data to be stored is the hash type, generating a data mark corresponding to each element according to the name of the data to be stored aiming at each element in the data to be stored; and taking the name of the element as a data key, taking the data mark corresponding to the element and the element value of the element as data values corresponding to the data key, and constructing conversion data in a data key value pair form.
For example, a data type of a certain data to be stored is a hash type, a name of the data to be stored is H, the data to be stored includes two elements in total, names of the two elements are I and J, each element has a corresponding element value, in a specific embodiment, corresponding data markers generated for the element I and the element J may both be H, where H in the data marker H represents a name of data to which the element belongs, and then, for the element I, the name I of the element is used as a data key, and the data marker H corresponding to the element and the element value of the element I are used as data values corresponding to the data key, so as to construct conversion data in a data key value pair form corresponding to the element I; and regarding the element J, taking the name J of the element as a data key, taking the data mark H corresponding to the element and the element value of the element J as data values corresponding to the data key, and constructing conversion data in a data key value pair form corresponding to the element J. In this way, the data to be stored is converted into converted data in the form of two data key value pairs. In order to distinguish the data mark corresponding to the element and the element value of the element, a separator may be set between the data mark corresponding to the element and the element value of the element, and then the data mark and the element value may be used as the data value corresponding to the data key.
(3) If the data type of the data to be stored is a set type or an ordered set type, generating a data mark corresponding to each element according to the name of the data to be stored and the element value of the element aiming at each element in the data to be stored; and taking the name of the element as a data key, taking the data mark corresponding to the element as a data value corresponding to the data key, and constructing conversion data in a data key value pair form.
For example, the data type of a certain data to be stored is a set type or an ordered set type, the name (name) of the data to be stored is X, the data to be stored includes a total of two elements, the names of the two elements are E and F, respectively, where the element value of the element E is 5, and the element value of the element F is 10. In a specific embodiment, the corresponding data tag generated for the element E may be X5, where X in the data tags X5 represents the name of the data to which the element belongs, and 5 represents the element value of the element, then the name E of the element is used as a data key, and the data tag X5 corresponding to the element is used as a data value corresponding to the data key, so as to construct the conversion data in the form of a data key value pair corresponding to the element E; the corresponding data tag generated for the element F may be X10, where X in the data tags X10 represents the name of the data to which the element belongs, and 10 represents the element value of the element, then the name F of the element is used as the data key, and the data tag X10 corresponding to the element is used as the data value corresponding to the data key, to construct the conversion data in the form of the data key value pair corresponding to the element F.
Step S205, compressing the converted data in the form of at least one data key value pair by using a preset compression algorithm.
In order to reduce the storage space required for the data to be stored, in step S205, the converted data in the form of at least one data key value pair is compressed by using a preset compression algorithm.
Step S206, storing the compressed converted data in the second database.
After the conversion data is compressed, the compressed conversion data is stored in the second database, so that the storage pressure of the first database is reduced, and the storage space occupied by data storage is effectively reduced.
According to the database storage processing method provided by the embodiment, data conversion can be performed on the data according to the data type under the condition that normal access of the client is not influenced, and the data is stored in the second database, so that the storage pressure of the first database is reduced, the data storage cost is effectively reduced, the data is compressed by using a compression algorithm, the storage space occupied by data storage is effectively reduced, and the database storage mode is further optimized.
Fig. 3 is a schematic flow chart illustrating a data query in a database storage processing method according to another embodiment of the present invention, as shown in fig. 3, the method includes the following steps:
step S300, receiving a data query request applicable to the first database sent by the client.
The method is performed by an interaction layer built in the second database, the interaction layer supporting a data access protocol of the first database, and then when the client sends a data query request applicable to the first database, the data query request applicable to the first database sent by the client is received in step S300. Wherein, the data query request carries the name of the data to be queried.
Step S301, judging whether the second database inquires the storage data of which the data key is the name of the data to be inquired; if yes, go to step S302; if not, go to step S303.
And after receiving the data query request, querying the stored data with the data key as the name of the data to be queried from the second database. If the data key is the stored data of the name of the data to be inquired from the second database, executing step S302; if the stored data of which the data key is the name of the data to be queried is not queried from the second database, it indicates that the data to be queried may be data converted, then step S303 is performed.
Step S302, returning the queried storage data to the client as a query result corresponding to the data query request.
And under the condition that the inquired storage data with the data key as the name of the data to be inquired is inquired, returning the inquired storage data to the client as an inquiry result corresponding to the data inquiry request.
In step S303, the stored data having the data tag matching the name of the data to be queried is queried from the second database.
And in the case that the stored data of which the data key is the name of the data to be inquired is not inquired, inquiring the stored data with the data mark matched with the name of the data to be inquired from the second database. Specifically, a data tag including the name of the data to be queried may be used as the data tag matching the name of the data to be queried.
Step S304, all the inquired storage data with the data marks matched with the names of the data to be inquired are combined to obtain combined data.
For example, the name of the data to be queried is a, the data type of the data to be queried is a list type, a piece of stored data with a data tag a1, a piece of stored data with a data tag a2 and a piece of stored data with a data tag A3 are queried from the second database, and all three pieces of stored data are stored data with data tags matching the name of the data to be queried, then the three pieces of stored data are subjected to reduction processing according to the data conversion scheme provided by the invention to obtain three elements in the list with the name of a and element values corresponding to each element, then the three elements are sorted and combined according to the sequence of the elements tagged in the data tags to obtain combined data, the name of the obtained combined data is a, and the data type is the list type.
For another example, the name of the data to be queried is H, the data type of the data to be queried is a hash type, two pieces of stored data with data labels H are queried from the second database, and both the two pieces of stored data are stored data with data labels matching the name of the data to be queried, then the two pieces of stored data are subjected to reduction processing according to the data conversion scheme provided by the present invention, so as to obtain two elements in the hash data with the name of H and an element value corresponding to each element, and then the two elements are subjected to combination processing, so as to obtain combined data, the name of the obtained combined data is H, and the data type is the hash type.
For another example, the name of the data to be queried is X, the data type of the data to be queried is an ordered set type, a piece of storage data with a data tag X5 and a piece of storage data with a data tag X10 are queried from the second database, and both the two pieces of storage data are storage data with a data tag matching the name of the data to be queried, then the two pieces of storage data are reduced according to the data conversion scheme provided by the present invention, so as to obtain two elements in the ordered set named a and an element value corresponding to each element, then the two elements are sorted and combined according to the sorting of the element values of the elements, so as to obtain combined data, the name of the obtained combined data is X, and the data type is the ordered set type.
Step S305, returning the combined data to the client as a query result corresponding to the data query request.
And after the combined data is obtained, returning the combined data to the client as a query result corresponding to the data query request.
The database storage processing method provided by the embodiment can quickly query and combine the second database to obtain corresponding data according to the data query request applicable to the first database, so that the database storage mode is optimized.
Fig. 4 is a block diagram showing a structure of a database storage processing apparatus according to an embodiment of the present invention, wherein the apparatus is executed by an interaction layer built in a second database, and the interaction layer supports a data access protocol of a first database. As shown in fig. 4, the apparatus includes: a receiving module 410, a judging module 420, a converting module 430 and a storing module 440.
The receiving module 410 is adapted to: and receiving a data storage request which is sent by the client and is applicable to the first database.
Wherein, the data storage request carries the data to be stored. In one embodiment, the first database may be an in-memory database and the second database may be a disk database.
The determination module 420 is adapted to: and judging whether the data to be stored carried by the data storage request is data in a data key value pair form.
If the judging module 420 judges that the data to be stored carried by the data storage request is data in a data key value pair form, the storing module 440 is triggered; if the determining module 420 determines that the data to be stored carried by the data storage request is not data in the form of a data key value pair, the converting module 430 is triggered.
The conversion module 430 is adapted to: if the determining module 420 determines that the data to be stored is not in the form of data key value pairs, the data to be stored is converted into conversion data in the form of at least one data key value pair according to the data type of the data to be stored. Wherein the transformed data in the form of at least one data key-value pair has a data tag.
The storage module 440 is adapted to: storing the transformed data in the form of at least one data key-value pair in a second database.
The storage module 440 stores the converted data in the form of at least one data key-value pair converted by the conversion module 430 into the second database.
The storage module 440 is further adapted to: if the determining module 420 determines that the data to be stored is data in the form of data key value pairs, the data to be stored is stored in the second database in the form of data key value pairs.
According to the database storage processing device provided by this embodiment, the receiving module receives a data storage request which is sent by a client and is applicable to a first database, the judging module judges whether data to be stored carried by the data storage request is data in a data key value pair form, the converting module converts the data to be stored into converted data in at least one data key value pair form according to a data type of the data to be stored under the condition that the judging module judges that the data to be stored is not data in the data key value pair form, and the storing module stores the converted data in the at least one data key value pair form into a second database. According to the technical scheme provided by the invention, under the condition that the normal access of the client is not influenced, the data to be stored can be stored in the second database according to the data storage request applicable to the first database, so that the storage pressure of the first database is reduced, the data storage cost is effectively reduced, and the database storage mode is optimized.
Fig. 5 is a block diagram showing a configuration of a database storage processing apparatus according to another embodiment of the present invention, wherein the apparatus is implemented by an interaction layer built in a second database, and the interaction layer supports a data access protocol of a first database. As shown in fig. 5, the apparatus includes: a receiving module 510, a determining module 520, a converting module 530, a storing module 540, a querying module 550, a responding module 560, and a processing module 570.
The receiving module 510 is adapted to: and receiving a data storage request which is sent by the client and is applicable to the first database. Wherein, the data storage request carries the data to be stored.
The determination module 520 is adapted to: and judging whether the data to be stored carried by the data storage request is data in a data key value pair form.
The conversion module 530 is adapted to: if the determining module 520 determines that the data to be stored is not in the form of data key value pairs, the data to be stored is converted into conversion data in the form of at least one data key value pair according to the data type of the data to be stored. Wherein the transformed data in the form of at least one data key-value pair has a data tag.
In particular, the conversion module 530 is further adapted to: if the data type of the data to be stored is a list type, generating a data mark corresponding to each element according to the name of the data to be stored and the sequence of the elements in the list aiming at each element in the data to be stored; and taking the name of the element as a data key, taking the data mark corresponding to the element and the element value of the element as data values corresponding to the data key, and constructing conversion data in a data key value pair form.
The conversion module 530 is further adapted to: if the data type of the data to be stored is the hash type, generating a data mark corresponding to each element according to the name of the data to be stored aiming at each element in the data to be stored; and taking the name of the element as a data key, taking the data mark corresponding to the element and the element value of the element as data values corresponding to the data key, and constructing conversion data in a data key value pair form.
The conversion module 530 is further adapted to: if the data type of the data to be stored is a set type or an ordered set type, generating a data mark corresponding to each element according to the name of the data to be stored and the element value of the element aiming at each element in the data to be stored; and taking the name of the element as a data key, taking the data mark corresponding to the element as a data value corresponding to the data key, and constructing conversion data in a data key value pair form.
The storage module 540 is adapted to: compressing the converted data in the form of at least one data key value pair by using a preset compression algorithm; and storing the compressed conversion data into a second database.
The storage module 540 is further adapted to: if the judgment module 520 judges that the data to be stored is data in a data key value pair form, compressing the data to be stored by using a preset compression algorithm; and storing the compressed data to be stored into a second database.
The receiving module 510 is further adapted to: and receiving a data query request which is sent by a client and is applicable to the first database. Wherein, the data query request carries the name of the data to be queried.
The query module 550 is adapted to: and querying the stored data of which the data key is the name of the data to be queried from the second database.
The response module 560 is adapted to: if the query module 550 queries the storage data with the data key as the name of the data to be queried, returning the queried storage data to the client as a query result corresponding to the data query request.
The processing module 570 is adapted to: if the query module 550 does not query the storage data of which the data key is the name of the data to be queried, querying the storage data with the data tag matched with the name of the data to be queried from the second database; and combining all the inquired storage data with the data marks matched with the names of the data to be inquired to obtain combined data.
The response module 560 is further adapted to: and returning the combined data serving as a query result corresponding to the data query request to the client.
According to the database storage processing device provided by the embodiment, data conversion can be performed on the data according to the data type under the condition that normal access of the client is not influenced, and the data is stored in the second database, so that the storage pressure of the first database is reduced, the data storage cost is effectively reduced, the data is compressed by using a compression algorithm, the storage space occupied by data storage is effectively reduced, and the database storage mode is further optimized; in addition, according to the data query request applicable to the first database, corresponding data can be rapidly queried and combined from the second database.
The invention also provides a nonvolatile computer storage medium, wherein the computer storage medium stores at least one executable instruction, and the computer executable instruction can execute the database storage processing method in any method embodiment.
Fig. 6 is a schematic structural diagram of a computing device according to an embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the computing device.
As shown in fig. 6, the computing device may include: a processor (processor)602, a communication Interface 604, a memory 606, and a communication bus 608.
Wherein:
the processor 602, communication interface 604, and memory 606 communicate with one another via a communication bus 608.
A communication interface 604 for communicating with network elements of other devices, such as clients or other servers.
The processor 602 is configured to execute the program 610, and may specifically perform relevant steps in the foregoing database storage processing method embodiment.
In particular, program 610 may include program code comprising computer operating instructions.
The processor 602 may be a central processing unit CPU or an application Specific Integrated circuit asic or one or more Integrated circuits configured to implement embodiments of the present invention. The computing device includes one or more processors, which may be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.
And a memory 606 for storing a program 610. Memory 606 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The program 610 may specifically be configured to cause the processor 602 to execute the database storage processing method in any of the method embodiments described above. For specific implementation of each step in the program 610, reference may be made to corresponding steps and corresponding descriptions in units in the above database storage processing embodiments, which are not described herein again. It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described devices and modules may refer to the corresponding process descriptions in the foregoing method embodiments, and are not described herein again.
The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.
The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functionality of some or all of the components in accordance with embodiments of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.
Claims (20)
1. A database storage processing method, the method comprising:
receiving a data storage request which is sent by a client and is suitable for a first database; the data storage request carries data to be stored;
judging whether the data to be stored carried by the data storage request is data in a data key value pair form;
if the data to be stored is judged to be data in a data key value pair form, storing the data to be stored in a second database in the data key value pair form;
if the data to be stored is judged to be not in the data key value pair form, converting the data to be stored into conversion data in at least one data key value pair form according to the data type of the data to be stored; wherein the conversion data in the form of the at least one data key-value pair has a data tag, and the data tag is data for restoring the conversion data in the form of the at least one data key-value pair into the data to be stored;
storing the transformed data in the form of the at least one data key-value pair in a second database;
wherein converting the data to be stored into conversion data in the form of at least one data key-value pair according to the data type of the data to be stored further comprises:
if the data type of the data to be stored is a list type, generating a data mark corresponding to each element in the data to be stored according to the name of the data to be stored and the sequence of the elements in the list;
and taking the name of the element as a data key, taking the data mark corresponding to the element and the element value of the element as data values corresponding to the data key, and constructing conversion data in a data key value pair form.
2. The method of claim 1, wherein the converting the data to be stored into converted data in the form of at least one data key-value pair according to the data type of the data to be stored further comprises:
if the data type of the data to be stored is the hash type, generating a data mark corresponding to each element in the data to be stored according to the name of the data to be stored;
and taking the name of the element as a data key, taking the data mark corresponding to the element and the element value of the element as data values corresponding to the data key, and constructing conversion data in a data key value pair form.
3. The method of claim 1, wherein the converting the data to be stored into converted data in the form of at least one data key-value pair according to the data type of the data to be stored further comprises:
if the data type of the data to be stored is a set type or an ordered set type, generating a data mark corresponding to each element in the data to be stored according to the name of the data to be stored and the element value of the element;
and taking the name of the element as a data key, taking the data mark corresponding to the element as a data value corresponding to the data key, and constructing conversion data in a data key value pair form.
4. The method of claim 1, wherein said storing the transformed data in the form of the at least one data key-value pair into a second database further comprises:
compressing the converted data in the form of the at least one data key value pair by using a preset compression algorithm;
and storing the compressed conversion data into a second database.
5. The method of claim 1, wherein the method further comprises:
and if the data to be stored is judged to be data in a data key value pair form, storing the data to be stored into a second database in the data key value pair form.
6. The method of claim 5, wherein the storing the data to be stored in the form of data key-value pairs into a second database further comprises:
compressing the data to be stored by using a preset compression algorithm;
and storing the compressed data to be stored into a second database.
7. The method of claim 1, wherein the method further comprises:
receiving a data query request which is sent by a client and is suitable for a first database; the data query request carries the name of the data to be queried;
inquiring the stored data of which the data key is the name of the data to be inquired from a second database;
and if the queried data key is the storage data of the name of the data to be queried, returning the queried storage data to the client as a query result corresponding to the data query request.
8. The method of claim 7, wherein the method further comprises:
if the data key is not inquired as the storage data of the name of the data to be inquired, inquiring the storage data with the data mark matched with the name of the data to be inquired from a second database;
all the inquired storage data with the data marks matched with the names of the data to be inquired are combined to obtain combined data;
and returning the combined data to the client as a query result corresponding to the data query request.
9. The method of any of claims 1-7, wherein the method is performed by an interaction layer built into the second database, the interaction layer supporting a data access protocol of the first database.
10. A database storage processing apparatus, the apparatus comprising:
the receiving module is suitable for receiving a data storage request which is sent by a client and is suitable for a first database; the data storage request carries data to be stored;
the judging module is suitable for judging whether the data to be stored carried by the data storage request is data in a data key value pair form;
the conversion module is suitable for converting the data to be stored into conversion data in the form of at least one data key value pair according to the data type of the data to be stored if the judgment module judges that the data to be stored is not data in the form of the data key value pair; wherein the conversion data in the form of the at least one data key-value pair has a data tag, and the data tag is data for restoring the conversion data in the form of the at least one data key-value pair into the data to be stored;
a storage module adapted to store the transformed data in the form of the at least one data key-value pair into a second database; if the judging module judges that the data to be stored is data in a data key value pair form, storing the data to be stored into a second database in the data key value pair form;
wherein the conversion module is further adapted to: if the data type of the data to be stored is a list type, generating a data mark corresponding to each element in the data to be stored according to the name of the data to be stored and the sequence of the elements in the list; and taking the name of the element as a data key, taking the data mark corresponding to the element and the element value of the element as data values corresponding to the data key, and constructing conversion data in a data key value pair form.
11. The apparatus of claim 10, wherein the conversion module is further adapted to:
if the data type of the data to be stored is the hash type, generating a data mark corresponding to each element in the data to be stored according to the name of the data to be stored;
and taking the name of the element as a data key, taking the data mark corresponding to the element and the element value of the element as data values corresponding to the data key, and constructing conversion data in a data key value pair form.
12. The apparatus of claim 10, wherein the conversion module is further adapted to:
if the data type of the data to be stored is a set type or an ordered set type, generating a data mark corresponding to each element in the data to be stored according to the name of the data to be stored and the element value of the element;
and taking the name of the element as a data key, taking the data mark corresponding to the element as a data value corresponding to the data key, and constructing conversion data in a data key value pair form.
13. The apparatus of claim 10, wherein the storage module is further adapted to:
compressing the converted data in the form of the at least one data key value pair by using a preset compression algorithm;
and storing the compressed conversion data into a second database.
14. The apparatus of claim 10, wherein the storage module is further adapted to:
and if the judging module judges that the data to be stored is data in a data key value pair form, storing the data to be stored into a second database in the data key value pair form.
15. The apparatus of claim 14, wherein the storage module is further adapted to:
compressing the data to be stored by using a preset compression algorithm;
and storing the compressed data to be stored into a second database.
16. The apparatus of claim 10, wherein the receiving means is further adapted to: receiving a data query request which is sent by a client and is suitable for a first database; the data query request carries the name of the data to be queried;
the device further comprises:
the query module is suitable for querying the data key from the second database as the storage data of the name of the data to be queried;
and the response module is suitable for returning the inquired storage data to the client as an inquiry result corresponding to the data inquiry request if the inquiry module inquires the storage data of which the data key is the name of the data to be inquired.
17. The apparatus of claim 16, wherein the apparatus further comprises:
the processing module is suitable for querying the stored data with the data marks matched with the name of the data to be queried from a second database if the query module does not query the stored data with the data keys as the name of the data to be queried; all the inquired storage data with the data marks matched with the names of the data to be inquired are combined to obtain combined data;
the response module is further adapted to: and returning the combined data to the client as a query result corresponding to the data query request.
18. The apparatus according to any of claims 10-16, wherein the apparatus is performed by an interaction layer built into the second database, the interaction layer supporting the data access protocol of the first database.
19. A computing device, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the corresponding operation of the database storage processing method according to any one of claims 1-9.
20. A computer storage medium having at least one executable instruction stored therein, the executable instruction causing a processor to perform operations corresponding to the database storage processing method according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710555960.8A CN107391632B (en) | 2017-06-30 | 2017-06-30 | Database storage processing method and device, computing equipment and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710555960.8A CN107391632B (en) | 2017-06-30 | 2017-06-30 | Database storage processing method and device, computing equipment and computer storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107391632A CN107391632A (en) | 2017-11-24 |
| CN107391632B true CN107391632B (en) | 2021-02-23 |
Family
ID=60333855
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710555960.8A Active CN107391632B (en) | 2017-06-30 | 2017-06-30 | Database storage processing method and device, computing equipment and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107391632B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108228799B (en) * | 2017-12-29 | 2021-09-28 | 北京奇虎科技有限公司 | Object index information storage method and device |
| CN109032724B (en) * | 2018-06-29 | 2022-06-24 | 武汉斗鱼网络科技有限公司 | Method, device, terminal and storage medium for processing user-defined bullet screen message |
| CN109086430A (en) * | 2018-08-10 | 2018-12-25 | 北京奇虎科技有限公司 | Data storage handling method, calculates equipment and computer storage medium at device |
| CN110222093A (en) * | 2019-06-12 | 2019-09-10 | 中国神华能源股份有限公司 | Handle the method, apparatus and storage medium of train data |
| CN111008230B (en) * | 2019-11-22 | 2023-08-04 | 远景智能国际私人投资有限公司 | Data storage method, device, computer equipment and storage medium |
| CN111144926B (en) * | 2019-11-27 | 2023-07-18 | 泰康保险集团股份有限公司 | Service request processing method, device, system, electronic equipment and readable medium |
| CN111400308B (en) * | 2020-02-21 | 2023-05-26 | 中国平安财产保险股份有限公司 | Processing method of cache data, electronic device and readable storage medium |
| CN113868498A (en) * | 2021-09-29 | 2021-12-31 | 珠海格力电器股份有限公司 | Data storage method, electronic device, device and readable storage medium |
| CN114328453A (en) * | 2021-12-27 | 2022-04-12 | 奇安信科技集团股份有限公司 | KV database data management method and device, computing equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8447791B1 (en) * | 2012-04-02 | 2013-05-21 | Bmc Software, Inc. | Systems and methods for altering limit key values |
| CN104156278A (en) * | 2014-08-01 | 2014-11-19 | 江苏大学 | File version control system and file version control method |
| CN104980462A (en) * | 2014-04-04 | 2015-10-14 | 深圳市腾讯计算机系统有限公司 | Distributed computation method, distributed computation device and distributed computation system |
| CN105159985A (en) * | 2015-08-31 | 2015-12-16 | 努比亚技术有限公司 | Data query device and method based on redis cluster |
| CN105260464A (en) * | 2015-10-16 | 2016-01-20 | 北京奇虎科技有限公司 | Data storage structure conversion method and apparatus |
-
2017
- 2017-06-30 CN CN201710555960.8A patent/CN107391632B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8447791B1 (en) * | 2012-04-02 | 2013-05-21 | Bmc Software, Inc. | Systems and methods for altering limit key values |
| CN104980462A (en) * | 2014-04-04 | 2015-10-14 | 深圳市腾讯计算机系统有限公司 | Distributed computation method, distributed computation device and distributed computation system |
| CN104156278A (en) * | 2014-08-01 | 2014-11-19 | 江苏大学 | File version control system and file version control method |
| CN105159985A (en) * | 2015-08-31 | 2015-12-16 | 努比亚技术有限公司 | Data query device and method based on redis cluster |
| CN105260464A (en) * | 2015-10-16 | 2016-01-20 | 北京奇虎科技有限公司 | Data storage structure conversion method and apparatus |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107391632A (en) | 2017-11-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107391632B (en) | Database storage processing method and device, computing equipment and computer storage medium | |
| CN110674432A (en) | Second-level caching method and device and computer readable storage medium | |
| US9336001B2 (en) | Dynamic instrumentation | |
| CN105608159B (en) | The method and apparatus of data buffer storage | |
| CN110968478B (en) | Log acquisition method, server and computer storage medium | |
| CN111737564B (en) | Information query method, device, equipment and medium | |
| CN103731498B (en) | Big data real-time enquiry system load balancing method based on copy selection | |
| CN105373746B (en) | A kind of distributed data processing method and apparatus | |
| CN115113997A (en) | Task scheduling method and device, electronic equipment and storage medium | |
| CN114549068A (en) | Short link generation method, equipment, device and computer readable storage medium | |
| US20150248447A1 (en) | Searching entity-key associations using in-memory objects | |
| CN112598514B (en) | Cross-chain transaction management method, cross-chain platform and medium based on block chain | |
| CN112015806A (en) | Method and device for storing data by block chain | |
| CN109213972B (en) | Method, device, equipment and computer storage medium for determining document similarity | |
| CN106095511A (en) | A kind of server updating method and apparatus | |
| CN113377791A (en) | Data processing method, system and computing equipment | |
| CN110221778A (en) | Processing method, system, storage medium and the electronic equipment of hotel's data | |
| JP6634938B2 (en) | Analysis support method, analysis support program, and analysis support device | |
| CN115033549A (en) | File link storage method and device based on block chain | |
| CN1996256A (en) | Processing method for use in client terminal having an access to EJB through remote interface and device therefor | |
| CN112181825A (en) | Test case library construction method and device, electronic equipment and medium | |
| CN105389257A (en) | System parameter processing method, apparatus and system | |
| CN104572649A (en) | Data processing method, device and system for distributed storage system | |
| US12067073B2 (en) | URL exchange | |
| CN112905602B (en) | Data comparison method, computing device and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |