JP2020038623A - Method, device, and system for storing data - Google Patents

Abstract

To provide a method, a system, an electronic device, and a computer program for storing data in a dispersion-type object storage.SOLUTION: A method includes steps of: determining a type of a storage target record on the basis of a size of data of the storage target record (step 201); searching for current storage information of the type of the storage target record in a storage file (step 202); determining whether or not there is a remaining storage space in a current allocated data block on the basis of the current storage information (step 203); and allocating a new data bock to the type in response to the determination of the absence of the remaining storage space and storing the storage target record into a new data block (step 204).SELECTED DRAWING: Figure 2

Description

  Embodiments of the present invention relate to the technical field of distributed object storage, and in particular, to a method, apparatus, and system for storing data.

  Generally, a distributed storage system stores data in a distributed manner among a plurality of independent devices. Conventional network storage systems use a centralized storage server to store all data. At this time, the storage server becomes a bottleneck in system performance and also becomes a focus on reliability and security, and cannot meet the needs of large-scale storage applications. A distributed network storage system often uses a scalable system structure, uses a plurality of storage servers to share a storage load, and uses a location server to determine the location of storage information. This not only increases system reliability, availability, and access efficiency, but also facilitates scalability.

  Embodiments of the present invention provide methods, devices, and systems for storing data.

  In a first aspect, embodiments of the present invention provide a method for storing data. The method is used for a standalone storage engine of a distributed object storage, wherein a storage file is configured on a disk of the standalone storage engine, a storage space of the storage file is divided into at least two data blocks, and the at least two data Using a linked list structure between the blocks, the method determines the type of the storage target record based on the size of the data of the storage target record, and stores the current storage information of the type in the storage file. Searching, wherein the current storage information includes information of a currently allocated data block and information of a currently stored record of the data block, and The size of the data of the records stored in the same type is the same, and the size of the data of the records stored in the different types is different, and the data currently allocated based on the current storage information is different. Allocating a new data block to the type in response to determining whether there is any remaining storage space in the block and in response to determining that there is no remaining storage space, Storing in said new data block.

  In some embodiments, the method for storing data is such that, in response to the determination that the remaining storage space is present, the remaining storage space is not less than a size of the data of the stored record. Further determining whether or not the remaining storage space is smaller than the size of the data of the storage target record. Further comprising: storing in a space; and allocating a new data block to the type, and storing the remaining data of the record to be stored in the new data block.

  In some embodiments, the storage space of the at least two data blocks is the same, and the sizes of data of records stored in the different types are all integer multiples of a predetermined number.

  In some embodiments, the method for storing data includes updating the current storage information of the type, generating post-storage location information of the storage object record, and outputting the location information. Further, the location information includes at least one of the type of a record, an identifier of a record, and an identifier of a data block where the record is located.

  In some embodiments, at least two of the storage files are configured on the disk, and index information of each of the storage files is stored in a directory of the disk.

  In some embodiments, the method for storing data includes retrieving location information of the record to be read from the directory, and determining the type and the type of the record to be read based on the location information of the record to be read. Determining a position offset in a data block located in the corresponding storage file; determining a length of the record to be read based on the type of the record to be read; and determining a position indicated by the position offset. And reading data having a length corresponding to the length of the record to be read, and outputting the read data as the record to be read.

  In some embodiments, the method for storing data comprises retrieving location information of the record to be deleted from the directory, and, based on the location information of the record to be deleted, deleting the location in the corresponding storage file. Obtaining the current storage information of the type to which the target record belongs; reading the last record from the currently stored record based on the obtained current storage information; and deleting the read record as the deletion target. The method further includes transferring and storing the data at the position where the record is located, and clearing data in a data block before the transfer of the read record, and correcting position information of the read record.

  In some embodiments, after clearing the data in the data block prior to the transfer of the read record, the method for storing the data includes storing the read record in a data block located prior to the transfer. It further includes determining whether data exists or not, and if no data exists in the data block, retrieving the data block for reassignment.

  In some embodiments, prior to determining the type of the storage target record based on a size of the data of the storage target record, the method for storing the data may include dividing the waiting-for-storage object. For the sub-object in the at least one sub-object obtained, the sub-object is coded to obtain a duplicate, and the description information and data of the duplicate are subjected to an arraying process. Generating a storage target record.

  In some embodiments, generating the storage object record of the sub-object is such that the size of the duplicated data after arraying is the same as one of the sizes of the data corresponding to each of the types in the storage file. And determining whether or not the size of the duplicate is all different from the size of the data corresponding to each type, and smaller than the size of the data corresponding to some of the types, The data after the data is padded with zeros, the size of the data of the duplicate after the padding with zeros is the same as the size of the data corresponding to the target type, and one storage object record of the sub-object is generated. And when the size of the data of the duplicate is larger than the maximum value of the size of the data corresponding to each type, By filling with division zero, the size of the data of each duplicate after division is made to be the same as one of the sizes of the data corresponding to each type, and at least two storage target records of the sub-object are stored. Generating, wherein the target type is the type having the smallest size of corresponding data of the some of the types.

  In a second aspect, embodiments of the present invention provide an apparatus for storing data. The device is configured as a stand-alone storage engine of distributed object storage, a storage file is configured on a disk of the stand-alone storage engine, a storage space of the storage file is divided into at least two data blocks, and the at least two data blocks. Using a linked list structure between the blocks, the device comprising: a type determining unit configured to determine a type of the storage target record based on a size of the data of the storage target record; and A search unit configured to search for current storage information of a type, wherein the current storage information is information of a currently allocated data block and a currently stored one of the data blocks. A search unit, wherein the size of data of records stored in the storage file of the same type is the same, and the size of data of records stored in the different type of the storage file is different. Based on the storage information, a space determining unit configured to determine whether there is any remaining storage space in the currently allocated data block, and determining that the remaining storage space is not present. A first allocation unit configured to allocate a new data block to the type and store the storage target record in the new data block.

  In some embodiments, the apparatus for storing the data does not have the remaining storage space less than the size of the data of the record to be stored in response to the determination that there is remaining storage space. A size determining unit arranged to further determine whether or not the remaining storage space is smaller than the data size of the storage target record; A second allocation unit that is stored in the remaining storage space, allocates a new data block to the type, and is arranged to store the remaining data of the storage target record.

  In some embodiments, the storage space of at least two data blocks is the same, and the size of the data of records stored in different types are all integer multiples of a predetermined number.

  In some embodiments, an apparatus for storing the data is arranged to update current storage information of the type, generate location information after storing the record to be stored, and output the location information. The apparatus further includes a position generation unit, wherein the position information includes at least one of a record type, a record identifier, and an identifier of a data block where the record is located.

  In some embodiments, at least two storage files are configured on a disk, and index information of each of the storage files is stored in a directory of the disk.

  In some embodiments, the apparatus for storing the data comprises: a first location search unit arranged to retrieve location information of the record to be read from the directory; and A determination unit arranged to determine a position offset in a data block located in a type of the record to be read and a corresponding storage file, and a length of the record to be read is determined based on the type of the record to be read. A reading unit arranged to read data whose length is the length of the record to be read, starting from the position indicated by the position offset, and to output the read data as the record to be read. Prepare.

  In some embodiments, the apparatus for storing the data comprises: a second location search unit arranged to retrieve location information of the record to be deleted from the directory; and An acquisition unit arranged to acquire the current storage information of the type belonging to the storage file corresponding to the record to be deleted; and, based on the acquired current storage information, the last record from the currently stored record. A transfer unit arranged to transfer and store the read and read record to the position where the record to be deleted is located, and clear the data in the data block before transfer of the read record and read the read record. A correction unit arranged to correct the positional information of the record.

  In some embodiments, the apparatus for storing the data determines whether the data still exists in the data block located before the read record is transferred, and stores the data in the data block. If not, the system further comprises a collection unit arranged to collect the data block for reassignment.

  In some embodiments, the apparatus for storing the data, for a sub-object in at least one sub-object obtained by dividing the object to be stored, codes the sub-object to obtain a duplicate. And a record generation unit arranged to perform an arraying process on the description information and the data of the duplicate and generate a storage target record of the sub-object.

  In some embodiments, the record generation unit determines whether the size of the data of the duplicate after arrangement is the same as one of the sizes of the data corresponding to each type in the storage file. When the size of the data of the determined subunit and the data of the duplicate are all different from the size of the data corresponding to each type and smaller than the size of the data corresponding to some types, A first generation sub that fills the trailing part with zeros, makes the size of the data of the duplicate after the padding with zero the same as the size of the data corresponding to the target type, and generates one storage target record of the sub copy of the subobject. Unit, wherein the target type is the first generation sub-unit, which is the type of the corresponding type having the smallest data size. If the size of the data of the duplicate is larger than the maximum value of the size of the data corresponding to each type, the size of the data of each duplicate after division is set to one of the sizes of the data corresponding to each type. A second generation sub-unit arranged to split and zero-pad the duplicate and generate at least two storage target records of the sub-object.

  In a third aspect, an embodiment of the present invention comprises a first subsystem, a second subsystem, and a third subsystem having the stand-alone storage engine of any one embodiment of the first aspect installed. Provide system. The first subsystem receives a storage request including a storage-waiting object transmitted by a user, divides the storage-waiting object into at least one sub-object, and performs a communication between the storage-waiting object and the at least one sub-object. Transmitting the correspondence between the second subsystem and the at least one sub-object to the third subsystem, wherein the second subsystem is configured to transmit the at least one sub-object and the at least one sub-object. The third subsystem is configured to store a correspondence between the sub-object and the sub-object by performing coding and arrangement processing on sub-objects in the at least one sub-object. Generate records to be stored Storing the the storage target record.

  In some embodiments, the third subsystem is further configured to send response information to the first subsystem to indicate data storage completion, wherein the first subsystem further comprises the response If information is received, a query identifier of the object to be stored is generated, and the query identifier is fed back to the user.

  In some embodiments, the first subsystem is further configured to receive a read request including a query identifier transmitted by the user, and to transmit the query identifier in the read request to the second subsystem. Wherein the second subsystem is further configured to obtain a sub-object list corresponding to an object indicated by a query identifier in the read request, and to transmit the sub-object list to the third subsystem, The third subsystem reads a corresponding record based on the sub-object list, analyzes the read record to obtain object data, and the first subsystem feeds back the object data to the user. To the object data Further configured to transmit to the subsystem.

  In a fourth aspect, an embodiment of the present invention includes one or more processors and a storage device for storing one or more programs, wherein the one or more programs store the one or more programs. The one or more processors provide an electronic device that, when executed by a plurality of processors, implements the method of any embodiment of the first aspect.

  In a fifth aspect, an embodiment of the present invention relates to a computer readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any embodiment of the first aspect is implemented. I will provide a.

  A method, apparatus, and system for storing data provided by an embodiment of the present invention can determine a type of a storage target record based on a size of the data of the storage target record. Thereby, the current storage information of the type can be searched in the storage file. The current storage information may include information on a currently allocated data block and information on a currently stored record of the data block. Here, the data size of records stored in the same type in the storage file is the same, and the data size of records stored in different types is different. Further, based on the current storage information, it can be determined whether there is any remaining storage space in the currently allocated data block. If it is determined that there is no remaining storage space, a new data block can be assigned to the type. Also, the record to be stored can be stored in a new data block. Thereby, the occurrence of space debris can be reduced, and the space utilization rate of the disk can be improved. This is also advantageous for improving the read / write performance of the disk.

Other features, objects and advantages of the present invention will become more apparent from the detailed description given to non-limiting embodiments with reference to the drawings.
FIG. 2 is an exemplary system architecture diagram to which one embodiment of the present invention is applied. 5 is a flowchart of one embodiment of a method for storing data according to the present invention. 5 is a flowchart of another embodiment of a method for storing data according to the present invention. 5 is a flowchart of another embodiment of a method for storing data according to the present invention. FIG. 3 is a schematic diagram of a logical structure of one embodiment of a storage file according to the present invention. FIG. 3 is a schematic structural diagram of one embodiment of a record in the present invention. FIG. 2 is a schematic structural view of one embodiment of a disk according to the present invention. 1 is a schematic structural diagram of one embodiment of an apparatus for storing data according to the present invention. FIG. 3 is a timing diagram of one embodiment of a system for storing data according to the present invention. FIG. 1 is a schematic structural diagram of a computer system for realizing an electronic device according to an embodiment of the present invention.

  Hereinafter, the present invention will be described in more detail with reference to the drawings and embodiments. It is understood that the specific embodiments described herein are not intended to be limiting on the invention, but merely to interpret the related invention. In addition, for convenience of explanation, the drawings show only parts related to the present invention.

  It should be pointed out that, where there is no inconsistency, the embodiments of the present invention and the features in the embodiments can be combined with each other. Hereinafter, the present invention will be described in detail with reference to the drawings in connection with embodiments.

  FIG. 1 illustrates an exemplary system architecture 100 of a method, apparatus, and system for storing data to which embodiments of the present invention may be applied.

  As shown in FIG. 1, the system architecture 100 can include terminals 101, 102, and 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the terminals 101, 102, 103 and the server 105. Network 104 may include various connections, such as a wired, wireless communication link, or fiber optic cable.

  The user can use the terminals 101, 102, and 103 to interact with the server 105 via the network 104 to send and receive messages and the like. Various client applications such as a web browser, a shopping application, a video application, a mailbox, and an instant messaging tool can be installed in the terminals 101, 102, and 103.

  Here, the terminals 101, 102, and 103 can be hardware or software. When the terminals 101, 102, and 103 are hardware, various electronic devices having a display screen may be used. The electronic device may be a smartphone, a tablet computer, a smart TV, an electronic book reader, an MP3 (Moving Picture Experts Group Audio Layer III, a video professional group audio layer 3) player, a laptop computer, a desktop computer, or the like. However, it is not limited to these. If the terminals 101, 102, and 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (eg, used to provide distributed services) or as a single software or software module. There is no specific limitation here.

  The server 105 can be a server that provides various services, for example, a background server that provides support for various applications installed on the terminals 101, 102, and 103. The background server can perform analysis processing on a data processing request (for example, a data storage request) transmitted by the user via the terminals 101, 102, and 103, and can process the processing result (for example, that the storage is completed). The feedback information to be displayed or the position information after data storage) can be returned to the terminals 101, 102, and 103.

  Here, the server 105 can be the same hardware, and can also be software. When the server 105 is hardware, it can be realized by a distributed server group composed of a plurality of servers, or can be realized by one server. When the server 105 is software, the server 105 may be implemented by a plurality of software or software modules (for example, used for providing a distributed service), or may be implemented by one software or software module. There is no specific limitation here.

  For example, in the server 105, a plurality of storage files 1051 and 1052 can be configured on a disk (that is, a storage space) for storing data. These storage files can be actual containers for storing data, thereby providing distributed storage of data.

  It should be noted that the method for storing data provided by embodiments of the present invention is generally performed by server 105. Correspondingly, a device for storing data is generally installed in server 105.

  It should be understood that the numbers of terminals, networks, servers and storage files in the servers in FIG. 1 are merely exemplary. Depending on the realization demand, any number of terminals, networks, servers and storage files in the servers can be provided.

  With continued reference to FIG. 2, a flow 200 of one embodiment of a method for storing data according to the present invention is shown. A method for storing such data may include the following steps.

  In step 201, the type of a storage target record is determined based on the data size of the storage target record.

  In this embodiment, the method for storing data can be used in a standalone storage engine for distributed object storage. Also, a storage file can be configured on the disk of the standalone storage engine. Here, the storage file may be an actual container for storing data. That is, the data is stored in a storage file. Here, the storage space of the storage file can be divided into at least two data blocks. The storage space of the storage file can be further divided into a plurality of storage sections (ie, data blocks). Also, a linked list structure can be used between at least two data blocks to ensure continuity of storage space. That is, each data block in the same storage file is linked by the link list method.

  It should be noted that the storage capacity of the storage file can be set according to the actual situation, for example, 32G. Also, the storage space of each data block in the same storage file can be the same (eg, 2M) or different. In the storage file, the storage target data can be classified and stored based on the size of the storage target data. Also, data of only one size can be stored for each type. That is, the size of data stored of the same type can be the same. The size of data stored in different types can be different. Here, the classification method is not limited in the present invention.

  In the present embodiment, the entity that executes the method for storing data (for example, the server 105 shown in FIG. 1) transmits the data to the terminal (for example, shown in FIG. The storage target record can be obtained from the terminals 101, 102, 103) or the cloud. In addition, the type of the storage target record is determined by a predetermined classification method based on the data size of the storage target record. For example, the data size of the storage target record is 4 KB. If the data size is larger than 0 KB and not larger than 4 KB, the data belongs to the first type. In this case, the execution subject can determine the type of the storage target record as the first type. Here, the storage target record can be any data to be stored using the disk distributed object storage technology. The content may include a character string of at least one character code such as a number, a character, an alphabet or a symbol.

  In certain optional embodiments of the present embodiment, for convenience of management, the storage space of at least two data blocks in the same storage file may be the same. The sizes of data of records stored in different types in the same storage file can all be integer multiples of a predetermined numerical value. This helps to simplify the calculation process and improve processing efficiency. As shown in FIG. 5A, the storage space of the data blocks is all 2M. Considering that the minimum value of the size of data is generally 4 KB, the sizes of data corresponding to different types are all integral multiples of 4 KB. In consideration of the read / write performance of the disk, the maximum value of the data size can be set according to the actual situation, and can be set to, for example, 512 KB. In other words, the size of the record data in the storage file can be divided into a plurality of types such as 4 KB, 8 KB,..., 512 KB.

  In step 202, the storage file is searched for current storage information of the type, the current storage information including information of a currently allocated data block and information of a currently stored record of the data block.

  In this embodiment, the execution subject can search the storage file for the current storage information of the type to which the record to be stored belongs. Here, the current storage information may include information of a currently allocated data block and currently stored record information of the data block. That is, the information of the data block currently allocated to the type and the currently stored record information of the data block are searched. Here, the information of the data block includes at least one of a link order between the data blocks, an identifier of the data block, and a storage space of the data block, but is not limited thereto. The record information includes, but is not limited to, at least one of a record identifier, a record data size, a data block identifier where the record is located, and the like. Here, the identifier may be used to uniquely indicate a data block or a record, and may be at least one of a character code such as a numeral and an alphabet. In other words, after searching for the current storage information of the type, the execution subject knows what data blocks exist for the type and what records are stored.

  It should be noted that records of the same type are generally stored one by one next to each other. That is, these records logically occupy a continuous space as a whole. This can improve storage space utilization by preventing the occurrence of disk fragments.

  As an example, assuming that the storage target record in FIG. 5A is X, if the execution subject determines that the type is 128, the current storage information of the type can be searched. That is, the storage space is a 2M data block (block) 37, the number of records (records) is 4, and the data size of each record is 512 KB. It can be seen that numerical numbers (1, 2, 3, etc.) are used as record identifiers. In this way, not only can different records of the same type be distinguished, but also the storage order of each record can be easily determined.

  In step 203, based on the current storage information, it is determined whether there is any remaining storage space in the currently allocated data block.

  In this embodiment, based on the current storage information retrieved in step 202, the execution subject has a remaining storage space in the currently allocated data block of the type (ie, the type to which the record to be stored belongs). Can be determined. By way of example, as shown in FIG. 5A, based on the current storage information during type = 128, the execution subject can determine that there is no remaining storage space in block 37, ie, 512KB × 4 = 2M. is there.

  Here, when the remaining storage space exists in the currently allocated data block, the execution subject can store the storage target record in the remaining storage space. If there is no remaining storage space in the currently allocated data block, the execution entity can subsequently execute step 204.

  Optionally, if it is determined that the remaining storage space is present in the allocated data block, the execution entity further determines whether the remaining storage space is not smaller than the size of the data of the record to be stored. Can be. If it is determined that the remaining storage space is smaller than the size of the data of the storage target record, a part of the data of the storage target record can be stored in the remaining storage space. Then, a new data block can be assigned to the type, and the remaining storage target records can be stored in the new data block.

  As an example, as shown in FIG. 5A, the size of the data is 504 KB of the storage target record Y. The execution subject can determine from the current storage information of type = 126 that the remaining storage space of block3 is 64 KB (4M−504 KB × 8). At this time, the execution subject can allocate one new 2M data block (block 4) under the type. In this way, the 64 KB before the storage target record Y can be stored in block3. Further, 440 KB after the storage target record Y can be stored in block4. That is, the storage target record Y can be stored in two adjacent data blocks.

  In step 204, in response to the determination that there is no remaining storage space, a new data block is assigned to the type and the record to be stored is stored in the new data block.

  In this embodiment, when it is determined that the remaining storage space does not exist in the currently allocated data block, the execution entity can allocate a new data block to the type. Also, the record to be stored can be stored in a new data block. As an example, it is the record X to be stored in FIG. 5A. Since there is no remaining storage space in block 37, the executing entity can allocate one new block 49. At this time, the storage target record X can be stored in the block 49.

  In the method for storing data in the present embodiment, the storage target records can be classified and stored based on the data size of the storage target records. The continuous storage of data can be ensured, and the generation of disk fragments can be reduced or prevented. In this way, it is useful for improving the utilization rate of the storage space and improving the read / write performance of the disk. The distributed object storage method will also be enhanced.

  In one selectable embodiment of this embodiment, after storing the record to be stored, the executing entity may also update the current storage information of the type to which it belongs. Then, the position information after the storage target record is stored can be generated. Further, the position information can be output. Here, the position information may be for explaining the storage position of the storage target record. For example, the location information may include at least one of a record type, a record identifier, and an identifier of a data block where the record is located, and the location information is output. Here, the output may be to store and output. For example, the position information can be stored locally in the execution subject or in another electronic device. For example, a transmission output for transmitting the position information to a terminal or the like can be used.

  As an example, in the case of the record X to be stored in FIG. 5A, the position information can be [record_type = 128, record_id = 5, block_id = 49, next_block_id = 0]. Even when the position information is [record_type = 128, record_id = 5] or [record_id = 5, block_id = 49], it can be understood that the execution subject can search for the record X.

  It should be noted that, in order to ensure that the data size of the records stored in the same type is the same, the execution subject can selectably execute the data of the storage target record before storing the storage target record. It can be determined whether or not the size is the size of the data corresponding to the type. When it is determined that the size of the data of the storage target record is the size of the data corresponding to the type, the execution subject can perform the storage process. When it is determined that the size of the data of the storage target record is not the size of the data corresponding to the type, and is generally smaller than the size of the data corresponding to the type, the execution subject determines the predetermined position of the storage target record (for example, the front or the front). The rear side) can be supplemented with predetermined data (for example, 0 or another character string, etc.). Thereby, the data size of the storage target record after the replenishment becomes the data size corresponding to the type. Thereafter, the execution subject can store the storage target record after the supplement.

  In some application scenarios, before determining the type of the storage target record based on the size of the data of the storage target record, the executing entity can process the data to obtain the storage target record. For example, for a sub-object in at least one sub-object obtained by dividing a storage-waiting object, the execution subject codes the sub-object (for example, EC coding) to obtain a duplicate. After that, the description information and the data of the duplicate are arrayed to generate a storage target record of the subobject.

  The point to be described is that the division process of the object to be stored can be completed by the executing subject or can be completed by another electronic device. Another electronic device can transmit at least one sub-object obtained by division to the execution subject.

  Further, the execution subject analyzes the duplicated data after the arrangement so that the size of the record data stored in the same type is the same, and determines the required data size (that is, the size of the data corresponding to each type). (One in the size) storage target record can be generated.

  Specifically, first, the execution subject can determine whether or not the size of the data of the duplicate after arrangement is the same as one of the sizes of the data corresponding to each type in the storage file. . When the size of the data of the duplicate is the same as one of the sizes of the data corresponding to each type, the execution subject can directly generate the storage target record of the subobject.

  The size of the duplicate data (for example, 5 KB) is different from the size of the data corresponding to each type (for example, 4 KB, 8 KB and 12 KB), and the size of the data corresponding to some types (for example, 8 KB and If it is smaller than 12 KB), the trailing part of the data of the duplicate is padded with zeros. Thereby, the size of the data of the duplicate after the padding with zeros and the size of the data corresponding to the target type can be made the same. Then, one storage target record of the sub-object can be generated. Here, the target type may be a type in which the size of the corresponding data of the partial type is the smallest (for example, 8 KB in 8 KB and 12 KB). It should be noted that the present invention does not limit the replenishment location and replenishment content unless it affects conventional data.

  If the size of the data of the duplicate is larger than the maximum value of the data size corresponding to each type (for example, 512 KB), the duplicate can be divided and filled with zeros. As a result, the size of the data of each duplicate after the division can be made the same as one of the sizes of the data corresponding to each type, and at least two storage target records of the sub-object are generated. For example, the size of the duplicate data is 518 KB, which can be divided into two sub-duplicates of 512 KB and 6 KB, and the 6 KB sub-duplicates are padded with zeros to be 8 KB. Alternatively, it can be divided into three sub-duplicates of 512 KB, 4 KB and 2 KB, and the 2 KB sub-duplicates are padded with 4 KB to zero. Alternatively, based on the size of data corresponding to other types, the duplicate can be divided into a plurality of sub-duplicates of the corresponding data size, and there is no need to perform a zeroing operation.

  By way of example, the record to be stored may include a field as shown in FIG. 5B. Here, part of the meta information (Meta information) of the duplicate can be recorded in the Shared Record (shared recording), and a cyclic redundancy code (crc: Cyclic Redundancy Code), a length (length), and the like can be recorded. Content can be included. The description information designated by the user can be recorded in Key + meta (key meta). Binary data can generally be the actual data of a duplicate. At the back of the data, there may be a field of unfixed length, padded with zeros (ie, filled zero). This makes it possible to ensure that each record (record) is an integral multiple of 4 kb.

  It can be understood that each storage file can also perform self-management of the internal storage space in order to reduce the load on the execution subject. For example, one data block management unit (BlockManager) 501 and a plurality of type management units (BlockLinkList) 502 can be installed for one storage file. Here, the data block management unit can be used to allocate and collect data blocks. Each type management unit can be for managing one type of storage information. For example, the type management unit of type = 128 in FIG. 5A can store a list of data blocks allocated by itself (for example, block 37) and the number of records input by itself (for example, four). In this way, the recorded number can be used to calculate how much remaining storage space is available for the last data block (eg, block 37). If the storage space is insufficient, information for indicating that the data block has been allocated can be transmitted to the data block management unit. At this time, the data block management unit can assign one new data block (eg, block 49) to it. In this way, it is useful to improve the processing efficiency and performance of the entire execution subject.

  At this time, the internal structure of each storage file can be constructed by the position information (Location) already stored and recorded by the user. At startup, each storage file can scan all its records once. As a result, all the data blocks assigned by the user can be displayed on the BlockManager. Further, it is possible to restore the position information of each record by using BlockLinkList. After completing the scan, the BlockManager will know its own data block allocation information. All types of BlockLinkList can also know their own data block information and their current record information.

  In the method for storing data in the present embodiment, the type of the storage target record can be determined based on the data size of the storage target record. Thus, the current storage information of the type can be searched in the storage file. The current storage information may include information on a currently allocated data block and currently stored record information of the data block. Here, the data size of records stored in the same type in the storage file is the same, and the data size of records stored in different types is different. Further, based on the current storage information, it can be determined whether there is any remaining storage space in the currently allocated data block. If it is determined that the remaining storage space does not exist, a new data block can be assigned to the type. Also, the record to be stored can be stored in a new data block. In this way, the generation of space fragments can be reduced, and the space utilization rate of the disk can be improved. This is also advantageous for improving the read / write performance of the disk.

  FIG. 3 shows a flow 300 of another embodiment of the method for storing data according to the present invention. In this embodiment, at least two storage files can be configured on the disk. Also, index information of each storage file on the disk can be stored in a directory of the disk. In other words, each storage file can include index information and data information. Here, in the index information, what kind of record and the state of the record are stored in the storage file. Data information mainly refers to actual data stored inside the storage file. That is, for all records input to the disc, the index information of the record is recorded in the directory. Here, the index information may include record position information.

  By way of example, the structure of the disc is shown in FIG. 5C. Here, the Rocksdb 503 displays a directory of the disc, uses a key-value storage method, and the Vlet 504 (Vlet_110_3 and Vlet...) Can display a storage file. It should be noted that a disc generally has only one directory instance. By storing the index information and the data information separately, it is possible to ensure that a record can be searched for quickly. Optionally, the index information can be stored in a full memory manner, but is limited by the size of the memory. Here, if the data is stored using the directory, another IO can occur although the memory is not limited. At this time, a cache memory can be configured to preferably balance use of the memory with use of the IO.

  In the present embodiment, the method for storing data may further include the following steps.

  In step 301, the directory is searched for the position information of the record to be read.

  In the present embodiment, the entity that performs the method for storing data (for example, the server 105 illustrated in FIG. 1) can search the directory for a record to be read. When the record to be read is searched, the position information can be obtained from the directory.

  In step 302, based on the position information of the record to be read, the type of the record to be read and the position offset in the data block located in the corresponding storage file can be determined.

  In this embodiment, when the type of the record is included in the position information, the execution subject can determine the type of the record to be read based on the position information of the record to be read. Alternatively, the execution subject can determine the storage file and the data block in which the record to be read is located based on the position information, and thereby can determine the type. At the same time, the position offset of the record to be read in the data block is determined based on the storage space of the data block where the data block is located and the information of the record stored in the data block. Here, the position offset can be used to indicate the starting position of the record in the data block.

  As an example, as shown in FIG. 5A, the execution subject is that the position information of the Y of the record from the disk directory is [vlet_id = 110_3, record_type = 126, record_id = 9, block_id = 3, next_block_id = 4]. That can be searched. At this time, based on the storage space of the data block in Vlet_110_3, it can be calculated that the position offset in block 3 of record Y is 2M-64KB (64KB = 4M-8 × 504KB).

  In step 303, the length of the record to be read is determined based on the type of the record to be read, and data having a length corresponding to the length of the record to be read is read starting from the position indicated by the position offset.

  In this embodiment, the execution subject can determine the length of the record to be read based on the type of the record to be read. Further, the execution subject can read data having a length corresponding to the length of the record to be read, starting from the position indicated by the position offset determined in step 302. For example, if the type of the record Y is 126, the length can be 504 KB. At this time, the execution subject can read data having a length of 504 KB into the memory at one time, starting from a position of 2M-64 KB in block 3.

  In step 304, the read data is output as a record to be read.

  In the present embodiment, the execution subject can use the read data as a record to be read, and can perform coding analysis on the record. As a result, the analyzed data can be output and transmitted to, for example, terminals (for example, terminals 101, 102, and 103 shown in FIG. 1).

  In some embodiments, each record can include a field of its stored location information to restore the index information if all index information is lost. As shown in FIG. 5B, the last position of the record (eg, fixed 24 bytes) uses the Record Guard data structure.

  Optionally, the performing entity can verify the Record Guard information before coding analysis. At the same time, it is possible to detect whether or not predetermined data (for example, 0) is present at a predetermined position of the read data. When the predetermined data exists, the predetermined data in the read data can be removed or ignored. Further, the read data after the verification that does not include the predetermined data is analyzed.

  In the method for storing data in the present embodiment, the step of reading data has been increased, and the reading process has been described in detail. The method for storing data in the present invention has been enhanced and completed. In addition, the reading and inputting process is all one IOPS (Input / Output Operations Per Second, the number of times of performing reading and inputting (I / O) operations per second). In this way, it helps to extend the scope of the method.

  FIG. 4 also shows a flow 400 of another embodiment of the method for storing data according to the present invention. The method for storing the data includes the following steps.

  In step 401, the directory is searched for the position information of the record to be deleted.

  In the present embodiment, the entity that executes the method for storing data (for example, the server 105 illustrated in FIG. 1) can search the directory for a record to be deleted. When a record to be deleted is found, its location information can be obtained from the directory.

  In step 402, based on the position information of the record to be deleted, the current storage information of the type to which the record to be deleted belongs in the corresponding storage file is obtained.

  In the present embodiment, the execution subject can determine the storage file where the record is located based on the position information of the record to be deleted. Then, the current storage information of the type to which the file belongs can be obtained from the storage file.

  As an example, as illustrated in FIG. 5A, the execution subject determines that the position information of the record Z is [vlet_id = 110_3, record_type = 128, record_id = 2, block_id = 37, next_block_id = 0] from the directory of the disk. Can be detected. At this time, the execution subject can acquire the current storage information of type = 128 out of Vlet_110_3, that is, block37 (record1-4) and block49 (record5).

  In step 403, based on the obtained current storage information, the last one record can be read from the stored record, and the read record can be transmitted and stored at a position where the record to be deleted is located. it can.

  In this embodiment, based on the obtained current storage information, the execution subject can read the last one record from the currently stored records. Further, the read record can be transmitted to the position where the record to be deleted is located and stored.

  For example, as shown in FIG. 5A, the execution subject can read the record X of record_id = 5. Thereby, the record X can be moved (Move) and input to the position of record_id = 2. Since the data lengths of the two records are the same, overwriting is possible.

  In step 404, the data in the data block before the transfer of the read record is cleared, and the position information of the read record is corrected.

  In the present embodiment, the execution subject can clear the data (for example, record5) in the data block (for example, block_id = 49) before the transfer of the read record (for example, record X). Further, it is necessary to correct the position information of the read record. For example, the corrected position information of the record X is [record_type = 128, record_id = 2, block_id = 37, next_block_id = 0]. It should be pointed out that after deleting the record to be deleted, the related information (for example, the directory of the disk, the current storage information under the corresponding type, the Record Guard of record X, etc.) needs to be further updated.

  In step 405, it is determined whether or not data exists in the data block located before the transfer of the read record.

  In this embodiment, the execution subject can further determine whether or not data exists in the data block located before the read record is transferred. For example, as shown in FIG. 5A, after transferring and storing the record X, the execution subject determines whether or not other data is stored in the block 49.

  If it is determined in step 406 that no data is stored in the data block, the data block is collected for reassignment.

  In this embodiment, when it is determined that no data is stored in the data block, the execution subject can collect the data block for re-assignment. For example, an already assigned state of a data block can be modified to an unassigned state. For example, as can be seen from FIG. 5A, after transferring and storing record X, the storage space of block 49 is completely empty. At this time, the BlockManager can collect the data block (FreeBlock 49).

  In the method for storing data in the present embodiment, the step of deleting data has been increased and the deletion process has been described in detail. The method for storing data in the present invention has been further enhanced and completed. In addition, the deletion process is equivalent to adding one writing process to one reading process, and is useful for improving the overall performance of the disc. Also, the continuity of the remaining data storage is ensured so that no fragments of space are generated.

  It can be understood that the size of each storage file constructed in the disk in each of the above embodiments can be the same, which is advantageous in that it is easy to manage and improves the efficiency of data processing. The size of each storage file can be different, which helps to meet the needs of different users. Note that the sizes of the data blocks in different storage files can be the same or different. For convenience of management, a configuration file can be installed in each storage file. Here, the configuration file can be used to describe configuration parameter information of the storage file.

  As shown in FIG. 5A, one 512 KB file (ie, VletInfo) may exist at the end of the storage file. The file can store its own configuration parameter information, can use a Protocol (Google Protocol Buffer), and is a simple and efficient storage format of structured data. It is scalable, easy to use, and fast to analyze, regardless of platform and language. It should be pointed out that the content in this file is generated only during the construction of the storage file and cannot be modified later.

  Here, the configuration parameter information includes a storage file size (for example, 32 GB), a data block size (for example, 2M), a minimum type in a record (for example, 4 kB), a maximum type in a record (for example, 512 KB), It may include at least one of intervals between record types (for example, 4 kB), but is not limited thereto. In other words, in addition to storing vletinfo in the last one page of the storage file, all other parts can be data blocks of the same size.

  It should be noted that storing a large number of small files is a problem that is difficult to solve with a distributed storage system. Although the user generally needs to compromise on read performance, space utilization, and deletion efficiency, in the data storage method according to each embodiment of the present invention, data writing is performed on one disk. It can correspond to random reading and writing times. Deletion of data can be equivalent to adding writing to one reading. In this way, it is useful to secure the read / write performance and the deletion efficiency of the disk. In addition, when writing or deleting data, no fragments of space are generated. In addition, an empty storage space can be recovered immediately. In this way, it is useful to improve the space utilization rate of the disk. In other words, the method according to the present invention can balance the above demands of the user, can reach good processing effect during actual use, and can be advantageous for improving the user experience. .

  In the method for storing data in an embodiment of the present invention, all data reading and writing is performed once by IOPS, unless the data spans two data blocks. Here, by taking measures such as adjusting the size of the data block or dividing the data input, it is possible to reduce or prevent a situation where two data blocks are straddled. The read / write performance is basically the same as the time required for one random read / write of each disk. Further, no fragments are generated in the process of writing and deleting, and the space can be recovered immediately. One additional IO write may occur at the time of deletion, but IO consumption is still relatively low compared to the prior art solutions.

  In addition, there are mainly two parts in the aspect of space waste. One of them is to make the sizes of the same type of data uniform. The waste of this part depends on the average size of the input data. As can be seen by statistics, the average input data of a user is typically 256 KB. The resulting waste of space is about 0.7%. Another is that the last one data block allocated in various types is not filled. In the worst case, there are only about 127 data blocks in one record. In this case, the waste of space is about (2M × 127) ÷ 32G = 0.7%. Through analysis, it can be seen that the waste of these two subspaces is all acceptable. Therefore, from a comprehensive comparison of aspects such as throughput, delay, and space utilization, the above method is superior to existing techniques.

  It should be noted that the method of the embodiment of the present invention can be used mainly when the data size is EB class, when the read is more than the write, and when the write is more than the delete. This places very stringent requirements on space utilization (ie, cost savings). In addition, relatively high demands are placed on read / write delay and throughput.

  As shown in FIG. 6, as an implementation of the method shown in the above figures, the present invention provides one embodiment of an apparatus for storing data. The embodiment of the device corresponds to the embodiment of the method shown in each of the above embodiments, and the device can be specifically applied to various electronic devices.

  As shown in FIG. 6, an apparatus 600 for storing data according to the present embodiment can be configured as a standalone storage engine of a distributed object storage. Storage files can be configured on the disk of the standalone storage engine. Here, the storage space of the storage file is divided into at least two data blocks, and a link list structure can be used between the at least two data blocks. The apparatus 600 is configured to determine a type of a storage target record based on a data size of a storage target record, and a type determination unit 601 configured to search a storage file for current storage information of the type. The current storage information includes information on a currently allocated data block and record information currently stored on the data block, and records stored under the same type on the storage file. The size of the data in the records stored under the different types is the same, and the size of the data in the records stored under different types is different. Based on the current storage information, the currently allocated data block has the remaining storage space. A scan configured to determine if it exists Source determination unit 603 and a first allocation configured to allocate a new data block to the type in response to the determination that there is no remaining storage space, and store the record to be stored in the new data block. And a unit 604.

  In one selectable embodiment of the present embodiment, the device 600 responds to the determination that the remaining storage space is present and that the remaining storage space is not smaller than the size of the data of the record to be stored. A size determination unit (not shown in FIG. 6) configured to further determine whether the storage space is smaller than the size of the data of the record to be stored. A second allocation unit (shown in FIG. 6) configured to store some data of the target record in the remaining storage space, allocate a new data block to the type, and store the remaining data of the storage target record. ) Can be further provided.

  Optionally, the storage space of at least two data blocks can be the same, and the size of the data of records stored in different types can all be an integer multiple of a predetermined number.

  Further, the apparatus 600 may update the current storage information of the type, generate the post-storage record storage location information, and output the location information (shown in FIG. 6). ) Can be further provided. Here, the position information includes at least one of a record type, a record identifier, and an identifier of a data block where the record is located.

  In some embodiments, at least two storage files can be configured on the disk, and index information of each storage file can be stored in a directory on the disk.

  Optionally, the device 600 may include a first location search unit (not shown in FIG. 6) configured to search the directory for location information of the record to be read, and based on the location information of the record to be read. A determination unit (not shown in FIG. 6) configured to determine a position offset in a data block located among a type of a record to be read and a corresponding storage file; Determining the length of the target record, starting from the position indicated by the position offset, reading data whose length is the length of the read target record, and outputting the read data as the read target record. And a reading unit (not shown in FIG. 6). Rukoto can.

  Further, the device 600 includes a second position search unit (not shown in FIG. 6) configured to search the directory for the position information of the record to be deleted, and a deletion target based on the position information of the record to be deleted. An acquisition unit (not shown in FIG. 6) configured to acquire the current storage information of the type to which the record belongs in the corresponding storage file, and the currently stored information based on the obtained current storage information. A transfer unit (not shown in FIG. 6) configured to read the last record from the record, and transfer and store the read record to the position where the record to be deleted is located; Correction unit configured to clear the data in the data block of the row and to correct the positional information of the read record And not shown) in FIG. 6 may further include a.

  In some embodiments, the apparatus 600 determines whether data is still present in the data block located before the read record is transferred, and the data is not stored in the data block. , May further comprise a recovery unit (not shown in FIG. 6) configured to recover the data block for reassignment.

  In some embodiments, the apparatus 600 includes, for a sub-object in at least one sub-object obtained by dividing the object to be stored, coding the sub-object to obtain a duplicate. A unit (not shown in FIG. 6) and a record generating unit (not shown in FIG. 6) configured to perform an arraying process on the description information and data of the duplicate and generate a storage target record of the sub-object. ) Can be further provided.

  Optionally, the record generation unit is configured to determine whether the size of the data of the duplicate after arrangement is the same as one of the sizes of the data corresponding to each type in the storage file. If the size of the determined subunit and the data of the duplicate are all different from the size of the data corresponding to each type and smaller than the size of the data corresponding to some types, the trailing part of the data of the duplicate is zero. And the size of the data of the duplicate after padding with zeros is the same as the size of the data corresponding to the target type, which is the smallest type of the corresponding data among the partial types, and The first generation sub-unit configured to generate one storage target record and the size of the data of the duplicate correspond to each type. If the data size is larger than the maximum value, the duplicate is divided and padded with zeros so that the data size of each duplicate after the division is the same as one of the data sizes corresponding to each type. And a second generation sub-unit configured to generate at least two storage target records of the sub-object.

  It can be appreciated that some units described in the apparatus 600 correspond to steps in the method described in FIGS. Accordingly, the operations, functions, and generated beneficial effects described for the above methods also apply to the apparatus 600 and the units contained therein, and will not be described again here.

  Referring to FIG. 7, there is shown a timing diagram of one embodiment of a system for storing data according to the present invention.

  The system for storing data in the present embodiment may include a first subsystem, a second subsystem, and a third subsystem in which the standalone storage engine described in each of the above embodiments is installed. The first subsystem receives a storage request including a storage object sent by a user, divides the storage object into at least one sub-object, and associates the storage object with the at least one sub-object. To the second subsystem and at least one sub-object to the third subsystem. The second subsystem is configured to store a correspondence between the object to be stored and the at least one sub-object in a list. The third subsystem is configured to perform coding and arraying processing on sub-objects in at least one sub-object to generate a storage target record of the sub-object, and store the generated storage target record. .

  As shown in FIG. 7, in step 701, the first subsystem can receive a storage request including a storage-waiting object transmitted by a user.

  In the present embodiment, the first subsystem can receive a storage request including a storage-waiting object transmitted by a user through a wired connection method or a wireless connection method. Here, the object waiting to be stored can be data in the distributed object storage system.

  In step 702, the first subsystem can divide the object to be stored into at least one sub-object. Here, the division method can be set according to actual demand.

  In step 703, the first subsystem can send a correspondence between the object to be stored and the at least one sub-object to the second subsystem, and sends the at least one sub-object to the third subsystem. be able to.

  In step 704, the second subsystem can store the correspondence between the object to be stored and the at least one sub-object in a list.

  In step 705, the third subsystem performs coding and arrangement processing on the sub-objects in the at least one sub-object, generates a storage target record of the sub-object, and stores the generated storage target record. can do. This can be referred to the relevant description of the embodiment of FIG. 2 and will not be repeated here.

  In some selectable embodiments of this example, for example, at step 706, the third subsystem may send response information to the first subsystem to indicate completion of data storage.

  Thereafter, in step 707, when receiving the response information, the first subsystem generates a query identifier of the object to be stored and feeds back the query identifier to the user. In this way, the user can access the object data indicated by the query identifier. Here, the query identifier includes at least one character code such as a number, an alphabet, and a character, but is not limited thereto.

  Optionally, for example, at step 708, the first subsystem can receive a read request including the query identifier sent by the user.

  Subsequently, in step 709, the first subsystem can send the query identifier in the read request to the second subsystem.

  Thereafter, in step 710, the second subsystem can obtain a sub-object list corresponding to the object indicated by the query identifier in the read request, and can transmit the sub-object list to the third subsystem. .

  Thereafter, in step 711, the third subsystem reads a corresponding record according to the sub-object list, analyzes the read record to obtain object data, and transmits the object data to the first subsystem. it can. This can be referred to the relevant description of the embodiment of FIG. 3 and will not be repeated here.

  Finally, at step 712, the first subsystem can feed back the object data to the user.

  In some application scenarios, the first subsystem may further receive a delete request including the query identifier sent by the user. In this way, the third subsystem can delete the object data indicated by the deletion request according to the description related to the embodiment of FIG. It will not be described again here.

  It can be understood that the first subsystem, the second subsystem and the third subsystem can be located on different electronic devices (for example, three servers) and the same electronic device (shown in FIG. 1). Server 105). Further, in some embodiments, when the first subsystem has the function of the second subsystem, the system in the present embodiment does not need to install the second subsystem.

  In the data storage system of the present embodiment, a new data storage method is applied, that is, the data is classified and stored according to the data size. In this way, not only can distributed storage of data be realized, but also data security can be improved. At the same time, the overall data processing performance can be improved and the operating cost can be reduced.

  Referring now to FIG. 8, a schematic structural diagram of a computer system 800 applied to implement an electronic device (eg, the server 105 shown in FIG. 1) according to an embodiment of the present application is shown. The electronic device shown in FIG. 8 shows only one embodiment, and does not limit the function and use range of the embodiment of the present invention.

  As shown in FIG. 8, computer system 800 performs various suitable operations and processes according to programs loaded in read-only memory (ROM) 802 or programs loaded from storage portion 808 into random access memory (RAM) 803. It comprises a central processing unit (CPU) 801 that can execute. The RAM 803 further stores various programs and data necessary for operating the system 800. The CPU 801, the ROM 802, and the RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

  An input unit 806 including a keyboard, a mouse, and the like, an output unit 807 including a cathode-ray tube (CRT), a liquid crystal display (LCD), and a speaker, a storage unit 808 including a hard disk, a LAN card, a modem, and the like Is connected to the I / O interface 805. The communication unit 809 executes a communication process via a network such as the Internet. The driver 810 is also connected to the I / O interface 805 as needed. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is mounted on a driver 810 as needed, and a computer program read therefrom is installed in the storage unit 808 as needed.

  In particular, in the embodiment according to the present invention, the processes described above with reference to the flowcharts can be realized as a computer software program. For example, an embodiment of the present disclosure comprises a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising a program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication portion 809 and / or can be installed from removable media 811. When the computer program is executed by the central processing unit (CPU) 801, it performs the functions limited to the method of the present invention. It should be noted that the computer-readable medium according to the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of both. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer readable storage media are electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only It includes, but is not limited to, a memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage element, a magnetic storage element, or any combination thereof. In the present invention, a computer-readable medium can be any tangible medium that includes or stores a program, which program is used by, or in conjunction with, the command execution system, apparatus, or element. Is done. In the present invention, a computer readable signal medium carries computer readable program code and may be provided in baseband or in a data signal propagated as part of a carrier wave. Data signals propagated in this manner can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, wherein the computer readable medium is used by, or in conjunction with, a command execution system, apparatus, or element. Programs can be sent, propagated, or transported. The program code contained on the computer readable medium may be transmitted by any suitable medium including, but not limited to, wireless, wire, fiber optic cable, RF, etc., or any suitable combination thereof.

  The flowcharts and block diagrams in the drawings illustrate the structures, functions, and operations of possible embodiments of the systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, a program segment, or a portion of code, where the module, program segment, or portion of the code implements a defined logical function. One or more executable instructions. Also note that in alternate embodiments, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks displayed in succession may, in fact, be performed substantially in parallel, or they may be performed in the reverse order, depending on the functionality involved. Also, each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented in dedicated hardware-based systems that perform specified functions or operations, or may be implemented using dedicated hardware. It can also be realized by a combination of and computer instructions.

  The units mentioned in the embodiments of the present invention may be implemented by software or may be implemented by hardware. The described unit can also be installed on a processor described as a processor comprising, for example, a type determination unit, a search unit, a space determination unit and a first allocation unit. Here, the names of these units may not limit the units themselves.For example, the type determination unit may also specify “the type of the storage target record according to the data size of the storage target record. Unit to determine ".

  In another aspect, the present invention further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment, or may be separately provided and installed on the electronic device. Can not. The computer-readable medium can carry one or more programs, and when the one or more programs are executed by the server, the electronic device is configured to store data according to the size of the data of the record to be stored. Determine the type of record to be stored, and search the storage file for current storage information under the type, comprising information of the currently allocated data block and information of the currently stored record of the data blocks, In a storage file, the data size of records stored under the same type is the same, the data size of records stored under different types is different, and according to the current storage information, Determine if there is any remaining storage space in the allocated data block In response to the remaining storage space is determined not by allocating a new data block under type, storing the storage object records in the new data block.

The above description is only the description of the preferred embodiment of the present invention and the principle of the applied technology. The scope of the present invention as referred to in the present invention is not limited to the specific combination of the technical features described above, but may be combined with the technical features or equivalent features without departing from the inventive concept. It was formed in any combination. For example, the technical features having functions similar to the technical features disclosed in the present invention and the technical features disclosed in, but not limited to, the present invention should be exchanged with each other to cover other technical solutions. Should be understood by those skilled in the art.

Claims (16)

A method for storing data, wherein the method is used in a standalone storage engine of a distributed object storage, wherein a storage file is configured on a disk of the standalone storage engine, and the storage space of the storage file is at least two data spaces. Divided into blocks and using a linked list structure between said at least two data blocks,
The method comprises:
Determining the type of the storage target record based on the size of the data of the storage target record;
Retrieving current storage information of the type in the storage file, wherein the current storage information includes information of a currently allocated data block and information of a currently stored record of the data block. The size of data of records stored in the same type of the storage file is the same, the size of data of records stored in the different type is different,
Determining whether there is any remaining storage space in the currently allocated data block based on the current storage information;
Allocating a new data block to the type in response to the determination that the remaining storage space is absent, and storing the storage target record in the new data block. Method. Responsive to the determination that the remaining storage space is present, further determining whether the remaining storage space is not smaller than the size of the data of the storage target record;
Storing, in response to the determination that the remaining storage space is smaller than the size of the data of the storage target record, some data of the storage target record in the remaining storage space;
2. A method for storing data according to claim 1, further comprising: allocating a new data block to the type and storing remaining data of the record to be stored in the new data block. . The storage space of the at least two data blocks is the same, and the sizes of data of records stored in the different types are all integer multiples of a predetermined numerical value. A way to remember. Updating the current storage information of the type, generating stored location information of the storage target record, and outputting the location information,
The data according to any one of claims 1 to 3, wherein the position information includes at least one of the type of a record, an identifier of a record, and an identifier of a data block where the record is located. Way to do. The method for storing data according to claim 4, wherein at least two storage files are configured on the disk, and index information of each of the storage files is stored in a directory of the disk. Searching for the position information of the record to be read from the directory;
Based on the position information of the record to be read, determining the type of the record to be read and a position offset in a data block located in the corresponding storage file;
Based on the type of the record to be read, determine the length of the record to be read, and read data having a length corresponding to the length of the record to be read, starting from the position indicated by the position offset. The method for storing data according to claim 5, further comprising: outputting the read data as the record to be read. Searching for the position information of the record to be deleted from the directory;
Acquiring the current storage information of the type to which the record to be deleted belongs in the corresponding storage file based on the position information of the record to be deleted;
Reading the last record from the currently stored record based on the obtained current storage information, transferring the read record to a position where the record to be deleted is stored, and
6. The method according to claim 5, further comprising: clearing data in a data block before transferring the read record, and correcting position information of the read record. Way for. After clearing the data in the data block before the transfer of the read record, the method for storing the data comprises:
Determining whether the read record has data in the data block located before the transfer,
The method of claim 7, further comprising: retrieving the data block for reassignment if the data block does not have data. Before determining the type of the storage target record based on the size of the data of the storage target record, a method for storing the data,
Coding a sub-object for a sub-object in at least one sub-object obtained by dividing the object to be stored, to obtain a duplicate;
The method according to any one of claims 1 to 8, further comprising: performing an arraying process on the description information and the data of the duplicate and generating a storage target record of the sub-object. Method for storing data. Generating the storage target record of the sub-object includes:
Determining whether or not the size of the duplicate data after arraying is the same as one of the sizes of the data corresponding to each of the types in the storage file;
If the size of the duplicate data is different from the size of the data corresponding to each of the types, and is smaller than the size of the data corresponding to some of the types, the trailing portion of the duplicate data is padded with zeros, and The size of the data of the duplicate after filling is the same as the size of the data corresponding to the target type, and one storage target record of the sub-object is generated.
When the size of the duplicate data is larger than the maximum value of the data size corresponding to each type, the duplicate is divided and filled with zeros, so that the size of each duplicate data after division is set to each of the types. Generating at least two storage target records of said sub-object, each being equal to one of the corresponding data sizes;
The method for storing data according to claim 9, wherein the target type is the type having the smallest size of corresponding data of the some of the types. An apparatus for storing data, wherein the apparatus is configured as a standalone storage engine of a distributed object storage, a storage file is configured on a disk of the standalone storage engine, and a storage space of the storage file has at least two data spaces. Divided into blocks and using a linked list structure between said at least two data blocks,
The device comprises:
A type determination unit configured to determine the type of the storage target record based on the size of the data of the storage target record;
A search unit configured to search the storage file for current storage information of the type, wherein the current storage information is information of a currently allocated data block and a currently stored data block of the data block. A search unit, comprising information of records, wherein the size of data of records stored in the storage file of the same type is the same, and the size of data of records stored in different types of the storage file is different,
A space determination unit configured to determine whether there is any remaining storage space in the currently allocated data block based on the current storage information;
A first allocation unit configured to allocate a new data block to the type in response to the determination that the remaining storage space is absent, and to store the record to be stored in the new data block; An apparatus for storing data comprising: A system for storing data,
A first subsystem, a second subsystem, and a third subsystem in which the standalone storage engine according to any one of claims 1 to 10 is installed,
The first subsystem receives a storage request including a storage-waiting object transmitted by a user, divides the storage-waiting object into at least one sub-object, and performs a communication between the storage-waiting object and the at least one sub-object. Transmitting the correspondence between the second subsystem and the at least one sub-object to the third subsystem;
The second subsystem is configured to store a correspondence between the waiting-for-storage object and the at least one sub-object in a list,
The third subsystem performs coding and arrangement processing on sub-objects in the at least one sub-object, generates a storage target record of the sub-object, and stores the generated storage target record. For storing data configured. The third subsystem is further configured to send response information for indicating data storage completion to the first subsystem,
The method according to claim 12, wherein the first subsystem is further configured to, when receiving the response information, generate a query identifier of the object to be stored and feed back the query identifier to the user. A system for storing the described data. The first subsystem is further configured to receive a read request including a query identifier transmitted by the user, and to transmit the query identifier in the read request to the second subsystem,
The second subsystem is further configured to obtain a sub-object list corresponding to an object indicated by the query identifier in the read request, and to transmit the sub-object list to the third subsystem;
The third subsystem reads a corresponding record based on the sub-object list, analyzes the read record to obtain object data, and the first subsystem feeds back the object data to the user. The system for storing data according to claim 13, further configured to transmit the object data to the first subsystem. An electronic device,
One or more processors,
A storage device for storing one or more programs,
An electronic device that, when the one or more programs are executed by the one or more processors, causes the one or more processors to implement the method according to any one of claims 1 to 10. A computer readable medium storing a computer program,
A computer readable medium on which the method according to any one of claims 1 to 10 is implemented when the computer program is executed by a processor.