WO2020024933A1 - Data writing method and server - Google Patents
Data writing method and server Download PDFInfo
- Publication number
- WO2020024933A1 WO2020024933A1 PCT/CN2019/098338 CN2019098338W WO2020024933A1 WO 2020024933 A1 WO2020024933 A1 WO 2020024933A1 CN 2019098338 W CN2019098338 W CN 2019098338W WO 2020024933 A1 WO2020024933 A1 WO 2020024933A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- file system
- area
- stored
- writing position
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0662—Virtualisation aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Definitions
- the present application relates to the field of computer technology, and in particular, to a data writing method and a server.
- a block device is a device that can read a certain length of data from any location on the device, including hard disks, disks, USB flash drives, and flash memory.
- the operating system uses a part of the system memory as a cache area. When receiving the data to be stored, the operating system can first save the data to be stored in the cache area, and then write some data in the cache area according to the existing elimination algorithm. Block device saved.
- the software organization responsible for managing and storing file information in the operating system is called a file system, and the data in the file system includes general data and metadata.
- general data refers to the actual data in ordinary files
- metadata is system data used to describe the characteristics of a file, such as access rights, file owner, and distribution information of file data blocks (such as Inode).
- storing metadata and some popular general data in the cache area can speed up the read and write operations of block devices.
- the block device is generally used to mount the file system, when the block device has been formatted as a file system, there is no file attribute information after the data reaches the block device, and the operating system cannot accurately distinguish the received data.
- Data is also metadata, which may cause important metadata to be eliminated from the cache.
- the present application provides a data writing method and a server, which are used to distinguish metadata from general data, and write the distinguished metadata into a cache area to optimize the read and write performance of a block device.
- a data writing method is provided.
- the method is applied to a server and includes: receiving first to-be-stored data sent by a client and the first to-be-stored data in a block device.
- each of the file system's area location information is generated according to the storage space size of the block device, and each of the file system's area location information includes the file system's metadata area and general data Location information of the area; when it is determined that the writing position of the first to-be-stored data is located in the metadata area of the first file system, writing the first to-be-stored data into a cache area of the server.
- the second aspect of the embodiments of the present application provides a server, which has a function of implementing the method provided in the first aspect.
- the functions may be implemented by hardware, and may also be implemented by hardware executing corresponding software.
- the hardware or software includes one or more modules or units corresponding to the functions described above.
- the server may include: a receiving unit, configured to receive the first to-be-stored data and the writing position of the first to-be-stored data sent by the client in a block device; a comparison unit, Comparing the writing position of the first to-be-stored data with the area location information of the first file system that has been generated in the server; the first file system is a plurality of files that have been generated by the server One of the file systems, the area location information of each of the file systems is generated according to the storage space size of the block device, and the area location information of each of the file systems includes the location of a metadata area and a general data area of the file system Information; a write data unit, configured to write the first data to be stored when the comparison unit determines that a write position of the first data to be stored is located in a metadata area of the first file system A cache area of the server.
- a receiving unit configured to receive the first to-be-stored data and the writing position of the first to-be-stored data sent by the client in a block
- the server includes a processor, a communication interface, a memory, and a bus, where the processor, the communication interface, and the memory are connected to each other through a bus; the memory stores machine-readable instructions, and the processor The method provided by the first aspect of the embodiments of the present application is executed by calling the machine-readable instructions.
- metadata is distinguished from general data, and the distinguished metadata is stored in the buffer area for a long time.
- the distinguished general data is written into the block device according to the method, which can avoid the mixed writing of metadata and general data, which leads to the magnetic head. Frequent movement between general data storage locations and metadata storage locations of block devices improves data read and write performance.
- FIG. 1 is a schematic diagram of an architecture provided by an embodiment of the present application.
- FIG. 3 is a schematic diagram of partitioning a block device in an EXT2 system according to an embodiment of the present application
- FIG. 4 is a schematic diagram of a logical structure of a block group in an EXT2 system according to an embodiment of the present application
- FIG. 5a, FIG. 5b, FIG. 5c, and FIG. 5d are schematic diagrams of an anti-check operation provided by an embodiment of the present application.
- FIG. 6 is a module block diagram of a server provided in an embodiment of the present application.
- FIG. 7 is a hardware structural diagram of a server provided in an embodiment of the present application.
- Metadata includes the data structure that manages the entire file system, the data structure that manages the allocation of space, the attributes and pointers of files, the attributes and contents of directories.
- EXT2 second extended filesystem
- superblocks, block group descriptors, data block bitmaps, inode bitmaps, inode tables, and directory entry pages all belong to the metadata of the file system.
- General data refers to the content of the file. Taking the EXT2 file system as an example, the general data of the file system is the file content stored in the data area.
- Metadata area refers to a contiguous space of cache addresses allocated in the cache. It is used to cache superblocks, block group descriptors, inode bitmaps, data block bitmaps, and inode information created during copy directories and compressed file decompression operations. Cache refers to a part of space divided from system memory.
- General data area refers to a contiguous space of cache addresses allocated in the cache. It is used to cache data information created during the copy directory and compressed file decompression operation.
- FIG. 1 is an overall architecture diagram of an embodiment of the present application.
- the embodiment of the present application includes a client and a server.
- the server is connected to multiple block devices, and the server is connected between the client and the block device.
- the connection between the server and the client and between the server and the block device can be It is a network connection or an electrical connection.
- the server can set up a cache area for each of these block devices, and create a virtual device corresponding to the block device on the client.
- the process of the client saving data to the virtual device is equivalent to sending a write instruction to the server, and the server writes the data carried by the write instruction to the buffer area or block device.
- the process of the client extracting data from the virtual device is equivalent to Send a read instruction to the server, and the server extracts the corresponding data from the block device or the buffer area according to the access address of the read instruction and returns it to the client.
- this application mainly improves the process of writing data, which is manifested in the server's separation and identification of the data to be stored, distinguishing between metadata and general data, and making the distinguished metadata permanent. Save to buffer area.
- FIG. 2 it is a schematic flow chart of the operation of the data writing method provided by the present application.
- the server receives the data to be stored from the client for the first time, perform the following steps:
- Step 201 The server receives the first to-be-stored data and the writing position of the first to-be-stored data sent by the client in the block device.
- the writing position of the first to-be-stored data in the block device is indicated by the following information: the starting writing position of the first to-be-stored data in the block device and the data size of the first to-be-stored data.
- the start writing position and the end writing position of the first data to be stored in the block device may be determined.
- Step 202 The server compares the writing position of the first to-be-stored data with the regional location information of the first file system generated in the server.
- the first file system is a plurality of file systems generated in the server.
- the area location information of each file system is generated according to the storage space size of the block device.
- the area location information of each file system includes the location information of the metadata area and the general data area of the file system.
- the client Before step 201, the client applies for a block device from the server in advance.
- the size of the storage space of the block device that can be used by the client can be requested by the client or allocated uniformly by the server.
- the server accepts the client's application for the use of the block device, it determines the block device for use by the client among the multiple block devices connected to the server, and creates a virtual device on the client. Creating a virtual device on the client is equivalent to mounting a block device on the client. This virtual device can be regarded as a mapping of the block device.
- the client formatting the virtual device is equivalent to sending formatting instructions to the server.
- the server formats the block device allocated to the client; the client saves the data to the virtual device, which is equivalent to sending a write instruction to the server, and the server writes the data carried by the write instruction to the cache area or Block device.
- the client wants to use the block device as a file, it needs to format the block device into a file system.
- the design idea of this application is to judge the type of data by comparing the location information of the block device formatted with the write position of the data. If the data write location is in the metadata area of the formatted block device, the data is considered as metadata; if the data write location is in the general data area of the formatted block device, the data to be considered is General data.
- the server is not sure which file system the client uses to format the block device it uses. Therefore, in the embodiment of the present application, the server can set up a file system template library.
- File system structure such as EXT2 (second extended filesystem), EXT3 (third extended filesystem), XFS, FAT (File Allocation Table), NTFS (New Technology File System) and so on.
- the server can use the storage space size as an input to the file system template library, and use the file system template library to establish a set that matches the size of the storage space.
- the regional location information of each file system the size of the input storage space is different, and the regional location information of each file system is also different.
- the server can establish a set of area location information of the file system corresponding to the storage space size of the block device used by the client.
- the block device when a block device is formatted as an EXT2 file system, the block device is divided into a boot block and multiple block groups; the boot block refers to the boot sector of the file system. Can be used to install the boot program; each block group has its own metadata area and general data area, and the internal logical results of each block group are shown in Figure 4, including the general data area and the metadata area.
- the logical structure of the metadata area and the general data area in the block group can be seen from FIG. 4.
- the structure of the data area is relatively single and consists of many blocks of the same size.
- the metadata area includes multiple sub-metadata areas.
- the metadata area of the EXT2 file system consists of a Super Block, a GDT (group descriptor), an inode table, an inode bitmap, and data.
- a block bitmap is composed of five sub-metadata areas.
- Step 203 When the server determines that the write position of the first data to be stored is located in the metadata area of the first file system, the server writes the first data to be stored into the cache area of the server.
- the server compares the writing position of the first to-be-stored data with the area position information of multiple file systems that have been generated in the server in sequence.
- the first to-be-stored data is found.
- the start writing position and the ending writing position of the data are located in the same sub-metadata area of the first file system, and it is determined that the writing position of the first to-be-stored data is located in the metadata area of the first file system.
- the server determines that the first file system is a file system type that the block device used by the client may adopt, and determines that the first data to be stored may be metadata. After the first file system is found, the server can stop the comparison and directly write the first data to be stored into the cache area.
- the server can determine the first The stored data is general data.
- the server can process the first to-be-stored data according to a set general data processing strategy. For example, the first to-be-stored data can be written directly to the block device, or the first to-be-stored data can be written to the cache area first. When the available space in the cache area is less than a certain value, the general data in the cache area is eliminated.
- a specific implementation method is as follows: When it is determined that the write position of the first data to be stored is located in multiple generated file systems In the general data area of each file system, determine the first to-be-stored data as general data, and determine whether the available space in the cache area of this server is larger than the occupied space of the first to-be-stored data; if so, The stored data is written into the cache area of this server; if not, according to the preset elimination algorithm, such as FIFO (First In, First Out) algorithm, LRU (Least, Recently Used) algorithm, LFU (Least Frequently Used, least frequently used) algorithm, etc., determine the obsolete data in the general data stored in the cache area of this server; according to the obsolete data, In position, selecting at least one data having a writing position adjacent from the buffer; writes a block of the present server apparatus together with at least one data out of the data having adjacent writing position.
- FIFO First In, First Out
- LRU Least, Recently Used
- LFU Least Frequently Used, least frequently used
- the aggregation method can also be used to write the metadata with adjacent write positions to the block device together.
- the written data after the data in the cache area is written to the block device, the written data also needs to be deleted from the cache area; when the data in the block device is written to the cache area, there is no need to When the delete operation is performed, the corresponding data remains in the block device.
- the server receives the data to be stored from the client for the first time. After that, when the server receives the data to be stored from the client again, the server can perform steps 201-203 again. Or, in another embodiment, in order to further improve the read and write performance of the block device, when the server performs steps 201-203 once and has determined the type of file system that the block device used by the client may adopt, the server When receiving the data to be stored subsequently sent by the same client, such as the second data to be stored, the following process may be performed:
- the server receives the second to-be-stored data and the writing position of the second to-be-stored data sent by the client in the block device; the writing position of the second to-be-stored data in the block device includes the second to-be-stored data in the block device Start writing position and ending writing position.
- the server compares the writing position of the second data to be stored with the area position information of the first file system.
- the first file system is the file system type that the block device used by the client determined in step 203 may adopt.
- the write position of the second to-be-stored data is directly compared with the area position information of the first file system. Compare the write position of the second to-be-stored data with the regional position information of the multiple file systems corresponding to the client that has been generated.
- the server still judges the first file system as the type of file system that the block device used by the client may use, and determines that the second data to be stored may be metadata, and directly writes the second data to be stored in the cache. Area.
- the server still determines that the first file system is a block used by the client.
- the file system type that the device may use, and determines that the second data to be stored may be general data.
- the server can process the second data to be stored according to a set general data processing policy, and the specific processing process is not described in detail.
- the start writing position and the end writing position of the second data to be stored are in different sub-metadata areas of the first file system, or the start writing position of the second data to be stored is located in the first file
- the metadata area of the system and the end write location of the second to-be-stored data are outside the metadata area of the first file system, indicating that the first file system is not the file system type used by the block device used by the client.
- the server needs to re-compare the writing position of the second to-be-stored data with the regional position information of other file systems (except the first file system) in the multiple file systems that have been generated.
- the start writing position and the end writing position of the second data to be stored are located in a metadata area of the file system, it is determined that the second data to be stored is normal data.
- the second file system may be determined to be a block used by the client.
- the file system type that the device may adopt, and determines that the second data to be stored may be metadata.
- the server can stop the comparison and directly write the second data to be stored into the cache area of the server.
- the server may also perform a back-check operation to transfer the data previously misjudged as metadata to the block device, and Data that was previously misjudged as general data is rewritten into the cache area, as follows:
- the server can save the start in the cache area of the server
- the data at the write position and / or the end write position are in [x1, x2]
- the data at the start write position and / or the end write position are in [y2, y1] are written to the block device of this server.
- the server can set Data stored in the start and / or end write locations of the server's buffer area in [y2, y1] is written to the block device of the server, and the start of write is stored in the block device of the server.
- the data at the input position and the end write position are both in [x2, x1] and written into the cache area of this server.
- the server can set Data stored in the start and / or end write locations of the server's cache area in [x1, x2] is written to the block device of the server, and the start of write is stored in the block device of the server The data at the entry position and the end writing position are located in [y1, y2] and written into the cache area of this server.
- the server can save the start of the block device in the server
- the data where the write position and the end write position are both in [x2, x1], and the data where the start write position and the end write position are both in [y1, y2] are written into the cache area of this server.
- the block device After the block device is formatted as a file system, general data and metadata are usually stored separately in the block device.
- the metadata is distinguished from the general data, and the distinguished metadata is stored in the cache area for a long time.
- the magnetic head by aggregating a plurality of data having adjacent write positions into a large data and writing it to the block device at one time, compared with writing a plurality of data into the block device in batches, the magnetic head is more reduced.
- FIG. 6 is a functional module diagram of a server provided in an embodiment of the present application. As shown in Figure 6, the server includes:
- the receiving unit 601 is configured to receive first data to be stored and a writing position of the first data to be stored in a block device sent by a client;
- a comparison unit 602 is configured to compare the writing position of the first to-be-stored data with the regional position information of the first file system generated in the server;
- the first file system is the service One of a plurality of file systems that have been generated in the client, each of the file system's area location information is generated according to the storage space size of the block device, and each of the file system's area location information includes metadata of the file system Location information of the data area and general data area;
- the data writing unit 603 is configured to write the first data to be stored when the comparison unit 602 determines that the writing position of the first data to be stored is located in a metadata area of the first file system.
- the cache area of this server is configured to write the first data to be stored when the comparison unit 602 determines that the writing position of the first data to be stored is located in a metadata area of the first file system. The cache area of this server.
- the metadata area of each file system includes multiple sub-metadata areas
- the writing position of the first data to be stored in the block device includes the first data to be stored in the block device.
- the starting writing position and the ending writing position; the comparison unit 602 is configured to determine that the starting writing position and the ending writing position of the first to-be-stored data are located in the same sub-unit of the first file system When in the data area, it is determined that the writing position of the first data to be stored is located in the metadata area of the first file system.
- the receiving unit 601 is further configured to receive the second to-be-stored data and the writing position of the second to-be-stored data in the block device sent by the client;
- the second The writing position of the data to be stored in the block device includes a start writing position and an end writing position of the second data to be stored in the block device;
- the comparison unit 602 is further configured to compare the writing position of the second data to be stored with the area position information of the first file system; if the starting writing position of the second data to be stored is And the end writing position are located in different sub-metadata areas of the first file system, or the start writing position of the second to-be-stored data is located in the metadata area of the first file system and the second The end writing position of the data to be stored is outside the metadata area of the first file system, and then the writing position of the second data to be stored and the area position of the second file system in the plurality of file systems Information comparison
- the writing data unit 603 is further configured to: when the comparing unit 602 determines that the start writing position and the ending writing position of the second to-be-stored data are located in a same sub-metadata area of the second file system , Writing the second to-be-stored data into a cache area of the server.
- the write data unit 603 is further configured to: if the metadata area range [x1, y1] of the first file system includes the metadata area range [x2, y2], the data stored in the start write position and / or the end write position in [x1, x2] stored in the cache area of this server, and the start write position and / Or the end write data located in [y2, y1] is written to the block device of this server.
- the write data unit 603 is further configured to: if the range of the metadata area [x1, y1] of the first file system and the range of the metadata area [x2, y2] if there is an intersection and x1> x2, y1> y2, then write the data stored in the cache area of the server and / or the end write location in [y2, y1] to the block of the server
- the device, and the data stored in the start writing position and the end writing position stored in the block device of the server are both in [x2, x1] and written to the buffer area of the server.
- the write data unit 603 is further configured to: if the range of the metadata area [x1, y1] of the first file system and the range of the metadata area [x2, y2] if there is an intersection and x1 ⁇ x2, y1 ⁇ y2, the data stored in the start and / or end write locations in the cache area of this server is written to the block of this server
- the device, as well as the data stored in the block device of the server at the beginning and end of writing, are located in the buffer area of the server.
- the write data unit 603 is further configured to: if the metadata area range [x2, y2] of the second file system includes the metadata area range [x1, y1], the data stored in the start block and end write positions of the server block device are both located in [x2, x1], and the start write position and end are stored in the block device of this server end The data whose write positions are located in [y1, y2] are written into the cache area of this server.
- the write data unit 603 is further configured to, when it is determined that the write position of the first data to be stored is located in a general data area of each file system of the multiple file systems, Determine that the first data to be stored is general data, and determine whether the available space in the cache area of the server is larger than the occupied space of the first data to be stored; if so, write the first data to be stored in the service The cache area on the server side; if not, the elimination data is determined from the general data stored in the cache area of the server side according to a preset elimination algorithm; according to the write position of the elimination data, an adjacent area is selected from the cache area on the server side Write at least one data at a location; write the obsolete data together with the at least one data with an adjacent write location into a block device on the server side.
- the division of the units in the embodiments of the present application is schematic, and is only a logical function division. There may be another division manner in actual implementation.
- the functional units in the embodiments of the present application may be integrated into one processing unit, or each of the units may exist separately physically, or two or more units may be integrated into one unit.
- the above integrated unit may be implemented in the form of hardware or in the form of software functional unit.
- FIG. 7 is a schematic diagram of a hardware structure of a server provided in an embodiment of the present application.
- the server includes a processor 701, a communication interface 702, a memory 703, and a bus 704. Among them, the processor 701, the communication interface 702, and the memory 703 Connected to each other via a bus.
- the processor 701 executes the method shown in FIG. 2 by calling the machine-readable instructions.
- An embodiment of the present application further provides a machine-readable storage medium that stores machine-readable instructions.
- the machine-executable instructions When the machine-readable instructions are called and executed by a processor, the machine-executable instructions cause The processor implements the method shown in FIG. 2.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Claims (11)
- 一种数据写入方法,所述方法应用于服务端,包括:A data writing method applied to a server includes:接收客户端发送的第一待存储数据和所述第一待存储数据在块设备中的写入位置;Receiving the first to-be-stored data and the writing position of the first to-be-stored data in the block device sent by the client;将所述第一待存储数据的写入位置与所述服务端中已生成的第一文件系统的区域位置信息进行比对;所述第一文件系统为所述服务端中已生成的多个文件系统之一,每个所述文件系统的区域位置信息根据所述块设备的存储空间大小生成,每个所述文件系统的区域位置信息包括该文件系统的元数据区和一般数据区的位置信息;Comparing the writing position of the first to-be-stored data with the regional position information of the first file system that has been generated in the server; the first file system is a plurality of files that have been generated in the server One of the file systems, the area location information of each of the file systems is generated according to the storage space size of the block device, and the area location information of each of the file systems includes the location of a metadata area and a general data area of the file system information;在确定所述第一待存储数据的写入位置位于所述第一文件系统的元数据区内时,将所述第一待存储数据写入所述服务端的缓存区。When it is determined that the writing position of the first to-be-stored data is located in a metadata area of the first file system, the first to-be-stored data is written into a cache area of the server.
- 如权利要求1所述的方法,其特征在于,The method according to claim 1, wherein:每个所述文件系统的元数据区包括多个子元数据区,The metadata area of each of the file systems includes a plurality of sub-metadata areas,所述第一待存储数据在所述块设备中的写入位置包括所述第一待存储数据在所述块设备中的开始写入位置和结束写入位置;The writing position of the first data to be stored in the block device includes a start writing position and an end writing position of the first to be stored data in the block device;确定所述第一待存储数据的写入位置位于所述第一文件系统的元数据区内,包括:Determining that the writing position of the first to-be-stored data is located in a metadata area of the first file system includes:如果所述第一待存储数据的开始写入位置和结束写入位置位于所述第一文件系统的同一个子元数据区内,则确定所述第一待存储数据的写入位置位于所述第一文件系统的元数据区内。If the start writing position and the ending writing position of the first to-be-stored data are located in the same sub-metadata area of the first file system, determine that the writing position of the first to-be-stored data is located in the first A file system's metadata area.
- 如权利要求2所述的方法,还包括:The method of claim 2, further comprising:接收所述客户端发送的第二待存储数据和所述第二待存储数据在所述块设备中的写入位置;所述第二待存储数据在所述块设备中的写入位置包括所述第二待存储数据在所述块设备中的开始写入位置和结束写入位置;Receiving the second to-be-stored data and the writing position of the second to-be-stored data in the block device sent by the client; the writing position of the second to-be-stored data in the block device includes the The start writing position and the end writing position of the second to-be-stored data in the block device;将所述第二待存储数据的写入位置与所述第一文件系统的区域位置信息进行比对;Comparing the writing position of the second data to be stored with the area position information of the first file system;如果所述第二待存储数据的开始写入位置和结束写入位置位于所述第一文件系统的不同子元数据区内,或者所述第二待存储数据的开始写入位置位于所述第一文件系统的元数据区内且所述第二待存储数据的结束写入位置位于所述第一文件系统的元数据区外,则将所述第二待存储数据的写入位置与所述多个文件系统中的第二文件系统的区域位置信息进行比对;If the start writing position and the ending writing position of the second to-be-stored data are located in different sub-metadata areas of the first file system, or the start writing position of the second to-be-stored data is located in the first A file system's metadata area and the end writing position of the second to-be-stored data is outside the metadata area of the first file system, then writing the second to-be-stored data's writing position with the Comparing the location information of the second file system in the multiple file systems;在确定所述第二待存储数据的开始写入位置和结束写入位置位于所述第二文件系统的同一个子元数据区内时,将所述第二待存储数据写入所述服务端的缓存区。When it is determined that the start writing position and the ending writing position of the second to-be-stored data are located in the same sub-metadata area of the second file system, write the second to-be-stored data into a cache of the server Area.
- 如权利要求3所述的方法,其特征在于,在确定所述第二待存储数据的开始写入位置和结束写入位置位于所述第二文件系统的同一个子元数据区内之后,所述方法还 包括:The method according to claim 3, wherein after determining that the start writing position and the ending writing position of the second to-be-stored data are located in the same sub-metadata area of the second file system, the method The method also includes:如果所述第一文件系统的元数据区范围[x1,y1]包含所述第二文件系统的元数据区范围[x2,y2],则将保存在所述服务端的缓存区中的开始写入位置和/或结束写入位置位于[x1,x2]内的数据、以及保存在所述服务端的缓存区中的开始写入位置和/或结束写入位置位于[y2,y1]内的数据写入所述块设备。If the range [x1, y1] of the metadata area of the first file system includes the range [x2, y2] of the metadata area of the second file system, start writing to the cache area of the server Data whose position and / or end write position is in [x1, x2], and data write start and / or end write positions stored in the cache area of the server are written in [y2, y1] Into the block device.
- 如权利要求3所述的方法,其特征在于,在确定所述第二待存储数据的开始写入位置和结束写入位置位于所述第二文件系统的同一个子元数据区内之后,所述方法还包括:The method according to claim 3, wherein after determining that the start writing position and the ending writing position of the second to-be-stored data are located in the same sub-metadata area of the second file system, the method The method also includes:如果所述第一文件系统的元数据区范围[x1,y1]与所述第二文件系统的元数据区范围[x2,y2]存在交集且x1>x2,y1>y2,则If the metadata area range [x1, y1] of the first file system and the metadata area range [x2, y2] of the second file system intersect and x1> x2, y1> y2, then将保存在所述服务端的缓存区中的开始写入位置和/或结束写入位置位于[y2,y1]内的数据写入所述块设备,以及Writing data in the start writing position and / or ending writing position in [y2, y1] stored in the cache area of the server to the block device, and将保存在所述块设备中的开始写入位置和结束写入位置均位于[x2,x1]内的数据写入所述服务端的缓存区。Write the data stored in the block device at the start writing position and the end writing position in [x2, x1] into the cache area of the server.
- 如权利要求3所述的方法,其特征在于,在确定所述第二待存储数据的开始写入位置和结束写入位置位于所述第二文件系统的同一个子元数据区内之后,所述方法还包括:The method according to claim 3, wherein after determining that the start writing position and the ending writing position of the second to-be-stored data are located in the same sub-metadata area of the second file system, the method The method also includes:如果所述第一文件系统的元数据区范围[x1,y1]与所述第二文件系统的元数据区范围[x2,y2]存在交集且x1<x2,y1<y2,则If the metadata area range [x1, y1] of the first file system and the metadata area range [x2, y2] of the second file system intersect and x1 <x2, y1 <y2, then将保存在所述服务端的缓存区中的开始写入位置和/或结束写入位置位于[x1,x2]内的数据写入所述块设备,以及Writing data in the start writing position and / or the end writing position in [x1, x2] stored in the cache area of the server to the block device, and将保存在所述块设备中的开始写入位置和结束写入位置均位于[y1,y2]内的数据写入所述服务端的缓存区。Write the data stored in the block device at the start writing position and the end writing position in [y1, y2] into the cache area of the server.
- 如权利要求3所述的方法,其特征在于,在确定所述第二待存储数据的开始写入位置和结束写入位置位于所述第二文件系统的同一个子元数据区内之后,所述方法还包括:The method according to claim 3, wherein after determining that the start writing position and the ending writing position of the second to-be-stored data are located in the same sub-metadata area of the second file system, the method The method also includes:如果所述第二文件系统的元数据区范围[x2,y2]包含所述第一文件系统的元数据区范围[x1,y1],则将保存在所述块设备中的开始写入位置和结束写入位置均位于[x2,x1]内的数据、以及保存在所述块设备中的开始写入位置和结束写入位置均位于[y1,y2]内的数据写入所述服务端的缓存区。If the range [x2, y2] of the metadata area of the second file system includes the range [x1, y1] of the metadata area of the first file system, the start writing position and The data whose end writing position is located in [x2, x1], and the data whose start writing position and end writing position are stored in [y1, y2] stored in the block device are written into the cache of the server Area.
- 如权利要求1所述的方法,还包括:The method of claim 1, further comprising:在确定所述第一待存储数据的写入位置位于所述多个文件系统中每个所述文件系统的一般数据区内时,When it is determined that the writing position of the first to-be-stored data is located in a general data area of each of the file systems in the plurality of file systems,确定所述第一待存储数据为一般数据,Determining that the first data to be stored is general data,判断所述服务端的缓存区的可用空间是否大于所述第一待存储数据的占用空间;Determining whether the available space in the cache area of the server is larger than the occupied space of the first data to be stored;如果所述缓存区的可用空间大于所述第一待存储数据的占用空间,则将所述第一待存储数据写入所述服务端的缓存区;If the available space of the cache area is larger than the occupied space of the first data to be stored, writing the first data to be stored into the cache area of the server;如果所述缓存区的可用空间小于或等于所述第一待存储数据的占用空间,则If the available space of the buffer area is less than or equal to the occupied space of the first data to be stored, then根据预设的淘汰算法在所述服务端的缓存区保存的一般数据中确定淘汰数据;Determining elimination data from general data stored in a cache area of the server according to a preset elimination algorithm;根据所述淘汰数据的写入位置,从所述服务端的缓存区中选择具有相邻写入位置的至少一个数据;并Selecting at least one data with an adjacent write position from a cache area of the server according to the write position of the obsolete data; and将所述淘汰数据与所述至少一个数据一起写入所述块设备。Write the obsolete data together with the at least one data into the block device.
- 如权利要求1所述的方法,还包括:The method of claim 1, further comprising:建立文件系统模板库;Establish a file system template library;将所述块设备的存储空间大小作为输入参数传入所述文件系统模板库;Sending the size of the storage space of the block device as an input parameter into the file system template library;利用所述文件系统模板库生成与所述块设备的存储空间大小相匹配的所述多个文件系统的区域位置信息。Use the file system template library to generate region location information of the multiple file systems that match the storage space size of the block device.
- 一种服务端,包括:A server, including:接收单元,用于接收客户端发送的第一待存储数据和所述第一待存储数据在块设备中的写入位置;A receiving unit, configured to receive the first to-be-stored data sent by the client and a writing position of the first to-be-stored data in the block device;比对单元,用于将所述第一待存储数据的写入位置与所述服务端中已生成的第一文件系统的区域位置信息进行比对;所述第一文件系统为所述服务端中已生成的多个文件系统之一,每个所述文件系统的区域位置信息根据所述块设备的存储空间大小生成,每个所述文件系统的区域位置信息包括该文件系统的元数据区和一般数据区的位置信息;A comparison unit, configured to compare the writing position of the first to-be-stored data with the area position information of the first file system that has been generated in the server; the first file system is the server One of a plurality of file systems that have been generated in the system, each of the file system's area location information is generated according to the storage space size of the block device, and each of the file system's area location information includes a metadata area of the file system And the location information of the general data area;写数据单元,用于在所述比对单元确定所述第一待存储数据的写入位置位于所述第一文件系统的元数据区内时,将所述第一待存储数据写入所述服务端的缓存区。A data writing unit, configured to write the first data to be stored when the comparison unit determines that a writing position of the first data to be stored is located in a metadata area of the first file system; The cache area of the server.
- 一种服务端,包括处理器、通信接口、存储器和总线,其中,所述处理器、所述通信接口、所述存储器通过所述总线相互连接;A server includes a processor, a communication interface, a memory, and a bus, wherein the processor, the communication interface, and the memory are connected to each other through the bus;所述存储器中存储有机器可读指令,所述处理器通过调用所述机器可读指令执行如权利要求1至9任一项所述的方法。Machine-readable instructions are stored in the memory, and the processor executes the method according to any one of claims 1 to 9 by calling the machine-readable instructions.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810857497.7A CN110795386B (en) | 2018-07-31 | 2018-07-31 | Data writing method and server |
CN201810857497.7 | 2018-07-31 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020024933A1 true WO2020024933A1 (en) | 2020-02-06 |
Family
ID=69230981
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2019/098338 WO2020024933A1 (en) | 2018-07-31 | 2019-07-30 | Data writing method and server |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN110795386B (en) |
WO (1) | WO2020024933A1 (en) |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113296700A (en) * | 2021-04-20 | 2021-08-24 | 新华三信息技术有限公司 | Storage space management method and device and server |
CN113986838B (en) * | 2021-12-28 | 2022-03-11 | 成都云祺科技有限公司 | Mass small file processing method and system based on file system and storage medium |
CN114356241B (en) * | 2021-12-28 | 2023-11-03 | 天翼云科技有限公司 | Small object data storage method, small object data storage device, electronic equipment and readable medium |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103399823A (en) * | 2011-12-31 | 2013-11-20 | 华为数字技术(成都)有限公司 | Method, equipment and system for storing service data |
CN105117351A (en) * | 2015-09-08 | 2015-12-02 | 华为技术有限公司 | Method and apparatus for writing data into cache |
WO2015183240A1 (en) * | 2014-05-27 | 2015-12-03 | Hewlett-Packard Development Company, L.P. | Storing compressed data |
CN107153508A (en) * | 2016-03-03 | 2017-09-12 | 杭州海康威视数字技术股份有限公司 | A kind of date storage method and device |
CN108021513A (en) * | 2016-11-02 | 2018-05-11 | 杭州海康威视数字技术股份有限公司 | A kind of date storage method and device |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7752391B2 (en) * | 2006-01-20 | 2010-07-06 | Apple Inc. | Variable caching policy system and method |
CN102103880B (en) * | 2009-12-22 | 2014-02-19 | 鸿富锦精密工业(深圳)有限公司 | Universal serial bus (USB) flash disk and memory capacity display method thereof |
CN103593262B (en) * | 2013-11-15 | 2016-05-25 | 上海爱数信息技术股份有限公司 | A kind of virtual machine backup method based on classification |
CN107451138A (en) * | 2016-05-30 | 2017-12-08 | 中兴通讯股份有限公司 | A kind of distributed file system storage method and system |
CN107491523B (en) * | 2017-08-17 | 2020-05-05 | 三星(中国)半导体有限公司 | Method and device for storing data object |
-
2018
- 2018-07-31 CN CN201810857497.7A patent/CN110795386B/en active Active
-
2019
- 2019-07-30 WO PCT/CN2019/098338 patent/WO2020024933A1/en active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103399823A (en) * | 2011-12-31 | 2013-11-20 | 华为数字技术(成都)有限公司 | Method, equipment and system for storing service data |
WO2015183240A1 (en) * | 2014-05-27 | 2015-12-03 | Hewlett-Packard Development Company, L.P. | Storing compressed data |
CN105117351A (en) * | 2015-09-08 | 2015-12-02 | 华为技术有限公司 | Method and apparatus for writing data into cache |
CN107153508A (en) * | 2016-03-03 | 2017-09-12 | 杭州海康威视数字技术股份有限公司 | A kind of date storage method and device |
CN108021513A (en) * | 2016-11-02 | 2018-05-11 | 杭州海康威视数字技术股份有限公司 | A kind of date storage method and device |
Also Published As
Publication number | Publication date |
---|---|
CN110795386B (en) | 2022-07-01 |
CN110795386A (en) | 2020-02-14 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10635310B2 (en) | Storage device that compresses data received from a host before writing therein | |
US7788460B2 (en) | Defragmenting objects in a storage medium | |
US7120767B2 (en) | Snapshot creating method and apparatus | |
US6598129B2 (en) | Storage device and method for data sharing | |
US9311002B1 (en) | Systems, methods, and computer readable media for compressing data at a virtually provisioned storage entity | |
US10635329B2 (en) | Method and apparatus for performing transparent mass storage backups and snapshots | |
US9778860B2 (en) | Re-TRIM of free space within VHDX | |
US8694563B1 (en) | Space recovery for thin-provisioned storage volumes | |
US6205529B1 (en) | Method and apparatus for defragmenting a storage device using a copy function in the device control logic | |
WO2020024933A1 (en) | Data writing method and server | |
US11403021B2 (en) | File merging method and controller | |
US10585594B1 (en) | Content-based caching using digests | |
US9922039B1 (en) | Techniques for mitigating effects of small unaligned writes | |
CN103577470B (en) | A kind of file system and method for lifting web server performance | |
US9398073B1 (en) | Methods and apparatus for providing content | |
US9367256B2 (en) | Storage system having defragmentation processing function | |
US9430492B1 (en) | Efficient scavenging of data and metadata file system blocks | |
US11132145B2 (en) | Techniques for reducing write amplification on solid state storage devices (SSDs) | |
US20050108296A1 (en) | File system preventing file fragmentation | |
US8255611B1 (en) | Methods and apparatus for accessing content | |
US10180901B2 (en) | Apparatus, system and method for managing space in a storage device | |
WO2015058641A1 (en) | Hybrid hard disk implementation method and device | |
US6601135B1 (en) | No-integrity logical volume management method and system | |
CN110795031A (en) | Data deduplication method, device and system based on full flash storage | |
US10089125B2 (en) | Virtual machines accessing file data, object data, and block data |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19843206 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19843206 Country of ref document: EP Kind code of ref document: A1 |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19843206 Country of ref document: EP Kind code of ref document: A1 |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 11.08.2021) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19843206 Country of ref document: EP Kind code of ref document: A1 |