CN117806986B

CN117806986B - Multi-stream management method of application data and flash memory device

Info

Publication number: CN117806986B
Application number: CN202311863315.4A
Authority: CN
Inventors: 李文捷; 吕涛; 单野林; 杨颖�; 陈祥
Original assignee: Shenzhen Dapu Microelectronics Co Ltd
Current assignee: Shenzhen Dapu Microelectronics Co Ltd
Priority date: 2023-12-29
Filing date: 2023-12-29
Publication date: 2024-10-15
Anticipated expiration: 2043-12-29
Also published as: CN117806986A

Abstract

The embodiment of the application relates to the technical field of storage management and discloses a multi-stream management method of application data and a flash memory device, wherein the method generates a plurality of software streams by clustering data with similar compression rate, estimates the life cycle of each software stream, maps the software streams with similar life cycles into the same hardware stream, and the data corresponding to the software flow is stored in the storage position corresponding to the hardware flow, and the application can further store the application data with similar life cycle in the storage position corresponding to the hardware flow based on the compression rate so as to optimize the write amplification problem of the flash memory device.

Description

Multi-stream management method of application data and flash memory device

Technical Field

The present application relates to the field of storage management technologies, and in particular, to a method for managing multiple streams of application data and a flash memory device.

Background

The solid state disk (SolidStateDisk, SSD) based on the nonvolatile storage medium (NAND FLASH) can support ' write-in-place ', namely, write-in-place ' directly writing data into the storage medium, while NAND FLASH needs to erase old data before writing operation, and recovery is performed on invalid pages through a garbage recovery mechanism, but when garbage recovery is performed on the data, a large number of valid pages still remain in a block to be recovered, and the valid pages must be written into an idle position first, so that the erase operation of the block and the recovery of the block can be performed for subsequent use.

However, the write-back operation of the effective page causes additional write operation, which results in write amplification, the write amplification not only consumes bandwidth resources in storage and reduces the read and write performance, but also reduces the service life of the solid state disk, so that the solid state disk capable of supporting multi-stream characteristics is required to support data isolation on a hardware level, application software is allowed to shunt data with different update modes (such as additional update and in-situ update) and different life cycles, so that data with close life cycles are placed in the same physical block, data with large life cycle difference are placed in different physical blocks, and when garbage recovery is performed, the life cycle of the data of the recovered physical block is basically invalid, thereby reducing the data quantity required to write back and further reducing the write amplification problem of the solid state disk, and being important for how to perform accurate, efficient and extensible multi-stream management of the solid state disk supporting multi-stream characteristics.

At present, a common mechanism for multi-stream management is mainly an automatic shunt management mechanism based on statistical information and opaque host, and the automatic shunt management mechanism predicts the life cycle of data based on the characteristic information of the data by counting the characteristic information of the data corresponding to a write request, and further maps the data with different life cycles into different stream identifications through an algorithm, which has the defects that:

(1) A large amount of characteristic information of input or output data needs to be counted, and memory resources of a host need to be consumed.

(2) There remains a need to modify the kernel of an operating system, for example: the kernel of the operating system comprises a virtual file system, a device driver and the like, the transparency of the host cannot be realized, the mode has larger dependence on application software and is opaque to the host, and the problem of write amplification of the solid-state disk cannot be effectively improved.

Disclosure of Invention

The embodiment of the application provides a multi-stream management method of application data and a flash memory device, which solve the technical problem of data write amplification and can store application data with similar life cycle in a storage position corresponding to a hardware stream so as to optimize the problem of the write amplification of the flash memory device.

In order to solve the technical problems, the embodiment of the application provides the following technical scheme:

in a first aspect, an embodiment of the present application provides a method for managing multiple streams of application data, where the method includes:

acquiring the compression rate of application data;

Clustering the application data according to the compression rate of the application data to generate a plurality of software streams;

estimating a lifecycle of each software flow;

clustering the software streams according to the life cycle of each software stream to establish a first mapping relation between each software stream and the hardware stream;

And storing the data corresponding to the software stream to a storage position corresponding to the hardware stream according to the first mapping relation.

In some embodiments, obtaining the compression rate of the application data includes:

Determining a compression algorithm corresponding to the application data according to the data type corresponding to the application data;

and compressing the application data according to a compression algorithm to obtain the compression rate of the application data.

In some embodiments, each software stream corresponds to at least one logical block address, the logical block address includes a plurality of logical pages, and data corresponding to the software stream is stored in the logical pages;

estimating a lifecycle of each software flow, comprising:

estimating the life cycle of all logic pages corresponding to each software stream;

the average value of the life cycles of all the logical pages corresponding to each software stream is calculated to determine the life cycle of each software stream.

In some embodiments, estimating the lifecycle of all logical pages corresponding to each software stream includes:

Counting the time stamp of each logic page access, wherein the time stamp comprises a first time stamp and a second time stamp, and the first time stamp and the second time stamp are adjacent time stamps;

The difference between the first timestamp and the second timestamp is calculated to determine the lifecycle of each logical page.

In some embodiments, each hardware flow corresponds to a preset life cycle one by one, and the software flows are clustered according to the life cycle of each software flow to establish a first mapping relationship between the software flows and the hardware flows, including:

if the life cycle of the software flow is within the preset life cycle, mapping the software flow to a hardware flow corresponding to the preset life cycle to determine a first mapping relation between the software flow and the hardware flow, wherein each hardware flow corresponds to a plurality of software flows.

In some embodiments, the storage location comprises a plurality of physical pages, the method further comprising:

the method for recycling the valid data in the physical page corresponding to the storage position corresponding to the hardware stream comprises the following steps:

counting feedback information of physical pages corresponding to storage positions corresponding to each hardware flow;

and according to the feedback information, moving the effective data in the storage position to a preset storage position.

In some embodiments, the feedback information includes a write back frequency of the physical page, a hardware stream number where the physical page is located;

according to the feedback information, moving the effective data in the storage position to a preset storage position comprises the following steps:

determining the write-back frequency of the physical page in a period of time according to the write-back frequency of the physical page, and moving the effective data in the physical page corresponding to the storage position to a preset storage position according to the write-back frequency;

And/or the number of the groups of groups,

According to the hardware stream number of the physical page, moving the effective data in the storage position to a preset storage position according to a state machine;

Before moving the valid data in the storage location to the preset storage location, the method further comprises:

And establishing a second mapping relation between each hardware stream and a preset hardware stream, wherein the preset hardware stream corresponds to a preset storage position.

In some embodiments, moving valid data in a physical page corresponding to a storage location to a preset storage location according to write-back frequency includes:

And if the write-back frequency is greater than or equal to the frequency threshold, moving the effective data in the physical page corresponding to the storage position to a preset storage position according to the second mapping relation.

In a second aspect, an embodiment of the present application provides a flash memory device, including:

The compression module is used for determining a compression algorithm corresponding to the application data according to the data type corresponding to the application data and compressing the application data according to the compression algorithm so as to obtain the compression rate of the application data;

the first clustering module is used for clustering the application data according to the compression rate of the application data to generate a plurality of software streams;

A life cycle estimating module for estimating a life cycle of each software flow;

The second clustering module is used for clustering the software streams according to the life cycle of each software stream so as to establish a first mapping relation between each software stream and the hardware stream;

The flash memory conversion module is used for storing data corresponding to the software stream to a storage position corresponding to the hardware stream according to the first mapping relation;

The remapping module is used for recycling the effective data in the storage position corresponding to the hardware stream so as to move the effective data in the storage position to a preset storage position;

And the flash memory medium is used for storing application data.

In a third aspect, an embodiment of the present application provides a flash memory device, including:

at least one processor; and

A memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of multi-stream administration of application data as in the first aspect

The embodiment of the application has the beneficial effects that: in comparison with the prior art, the method for managing the multiple streams of the application data provided by the embodiment of the application comprises the following steps: acquiring the compression rate of application data; clustering the application data according to the compression rate of the application data to generate a plurality of software streams; estimating a lifecycle of each software flow; clustering the software streams according to the life cycle of each software stream to establish a first mapping relation between each software stream and the hardware stream; and storing the data corresponding to the software stream to a storage position corresponding to the hardware stream according to the first mapping relation.

Generating a plurality of software streams by clustering data with similar compression ratios, estimating the life cycle of each software stream, mapping the software streams with similar life cycles into the same hardware stream, and stores the data corresponding to the software stream to the storage location corresponding to the hardware stream, the application can store the application data with similar life cycle in the storage position corresponding to the hardware flow so as to optimize the write amplification problem of the flash memory device.

Drawings

One or more embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements, and in which the figures of the drawings are not to be taken in a limiting sense, unless otherwise indicated.

FIG. 1 is a schematic diagram of a connection relationship between a host and a flash memory device according to an embodiment of the present application;

FIG. 2 is a schematic flow chart of a manual bypass mechanism according to an embodiment of the present application;

FIG. 3 is a flow chart of an automated offloading mechanism based on statistics according to an embodiment of the present application;

FIG. 4A is a schematic flow chart of clustering and splitting application data based on compression rate according to an embodiment of the present application;

FIG. 4B is a schematic flow chart of two-stage clustering and splitting of application data according to an embodiment of the present application;

FIG. 4C is a flow chart of a two-stage based multi-stream mapping method according to an embodiment of the present application;

FIG. 5 is a flow chart of a method for managing multiple streams of application data according to an embodiment of the present application;

Fig. 6 is a schematic diagram of a refinement flow of step S501 of fig. 5;

FIG. 7 is a schematic diagram illustrating an example of compression ratios corresponding to different types of data according to an embodiment of the present application;

fig. 8 is a schematic diagram of a refinement flow of step S503 of fig. 5;

Fig. 9 is a schematic diagram of a refinement flow of step S5031 of fig. 8;

Fig. 10 is a schematic diagram of a refinement flow of step S504 of fig. 5;

FIG. 11 is a schematic flow chart of recovering valid data in a physical page according to an embodiment of the present application;

Fig. 12 is a flowchart illustrating a process of establishing a second mapping relationship between a hardware flow and a preset hardware flow according to an embodiment of the present application;

fig. 13 is a schematic diagram of a refinement flow of step S1101 of fig. 11;

fig. 14 is a schematic diagram of a refinement flow of step S1112 of fig. 13;

fig. 15 is a schematic diagram of a refinement flow of step S1121 of fig. 14;

FIG. 16 is a schematic diagram of the overall structure of a flash memory device according to an embodiment of the present application;

fig. 17 is a schematic structural diagram of another flash memory device according to an embodiment of the present application.

Reference numerals illustrate:

Reference numerals	Name of the name	Reference numerals	Name of the name
				100	Flash memory device	106	Flash memory medium
101	Compression module	107	Remapping module
				102	First clustering module	108	Processor and method for controlling the same
103	Lifecycle estimation module	109	Memory device
				104	Second aggregation module	200	Host machine
105	Flash memory conversion module

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

It should be noted that, if not in conflict, the features of the embodiments of the present application may be combined with each other, which is within the protection scope of the present application. In addition, while functional block division is performed in a device diagram and logical order is shown in a flowchart, in some cases, the steps shown or described may be performed in a different order than the block division in the device, or in the flowchart. Furthermore, the words "first," "second," "third," and the like as used herein do not limit the order of data and execution, but merely distinguish between identical or similar items that have substantially the same function and effect.

In addition, the technical features of the embodiments of the present application described below may be combined with each other as long as they do not collide with each other.

Prior art known to the present inventors will be briefly described before describing embodiments of the present application, so that the following description is convenient for understanding the embodiments of the present application.

The multi-streaming technique is a solid state disk interface that allows a host to pass data semantic information. According to the semantic information of the data provided by the host, the multi-stream solid-state disk can reasonably place the written data. When the semantic information of the data is the expected service life of the data, the solid-state disk can store the data with the near service life in the same erasing block, so that the data of the same erasing block are failed as much as possible, the number of the migrated effective pages and the write amplification during garbage recovery are reduced, and the performance and the service life of the solid-state disk are finally improved.

The technical scheme of the application is specifically described below with reference to the accompanying drawings of the specification:

referring to fig. 1, fig. 1 is a schematic diagram illustrating a connection relationship between a host and a flash memory device according to an embodiment of the present application;

As shown in fig. 1, the flash memory device 100 is configured to process application data of the host 200, to classify the application data of the host 200 according to data types, to generate a plurality of software flows by clustering the compressed application data, to perform life cycle estimation on each software flow, and further, to establish a mapping relationship between the software flow corresponding to the application data of the host 200 and a hardware flow of the flash memory device 100, to store the application data corresponding to the software flows with different life cycles into different hardware flows according to the mapping relationship, so as to realize separate storage of application data with larger life cycle differences, and to optimize the problem of write amplification of data in the flash memory device 100.

The flash memory device 100 includes a Solid state disk (Solid STATE DRIVES, SSD), a usb disk, a memory card, a flash disk (a storage device integrated with the usb disk and the memory card), and other storage devices for storing application data.

The solid state disk includes a Flash memory medium, which is used as a storage medium of the solid state disk, and also referred to as a Flash memory, a Flash, a NAND FLASH memory or a Flash granule, and the Flash memory medium is used as a storage unit for application data, system data and the like.

The NAND FLASH memory is a nonvolatile memory, and can store data for a long time under the condition of no current supply. The NAND FLASH memory takes a single transistor as a memory unit of binary signals, and the structure of the memory is very similar to that of a common semiconductor transistor, wherein the single transistor of the NAND FLASH memory is provided with a floating gate and a control gate, the floating gate is used for storing electrons, the surface of the floating gate is covered by a layer of silicon oxide insulator, the surface of the floating gate is coupled with a control gate through a capacitor, when negative electrons are injected into the floating gate under the action of the control gate, the storage state of a single crystal of the NAND FLASH memory 12 is changed from '1' to '0', and when the negative electrons are removed from the floating gate, the storage state is changed from '0' to '1', and the insulator covered on the surface of the floating gate is used for trapping the negative electrons in the floating gate, so that data storage is realized. I.e., NAND FLASH, memory 12 is a floating gate transistor that is used to store data in the form of electrical charges. The amount of charge stored is related to the magnitude of the voltage applied by the floating gate transistor, and in particular, whether data is stored in the memory cell depends on whether the voltage of the charge stored therein is greater than a preset voltage threshold Vth.

The NAND FLASH memories can be classified into SLC, MLC, TLC and QLC according to the different levels of voltages of the memory cells. When the memory cell of NAND FLASH is of the SLC type, the single memory cell only stores one bit of data 1 or 0, and when the voltage of the charge stored in the single memory cell is greater than a preset voltage threshold Vth, the data representing the memory cell is 1; when the voltage of the electric charge stored in the individual memory cell is less than the preset voltage threshold Vth, the data indicating the memory cell is 0. Since the default data of the memory cell is 1, the data of the memory cell being 0 means that the charge stored in the single memory cell is discharged, and when the charge is discharged to such an extent that the voltage of the floating gate transistor is less than the preset voltage threshold Vth, the operation of writing the data 0 is completed. When the memory cell of NAND FLASH memory 12 is of MLC type, TLC type or QLC type, a single memory cell can store multi-bit data, for example, 2-bit data, by controlling the amount of charges in the memory cell, defining a plurality of preset voltage thresholds Vth, and for writing data, controlling the amount of charges in the memory cell by charging process, so that the charges fall into different adjacent two intervals of preset voltage thresholds Vth, corresponding to the data 00, 01, 10, 11; for reading data, the data 00, 01, 10, 11 stored in the memory cell are analyzed by acquiring the current in the corresponding memory cell and then completing reading through a series of decoding circuits.

One NAND FLASH memory includes at least one Chip, each Chip is composed of a plurality of Block physical blocks, each Block physical Block includes a plurality of Page pages. The Block physical Block is the minimum unit of the NAND FLASH memory for executing the erasing operation, the Page is the minimum unit of the NAND FLASH memory for executing the reading and writing operation, and the capacity of one NAND FLASH memory is equal to the number of the Block physical Block and the number of Page pages contained in the Block physical Block and the capacity of one Page.

Since the memory cell of NAND FLASH is implemented by floating gate technology, the charge on the floating gate layer can change the distribution state of electrons, thereby changing the state of the memory cell, but the charge on the floating gate layer leaks over time, so that periodic erasing and refreshing of the charge is required to maintain the endurance of the data.

Referring to fig. 2, fig. 2 is a schematic flow chart of a manual shunting mechanism according to an embodiment of the present application;

The manual distribution mechanism mainly judges the life cycle corresponding to different types of application data based on experience of a developer, namely, the life cycle of the application data is determined before the application data is input into storage positions corresponding to different hardware flows, and the data of different life cycles are mapped into the corresponding hardware flows according to the storage positions corresponding to different hardware flow identifications, wherein the life cycles of the application data stored in the storage positions corresponding to each hardware flow are different.

It will be appreciated that, due to the physical characteristics of the flash memory device and the configuration of the flash memory device, the data transmission speed and the concurrent streaming support capability of the flash memory device are limited, for example: the data transfer of a flash memory device is required to follow certain protocols and standards, such as SATA, PCI, etc., which limit the number of streams supported by the flash memory device, and thus the number of hardware streams supported by different flash memory devices is determined by hardware or other techniques.

As shown in fig. 2, the manual bypass mechanism is further described by taking RocksDB key-value database as an example, rocksDB key-value database is operated on a host, before data files in RocksDB key-value database are stored in a multi-stream solid-state disk in a bypass mode, life cycles of different data files in RocksDB key-value database are determined based on experience of a developer, by modifying RocksDB source codes corresponding to the key-value database, different write prompts (WRITE HINT) are set for software streams corresponding to different files, namely hardware streams corresponding to the software streams are manually modified, wherein the data files in RocksDB key-value database comprise data in write-ahead log (WAL) and Memtable, memtable is a data structure in a host memory for storing memory data, before data in Memtable is flushed to the multi-stream solid-state disk, files in different life cycles are required to be read from Memtable by sorting the files in a string table (Sorted Strings Table, SSTable) and the memory data are respectively stored in different life cycles in the memory layer, when the corresponding write prompt is written to the memory layer of the corresponding to the memory stream, the write prompt is written to the corresponding memory layer in the memory layer, and the memory is also written to the corresponding to the memory layer in the memory layer along with the write prompt layer in the memory stream, and the memory is written to the corresponding to the memory layer in the memory stream, and the hardware is identified when the write prompt is written to the corresponding to the hardware stream in the corresponding memory stream, and the hardware in the hardware is written to the memory stream is written.

For example: when the write prompt is a long life cycle, converting the write prompt into a hardware stream identifier 3, wherein the hardware stream identifier 3 corresponds to a storage position for storing a data file of the long life cycle; when the write prompt is in the middle life cycle, converting the write prompt into a hardware flow identifier 2, wherein a storage position corresponding to the hardware flow identifier 2 is used for storing a data file in the middle life cycle; when the write hint is in a short life cycle, converting the write hint into a hardware flow identifier 1, wherein a storage position corresponding to the hardware flow identifier 1 is used for storing a data file in the short life cycle, for example: the log file is often required to be refreshed, namely, the life cycle of the log file is short, the log file is stored in a storage position corresponding to the hardware flow mark 1; when the write hint is default, converting the write hint into a hardware flow identifier, where the hardware flow identifier is 0, and a storage location corresponding to the hardware flow identifier 0 is used for storing default data of the host, for example: system configuration information, application configuration information, default files or data, etc.

The above-mentioned manual bypass mechanism in fig. 2 is used to store host data on the multi-stream solid-state disk, which has the following disadvantages that (1) a developer needs to deeply understand host data (2) needs to modify source codes of application programs, the application programs are more invasive (3) when data generated by different application programs are stored on the multi-stream solid-state disk, each application program needs to be adapted to the multi-stream solid-state disk, and when the application programs are too many, the complexity of multi-stream management of the data of the application programs is increased. Based on this, the embodiment of the application provides a method for performing distribution management on application data by an automatic distribution mechanism so as to overcome the defects.

Specifically, referring to fig. 3, fig. 3 is a flow chart of an automatic shunting mechanism based on statistical information according to an embodiment of the present application;

As shown in fig. 3, the operating system kernel includes a statistics information collection module, a life cycle estimation module, and a multi-stream mapping module, where the statistics information collection module is used to count feature information of application data, for example: the feature information includes access frequency, rewriting interval and program counter digest of the logical block address corresponding to the application data, wherein the program counter digest includes values recorded by the program counter, such as the number of times of program execution, average execution time of the program, the longest execution time of the program, and the like.

The life cycle estimation module estimates the life cycle of the data of the write request based on the characteristic information, and automatically maps the data into hardware streams with different priorities according to the estimated life cycle, for example: in a drive layer in the multi-stream solid-state disk, by counting logical block addresses of a write request, feature information such as access heat (frequency), recency (recency), and sequency (sequentiality) of data corresponding to the logical block addresses can be obtained, and the feature information can reflect the life cycle of the data to a certain extent, for example, the higher the access heat is, the shorter the life cycle of the data is, the longer the life cycle of the data which is not accessed for a long time is, and the closer the life cycle of the data adjacent to the address is.

The multi-stream mapping module is used for mapping the data with similar life cycle to the same hardware stream identifier through algorithms such as multi-queues, clustering and the like so as to store the data with similar life cycle to the storage position corresponding to the same hardware stream identifier, thereby realizing automatic data distribution.

The automatic distribution mechanism can realize automatic distribution of data, but the following disadvantages still exist: the method comprises the steps of (1) modifying in an operating system kernel, such as a virtual file system, a block layer, a device driver and other modules, wherein the complexity is high (2) a large amount of I/O characteristic information is required to be counted, a large amount of memory resources of a host are consumed (3) the write amplification problem of the multi-stream solid state disk is not considered (4) and the shunt strategy is not evaluated.

Based on the above, the application provides a multi-stream management method of application data, which does not need to count a large amount of I/O characteristic information and saves memory resources of a host.

Referring to fig. 4A, fig. 4A is a schematic flow chart of application data clustering and splitting based on compression rate according to an embodiment of the present application;

As shown in fig. 4A, the flash memory device 100 includes a compression module 101, a first clustering module 102, a flash conversion module 105, and a flash memory medium 106.

The compression module 101 receives application data sent by a host, and compresses the application data, where the application data includes multiple types of data files, and compression rates corresponding to the different types of data files are different.

The first clustering module 102 clusters application data according to the compression rates corresponding to different types of data files, and maps data files with similar compression rates into the same hardware stream to establish a mapping relationship between the data files and the hardware stream.

The flash memory conversion module 105 stores the data file into the flash memory medium 106 according to the mapping relation between the data file and the hardware flow, wherein the flash memory medium 106 comprises a plurality of super blocks, each hardware flow corresponds to one super block, each hardware flow corresponds to one hardware flow identifier, the data file is stored into the flash memory medium 106, namely, according to the hardware flow identifier corresponding to the data file, the data file is stored into the super block corresponding to the hardware flow identifier corresponding to the data file.

In the embodiment of the application, the application data without the compression rate is clustered through the first clustering module to realize multi-stream mapping of the solid-state flash memory device, but the write amplification problem of the flash memory device cannot be optimized by only clustering the application data based on the compression rate.

In view of this, the embodiment of the application provides a method for performing two-stage clustering and splitting on application data, which can further cluster application data with similar life cycle in the same hardware stream based on the compression rate, so as to optimize the write amplification problem of the flash memory device.

Specifically, referring to fig. 4B, fig. 4B is a schematic flow chart of two-stage clustering and splitting of application data according to an embodiment of the present application;

As shown in fig. 4B, the flash memory device 100 includes a compression module 101, a first clustering module 102, a life cycle estimation module 103, a second clustering module 104, a flash conversion module 105, and a flash medium 106.

The first clustering module 102 clusters application data according to compression rates corresponding to different types of data files to generate a plurality of software streams, wherein data files with similar compression rates are mapped into the same software stream.

The life cycle estimation module 103 estimates the life cycle of each software flow based on the compression rate of the data file.

The second clustering module 104 clusters the software flows according to the lifecycle of the software flows, so as to map the software flows with the lifecycle close to each other into the same hardware flow, that is, establish a mapping relationship between the software flows and the hardware flows, where each hardware flow corresponds to a plurality of software flows. It will be appreciated that the number of hardware streams is determined by the type of flash memory device and that the number of hardware streams is typically less than the number of software streams, and therefore mapping software streams into hardware streams enables matching software streams to hardware streams.

The flash memory conversion module 105 stores the data in the software stream into the super block corresponding to the hardware stream according to the mapping relation between the software stream and the hardware stream.

In the embodiment of the application, the application data with similar compression rate are clustered through the first clustering module so as to map the application data with the same compression rate into the same software stream, and further, the software stream is clustered through the second clustering module so as to map the software stream with similar life cycle into the same hardware stream, and the application data with similar life cycle can be placed together through clustering the application data in two stages, thereby avoiding the occupation of multi-stream resources and solving the problem that the software stream is not matched with the hardware stream.

According to the multi-stream management method, application data with the life cycle being close to that of the application data are placed in the same hardware stream, the problem of write amplification of the flash memory device can be effectively solved, but the data with the compression rate being close to that of the application data but with the life cycle being longer still exist in the same hardware stream, so that the problem of write amplification is caused by repeated rewriting in the subsequent garbage recycling process, and the problem is further improved.

Referring to fig. 4C, fig. 4C is a flow chart of a two-stage-based multi-stream mapping method according to an embodiment of the application;

As shown in fig. 4C, fig. 4C is an improvement on the basis of fig. 4B, in order to solve the problem of write amplification in the garbage collection process mentioned in fig. 4B, a remapping module 107 is added in the flash memory device 100, where the remapping module 107 is used to map all hardware streams into a new hardware stream, and in the garbage collection process, the life cycle of the data in the physical pages in the super blocks corresponding to each hardware stream is written back into the new hardware stream, so that the problem of write amplification can be effectively optimized.

In the embodiment of the application, the application data with similar compression rate is clustered through the first clustering module so as to map the application data with the same compression rate into the same software stream, and further, the software stream is clustered through the second clustering module so as to map the software stream with similar life cycle into the same hardware stream, and the application data with similar life cycle can be placed together through clustering the application data in two stages, so that the occupation of multi-stream resources is avoided, the problem that the software stream is not matched with the hardware stream is solved, and meanwhile, the remapping is carried out on the data with longer life cycle and effectiveness in the data block corresponding to the hardware stream through the remapping module, so that the write amplification problem of the flash memory device can be effectively improved.

Referring to fig. 5, fig. 5 is a flow chart of a multi-stream management method for application data according to an embodiment of the present application;

as shown in fig. 5, the multi-stream management method of application data includes:

Step S501: acquiring the compression rate of application data;

In the embodiment of the application, the compression rate of the application data is internally related to the life cycle of the application data, for example, in RocksDB, cassandran application scenes, the compression rate of the WAL file is lower, but the life cycle of the WAL file is shorter, the compression rate of the SSTable file is higher, but the life cycle of the SSTable file is longer, namely, in general, the life cycle corresponding to the file with low compression rate is shorter, and the life cycle corresponding to the file with high compression rate is longer, so the compression rate can be used for indirectly representing the life cycle, thereby firstly carrying out data distribution on the application data based on the compression rate, and being capable of preliminarily mapping the application data with the life cycle close to one software stream.

Specifically, the application data is compressed by a compression algorithm to obtain a compression rate of the application data, wherein the compression algorithm comprises compression algorithms such as LZ4 and ZSTD, snappy, DEFLATE.

In the embodiment of the application, the total data volume written into the flash memory device can be reduced by compressing the application data.

Referring to fig. 6 again, fig. 6 is a schematic diagram of a refinement flow chart of step S501 in fig. 5;

As shown in fig. 6, this step S501 includes:

Step S5011: determining a compression algorithm corresponding to the application data according to the data type corresponding to the application data;

In the embodiment of the application, the application data has a plurality of data types, and in order to improve the compression rate and compression rate of the application data and improve the writing performance of the flash memory device, different compression algorithms are selected for the application data with different data types, for example: the log file and the temporary data are data which are transmitted and stored quickly, so that the log file and the temporary data can be processed more quickly by using an LZ4 algorithm, and for large data sets such as pictures, videos and the like, the large data sets can be processed in batches by using a ZSTD algorithm so as to improve compression efficiency.

Specifically, a compression algorithm corresponding to the application data is determined according to a data type corresponding to the application data, wherein the data type comprises structured data, unstructured data, real-time data, text data, log data and the like. It will be appreciated that the speed efficiency of processing different types of data by different compression algorithms may be different, for example: the LZ4 algorithm is more suitable for processing data which needs to be quickly transmitted and stored, such as log files, temporary data and the like; the ZSTD algorithm is more suitable for processing large data sets, such as: video, audio, pictures, etc.; the snappy algorithm is suitable for processing data of file type, such as text files, binary files, etc.

Referring to fig. 7 again, fig. 7 is a schematic diagram illustrating an example of compression ratios corresponding to different types of data according to an embodiment of the present application;

As shown in fig. 7, the compression rate of each data block in the first database file is substantially distributed around 0.17, the first database file includes nci _database database file, the data compression rate of the web page file is substantially distributed around 0.4, the web page file includes webster web page, the data compression rate of the text file is substantially distributed around 0.5, the text file includes dickens _txt text file, the data compression rate of each data block in the second database file is substantially distributed around 0.74, the second database file includes osdb _database database file, the data compression rate of the image file is substantially distributed around 0.82, the image file includes xray _image picture, the data compression rate of the binary file is substantially distributed around 0.85, and the binary file includes sao_bin text file.

As can be seen from the above data, the compression ratios of the data of different data types are different, and the compression ratio of the data of some different data types is relatively large, for example, the compression ratio of the data between the nci _database database file and the xray _image picture is relatively large, while the compression ratio of the data of some different data types is relatively small, for example, the compression ratio of the data between the osdb _database file and the sao_bin text file is relatively small.

In the embodiment of the application, in the process of compressing data, a proper compression algorithm is selected to consider the compression ratio of the data, the performance requirement of the flash memory device and the life cycle of the data, for example: whether over-compression of the data affects I/O performance, further for example: for data (data with a short life cycle) which are frequently read, the data can be decompressed faster during reading by adopting lightweight compression, for example, in a Cassandra scene, the pre-write log is frequently used, compression of the pre-write log is not needed, the SSTable is used as a data structure in the Cassandra, the SSTable is divided into a plurality of layers, the heat of the data stored in each layer of SSTable is different, for example, the heat of the data in the 0 th layer of SSTable is highest, the heat of the data in the n layer of SSTable (the last layer of SSTable) is lowest, a lightweight compression algorithm is adopted for the data in the 0 th layer of SSTable, and a heavyweight compression algorithm is adopted for the data in the n layer of SSTable, wherein the higher the heat of the data is more times the data is accessed, the shorter the life cycle of the data is, the lower the times the data is accessed are less, and the life cycle of the data is longer.

In the embodiment of the application, a lightweight compression algorithm is adopted for the data with high heat, the compression rate of the data with high heat is low, and a heavyweight compression algorithm is adopted for the data with low heat, so that the compression rate of the data with low heat is high, namely the life cycle corresponding to the data with low compression rate is lower, and the life cycle corresponding to the data with high compression rate is longer, thereby indirectly representing the life cycle of the data through the compression rate and further carrying out shunt management on the application data according to the compression rate of the data.

Step S5012: compressing the application data according to a compression algorithm to obtain the compression rate of the application data;

specifically, the application data is compressed according to a compression algorithm determined by the data type of the application data, so as to obtain the compression rate of the compressed application data.

In the embodiment of the application, different compression algorithms are adopted for the data with different data types, so that the extra delay of the compression algorithm on a write key path can be avoided, and the influence of the compression algorithm on the write performance of the flash memory device is avoided.

Step S502: clustering the application data according to the compression rate of the application data to generate a plurality of software streams;

Specifically, according to the compression rate of the application data, the application data are clustered, namely, according to the compression rate of the application data, the application data with similar compression rate are divided into the same type of data by using a clustering algorithm to generate a plurality of software streams, wherein the clustering algorithm comprises a K-means algorithm, a DBSCAN algorithm, a hierarchical clustering algorithm, a spectral clustering algorithm, a collaborative filtering algorithm, a random forest algorithm, a Gaussian process regression clustering algorithm and other clustering algorithms, and the number (the number of types) of the software streams output by the clustering algorithm can be set according to specific requirements. It should be noted that whether the compression rates are similar or not may be determined by whether the compression rates of the application data are within the same preset compression rate range, for example, the preset compression rate range is [0.1,0.2], if the compression rate of the text file is 0.17 and the compression rate of the picture is 0.1, the compression rates of the text file and the picture are both within the preset compression rate range, and the compression rates of the text file and the picture are similar.

As shown in fig. 7, the compression rate of each data block in the first database file (such as nci _database file) is substantially distributed around 0.17, the compression rate of the web page file (such as webster web page) is substantially distributed around 0.4, the compression rate of the text file (such as dickens _txt text file) is substantially distributed around 0.5, the compression rate of each data block in the second database file (such as osdb _database file) is substantially distributed around 0.74, the compression rate of the image file (such as xray _image picture) is substantially distributed around 0.82, and the compression rate of the binary file (such as sao_bin binary file) is substantially distributed around 0.85.

As can be seen from the above data display, the data compression rates of the data of the same type of file are basically the same, the data of the same type of file are clustered in the same software stream, and the data compression rates of the image file, the data compression rates of the data blocks in the second database file and the data compression rates of the binary file in fig. 7 are all higher and the compression rates are similar, i.e. the corresponding life cycles of the image file, the second database file and the binary file are closer, and the image file, the second database file and the binary file are clustered in the same software stream.

In the embodiment of the application, the data with different compression rates are clustered through a clustering algorithm to gather the data with the same file type in the same software flow, the data with different file types in different software flows, the application data with a relatively close life cycle in the same software flow and the application data with a relatively far life cycle in different software flows, so as to realize multi-flow identification of the flash memory device.

Step S503: estimating a lifecycle of each software flow;

In the embodiment of the application, the compression rate is only an indirect representation of the life cycle of the application data, and the compression rate of the application data corresponding to different software flows is larger, the life cycles corresponding to the software flows with larger compression rate differences are likely to be closer, at this time, the software flows with closer life cycles are dispersed into different hardware flows, which can cause the waste of multi-flow resources, therefore, the life cycle of each software flow needs to be estimated, so that the software flows with closer life cycles are mapped into the same hardware flow, and the multi-flow resources are saved. It should be noted that the number of hardware streams is limited to the flash memory device, and the number of hardware streams is not excessive.

In the embodiment of the application, each software stream corresponds to at least one logic block address, the logic block address comprises a plurality of logic pages, and data corresponding to the software stream is stored in the logic pages.

Specifically, the time interval of writing the logical page twice of each logical page is recorded, the life cycle of each logical page is estimated according to the time interval corresponding to each logical page, and the average life cycle of all the logical pages corresponding to each software stream is calculated to estimate the life cycle corresponding to each software stream.

Referring to fig. 8 again, fig. 8 is a schematic diagram of the refinement flow of step S503 in fig. 5;

as shown in fig. 8, this step S503 includes:

step S5031: estimating the life cycle of all logic pages corresponding to each software stream;

Specifically, the time interval of two adjacent write operations closest to each logical page is recorded, and the life cycle of all the logical pages corresponding to each software stream is estimated according to the time interval of two adjacent write operations closest to each logical page.

Referring to fig. 9 again, fig. 9 is a schematic diagram of the refinement process of step S5031 in fig. 8;

as shown in fig. 9, this step S5031 includes:

step S5311: counting the time stamp of each logic page access, wherein the time stamp comprises a first time stamp and a second time stamp, and the first time stamp and the second time stamp are adjacent time stamps;

Specifically, the time stamp of each logic page access is counted, namely, the time stamp of each logic page for writing operation is recorded, the time stamp of the last two adjacent times for writing operation is taken, the time stamp comprises a first time stamp and a second time stamp, the first time stamp and the second time stamp are adjacent time stamps, and the first time stamp is smaller than the second time stamp.

Step S5312: calculating a difference between the first timestamp and the second timestamp to determine a lifecycle of each logical page;

Specifically, the difference between the first timestamp and the second timestamp is calculated, that is, the value obtained by subtracting the first timestamp from the second timestamp is the life cycle of the logical page, and the life cycle of each logical page can be determined by using the calculation method.

Step S5032: calculating the average value of the life cycles of all the logical pages corresponding to each software stream to determine the life cycle of each software stream;

specifically, an average value of the life cycles of all the logical pages corresponding to each software flow is calculated, the life cycles of all the logical pages corresponding to each software flow are added to obtain a total life cycle, the total life cycle is divided by the number of pages of the logical pages corresponding to the software flow, and the average value of the life cycles of all the logical pages is obtained, wherein the average value is the life cycle of the software flow.

It should be noted that the lifecycle of the software flow may also be determined by other methods, which are not limited herein.

Step S504: clustering the software streams according to the life cycle of each software stream to establish a first mapping relation between each software stream and the hardware stream;

In the embodiment of the present application, each hardware flow corresponds to a preset life cycle one by one, for example: assuming that 3 hardware flows exist, a storage position corresponding to the 1 st hardware flow is used for storing data with a life cycle of 1 day, a storage position corresponding to the 2 nd hardware flow is used for storing data with a life cycle of 7 days, a storage position corresponding to the 3 rd hardware flow is used for storing data with a life cycle of one month, before a first mapping relation between each software flow and the hardware flow is established, whether the life cycle of the current software flow is within a preset period corresponding to the current hardware flow or not needs to be judged, and if the life cycle of the current software flow is 5 days, the current software flow is mapped into the 2 nd hardware flow.

In the embodiment of the present application, each hardware flow corresponds to a hardware flow number, for example: there are three hardware streams numbered 0, 1, 2, respectively. It should be noted that, the preset life cycle corresponding to the hardware flow number is not affected by the ordering of the hardware flow numbers, for example: the preset life cycle of the hardware stream number 1 may be greater than the preset life cycle of the hardware stream number 0, or the preset life cycle of the hardware stream number 1 may be less than the preset life cycle of the hardware stream number 0, which is specifically set according to the actual requirement, and is not limited herein.

Specifically, according to the life cycle of each software flow, the software flows are clustered through a clustering algorithm, whether the life cycle of each software flow is in a preset period corresponding to the hardware flow is judged, if so, a first mapping relation between each software flow and the hardware flow is established, namely, each software flow corresponds to a hardware flow number corresponding to one hardware flow, for example: assuming that there are currently 10 software streams, 4 hardware streams, with the lifecycle of the software streams 1-4 being within the preset lifecycle of the hardware stream with hardware stream number 0, the software streams 1-4 are mapped into the hardware stream with hardware stream number 0. Wherein the number of clusters of the clustering algorithm is defined according to the number of hardware streams, for example: currently there are 8 hardware flows, and the clustering number of the clustering algorithm is set to 8.

In the embodiment of the application, the first mapping relation between each software flow and the hardware flow is established, so that the software flows with the relatively close life cycle are gathered in the same hardware flow, and one hardware flow corresponds to a plurality of software flows.

In the embodiment of the application, the life cycle of each software stream is estimated, so that the software streams with larger compression ratio difference and closer life cycle are mapped into the same hardware stream, the occupation of multi-stream resources is avoided, and meanwhile, a plurality of software streams are mapped into one hardware stream, so that the problem of mismatching of the number of the software streams and the number of the hardware streams is solved.

Referring to fig. 10 again, fig. 10 is a schematic diagram of the refinement flow of step S504 in fig. 5;

as shown in fig. 10, the S504 includes:

step S5041: judging whether the life cycle corresponding to the current software flow is within the preset life cycle corresponding to the current hardware flow;

Specifically, it is determined whether the life cycle corresponding to the current software flow is within the preset life cycle corresponding to the current hardware flow, if so, the step SS5042 is skipped, otherwise, the step S5044 is skipped.

Step S5042: mapping the software stream to a hardware stream corresponding to a preset life cycle to determine a first mapping relation between the software stream and the hardware stream;

specifically, if the life cycle corresponding to the current software flow is within the preset life cycle corresponding to the current hardware flow, the current software flow is mapped to the current hardware flow through a clustering algorithm to determine the mapping relationship between the current software flow and the current hardware flow, and the step S5043 is skipped.

Step S5043: move to the next software flow;

Specifically, after the mapping relationship between the current software flow and the current hardware flow has been determined, the process moves to the next software flow, and step S5041 is continued until all the software flows are clustered into corresponding hardware flows.

Step S5044: the current software flow is not mapped;

Specifically, if the life cycle corresponding to the current software flow is not within the preset life cycle corresponding to the current hardware flow, the process goes to step S5045.

Step S5045: move to the next hardware flow;

specifically, it is determined that the life cycle corresponding to the current software flow is not within the preset life cycle corresponding to the current hardware flow, and the process moves to the next hardware flow, and step S5041 is continued until all the software flows are clustered into corresponding hardware flows.

In the embodiment of the application, the multi-stream management of the application data is realized by establishing the first mapping relation between the software stream and the hardware stream.

Step S505: according to the first mapping relation, storing data corresponding to the software flow to a storage position corresponding to the hardware flow;

Specifically, each hardware flow corresponds to one super block, and according to a first mapping relation between the software flow and the hardware flow, data corresponding to the software flow is stored in a storage position corresponding to the hardware flow, namely, the data corresponding to the software flow is stored in the super block corresponding to the hardware flow.

In the embodiment of the application, the data with similar compression ratio are clustered to generate a plurality of software streams, the life cycle of each software stream is estimated, so that the software streams with similar life cycles are mapped into the same hardware stream, the data corresponding to the software streams are stored in the storage positions corresponding to the hardware streams, the application data with similar life cycles can be stored in the storage positions corresponding to the hardware streams, the moving times of the data on the flash memory device can be reduced, and the write amplification problem of the flash memory device is optimized.

In the embodiment of the application, the application data of the host is subjected to split management and is directly realized in the flash memory device, the application program of the host is not required to be relied on, and the host is only responsible for transmitting the data, so that the effect of transparency of the host is achieved.

Referring to fig. 11 again, fig. 11 is a schematic flow chart of recovering valid data in a physical page according to an embodiment of the present application;

as shown in fig. 11, reclaiming valid data in a physical page includes:

step S1101: recovering effective data in a physical page corresponding to a storage position corresponding to the hardware stream;

In the embodiment of the application, when the garbage collection mechanism in the flash memory device is started, the unused storage space is automatically cleaned, so that the continuity and the usability of the storage space are ensured.

In the embodiment of the application, in the garbage collection process, a small part of data with longer life cycle and effectiveness may still exist in the storage position corresponding to the hardware flow, and the data with longer life cycle and effectiveness in the storage position corresponding to the hardware flow needs to be moved to another storage position to be erased.

Specifically, the storage position corresponding to the hardware flow is a super block, the super block is composed of a plurality of physical blocks, each physical block comprises a plurality of physical pages, the physical pages are used for storing data, and the effective data in the storage position corresponding to the hardware flow is recovered, namely the effective data in the physical pages corresponding to the storage position corresponding to the hardware flow is recovered.

Referring to fig. 12 again, fig. 12 is a flowchart illustrating a process of establishing a second mapping relationship between a hardware flow and a preset hardware flow according to an embodiment of the present application;

as shown in fig. 12, establishing a second mapping relationship between a hardware flow and a preset hardware flow includes:

Step S1201: establishing a second mapping relation between each hardware flow and a preset hardware flow, wherein the preset hardware flow corresponds to a preset storage position;

Specifically, before recovering the valid data in the physical page corresponding to the storage position corresponding to the hardware flow, a second mapping relation between each hardware flow and a preset hardware flow is pre-established, and the preset hardware flow is used for storing the valid data with a longer life cycle in the hardware flow into the preset storage position corresponding to the preset hardware flow.

For example, for a physical page of a normal hardware stream, when the write back frequency exceeds a threshold, data is written into hardware stream 0 with hardware stream number 0, where hardware stream 0 is used to store "colder" data, and if the physical page of hardware stream 0 is written back again at garbage collection, the data in the physical page is written into hardware stream 1 with hardware stream number 1, where hardware stream 1 is used to store "colder" data. If the physical page of hardware stream 1 is written back again at garbage collection, the data is still written back to hardware stream 1; if the physical page of hardware stream 1 is overwritten, new data will be written to dedicated hardware stream 0, and if the physical page of hardware stream 0 is overwritten, new data will be written to the original hardware stream, i.e., the hardware stream given by the second aggregation module.

By storing the cold data in the preset storage position corresponding to the preset hardware flow, the cold data is data with a longer life cycle or data which is not frequently used.

Referring to fig. 13 again, fig. 13 is a schematic diagram of a refinement flow chart of step S1001 of fig. 10;

As shown in fig. 13, this step S1101 includes:

step S1111: counting feedback information of physical pages corresponding to storage positions corresponding to each hardware flow;

Specifically, feedback information of the physical pages corresponding to the storage positions corresponding to each hardware stream is counted, wherein the feedback information comprises the write-back frequency of the physical pages and the hardware stream numbers of the physical pages, and the write-back frequency of the physical pages represents the use frequency of the physical pages in write operation and erase operation. Where write back refers to the data that is still valid for a physical page when garbage is reclaimed, the valid data needs to be written to a new location.

In the embodiment of the present application, the improvement effect of the write amplification of each hardware stream may be evaluated according to the feedback information of the physical page, for example: the write-back frequency determined by the write-back frequency of one hardware stream is larger than or equal to the frequency threshold value, which means that the data with longer life cycle and effectiveness exists in the hardware stream, and the data with longer life cycle and effectiveness in the hardware stream needs to be moved into a preset hardware stream so as to improve the write amplification problem of the hardware stream.

Step S1112: according to the feedback information, moving the effective data in the storage position to a preset storage position;

specifically, the feedback information includes a write-back frequency of the physical page and a hardware stream number of the physical page, and whether to move the valid data in the physical page to a preset storage position can be judged according to the write-back frequency of the physical page, or whether to move the valid data in the physical page to the preset storage position according to a state machine can be judged according to the hardware stream number of the physical page.

Referring to fig. 14 again, fig. 14 is a detailed flowchart of step S1112 of fig. 13;

as shown in fig. 14, this step S1112 includes:

Step S1121: determining the write-back frequency of the physical page in a period of time according to the write-back frequency of the physical page, and moving the effective data in the physical page corresponding to the storage position to a preset storage position according to the write-back frequency;

Specifically, according to the write-back frequency of the physical page, the write-back frequency of the physical page in a period of time, that is, the number of times the physical page is referred to in a period of time, according to the write-back frequency, that is, according to the number of times the physical page is referred to in a period of time, whether the number of times the physical page is referred to is greater than or equal to a frequency threshold value is judged, if the number of times the physical page is referred to is greater than or equal to the frequency threshold value, the effective data in the physical page is remapped to the same hardware stream when garbage is recovered, and if the number of times the physical page is referred to is less than the frequency threshold value, the effective data of the physical page is remapped to a preset storage position corresponding to a preset hardware stream, wherein the preset storage position comprises a physical block, the physical block comprises a plurality of physical pages, and the effective data of the physical page is moved to the physical page in the physical block. Wherein, a period of time refers to a preset time threshold, which is set according to specific needs, for example: values of 0.5s, 1s, etc. are set, and are not limited herein.

In the embodiment of the application, the write amplification improvement effect of the flash memory device can be effectively evaluated by counting the feedback information of the physical page.

Step S1122: according to the hardware stream number of the physical page, moving the effective data in the storage position to a preset storage position according to a state machine;

In the embodiment of the application, the state machine is used for recording the states of the physical pages, wherein the states of the physical pages comprise an idle state, a use state and a recovery state.

Specifically, according to the hardware stream number of the physical page, the state of the physical page in the super block corresponding to the hardware stream number is searched in a state machine, and if the state of the physical page is the recovery state, the valid data in the physical page corresponding to the recovery state in the storage position is moved to a preset storage position.

Referring to fig. 15 again, fig. 15 is a schematic diagram of the refinement flow of step S1121 in fig. 14;

as shown in fig. 15, this step S1121 includes:

step S1123: judging whether the write-back frequency of the physical page is larger than or equal to a frequency threshold value;

Specifically, it is determined whether the write back frequency of the physical page is greater than or equal to the frequency threshold, if the write back frequency of the physical page is greater than or equal to the frequency threshold, the process goes to step S1124, otherwise, the process goes to step S1125.

Step S1124: according to the second mapping relation, moving the effective data in the physical page corresponding to the storage position to a preset storage position;

Specifically, for a physical page of a common hardware stream, when the number of write backs exceeds a threshold, data is written into a dedicated hardware stream 0 with a hardware stream number of 0, where the hardware stream 0 is used for storing "colder" data, and if the physical page of the hardware stream 0 is written back again during garbage collection, the data is written into a hardware stream 1 with a hardware stream number of 1, where the hardware stream 1 is used for storing "colder" data. If the physical page of the special hardware stream 1 is written back again when garbage is recovered, the data is still written back to the special hardware stream 1; if the physical page of hardware stream 1 is overwritten, new data will be written to dedicated hardware stream 0, and if the physical page of hardware stream 0 is overwritten, new data will be written to the original hardware stream, i.e., the hardware stream given by the second aggregation module.

It will be understood that "colder" data refers to data that is not frequently used but that is occasionally used, and "very cold" data refers to data that is not substantially used. For example: the data may be determined to be first cold data or second cold data according to a time interval between two adjacent writing operations of each logical page, where the first cold data is "colder" data and the second cold data is "colder" data, such as: and if the time interval exceeds the first time threshold, determining the data as first cold data, and further, if the time interval exceeds the second time threshold, determining the data as second cold data, wherein the first time threshold is smaller than the second time threshold.

Step S1125: the data of the current physical page is not moved;

Specifically, if the write-back frequency of the physical page is less than the frequency threshold, which indicates that the physical page is often referenced, the data of the current physical page does not need to be moved.

In the embodiment of the application, the hardware stream is remapped to the preset hardware stream, so that the data with longer life cycle and effectiveness in the hardware stream is moved to the preset storage position corresponding to the preset hardware stream, and the moving times of the data with longer life cycle are reduced, thereby improving the write amplification problem of the flash memory device.

In the embodiment of the application, the data with similar compression ratio are clustered to generate a plurality of software streams, the life cycle of each software stream is estimated, so that the software streams with similar life cycles are mapped into the same hardware stream, and the data corresponding to the software streams are stored in the storage positions corresponding to the hardware streams.

Referring to fig. 16, fig. 16 is a schematic diagram illustrating an overall structure of a flash memory device according to an embodiment of the application;

as shown in fig. 16, the flash memory device 100 includes a compression module 101, a first clustering module 102, a life cycle estimation module 103, a second clustering module 104, a flash conversion module 105, a flash medium 106, and a remapping module 107.

The compression module 101 is connected to the first clustering module 102, and is configured to obtain application data, determine a compression algorithm corresponding to the application data according to a data type corresponding to the application data, and compress the application data according to the compression algorithm to obtain a compression rate of the application data. The compression algorithm comprises an LZ4 compression algorithm and an ZSTD, snappy, DEFLATE compression algorithm.

The first clustering module 102 is connected to the compression module 101 and the life cycle estimation module 103, and is configured to cluster application data according to a compression rate of the application data to generate a plurality of software flows, where the first clustering module 102 is configured to run a clustering algorithm, and the clustering algorithm includes a K-means algorithm, a DBSCAN algorithm, a hierarchical clustering algorithm, a spectral clustering algorithm, a collaborative filtering algorithm, a random forest algorithm, a gaussian process regression clustering algorithm, and other clustering algorithms.

Preferably, the first clustering module 102 includes a K-means clustering module, where K is the number of software streams. The K-means clustering module is used for running a K-means algorithm, setting the clustering number as the number of software streams, and clustering application data to generate K software streams. When the K-means clustering algorithm is initialized, the compression rate (central point) of the software stream is preset as an initial value, but in the running process of the K-means clustering algorithm, the compression rate (central point) of the software stream can adaptively change or update along with the compression rate of the logic page data of the software stream.

It will be appreciated that the number of software streams is determined based on the experience of the developer, and is typically a relatively large value, such as 32. The user data, i.e. the write data, is divided into 32 clusters, i.e. 32 software streams, based on the data compression rate.

The life cycle estimation module 103 is connected to the first clustering module 102 and the second clustering module 104, and is configured to estimate a life cycle of each software flow.

The second clustering module 104 is connected to the life cycle estimation module 103 and the flash memory conversion module 105, and is configured to cluster the software flows according to the life cycle of each software flow, so as to establish a first mapping relationship between each software flow and the hardware flow.

Preferably, the second clustering module 104 includes a K-means clustering module, where K is the number of hardware streams. The K-means clustering module is used for running a K-means algorithm, setting the clustering number as the number of hardware streams, and clustering application data to generate K hardware streams. When the K-means clustering algorithm is initialized, the life cycle (center point) of the hardware flow is preset as an initial value, but in the running process of the K-means clustering algorithm, the life cycle (center point) of the hardware flow can adaptively change or update along with the life cycle of the software flow.

It will be appreciated that the number of hardware streams is less than the number of software streams, for example: if the hardware actually supports only 8 streams, then 32 software streams still need to be clustered into the actual 8 hardware streams, the K-means algorithm is still adopted for the second clustering, and the clustering distance is calculated through the estimated stream life cycle. The second clustering output is 8 streams, matching the actual hardware.

The flash memory conversion module 105 is connected to the second aggregation module 104 and the flash memory medium 106, and is configured to store data corresponding to the software flow to a storage location corresponding to the hardware flow according to the first mapping relationship. The flash translation module 105 includes a flash translation layer, and the various functions of the flash translation module 105 are performed by the flash translation layer (FlashTranslation Layer, FTL).

The flash memory medium 106 is connected to the flash memory conversion module 105 and the remapping module 107, and is used for storing application data.

The remapping module 107 is connected to the flash conversion module 105 and the flash medium 106, and is configured to recycle the valid data in the storage location corresponding to the hardware stream, so as to move the valid data in the storage location to a preset storage location.

In the embodiment of the application, the compression rate of the application data is obtained; clustering the application data according to the compression rate of the application data to generate a plurality of software streams; estimating a lifecycle of each software flow; clustering the software streams according to the life cycle of each software stream to establish a first mapping relation between each software stream and the hardware stream; and storing the data corresponding to the software stream to a storage position corresponding to the hardware stream according to the first mapping relation.

Referring to fig. 17, fig. 17 is a schematic structural diagram of another flash memory device according to an embodiment of the present application;

As shown in fig. 17, the flash memory device 100 includes one or more processors 108 and a memory 109. In fig. 17, a processor 101 is taken as an example.

The processor 108 and the memory 109 may be connected by a bus or otherwise, for example in fig. 17.

A processor 108 for providing computing and control capabilities to control the flash memory device 100 to perform corresponding tasks, for example, to control the flash memory device 100 to perform a multi-stream management method of application data in any of the method embodiments described above, the method comprising: acquiring the compression rate of application data; clustering the application data according to the compression rate of the application data to generate a plurality of software streams; estimating a lifecycle of each software flow; clustering the software streams according to the life cycle of each software stream to establish a first mapping relation between each software stream and the hardware stream; and storing the data corresponding to the software stream to a storage position corresponding to the hardware stream according to the first mapping relation.

The processor 108 may be a general-purpose processor including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), a hardware chip, or any combination thereof; but also a digital signal processor (DIGITAL SIGNAL Processing, DSP), application specific integrated circuit (Application SpecificIntegrated Circuit, ASIC), programmable logic device (programmable logic device, PLD), or a combination thereof. The PLD may be a complex programmable logic device (complex programmable logic device, CPLD), a field-programmable gate array (FPGA) GATE ARRAY, generic array logic (GENERIC ARRAY logic, GAL), or any combination thereof.

The memory 109 is used as a non-transitory computer readable storage medium for storing non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the multi-stream management method of application data in the embodiment of the present application. The processor 108 may implement the multi-stream management method of application data in any of the method embodiments described above by running non-transitory software programs, instructions, and modules stored in the memory 109. In particular, memory 109 may include Volatile Memory (VM), such as random access memory (random access memory, RAM); memory 109 may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory (flash memory), hard disk (HARD DISK DRIVE, HDD) or solid-state disk (solid-state drive-STATE DRIVE, SSD) or other non-transitory solid state storage device; memory 109 may also include a combination of the above types of memory.

Memory 109 may include high-speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 109 may optionally include memory located remotely from processor 108, which may be connected to processor 108 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

One or more modules are stored in the memory 109 that, when executed by the one or more processors 108, perform the method of multi-stream management of application data in any of the method embodiments described above, for example, performing the various steps shown in fig. 5 described above.

Embodiments of the present application also provide a non-transitory computer readable storage medium, such as a memory including program code executable by a processor to perform the multi-stream management method of application data in the above embodiments. For example, the non-volatile computer readable storage medium may be a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a compact disc Read-Only Memory (CDROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

Embodiments of the present application also provide a computer program product comprising one or more program codes stored in a non-volatile computer-readable storage medium. The processor of the flash memory device reads the program code from the non-volatile computer readable storage medium, and the processor executes the program code to complete the method steps of the multi-stream management method for application data provided in the above-described embodiments.

It will be appreciated by those of ordinary skill in the art that all or part of the steps of implementing the above embodiments may be implemented by hardware, or may be implemented by program code related hardware, and the program may be stored in a non-volatile computer readable storage medium, where the storage medium may be a read only memory, a magnetic disk or optical disk, etc.

From the above description of embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus a general purpose hardware platform, or may be implemented by hardware. Based on such understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the related art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., and include several instructions for up to a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method of each embodiment or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and are not limiting; the technical features of the above embodiments or in the different embodiments may also be combined within the idea of the application, the steps may be implemented in any order, and there are many other variations of the different aspects of the application as above, which are not provided in details for the sake of brevity; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the application.

Claims

1. A method of multi-stream management of application data, the method comprising:

Acquiring the compression rate of the application data;

Estimating a lifecycle of each of the software flows;

clustering the software streams according to the life cycle of each software stream to establish a first mapping relation between each software stream and each hardware stream;

according to the first mapping relation, storing data corresponding to the software flow to a storage position corresponding to the hardware flow, wherein the storage position comprises a plurality of physical pages;

The method further comprises the steps of:

The recovery of the valid data in the physical page corresponding to the storage position corresponding to the hardware flow comprises the following steps:

counting feedback information of a physical page corresponding to a storage position corresponding to each hardware stream, wherein the feedback information comprises write-back frequency of the physical page and hardware stream numbers of the physical page;

According to the feedback information, moving the effective data in the storage position to a preset storage position;

And moving the effective data in the storage position to a preset storage position according to the feedback information, wherein the method comprises the following steps:

And/or the number of the groups of groups,

And moving the effective data in the storage position to a preset storage position according to a state machine according to the hardware stream number of the physical page.

2. The method of claim 1, wherein the obtaining the compression rate of the application data comprises:

and compressing the application data according to the compression algorithm to obtain the compression rate of the application data.

3. The method of claim 2, wherein each of the software streams corresponds to at least one logical block address, the logical block address comprising a plurality of logical pages, the data corresponding to the software streams being stored in the logical pages;

said estimating a lifecycle of each of said software flows comprising:

And calculating the average value of the life cycles of all the logic pages corresponding to each software stream to determine the life cycle of each software stream.

4. A method according to claim 3, wherein said estimating the lifecycle of all logical pages corresponding to each of said software flows comprises:

a difference between the first timestamp and the second timestamp is calculated to determine a lifecycle of each of the logical pages.

5. The method of claim 4, wherein each of the hardware flows corresponds to a predetermined life cycle, and the clustering the software flows according to the life cycle of each of the software flows to establish the first mapping relationship between the software flows and the hardware flows includes:

And if the life cycle of the software flow is within the preset life cycle, mapping the software flow to a hardware flow corresponding to the preset life cycle to determine a first mapping relation between the software flow and the hardware flow, wherein each hardware flow corresponds to a plurality of software flows.

6. The method of claim 1, wherein the step of determining the position of the substrate comprises,

Before the moving the valid data in the storage location to the preset storage location, the method further includes:

And establishing a second mapping relation between each hardware stream and a preset hardware stream, wherein the preset hardware stream corresponds to the preset storage position.

7. The method of claim 6, wherein the moving the valid data in the physical page corresponding to the storage location to the preset storage location according to the write-back frequency comprises:

And if the write-back frequency is greater than or equal to a frequency threshold, moving the effective data in the physical page corresponding to the storage position to a preset storage position according to the second mapping relation.

8. A flash memory device, comprising:

The compression module is used for determining a compression algorithm corresponding to the application data according to the data type corresponding to the application data, and compressing the application data according to the compression algorithm to obtain the compression rate of the application data;

A life cycle estimating module for estimating a life cycle of each of the software streams;

The second clustering module is used for clustering the software streams according to the life cycle of each software stream so as to establish a first mapping relation between each software stream and each hardware stream;

the flash memory conversion module is used for storing the data corresponding to the software stream to a storage position corresponding to the hardware stream according to the first mapping relation, and the storage position comprises a plurality of physical pages;

the remapping module is used for recycling the effective data in the storage position corresponding to the hardware flow so as to move the effective data in the storage position to a preset storage position;

The remapping module is further configured to recycle valid data in a physical page corresponding to a storage location corresponding to the hardware flow;

The remapping module is specifically configured to count feedback information of a physical page corresponding to a storage position corresponding to each hardware flow, where the feedback information includes a write-back frequency of the physical page and a hardware flow number where the physical page is located, and move valid data in the storage position to a preset storage position according to the feedback information;

The remapping module is specifically configured to determine a write-back frequency of the physical page in a period of time according to the write-back frequency of the physical page, and move the valid data in the physical page corresponding to the storage position to a preset storage position according to the write-back frequency; and/or, according to the hardware stream number of the physical page, moving the effective data in the storage position to a preset storage position according to a state machine;

and the flash memory medium is used for storing the application data.

9. A flash memory device, comprising:

at least one processor; and

A memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of multi-stream management of application data according to any one of claims 1-7.