CN117406923A - Data deleting and managing system based on log structure merging tree - Google Patents
Data deleting and managing system based on log structure merging tree Download PDFInfo
- Publication number
- CN117406923A CN117406923A CN202311332376.8A CN202311332376A CN117406923A CN 117406923 A CN117406923 A CN 117406923A CN 202311332376 A CN202311332376 A CN 202311332376A CN 117406923 A CN117406923 A CN 117406923A
- Authority
- CN
- China
- Prior art keywords
- data
- value
- log
- deduplication
- lsmt
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012986 modification Methods 0.000 claims abstract description 15
- 230000004048 modification Effects 0.000 claims abstract description 15
- 238000012545 processing Methods 0.000 claims abstract description 7
- 238000005056 compaction Methods 0.000 claims abstract description 5
- 238000000034 method Methods 0.000 claims description 29
- 230000008569 process Effects 0.000 claims description 24
- 230000026676 system process Effects 0.000 claims description 10
- 230000003321 amplification Effects 0.000 abstract description 4
- 238000012217 deletion Methods 0.000 abstract description 4
- 238000003199 nucleic acid amplification method Methods 0.000 abstract description 4
- 230000006870 function Effects 0.000 abstract description 3
- 238000013500 data storage Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 10
- 238000007726 management method Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000013523 data management Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
- G06F3/0641—De-duplication techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data deleting and managing system based on a log structure merging tree, which has two structures for managing data, wherein the structures are as follows: the log structure merging tree LSMT structure is used for storing initial data of the system, performing repeated deleting operation and processing operation requests of users; the value file table vFT structure is used to hold unrendered data that is deleted by the log structured merge tree LSMT structure but still referenced by the deduplicated data. The log structure merging tree LSMT performs double re-deletion, and one re-deletion is performed when the data on the memory is persisted to a disk; the secondary deduplication is performed during the compaction operation of the system on the plurality of ordered string tables sstables. According to the invention, through double erasure and fingerprint index storage in the disk, the data storage cost and the memory overhead are reduced; through the DedupVal index mode and vFT structure, the efficient query and modification functions after the deduplication are provided; the garbage collection of vFT can further reduce the disk space occupation and the read amplification of DedupVal.
Description
Technical Field
The invention relates to the field of data management, in particular to a data deduplication and management system based on a log structure merge tree.
Background
A Log Structured Merge Tree (LSMT) is a widely used index structure for persistent key-value storage, which caches all writing or modifying operations in memory first, then converts them into sequential writing on disk, fully utilizes the writing bandwidth of disk, and has good writing performance.
The data compression technology is that the same data is identified by fingerprint, and the stored same data is referenced to achieve the effect that only one part of the same data is stored on a disk, so that the storage cost and the network flow are reduced. It may further reduce the amount of data that needs to be stored compared to local compression.
In the information age, the data volume of each industry has a explosive growth trend, and reports show that the global data volume reaches 175ZB by 2025, and the data growth speed exceeds the hardware cost reduction speed. Key value storage is an important infrastructure that can be used as a storage engine for the bottom layer of a database. Therefore, how to perform data deduplication in the method to reduce hardware cost, and maintain the original characteristics of low latency and high throughput after deduplication is a problem to be solved by those skilled in the art.
Some existing deduplication methods designed in log-structured merge trees perform global deduplication on all key-value pairs, and physically separate keys and values. However, performing global deduplication requires storing all indexes of non-duplicate values in memory, with large storage overhead and index overhead; at the same time, key-value separation also causes that keys and values cannot be cooperatively managed, and additional garbage collection of values causes a great deal of expenditure and cannot provide efficient functions such as inquiry, updating and deletion. Therefore, in order to perform deduplication in the log-structured tree to save hardware costs while maintaining the low latency and high throughput characteristics of the original system, redesign is required according to the characteristics of the log-structured tree.
Disclosure of Invention
The invention aims to solve the problems in the prior art and provides a data deduplication method and a deduplication data management system which are more suitable for log-structured merge trees.
The technical scheme adopted for solving the technical problems is as follows: the utility model provides a data deleting and management system based on log structure merge tree, which is characterized in that the system data is KV data, KV data includes Key Key and Value, initial Key Key is OrgKey, initial Value is OrgVal, the system has two kinds of management data's structure, respectively:
the log structure merging tree LSMT structure is used for storing initial data of the system, performing repeated deleting operation and processing operation requests of users, wherein the operation requests of the users comprise writing requests, inquiry requests and modification requests; the deduplication operation refers to that for a plurality of data with a duplicate Value, one data is reserved as unrepeated data, other data is used as deduplicated data, the unrepeated data is not processed, the Value of the deduplicated data is replaced by a reference to the unrepeated data, and the number of times that the unrepeated data is referenced is represented by the number of times that the data is referenced DedupCnt;
the value file table vFT is configured to store an ordered string table SSTable in which the LSMT structure of the log-structured merge tree is deleted in the process of compaction but the number of times of file referencing, dedupRefCnt, is not zero, and the value file table vFT is configured to store a file of data as a value ordered string table vSSTable; the number of times of file quoted DedupRefCnt of the ordered string table SSTable is the sum of the number of times of data quoted DedupCnt of all unrendered data in the ordered string table SSTable.
Preferably, in the log structure merge tree LSMT structure, the implementation of the deduplication operation is as follows:
detecting whether repeated Value appears in KV data, if so, reserving one KV data as unreliated data and the other KV data as repeated data for a plurality of KV data with repeated Value; the un-deleted data is not processed, the Value of the deleted data is replaced by an OrgVal Value to a deleted Value DedupVal, and the deleted Value DedupVal contains index information of the Value of the un-deleted data;
the log-structured merge tree LSMT also caches the relevant index of fingerprint information in the deduplication process, and the fingerprint information cache is maintained by using a least recently used algorithm LRU.
Preferably, the log-structured merge tree LSMT performs double deduplication, which is respectively:
the data on the memory is subjected to one-time erasure when being persisted to the disk, specifically: the system performs primary re-deleting on the data of the immutable memory table im-memtable, stores the re-deleted data into a disk, generates an ordered string table SSTable, and stores related fingerprint information into the generated ordered string table SSTable;
the secondary repeating deleting is performed in the process of performing the compaction operation on a plurality of ordered string tables SSTable by the system, and specifically comprises the following steps: and reading out part of the ordered string tables SSTable, and performing secondary erasure on KV data according to the key values among the ordered string tables SSTable and the relevant fingerprint information stored in the ordered string tables SSTable.
Preferably, when the system processes a query request of a user, when the query request is a point query request, searching a corresponding Value in the log structure merging tree LSMT according to a Key Key sent by the user; if the read Value is the OrgVal Value, directly returning the Value; and if the read Value is the deduplicated Value DedupVal, reading the Value of the corresponding unrendered data according to the index information.
Preferably, when the system processes a query request of a user, when the query request is a range query request, searching a plurality of Value values corresponding to the Key range sent by the user in the log structure merging tree LSMT; if the read Value is the OrgVal Value, directly returning the Value; if the read Value is the deduplicated Value DedupVal, reading the Value of the corresponding unrendered data according to the index information; if a plurality of DedupVal are inquired, the DedupVal is put into a read request queue RQ, and the Value of the corresponding unrendered KV data is read by using multithreading.
Preferably, when the system processes the modification request of the user, if the modified Value is the OrgVal Value, determining how to modify according to the quoted times DedupCnt of the data; if the modified Value is DedupVal, directly modifying the data, and performing 1 subtracting operation on the data quoted by the DedupCnt of the unremoved data quoted before the data is modified; the modification request is completed in the compatibility process.
Preferably, the modification request includes an update request and a delete request.
Preferably, the value file table vFT combines the number of file references DedupRefCnt with the least recently used K-time algorithm LRU-K to cache data in a partial ordered string table vSSTable in memory, thereby accelerating data query.
Preferably, the value file table vFT performs garbage collection when the total size exceeds a preset upper limit, specifically: selecting N value ordered string tables vSSTable with the largest number of times of the file being referenced, performing garbage collection, generating a new value ordered string table vSSTable, and deleting the old value ordered string table vSSTable; the upper limit of the total size and N of the value file table vFT are both configurable values. The invention has the following beneficial effects:
1. by deleting the repeated data, the data volume is reduced, so that the number of times of comparison is reduced, the blockage of foreground writing is reduced, the writing performance is improved, and the hardware cost of data storage is reduced;
2. the index memory overhead of the data deduplication technology is reduced by storing fingerprint information required by deduplication in the SSTable;
3. through the deduplication technology in the comparison process, the data volume to be written is reduced, and then the write amplification of the system based on the log structure merging tree is reduced;
4. through the index mode of DedupVal and vFT structure, the efficient query and modification functions after the deduplication are provided; the garbage collection of vFT can further reduce disk space occupation while reducing the read amplification of DedupVal.
The present invention will be described in further detail with reference to the drawings and examples, but the present invention is not limited to the examples.
Drawings
FIG. 1 is a system block diagram of an embodiment of the present invention;
FIG. 2 is a schematic diagram of a two-time data deduplication process according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of processing a user query request according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating processing of a user modification request according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of vFT according to an embodiment of the present invention.
Detailed Description
Referring to fig. 1, a system structure diagram of an embodiment of the present invention includes two structures, i.e., a log structure merge tree LSMT and a value file table vFT, for managing files of different referenced cases; the file SSTable managed by the log structure merge tree LSMT and the file vssttable managed by the value file table vFT use the same format.
The memory of the LSMT comprises a buffer and a cache, the buffer part comprises a variable memory table memtable and an invariable memory table im-memtable, and the memory is used for bearing the writing and modifying requests of users; the cache part is used for accelerating the cache of the user query request, and also comprises an LRU FPcache used for caching the mapping from the Value to the FP fingerprint thereof (namely, the FP thereof can be quickly acquired through the Value) in the process of deleting the data, and the LRU FPcache can accelerate the deleting process.
The memory of the value file table vFT is also divided into two parts, and one part stores vFT metadata, including basic information such as the number of vSSTable and the like; the other part is vFT cache, which is used for caching part of data and accelerating the reading of vSSTable by a user.
Represented on the arrow of disk section from SSTable to vSSTable is the conversion condition of the file from log structured merge tree LSMT to value file table vFT: i.e., SSTable data is no longer owned by LSMT (embodied as refcnt=0) but is still referenced by the deduplicated data (embodied as file referenced number dedupefcnt > 0).
Referring to fig. 2, a schematic diagram of two data deduplication steps according to an embodiment of the present invention is shown.
The first deduplication is to delete im-memtable, and then save the FP-to-OrgKey key index associated with the im-memtable in a persisted SSTable. The system persists im-memtable into SSTable on the disk in the background, and in the process of establishing SSTable, the system deletes the in-table KV data again and stores relevant fingerprint information into the SSTable. Im-memable in LSMT cannot be modified by new requests.
The second deduplication is to deduplicate all selected SSTable in the process of performing compatibility on a plurality of SSTable in the background, integrate all FP index items in the process of deduplication, if one FP is related to a plurality of OrgKey, the process indicates that a plurality of OrgVal appear, only one part is reserved, and the process is reserved in the SSTable with the maximum total reference number DedupRefCnt
And the repeating deleting process caches the relevant index of the FP information in the memory, and the FP cache is maintained by using an LRU algorithm. The re-deleting process maintains the index buffer from Value to FP, reduces the fingerprint calculation times of OrgVal, and accelerates the re-deleting process. And the index cache decides the obsolete data item when the cache is full according to the access heat information of the index item.
The repeated deleting process detects a plurality of KV data which are repeated for the Value, one KV data is reserved as unredeleted data, and other KV data are reserved as repeated data; the un-deleted data is not processed, the Value of the deleted data is replaced by the OrgVal Value to be deleted Value DedupVal, and the deleted Value DedupVal is index information of the Value of the un-deleted data.
DedupVal holds an index information from which its OrgVal value can be found. Specifically, the structure of DedupVal is: unique fn+orgkey, where unique fn represents the globally unique serial number of each SSTable or vSSTable, while OrgKey is the Key of unrendered data that is repeated with its Value.
Referring to fig. 3, a schematic diagram of processing a user query according to an embodiment of the present invention is shown.
Specifically, when the system processes a point query request, the Value is searched according to the Key Key:
firstly searching in a cache part, directly returning if a Value corresponding to a Key Key is found, otherwise, continuing to search in an SSTable. When inquiring in the SSTable, if the Key Key is not found, returning a result that the Key does not exist, otherwise, reading a Value corresponding to the Key Key; if the Value is the OrgVal Value, directly returning, if the Value is the DedupVal Value, analyzing the Value, and searching the stored OrgKey in the corresponding file; the analysis specifically comprises the following steps: first, resolving fn, if resolved fn does not exist, searching according to OrgKey in vFT. Because fn in DedupVal is invalid after vSSTable is deleted, recovery through OrgKey in the vSSTable is needed, and in order to reduce the number of vSSTable to be searched, the vSSTable is searched according to the sequence of fn from large to small; if fn exists, reading out OrgValue corresponding to OrgKey directly in the corresponding file.
Specifically, for a range query request, values corresponding to all relevant keys are obtained in the LSMT, all the dupval are put into RQ, and each dupval is recovered from the LSMT or vFT by using one thread. It will be appreciated that to fully exploit the multi-core processor and SSTable read-only features, multiple threads are used for DedupVal recovery within RQ.
Referring to fig. 4, a schematic diagram of processing a user modification request according to an embodiment of the present invention is shown. The modification request comprises Value update of a certain KV pair or deletion of the KV pair, specifically: the system processes the modification request in the LSMT compatibility process, at this time, the number of references to the OrgKey at the corresponding OrgVal is reduced by 1, and whether the number of references to the OrgKey is 1 is checked to determine whether to delete the OrgKey.
Specifically, when the system processes the update request, if the request is to update OrgVal, checking the number of times DedupCnt of data to be referenced, and determining whether the data is retained in the original SSTable; if the update to DedupVal is requested, the update is directly performed, and the data of the unremoved data which is referenced before the update is subtracted by 1 by the reference number DedupCnt.
Specifically, when the system processes the delete request, a KV pair is written in the LSMT, where Value identifies that the Key is deleted. In the process of the compatibility, if the OrgVal is deleted, determining whether to keep the original SSTable according to the cited condition of the data; if DedupVal is deleted, the number of references DedupCnt to unremoved data referenced before it is deleted is decremented by 1.
Referring to fig. 5, which is a schematic diagram of a vFT structural design of an embodiment of the present invention, sstables in the system are initially managed by LSMT, and due to the compatibility process, part of sstables are deleted by LSMT, sstables in which DedupRefCnt is not zero are stored vFT, and for distinction from files in LSMT, management files thereof are called vssttable (value Sorted String Table, ordered string table storing original values), and the formats of both are the same. The valid data in vSSTable are KV pairs that are deleted or modified by LSMT, but still referenced by part of dupval.
For the query request of the user, the vSSTable has no locality, and vFT combines the DedupRefCnt and the LRU-K algorithm to cache partial data in the memory in order to accelerate the request completion speed.
The metadata (metadata) in vFT holds the total size of all vSSTable at present, the preset maximum file size upper limit, and the number of times each vSSTable is referenced. And vFT decides vcompaction time according to preset size limitation, and selects a plurality of vSSTable with the most references according to DedupRefCnt to recycle garbage during vcompaction, so as to generate new vSSTable, thereby reducing the number of vSSTable to be read for inquiry. Specifically, in vFT, according to the sequence of the times of being referenced, selecting a plurality of vSSTable read-outs with the most references, merging and sequencing in a similar way by adopting the compatibility, deleting Key value pairs which are not referenced, and generating a new vSSTable by combining the sequence of the times of being referenced and the keys. The number of the vSSTable selected for garbage collection can be manually configured, and can also be determined according to the current idle resource quantity and the vFT total size; the newly generated vSSTable uses the same global fn allocation as the files in LSMT, with fn for each vSSTable being unique.
Therefore, the data deleting and managing system based on the log structure merging tree is a log structure merging tree friendly data deleting method, combines the characteristics and the limitations of the log structure merging tree, designs a data deleting method which is more suitable for the log structure merging tree, can simultaneously ensure efficient data writing, inquiring, updating and deleting operations, reduces storage cost and write amplification, and provides a log structure merging tree friendly, complete and comprehensive data deleting mode and a deleted data managing mode.
The foregoing is only illustrative of the present invention and is not to be construed as limiting thereof, but rather as various modifications, equivalent arrangements, improvements, etc., within the spirit and principles of the present invention.
Claims (9)
1. The utility model provides a data deduplication and management system based on log structure merge tree which characterized in that, the data of system is KV data, and KV data includes Key Key and Value, and initial Key Key is the OrgKey, and initial Value is the orgVal, and the system has two kinds of management data's structure, and is respectively:
the log structure merging tree LSMT structure is used for storing initial data of the system, performing repeated deleting operation and processing operation requests of users, wherein the operation requests of the users comprise writing requests, inquiry requests and modification requests; the deduplication operation refers to that for a plurality of data with a duplicate Value, one data is reserved as unrepeated data, other data is used as deduplicated data, the unrepeated data is not processed, the Value of the deduplicated data is replaced by a reference to the unrepeated data, and the number of times that the unrepeated data is referenced is represented by the number of times that the data is referenced DedupCnt;
the value file table vFT is configured to store an ordered string table SSTable in which the LSMT structure of the log-structured merge tree is deleted in the process of compaction but the number of times of file referencing, dedupRefCnt, is not zero, and the value file table vFT is configured to store a file of data as a value ordered string table vSSTable; the number of times of file quoted DedupRefCnt of the ordered string table SSTable is the sum of the number of times of data quoted DedupCnt of all unrendered data in the ordered string table SSTable.
2. The system for data deduplication and management based on log-structured merge tree according to claim 1, wherein the implementation of deduplication operation in the log-structured merge tree LSMT structure is as follows:
detecting whether repeated Value appears in KV data, if so, reserving one KV data as unreliated data and the other KV data as repeated data for a plurality of KV data with repeated Value; the un-deleted data is not processed, the Value of the deleted data is replaced by an OrgVal Value to a deleted Value DedupVal, and the deleted Value DedupVal contains index information of the Value of the un-deleted data;
the log-structured merge tree LSMT also caches the relevant index of fingerprint information in the deduplication process, and the fingerprint information cache is maintained by using a least recently used algorithm LRU.
3. The system for data deduplication and management based on log-structured merge tree according to claim 2, wherein the log-structured merge tree LSMT performs deduplication twice, respectively:
the data on the memory is subjected to one-time erasure when being persisted to the disk, specifically: the system performs primary re-deleting on the data of the immutable memory table im-memtable, stores the re-deleted data into a disk, generates an ordered string table SSTable, and stores related fingerprint information into the generated ordered string table SSTable;
the secondary repeating deleting is performed in the process of performing the compaction operation on a plurality of ordered string tables SSTable by the system, and specifically comprises the following steps: and reading out part of the ordered string tables SSTable, and performing secondary erasure on KV data according to the key values among the ordered string tables SSTable and the relevant fingerprint information stored in the ordered string tables SSTable.
4. The system for data deduplication and management based on a log-structured merge tree according to claim 2, wherein when the system processes a query request of a user, when the query request is a point query request, a corresponding Value is searched in a log-structured merge tree LSMT according to a Key sent by the user; if the read Value is the OrgVal Value, directly returning the Value; and if the read Value is the deduplicated Value DedupVal, reading the Value of the corresponding unrendered data according to the index information.
5. The system for data deduplication and management based on a log-structured merge tree according to claim 2, wherein when the system processes a query request of a user, when the query request is a range query request, searching a plurality of Value values corresponding to the Key range sent by the user in the log-structured merge tree LSMT; if the read Value is the OrgVal Value, directly returning the Value; if the read Value is the deduplicated Value DedupVal, reading the Value of the corresponding unrendered data according to the index information; if a plurality of DedupVal are inquired, the DedupVal is put into a read request queue RQ, and the Value of the corresponding unrendered KV data is read by using multithreading.
6. The system for data deduplication and management based on a log-structured merge tree according to claim 2, wherein when the system processes a user's modification request, if the modified Value is the OrgVal Value, determining how to modify the data according to the number of times the data is referenced DedupCnt; if the modified Value is DedupVal, directly modifying the data, and performing 1 subtracting operation on the data quoted by the DedupCnt of the unremoved data quoted before the data is modified; the modification request is completed in the compatibility process.
7. The log structured merge tree based data deduplication and management system as recited in claim 6, wherein the modification request comprises an update request and a delete request.
8. The system of claim 1, wherein the value file table vFT combines the number of file references, ddroprefcnt, with the least recently used K-time algorithm LRU-K, to cache data in a partial ordered string table vSSTable in memory, to accelerate data queries.
9. The system for data deduplication and management based on a log structured merge tree of claim 1, wherein the value file table vFT performs garbage collection when the total size exceeds a preset upper limit, specifically: selecting N value ordered string tables vSSTable with the largest number of times of the file being referenced, performing garbage collection, generating a new value ordered string table vSSTable, and deleting the old value ordered string table vSSTable; the upper limit of the total size and N of the value file table vFT are both configurable values.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311332376.8A CN117406923A (en) | 2023-10-16 | 2023-10-16 | Data deleting and managing system based on log structure merging tree |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311332376.8A CN117406923A (en) | 2023-10-16 | 2023-10-16 | Data deleting and managing system based on log structure merging tree |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117406923A true CN117406923A (en) | 2024-01-16 |
Family
ID=89491855
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311332376.8A Pending CN117406923A (en) | 2023-10-16 | 2023-10-16 | Data deleting and managing system based on log structure merging tree |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117406923A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118643042A (en) * | 2024-08-15 | 2024-09-13 | 浙江智臾科技有限公司 | Main key storage engine and storage method supporting real-time updating and impromptu query |
-
2023
- 2023-10-16 CN CN202311332376.8A patent/CN117406923A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118643042A (en) * | 2024-08-15 | 2024-09-13 | 浙江智臾科技有限公司 | Main key storage engine and storage method supporting real-time updating and impromptu query |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110825748B (en) | High-performance and easily-expandable key value storage method by utilizing differentiated indexing mechanism | |
| CN111656341B (en) | Cache for efficient record lookup in LSM data structures | |
| US10620862B2 (en) | Efficient recovery of deduplication data for high capacity systems | |
| US10394757B2 (en) | Scalable chunk store for data deduplication | |
| US9043334B2 (en) | Method and system for accessing files on a storage system | |
| US7418544B2 (en) | Method and system for log structured relational database objects | |
| US20150089125A1 (en) | Framework for numa affinitized parallel query on in-memory objects within the rdbms | |
| US10521117B2 (en) | Unified table delta dictionary memory size and load time optimization | |
| WO2012083267A2 (en) | Garbage collection and hotspots relief for a data deduplication chunk store | |
| US10289709B2 (en) | Interleaved storage of dictionary blocks in a page chain | |
| CN117406923A (en) | Data deleting and managing system based on log structure merging tree | |
| CN101789027A (en) | Metadata management method based on DBMS and metadata server | |
| CN111984604B (en) | Method for reducing fragments of log-structured file system and flash memory storage system | |
| Amur et al. | Design of a write-optimized data store | |
| US20240281411A1 (en) | Defragmentation for log structured merge tree to improve read and write amplification | |
| US20230022756A1 (en) | Efficient in-memory multi-version concurrency control for a trie data structure based database | |
| CN114969069B (en) | Heat perception local updating method applied to key value storage system | |
| CN113553476A (en) | Key value storage method for reducing write pause by utilizing Hash | |
| US20230350810A1 (en) | In-memory hash entries and hashes used to improve key search operations for keys of a key value store | |
| WO2024020070A1 (en) | High density data storage based on log structured storage techniques | |
| CN109002400B (en) | Content-aware computer cache management system and method | |
| KR101806394B1 (en) | A data processing method having a structure of the cache index specified to the transaction in a mobile environment dbms | |
| CN118535578A (en) | Hash and LSM Tree-based hybrid index method and key value storage system | |
| US10558636B2 (en) | Index page with latch-free access | |
| CN112799590B (en) | Differentiated caching method for online main storage deduplication |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |