CN110147203A - A kind of file management method, device, electronic equipment and storage medium - Google Patents
A kind of file management method, device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN110147203A CN110147203A CN201910411298.8A CN201910411298A CN110147203A CN 110147203 A CN110147203 A CN 110147203A CN 201910411298 A CN201910411298 A CN 201910411298A CN 110147203 A CN110147203 A CN 110147203A
- Authority
- CN
- China
- Prior art keywords
- storage
- file
- storage region
- data
- small documents
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/134—Distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a kind of file management method, device, electronic equipment and storage mediums, wherein file management method includes: to obtain merging file to be written；The merging file is stored to the first storage region, and obtains the index data for each small documents in the merging file；By the index datastore to the second storage region, the reading and writing data performance of second storage region is higher than the reading and writing data performance of first storage region.The embodiment of the present invention can also improve the reading data performance of cloud storage system while saving memory space.
Description
Technical field
The present invention relates to technical field of data storage, more particularly to a kind of file management method, device, electronic equipment and
Storage medium.
Background technique
Cloud storage technology is a kind of technology for the storage of data cloud, and cloud storage system can receive client and pass through net
The data that network is sent, to be stored to data.
In cloud storage system, data usually store in the form of a file, specifically, the storage service in cloud storage system
File is carried out piecemeal according to fixed size by device, and each piece is known as a data block, and each data block is stored to existing in advance
In the storage space block divided on storage medium, that is, storage space block one data block of corresponding storage, and for size
One small documents of insufficient above-mentioned fixed size then need individually to occupy a storage space block in storage, when small documents number
When measuring excessive, lead to the problem of waste of storage space occur.
The prior art generallys use following scheme and avoids waste of storage space: large amount of small documents is merged into one big text
Part stores the index information of the big file and each small documents to the mechanical hard disk of storage server respectively.Reading small text
When part, the index information of small documents is first read, then data are read from big file based on index information.
But since small documents are large number of, the prior art will first read index information when reading small documents every time
Read data again, that is, mechanical hard disk will be read twice, therefore storage server single machine QPS (Query Per Second,
Query rate per second) up to 1000 or more, and the maximum QPS that mechanical hard disk can provide is typically only 90, leads to data read capability
It is unable to satisfy reading data demand, to cause the reading data degraded performance of storage server.
Summary of the invention
The embodiment of the present invention is designed to provide a kind of file management method, device, electronic equipment and storage medium, with
The reading data performance of storage server is improved while saving memory space.Specific technical solution is as follows:
In a first aspect, the embodiment of the invention provides a kind of file management method, applied to the management in cloud storage system
Server, the management server are used to manage more storage servers in the cloud storage system, the storage server
For storing data, which comprises
Merging file to be written is obtained, the file that merges is the file that multiple small documents obtain after merging, described
Small documents are the file that size is lower than preset threshold；
The merging file is stored to the first storage region, and obtains the rope for each small documents in the merging file
Argument evidence, wherein the storage location information of the small documents is carried in every index data；
By the index datastore to the second storage region, wherein the reading and writing data performance of second storage region
Higher than the reading and writing data performance of first storage region.
Optionally, described the step of storing the merging file to the first storage region, comprising:
The merging file is stored into the corresponding storage region of the first storage medium；
It is described by the index datastore to the second storage region the step of, comprising:
By the index datastore into the corresponding storage region of the second storage medium, wherein second storage is situated between
The reading and writing data performance of matter is higher than the reading and writing data performance of first storage medium.
Optionally, first storage medium is mechanical hard disk, and second storage medium is solid-state hard disk SSD.
Optionally, described the step of storing the merging file to the first storage region, comprising:
The merging file is stored to the first storage region of the first storage server, first storage server is
Wherein one in the more storage servers；
It is described by the index datastore to the second storage region the step of, comprising:
By the index datastore to the second storage region of first storage server.
Optionally, first storage region and second storage region are divided into the identical memory space of size
Block, for each storage space block for storing a data block, a data block includes: multiple storage catalogues, Yi Jicun
Storage space sets specified directory, and for storing data, record has default value to the storage catalogue under the storage location specified directory,
The different default values stores data to different storage catalogues for specified；
Described the step of storing the merging file to the first storage region, comprising:
If the size for merging file is not more than the size of the storage space block, the merging file is stored
Under the specified storage catalogue of data block corresponding to one of storage space block to first storage region；
If it is described merge file size be greater than the storage space block size, by the merging file store to
Under the specified storage catalogue of the corresponding data block of multiple storage space blocks of first storage region.
Optionally, the method also includes:
From second storage region, the index data of the small documents to read from first storage region is read；
According to the index data, storage location of the data in first storage region of the small documents is positioned；
According to the storage location positioned, read from the merging file that first storage region is stored
The small documents.
Optionally, described from second storage region, read the small documents to read from first storage region
The step of index data, comprising:
From the corresponding storage region of second storage medium, obtain to from the corresponding storage of first storage medium
The index data of the small documents read in region；
Described the step of reading the small documents from the merging file that first storage region is stored, packet
It includes:
From the merging file that the corresponding storage region of first storage medium is stored, the small text is read
Part.
Second aspect, the embodiment of the invention provides a kind of document management apparatus, applied to the management in cloud storage system
Server, the management server are used to manage more storage servers in the cloud storage system, the storage server
For storing data, described device includes:
Module is obtained, for obtaining merging file to be written, the merging file is that multiple small documents obtain after merging
The file arrived, the small documents are the file that size is lower than preset threshold；
First memory module for storing the merging file to the first storage region, and obtains and is directed to the merging
The index data of each small documents in file, wherein the storage position of the small documents is carried in every index data
Confidence breath；
Second memory module is used for the index datastore to the second storage region, wherein second memory block
The reading and writing data performance in domain is higher than the reading and writing data performance of first storage region.
Optionally, first memory module, is specifically used for:
The merging file is stored into the corresponding storage region of the first storage medium；
Second memory module, is specifically used for:
By the index datastore into the corresponding storage region of the second storage medium, wherein second storage is situated between
The reading and writing data performance of matter is higher than the reading and writing data performance of first storage medium.
Optionally, first storage medium is mechanical hard disk, and second storage medium is solid-state hard disk SSD.
Optionally, first memory module, is specifically used for:
The merging file is stored to the first storage region of the first storage server, first storage server is
Wherein one in the more storage servers；
Second memory module, is specifically used for:
By the index datastore to the second storage region of first storage server.
Optionally, first storage region and second storage region are divided into the identical memory space of size
Block, for each storage space block for storing a data block, a data block includes: multiple storage catalogues, Yi Jicun
Storage space sets specified directory, and for storing data, record has default value to the storage catalogue under the storage location specified directory,
The different default values stores data to different storage catalogues for specified；
First memory module, comprising:
First sub-module stored, if being not more than the size of the storage space block for the size for merging file,
The merging file is then stored into the specified of data block corresponding to one of storage space block to first storage region
Under storage catalogue；
Second sub-module stored, if being greater than the size of the storage space block for the size for merging file,
The merging file is stored to specified the depositing of the corresponding data block of multiple storage space blocks of first storage region
It stores up under catalogue.
Optionally, described device further include:
First read module, for reading small to be read from first storage region from second storage region
The index data of file；
Locating module, for positioning the data of the small documents in first storage region according to the index data
Storage location；
Second read module, for being stored from first storage region according to the storage location positioned
The small documents are read in the merging file.
Optionally, first read module, is specifically used for:
From the corresponding storage region of second storage medium, obtain to from the corresponding storage of first storage medium
The index data of the small documents read in region；
Second read module, is specifically used for:
From the merging file that the corresponding storage region of first storage medium is stored, the small text is read
Part.
The third aspect, the embodiment of the invention provides a kind of electronic equipment, including processor and machine readable storage medium,
The machine readable storage medium is stored with the machine-executable instruction that can be executed by the processor, and the processor executes
The machine-executable instruction is to realize the method and step of file management method that above-mentioned first aspect provides.
Fourth aspect, the embodiment of the invention provides a kind of computer readable storage medium, the computer-readable storage
Dielectric memory contains computer program, when the computer program is executed by processor, realizes the text that above-mentioned first aspect provides
The method and step of part management method.
5th aspect, the embodiment of the invention also provides a kind of computer program products comprising instruction, when it is being calculated
When being run on machine, so that computer executes the method and step for the file management method that above-mentioned first aspect provides.
6th aspect, the embodiment of the invention also provides a kind of computer programs, when run on a computer, so that
Computer executes the method and step for the file management method that above-mentioned first aspect provides.
A kind of file management method, device, electronic equipment and storage medium provided in an embodiment of the present invention, it is to be written obtaining
It after the merging file entered, is stored by the way that file will be merged to the first storage region, and obtained for each small documents in merging file
Index data, then by index datastore to the second storage region, since the reading and writing data performance of the second storage region is higher than
The reading and writing data performance of first storage region, therefore when reading small documents, the high read-write property of the second storage region can be utilized
The index data of each small documents can be more rapidly read, and then according to read index data, from the first storage region institute
Each small documents are read in the merging file of storage, to avoid reading data degraded performance existing for existing file management method
Problem can also improve the reading data performance of cloud storage system while saving memory space.Certainly, implement of the invention
Any product or method must be not necessarily required to reach all the above advantage simultaneously.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with
It obtains other drawings based on these drawings.
Fig. 1 is a kind of flow diagram of file management method provided in an embodiment of the present invention；
Fig. 2 is the schematic diagram of the data management topological structure in the cloud storage system of the embodiment of the present invention；
Fig. 3 is the flow diagram of another file management method provided in an embodiment of the present invention；
Fig. 4 is the flow diagram of another file management method provided in an embodiment of the present invention；
Fig. 5 is the flow diagram of the 4th kind of file management method provided in an embodiment of the present invention；
Fig. 6 is the memory topology structural schematic diagram of the storage server in the embodiment of the present invention；
Fig. 7 is the storage content schematic diagram of data block in the embodiment of the present invention；
Fig. 8 is the flow diagram of the 5th kind of file management method provided in an embodiment of the present invention；
Fig. 9 is a kind of structural schematic diagram of document management apparatus provided in an embodiment of the present invention；
Figure 10 is the structural schematic diagram of the first memory module in the embodiment of the present invention；
Figure 11 is the structural schematic diagram of another document management apparatus provided in an embodiment of the present invention；
Figure 12 is the structural schematic diagram of a kind of electronic equipment provided in an embodiment of the present invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall within the protection scope of the present invention.
Existing file management method can also usually use following scheme:
For file to be written, judge whether file size is less than preset threshold value, if this document is not more than threshold value,
Then this document can directly be stored into the storage server based on SSD (Solid State Disk, solid state hard disk)；If
This document is greater than threshold value, then can store this document into the storage server based on mechanical hard disk.But this manager
Method makes all small documents be both needed to storage into SSD, and SSD is at high price at present, and it is high to will lead to cloud storage system operation cost.
Alternatively, the caching of default size can be distributed for this document in memory if this document is not more than threshold value, it will
This document write-in caching；If this document is greater than threshold value, this document can be stored to the storage service based on mechanical hard disk
In device.But this management method will will lead to loss of data when mechanical disorder since small documents are stored in memory.
Embodiment of the method 1
Have in view of that, as shown in Figure 1, the embodiment of the present invention provides firstly a kind of file management method, this method can be with
Applied to the management server in cloud storage system, wherein management server can be used for managing more in cloud storage system
Storage server, for storing data, this method may comprise steps of storage server:
S101 obtains merging file to be written.
In the embodiment of the present invention, as shown in Fig. 2, for the schematic diagram of the data management topological structure in cloud storage system,
In, it may include multiple storage clusters in a cloud storage system, a management server be laid in each storage cluster (i.e.
Nameserver) and several (N platform, N be greater than 2) storage server (i.e. dataserver), management server can be to upper
Several storage servers are stated uniformly to be managed, storage server for storing data, illustratively, when obtaining number to be written
According to rear, management server can control above-mentioned storage server and store to the data.
Merging file therein can refer to a big file for obtaining multiple small documents after merging treatment, can be with
Understand, the obtained size for merging file is bigger compared to each small documents, due to that can be deposited in the form of one file
Storage, therefore can be avoided a case where small documents deposit discontented storage space block generation, that is, avoid waste of storage space
Problem.
Also, the data for merging each small documents in file, which can be, to be saved in order into merging file, therefore more
Be conducive to traditional mechanical hard disk read-write (the random reading performance of traditional mechanical hard disk is far below continuous reading performance).
Also, it is stored not yet into storage server due to merging file at this time, it can be by above-mentioned merging file
Merging file referred to as to be written.
Small documents therein can refer to that size is lower than the file of preset threshold, for example, size of data is lower than 2KB
The file of (Kilobyte, kilobytes).Certainly, the developer of cloud storage system can be flexibly reasonable according to practical business demand
Ground is arranged above-mentioned threshold value, the embodiment of the present invention to the specific value of above-mentioned threshold value without limitation.Multiple small documents are merged into one
A process for merging file can be obtained by existing file mergences method, and the detailed process embodiment of the present invention is no longer superfluous
It states.
S102 will merge file and store to the first storage region, and obtains for the index for merging each small documents in file
Data.
Different storage regions can be set in a storage cluster, it is corresponding for different storage regions
Reading and writing data performance can be different, and therefore, the embodiment of the present invention can will merge file and store to the first storage region, and deposit
Chu Hou, obtains the index data of each small documents in the merging file, wherein a small documents are carried in every index data
Storage location information, that is, the storage location information of the corresponding small documents of an index data, storage location information is for remembering
Small documents are recorded in the storage location of the first storage region, for example, the identification number of the storage space block stored, and can read
Play the role of positioning when the small documents.
The process of the above-mentioned index data for obtaining each small documents can be obtained by existing file index information generation method
It arrives, the embodiment of the present invention repeats no more.
The reading and writing data performance of index datastore to the second storage region, the second storage region is higher than first by S103
The reading and writing data performance of storage region.
After obtaining index data, the embodiment of the present invention can be deposited index datastore to the second storage region, second
Storage area domain is the region different from the first storage region, also, the reading and writing data performance of the second storage region is higher than the first storage
The reading and writing data performance in region.By the way that index datastore into higher second storage region of readwrite performance, is being read in this way
When taking large amount of small documents, it can use the high readwrite performance of the second storage region, more rapidly read the index of each small documents
Data, and then each small documents are further read from merging file according to read index data, it can be avoided due to tradition
The problem of reading data degraded performance caused by mechanical hard disk performance limits.In addition, since index data is relative to file data
Required memory space is smaller, therefore, only can save index datastore in the second high storage region of reading and writing data performance
Cost.
A kind of file management method provided in an embodiment of the present invention, after obtaining merging file to be written, by that will close
And file is stored to the first storage region, and is obtained for the index data for merging each small documents in file, then by index data
It stores to the second storage region, since the reading and writing data performance of the second storage region is higher than the reading and writing data of the first storage region
Can, therefore when reading small documents, each small documents can be more rapidly read using the high readwrite performance of the second storage region
Index data read from the merging file that the first storage region is stored each small and then according to read index data
File is saving the same of memory space to avoid the problem that reading data degraded performance existing for existing file management method
When can also improve the reading data performance of cloud storage system.
Embodiment of the method 2
As shown in figure 3, this method may include following step the embodiment of the invention also provides a kind of file management method
It is rapid:
S201 obtains merging file to be written.
The step is identical as the step S101 of embodiment of the method 1, and details are not described herein for the embodiment of the present invention.
S202 will merge file and store into the corresponding storage region of the first storage medium, and obtain for merging file
In each small documents index data.
The step is similar to the step S102 of embodiment of the method 1, is different in, and the embodiment of the present invention can will merge text
Part is stored into the corresponding storage region of the first storage medium, that is, the first storage region is specifically as follows the first storage medium pair
The storage region answered.
As a kind of optional embodiment of the embodiment of the present invention, above-mentioned first storage medium for example can be mechanical hard
Disk, then the above-mentioned file that will merge stores the step into the corresponding storage region of the first storage medium, that is, can be that will merge text
Part is stored into the corresponding storage region of mechanical hard disk.
S203, by index datastore into the corresponding storage region of the second storage medium.
The step is similar to the step S103 of embodiment of the method 1, is different in, and the embodiment of the present invention can be by index number
According to storing into the corresponding storage region of the second storage medium, that is, the second storage region is specifically as follows the second storage medium pair
The storage region answered.
As a kind of optional embodiment of the embodiment of the present invention, above-mentioned second storage medium for example can be SSD, then on
State the step by index datastore into the corresponding storage region of the second storage medium, that is, can be by index datastore
Into the corresponding storage region of SSD.
It is appreciated that the reading and writing data performance of the second storage medium is higher than the reading and writing data performance of the first storage medium, because
, when reading small documents, the embodiment of the present invention can read the index data of each small documents more quickly for this.
A kind of file management method provided in an embodiment of the present invention, since the reading and writing data performance of the second storage medium is higher than
The reading and writing data performance of first storage medium, therefore when reading small documents, the high read-write property of the second storage medium can be utilized
The index data of each small documents can be more rapidly read, and then according to read index data, from the first storage region institute
Each small documents are read in the merging file of storage, to further increase the reading data performance of cloud storage system.
Embodiment of the method 3
As shown in figure 4, this method may include following step the embodiment of the invention also provides a kind of file management method
It is rapid:
S301 obtains merging file to be written.
The step is identical as the step S101 of embodiment of the method 1, and details are not described herein for the embodiment of the present invention.
S302 will merge file and store to the first storage region of the first storage server.
The step is similar to the step S102 of embodiment of the method 1, is different in, in the embodiment of the present invention, management service
Device can will merge file and store in the wherein storage server into a storage cluster, specifically, can will merge
File is stored to the first storage region of the first storage server, and above-mentioned first storage server is any in storage cluster
One storage server.
Optionally, can will merge file to store to the corresponding memory block of the first storage medium of the first storage server
Domain, above-mentioned first storage region are specifically as follows storage region corresponding to the first storage medium in the first storage server.
Optionally, the first storage medium can be mechanical hard disk.
S303, by the second storage region of index datastore to the first storage server.
The step is similar to the step S103 of embodiment of the method 1, is different in, and the embodiment of the present invention can be by index number
It, specifically, can be by index datastore to first according in the wherein storage server stored into a storage cluster
Second storage region of storage server, above-mentioned first storage server are any one storage service in storage cluster
Device, the reading and writing data performance of the second storage region of the storage server are higher than the reading and writing data performance of the first storage region.
It optionally, can be by the corresponding memory block of the second storage medium of index datastore to the first storage server
Domain, above-mentioned second storage region are specifically as follows storage region corresponding to the second storage medium in the first storage server.
Optionally, the second storage medium can be SSD.
In compared with the prior art, small documents are all stored in SSD disk, or are stored in point of SSD disk composition
In cloth storage system, the cost of the embodiment of the present invention is lower, the reason is that SSD is more than mechanical hard disk price under identical capacity
It is expensive, in a storage server, if disk is all changed to SSD, compared to the file management method of the embodiment of the present invention,
Carrying cost at least rises 100 times.
As a kind of optional embodiment of the embodiment of the present invention, on the basis of process shown in Fig. 4, as shown in figure 5,
Before step S302, the file management method of the embodiment of the present invention can also include:
S301 ', judges whether the size for merging file is greater than the size of storage space block.
Then above-mentioned steps S302 can specifically include:
S3021 will merge file and store to first if the size for merging file is not more than the size of storage space block
Under the specified storage catalogue of data block corresponding to one of storage space block of storage region.
In the embodiment of the present invention, if the size for merging file is less than or equal to the size of storage space block, show only to need
Memory space provided by a storage space block is wanted, the memory space of merging file needs can be met.
Illustratively, above-mentioned merging file can be stored in the storage catalogue of data block corresponding to a storage space block
Under 0 or under storage catalogue 1.
Merging file is stored to first and is deposited if the size for merging file is greater than the size of storage space block by S3022
Under the specified storage catalogue of the corresponding data block of multiple storage space blocks in storage area domain.
In the embodiment of the present invention, if the size for merging file is greater than the size of storage space block, show that a storage is empty
Between memory space provided by block, be unable to satisfy memory space required for the merging file, therefore can be used multiple continuous
Storage space block the merging file is stored.
Likewise it is possible to store data to the specified storage mesh of above-mentioned multiple corresponding data blocks of storage space block
Under record.
As shown in fig. 6, for the memory topology structural schematic diagram of the storage server in the embodiment of the present invention, a storage clothes
It is usually provided with muti-piece disk in business device, forms disk array.The storage region of every piece of disk, can according to preset size,
Being divided into the identical storage space block of multiple sizes, (N number of, 2) N is greater than, each storage space block can be used for storing data.
In the embodiment of the present invention, illustratively, file to be written can indicate in the storage path of disk are as follows:/data/
Vols/vol1/phenix_data/00000000000000709106501, wherein data/, vols/, vol1/, phenix_
Data/ indicates store path step by step, and in the string number after phenix_data/, the first bit digital indicates data block
Type, 0 indicates three copy types；20 digit word bits after first indicate the identification number of storage space block, the i.e. memory space
Block a storage cluster unique identification, when data (for example, with idx format save data, alternatively, with dat format preservation
Data) store to data block corresponding to storage space block, management server can recorde the storage that the data are stored
The identification number of space block, so that index data is generated, in order to search and read later；Last two indicate the copy in three pairs
Unique identifying number in this, value is usually one in 0-2, the case where if there is 3 copies are greater than, is then likely larger than 2.
As shown in fig. 7, for the storage content schematic diagram of data block in the embodiment of the present invention, it can under each data block catalogue
To include different catalogues, these catalogues can be with are as follows: the first storage catalogue indicates the title of the catalogue with number 0；Second storage
Catalogue indicates the title of the catalogue with number 1；And a storage location specified directory, which is indicated with English CURRENT
Title.
Wherein, storage catalogue (i.e. storage catalogue 0 and storage catalogue 1) can be used for storing data, in each storage catalogue
Index data, the i.e. data of idx format can be saved, file data, the i.e. data of dat format can also be saved.
Storage location specified directory (i.e. CURRENT catalogue) stores data to which storage catalogue for specified,
That is, storing data to storage catalogue 0 for specified, or data are stored to storage catalogue 01.
The specified principle of storage location specified directory are as follows: number 0 or 1 can be saved under storage location specified directory, when
When the numerical value saved in storage location specified directory is 0, expressions storage catalogue 0 is current storage catalogue, can will be current
Data are stored to storage catalogue 0；When the numerical value saved in storage location specified directory is 1, indicate that storage catalogue 1 is current
Storage catalogue, current data can be stored to storage catalogue 1.
In the embodiment of the present invention, illustratively, index data can indicate in the storage path of disk are as follows:/data/
Phenix_idx/ { 0..35 }/00000000000000709106501/0/idx, wherein data/, phenix_idx/ are indicated
Store path step by step；The serial number of disk where { 0..35 } indicates file data corresponding to index data；phenix_idx/
In string number later, the first bit digital indicates the type of data block, and 0 indicates three copy types；20 digits after first
Word bit indicates the identification number of storage space block corresponding to dat data, i.e., the storage space block is in the unique of storage cluster
Mark；Last two indicate unique identifying number of the copy in three copies, and value is usually one in 0-2, if there is
The case where greater than 3 copies, then it is likely larger than 2.
Above-mentioned disk array, such as can be the disk array being made of 36 pieces of mechanical hard disks and 1 piece of SSD, wherein 36
Block mechanical hard disk is used to store the file of dat format, and 1 piece of SSD is used to store the file of idx format.Due to the embodiment of the present invention
Index data is stored respectively with file data, therefore, storage catalogue corresponding for data block each in mechanical hard disk can be only
File data is saved, storage catalogue corresponding for data block each in SSD can only save index data, optionally, above-mentioned rope
Argument evidence and file data can save in a binary format.
According to the specified principle of above-mentioned storage location specified directory, compression behaviour is being carried out to the data under currently stored catalogue
When making, the data compressed can be stored under another storage catalogue, while modifying storage location after squeeze operation is complete
Numerical value in specified directory to be directed toward another storage catalogue, therefore will not influence the data under currently stored catalogue.
A kind of file management method provided in an embodiment of the present invention, after obtaining merging file to be written, by that will close
And file stores the first storage region of the wherein storage server into cloud storage system, and obtains for merging file
In each small documents index data, then by the second storage region of index datastore to the storage server, due to the storage
The reading and writing data performance of second storage region of server is higher than the reading and writing data performance of the first storage region, therefore small reading
When file, the rope of each small documents can be more rapidly read using the high readwrite performance of second storage region of storage server
Argument evidence, and then according to read index data, each small documents are read from the merging file that the first storage region is stored,
It, can also while saving memory space to avoid the problem that reading data degraded performance existing for existing file management method
Enough improve the reading data performance of cloud storage system.
Embodiment of the method 4
The embodiment of the invention also provides a kind of file management methods, based on any of the above embodiments, such as Fig. 8 institute
Show, this method may comprise steps of:
S 401 reads the index data of the small documents to read from the first storage region from the second storage region.
According to previous embodiment it is found that the first storage region is stored with merging file, the second storage region is stored with small text
The index data of part therefore, can be with for some small documents to read from the first storage region in the embodiment of the present invention
The index data of the small documents is read, from the second storage region first to obtain the storage location information of the small documents.
It, can be from the corresponding storage region of the second storage medium as a kind of optional embodiment of the embodiment of the present invention
In, the index data of the small documents to read from the corresponding storage region of the first storage medium is obtained, the second storage medium can
Think SSD.
S402 positions storage location of the data in the first storage region of small documents according to index data.
After reading index data, it can be deposited according to storage location information therein, the data for positioning small documents first
The storage location in storage area domain.
Specifically, according to previous embodiment it is found that can recorde the identification number of storage space block in index data, because
Which this can be stored according to the file data that identification number positions small documents and deposit after determining the identification number of storage space block
It stores up in the corresponding data block of space block.
It, can be from the corresponding storage region institute of the first storage medium as a kind of optional embodiment of the embodiment of the present invention
In the merging file of storage, small documents are read, optionally, the first storage medium can be mechanical hard disk.
S403 reads small documents from the merging file that the first storage region is stored according to the storage location positioned.
The merging file that first storage region is stored is the file that multiple small documents obtain after merging, therefore merges text
It include the file data of each small documents in part, when to be stored in which storage space block corresponding for the file data for navigating to small documents
After in data block, the file data of small documents can be read from the data block, that is, the conjunction stored from the first storage region
And small documents are read in file.
A kind of file management method provided in an embodiment of the present invention is stored by that will merge file to the first storage region,
By index datastore to the second storage region, since the reading and writing data performance of the second storage region is higher than the first storage region
Reading and writing data performance, therefore when reading small documents, it can more rapidly be read using the high readwrite performance of the second storage region
The index data of each small documents is taken, and then according to read index data, the merging file stored from the first storage region
Middle each small documents of reading are deposited to avoid the problem that reading data degraded performance existing for existing file management method in saving
The reading data performance of cloud storage system can also be improved while storing up space.
Corresponding to above method embodiment, embodiment that the embodiment of the present invention also provides corresponding devices.
Installation practice 1
As shown in figure 9, can be applied in cloud storage system the embodiment of the invention provides a kind of document management apparatus
Management server, management server are used to manage more storage servers in cloud storage system, and storage server is for storing
Data may include multiple storage clusters in one cloud storage system, a management server be laid in each storage cluster
And several storage servers, management server can uniformly be managed above-mentioned several storage servers, storage clothes
Device be engaged in for storing data, illustratively, after obtaining data to be written, management server can control above-mentioned storage server
The data are stored, which includes:
Module 501 is obtained, for obtaining merging file to be written, merging file is that multiple small documents obtain after merging
File, small documents be size be lower than preset threshold file.
Merging file in the embodiment of the present invention can refer to one that obtains multiple small documents after merging treatment big
File, it will be understood that the obtained size for merging file is bigger compared to each small documents, due to can be with the shape of a file
Formula is stored, therefore can be avoided a case where small documents deposit discontented storage space block generation, that is, avoids storage empty
Between the problem of wasting.
Also, the data for merging each small documents in file, which can be, to be saved in order into merging file, therefore more
Be conducive to traditional mechanical hard disk read-write (the random reading performance of traditional mechanical hard disk is far below continuous reading performance).
Also, it is stored not yet into storage server due to merging file at this time, it can be by above-mentioned merging file
Merging file referred to as to be written.
Small documents therein can refer to that size is lower than the file of preset threshold, for example, size of data is lower than 2KB
The file of (Kilobyte, kilobytes).Certainly, the developer of cloud storage system can be flexibly reasonable according to practical business demand
Ground is arranged above-mentioned threshold value, the embodiment of the present invention to the specific value of above-mentioned threshold value without limitation.Multiple small documents are merged into one
A process for merging file can be obtained by existing file mergences method, and the detailed process embodiment of the present invention is no longer superfluous
It states.
First memory module 502 is stored to the first storage region for that will merge file, and is obtained in merging file
The index data of each small documents, wherein the storage location information an of small documents is carried in every index data.
Different storage regions can be set in a storage cluster, it is corresponding for different storage regions
Reading and writing data performance can be different, and therefore, the embodiment of the present invention can will merge file and store to the first storage region, and deposit
Chu Hou, obtains the index data of each small documents in the merging file, wherein a small documents are carried in every index data
Storage location information, that is, the storage location information of the corresponding small documents of an index data, storage location information is for remembering
Small documents are recorded in the storage location of the first storage region, for example, the identification number of the storage space block stored, and can read
Play the role of positioning when the small documents.
The process of the above-mentioned index data for obtaining each small documents can be obtained by existing file index information generation method
It arrives, the embodiment of the present invention repeats no more.
Second memory module 503 is used for index datastore to the second storage region, wherein the second storage region
Reading and writing data performance is higher than the reading and writing data performance of the first storage region.
After obtaining index data, the embodiment of the present invention can be deposited index datastore to the second storage region, second
Storage area domain is the region different from the first storage region, also, the reading and writing data performance of the second storage region is higher than the first storage
The reading and writing data performance in region.By the way that index datastore into higher second storage region of readwrite performance, is being read in this way
When taking large amount of small documents, it can use the high readwrite performance of the second storage region, more rapidly read the index of each small documents
Data, and then each small documents are further read from merging file according to read index data, it can be avoided due to tradition
The problem of reading data degraded performance caused by mechanical hard disk performance limits.In addition, since index data is relative to file data
Required memory space is smaller, therefore, only can save index datastore in the second high storage region of reading and writing data performance
Cost.
As a kind of optional embodiment of the embodiment of the present invention, the first memory module specifically can be used for:
File will be merged to store into the corresponding storage region of the first storage medium；
Second memory module, specifically can be used for:
By index datastore into the corresponding storage region of the second storage medium, wherein the data of the second storage medium
Readwrite performance is higher than the reading and writing data performance of the first storage medium.
As a kind of optional embodiment of the embodiment of the present invention, the first storage medium is mechanical hard disk, and the second storage is situated between
Matter is solid-state hard disk SSD.
As a kind of optional embodiment of the embodiment of the present invention, the first memory module specifically can be used for:
File will be merged to store to the first storage region of the first storage server, the first storage server is more storages
Wherein one in server；
Second memory module, specifically can be used for:
By the second storage region of index datastore to the first storage server.
As a kind of optional embodiment of the embodiment of the present invention, the first storage region and the second storage region are divided into
The identical storage space block of size, for each storage space block for storing a data block, a data block includes: multiple deposits
Catalogue and storage location specified directory are stored up, for storing data, there is default record storage catalogue under storage location specified directory
Numerical value, different default values store data to different storage catalogues for specified, as shown in Figure 10, then the first storage
Module may include:
First sub-module stored 5021 will if the size for merging file is not more than the size of storage space block
Merge under the specified storage catalogue that file stores data block corresponding to one of storage space block to the first storage region.
Second sub-module stored 5022 will close if the size for merging file is greater than the size of storage space block
And file is stored to the specified storage catalogue of the corresponding data block of multiple storage space blocks of the first storage region.
A kind of document management apparatus provided in an embodiment of the present invention, after obtaining merging file to be written, by that will close
And file is stored to the first storage region, and is obtained for the index data for merging each small documents in file, then by index data
It stores to the second storage region, since the reading and writing data performance of the second storage region is higher than the reading and writing data of the first storage region
Can, therefore when reading small documents, each small documents can be more rapidly read using the high readwrite performance of the second storage region
Index data read from the merging file that the first storage region is stored each small and then according to read index data
File is saving the same of memory space to avoid the problem that reading data degraded performance existing for existing file management method
When can also improve the reading data performance of cloud storage system.
Installation practice 2
The embodiment of the invention also provides a kind of document management apparatus, on the basis of embodiment shown in Fig. 9, such as Figure 11 institute
Show, can also include:
First read module 601, for reading the small documents to read from the first storage region from the second storage region
Index data.
Locating module 602, for positioning storage position of the data in the first storage region of small documents according to index data
It sets.
Second read module 603, the merging text for being stored from the first storage region according to the storage location positioned
Small documents are read in part.
As a kind of optional embodiment of the embodiment of the present invention, the first read module specifically can be used for:
From the corresponding storage region of the second storage medium, obtain to be read from the corresponding storage region of the first storage medium
The index data of the small documents taken；
Second read module, specifically can be used for:
From the merging file that the corresponding storage region of the first storage medium is stored, small documents are read.
A kind of document management apparatus provided in an embodiment of the present invention is stored by that will merge file to the first storage region,
By index datastore to the second storage region, since the reading and writing data performance of the second storage region is higher than the first storage region
Reading and writing data performance, therefore when reading small documents, it can more rapidly be read using the high readwrite performance of the second storage region
The index data of each small documents is taken, and then according to read index data, the merging file stored from the first storage region
Middle each small documents of reading are deposited to avoid the problem that reading data degraded performance existing for existing file management method in saving
The reading data performance of cloud storage system can also be improved while storing up space.
The embodiment of the invention also provides a kind of electronic equipment, are specifically as follows server, as shown in figure 12, the equipment
700 include processor 701 and machine readable storage medium 702, and machine readable storage medium, which is stored with, to be executed by processor
Machine-executable instruction, processor execute machine-executable instruction perform the steps of
Merging file to be written is obtained, merging file is the file that multiple small documents obtain after merging, and small documents are
Size is lower than the file of preset threshold；
File will be merged to store to the first storage region, and obtained for the index data for merging each small documents in file,
Wherein, the storage location information an of small documents is carried in every index data；
By index datastore to the second storage region, wherein the reading and writing data performance of the second storage region is higher than first
The reading and writing data performance of storage region.
Machine readable storage medium may include random access memory (Random Access Memory, abbreviation RAM),
It also may include nonvolatile memory (non-volatile memory), for example, at least a magnetic disk storage.Optionally,
Memory can also be that at least one is located remotely from the storage device of aforementioned processor.
Above-mentioned processor can be general processor, including central processing unit (Central Processing Unit,
Abbreviation CPU), network processing unit (Network Processor, abbreviation NP) etc.；It can also be digital signal processor
(Digital Signal Processing, abbreviation DSP), specific integrated circuit (Application Specific
Integrated Circuit, abbreviation ASIC), field programmable gate array (Field-Programmable Gate Array,
Abbreviation FPGA) either other programmable logic device, discrete gate or transistor logic, discrete hardware components.
Electronic equipment provided in an embodiment of the present invention is deposited after obtaining merging file to be written by that will merge file
Storage is obtained for the index data for merging each small documents in file to the first storage region, then by index datastore to the
Two storage regions, since the reading and writing data performance of the second storage region is higher than the reading and writing data performance of the first storage region,
When reading small documents, the index number of each small documents can be more rapidly read using the high readwrite performance of the second storage region
According to, and then according to read index data, each small documents are read from the merging file that the first storage region is stored, thus
It avoids the problem that reading data degraded performance existing for existing file management method, can also be mentioned while saving memory space
The reading data performance of high cloud storage system.
The embodiment of the invention also provides a kind of computer readable storage medium, it is stored in computer readable storage medium
Computer program, when computer program is executed by processor, to execute following steps:
Merging file to be written is obtained, merging file is the file that multiple small documents obtain after merging, and small documents are
Size is lower than the file of preset threshold；
File will be merged to store to the first storage region, and obtained for the index data for merging each small documents in file,
Wherein, the storage location information an of small documents is carried in every index data；
By index datastore to the second storage region, wherein the reading and writing data performance of the second storage region is higher than first
The reading and writing data performance of storage region.
Computer readable storage medium provided in an embodiment of the present invention, after obtaining merging file to be written, pass through by
Merge file to store to the first storage region, and obtain for the index data for merging each small documents in file, then by index number
According to storing to the second storage region, since the reading and writing data performance of the second storage region is higher than the reading and writing data of the first storage region
Performance, therefore when reading small documents, each small text can be more rapidly read using the high readwrite performance of the second storage region
The index data of part, and then according to read index data, it is read from the merging file that the first storage region is stored each
Small documents are saving memory space to avoid the problem that reading data degraded performance existing for existing file management method
The reading data performance of cloud storage system can also be improved simultaneously.
The embodiment of the invention also provides a kind of computer program products comprising instruction, when it runs on computers
When, so that computer executes following steps:
Merging file to be written is obtained, merging file is the file that multiple small documents obtain after merging, and small documents are
Size is lower than the file of preset threshold；
File will be merged to store to the first storage region, and obtained for the index data for merging each small documents in file,
Wherein, the storage location information an of small documents is carried in every index data；
By index datastore to the second storage region, wherein the reading and writing data performance of the second storage region is higher than first
The reading and writing data performance of storage region.
Computer program product provided in an embodiment of the present invention comprising instruction, after obtaining merging file to be written,
It is stored by the way that file will be merged to the first storage region, and obtained for the index data for merging each small documents in file, then will
Index datastore is to the second storage region, since the reading and writing data performance of the second storage region is higher than the number of the first storage region
According to readwrite performance, therefore when reading small documents, can more rapidly be read using the high readwrite performance of the second storage region
The index data of each small documents, and then according to read index data, from the merging file that the first storage region is stored
Each small documents are read, to avoid the problem that reading data degraded performance existing for existing file management method, are stored saving
The reading data performance of cloud storage system can also be improved while space.
The embodiment of the invention also provides a kind of computer programs, when run on a computer, so that computer is held
Row following steps:
Merging file to be written is obtained, merging file is the file that multiple small documents obtain after merging, and small documents are
Size is lower than the file of preset threshold；
File will be merged to store to the first storage region, and obtained for the index data for merging each small documents in file,
Wherein, the storage location information an of small documents is carried in every index data；
By index datastore to the second storage region, wherein the reading and writing data performance of the second storage region is higher than first
The reading and writing data performance of storage region.
Computer program provided in an embodiment of the present invention comprising instruction passes through after obtaining merging file to be written
File will be merged to store to the first storage region, and obtained for the index data for merging each small documents in file, then will index
Data are stored to the second storage region, since the data that the reading and writing data performance of the second storage region is higher than the first storage region are read
Write performance, therefore when reading small documents, it can more rapidly be read using the high readwrite performance of the second storage region each small
The index data of file, and then according to read index data, it is read from the merging file that the first storage region is stored
Each small documents are saving memory space to avoid the problem that reading data degraded performance existing for existing file management method
While can also improve the reading data performance of cloud storage system.
For device/electronic equipment/storage medium embodiment, since it is substantially similar to the method embodiment, so
It is described relatively simple, the relevent part can refer to the partial explaination of embodiments of method.
It should be noted that, in this document, relational terms such as first and second and the like are used merely to a reality
Body or operation are distinguished with another entity or operation, are deposited without necessarily requiring or implying between these entities or operation
In any actual relationship or order or sequence.Moreover, the terms "include", "comprise" or its any other variant are intended to
Non-exclusive inclusion, so that the process, method, article or equipment including a series of elements is not only wanted including those
Element, but also including other elements that are not explicitly listed, or further include for this process, method, article or equipment
Intrinsic element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that
There is also other identical elements in process, method, article or equipment including the element.
Each embodiment in this specification is all made of relevant mode and describes, same and similar portion between each embodiment
Dividing may refer to each other, and each embodiment focuses on the differences from other embodiments.Especially for system reality
For applying example, since it is substantially similar to the method embodiment, so being described relatively simple, related place is referring to embodiment of the method
Part explanation.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the scope of the present invention.It is all
Any modification, equivalent replacement, improvement and so within the spirit and principles in the present invention, are all contained in protection scope of the present invention
It is interior.
Claims (16)
1. a kind of file management method, which is characterized in that applied to the management server in cloud storage system, the management service
Device is used to manage more storage servers in the cloud storage system, the storage server for storing data, the side
Method includes:
Merging file to be written is obtained, the file that merges is the file that multiple small documents obtain after merging, the small text
Part is the file that size is lower than preset threshold；
The merging file is stored to the first storage region, and obtains the index number for each small documents in the merging file
According to, wherein the storage location information of the small documents is carried in every index data；
By the index datastore to the second storage region, wherein the reading and writing data performance of second storage region is higher than
The reading and writing data performance of first storage region.
2. the method according to claim 1, wherein described store the merging file to the first storage region
The step of, comprising:
The merging file is stored into the corresponding storage region of the first storage medium；
It is described by the index datastore to the second storage region the step of, comprising:
By the index datastore into the corresponding storage region of the second storage medium, wherein second storage medium
Reading and writing data performance is higher than the reading and writing data performance of first storage medium.
3. according to the method described in claim 2, it is characterized in that, first storage medium be mechanical hard disk, described second
Storage medium is solid-state hard disk SSD.
4. method according to claim 1-3, which is characterized in that described to store the merging file to first
The step of storage region, comprising:
The merging file is stored to the first storage region of the first storage server, first storage server is described
Wherein one in more storage servers；
It is described by the index datastore to the second storage region the step of, comprising:
By the index datastore to the second storage region of first storage server.
5. according to the method described in claim 4, it is characterized in that, first storage region and the second storage region quilt
It is divided into the identical storage space block of size, each storage space block is for storing a data block, a data block
Include: multiple storage catalogues and storage location specified directory, the storage catalogue for storing data, the storage location
Record has default value under specified directory, and the different default values stores data to different storage catalogues for specified
Under；
Described the step of storing the merging file to the first storage region, comprising:
If the size for merging file is not more than the size of the storage space block, the merging file is stored to institute
Under the specified storage catalogue for stating data block corresponding to one of storage space block of the first storage region；
If the size for merging file is greater than the size of the storage space block, the merging file is stored to described
Under the specified storage catalogue of the corresponding data block of multiple storage space blocks of first storage region.
6. according to the method described in claim 5, it is characterized in that, the method also includes:
From second storage region, the index data of the small documents to read from first storage region is read；
According to the index data, storage location of the data in first storage region of the small documents is positioned；
According to the storage location positioned, from the merging file that first storage region is stored described in reading
Small documents.
7. according to the method described in claim 6, it is characterized in that, described from second storage region, read to from described
The step of index data for the small documents that first storage region is read, comprising:
From the corresponding storage region of second storage medium, obtain to from the corresponding storage region of first storage medium
The index data of the small documents of middle reading；
Described the step of reading the small documents from the merging file that first storage region is stored, comprising:
From the merging file that the corresponding storage region of first storage medium is stored, the small documents are read.
8. a kind of document management apparatus, which is characterized in that applied to the management server in cloud storage system, the management service
Device is used to manage more storage servers in the cloud storage system, the storage server for storing data, the dress
It sets and includes:
Module is obtained, for obtaining merging file to be written, the multiple small documents of the merging file obtain after merging
File, the small documents are the file that size is lower than preset threshold；
First memory module for storing the merging file to the first storage region, and obtains and is directed to the merging file
In each small documents index data, wherein carried in every index data the small documents storage location letter
Breath；
Second memory module is used for the index datastore to the second storage region, wherein second storage region
Reading and writing data performance is higher than the reading and writing data performance of first storage region.
9. device according to claim 8, which is characterized in that first memory module is specifically used for:
The merging file is stored into the corresponding storage region of the first storage medium；
Second memory module, is specifically used for:
By the index datastore into the corresponding storage region of the second storage medium, wherein second storage medium
Reading and writing data performance is higher than the reading and writing data performance of first storage medium.
10. device according to claim 9, which is characterized in that first storage medium be mechanical hard disk, described second
Storage medium is solid-state hard disk SSD.
11. according to the described in any item devices of claim 8-10, which is characterized in that first memory module is specifically used for:
The merging file is stored to the first storage region of the first storage server, first storage server is described
Wherein one in more storage servers；
Second memory module, is specifically used for:
By the index datastore to the second storage region of first storage server.
12. device according to claim 11, which is characterized in that first storage region and second storage region
It is divided into the identical storage space block of size, each storage space block is for storing a data block, the data
Block includes: multiple storage catalogues and storage location specified directory, the storage catalogue for storing data, the storage position
Setting record under specified directory has default value, and the different default values stores data to different storage mesh for specified
Under record；
First memory module, comprising:
First sub-module stored will if being not more than the size of the storage space block for the size for merging file
The specified storage for merging file and storing data block corresponding to one of storage space block to first storage region
Under catalogue；
Second sub-module stored, if being greater than the size of the storage space block for the size for merging file, by institute
Merging file is stated to store to the specified storage mesh of the corresponding data block of multiple storage space blocks of first storage region
Under record.
13. device according to claim 12, which is characterized in that described device further include:
First read module, for reading the small documents to read from first storage region from second storage region
Index data；
Locating module, for positioning data the depositing in first storage region of the small documents according to the index data
Storage space is set；
Second read module, for according to the storage location positioned, stored from first storage region described in
Merge in file and reads the small documents.
14. device according to claim 13, which is characterized in that first read module is specifically used for:
From the corresponding storage region of second storage medium, obtain to from the corresponding storage region of first storage medium
The index data of the small documents of middle reading；
Second read module, is specifically used for:
From the merging file that the corresponding storage region of first storage medium is stored, the small documents are read.
15. a kind of electronic equipment, which is characterized in that including processor and machine readable storage medium, the machine readable storage
Media storage has the machine-executable instruction that can be executed by the processor, and the processor executes the executable finger of the machine
Enable the method and step to realize the described in any item file management methods of claim 1-7.
16. a kind of computer readable storage medium, which is characterized in that be stored with computer in the computer readable storage medium
Program realizes the side of the described in any item file management methods of claim 1-7 when the computer program is executed by processor
Method step.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910411298.8A CN110147203B (en) | 2019-05-16 | 2019-05-16 | File management method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910411298.8A CN110147203B (en) | 2019-05-16 | 2019-05-16 | File management method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110147203A true CN110147203A (en) | 2019-08-20 |
| CN110147203B CN110147203B (en) | 2022-11-04 |
Family
ID=67595693
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910411298.8A Active CN110147203B (en) | 2019-05-16 | 2019-05-16 | File management method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110147203B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112416880A (en) * | 2021-01-22 | 2021-02-26 | 南京群顶科技有限公司 | Method and device for optimizing storage performance of mass small files based on real-time merging |
| CN113420025A (en) * | 2021-06-11 | 2021-09-21 | 广联达科技股份有限公司 | Component data processing method and device and electronic equipment |
| CN113495681A (en) * | 2020-04-07 | 2021-10-12 | 杭州萤石软件有限公司 | NAND FLASH file data access method, device and storage medium |
| CN114218161A (en) * | 2021-12-29 | 2022-03-22 | 北京百度网讯科技有限公司 | Index storage method and device, retrieval engine, electronic equipment and storage medium |
| WO2022083287A1 (en) * | 2020-10-20 | 2022-04-28 | 百果园技术(新加坡)有限公司 | Storage space management method and apparatus, device, and storage medium |
| CN117632039A (en) * | 2024-01-25 | 2024-03-01 | 合肥兆芯电子有限公司 | Memory management method, memory storage device and memory control circuit unit |
Citations (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5454103A (en) * | 1993-02-01 | 1995-09-26 | Lsc, Inc. | Method and apparatus for file storage allocation for secondary storage using large and small file blocks |
| CA2596434A1 (en) * | 2006-08-25 | 2008-02-25 | Dan Dodge | File system having variable logical storage block size |
| US20130111182A1 (en) * | 2011-10-26 | 2013-05-02 | International Business Machines Corporation | Storing a small file with a reduced storage and memory footprint |
| CN103577123A (en) * | 2013-11-12 | 2014-02-12 | 河海大学 | Small file optimization storage method based on HDFS |
| CN103605726A (en) * | 2013-11-15 | 2014-02-26 | 中安消技术有限公司 | Method and system for accessing small files, control node and storage node |
| CN103678579A (en) * | 2013-12-12 | 2014-03-26 | 浪潮电子信息产业股份有限公司 | Optimizing method for small-file storage efficiency |
| CN104462563A (en) * | 2014-12-26 | 2015-03-25 | 浙江宇视科技有限公司 | File storage method and system |
| CN104536959A (en) * | 2014-10-16 | 2015-04-22 | 南京邮电大学 | Optimized method for accessing lots of small files for Hadoop |
| CN104572670A (en) * | 2013-10-15 | 2015-04-29 | 方正国际软件(北京)有限公司 | Small file storage, query and deletion method and system |
| CN104991747A (en) * | 2015-07-30 | 2015-10-21 | 湖南亿谷科技发展股份有限公司 | Method and system for data management |
| CN105069048A (en) * | 2015-07-23 | 2015-11-18 | 东方网力科技股份有限公司 | Small file storage method, query method and device |
| CN105095421A (en) * | 2015-07-14 | 2015-11-25 | 南京国电南自美卓控制系统有限公司 | Distributed storage method for real-time database |
| CN105138571A (en) * | 2015-07-24 | 2015-12-09 | 四川长虹电器股份有限公司 | Distributed file system and method for storing lots of small files |
| US9286261B1 (en) * | 2011-11-14 | 2016-03-15 | Emc Corporation | Architecture and method for a burst buffer using flash technology |
| CN105868286A (en) * | 2016-03-23 | 2016-08-17 | 中国科学院计算技术研究所 | Parallel adding method and system for merging small files on basis of distributed file system |
| CN105956183A (en) * | 2016-05-30 | 2016-09-21 | 广东电网有限责任公司电力调度控制中心 | Method and system for multi-stage optimization storage of a lot of small files in distributed database |
| CN106021585A (en) * | 2016-06-02 | 2016-10-12 | 同济大学 | Traffic incident video access method and system based on time-space characteristics |
| CN106294603A (en) * | 2016-07-29 | 2017-01-04 | 北京奇虎科技有限公司 | File memory method and device |
| CN106775446A (en) * | 2016-11-11 | 2017-05-31 | 中国人民解放军国防科学技术大学 | Based on the distributed file system small documents access method that solid state hard disc accelerates |
| CN107247714A (en) * | 2016-06-01 | 2017-10-13 | 国家电网公司 | A kind of small documents access system and method based on distributed storage technology |
| CN107766374A (en) * | 2016-08-19 | 2018-03-06 | 上海凯翔信息科技有限公司 | The optimization method and system that a kind of mass small documents storage is read |
| CN108234594A (en) * | 2017-11-28 | 2018-06-29 | 北京市商汤科技开发有限公司 | File memory method and device, electronic equipment, program and medium |
-
2019
- 2019-05-16 CN CN201910411298.8A patent/CN110147203B/en active Active
Patent Citations (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5454103A (en) * | 1993-02-01 | 1995-09-26 | Lsc, Inc. | Method and apparatus for file storage allocation for secondary storage using large and small file blocks |
| CA2596434A1 (en) * | 2006-08-25 | 2008-02-25 | Dan Dodge | File system having variable logical storage block size |
| US20130111182A1 (en) * | 2011-10-26 | 2013-05-02 | International Business Machines Corporation | Storing a small file with a reduced storage and memory footprint |
| US9286261B1 (en) * | 2011-11-14 | 2016-03-15 | Emc Corporation | Architecture and method for a burst buffer using flash technology |
| CN104572670A (en) * | 2013-10-15 | 2015-04-29 | 方正国际软件(北京)有限公司 | Small file storage, query and deletion method and system |
| CN103577123A (en) * | 2013-11-12 | 2014-02-12 | 河海大学 | Small file optimization storage method based on HDFS |
| CN103605726A (en) * | 2013-11-15 | 2014-02-26 | 中安消技术有限公司 | Method and system for accessing small files, control node and storage node |
| CN103678579A (en) * | 2013-12-12 | 2014-03-26 | 浪潮电子信息产业股份有限公司 | Optimizing method for small-file storage efficiency |
| CN104536959A (en) * | 2014-10-16 | 2015-04-22 | 南京邮电大学 | Optimized method for accessing lots of small files for Hadoop |
| CN104462563A (en) * | 2014-12-26 | 2015-03-25 | 浙江宇视科技有限公司 | File storage method and system |
| CN105095421A (en) * | 2015-07-14 | 2015-11-25 | 南京国电南自美卓控制系统有限公司 | Distributed storage method for real-time database |
| CN105069048A (en) * | 2015-07-23 | 2015-11-18 | 东方网力科技股份有限公司 | Small file storage method, query method and device |
| CN105138571A (en) * | 2015-07-24 | 2015-12-09 | 四川长虹电器股份有限公司 | Distributed file system and method for storing lots of small files |
| CN104991747A (en) * | 2015-07-30 | 2015-10-21 | 湖南亿谷科技发展股份有限公司 | Method and system for data management |
| CN105868286A (en) * | 2016-03-23 | 2016-08-17 | 中国科学院计算技术研究所 | Parallel adding method and system for merging small files on basis of distributed file system |
| CN105956183A (en) * | 2016-05-30 | 2016-09-21 | 广东电网有限责任公司电力调度控制中心 | Method and system for multi-stage optimization storage of a lot of small files in distributed database |
| CN107247714A (en) * | 2016-06-01 | 2017-10-13 | 国家电网公司 | A kind of small documents access system and method based on distributed storage technology |
| CN106021585A (en) * | 2016-06-02 | 2016-10-12 | 同济大学 | Traffic incident video access method and system based on time-space characteristics |
| CN106294603A (en) * | 2016-07-29 | 2017-01-04 | 北京奇虎科技有限公司 | File memory method and device |
| CN107766374A (en) * | 2016-08-19 | 2018-03-06 | 上海凯翔信息科技有限公司 | The optimization method and system that a kind of mass small documents storage is read |
| CN106775446A (en) * | 2016-11-11 | 2017-05-31 | 中国人民解放军国防科学技术大学 | Based on the distributed file system small documents access method that solid state hard disc accelerates |
| CN108234594A (en) * | 2017-11-28 | 2018-06-29 | 北京市商汤科技开发有限公司 | File memory method and device, electronic equipment, program and medium |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113495681A (en) * | 2020-04-07 | 2021-10-12 | 杭州萤石软件有限公司 | NAND FLASH file data access method, device and storage medium |
| WO2022083287A1 (en) * | 2020-10-20 | 2022-04-28 | 百果园技术(新加坡)有限公司 | Storage space management method and apparatus, device, and storage medium |
| CN112416880A (en) * | 2021-01-22 | 2021-02-26 | 南京群顶科技有限公司 | Method and device for optimizing storage performance of mass small files based on real-time merging |
| CN113420025A (en) * | 2021-06-11 | 2021-09-21 | 广联达科技股份有限公司 | Component data processing method and device and electronic equipment |
| CN114218161A (en) * | 2021-12-29 | 2022-03-22 | 北京百度网讯科技有限公司 | Index storage method and device, retrieval engine, electronic equipment and storage medium |
| CN117632039A (en) * | 2024-01-25 | 2024-03-01 | 合肥兆芯电子有限公司 | Memory management method, memory storage device and memory control circuit unit |
| CN117632039B (en) * | 2024-01-25 | 2024-05-03 | 合肥兆芯电子有限公司 | Memory management method, memory storage device and memory control circuit unit |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110147203B (en) | 2022-11-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10649696B2 (en) | Data set management | |
| CN110147203A (en) | A kind of file management method, device, electronic equipment and storage medium | |
| US10346081B2 (en) | Handling data block migration to efficiently utilize higher performance tiers in a multi-tier storage environment | |
| CN103064639B (en) | Date storage method and device | |
| US9329957B2 (en) | Method for managing storage system using flash memory, and computer | |
| US11675499B2 (en) | Synchronous discovery logs in a fabric storage system | |
| CN106446159B (en) | A kind of method of storage file, the first virtual machine and name node | |
| CN110134338B (en) | Distributed storage system and data redundancy protection method and related equipment thereof | |
| US20180059949A1 (en) | Tier based data file management | |
| US11531491B2 (en) | Systems and methods for random to sequential storage mapping | |
| US9189494B2 (en) | Object file system | |
| CN103929500A (en) | Method for data fragmentation of distributed storage system | |
| US8386707B2 (en) | Virtual disk management program, storage device management program, multinode storage system, and virtual disk managing method | |
| US20140181455A1 (en) | Category based space allocation for multiple storage devices | |
| CN103037004A (en) | Implement method and device of cloud storage system operation | |
| US10365845B1 (en) | Mapped raid restripe for improved drive utilization | |
| CN109582213A (en) | Data reconstruction method and device, data-storage system | |
| CN109918352B (en) | Memory system and method of storing data | |
| CN111435286B (en) | Data storage method, device and system | |
| CN104021088A (en) | Log storage method and device | |
| CN108304142A (en) | A kind of data managing method and device | |
| US11507278B2 (en) | Proactive copy in a storage environment | |
| CN105426125B (en) | A kind of date storage method and device | |
| CN109508140B (en) | Storage resource management method and device, electronic equipment and system | |
| CN113853778A (en) | Cloning method and device of file system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |