JP5303213B2 - Data management method with data compression processing - Google Patents

Description

  The present invention relates to a data management method for compressing and expanding a large amount of data including many redundant elements at high speed.

  First, terms used in the following sentences are defined.

  Row, column, field: As shown in FIG. 2, the terms row 203, column 204, and field 202 are used as elements constituting the table 201.

First normal form: One of RDBMS normalization levels. The fields that make up a row are defined as scalar values, that is, not including sets or arrays.
Next, I will describe the details of the background art.

  As mobile devices become more sophisticated, the amount of data handled has increased, and search patterns have diversified, and DBMS has come to be used in embedded environments. In particular, the RDBMS used in the conventional server system is already attracting attention because the query language is low because the query language is already standardized.

  In the device embedded environment, there is a strong demand for hardware cost reduction, and resource constraints are more severe than in the server system environment. Therefore, there is a demand for RDBMS that the smaller the storage usage and the memory usage, the better. However, on the other hand, there are many DB embedded users and there is not much need to consider update inconsistencies, so there are many operations using the first normal form schema, resulting in a lot of redundancy. There is a need for a database compression function.

  Many of the conventional techniques for database compression are for data warehouses that are referenced more frequently than updates. Among them, a data compression technique that applies a compression code such as an LZ code for each row is representative, and is disclosed in Non-Patent Document 1. This method is fast because no extra storage I / O occurs during compression / decompression processing, but the compression rate is difficult to increase because the original data is small. Furthermore, not only in the first normal form schema, database redundancy tends to be less in the row direction and more in the column direction. Therefore, compression by row unit is less likely to increase the compression rate.

  Another typical technique for database compression is column-wise compression, which is also disclosed in Non-Patent Document 1. In consideration of the fact that database redundancy tends to occur in the column direction, this is a technique that applies a compression technique such as LZ code in the column direction. One of the proposed implementation methods for column-wise compression is to divide the entire table into columns and compress it, but when compressing a table consisting of a large amount of data, the compression ratio is good. The decompression process is slow because storage I / O for reading the dictionary occurs. In addition, there is a drawback that the cost for deleting and updating is high and very slow.

  On the other hand, in the database compression method described in Non-Patent Document 2, when fields included in a single page overlap, compression is performed by replacing with a pointer. This method is fast because no extra storage I / O occurs during compression / decompression. On the other hand, since compression is performed only when the fields completely match, it is difficult to obtain the compression effect for a freely described character string type that stores notes and comments.

“Data Compression Support in Databases”, Balakrishna R. Iyer, VLDB94 “Data Compression in Oracle”, Meikel Poessv, VLDB2003

  Therefore, an object of the present invention is to provide a compression method capable of high-speed compression / decompression even when a large amount of data including redundancy is compressed and handled.

  The present invention pays attention to the fact that there is much redundancy in the column direction particularly in the data of the first normal form schema, and makes it possible to designate a redundant column as a user.

  To summarize the points of the data management method in the representative embodiment disclosed in this specification, data compression processing based on a compression value management table on the storage is performed on data in which a set of values of a plurality of items repeatedly appears. A data management method for performing management, wherein for each item in the data, designation by a user of whether or not compression is received, and for each item designated as being compressed, an identifier of the item and an identifier of data including the item Is stored in the compressed value management table, and only the value specified by the identifier of the item in the data is subjected to compression processing and managed on the storage.

  In other words, in the relational database field, the management target is a table, and the above-mentioned value set of multiple items is a row constituting the table, each item is a column constituting the table, and each item value is It is a field. That is, for each column of the user table, the user accepts designation of whether or not compression is performed, and for each column designated as being compressed, the column name and table name are stored in the compressed value management table, and each field of the user table is stored. Only the data in the field of the column designated as being compressed in the compressed value management table is subjected to compression processing and managed on the storage.

  In the data management method in another representative embodiment to be disclosed, a compression unit is introduced so that compression and expansion can be performed at a higher speed.

  The data management method of the present invention has an advantage that only a column specified by the user is compressed in the column direction for each compression unit, so that both the calculation amount and the number of storage I / O are small, and a high speed and high compression rate can be realized. .

  In the computer system of this embodiment, data in which a set of values of a plurality of items such as a table and a set of structures repeatedly appear is managed on the storage as a management target. In particular, only the value of one or more items specified by the user in each set is subjected to data compression processing, stored in storage, and managed.

  FIG. 1 shows the configuration of a computer system of this embodiment. The computer system 101 includes a CPU 112, a main storage device 113, and a storage 103 as main components. On the main storage device 113, the operating system 111 read from the illustrated storage 103 or another storage device is expanded on the main storage device 113. Further, a relational database execution unit 102 and an AP execution unit 110 are expanded on the main storage device 113. Under the control of the operating system 111, the CPU 112 executes processes of the relational database execution unit and the AP execution unit, thereby fulfilling a data management function and an application function.

Examples of realization of the storage 103 include a hard disk drive, a semiconductor memory, and a tape drive. The storage 103 is provided with a user data area 104 and a management information area 105. Management information used for data management by the relational database execution unit 102 is stored and managed in the management information area 104. Here, the management information is managed in a table format. What is specific to the present embodiment is that the management information stored in the management information area 104 includes a compression location management table 106 used for data compression / decompression processing. That is, it has a compressed value management table on the storage. Details of the compression value management table 106 will be described later. The user data area 104 stores a table that is a management target of the above-described data management and that is a target of data compression and expansion. These tables to be managed are called user tables for distinction.
FIG. 2 shows the structure of a general user table. The user table 201 includes a plurality of rows 203 and columns 204, and an element specified by a certain row and a certain column is referred to as a field 202.

  The relational database execution unit 108 includes a query processing unit 107 and a compressed value definition unit 108. The inquiry processing unit 107 has a function of receiving an instruction such as an inquiry or data definition from the user 109 and executing data management according to the instruction. The compressed value definition unit 108 manages management information related to the compressed value in the management information area 105 of the storage 104. Details of the compression value definition processing unit 108 will be described later.

  In this embodiment, when defining the structure of the user table as advance preparation, the user 109 designates whether or not compression is performed for each column, and after the query processing unit 107 interprets it, the compressed value definition processing unit 108 receives the compressed value. Save to the management table. That is, the user specifies whether or not compression is performed for each data item, and the compression value is an item to be compressed (here, the column name of the user table) and the identifier of the data including the item (here, the user table name). The definition part receives it and stores it in the compressed value management table.

  For reference when defining the structure of a user table, for example, in a relational database program that is widely used as a data management program for managing user tables, create table statements and table structure change statements that are SQL table structure definition statements Define the data structure using the alter table statement. For this reason, in this embodiment, the grammar is extended so that the presence / absence of compression for each column can also be specified by a create table statement or an alter table statement. FIG. 3 shows an example of a user table, a corresponding create table statement, and a compressed column management table. The songs table 301 shown in FIG. 3 (a) is a user table for storing music player music information, an id column representing a serial number, an artist_name column representing a singer name, an album_name column representing an album name, and a release column representing a release year. , A track_num column representing the order of the songs in the album, a genre column representing the genre of the music, and a song_title column representing the music name. Here, FIG. 3B shows an extension of the create table statement used for the table structure definition when the artist_name column, the album_name column, and the release column are designated as compressed columns. In this create table statement extension, when defining the artist_name column, album_name column, and release column, “compress” is specified to indicate that the column is a compressed column following the data type definition. In the compressed value management table 303 shown in FIG. 3C, user table identifiers and column identifiers managed by compression are managed. That is, the data identifier and the compressed value identifier are managed in the compressed value management table. In this embodiment, the user table name is assumed to be unique, and the user table name is used as the user table identifier. As a column identifier, the column name is used on the assumption that the column name is unique for each user table.

  As described above, in this embodiment, in a system in which a table is managed on the storage, the user designates whether or not to compress for each column constituting the table, registers it in the compressed value management table, and stores the table data. At the time of saving, the data of the field in the column designated as compressed is compressed and saved by the designated compression method. This method can be applied not only to the system that manages the table as the management target as in the above example, but also to the system that manages the data such as a set or array of structures, and other data in which multiple item value pairs appear repeatedly It is. FIG. 4 shows an example of expanding the structure array of C language. In structure declaration and structure array variable declaration 401, structure array variable songs [] corresponding to songs table 301 and structure song corresponding to rows in songs table are defined, and a set of 1,000 structures is defined doing. However, the id column in the songs table 301 corresponds to the subscript of the structure array variable songs []. Even when managing structure array variables defined in 401, it is possible to manage data in the same way as when defining using the extension 302 of the SQL create table statement.

  In the second embodiment, it is specified whether or not each column is a compression pair column, and the user also specifies the compression unit of the compression. The management target on the storage is not limited to the table, but the management target can be applied to a system that is data in which a set of values of a plurality of items repeatedly appears as in the first embodiment.

  FIG. 5 shows a data management system system configuration of the second embodiment.

  The computer system 501 has a data management program 502 and a storage 503 for storing management information related to tables and tables. The AP execution unit 515, the CPU 517, and the operating system 516 are the same as the AP execution unit (110 in FIG. 2), the CPU (112 in FIG. 2), and the operating system (111 in FIG. 2) in the data management system of the first embodiment. is there. The relational database execution unit 502 expanded on the main storage device 518 includes a query processing unit 507, a compressed value definition processing unit 508, an I / O distribution processing unit 509, an uncompressed table-oriented insert statement processing unit 510, and a compressed string. Insert statement processing unit 511 for a table including a select statement processing unit 519 for a table including a compressed column, a select statement processing unit 520 for an uncompressed table, a delete statement processing unit 513 for a compressed column, and a normal delete statement processing unit 514 included. In the compression value management table 506 stored in the management information area 505 of the storage 503, the compression unit is registered corresponding to the column name to be compressed.

  In this embodiment, all the fields of the column to be compressed designated by the user 512 are divided into a plurality of subsections designated by the user 512 and compressed. The size of this small section is called a compression unit, and the presence / absence of compression and the unit of compression are specified for each column by the user 512 when defining the structure of the user table as preparation for data management. The presence / absence and unit of compression designated for each column by the user 512 are interpreted by the query processing unit 507, received by the compression value definition processing unit 508, and stored in the compression value management table 506. That is, for each value included in the data, the user specifies whether or not compression is performed and the compression unit, and the item (column name) to be compressed specified by the user, the identifier of the data including the item (table name), and the compression unit The compressed value definition processing unit receives it and stores it in the compressed value management table. As an example of the unit of compression, a unit such as the size of data, the number of rows of data, the number of fields, the number of pages on the storage can be raised. In this embodiment, the number of pages is used. If the compression unit is n pages and the offset from the first page number of the user data area 504 of a page A is a, the offset of the page included in the same subsection as A is n × (a div n ) ≦ x <n × {(a div n) +1} is assumed to be a natural number x.

  FIG. 6 shows an example of a user table, a corresponding create table statement, and a compressed column management table. A songs table 601 in FIG. 6A is a user table for storing music information of the same music player as the user table in the first embodiment shown in FIG. That is, the table 601 includes an id column representing a serial number, an artist_name column representing a singer name, an album_name column representing an album name, a release column representing a release year, a track_num column representing the order of songs in the album, a genre column representing a song genre, It consists of a song_title column that represents the name. Here, the artist_name column, the album_name column, and the release column are specified as compression columns, and the artist_name column is expanded by 2 pages, and the album_name column and release column are expanded by 4 pages. 602. In the extension of the create table statement shown in FIG. 6B, when defining the artist_name column, the album_name column, and the release column, “compress” indicating that the column is a compression column following the data type definition and “2” that is a compression unit. "And" 4 "are specified. In the compressed value management table 603 shown in FIG. 6C, the user table identifier, column identifier, and compression unit to be managed by compression are managed. That is, in the compression value management table, a data identifier, a compression value identifier, and a compression unit are designated. In this embodiment, the user table name is assumed to be unique, and the user table name is used as the user table identifier. As a column identifier, the column name is used on the assumption that the column name is unique for each user table.

  In this embodiment, when inserting a row, an insert statement in SQL is used. The query processing unit 507 receives and processes the insert statement issued by the user 512. An insert statement processing instruction from the inquiry processing unit is passed to the I / O distribution processing unit 509.

  FIG. 7 shows a processing flow when the I / O distribution processing unit 509 executes the insert statement processing instruction. After the processing is started (701), the compressed value definition processing unit 508 is referred to (702), and it is determined whether or not the table to be inserted is a table including a compressed column (703). If the table into which the row is to be inserted is a table including a compressed column, the table insert statement processing unit 511 including the compressed column is called (704). On the other hand, if the table to be inserted does not include a compressed column, the insert statement processing unit 510 for uncompressed table is called (705), and the processing is terminated (706).

  The table insert statement processing unit 511 including the compressed column refers to the compression value definition processing unit 508 and compresses the requested field based on the designated compression unit. That is, compression is performed for each compression unit in accordance with the definition recorded in the compression value management table 506.

  When using a method that requires a dictionary typified by a Huffman code to be stored separately as a compression method, and when using a method that does not need to store a dictionary typified by an LZ code separately, a compressed string is included. The processing procedure of the insert statement processing unit 511 for the table is different. FIG. 8 shows a processing procedure in the case of the former compression method. First, in order to sequentially search a plurality of compression units in a table into which a row is inserted, the target compression unit number a is set to 1 (801), and the head position of the first compression unit is acquired (802). Next, the dictionary of the compression unit is read (803). In a typical example, the first page before the page storing the user table data in the compression unit is the page storing the dictionary. Next, the data of the insertion target line is compressed using the read dictionary (804). Next, the page is acquired from the first page in the compression unit (805, 806), and it is determined whether the acquired page has an empty area (807). If there is no empty area, it is determined whether there are still unconfirmed pages in the compression unit (809), the number of pages is incremented (810), and the determination of whether there is an empty area is repeated. If there is an empty area, the compressed data, that is, the data of the entire row that combines the data of the field corresponding to the compressed column obtained by compression and the raw data of the field corresponding to the uncompressed column. Write (814), the process ends. If all the pages in the compression unit are sequentially judged and there is no empty area, that is, if there are no more unconfirmed pages in step 809, it is determined whether there is an unconfirmed compression unit in the table (812). The compression unit number a is incremented, and for the next compression unit, the above-mentioned head position acquisition (802), dictionary reading (803), and empty area confirmation of each page (805 to 810) are performed. In this manner, pages in which rows can be inserted are searched in order from the first page of the table data over each compression unit, and writing is performed on the page where the empty area is found. If it is determined in step 812 that there is no unconfirmed compression unit without finding a page with an empty area, it is reported that row insertion has failed due to insufficient area, and the process is terminated.

  As described above, in the processing flow shown in FIG. 8, in the insert statement processing, a procedure for sequentially searching for insertable pages from the first page of the first table is taken, but a means for separately managing writable pages is prepared. It is possible to omit the loop processing of FIG.

  FIG. 9 shows the procedure of insert statement processing when using a compression method that does not require a separate dictionary. Also in the flow of FIG. 9, first, the compression unit number a is set to 1 (901), and the head position of the head compression unit in the table is acquired (902). Next, the attention page number b in the compression unit is set to 1 (903), and the attention page is acquired (904). The data of the compressed column in the data is expanded from the data in the first row in the page, and the expansion is reflected in the dictionary (905, 906). The page number of interest is sequentially incremented (908), and the process of expansion and reflection to the dictionary is repeated. If it is determined in step 907 that all the expansion processing has been completed for the rows in the page, data corresponding to the compression column of the insertion target row is compressed (909), and an empty area for storing the compressed data in the page It is determined whether there is (910). If there is an empty area, the compressed data is written to the page (916), and the row insertion process is terminated. If there is no empty area, the page number b of interest is incremented (912), and the decompression process for each line and the compression process for the insertion target line are performed for the next page. If there is an empty area in the page in step 910, the compressed data in step 916 is written, and the process ends. If all the pages in the compression unit are sequentially determined and there is no empty area, it is determined whether there is an unconfirmed compression unit in the table (913). If there is an unconfirmed compression unit in the table, the compression unit number a is incremented (914), and the processing of steps 902 to 912 is performed for the next compression unit. When each page of each compression unit is checked until a page having an empty area into which a row can be inserted is found in this way, and when it is determined in step 913 that there is no more unconfirmed compression unit without finding a page with an empty area. Then, it reports that the insertion of the line has failed due to a lack of area and ends the processing (915).

  The uncompressed table insert statement processing unit 510 inserts the field of the insertion target row without performing compression and expansion processing. Note that the processing method of the uncompressed table insert statement processing unit 510 is a known processing method, and thus the details are omitted.

  In this embodiment, when reading all or part of the table, a select statement in SQL is used. The select statement issued by the user 512 is received and processed by the query processing unit 507. A select statement processing instruction from the inquiry processing unit is passed to the I / O distribution processing unit 509. FIG. 10 shows an operation flow when the I / O distribution processing unit 509 executes the select statement processing instruction. After the Select statement processing start (1001), the compressed value definition table 508 is referred to (1002), and it is determined whether the table to be read includes a compressed column (1003). If the table includes a compressed column, the table select statement processing unit 519 including the compressed column is called (1004), and the processing is completed (1006). On the other hand, if a table that does not include a compressed column is a read target, the select statement processing unit 520 for uncompressed tables is called (1005), and the processing is completed (1006).

  The processing flow of the table-specific select statement processing unit 519 including a compressed column is shown in FIGS. FIG. 11 is a processing flow of the table-specific select statement processing unit 519 including a compressed string when a compression method that requires a dictionary to be stored separately is used. First, in order to sequentially search a plurality of compression units in the user table, the compression unit number a of interest is set to 1 (1101), and the head position of the first compression unit is acquired (1102). Next, the dictionary of the compression unit is read (1103). The page is acquired from the first page in the compression unit (1104, 1105). The compressed data is decompressed in order from the data of the first line in the acquired page (1106, 1107), and it is determined whether or not the condition of the select statement is satisfied (1108). Processing 1109 for outputting data is performed. After this processing, the line number c is incremented (1111), and the process is sequentially performed for each line. If it is determined in step 1110 that there is no unsearched line in the page, the page number b is incremented in step 1114 through step 1112, and steps 1105 to 1111 are repeatedly performed for the next page. Repeat for each page until there are no more unsearched pages. If it is determined in step 1112 that there is no unsearched page in the compression unit, step 1114 is followed by step 1115 to increment the compression unit number a, and the search for the next compression unit in the user table is similarly performed. If it is determined in step 1114 that there is no unsearched compression unit in the user table, the select statement processing is terminated. FIG. 12 shows the procedure of select statement processing in the case of using a compression method that does not need to store a dictionary separately. Since the processing is repeated for each page included in each of the compression units of the user table in the same manner as the flow of FIG. 11, the flow feedback from step 1213 to step 1214 and the steps 1211 to 1212 are performed. The explanation of the return of the flow through is omitted. In the processing for each page, the page is first acquired (1204), this line is expanded from the data of the first line in the page, and the expansion is reflected in the dictionary created at the time of expansion (1205, 1206). Next, it is determined whether the line satisfies the condition of the select statement (1207). If the judgment is Yes, the line is output (1208). If the determination is No, the process proceeds to step 1209 as it is. If there is an unsearched line in the page, the line number is incremented (1210), the above processing is performed again from the acquisition of the page, and the process is repeated until there is no unsearched page.

  A detailed description of the processing method of the select statement processing unit 520 for uncompressed tables is omitted.

  When deleting rows from the user table, use the delete statement in SQL. The query processing unit 507 receives and processes the delete statement issued by the user 512. FIG. 13 shows an operation flow when the query processing unit 507 processes the delete statement. After starting the delete statement processing 1301, the compression value definition table 506 is referred to (1302), and it is determined whether the table that is the target of the delete statement includes a compressed column (1303). That is, the query processing unit 507 refers to the compressed value definition table 506 and determines whether the deleted data includes a compressed value. If the compressed string is included, the delete statement processing unit 513 for the compressed string is called (904), and the process is completed (906). On the other hand, if the compressed string is not included, normal delete statement processing is called (1305), and the processing is completed.

  The delete statement processing for the compressed string called in step 1304 will be described in detail with reference to FIGS. FIG. 14 shows a flow of the delete statement processing for the compressed string executed by the delete statement processing unit 513 in the case where a compression method that needs to store a dictionary separately, for example, represented by Huffman code compression, is used. In this case, the contents of the dictionary are not changed, and the process of deleting the part where the line actually exists and setting the dictionary deterioration flag is executed. The flow in FIG. 12 is based on the assumption that the target user table includes a plurality of compression units, and the compression units include a plurality of pages. First, the compression unit number a is set to 1 (1401), and the head position of the compression unit is acquired (1402). In this example, the dictionary of the compression unit is stored at the head position of the compression unit, and this dictionary is read in step 1403. Next, the page number b in the compression unit is set to 1 (1404), and the page is acquired (1405). Next, the number of the line in the acquired page is set to 1 (1406), and this line is expanded using the previous dictionary (1407). Next, it is determined whether or not the relevant line is the target of the delete statement (1408). If yes, the relevant line is deleted (1409). If the first line is erased within the compression unit, the deterioration flag is stored in the above dictionary. (1410). If the line is not the object of the delete statement, the process proceeds directly to 1411. If there is an unprocessed line in the page, the line number is incremented (1412), and steps 1407 to 1510 are repeated. If it is determined in step 1413 that there is an unprocessed page in the compression unit, the page number is incremented (1414), the page is acquired, and steps 1407 to 1410 are repeated for each row of the page. As a result of the above processing, all the lines corresponding to the delete statement in the compression unit are deleted, and a deterioration flag is attached to the dictionary of the compression unit. The state at this time is shown in FIG. The above processing is executed for all of the plurality of compression units of the target user table by repeating the determination whether there is an unprocessed compression unit (1415) and the increment of the compression unit number a (1416). The

  When a method that does not need to store a dictionary typified by an LZ code as a compression method is used, it is possible to employ a process in which a deletion flag is set in a row that is a target of a delete statement and no deletion is actually performed. This method can be realized, for example, by providing a 1-bit deletion flag area at the beginning of all lines. FIG. 15 shows a flow of delete statement processing for a compressed string in the case of a compression method that does not require another dictionary to be stored. Since the processing is repeated for each of the pages included in each of the plurality of compression units of the user table, it is exactly the same as the flow in FIG. 14, so the flow is returned from step 1513 to step 1514, and step 1511 to step 1512. The explanation of the flow feedback through the is omitted. In the processing for each page, first, the page is acquired (1504), the number of the line in the page is set to 1 (1505), this line is expanded, and the expansion is reflected in the dictionary created at the time of expansion (1506). ). Next, it is determined whether or not the line is a target line of a delete statement (1507). If the determination is Yes, a deletion flag indicating that the line has been deleted is attached (1508), and the process proceeds to step 1509 without actually erasing the page. If the determination is No, the process proceeds to step 1509 as it is. If there is an unprocessed line in the page, the line number is incremented (1510), the above process is performed again from the acquisition of the page, and the process is repeated until there is no unprocessed page. FIG. 17A shows the state after this delete statement process is performed several times.

  In either method, it is necessary to appropriately perform maintenance processing with the execution of the delete statement. In the delete statement processing described with reference to FIG. 11 in the case of a compression method in which a dictionary needs to be stored separately, when a line is erased within each compression unit, an empty area is included in each page as shown in FIG. 1601 is created, and an empty area can be used when there is a request to insert a new row in the table. However, in the dictionary stored in the dictionary storage page 1602 in relation to the compressed data in the compression unit, dictionary data that is no longer used for compression of the deleted line remains, that is, the dictionary deteriorates. Therefore, for example, when the delete statement process is executed a specific number of times, the maintenance process is started. In the maintenance process, it is searched whether the deterioration flag 1603 is attached to the dictionary of each compression unit, the line included in each page of the compression unit to which the deterioration flag is attached is expanded and the deteriorated dictionary is discarded. As shown in (b), the data in each row is rearranged to the left, recompressed, and written to each page. The dictionary generated by the recompression process is stored again in the dictionary storage page 1602.

  In the delete statement processing described with reference to FIG. 12 in the case of a compression method that does not require another dictionary to be stored, even if a row is deleted, only the deletion flag 1701 shown in FIG. Left behind. Therefore, since the empty area cannot be created even if the delete statement process is advanced as it is, it is preferable to perform the maintenance process in order to secure the empty area. As a maintenance process in this case, the compression unit including the line with the deletion flag 1701 is searched, the line with the deletion flag of each page in the compression unit is deleted, and as shown in FIG. The procedure of rearranging the data in each row to the left, recompressing and writing can be employed.

  Note that the data management method handled in this embodiment is further expanded, and the user specifies the presence / absence of compression, the compression unit, and the compression method for each value included in the data. In the compression value management table, the data identifier, the compression identifier, and the compression unit are specified. By managing the compression method, it is possible to specify the optimum compression method for each column.

  FIG. 18 shows an example of the user table and the corresponding create table statement expansion and compression column management table when the compression method is also specified. A songs table 1801 shown in FIG. 18A is a user table for storing music player music information, an id column representing a serial number, an artist_name column representing a singer name, an album_name column representing an album name, and a release column representing a release year. , A track_num column representing the order of the songs in the album, a genre column representing the genre of the music, and a song_title column representing the music name. Here, the artist_name column, the album_name column, and the release column are designated as compressed columns, and the artist_name column is used for LZ code compression for two pages, and the album_name column and release column are used for table structure definition when compressing four pages for Huffman code. The extension of the create table statement is shown in FIG. In this create table statement extension, when defining the artist_name column, album_name column, and release column, the data type definition is followed by “compress” indicating that it is a compressed column, and compression units “2” and “4”. The compression method “LZ” and “Huffman” are specified. Further, a compressed column management table 1803 in FIG. 18C manages a data identifier, a compressed column identifier, a compression unit, and a compression method. In other words, the compression value definition processing unit receives the compression value specified by the user, the identifier of the data including the value, the compression unit, and the compression method, and stores them in the compression value management table. Further, when processing the insert statement for a table including a compressed column in the procedure 704 of FIG. 7, the insert statement processing unit 511 refers to the compression value management table and requests the field requested in accordance with the compression unit and compression method specified there. Perform decompression processing and compression processing for insertion. Similarly, in the case of the select statement processing for a table including a compressed column in the procedure 1004 of FIG. 10, the select statement processing unit 520 refers to the compression value management table 508 and performs the expansion processing based on the designated compression unit and compression method. To output the requested field.

  By compressing data, it can be used on devices with severe storage capacity constraints. In addition, the amount of calculation can be reduced by specifying the column to be compressed by the user, and it can be used in a device with severe calculation capacity and memory restrictions. In particular, it is a relative relationship between the amount of data to be handled and the system scale for both computing capacity and memory constraints, and can be applied not only to portable devices but also to large-scale servers.

1 is a system configuration diagram of a computer system according to a first embodiment that implements a data management method involving data compression processing according to the present invention. Example 1 It is explanatory drawing which showed the component of the table | surface which is the management object of the Example of this invention. It is explanatory drawing which showed the user table in the said Example 1, the data definition sentence corresponding to it, and a compression column management table. It is explanatory drawing which showed the example which expanded the arrangement | sequence of the structure in the said Example 1. FIG. It is a system of the computer system of Example 2 which implements the data management method accompanied by the data compression process of this invention. It is explanatory drawing which showed the user table in the said Example 2, the data definition sentence corresponding to it, and a compression column management table. It is a flowchart which shows the processing flow at the time of the I / O distribution process part of the said Example 2 executing an insert statement processing command. It is a flowchart which shows the procedure of an example of the insert statement process which the insert statement processing part for tables containing the compression column of the said Example 2 performs. It is a flowchart which shows the procedure of the other example of the insert sentence process which the insert sentence process part for tables containing the said compression column performs. It is the flowchart which showed the processing flow at the time of the said I / O distribution processing part performing a select sentence processing command. It is a flowchart which shows the procedure of an example of the select sentence process which the select sentence process part for tables containing the compression sequence of the said Example 2 performs. It is a flowchart which shows the procedure of the other example of the select sentence process which the select sentence process part for tables containing the said compression string performs. It is the flowchart which showed the processing flow at the time of the inquiry process part of the said Example processing a delete sentence. It is a flowchart which shows the procedure of an example of the delete sentence process which the delete sentence process part for compression strings of the said Example 2 performs. It is a flowchart which shows the procedure of the other example of the delete sentence process which the said delete sentence process part for compressed strings performs. It is a conceptual diagram which shows the state of the table data before and after the maintenance process employ | adopted when performing the delete sentence process part of the said FIG. It is a conceptual diagram which shows the state of the table data before and after the maintenance process employ | adopted when performing the delete sentence process part of the said FIG. It is explanatory drawing which showed the user table in the modification of Example 2, the data definition sentence corresponding to it, and a compression column management table.

Explanation of symbols

101, 501 ... computer system,
102, 502 ... relational database execution unit,
103, 503 ... Storage,
104, 504 ... user data area,
105, 505 ... management information area,
106, 506, 603, 1803 ... compression value management table,
107, 507 ... inquiry processing unit,
108, 508 ... compression value definition processing unit,
109, 512 ... user,
110, 515 ... AP execution unit,
111, 516 ... operating system,
112, 517 ... CPU,
509 ... IO distribution processing unit,
510 ... insert statement processing unit for uncompressed table,
511 ... insert statement processing unit for a table including a compressed column,
513 ... delete statement processing unit for compressed columns,
514: Normal delete statement processing unit,
519 ... select statement processing unit for tables including compressed columns,
520: Select statement processing unit for uncompressed table.

Claims (8)

  A data management method for managing data by repeatedly performing data compression processing based on a compression value management table on a storage for data in which a set of multiple item values repeatedly appears.
  For each item in the data, accept the designation by the user of the presence or absence of compression and the compression unit,
  For each item designated as being compressed, the identifier of the item, the identifier of the data including the item, and the compression unit are stored in the compressed value management table,
  A data management method involving a data compression process in which only the data specified by the identifier of the item among the data is compressed and managed on the storage within a compression unit range designated.   When receiving a request to insert a set of values into the data, it is determined whether or not each item of the set of values is a compressed item with reference to the compressed value management table. A value within the compression unit range of the item stored in the compression value management table is read from the storage, the data compression process is executed, and the result is inserted. If not, the requested value is inserted as it is. The data management method with data compression processing according to claim 1.   When a request to read all or part of the data is received, it is determined whether or not the value to be read is an item with compression by referring to the compression value management table. 2. The data management method with data compression processing according to claim 1, wherein a value within the range of the compression unit of the item stored in the table is read from the storage and data decompression processing is performed.   A procedure for determining whether the value of the item to be compressed is included in the data requested to be deleted with reference to the compression value management table when receiving a request to delete all or part of the data; and The data management method according to claim 1, further comprising a procedure of performing data deletion processing specific to the compressed value when the value of the item to be compressed is included.   The compression process is a compression method in which a dictionary for each compression unit is created at the time of compression and the dictionary is stored at the beginning of the compression unit, and the data deletion process specific to the compression value is performed by calculating the compression value from the beginning of the compression unit. 5. The data management method according to claim 4, further comprising: a procedure of repeating a process of decompressing and deleting data to be deleted; and a procedure of attaching a deterioration flag to the dictionary of the compression unit.   The compression process is a compression method that does not involve the process of saving the dictionary created at the time of compression, and the data deletion process specific to the compression value is the data to be deleted if the compression value is expanded from the beginning of the compression unit. 5. The data management method according to claim 4, further comprising a step of repeatedly attaching the data deletion flag instead of deleting the data from the storage.   In the compressed value management table, in addition to the item identifier, the identifier of the data including the item, and the compression unit designation for each item designated as compressed, a compression method designation is stored, and the item of the data 2. A data management method involving data compression processing according to claim 1, wherein only the data specified by the identifier is managed on the storage by performing compression processing using a compression method specified within a specified compression unit range.   2. The compression unit is a size of a partition to which the data is subjected to compression processing, and is the size of the data, the number of rows, the number of fields, or the number of pages on the storage. Data management method described in 1.

Images