CN103412917B - The Database Systems of a kind of extendible polymorphic type FIELD Data coordinated management and management method - Google Patents

Abstract

An extensible database system and management method for coordinated management of multi-type domain data, including a data resource ontology library module, a hierarchical domain database module, a network-type domain database module, and a domain data evolution module, wherein the data resource ontology library module and multiple The species type database, the hierarchical domain database module and the network domain database module together form the database set. The present invention can build a large number of business-oriented data storage libraries, build an expandable data resource ontology library system on this basis, and quickly expand different types of domain-level databases and network-type domain databases. Extract new data objects from structured raw text data to construct new domain data.

Description

A database system and management method for scalable multi-type domain data coordinated management

technical field

The invention relates to an extensible database system and a management method for coordinated management of data in multiple types of fields, belonging to the fields of databases and artificial intelligence.

Background technique

A database is a warehouse that organizes, stores and manages data according to the data structure, and is a general data processing system for a unit or an application field. With the development of information technology and market, data management is no longer just storing and managing data, but has transformed into various data management methods required by users. There are many types of databases, ranging from the simplest tables storing various data to large database systems capable of storing massive amounts of data, which have been widely used in various aspects. With the acceleration of the informatization process and the arrival of the "big data" era, enterprise data tends to be more and more massive, unstructured and complex. The organic combination of two computer technologies, artificial intelligence and database, has promoted the intelligent development of database. General application program implicitly encodes the knowledge of problem solving in the program, while the system based on intelligent database explicitly expresses the problem solving elements in the application field, and forms a relatively independent program entity separately.

With the acceleration of the informatization process, the management of massive and complex data has attracted more and more attention from enterprises. However, in the process of resource management, enterprises often encounter the following problems: the storage and management of massive enterprise data is difficult; the search is slow and inefficient; Domain data version management is chaotic; data security is not guaranteed; databases in various domains cannot be effectively collaborated and shared. Therefore, to deal with the management of massive and complex unstructured data, an intelligent database that is scalable, evolvable, and collaboratively managed in multiple fields is required to store, process, and analyze these data.

Contents of the invention

The technical solution of the present invention is to solve the problems of efficient storage organization, indexing and query of unstructured data with multiple complex relationships, and provide an expandable database system and management method for coordinated management of multi-type field data.

The technical solution of the present invention: an expandable multi-type domain data coordinated management database system, including: a data resource ontology library module, a network-type domain database module, a hierarchical domain database module and a domain data evolution module, wherein:

The data resource ontology library module defines the top-level data resource model, realizes the logical view design and storage structure design of basic data units, provides basic support capabilities for data storage and access, and establishes a database containing a large number of business data objects, relationships and concepts; data resources Ontology library module provides top-level data abstraction rules and data access rules for network domain database module and hierarchical domain database module;

The network-type domain database module, according to the attributes of data objects, relational networks and other special attributes, builds a database based on network-type data attributes on the basis of the data resource ontology library, and realizes the data structure design, storage design and indexing of network-type data objects Design and form a relational network containing a large number of network-type data objects, and realize the external access interface of the network-type database; the network-type domain data module is the inheritance of the data resource ontology library and the instantiation realization on the network-type domain data; Provide users, other modules and external systems with query interfaces based on network-based domain data;

Hierarchical domain database module, according to the relationship characteristics of hierarchical data objects such as subordination, adjacency, intersection, and same level, builds a database that specifically represents data objects and their hierarchical affiliation related information, and realizes the external provision of data objects and their related information. The access interface of the hierarchical subordinate database; the hierarchical domain database module is a further evolution of the network domain data module, which stores and organizes domain data with only hierarchical structure in the form of a tree, realizes hierarchical semantics, and provides information to users, other modules and The external system provides a query interface based on hierarchical domain data;

The domain data evolution module tracks and controls the changes of the domain data in the data resource ontology library, network domain database and hierarchical domain database during the user's use, establishes the version history of the data object, and combines the original data set provided by the user with the Data is analyzed to obtain new domain data and input into the domain database through screening. The domain data evolution module provides record-based data version control for the above three libraries, and automatically discovers new domain data from the original data input by users , and use its interface for corresponding evolution management.

The data resource ontology library module includes a data persistence module, a bottom layer lexicon establishment module, a relationship definition module, a data index module and an interface module;

The data persistence module defines an interface-oriented implementation method, and flexibly configures data persistence implementation according to different hardware environments, context environments, and other requirements; based on object serialization technology, defines serialization and deserialization protocols for domain data-related objects , when the data is persisted, output the binary stream obtained by serializing the object to a file, database or network location through the file organization protocol; when it is necessary to load an object that has not been loaded into the object buffer pool, read it according to the logical address information sent by the upper layer request Take the corresponding data stream, and then reconstruct the object through the deserialization protocol of the object; the logical organization method of data in the file is the block storage method, and the block management adopts the heap structure for management. The data persistence module of the data resource ontology library is also the data persistence abstraction of the network domain database and the hierarchical domain database. The data storage functions of the latter two modules are customized and extended according to different persistence protocols based on the persistence module. Form a persistence library of a specific data type;

The underlying lexicon building module stores data objects without extended attributes and relationships, and establishes basic domain data objects, serialization protocols, deserialization protocols, and storage managers; the single data form of the underlying lexicon is network-type domain data and The definition of hierarchical domain data provides the data basis. The network domain database module and the hierarchical domain database module implement the serialization and deserialization interfaces defined here;

The relationship definition module, based on the implementation of the underlying lexicon, uses new files to establish synonymous, antisense, and subordinate relationships for entries in the underlying data object database; highly abstract and generalized relationship definition, organization, and storage And management, so that the network domain database can be flexibly expanded on this basis;

The data index module first defines the abstract of the domain data object, and maps the domain data abstract and the logical storage information of the domain data object through a fast double-coding algorithm to achieve the purpose of fast retrieval and access control; the network domain database module and Hierarchical domain database modules all contain an index part, in which keywords are realized by obtaining a pair of long integer numbers after double-coding calculations;

The interface module is implemented based on the EJB3.0 specification and published in the form of EJB interface and Web Service interface to realize cross-platform services. The network-type domain database and hierarchical domain database inherit the interface module of the data resource ontology library to realize the customized interface publishing function .

The implementation process of the network domain database module is as follows:

(1) In terms of storage design, based on the storage management layer of the data resource ontology library, define domain data objects and persistence protocols related to network domain data;

(2) Establish the storage layer on the basis of the defined network-type domain data objects, and define the basic structure and process of the attribute part. The attribute part is divided into two parts, one part is the attribute that exists when the database is designed, called the basic attribute; the other part is the user-defined attribute, called the extended attribute;

(3) Establish a data index on top of the data object storage structure in the network domain, based on the B-tree and Bloom Filter implementation with a fast buffer, dynamically generate a B-tree when inserting a network-type data object, and do not limit the B-tree The maximum number of layers. For the case of network data objects with the same name, the attribute blocks are linked together with pointers to form a list of attribute blocks. When the name is used as the keyword to query the network data object, the list of network data objects with the same name can be quickly obtained. ;

(4) After the data index is realized, for the update of data records, checkpoints and log files are implemented to ensure efficient access and high fault tolerance of the system.

The implementation process of the hierarchical domain database module is as follows:

(1) In terms of storage design, based on the storage management layer of the data resource ontology library, define the domain data objects and persistence protocols related to the hierarchical domain database;

(2) On the basis of the hierarchical domain data object and the extension of the hierarchical domain data structure to the persistence protocol, the storage design of the hierarchical structure is carried out;

(3) After the storage layer is completed, organize the relationship structure between hierarchical data objects through a limited binary tree. The keywords in the index file are composed of a double-coded number pair of a data object. Due to the uniqueness of this number pair, there is no need to consider the problem of conflicts. In the attribute file, the parent data object and the child data object are also stored using this number pair; when searching, by calculating the number pair of the data object, matching the same number pair in the index file, and obtaining the corresponding attribute pointer, read If there are multiple child data objects, all the subordinate data objects can be found by the pointer of the attribute pointing to the next attribute;

(4) After the index is completed, based on the checkpoint and log file functions of the network domain database module, a simplified log file suitable for the hierarchical structure is constructed.

The implementation process of the domain data evolution module is as follows:

(1) First collect user activity records of each database, and monitor changes in the activity level of domain data objects.

(2) Analyze the collected data object activity change data, and include domain data objects whose activity is lower than the system threshold into the alert backup database;

(3) Further analyze user activity records, and establish version change records for domain data objects whose core attributes have changed;

(4) The system also analyzes the text data provided by the user or on the Internet to build a huge data object analysis library; when new data is input into the data object analysis library, it will trigger the reading of the version information of the associated data object , and then by analyzing the relationship between the data object and the version of the associated data object, the probability that the current data object is new domain data is calculated, and the new domain data is automatically or manually modified by the user and added to the corresponding domain database.

An extensible database management method for coordinated management of multi-type domain data, the implementation steps are as follows:

(1) Preprocess the text data files provided by users, remove non-core field data including stop words, modal particles, and punctuation marks, and obtain preprocessed text data;

(2) Input the preprocessed data output in step (1) into the LDA probability model, match with the established data model, and obtain the domain-related data objects;

(3) Construct the suffix tree of the domain-related data objects output in step (2), merge the existing suffix tree, and traverse the merged suffix tree step by step to obtain high-frequency strings, and then initialize a domain related data objects;

(4) Input the field-related data objects obtained in step (3) into the data resource ontology library for type and relationship judgment and matching, and obtain the type of data objects related to the field, that is, hierarchical type, network type, or user-defined type, and other domain data objects associated with this data object;

(5) Input the domain-related data objects and their associated data output in step (4) into the corresponding type of domain database, create data change log records, and input the domain-related data objects into the double-encoding algorithm to obtain the corresponding index number I;

(6) Combine the number pair obtained in step (5) with the data objects and related domain data objects, and finally output domain data objects containing domain correlation, multiple relationships, and multiple attributes.

The advantage of the present invention compared with prior art is:

(1) The present invention is based on block storage, heap management and multi-log group storage technology to ensure the efficiency and security of the underlying storage;

(2) Based on the scalable design method, by customizing the data relationship, the domain database of multiple sub-domains can be expanded on the data resource ontology database;

(3) The present invention realizes the functions of automatic detection, analysis and extraction of a large amount of text data, and evolves the acquired data objects into new field data on the basis of the latest version of the data resource ontology library.

Description of drawings

Fig. 1 is a block diagram of the system of the present invention;

Fig. 2 is a schematic diagram of the concept relationship of the module of the data resource ontology library in the present invention;

Fig. 3 is a schematic diagram of the basic data model of triples in the data resource ontology library module of the present invention;

Fig. 4 is a schematic diagram of the index of the data resource ontology database of the present invention;

Fig. 5 is a schematic diagram of the access process of the data resource ontology library module of the present invention;

Fig. 6 is a schematic diagram of the process of defining a new field name in the data resource ontology library of the present invention;

Fig. 7 is a schematic diagram of the process of inserting a new relationship into the data resource ontology database of the present invention;

Fig. 8 is a schematic diagram of the process of retrieving data objects in the data resource ontology database of the present invention;

Fig. 9 is a schematic diagram of the basic structure and relationship of the network database of the present invention;

Fig. 10 is a schematic diagram of the basic structure and relationship of the hierarchical domain database of the present invention;

Fig. 11 is a schematic diagram of the query process of the hierarchical data domain database of the present invention;

Fig. 12 is a schematic diagram of the flow of new domain data objects in the hierarchical domain database of the present invention;

Fig. 13 is a representation diagram of the LDA model of the data evolution module in the field of the present invention;

Fig. 14 is a schematic diagram of the data evolution version control structure in the field of the present invention.

detailed description

The invention realizes effective storage and query of data objects and relationships through an efficient data storage organization method. And new data objects and domain data can be found through a large amount of data provided by users; data resource ontology database, network domain database, and hierarchical domain database can be realized, and domain database can be expanded.

The present invention includes a data resource ontology library module, a hierarchical domain database module, a network-type domain database module and a domain data evolution module, wherein the data resource ontology library module, the hierarchical domain database module and the network-type domain database module together form a database set, providing Carry out information extraction and data object retrieval, establish various related data object libraries required by the domain database system, and provide data storage and access basic support capabilities for multi-type domain data coordination management; domain data evolution module tracks and controls changes in domain data , to realize the evolution of the domain database itself and domain data version management.

1. Data resource ontology library module

Definition of attribute: The nature and relationship of a thing is called the attribute of this thing. The unique attributes of a certain type of things are abstracted from specific things, such as human voice and thinking are unique attributes. The accidental attribute of a certain type of thing is that some things of a certain type have attributes, but not all things have attributes. For example, the skin color and nationality of a person are all accidental attributes.

Definition of concept: A concept is a thinking form that reflects the unique attributes (inherent attributes or essential attributes) of things. Concepts are abstract and universal. Concepts can be divided into true and false, and a true concept is a concept that correctly reflects the unique attributes of things. The connotation of a concept is the unique attribute of the thing reflected by the concept. The extension of a concept is something that has the unique attributes reflected by the concept. The "is-a" relationship is generally used to indicate that a certain conceptual model is an extension of a certain conceptual model. A relationship is an extension of a concept. A role is also an extension of a concept. The concept requirements are clear, that is, it is required to clarify a concept from two aspects of connotation and extension. According to whether the extension of the concept is one thing or multiple things, the concept can be divided into individual concept and general concept. The extension of a single concept is a unique thing, such as a specific time and a specific space. The extension of the general concept can include many things. For example, concepts such as "city" and "commodity". "City" includes many specific cities, and "commodity" is also a conceptual collection of a large number of physical commodities. Both "is" and "is-a" are unique concepts. Concepts can also be divided into aggregate concepts and non-aggregate concepts. The aggregate concept is a concept that reflects an aggregate. A non-aggregate concept is a concept that does not reflect an aggregate. Concepts can also be divided into positive concepts and negative concepts. Positive concepts are concepts that reflect things with certain attributes. Negative concepts are concepts that reflect things that do not have a certain attribute. Concepts are also divided into relative concepts and absolute concepts. Relative concepts are concepts that reflect things that have a certain relationship. An absolute concept is a concept that reflects something with a certain quality.

The relationship between concepts has the following four basic relationships, as shown in Figure 2:

(1) Identity relationship: If all a is b, and b is a at the same time, then a and b have identity relationship. If two concepts have an identical relationship, then the extensions of the two concepts are the same.

(2) Subordination relationship: If all b are a, but there is a that does not belong to b, then the relationship between a and b is a superior relationship, and the relationship between b and a is a subordinate relationship.

(3) Cross relation: If some a is b, and some a is not b, and some b is not a, then a and b are cross relation.

(4) Disparate relationship: All a is not b, then a and b are disparate relations.

The basic data model of the triplet "Concept A-Relation-Concept B" is the basis of the domain database, which is the basic logical structure and the basic physical storage structure (Figure 3). "Concept" is the basic element in the domain database, which expresses people's cognition of things, and is a thinking form that reflects the unique attributes of things. A concept can be a real object, or a human imagination or design, that is, it can be an attribute or an activity. The logical meaning expressed by the triplet is the "relationship (Relation)" between "concept A" and "concept B", "concept A" in this triplet is the identity of the owner, "concept B" Within this triplet is the identity of the participant. "Concept A" and "Concept B" may also have or participate in other relationships. Thus, a very complex network structure is established in n concepts through relationships, and the complexity of this structure is determined by the user or the actual operating environment. "Concept" can be divided into "Class" (Class) and "Individual" (Individual) according to the level of abstraction. "Relationship" is used to establish a connection between concepts, which can be divided into "inter-class relationship", "class-individual relationship" and "inter-individual relationship". In data model design, all "relationships" are regarded as a "concept". Therefore, everything in the domain database is a "concept", and each "concept" has a unique identifier. All domain data can be expressed in this concept-relationship-concept way. If the data composition of a certain domain is very complex, then the structure of the domain data table is likely to be a very complex network structure, but the context But very clear. Many fields of data in the world are directly or indirectly related to each other. Through the expression structure of domain data and relations, it is easy to find related data. Unlike relational data structures, the structure of concept relations here can accommodate almost all The relationship and data of the database can modify and create new relationships and data at will, unlike relational databases. After the database design is completed, the relationship between data and data will not be changed. From this point of view, the conceptual relationship This method of data composition has incomparable advantages when performing model search or data mining.

1.1 Data resource ontology library structure description

In the ontology library of data resources, the ontology data stored in the library have the characteristics of clear concept, concise word form, and univocality. An ontology only expresses a concept, and it is mainly based on nouns and noun phrases. Some data objects have specific relationships, such as synonymous, antonymous, and affiliation. First, design an underlying database, which contains data objects without extended attributes and relationships. The underlying database provides services for realizing the three relationships of synonym, antonym and affiliation. Another important part is the memory index. Its structure includes object Hash value and disk block address; the object Hash value is a unique number. If two data objects conflict, that is, the Hash value is equal, they will be stored in the same block on the disk. Such a block can store multiple records.

Disk file: Divide the disk space of the specified size into one block; one block stores multiple records, and if the record exceeds the upper limit of the block, the pointer points to the next block. Through the program, the source data object file is written into the binary data file through mapping. In the data file, the data block is used as the unit, and a data block stores data objects with the same Hash value, that is, a Hash conflict occurs. This data block stores multiple attribute fields of the data object. If necessary, other fields can be added, such as adding the address field of its synonymous data object for the synonymous database. When looking for a data object, use the Hash function to calculate its Hash value. According to the mapping relationship, you can directly find the relative physical block number of the data object, and then hand it over to the database manager and object manager for data reading and unpacking . In this method, a database fast buffer is set in the middle, which avoids transferring a large number of data objects into memory, saves storage space, and has high access efficiency.

Independent data object: That is, the data object has no relationship with other data objects. When querying, only the data object is returned, and no other relationships are displayed. This part makes direct use of the underlying database, which does not implement relationships between data objects. The logical structure of domain data storage includes (the structure is shown in Figure 4):

(1) Index: A value calculated by the data object through the hash function, which is used to locate the address of the data object.

(2) Block address: the relative physical address of the block where the data object is stored.

(3) Data object record: the data object itself.

Data objects with relation: This relation includes synonymous relation, antonymous relation and affiliation relation. These relationships are implemented on the basis of the underlying database using the addresses of the data objects it provides.

The realization of the data object relationship is as follows: on the basis of the underlying database, the index table file and the record set file are respectively established. The index table file and the record set file are divided into continuous blocks of a specific size in the physical storage structure. Here, a block is referred to as a record, and each record corresponds to a unique number (that is, the first record number is 0, the second record record number 1, and so on). In the index table file, the index records include the number of the positive association group, the number of the reverse association group, the number of the lower data object group, and the relative physical address of the upper data object (these numbers refer to the numbers in the record set file). In the record set file, each record has two identifiers to record the previous record number and subsequent record number of this record, and can store a certain number of relative physical addresses of data objects. When a record space is used up, it can be used according to Its subsequent records need to be reassigned. When adding a positive association (reverse association/lower association) data object to a data object, first process the data object with the Hash function to generate a unique relative physical address of the data object, and then assign an index to the data object in the index table file Record, record the index record number of the data object at a specific location in the data block management layer according to the address of the data object. Next, allocate a record in the record set file, which is dedicated to saving the relative physical address of the synonym positive association address (reverse association/subordinate association) data object of this data object, and then write the newly allocated record number into the index record of this data object in the word. After that, add the relative physical address of the synonym positive association address (antisense reverse association/subordinate association) data object to the corresponding record in the recordset file. If adding a subordinate word-associated data object, you need to set the relative physical address of its superior word-associated data object in the index word record of the subordinate word-associated data object. After performing operations such as deleting positive associations, reverse associations, and lower-level associations, check the corresponding recordset file records and corresponding index records. If the corresponding record content is empty or the index record content is empty, record their Numbers, and release the relationship between them and the corresponding data objects, so that the free disk space can be recycled, greatly improving the utilization of disk space.

1.2 Define the database

The purpose of setting up this module is mainly to facilitate the management and expansion of the data object library. Only after a field name is defined, the data object in the data object library can add the field and its field value; if a field name is modified, the data object that owns the field in the library will automatically modify the field name to a new one The field name, but the corresponding field value remains unchanged; if a field name defined in the library is deleted, all data objects in the database that have the field will automatically delete the field and its field value. Similarly, after defining a relationship name, the data objects in the database can establish the relationship; if a relationship name is modified, the relationship will be automatically modified into a new relationship between the data objects in the database that own the relationship; if If you delete a relationship name already defined in the database, all data objects that have the relationship in the database will automatically cancel the relationship. For the default field names and relationship names defined in the database, operations such as modification and deletion cannot be performed. For user-defined field names and relationship names, users can perform operations such as adding, modifying, and deleting. Figure 5 shows the abstract architecture and access process of the data resource ontology library. Take the defined field name as an example,

Figure 6 is the execution flow for defining field names. Proceed as follows:

(1) The client interface service component sends a request for defining field names to the server service component.

(2) The server calls the method of the domain database manager to define the field name. The database manager calls the database access object to check whether the field name has been defined. The database access object returns the result of the check. If the database library has defined the field name, return the integer 0 to the server. The server returns the operation result 0 to the client, and the process ends.

(3) The database library manager sends a request to the database access object to read the old checkpoint in the record log (the position at the end of the valid segment of the record log).

(4) The database accessor returns the old checkpoint and assigns it to startPos. The database manager issues a request to the abstract data access object to read the current database valid backup version number. The database access object returns the request result, and the database manager assigns the returned result to the variable version. Copy the database registry, referred to as the new registry.

(5) If the field name recorded in the new registry is empty, directly add the field name to the new registry, and the field name code is 1; otherwise: read the current maximum field name code in the new registry. In the new registration form, if there is a free field name code i that is smaller than the current maximum field name code, this code is assigned to the newly added field name; otherwise, the current maximum field name code is automatically increased by 1. Assign the current maximum field name encoding to the newly added field name. Reset the current maximum field name encoding.

(6) Set the current maximum field name code in the registry to 1. Create a logging object. Assign values to each variable identifier of the log object, including the data it loads and the action to be performed, and then write the log record to the log file.

(7) Return the new valid checkpoint in the log file (referred to as the new checkpoint). Write a new checkpoint to the head of the log file. Write the newly written log record content in the log file to the data file (commit for short).

(8) Return the submission result. If there is an error or failure in the submission, the database manager will return the operation result -1 to the server and end the process. If the submission is successful: check the size of the log file, if it exceeds a certain size, the log file will be reset. The database manager returns the operation result 1 to the server. The server returns the request operation result 1 to the client, and the field name is defined successfully.

1.3 Manage data object information

It is mainly responsible for managing data object information, providing functions such as adding, deleting, and modifying data object information. When destroying a data object, all information of the data object (including the fields owned by the data object and its relationship with other words) is completely deleted from the database. There are three ways to add a database to the database: one is to directly add a data object without any information; the other is to add a field to a data object, if the data object does not exist in the database, the database The data object will be added automatically, and then new field information will be added for it; third, when establishing a relationship between two data objects, if one or both data objects do not exist in the database, the system will automatically Data objects that do not exist in the database are added to the library first, and then the corresponding relationships are established for them. When adding or modifying a field for a data object, the field name must be a field name already defined in the database; similarly, when establishing a relationship between two data objects, the relationship name must also be a relationship name already defined in the database. At the same time, the frequency and word frequency of data objects can be set through this module, and database files can be imported and exported. The following takes inserting a data object field as an example to briefly introduce the relevant operation process. FIG. 7 is a sequence diagram for inserting a data object field. The corresponding steps in the sequence diagram are as follows:

(1) The user service interface sends to the server to add a field to the keyword, and the field value is content.

(2) The server calls the method of the thesaurus manager to add fields and field contents to the data object.

(3) The database manager invokes the double coder to calculate the double codes of keywords.

(4) The double encoder returns the keyword double-encoded object key (referred to as the index key); if the returned result is empty, go to step 5; otherwise, go to step 7.

(5) The calculation of the double code for the keyword fails, and the data object manager returns the operation result -1 to the server.

(6) The server returns the operation result -1 to the client, declaring that the requested operation failed, and go to step 40.

(7) The data object manager sends an encoded request to the abstract data access object to obtain the field name in the registry.

(8) The database access object returns the code of the corresponding field name, if the return value is not empty (that is, the field name has been defined). Then go to step 11.

(9) The database manager returns the operation result 0 to the server, declaring that the field name fieldName is not defined in the database, and the field and its content (field value) cannot be inserted for keywords.

(10) The server returns the operation result 0 to the client, declaring that the thesaurus does not define the field name fieldName, and the requested operation fails.

(11) The database manager sends the index value of the index key key in the index table to the database access object.

(12) The database access object returns the index value mapped to the index key key. If the value is empty, it means that the database does not contain keywords, and go to step 13; otherwise, go to step 16.

(13) Add keywords to the database and return the added result (integer); if the addition fails, go to step 6, otherwise go to step 14.

(14) Send the index value of the index key key in the index table to the data repository access object again.

(15) The database access object returns the index value value of the index key key.

(16) The database manager sends a request to the database access object to read the old checkpoint in the record log (the position at the end of the valid segment of the record log).

(17) The database access object returns the old checkpoint, and the database manager assigns the returned result to the variable startPos.

(18) The database manager sends a request to the abstract data access object to read the effective backup version number of the current database.

(19) The database access object returns the read result, and the database manager assigns the returned result to the variable version.

(20) Copy the database registry and call it a new registry.

(21) The database manager sends the byte data information of the read keyword to the database access object.

(22) The database access object returns the byte data information of the keyword and the carrier of the disk address set (called a data cart).

(23) The database manager sends a request to the data processing factory to process the byte data of the keyword and convert it into a visual information object.

(24) The data processing factory returns the keyword content carrier to the database manager, and the database manager checks the keyword content carrier; if the field name and its corresponding field value to be added already exist, go to step 25, otherwise go to step 27 .

(25) The database manager returns the operation result 2 to the server, indicating that the content to be added already exists.

(26) The server returns the operation result 2 to the client, indicating that the content to be added already exists.

(27) Add fieldName and content to the keyword content carrier.

(28) The database manager sends to the data processing factory to process the data of keywords and convert them into byte data.

(29) The data processing factory returns the byte data of the keyword content to the database manager, and the database manager loads the returned byte data into the data cart.

(30) According to the data of the data car and the information of the new registry, the addresses of the required disk blocks are redistributed, and the information of the new registry is modified.

(31) Create a new log record object, and load the data cart, the new registry and the actions to be performed into the log record object.

(32) The database manager sends a request to the database access object to write the newly created logging object into the logging file.

(33) The database access object returns the new effective checkpoint of the log file (new checkpoint for short) to the database manager.

(34) The data manager sends a request to the database access object to write the new checkpoint into the head of the log file, and the database access object responds to the request.

(35) The database manager sends to the database access object to write the newly written log record content in the log file into the data file (submission for short).

(36) The database access object returns the submission result, if the submission is successful, go to step 38, otherwise go to step 37.

(37) If the submission fails or the above steps throw an exception, go to step 6.

(38) The database manager returns the operation result 1 to the server.

(39) The server returns the request operation result 1 to the client, and the field insertion is successful.

(40) END.

1.4 Retrieve data object information

The retrieval function is as follows:

(1) Check the existence of data objects: check whether a data object exists in the database;

(2) Retrieve data packets of data objects: retrieve all visualized data information (fields, field values, relations, relational words) of data objects, and encapsulate them into data packets for network or other forms of transmission;

(3) Retrieve the field value of the data object: retrieve the field value (field content) of a certain field of the data object;

(4) Retrieve data relational words: retrieve all relational words of a certain relation of data objects;

(5) Search by field name: It is divided into search by single field and search by combination of double fields. Retrieving by single field refers to retrieving data packets of all data objects with a certain field; retrieving by combination of double fields refers to retrieving data packets of all data objects having two fields at the same time;

(6) Retrieve by relation name: retrieve the data packets of all data objects with a certain relation;

(7) Backward matching retrieval: Retrieve the data packets of all data objects headed by a certain keyword;

(8) Fuzzy matching retrieval: Retrieve the data packets of all data objects containing a certain keyword;

(9) Retrieve high-frequency words: retrieve a specified number of data objects or data object packets according to the frequency from high to low;

(10) Retrieve low-frequency words: retrieve all data objects whose word frequency is lower than a certain frequency;

(11) Retrieval frequency Term frequency: the frequency of retrieving a certain data object (number of times retrieved);

(12) Retrieve all field names defined in the database;

(13) Retrieve all relationship names defined in the database.

Figure 8 shows a sequence diagram for retrieving a data object packet. Introduction to the sequence diagram:

(1) The client sends a request to the server to retrieve the data packet of the data object of the keyword.

(2) The server calls the method of the database retriever to retrieve the data object packet.

(3) The database manager invokes the double coder to calculate the double codes of keywords.

(4) The dual encoder returns the keyword double-encoded object key (referred to as the index key).

(5) The database retriever sends a request to the database access object to obtain the index value corresponding to the key in the index table.

(6) The database access object returns the index value mapped to the key to the database retriever. If the value is empty, it means that the keyword does not exist in the database, and go to step 7; otherwise, go to step 9.

(7) The database retriever returns the retrieval result null to the server.

(8) The server returns the retrieval result null to the client, go to step 14.

(9) Add 1 to the frequency of keywords, and then the database retriever sends a request to the database access object to update the frequency of keywords in the database index table, and the database access object automatically responds to the request.

(10) The database retriever sends a request to the database access object to update the frequency of keywords in the data file, and the database access object automatically responds to the request.

(11) The database retriever invokes its own method to retrieve the data packet of the keyword according to the first address of the keyword on the disk.

(12) The database retriever returns a packet of keywords to the server.

(13) The server returns the keyword packet to the client.

(14) END.

2. Network domain database module:

2.1 Basic Principles

Each network-type domain data collection is divided into an index part and an attribute part. The index part is stored in the name.dct file, and the attribute part is stored in the attr.dct file. The name index part of network data is an unlimited B-tree dynamically generated when inserting data objects.

Pointer operations are used in the index file, so N pointers are defined, which correspond to N commonly used Chinese characters in the GB2312-80 encoding. The pointer points to the root of the B-tree where the names starting with this word are located. All names starting with the same character are stored in the same B-tree.

When retrieving, you can use the name as a keyword to retrieve network-type data. The keyword is calculated by using the GB2312-80 code of the name through the Hash function. When searching, first find the B-tree, and then use the B-tree search algorithm to find the name. There is a one-to-one correspondence between the names in the index file and the attributes in the attribute file, that is, if the abstract information of the data object is found in the index file, the attribute of the network data object must exist in the attribute file. In the index file, the summary information is used as a keyword (key) to search, and the position of the network-type data attribute corresponding to the summary information in the property file is used as the search result. When the summary information is found in the index file, relevant properties can be directly read from the property file according to the corresponding property index before the summary information. Therefore, the operation on the attribute file is very fast, and the time is mainly spent on searching the index file. In the index file, Hash algorithm and B-tree algorithm are used to store and search summary information. This algorithm is based on hard disk retrieval. At the same time, the addressing operation is directly searched according to the pointer, so the algorithm is more efficient.

2.2 Index storage structure

In the index file, Hash algorithm and B-tree algorithm are used to store and search the summary information of network data. Here, the first character of the summary information is called "initial character", and the rest after removing the first character is called "suffix". First, create a location table with N entries in the index file, and each entry consists of a single character and its GB2312-80 value. The characters in each table entry only need to calculate its key value through the Hash function to obtain its address in the location table. The entry also stores a pointer to the root of the B-tree. The B-tree is used to store the suffix of the summary information.

When storing the name of network-type data, first find the B-tree corresponding to the word in the location table according to the first word of the summary information, and then insert its suffix into the B-tree. The search process of the summary information is similar to the storage. Firstly, according to the first word of the summary information, the corresponding B-tree is found in the location table, and the suffix of the summary information is searched in the B-tree.

B-tree structure: The B-tree is used to store the suffix of the summary information, and the suffix is stored as a key in the B-tree. In order to reduce the number of disk reads, n-order B-trees are used according to actual needs. Each node in the B-tree contains the following information:

(n,C ₀ ,A ₁ ,K ₁ ,C ₁ ,A ₂ ,K ₂ ,C ₂ ,…,A _n ,K _n ,C _n ,Father)

Where n is the number of keywords in the node; K _i (i=1,...,n) is the keyword (suffix of summary information), and K _i <K _i+1 (i=1,..... ,n); C _i (i=1,…..,n) is a pointer to the root node of the subtree, and the keywords in the subtree pointed to by the pointer C _i-1 are all smaller than K _i (i=1,… .., _n ), the keywords of all nodes in the subtree pointed to by C _n are greater than K _n ; The corresponding character is the position in the attribute file of the abstract information attribute with the first word and the suffix K _i ; Father is the pointer to the parent node.

When looking for a data object, first calculate the entry address of the word in the location table through Hash according to the first word of the given summary information, and then read the content of the entry to find the B-tree root address corresponding to the word , and then look up the suffix of the summary information in the B-tree. The time used for a search is the time of Hash calculation plus the time of B-tree search, so the retrieval algorithm is more efficient. Using the memory mapping technology, it is not necessary to read the index file into the memory, but only to read the used nodes into the memory, which greatly reduces the disk reading time and improves the memory utilization.

2.3 Attribute storage structure of network data

Except for the name, the properties of the network data object are all stored in the property file. The property is divided into two parts, one part is the property that exists when the database is designed, called the basic property, which is stored in the basic property file; the other part is the user Custom properties, called extended properties, are stored in extended properties files. The basic attributes of network data objects are stored in the form of attribute blocks. The attributes of a data object are stored in an attribute block, and the basic attribute blocks of data objects are stored in the basic attribute file in sequence according to the insertion order of the data objects. The data object extended attributes are saved in the form of linked list. The storage structure of the property file is shown in Figure 9. The basic attribute block is composed of the basic attribute block of the data object, pointer a, pointer b, etc. In the basic attribute block, the pointer a points to the attribute block with the same name, and the pointer b points to the first address of the extended attribute of the data object in the extended attribute file.

When looking for the attribute of a network data object, first use the name of the data object to search in the index file, and at the same time find the name of the data object, you can find the location pointer of the attribute block of the data object in the attribute file in the node storing the name, according to The pointer directly goes to the corresponding address of the attribute file to read the basic attribute, and then reads the extended attribute in the extended attribute file according to the relevant pointer in the attribute block. Therefore, the retrieval efficiency in the property file is very high.

3. Hierarchical domain database module:

3.1 Basic Principles

According to the characteristics of the relationship between hierarchical data, it can be known that there are mainly four relationships among hierarchical data: membership, adjacency, crossover, and the same reference. Among them, the subordination relationship is the main relationship. A hierarchical data object may have both superior and There may be lower levels. It is stipulated here that each hierarchical data object is directly responsible to the upper level, or leads the lower level. This design is based on the consideration that there are many basic master attributes in hierarchical data that are completely equal.

For each hierarchical data object, it can be used as a database key to uniquely identify the data object. The key is a pair of numbers calculated by a double-encoded Hash function as the logical address of the data object. That is, the key of each data object corresponds to its storage address on the disk. To find a certain data object, as long as its key value is calculated through the Hash function, it is equivalent to obtaining the logical address of the data object, and then The data access task is handed over to the object manager and the log manager to complete. This method avoids searching and matching, and the time consumption is mainly in the calculation of the Hash value, and it is not necessary to load all the data blocks into the memory, as long as the required data objects are read into the memory, regardless of the execution efficiency of the algorithm , or in the utilization of memory space, it is feasible. At the same time, because the addressing is directly searched according to the pointer, the retrieval efficiency of the data object is extremely high.

3.2 Structure of the index file

The index file structure mainly includes the following fields, and their functions are described as follows:

Key is the key value calculated by Hash, which is unique for each data object;

·Father/Son field indicates the affiliation relationship, Father indicates the Key of the upper-level hierarchical data object, and Son indicates the Key of the lower-level hierarchical data object;

The Neighbor field represents the adjacent relationship, and the neighbor represents the Key of the adjacent hierarchical data object in the relational structure, and this field may have more than one Key;

The Cross field indicates a cross relationship, and the cross indicates the Key of a hierarchical data object with a cross relationship in the relationship structure, and the field may also have more than one Key;

The Co-ref domain indicates the same-referring relationship, and the co-ref indicates that they all refer to hierarchical data objects in the same domain in terms of semantic understanding, and the domain may also have more than one Key;

·The 0/1 field indicates that the hierarchical data object may have the same name, that is, the name is the same, but it has two completely different meanings. If it is set to 0, it means that there is no duplicate name, and if it is 1, it means that there is a duplicate name, and the Fathers field immediately following it records all the upper-level data objects that contain this data object;

·Fathers field records all the upper-level data objects that contain this hierarchical data object when there is a phenomenon of duplicate names, so this field may also have more than one Key. NULL if there is no duplicate name.

Each of the above fields occupies four bytes.

3.3 Structure of data files

The data file is the file that stores the hierarchical data object itself, and the combination of it and the index file can realize the access operation of the data object. The data file is a linear table of data objects in the data logic, and the entries in the linear table—the data entries establish the connection with the fields in the structure through pointers, as shown in Figure 10. Among them, W _i and W _j are entries of a hierarchical data object, and the pointers of Father, Son, Neighbor, Cross, Co-ref, and Fathers fields point to the upper-level data object and the lower-level data object of the data object entry respectively. , adjacent data objects, intersecting data objects, same-referring data objects, and all the upper-level data objects of this data object when the basic attributes are completely equal. The two data objects can be distinguished by the superordinate data object.

When a data object needs to be searched, as long as the address of the data object is obtained through Hash calculation and loaded into the memory, the organizational structure of the data object and other related data objects can be easily constructed. Moreover, there is no need for search and matching processes, and the entire search time is mainly spent on Hash calculations. The algorithm has extremely high time efficiency, and the filling factor of Hash is above 0.8.

Figure 11 and Figure 12 show the related business algorithm process of the hierarchical domain database.

4. Domain data evolution module:

4.1 Information extraction based on LDA

The first step in domain data evolution is to analyze and process a large amount of valuable text information. This system adopts the text clustering mining technology based on the LDA (Latent Dirichlet Allocation) probability generation model, which helps to find relevant field data by automatically aggregating similar texts in the text set into different categories. The text is represented by the vector space model, and the text representation matrix usually has a high dimensionality, and the similarity measurement loses its meaning due to the "curse of dimensionality" in the clustering process. The LDA topic model has a good text representation ability, can mine the potential semantic information of the text, obtain the representation of the document in the topic space, and reduce the dimension of the document representation. By modeling the text, we can perform feature selection, topic classification, and similarity judgment on the text. The LDA model adopts the bag-of-words method, which treats each text data resource as a word frequency vector, thereby converting text information into digital information that is easy to model.

The three-layer Bayesian model representation of the LDA model is shown in Figure 13. Φ _k represents the term probability distribution in topic K, θ _m represents the topic probability distribution of the mth document, and Φ _k and θ _m are used as parameters of multinomial distribution to generate topics and words respectively. K represents the number of topics, M represents the number of documents, N _m represents the document length of the m-th document, ω _m,n and Z _m,n represent the n-th word and its topic in the m-th document, respectively. α and β are the parameters of the Dirichlet distribution, which are usually fixed values and symmetrically distributed, so they are represented by scalars. Both Φ _k and θ _m obey the Dirichlet distribution, and the distribution function is shown in the following formula:

$\mathrm{Dir}\left(\mathrm{\&mu;}\right|\mathrm{\&alpha;})=\frac{\mathrm{\&Gamma;}\left({\mathrm{\&alpha;}}_0\right)}{\mathrm{\&Gamma;}\left({\mathrm{\&alpha;}}_1\right)...\mathrm{\&Gamma;}\left({\mathrm{\&alpha;}}_k\right)}\underset{k=1}{\overset{K}{\mathrm{\&Pi;}}}{{\mathrm{\&mu;}}_k^{{\mathrm{\&alpha;}}_k}}^{-1}$ (Formula 1)

Among them, 0≤μ _k ≤1, Γ is the gamma function. The generation process of LDA is as follows.

(a) For topic sampling

(b) For the mth document in the corpus, m∈[1,M];

(c) Sampling topic probability distribution θ _m ~Dir(α);

(d) Use document length N _m ~Poiss(ξ);

(e) For the nth word in document m, n∈[1,N _m ];

(f) Select the hidden topic z _m,n ~Mult(θ _m );

(g) generate words

The parameter estimation of LDA first calculates the conditional probability of the topic sequence under the word sequence, the formula is as follows:

$p\left(z\right|w)=\frac{p(w,z)}{\underset{z}{\mathrm{\&Sigma;}}p(w,z)}$ (Formula 2)

Then perform Gibbs sampling on the topic sequence, and the sampling formula is as follows:

$p(z_i=k|z...i,w)\&Proportional;\frac{{\mathrm{no}}_{k,...,i}^{\left(t\right)}+{\mathrm{\&beta;}}_t}{[\underset{\mathrm{\&upsi;}=1}{\overset{V}{\mathrm{\&Sigma;}}}{\mathrm{no}}_k^{\left(\mathrm{\&upsi;}\right)}+{\mathrm{\&beta;}}_{\mathrm{\&upsi;}}]-1}\&Center\; Dot;\frac{{\mathrm{no}}_{m,...,i}^{\left(k\right)}+{\mathrm{\&alpha;}}_k}{[\underset{z=1}{\overset{K}{\mathrm{\&Sigma;}}}{\mathrm{no}}_m^{\left(z\right)}+{\mathrm{\&alpha;}}_z]-1}$ (Formula 3)

Obtain the label of the topic z of each word ω, and the final parameter calculation formula is expressed as follows:

(Formula 4)

${\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; no</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}}{\underset{\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; K</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; no</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}}$

The model M that has been trained is given to the new document The hidden topic sampling formula of each word is as follows:

$p({\tilde{z}}_t=k{|\widetilde{\mathrm{\&omega;}}}_i=t,{\tilde{z}}_{\&Right\; Arrow;i},{\widetilde{\mathrm{\&omega;}}}_{\&Right\; Arrow;i};m)=\frac{{\mathrm{no}}_k^{\left(t\right)}+{\mathrm{no}}_{k,\&Right\; Arrow;i}^{\left(t\right)}+{\mathrm{\&beta;}}_t}{\underset{\mathrm{\&upsi;}=1}{\overset{V}{\mathrm{\&Sigma;}}}{\mathrm{no}}_k^{\left(\mathrm{\&upsi;}\right)}+{\widetilde{\mathrm{no}}}_{k,\&Right\; Arrow;i}^{\left(\mathrm{\&upsi;}\right)}+{\mathrm{\&beta;}}_{\mathrm{\&upsi;}}}\&Center\; Dot;\frac{{\mathrm{no}}_{{\tilde{m}}_{,\&Right\; Arrow;i}}^{\left(k\right)}+{\mathrm{\&alpha;}}_k}{[\underset{z=1}{\overset{K}{\mathrm{\&Sigma;}}}{\mathrm{no}}_{\tilde{m}}^{\left(z\right)}+{\mathrm{\&alpha;}}_z]-1}$ (Formula 5)

in, represents a new document Corresponding subject vectors.

Through the above-mentioned Gibbs sampling method, the topic label of each word is obtained, and the value of each topic component of the document is calculated using formula 6, and the document of an item space is represented in the topic space.

${\mathrm{\&theta;}}_{\tilde{m},k}=\frac{{\mathrm{no}}_{\tilde{m}}^{\left(k\right)}+{\mathrm{\&alpha;}}_k}{\underset{z=1}{\overset{K}{\mathrm{\&Sigma;}}}{\mathrm{no}}_{\tilde{m}}^{\left(z\right)}+{\mathrm{\&alpha;}}_z}$ (Formula 6)

After the above steps, the clustering process can be performed. After using LDA to select a certain proportion of features, the K-means algorithm is used to cluster the text. The text clustering process is as follows:

(1) Preprocessing the original text, including word segmentation and removing stop words;

(2) Use the LDA model for feature selection;

(3) For the selected features, count the weight of each feature in each text, and the calculation formula of the weight W(d,w) of the feature in the text is as follows:

$W(d,w)=\frac{\log (\mathrm{tf}(d,w)+1)\&times;\log ((m+1)/(\mathrm{df}\left(w\right)+0.5))}{\mathrm{\&Sigma;}\log (\mathrm{tf}(d,w^{\&prime;})+1)\&times;\log ((m+1)/(\mathrm{df}\left(w^{\&prime;}\right)+0.5))}$ (Formula 7)

Among them, M is the total number of texts, tf(d,w) is the number of occurrences of word element w in text d, and df(w) is the text frequency of word element w. Once the representation of the text is obtained, a vector space model can be generated.

(4) Randomly select the initial point, and use the K-means algorithm to obtain the final clustering result. Among them, the K-means clustering algorithm needs to measure the distance between texts, and uses cosine similarity to calculate. For two texts d and d′, their similarity calculation formula is as follows:

$\mathrm{sim}(d,d^{\&prime;})=\frac{\underset{w\&Element;d,d^{\&prime;}}{\mathrm{\&Sigma;}}W(d,w)\&times;W(d^{\&prime;},w)}{d\&times;d^{\&prime;}}$ (Formula Eight)

4.2 Domain database version control

The domain database version control module refers to the theory and method of version control to realize the management and control of the evolution process of the database. Each evolution state of data can be regarded as a version. This module provides functions such as version generation, version restoration, and version deletion. Specifically, due to factors such as modification and evolution of the domain database, the domain database will continue to evolve with time, and the functions of this module record this series of evolution processes. On the one hand, it records the evolution process of data in a specific field, allowing users to view it at any time and restore data in a certain field to a certain version in the past. On the other hand, users can also mark a database in a certain state as a version, so that it can Sometime in the future reverts the entire database to this version. Users can delete a non-critical version when needed. Its structure is shown in Figure 14.

4.3 New Data Object Discovery

The discovery of new data objects requires the user to provide a large amount of basic text data. The system analyzes these data through the above-mentioned LDA-based analysis model to build a huge data object analysis library; when new data is input into the data object analysis library, it will trigger Read the version information of the associated domain data, and then calculate the possibility that the current data object is new domain data by analyzing the relationship between the data object and the associated domain data version, and automatically or manually update the new domain The data is modified and added to the domain database. The core structure of the data object analysis library is a suffix tree. Another important part of this part is the directory monitoring module, which is used for the system to automatically sense the arrival of new data, and then automatically perform evolution processing. The processing method is as follows:

(1) The system is started, and the configuration file is checked to obtain the domain data evolution data source directory.

(2) Start the directory monitor (AutoDector) to monitor the status changes of the data source directory. When a new file is added, the directory monitor will detect the change, then check its file format, and if it is one of text file, PDF file, HTML file and Word document, it will be read and analyzed.

(3) According to different types of input files, different file analyzers are implemented: TxtAnalyzer, PdfAnalyzer, HtmlAnalyzer, WordAnalyzer. Among them, PdfAnalyzer and WordAnalyzer are implemented using the open source tool Apache POI. After passing through the file analyzer, the text or text stream is obtained (when the amount of data is huge, the text stream is returned).

(4) Input the obtained text or text stream into the analysis library, that is, insert it into the latest suffix tree. When the suffix tree changes, it will trigger the check of the relevant changed entries: after the frequency of the entry reaches the threshold t, query the field data related to the entry from the data resource ontology database, and determine whether to Build new base domain data.

(5) After analyzing the file, rename the file to a file ending with ".analyzed" to distinguish it from unanalyzed files. After that, check the metadata files in the data source directory, and delete some of the earliest analyzed files if the number or size of the currently analyzed files reaches the upper limit.

The contents not described in detail in the description of the present invention belong to the prior art known to those skilled in the art.

The above is only a preferred embodiment of the present invention, it should be pointed out that, for those of ordinary skill in the art, without departing from the principle of the present invention, some improvements and modifications can also be made, and these improvements and modifications can also be made. It should be regarded as the protection scope of the present invention.

Claims (6)

1. An extensible database system for multi-type field data coordination management, comprising: data resource ontology library module, network type domain database module, hierarchical type domain database module and domain data evolution module, wherein:

the data resource ontology library module is used for defining a top-level data resource model, realizing logic view design and storage structure design of basic data units, providing data storage and access basic support capacity and establishing a database containing a large number of business data objects, relations and concepts; the data resource ontology library module provides top-level data abstraction rules and data access rules for the network type domain database module and the hierarchical type domain database module;

the network type domain database module is used for constructing a database based on network type data attributes on the basis of a data resource ontology base according to the attributes of the data objects, a relational network and other special attributes, realizing the data structure design, storage design and index design of the network type data objects, forming a relational network containing a large number of network type data objects and realizing the provision of a network type database access interface to the outside; the network type domain database module is used for realizing inheritance of a data resource ontology base and instantiation on network type domain data; providing a query interface based on network type field data for a user, other modules and an external system; the hierarchical domain database module is used for constructing a database specially representing data objects and related information of hierarchical membership thereof according to the characteristics of membership, adjacency, intersection and peer relationship among hierarchical data objects, and realizing an access interface for providing the data objects and the hierarchical membership database thereof to the outside; the hierarchical domain database module is used for further evolving the network domain database module, storing and organizing the data only having the hierarchical structure domain in a tree form, realizing hierarchical semantics, and providing a query interface based on the hierarchical domain data for a user, other modules and an external system;

the domain data evolution module tracks and controls the change of the domain data in the data resource body base, the network type domain database and the hierarchical domain database in the using process of the user, establishes data version history, analyzes an original data set provided by the user in combination with the existing data to obtain new domain data and inputs the new domain data into the domain database through screening, provides record-based data version control for the data resource body base module, the network type domain database module and the hierarchical domain database module, automatically discovers the new domain data from the original data input by the user, and uses an interface thereof to carry out corresponding evolution management.

2. The scalable multi-domain data orchestration management database system according to claim 1, wherein: the data resource ontology library module comprises a data persistence module, a bottom database establishing module, a relation defining module, a data index module and an interface module;

the data persistence module defines an interface-oriented implementation method and flexibly configures data persistence according to different hardware environments, context environments and other requirements; defining serialization and deserialization protocols of field data related objects based on an object serialization technology, and outputting binary streams obtained after object serialization to a file, a database or a network position through a file organization protocol during data persistence; when an object which is not loaded in the object buffer pool needs to be loaded, reading a corresponding data stream according to the logic address information sent by an upper layer request, and reconstructing the object through an deserialization protocol of the object; the logical organization mode of the data in the file is a block storage mode, and the management of the blocks adopts a heap structure for management; the data persistence module of the data resource ontology base is also a data persistence abstraction of the network type domain database and the hierarchical type domain database, and the data storage functions of the network type domain database and the hierarchical type domain database are customized and expanded based on the persistence module according to different persistence protocols to form a persistence base of a specific data type;

the bottom database establishing module is used for storing data object data without extended attributes and relations, and establishing a basic field data object, a serialization protocol, an anti-serialization protocol and a storage manager; the single data form of the bottom database provides a data basis for the definition of the network type field data and the hierarchical type field data; the network type domain database module and the hierarchical type domain database module realize defined serialization and deserialization interfaces;

the relation definition module is used for establishing a synonymy relation, an antisense relation and a membership relation for the entries of the bottom database by using a new file on the basis of the realization of the bottom database; the highly abstract and generalized relation definition, organization, storage and management enable the network type domain database to realize flexible extension on the basis;

the data index module is used for performing abstract definition on the field data object, and mapping the field data abstract and the logic storage information of the field data object through a quick double-coding algorithm so as to achieve the purposes of quick retrieval and access control; the network type domain database module and the hierarchical type domain database module both comprise index parts, wherein keywords are realized by obtaining a long and integer digital pair through double-coding calculation;

the interface module is realized based on EJB3.0 standard, and is published in the form of EJB interface and Web Service interface to realize cross-platform Service, and the network type domain database and the hierarchical type domain database realize the customized interface publishing function by inheriting the data resource ontology interface module.

3. The scalable multi-domain data orchestration management database system according to claim 1, wherein: the network type domain database module is realized by the following steps:

(1) defining a domain data object and a persistence protocol related to the network type domain data on the basis of a storage management layer of the data resource ontology library in storage design;

(2) performing storage design on the basis of a defined network type field data object, and firstly defining a basic structure and a process of an attribute part; dividing an attribute part into two parts, wherein one part is an existing attribute during database design and is called a basic attribute; the other part is a user-defined attribute which is called an extended attribute;

(3) establishing a data index on a network type domain data object storage structure, dynamically generating a B tree when inserting a network type data object based on the B tree with a fast buffer and a Bloom Filter, and not limiting the maximum layer number of the B tree, connecting attribute blocks together by using pointers to form an attribute block linked list aiming at the condition that the network type data object has the same keyword, and quickly obtaining a network type data object list with the same keyword when inquiring the network type data object by using the keyword;

(4) after the data index is realized, the data record is updated according to the updating of the data record, and the data update is recorded in the finest granularity through the check point and the log file, so that the high-efficiency access and the high fault tolerance of the system are guaranteed.

4. The scalable multi-domain data orchestration management database system according to claim 1, wherein: the hierarchical domain database module is realized by the following steps:

(1) defining a domain data object and a persistence protocol related to a hierarchical domain database on the basis of a storage management layer of the data resource ontology base in storage design;

(2) on the basis of the extension of the hierarchical domain data object and the persistent protocol based on the hierarchical domain data structure, the storage establishment of the hierarchical structure is carried out;

(3) after the storage and establishment of the hierarchical structure are completed, organizing a relationship structure between hierarchical data objects through a defined binary tree; the key words in the index file are formed by unique double-code number pairs of data objects, and the problem of conflict does not need to be considered; in the attribute file, storing the hierarchical data of the father data object and the son data object by adopting the even number; during retrieval, the number pairs of the data objects are calculated, the same number pairs are matched in the index file, pointers of corresponding attributes are obtained, the attributes are read out, and if a plurality of sub-data objects exist, all lower-level data objects can be found by the pointers of the attributes pointing to the next attribute;

(4) after the index is finished, a simplified log file suitable for a hierarchical structure is constructed on the basis of the functions of the check point and the log file of the network type domain database module.

5. The scalable multi-domain data orchestration management database system according to claim 1, wherein: the field data evolution module is realized by the following steps:

(1) firstly, collecting user activity records of various domain databases, and monitoring the change of activity degree of domain data objects;

(2) analyzing the activity change data of the collected data objects, and bringing the domain data objects with the activity lower than a system threshold value into a guard backup library;

(3) further analyzing the activity record of the user, and establishing a version change record of the data object for the field data object with the changed core attribute;

(4) the system analyzes the text data provided by the user or on the Internet to construct a huge data analysis library; when new data is input into the data analysis base, the version information of the associated data object is triggered to be read, then the probability that the current data object is new domain data is calculated by analyzing the relation between the data object and the associated data object version, the new domain data is automatically or manually modified by a user and added into the corresponding domain database.

6. An extensible database management method for data coordination management in multiple types of fields is characterized by comprising the following implementation steps:

(1) preprocessing text data provided by a user, removing non-core field data including stop words, tone words and punctuation marks to obtain preprocessed text data;

(2) inputting the preprocessed data output in the step (1) into an LDA (latent Dirichlet allocation) probability model, and matching the preprocessed data with the established data model to obtain a field-related data object;

(3) constructing a Suffix Tree (Suffix Tree) on the field-related data object output in the step (2), fusing the existing Suffix Tree, gradually traversing the merged Suffix Tree to obtain a high-frequency character string, and initializing a field-related data object;

(4) inputting the field-related data object obtained in the step (3) into a data resource ontology base for type and relationship judgment and matching, and obtaining the type of the field-related data object, namely the hierarchical type, the network type or the user-defined type, and other field data objects related to the data object;

(5) inputting the field-related data object and the associated data output in the step (4) into a field database of a corresponding type, establishing a data change log record, and inputting the field-related data object into a double-coding algorithm to obtain a corresponding index even;

(6) and (4) carrying out service combination on the number pairs obtained in the step (5), the data objects and the related field data objects, and finally outputting the field data objects containing field correlation, multiple relations and multiple attributes.