CN114911787B - Multi-source POI data cleaning method integrating position and semantic constraint - Google Patents

Abstract

The invention relates to a multi-source POI data cleaning method that integrates location and semantic constraints, and belongs to the field of data processing technology. This method performs the following steps: Step 1, perform GeoHash conversion on the collected multi-source POI data; Step 2, perform adjacent point query on the converted string; Step 3, perform redundancy on the windows with adjacent points in Step 2 Processing; Step 4, construct a word segmentation scheme; Step 5, perform redundant processing on the data processed in Step 4; Step 6, complete POI data rematching based on the word frequency statistics of the word segmentation scheme reconstructed in Step 5. This method can complete the data cleaning work more accurately and efficiently, and the cleaning results are better, more realistic and effective.

Description

A multi-source POI data cleaning method integrating location and semantic constraints

Technical field

The invention relates to a multi-source POI data cleaning method that integrates location and semantic constraints, and belongs to the field of data processing technology.

Background technique

With the continuous emergence of new information publishing methods represented by blogs, social networks, and location-based services (LBS), as well as the rise of technologies such as cloud computing and the Internet of Things, data is growing and accumulating at an unprecedented rate. Constantly try to dig out the hidden information under big data. However, as the amount of data increases significantly, the quality of data is also declining. In the big data environment, various types of data from heterogeneous systems have several problems: ① Complexity, the data of each application system lacks a unified standard definition, and has great inconsistency. ② Repeatability, there are two or more identical physical descriptions of the same objective thing in the database. ③Fuzziness, due to defects in system design and some human factors during use, leads to the phenomenon of uncertain loss of attribute values in data records.

It is precisely because of the above situation that data cleaning plays an increasingly important role in the process of data analysis and management. Data cleaning aims to identify and correct noise in data to minimize the impact of noise on data analysis results. As a component of big data, POI is an important carrier of location services and directly controls the quality of location services research. In order to obtain more and more comprehensive POI data, researchers/technicians try to obtain data from multiple data sources, but this brings about problems such as the increase of redundant data and the emergence of incomplete data.

Contents of the invention

The technical problem to be solved by this invention is: how to provide a multi-source POI data cleaning method.

In order to solve the above technical problems, the technical solution proposed by the present invention is: a multi-source POI data cleaning method that integrates location and semantic constraints, and performs the following steps:

Step 1: Perform GeoHash conversion on the collected multi-source POI data and convert the two-dimensional coordinate data into a string;

Step 2: Perform adjacent point query on the converted string;

Step 3: Perform redundant processing on the windows with adjacent points in Step 2, and perform redundant data processing, incomplete data processing, inconsistent data processing and high similarity data processing in sequence;

Step 4: Build a word segmentation solution based on the Chinese Language Model and the Hidden Markov Mode;

Step 5: Perform redundant processing on the data processed in Step 4;

Step 6: Complete the POI data re-matching based on the word frequency statistics of the word segmentation scheme reconstructed in step 5 to realize the multi-source POI data cleaning.

The improvement of the above technical solution is to conduct a neighbor point query based on the B+ tree method and prefix matching on the converted string.

The improvement of the above technical solution is: the processing of redundant data, incomplete data, inconsistent data and highly similar data in step 3 are as follows:

Redundant data processing, retaining one operation for duplicate data caused by continuous tracking of data on the same platform; when some attributes of a small amount of redundant data are consistent, the method of retaining the data with the highest data completeness based on location attributes is used;

Incomplete data processing, first make a redundancy judgment with the complete data. If it is defined as redundant data, it will be eliminated. If it is non-redundant data, it will be further judged whether it is inconsistent data or highly similar data, and it will be processed according to the corresponding method and appended at the same time. Corresponding label;

Inconsistent data processing, for inconsistent data of non-neighboring points, verify the POI point name and location through multiple geo-parsing and address parsing on different geographical service platforms; for inconsistent data of neighboring points, select the location data with the most parsed information as the Entity location information, eliminating other inconsistent data;

Highly similar data processing uses inconsistent data processing methods to segment entity description names into phrases, establishes similar data indexes and obtains address data based on the regional mapping database of designated regions, and selects POI data that is more comprehensive relative to geographical elements for storage.

The improvement of the above technical solution is: in step 4, the ChineseLanguage Model is used to split the POI names that the existing vocabulary depends on; for the words that are not included in the vocabulary but need to be divided, the Hidden MarkovMode is used to split the POI based on word formation. Names are divided into word segments.

The improvement of the above technical solution is that the redundancy processing process in step 5 is the same as that in step 3 except that the processing objects are different.

The improvement of the above technical solution is: in the reconstruction process of step 6, it is necessary to determine the keywords related to the name of the POI data and the word frequency of the corresponding keywords, and discard the reverse file frequency according to the word frequency to select high-probability keywords. To correspond to the corresponding POI data.

The beneficial effects of the present invention are: the present invention processes multi-source POI data such as redundancy, error, confirmation, and reclassification, and unifies data quality and standards with location constraints and semantic constraints to obtain highly available and highly credible POI data. set. This method can complete the data cleaning work more accurately and efficiently, and the cleaning results are better. It is a more practical and effective data cleaning method.

Description of the drawings

Figure 1 is a flow chart of a multi-source POI data cleaning method that integrates location and semantic constraints according to an embodiment of the present invention.

Detailed ways

Example

A multi-source POI data cleaning method that integrates location and semantic constraints in this embodiment, as shown in Figure 1, performs the following steps:

Step 1: Perform GeoHash conversion on the collected multi-source POI data to convert the two-dimensional coordinate data into a string; GeoHash, as a geographical location representation structure, can convert location information encoding such as longitude and latitude into characters composed of letters and numbers. String, that is, reducing two-dimensional data to one-dimensional data, and the longer the shared prefix between the GeoHash strings converted by two geographical locations, the closer the two points are in space.

Step 2: Perform adjacent point query on the converted string; use the B+ tree method to match the string prefix within a fixed window to implement adjacent point query. When selecting the length of GeoHash, you should conduct multiple experiments to calculate the latitude error, longitude error and meter error for each length, and choose the most appropriate length.

Step 3. If there is only one POI point (POI position) or no POI point in the data in Step 2 (or in the area), the data will be stored for subsequent research operations. If there are multiple adjacent POI points in the area, it is necessary to perform data similarity detection, analyze their data quality and data redundancy, and conduct inspection, screening and elimination to obtain a low-redundancy and high-availability data set in the area.

Step 4: Build a word segmentation solution based on the Chinese Language Model (CLM) and Hidden Markov Mode (HMM);

Step 4.1: Split the POI names that already depend on the vocabulary and use CLM for processing. First, use the prefix dictionary to scan the word graph composed of the POI name, and construct a directed acyclic graph based on all possible words in the name to obtain all the segmentation methods W of the POI name S. Based on dynamic programming, the conditional probability P(W|S) of each segmentation method is calculated, and the POI name segmentation method W ^* corresponding to the maximum conditional probability is taken, which is the final word segmentation result. According to the Bayesian formula, W ^* can be obtained by solving P(W), and P(W) can be obtained using CP(W)LM modeling. Taking Bi-gram as an example, the formula is as follows:

W ^* =argmax _w P(W|S)

Step 4.2. For words that are not included in the vocabulary but need to be divided, that is, non-login words, use HMM to divide the POI name into word segments mainly based on word formation, and specify the four morphemes of the word: beginning, middle, and end of the word. , a single character becomes a word. Taking the POI name as the input, the sequence string composed of the corresponding lexeme as the output, and dividing the lexeme sequence string, the division of the POI name can be obtained. For the discovered non-login words, add them to the vocabulary to improve the efficiency of the algorithm. The final POI name segmentation example is as follows.

POI name segmentation indication

Step 5. Since the data comes from multiple platforms, there will be multiple data records pointing to the same geographical entity in the captured data, thus generating redundant data.

This type of redundant data needs to be cleaned up in terms of its attributes and location. For redundant data with completely consistent attribute record rows, the method of retaining one record and eliminating other redundant records is adopted. For redundant data with partially consistent attribute record rows, since they are adjacent POI data located in the same window, the POI with the highest data completeness is retained based on location information, and other redundant data with consistent attributes are discarded.

For some incomplete data that lack category auxiliary information, redundancy judgment is first made with complete data, and if it is defined as redundant data, it is eliminated. If it is non-redundant data, calculate the similarity between the data and each type based on the analysis results of each type of data established in step 3, and assign the POI data type according to the data meeting the similarity threshold or keyword matching requirements of a certain category. Label.

For inconsistent data with inconsistent locations, select the location data with the most parsed information as the entity location information, and eliminate other inconsistent data; for data with similar names but consistent locations in nearby POI information, index the inconsistent data in the window and use The name represents at most one set of data as entity name information.

Example of inconsistent neighboring POI data

For POI data with similar but not completely identical names and adjacent but not completely overlapping coordinates, first make a preliminary judgment of similarity: use a word segmentation scheme to segment entity description names into phrases, and select POI groups with more than 1 group of identical words after segmentation. Or if there is only one set of word segmentation results and similar POI data, a similar data index will be established. Secondly, the address data is standardized for the data in the similar data index. For example, the Chinese administrative division mapping library is used to extract and map the original address information to obtain the standardized address data. Finally, based on the administrative region mapping library and word segmentation scheme, the POI address data is split and multiplied into 8 parts, namely province, city, district, road, house number, community, building, and text, and different splits are set for different POI types. The result comparison method selects POI data with complete geographical elements for storage.

Step 6: Since the POI data re-constructed in step 5 still exists. Since there are a small number of POI types in the multi-source POI data that are inconsistent with the actual types, it is necessary to perform type rematching on these POI data to achieve the multi-source POI data. of cleaning.

Due to the category attributes of POI data, the impact of data of the same category on the target POI is much greater than that of data of the same category. Therefore, a POI category corpus is constructed to simulate the corpus scenario where the POI is really located. First, count the number of keywords in the POI name (Term Count, referred to as TC), that is, the number of times a certain word appears in the database. After determining the TC, calculate the word frequency of the keyword. For the keyword J in the POI name of a certain i type, its term frequency (Term Frequency, TF for short) is as shown in the following formula, where TC _i,j is the frequency of the word in the i corpus the number of occurrences in , and is the sum of the number of occurrences of all words in the i corpus.

Based on the assumption that there is a certain relationship between the name directivity in POI data and the position of the keyword group in the name sequence, TF-IDF is improved. When scanning POI names, the sequence k in the phrase is split according to the POI where the keyword is located. Add a new position weight W. The definition rules of W are as follows: when the number of keywords is greater than 2, the weight of the last keyword is set to 2, the weight of the penultimate group of keywords is set to 1.5, and the rest are set to 1; when the keyword When there are only 2 keywords, the weight of the last keyword is set to 1.5, and the rest are set to 1; when the number of keywords is 1, the weight is set to 1; if the keyword is a single number or letter, there is no weight. The final weighted word frequency is obtained by multiplying it with TF. The formula is as follows:

Based on heap sorting and batch processing, the top n keywords in each type with weighted word frequency are selected as the core words of the category, and a core word dictionary tree is constructed to efficiently match the core words. Finally, the keywords of a certain category are matched according to the POI name. , calculate the sum of the weights corresponding to the keywords, that is, calculate the probability of the category to which it belongs, and finally select the category with the highest probability to assign it to the POI to complete the re-matching process.

Claims (5)

1. A multi-source POI data cleaning method integrating position and semantic constraint is characterized by executing the following steps:

step 1, performing GeoHash conversion on collected multi-source POI data, and converting two-dimensional coordinate data into character strings;

step 2, inquiring the adjacent points of the converted character string;

step 3, performing redundancy processing on the window with the adjacent point in the step 2, and sequentially performing redundancy data processing, incomplete data processing, inconsistent data processing and high-similarity data processing;

step 4, constructing a word segmentation scheme based on the Chinese language model Chinese Language Model and the hidden Markov model Hidden Markov Mode, wherein: processing using Chinese Language Model for existing vocabulary dependent POI name splitting; aiming at words which are not recorded by the vocabulary but need to be divided, using Hidden Markov Mode to divide the POI name into words based on word formation;

step 5, performing redundancy processing on the data processed in the step 4;

and 6, finishing POI data re-matching based on word frequency statistics of the word segmentation scheme reconstructed in the step 5, and realizing multi-source POI data cleaning.

2. The multi-source POI data cleansing method with fusion of location and semantic constraints according to claim 1, wherein: and carrying out neighbor point query on the converted character string by prefix matching based on a B+ tree method.

3. The multi-source POI data cleansing method with fusion of location and semantic constraints according to claim 1, wherein: the processing for redundant data, incomplete data, inconsistent data and highly similar data in step 3 is as follows,

redundant data processing, namely reserving one operation for repeated data caused by continuous tracking of the same platform data; the method comprises the steps of processing partial attributes of a small amount of redundant data in a consistent manner in a mode of reserving data with highest completeness based on the position attributes;

processing incomplete data, namely firstly carrying out redundancy judgment on the incomplete data and the complete data, removing if the complete data is defined as redundant data, further judging whether the complete data is inconsistent data or high similar data if the complete data is non-redundant data, and processing the complete data according to a corresponding mode while adding corresponding labels;

inconsistent data processing, namely verifying the names and positions of POI points by carrying out multiple times of geographic analysis and address analysis on different geographic service platforms for inconsistent data of non-adjacent points; for inconsistent data of adjacent points, selecting the position data with the most analyzed information as entity position information, and eliminating other inconsistent data;

and (3) processing high similarity data, namely performing phrase segmentation on entity description names by utilizing an inconsistent data processing mode, establishing a similarity data index, acquiring address data based on a region mapping library of a designated region, and selecting POI data more comprehensive relative to geographic elements for storage.

4. The multi-source POI data cleansing method with fusion of location and semantic constraints according to claim 1, wherein: the redundant processing procedure in the step 5 is consistent with the rest part of the step 3 except for the processing objects.

5. The multi-source POI data cleansing method with fusion of location and semantic constraints according to claim 1, wherein: in the reconstruction process of step 6, keywords related to the names of the POI data and word frequencies of the corresponding keywords are required to be determined, and the reverse file frequencies are abandoned according to the word frequencies, so that the keywords with high probability are selected to correspond to the corresponding POI data.