DomainNet: Homograph Detection and Understanding in Data Lake Disambiguation

Entity Resolution (ER) identifies records (also called tuples) across different datasets (or sometimes corpora) that represent the same real-world entities. ER is generally applied to structured and semi-structured data including tables and RDF triples [37]. Some ER approaches also identify ambiguous values as part of the resolution process. For example, using collective entity resolution over two types of tables (e.g., papers and authors) one can identify if a value, say, “X. Wang,” refers to different authors [9]. Similarly in familial networks, one can resolve synonyms (different values that refer to the same person) and identify homographs (same value used to refer to different people) [52].

ER assumes that the information to be resolved or disambiguated is of a single known type (e.g., resolving customer tuples or patient records) or from a small set of types (e.g., authors, their papers, and publishing venues). Some work, called schema-agnostic ER, does not require that all data be represented using the same schema [21]. However, all these approaches start with the assumption that two or more tables (or corpora) are describing the same type of entities [73, 74, 83].

In data lake disambiguation, we seek to find ambiguous values even when we do not know what type of entities a table is describing. We also do not know if different tables are describing the same or different entities. Hence, we cannot apply collective models or other resolution models that rely on this knowledge.

One possible approach to data lake disambiguation is to discover semantic types for all attributes (columns) and then label a value appearing in different semantic types a homograph. In the running example, identifying the semantic type of T1.At_Risk and T2.name as animal and mammal, respectively, and knowing that mammals are animals, one can infer that Jaguar is not a homograph there. In contrast, recognizing T3.C2 is of type “Car Manufacturer,” which is neither a sub- nor super-type of animals, implies that Jaguar in T3 and T1 represents a homograph. Here, we discuss different approaches to semantic type discovery and to what extent they could be used for homograph detection.

Knowledge-based Techniques. There has been considerable work on semantic type detection in the Semantic Web community that uses external knowledge from well-known ontologies including DBpedia [54], Yago [88], and Freebase [12]. Most solutions have been applied to Web tables [29, 30, 55] that are small (in comparison to tables in other data lakes [67]) and have rich metadata (table and attribute names).

Hassanzadeh et al. [39] use a map-reduce approach to find similarity between a (column, data value) pair from a table with a (class, instance label) pair from the Knowledge Base (KB). Ritze et al. [79] match Web tables to DBpedia to profile the potential of Web tables for augmenting knowledge bases with missing information. These approaches cannot infer type information for an attribute that it is not part of the KB. Unfortunately, the coverage of values from data lakes in Open KBs is low (a recent study reports about 13% [68]), limiting their applicability.

Supervised Techniques. An alternative approach is to use machine learning (ML) to infer the semantic type of attributes. ML solutions utilize a variety of graphical models (Conditional Random Fields [38], Markov Random Fields [60]) as well as Multi-level Classification [89] and Deep Learning [43]. Sherlock [43] uses features about the values in an attribute to classify some of the attributes in a data lake into one of 78 semantic types (such as address or horse jockey) [43]. A more recent solution, called SATO [100], augments this approach and shows that using row information can improve the classification accuracy for the same 78 semantic types. These approaches require large amounts of labeled training data and are limited to only a fixed pre-defined set of types. Nonetheless, we consider them as baseline approaches for identifying homographs in our experiments. Any value that appears in more than one semantic type is considered a homograph.

Unsupervised Techniques. Unsupervised semantic type discovery algorithms have only recently started to be studied. We discuss two unsupervised algorithms, one for semantic type discovery, \(D^4\) [71], and one for table unionability search [68].

\(D^4\) provides an unsupervised approach with a focus on assembling all the values of each semantic type in a data lake [71] (these values are called a “domain”). They propose a data-driven approach that leverages value co-occurrence information to cluster values that are from the same domain. Heuristics attempt to deal with ambiguous values that may appear in multiple domains. In our context, \(D^4\) can be used to label values that appear in multiple domains as homographs. This indeed serves as a baseline method for detecting homographs in our experiments. However, as we later show in Section 6.9, \(D^4\)’s performance can still be impacted negatively by the presence of homographs, suggesting that DomainNet can be used as a pre-processing step to improve \(D^4\)’s performance.

Table Union Search (TUS) [68] solves a different problem. Given a query table, they find a set of tables from the lake that are most unionable with it. To do so, they provide several similarity measures that are used collectively to calculate how unionable two attributes are. This work can use both ontological and semantic (word embedding) signals when present to determine unionability over heterogeneous attributes. TUS does not attempt to find/label homographs, but could be adapted for that problem by identifying shared values between a pair of non-unionable columns. However, such an approach would not be scalable, since all pairs of columns would have to be compared and labeled as unionable or non-unionable. One of our benchmarks has over 9,000 columns. Real table repositories reported in other studies have over 65,000 columns [104].

Word-sense disambiguation (WSD) [44, 69], i.e., the task of identifying which meaning of a word is used in a sentence, is an important problem in computational linguistics. Although a human can proficiently perform this task on a document, constructing algorithms that perform this task effectively is still an open research problem. Techniques proposed so far range from dictionary-based methods, which use the knowledge encoded in lexical resources (e.g., WordNet) [69], to more recent solutions in which a classifier is trained for each distinct word on a corpus of manually sense-annotated examples [75]. Additionally, completely unsupervised methods have been proposed that cluster occurrences of words, thereby inducing word senses, i.e., word embeddings [44]. The aforementioned solutions rely on information (or latent information) about the structure of sentences including grammatical rules. Finally, while solutions that do not rely on grammar also exist, they only operate on documents and not tables [10, 84].

Another relevant sub-task in Natural Language Processing is Named-Entity Recognition (NER), which has been proposed as a possible solution for disambiguation [93]. NER seeks to locate and classify named entities mentioned in unstructured text into pre-defined categories such as person names, organizations, locations, and so on. NER systems have been created that use linguistic grammar-based techniques as well as statistical models [3]. A special case of the NER problem is the author name disambiguation problem [33, 85]. Authors of scholarly documents often share names, which makes it hard to distinguish each author’s work. Hence, author name disambiguation aims to find all publications that belong to a given author and distinguish them from publications of other authors who share the same name. Different solutions have been proposed using graphs [57]. However, the graph structure proposed is largely domain-specific. The graph contains not only the information about the co-authorship and published papers, but also venue of the paper published, year of research activities, and so on. Overall, NER approaches are effective when they operate over specific domains and structured text, but they cannot be used directly over a large set of heterogeneous tables to identify homographs (especially if the homographs are null equivalent values or due to value misplacement). Also note that values in tables need not be named entities. The value ca may refer to the named entity California, but it may also be an abbreviation for the word circa or an initialism for the adjective closed-access. Similarly, a number like 10 may be a homograph referring to both a rating and to a street number.

Disambiguation of values also plays an important role in entity matching (EM) techniques, i.e., the problem of determining whether two data entries refer to the same real-world entity. For example, when comparing a pair of entries for products from two different tables, it is important to discern if the shared values and their respective contexts with other available attributes (e.g., price, product description, product code) can result in a match (same product) or not. With the emergence of deep learning techniques in NLP using modern transformer architectures (e.g., BERT [26], XLNet [95], RoBERTa [61], and DistilBERT [81]), EM techniques [14, 59] using them have seen their performance significantly improved, especially in benchmarks that are highly textual. One such state-of-the-art EM technique is DITTO [59], which uses fine-tuned pre-trained transformer-based language models and is further optimized by injecting domain knowledge, text summarization, and data augmentation. However, EM techniques are not suitable for discovering all homographs in a data lake, since two rows from different tables that contain the same value (with the same meaning) may not necessarily match, given the table context. For example, consider the two tuples A = [UK, Jaguar, 25.80] and B = [Bob, Smith, Jaguar, XE, London, 45 Sheffield St.]. Tuple A describes car brands and their revenue, and tuple B describes car owners and their address. Even though Jaguar refers to the car brand in both cases, the two tuples would not match using EM techniques, since they communicate different information. Moreover, EM techniques are focused on identifying matching and non-matching data entries from two tables (or a small number of tables) and not from a large and heterogeneous table repository like a data lake.

Natural language interfaces (NLI) for databases take advantage of user interaction to resolve ambiguities. NLIs are primarily designed for non-expert SQL users to facilitate querying by expressing queries in natural language [2]. Over the past few decades a large variety of NLI systems have been developed, such as keyword-based systems [11, 47, 82], pattern-based systems [25, 99], parsing-based systems [48, 58], grammar-based systems [1, 28, 86], as well as more recently machine learning approaches [15, 23, 90, 102]. Systems using different approaches are able to handle different levels of SQL complexities (e.g., keyword-based systems cannot handle aggregations or long-range dependencies, whereas parsing-based and grammar-based systems can). More importantly ambiguities (either from the specified values or from the query logic) are resolved mostly interactively by asking the user to confirm the values and/or logic understood by the system or by having the user choose from a set of options. Many NLIs are designed to operate over databases from a specific domain or using a specific type of queries and would not generalize very well to arbitrary databases and SQL queries. Although recent machine learning-based NLI approaches are more generalizable and designed without a specific domain in mind, such methods are usually evaluated over Spider [98], a large benchmark consisting of 10,181 questions and 5,693 SQL queries from 200 databases covering 138 domains. Despite the size of Spider, it is still far from the heterogeneity and size of data lakes that can contain thousands of domains and millions of unique values [67].

Another area of related work that can potentially be re-purposed to disambiguate values belonging in multiple communities is overlapping community detection [5, 24, 45, 92]. Overlapping community detection algorithms exploit the graph structure to identify nodes that can possibly belong to multiple communities (i.e., they overlap with two or more communities) [5, 92]. There is a large variety of techniques employed by overlapping community detection algorithms, such as clique expansion, link clustering, label propagation, and many others. We discuss a few of those methods that we later compare against DomainNet. Big-Clam [94] is a scalable overlapping community detection algorithm that combines non-negative matrix factorization methods with block stochastic gradient descent to identify node cluster affiliations. LPANNI [63] detects overlapping community structures by adopting a fixed label propagation sequence that is based on the ascending order of node importance and a label update strategy that is based on neighbor node influence. DANMF [96] is a deep autoencoder-like non-negative matrix factorization method that can learn hierarchical mappings between the original network and the final community assignment. It has implicit low-to-high level hidden attributes of the original network learned in the intermediate layers. Core Expansion [20] is a recent approach that tries to detect communities without computing the modularity score. It automatically detects the core of each possible community in the network and then iteratively expands each core by adding nodes to form the final communities. The expansion process is based on a neighborhood overlap measure. Overlapping community detection algorithms can be adapted to solve for the homograph discovery problem by labeling nodes that belong in more than one community as homographs. We compare the efficacy of overlapping community detection algorithms against DomainNet in Section 6.8.

In data lakes, attribute and table names can be missing or misleading (with many ambiguous terms such as “name,” “column 2,” or “detail”) [67]. Well-curated enterprise lakes may have more complete metadata, but even they do not follow the unique name assumption—which states that different attribute names always refer to different things. As a result, many data lake search approaches rely solely on the table contents [27, 32, 104, and others]. In a similar vein, in DomainNet, we investigate to what extent data values and the co-occurrence of data values within attributes can be used to determine if a value is a homograph, and if so, how many meanings it has in the data lake.

In a well-curated database or warehouse, we may know the domain of each attribute (e.g., “Animal Name” vs. “Company Name”) and can leverage it to identify homographs. However, in large non-curated table repositories with possibly missing and ambiguous table and attribute names, we cannot rely on such information to be readily available.

In data lakes, without a priori knowledge of table semantics or types, we take a network-based approach to understanding the meaning of repeated data values. We propose to detect homographs using network measures. For that purpose, we can interpret the co-occurrence information about values across different attributes using a network representation in which nodes represent data values and edges represent the fact that two values co-occur in at least one column (attribute) in the data lake. Notice that while we do not use the column names, we do use the table structure (the fact that a value appears in a certain set of columns and co-occurs with other specific values). So, we are not treating tables as documents (i.e., bags-of-words based on their cell values), as usually done in information retrieval tasks.

Whereas this representation allows us to apply straightforward metrics from community detection, it comes at a high cost: The representation uses more space than the original data lake. The incidence matrix is sparse and has as many entries, as there are cells in the data lake (Figure 2). In contrast, the co-occurrence graph increases quadratically in size with respect to the cardinality of attributes (the size of the vocabulary) in the data lake (Figure 3(a)). Consider a single column with 100 values. The incidence matrix represents this information with 1 row, 100 columns, and 100 entries. The co-occurrence graph represents this with 100*99/2 = 4,950 edges across 100 nodes.

Thus, we use a more compact network representation that allows us (after some modifications) to apply network metrics to discover pivotal points (Figure 3(b)). DomainNet uses a bipartite graph composed of (data) value nodes and attribute nodes. The attribute nodes represent the set of attributes and the value nodes the set of data values across all attributes in the lake. Every data value is treated as a single string; it is capitalized and has its leading and trailing white-space removed to ensure consistent comparison of data values across the lake. Notice that each data value, even if found in multiple attributes, is represented by one single value node in the graph. An edge is placed between a value node and an attribute node if the data value appears in the attribute (column) corresponding to that attribute node. Data values that appear in more than one attribute are candidates for being homographs. Notice that our bipartite graph representation is conceptually similar to an inverted index, as it allows us to quickly identify co-occurring nodes while saving a lot of space. Inverted indices have been widely used in information retrieval for efficient indexing as well as to encode term co-occurrences [34]. However, because our bipartite representation is still a graph, we can apply graph measures as well as graph compression techniques, as we later discuss in Section 3.4, to improve the efficiency of our method in identifying likely homographs.

In the DomainNet bipartite graph, we call two data values neighbors if they both appear in the same attribute (and hence there is a path of length two between them in the graph). Similarly, two attributes are neighbors if they have at least one data value in common (and hence there is a path of length two between them). For a data value node v, \(N(v)\) denotes the set of all its value neighbors. We also define the cardinality of a data value node v as the number of neighbors \(|N(v)|\), which is the number of unique data values that co-occur with v. If n is the number of value nodes and a the number of attribute nodes, then the number of edges in a DomainNet graph over real data tends to be much less than \(n\cdot a\).

Tables to Graph. Recent work on embedding algorithms in relational databases [6, 16, 53] use a graph representation of tables. Like DomainNet, they model values and columns as nodes. Depending on the problem addressed, some approaches also include nodes for rows and tables. Like in our approach, column names are not assumed to be present or unambiguous.

Koutras et al. [53] and Capuzzo et al. [16] use a tripartite graph representation in which every value node is connected with its column node and its row node. Such an approach works well for the tasks of tuple-level entity resolution and for schema matching (a similar task to semantic type discovery). In our example, Panda in \(T1\) and \(T2\) are not homographs, but the row information makes them seem quite different.

In contrast, Arora and Bedathur [6] use a homogeneous graph using only data value nodes that are connected with each other if they appear in the same row of the table. They do not use the value co-occurrence information within a column, making homograph detection using solely row context inappropriate in large heterogeneous datasets.

Intuitively, data values that frequently co-occur with each other will form an implicit domain or community in DomainNet, with many paths of varying length between them. Homographs will span two or more communities. Notice, however, that we do not know a priori what the communities are or even how many there are. While there is a rich literature on community detection, many approaches require knowledge of the possible communities such as the number of communities [18]. Others are parameter-free, meaning they can learn the number of communities [40, and others]. However, in our problem the number is not only unknown, it may be massive. A data lake from open data repositories with just a modest number of tables may have hundreds or thousands of attributes representing possibly different domains. Miller [66] states that in their study of open data, tables have, on average, 16 attributes but with a large variance, stating that some tables have many hundreds of attributes. Nevertheless, work in overlapping community detection could be utilized to identify homographs as cell value nodes that are part of multiple communities. We indeed make this comparison in our experiments (see Section 6.8).

As an alternative, what we propose in this article is to use network centrality measures that can be defined without prior knowledge of how many communities exist, their overlap, or the distribution of attribute cardinalities. The intuition behind centrality measures is to capture how well connected the neighbors of a given node are. We define variants of these measures appropriate for the DomainNet bipartite graph. We then discuss to what extent these measures may distinguish whether a data value has a single meaning or multiple meanings (the latter being a homograph).

Local Clustering Coefficient as a homograph score. The local clustering coefficient (LCC) [91] for a given value node measures the average probability that a pair of the node’s neighbors are also neighbors with each other, i.e., the fraction of value-neighbor triangles that actually exist over all possible triangles.

The LCC metric is usually defined over unipartite graphs (such as the co-occurrence graph in Figure 3(a)). We use the definition of value-neighbors (recall the set of all value neighbors of a value node u is \(N(u)\)) to generalize LCC to our bipartite graph.

The pairwise clustering coefficient of two data value nodes v and w is defined as the Jaccard similarity between their neighbors.Given a graph G and a value node u, the LCC is defined as the average pairwise clustering coefficient among all the node’s value neighbors:The LCC of a node u can be computed in time \({\mathcal {O}}(N(u)^2)\) and provides a notion of the importance of a node in connecting different communities.

Intuitively, we expect unambiguous values to appear with a set of values that co-occur often and thus have high LCC scores. This behavior should be less common for homographs, which may span values from different communities, as they appear in various contexts depending on their meaning.

Despite LCC’s computational simplicity, the measure as defined in Equation (1) is no more than the average Jaccard similarity between the set of attributes that a value co-occurs with. Unfortunately, it is well-known that Jaccard similarity is biased to small sets. As a consequence, the measure is not as effective in real data lakes, where attribute sizes are often considerably skewed. Our experiments will confirm this downside of LCC.

Betweenness Centrality as a homograph score. The LCC of a node is fast to compute, but it only considers the local neighborhood of a value. In a data lake, the local neighborhood may not be sufficient. In particular, the local neighborhood may not include values that are members of the same community but happen to not co-occur. To overcome these two problems (missing values in the neighborhood and attributes with very different cardinalities), we look at metrics that take a more global perspective on the network.

The betweenness centrality(BC) of a node measures how often a node lies on paths between all other nodes (not just the neighbors) in the graph [35]. One way to think of this measure is in a communication network setting where the nodes with highest betweenness are also the ones whose removal from the network will most disrupt communications between other nodes in the sense that they lie on the largest number of paths [70].

Consider two nodes v and w. Let \(\sigma _{vw}\) be the total number of shortest paths between v to w, and let \(\sigma _{vw}(u)\) be the number of shortest paths between v to w that pass through u (where u can be any node).⁴ The betweenness centrality of a node u is defined as follows, where v and w can be any node in the graph:By convention, \(\frac{\sigma _{vw}(u)}{\sigma _{vw}} = 0\) if \(\sigma _{vw}\) (and therefore \(\sigma _{vw}(u)\)) is 0.

Intuitively, a homograph appears with sets of values that do not or rarely co-occur across those sets, and thus the shortest paths between such non-co-occurring nodes would have to go through the homograph node. Conversely, unambiguous values appear with a set of values that also co-occur a lot, and thus the shortest path between them does not unnecessarily have to go through one or a few nodes.

Complexity of BC. Calculating the BC for all nodes in a graph is an expensive computation. A naive implementation takes \({\mathcal {O}}(n^3)\) time and \({\mathcal {O}}(n^2)\) space (n denotes the number of nodes in the graph). The most efficient algorithm to date is Brandes’ algorithm [13], which takes \({\mathcal {O}}(nm)\) time and \({\mathcal {O}}(n + m)\) space (for unweighted networks), where m is the number of edges in the graph. Notice that this algorithm is still expensive if the graph is dense (i.e., \(m \gt \gt n\)).

The high time complexity of BC motivated approximations, which usually sample a subset of nodes from the graph and thus do not calculate all shortest paths. One common sampling strategy is to pick nodes with a probability that is proportional to their degree (nodes with high degree are more likely to appear in shortest paths). Riondato and Kornaropoulos [78] provide an approximation algorithm via sampling with offset guarantees. Geisberger, Sanders, and Schultes [36] provide an approximation algorithm without guarantees that performs very well in practice. The complexity of the approximate BC is \({\mathcal {O}}(s m),\) where s is the number of nodes sampled. We chose Geisberger, Sanders, and Schultes [36] to approximate betweenness centrality to benefit most from its short runtime on large graphs.

Although our bipartite DomainNet graph is designed to be sparse as described in Section 3.2, for the purposes the BC computation there is topological information in the graph that we can leverage to further speedup the computation. More specifically there are many cell nodes in the bipartite DomainNet graph that are connected to the same set of attribute nodes. For example, the value “Lion” may always appear in attributes where the value “Tiger” also appears, so the cell nodes corresponding to values “Lion” and “Tiger” will be connected to the same set of attribute nodes. In the context of BC this translates to “Lion” and “Tiger” having the same BC scores, because for any pair of nodes where there is a shortest path between them through cell node “Lion,” there is another path with the same length that goes through the cell node “Tiger.” Nodes such as “Lion” and “Tiger” that always appear in the same set of attributes are quite common in large data lakes, so if we can group them together and reduce the graph size, then the computation of BC can be significantly improved.

Graph summarization techniques such as graph compression, grouping, and simplification have been extensively studied [8, 62, 65] and applied in various applications, such as query handling [64, 77], pattern discovery [22, 101], graph databases [46, 50], or for semi-supervised learning from few labels [72]. The idea of compressing the graph by identifying sets of nodes that are solely connected with another set of nodes for the purpose of speeding up the exact BC computation was first explored by Sariyüce et al. [17], where they propose various graph shattering⁵ and compression techniques that reduce the size of the input graph. Let \(\Gamma (v)\) denote the set of nodes connected to node v. We say two nodes \(u, v\) are identical if and only if \(\Gamma (v)=\Gamma (u)\). Given a graph G, Sariyüce et al. [17] generate a compressed graph \(G^{\prime }\) by compressing each set of identical values into a single compressed node. They then can run Brandes’ BC algorithm [13] on \(G^{\prime }\) with a few small modifications and obtain the BC scores for all nodes in G. In short, this is achieved by recording the number of nodes each compressed node (called the “ident” score [17]) in \(G^{\prime }\) corresponds to in G. The computation speedup comes from the fact that the compressed graph \(G^{\prime }\) will have fewer nodes and edges than the original graph G. Since Brandes’ algorithm has a complexity of \({\mathcal {O}}(nm),\) the speedup can be significant if the compression is large. For instance, if the number of nodes and edges is halved in the compressed graph, then we would expect a quadruple speedup.

Although Sariyüce et al. [17] use their graph compression techniques to compute the exact BC scores for each node in a graph, we can still re-purpose them to compute the approximate BC scores for each node if we want an even faster answer. As with many other BC approximation algorithms [7, 19, 36, 78], we can sample a subset of the nodes in the compressed graph \(G^{\prime }\) and compute all the shortest paths that start from our sampled nodes.

One way to sample for nodes in the compressed graphs is using uniform random sampling, but, as we will demonstrate in Section 6.6, that can lead to a biased sampling. This is because many compressed nodes correspond to multiple nodes in the original graph, but they are sampled with the same probability as uncompressed nodes. This can lead to large inaccuracies in the approximation, since compressed nodes that correspond to many nodes in the original graph can have a large impact in the BC scores if not sampled adequately. Therefore, to accommodate for this imbalance, we can use weighted random sampling over the compressed graph where the probability of sampling a node is proportional to its ident score.⁶

In this section, we describe the implementation of an end-to-end system that allows users to disambiguate data lakes using our proposed methodology. Our system has three steps, as illustrated in Figure 4: (1) construct DomainNet graph; (2) calculate measures; and (3) rank measures.

DomainNet graph construction. The input is a set of raw data tables from relational databases, CSV files, or any other open data format. It is important to note that we do not require any information in regards to types, attribute names, or the semantics of relationships between tables. We build our bipartite graph as described in Section 3.2.

Graph measure computation. Using the DomainNet graph constructed in the previous step, our system computes both LCC and BC scores for each value node (Section 3.3) or compressed value node (Section 3.4). We show empirically in Section 6.1 that BC outperforms LCC in homograph detection.

Graph measure ranking. Nodes are ranked by their centrality score (ascending order for LCC measures and descending order for BC measures) the top-ranked data values are presented to a user.

In this section, we describe our method for solving Problem 4.1. Let h be the homograph in question and let \(Attr(h)\) denote the set of attributes (columns) the homograph h appears in. Our goal is to find \(\mathcal {N^{\prime }}\) sets (or groups) of attributes, \(G_1,\dots , G_{\mathcal {N^{\prime }}}\), where \(2 \le \mathcal {N^{\prime }} \le |Attr(h)|\) such that \(\bigcup _{i=1}^{\mathcal {N^{\prime }}}G_i = Attr(h)\) and \(\bigcap _{i=1}^{\mathcal {N^{\prime }}}G_i = \varnothing\). The sets of attributes \(G_1,\dots ,G_{\mathcal {N^{\prime }}}\) correctly solve Problem 4.1 if and only if all the instances of the homograph h in each attribute group have the same meaning. So, if in the ground truth there are \(\mathcal {N}\) meanings for homograph h and our solution is correct, then \(\mathcal {N^{\prime }} = \mathcal {N}\) and the sets of attributes \(G_1,\dots ,G_{\mathcal {N^{\prime }}}\) will correspond to the correct clustering of h’s attributes based on their meaning.

One way to group the set of attributes of a homograph by their meaning is to find a measure for the similarity of the values between two attributes. Popular similarity measures in this context include Jaccard similarity [32] and containment [103]. More sophisticated measures such as unionability [68] that can overcome value heterogeneity (e.g., two columns whose values do not overlap but still belong to the same domain) have also been proposed.

We choose the Jaccard similarity, because it is inexpensive to compute and performed well in our experiments. More specifically, given a homograph h, we compute a distance matrix of size \(|Attr(h)| \times |Attr(h)|\) by computing the pairwise Jaccard distances⁷ between the cell values contained in each pair of attributes. We can think of this process as an assignment of each attribute to a point in \(\mathbb {R}^{|Attr(h)|}\) where ideally attributes that belong to the same domain will be spatially close to each other. Therefore, given the distance matrix, we would like to cluster its rows so each cluster corresponds to attributes where instances of homograph h in them have the same meaning. There are a myriad of clustering algorithms, but given that we want to identify spatially close points (i.e., in our case, attributes) a density-based clustering algorithm would be fitting. We selected DBSCAN [31], because it is efficient and it can automatically identify the number of clusters, unlike k-means and many other clustering algorithms. DBSCAN identifies a point in a dense region as an initial cluster and then expands this cluster by including neighboring points that are “sufficiently close.” Points in low-density regions that are not assigned to a cluster are labeled as outliers.

There are two key parameters in the DBSCAN algorithm: Epsilon (\(\varepsilon\)) and Minimum Points (minPts).

Notice that our choice of Jaccard similarity is not in contradiction with our claims about it in Section 3.3. Generating the pairwise Jaccard distances across all pairs of attributes in the dataset and running DBSCAN on this larger matrix will not be effective. Jaccard similarity cannot meaningfully express the differences between attributes at a global scale; however, if applied locally for the attributes of a value we have identified as a homograph, then it can yield much more meaningful results.

To motivate our algorithm that chooses \(\varepsilon\) in practice, we first discuss an ideal scenario where the ground truth about the semantic type of each relevant attribute is given. Having the semantic type ground truth means that we also know the correct groups of attributes \(G_1,\dots ,G_\mathcal {N^{\prime }}\) that solve Problem 4.1. So, using DBSCAN, how should we set \(\varepsilon\) to extract the same groups? Let \(D_{G_i}\) be the multiset of the pairwise distances between all attributes in \(G_i\), so if our distance measure is Jaccard \(D_{G_i} = \lbrace J(a,b) | \forall a, b \in G_i \rbrace\) where \(J(a,b)\) denotes the Jaccard distance between attribute a and b. We choose \(\varepsilon = \max (\max (D_{G_1}), \dots , \max (D_{G_n}))\); in other words, \(\varepsilon\) is set as the largest pairwise distance between attributes of the same type. Running DBSCAN with the following \(\varepsilon\) and minPts = 1 will extract the correct groups of attributes as long as there is no pairwise distance between attributes of different types that is a distance less than or equal to \(\varepsilon\). To see why, consider the pairwise Jaccard distances matrix shown in Example 4.2. If we are given the ground truth of the semantic types and choose our \(\varepsilon\) as \(\max (\max (\lbrace 0.15, 0.15\rbrace), \max (\lbrace 0.8, 0.8\rbrace))=0.8\), then when we run DBSCAN, we will correctly find groups {T1.country1, T2.country2} and {T3.title1, T4.title2}. However, assume now a case where \(J(\texttt {T2.country2, T3.title1})=0.50\), then even when we are given the ground truth and choose \(\varepsilon =0.8\), DBSCAN will not be able to extract the correct clusters. This is because there will be a pairwise distance between two attributes with different types (specifically, between T2.country2 and T3.title1) that is smaller than a pairwise distance between two attributes with the same type (specifically, between T1.title1 and T2.title2). Therefore, there is no \(\varepsilon\) value for which DBSCAN can correctly group the attributes.

In practice, we will not have the ground truth of the semantic types for each attribute and so we cannot exactly choose \(\varepsilon\) as we discussed above. However, there is a simple heuristic, as we briefly mentioned in Example 4.2 to choose an \(\varepsilon\). Notice that, ideally, our choice of epsilon should correspond to the largest pairwise distance between attributes of the same type, so, if we assume that distances between attributes of different types is typically very large (e.g., \(\ge 0.95\)), then we can take all the pairwise distances from the pairwise distances matrix, filter out those that are very large, and pick epsilon as the maximum of the remaining pairwise distances. More formally:

As we shall see in our experiments in Section 6.4, Heuristic 4.3 performs quite well when we choose a simple threshold (e.g., \(\tau =0.99\)), but we can come up with an even better way to choose \(\varepsilon\) that does not require us to specify a hard threshold. The main downside of Heuristic 4.3 is that even with a large threshold, \(\tau\), there are cases that it can filter out pairwise distances between attributes with the same type. For instance, homographs such as “Lincoln” that appear both as a first name but also as a city name suffer by the presence of an arbitrary threshold imposed by Heuristic 4.3. More specifically, the pairwise Jaccard distance between two city name columns can many times be larger than a specified threshold \(\tau\), since the domain of cities is much larger than that of first names in this benchmark. So, two columns that are both about cities may have a very small intersection. Nevertheless, even though the pairwise distance between attributes of the same type is very large, it most likely is not as large as the pairwise distance between attributes of different types, so there does exist an \(\varepsilon\) for which DBSCAN will identify the correct groups of attributes. We can find such an \(\varepsilon\) by estimating the density distribution of all the pairwise distances and choosing the left boundary of the last peak (the peak with the highest pairwise distance) in the density distribution as \(\varepsilon\). We can estimate the density distribution of the pairwise distances using Kernel Density Estimate (KDE), which fits a probability density distribution function over the input of the pairwise distances.

The reason we choose \(\varepsilon\) as the left boundary of the last peak is because we assume the last peak corresponds to the pairwise distances of attributes with different types, so if we choose an \(\varepsilon\) that corresponds to the pairwise distance just before the last peak (i.e., the left boundary of the last peak), then it should eliminate the clustering of attributes with different types together. We illustrate that process in Figure 6(b), which shows the density distribution found using KDE for the pair distances of all attributes that appear with the homograph “Lincoln.” Notice how close the last two peaks are. The last peak corresponds to the pairwise distances between attributes of different types, whereas the small second-to-last peak corresponds to the pairwise distances between city attributes where “Lincoln” appears in. The leftmost boundary of the last peak corresponds to a distance of about 0.988, so using Heuristic 4.4, we also set \(\varepsilon = 0.988\) and running DBSCAN correctly groups all attributes found with “Lincoln,” which in this case has three meanings, as it appears as a car brand, a first name, and as a city name.

An important parameter when running KDE over the pairwise distances is the number of samples (i.e., distinct distance positions on the x-axis in Figure 6) for which we estimate the density. In Figure 6(b), we have chosen 12,000 samples, whereas in Figure 6(a), we have chosen 1,000 samples. Notice that, because we did not estimate the density at enough points in Figure 6(a), the pairwise distances between the city attributes and pairs of attributes with different types are all represented in the last peak. Therefore, using Heuristic 4.4, we will extract an \(\varepsilon =0.918\), which is not optimal, since now the city attributes will no longer be grouped together. However, if we pick too many samples, then that is also a problem, as that can lead to overfitting to the pairwise distances and may create multiple peaks that correspond to the last peak in Figure 6(b), thus the selected \(\varepsilon\) would be too high and lead to too many groups. One way to handle this problem is to choose the number of samples to be dependent on the range of the non-zero pairwise distances. For instance, the pairwise distances in Figure 6(b) range from about 0.15 to 1, so a considerable number of samples is needed to detect the second-to-last peak separately from the last peak. Conversely, if the range is from 0.85 to 1, then we can separate the peaks with fewer samples and in fact choosing more samples will lead to overfitting. Empirically, we found a good way to dynamically assign the number of samples for KDE is to use the following formula:where range corresponds to the difference between the largest pairwise distance and the smallest non-zero pairwise distance. Notice that when the range is equal to 1, then the number of samples is just equal to the baseline number of samples and as the range decreases, we choose fewer samples to reduce overfitting.

We designed a small fully synthetic, but real-world inspired, data lake for a systematic validation of our approach. It consists of 13 tables generated using Mockaroo,¹⁰ which lets the data creator specify data sources from various categories.

Each table has 1,000 rows, except for two tables that contain countries and states. We used the real numbers of countries and US states of 193 and 50, respectively. There are 55 data values that are homographs, e.g., Sydney (city or name), Jamaica (city or country), Lincoln (first-name, car, or city), CA (country or state abbreviation), and Pumpkin (grocery product or movie title).

In the absence of homograph-labeled large real data lakes, we set out to find a closely related benchmark that we could adapt to our purposes. Unfortunately, while there are many table-based benchmarks, even some for data-semantics-related problems, they generally prove hard to adapt. For example, the VizNet corpus [41] used in semantic type detection in tables [43, 100] provided ground-truth labels for only a small fraction of the columns in the repository, making ground-truth discovery of all homograph labels practically impossible. We therefore selected the TUS benchmark [68], which contains real data and provides a ground-truth mapping for each column to the set of columns in the repository that it is unionable with. This enables us to automatically label all homographs. Let \(U(a)\) denote the set of columns (attributes) a given column a is unionable with and notice that a is always unionable with itself, hence \(a \in U(a)\). Let \(Attr(v)\) be the set of columns (attributes) a data value v appears in. Converting the TUS benchmark into our bipartite graph representation, we can automatically label data values as “unambiguous” or “homograph” based on the unionability ground truth.

Intuitively, a data value is a homograph if it appears in at least two different columns that are not unionable (and hence have different types). For instance, assume value “USA” appears in columns X1.country and X2.location. If the corresponding two columns are unionable, i.e., \(U(\texttt {X1.country})\) = \(U(\texttt {X2.location})\) = \(\lbrace \texttt {X1.country}, \texttt {X2.location}\rbrace\), then we can conclude that “USA” is an unambiguous value. In contrast, the columns containing the value “Jaguar” in the zoo or donor tables from Figure 1 are not unionable with either the company or car model tables, and hence “Jaguar” would be labeled a homograph.

Based on Definition 2 there are 164,364 unambiguous values and 26,035 homographs in the TUS benchmark, suggesting homographs are very abundant in real data lakes. Notice that attribute cardinalities in TUS are highly skewed, a common phenomenon in data lakes for open-data repositories [67]. Hence, this benchmark provides a “stress-test” for our approach. How well can it deal with both small and large cardinalities of attributes containing a homograph (in TUS, these cardinalities range from 3 to 22,703)?

Having real data is important, but we also need to understand the performance of our solution as the number of homographs in a data lake changes. To this end, we modified the TUS benchmark as follows: First, we removed all 26,035 homographs. Second, we carefully introduced artificial homographs with different properties. Since the artificial homographs are now the only ones in the data lake, we can measure how their properties affect our detection algorithm.

A homograph is injected by selecting two different data values from two columns that are not unionable. These original values are then replaced by a new unique value such as “InjectedHomograph1.” We only replaced string values with at least three characters. In our experiments, we vary the minimum allowed cardinality of the attributes containing values replaced with an injected homograph. We also vary the number of meanings of an injected homograph. This allows us to evaluate the effectiveness of our approach in identifying homographs with respect to the cardinality and number of meanings of the homographs.

We first use the SB to compare the homograph rankings obtained using the LCC and BC measures (Section 3) to study their ability to identify homographs. The bipartite graph for SB is relatively small, consisting of 17,672 nodes (17,633 data-value nodes and 39 attribute nodes) and 19,473 edges. We calculated the LCC and BC for each node in the graph and examined how these scores differ between homographs and unambiguous values.

Which measure is better at discovering homographs?. Figure 7 shows the top-55 data values based on LCC. For LCC, lower scores should in theory indicate a greater likelihood of being a homograph. Notice how more than 75% of the top-ranked data values are not homographs, meaning that a large number of unambiguous values have smaller LCC scores than the homographs. This is mainly caused by unambiguous values from small domains that do not co-occur often with many values in their domain. This confirms our hypothesis from Section 3 that LCC may not work well when homographs appear in small domains. In fact, the majority of the 55 homographs in the dataset have LCC scores significantly above 0.45, and so it is not necessarily true that homographs have low LCC cores. Overall, the results indicate that LCC scores do not provide an effective separation between homographs and unambiguous values.

However, the BC scores result in a vastly better top-55 result, as shown in Figure 8. Here, 38 out of the top-55 BC scores correspond to homographs. This is a much improved outcome over the LCC scores in Figure 7. But what happened to the remaining 17 homographs that are not in the top-55? We noticed that the remaining 17 homographs have betweenness scores of nearly zero and they all are values corresponding to homographs that are abbreviations of country and state names. Recall that these are the only two tables in SB with fewer than 1,000 tuples, where the state table contains only 50 tuples. This means that the BC score for values in these small domains cannot be very large, as there cannot be as many shortest paths that would pass through the homograph in question.

An explanation for the low BC scores for these homographs is the fact that there is considerable intersection between the country and state values, which is not the case with other homographs (e.g., the car brands and cities intersect only on the value Lincoln and Jaguar). This relatively large intersection also reduces the BC scores for those homographs, as the number of shortest paths connecting two nodes between cities and states is much larger. For example, going from the country code GR to the state code MA, the shortest path could be using the homograph AL (which is for Albania/Alabama) or CA (which is for Canada/California) or any other homograph between countries and states. As a result, those homographs receive lower BC scores, because the denominator in Equation (2) becomes large.

How good is previous work at finding homographs?. As discussed earlier, we compare DomainNet against a competitor based on \(D^4\) [71]. When applied to the SB dataset, \(D^4\) discovers four domains corresponding to Country, Country Code, Scientific Animal Name, and Scientific Plant Name. It maps the domains on 14 out of 39 table columns (attributes) in SB. Among these 14 attributes, there are 21 of the 55 homographs. Overall, when considering the top-55 results returned, the \(D^4\)-based algorithm disambiguates homographs in SB with a precision, recall, and F1-score of 38%. Using the BC score, DomainNet achieves for the top-55 results a precision, recall, and F1-score of 69%.

We now study the BC-score-based version of DomainNet in more detail on the large real-world dataset TUS-I with the injected homographs. Due to the cost of running BC for each node, all BC scores are approximated using 5,000 samples.¹³

How does cardinality affect homograph discovery?. Recall that after removing all original homographs in TUS, the TUS-I dataset only contains the homographs we methodically injected to study a specific effect on betweenness centrality. We ran our experiments by randomly selecting 50 pairs of values from different domains¹⁴ and replaced them with our 50 injected homographs. Each experiment was repeated four times with a different seed for selecting the values for replacement. Since the number of homographs in our experiment is always 50, in an ideal scenario the top-50 BC scores would correspond to exactly those injected homographs.

We found that cardinality has the expected impact on BC scores in terms of separating homographs and unambiguous values. If the data values chosen for replacement have a not-too-small cardinality (i.e., they co-occur with many other values), then the BC score of their injected homograph was notably higher. We confirmed this observation in Table 2, where we varied the cardinality threshold for the data values chosen for replacement. Overall, as we increased the cardinality threshold, a larger percentage of the injected homographs ranked in the top-50. In fact, if the replaced values had a cardinality of 500 or higher, then DomainNet consistently ranked at least 48 of the 50 injected homographs in the top 50. For reference, the largest attribute in TUS has 25,000 values and over half of all attributes have more than 500 values.

How does the number of meanings of a homograph affect homograph discovery?. In addition to varying the cardinality of the replaced values, we examined how the number of meanings of the injected homographs impacts their BC-based rankings. The number of meanings of an injected homograph is the number of values replaced for each injected homograph. The replaced values are all chosen from different domains to ensure that the injected homographs have consistently the specified number of meanings. We explored injected homographs with the number of meanings in the range 2 to 8 for replaced data values with a cardinality of 500 or higher. Table 3 shows that as we increase the number of meanings, DomainNet becomes better at discovering them. This is consistent with our intuition for betweenness centrality, since homographs with more meanings are more likely to be hub nodes that connect multiple sets of nodes with each other in our bipartite graph representation of the data lake.

Last, we explore the performance of DomainNet with betweenness centrality on the real TUS dataset with its 26,035 real homographs. Since the number of homographs is large, we not only report precision, recall, and F1-score for the top-26,035 results, but for all top-k with k from 1 all the way to the number of nodes in our graph, i.e., 190,399. We do not compare against the \(D^4\)-based algorithm for homographs, because \(D^4\) operates only on string attributes, and given the large number of numerical attributes, the \(D^4\) coverage will be even lower than in SB (where it only finds domains for 14 out of 39 attributes).

How does our approach perform on a real open-data benchmark?. Figure 9 shows the summary of our top-k evaluation results. Notice that for relatively small values of k such as \(k=200,\) our method can identify homograph values with high precision (0.89). Naturally, as we increase \(k,\) precision decreases and recall increases. At \(k=26,\!035\) (vertical line in Figure 9), which is the number of true homographs in the TUS benchmark, we achieve a precision, recall, and F1-score of 0.622. The highest F1-score occurs at \(k=29,\!633,\) where precision, recall, and F1-score are 0.615, 0.7, and 0.655, respectively.

It is important to emphasize that our approach is completely unsupervised and does not assume any external knowledge about the tables or their values. Existing state-of-the-art methods that tackle data integration tasks as described in Section 2 cannot be readily used for homograph identification or their coverage is severely limited (e.g., knowledge-based approaches like AIDA [97]).

Below, we report the top-10 values and their BC scores from the TUS benchmark:

All 10 data values are homographs based on the ground truth. Notice that, from a natural-language perspective, these 10 values do not seem to be homographs, but a closer look at the data revealed good reasons why they were labeled as homographs. For example, the value Music Faculty appears in two distinct contexts: as a geographic location/landmark in transportation-related tables as well as a department in university-related tables.

The value with the fifth-highest BC score is the period character. This may seem bizarre, but the period is used extensively as a null replacement in a large variety of tables and thus it acts as a homograph with a very large number of meanings. Finally, notice that we identify numerical values such as 50, 125, and 2, which appear in a variety of contexts such as addresses, identification numbers, quantity of products, and so on. Numerical values are traditionally difficult to deal with in many data-integration tasks, hence being able to identify some of them in a completely unsupervised manner is a notable step toward better coverage for numerical values.

We solve the problem of grouping attributes containing a homograph into groups that reflect the different meanings of the homograph using the method described in Section 4.1. We experiment with the choice of \(\varepsilon\) using both Heuristic 4.3 and Heuristic 4.4 as well as our choice of the number of samples that can either be fixed for any range of distances or assigned dynamically using Equation (3). In particular, we examine the precision in identifying the number of meanings for all homographs in the SB as well as the quality of the clusters of attributes (i.e., how well are attributes grouped together that contain the same meaning instances of a homograph) generated after running DBSCAN. Figure 10(a) shows the average precision in identifying the correct number of meanings for each homograph across three different methods: (1) Greatest assigns an \(\varepsilon\) using Heuristic 4.3 with threshold \(\tau =0.99\), (2) KDE assigns an \(\varepsilon\) using Heuristic 4.4, and (3) KDE (Dynamic number of samples) also assigns an \(\varepsilon\) using Heuristic 4.4, but the number of samples used by KDE is specified using Equation (3). Notice how the precision using Greatest in Figure 10(a) is constant, since it is not using KDE for which we vary the number of samples. Also the x-axis measure for KDE (Dynamic number of samples) in Figure 10(a) corresponds to the baseline number of samples, as the number of samples is determined using Equation (3).

As shown in Figure 10(a) both KDE and KDE (Dynamic number of samples) outperform Greatest for a large range of number of samples. Moreover, dynamically determining the number of samples rather than using a fixed number of samples when running KDE also seems to give a better precision, as we are less prone to overfitting the distributions with a small range of pairwise distances. This is further confirmed by looking at the deteriorating precision of KDE and KDE (Dynamic number of samples) as we increase the number of samples or baseline number of samples, respectively. In that case, excessive sampling can often lead to overfitting where there can be multiple distinct peaks in the KDE distribution that correspond to pairwise distances between attributes of different types, and as a result Heuristic 4.4 becomes less accurate. Notice also that KDE (Dynamic number of samples) retains a higher precision for a larger range of baseline number of samples than KDE, as it is able to reduce overfitting by using fewer samples for distributions of pairwise distances with a small range.

Figure 10(b) shows the average quality of the identified clusters for all homographs after running DBSCAN across the three different methods: Greatest, KDE, and KDE (Dynamic number of samples). Since we know the ground truth clusters in the SB, we evaluate the quality of our clusters using the adjusted Rand index [42]. The adjusted Rand index is an extrinsic evaluation measure for clustering that computes a similarity between two clustering assignments by considering all pairs of samples and counting pairs that are assigned to the same or different clusters in the predicted and ground truth clusters [76]. The score is further adjusted to account for chance and ranges from \(-\)1 to 1, where a score of 1 indicates a perfect prediction of clusters. As seen in Figure 10(b) the discovered clusters using DBSCAN are fairly close to the ground truth, which means that for most homographs, we can accurately group their attributes based on their type. The effectiveness of each of the three methods coincides with the pattern we observed for precision in Figure 10(a), where KDE (Dynamic number of samples) performs the best, as it reduces overfitting that can happen especially as we increase the number of samples for KDE.

The number of meanings of the homographs in SB is relatively small (only 2 or 3; see Table 1). Hence, to further stress-test our method, we also tested our precision and clustering quality by injecting null value homographs (meaning homographs with a large number of meanings) in the SB with results consistent with Figure 10. More specifically, we took the SB and injected null values by manually placing various null equivalent strings such as “missing” or “undefined” across multiple columns of different domains. In this way, SB contains some homographs (i.e., null values) with many more meanings than usual, and we can check if we can accurately identify the number of meanings and cluster the attributes for these injected null values. We found that, in most cases, we also predict a much larger number of meanings for the injected null values. Assuming that a large number of meanings often represents a null or a filler value, if we can find a number of meanings larger than a threshold, then we can still correctly identify this particular type of homograph. For example, the injected null value “missing” appears in 17 semantically different attributes, and our approach with KDE predicts that it appears in 16 distinct clusters, so even though the exact precision for the number of meanings is off, the clusters that we do find agree in a large part with the ground truth clusters.

As discussed in Section 3.5, Step 1 (graph construction) and Step 2 (centrality measure computation) are the most computationally expensive in our approach. In this section, we examine empirically the scalability of these steps.

The time to construct our bipartite graph is dependent on how long it takes to scan all input tables, which is a relatively fast operation. For example, the bipartite graph for the TUS dataset takes about 1.5 minutes to construct, which is how long it takes to read through each table in the dataset.

The runtime of Step 2 depends on the graph measure used. LCC is a local measure that is efficient to compute, but, as we demonstrated in Section 6.1, it is not as effective in finding homographs as BC is. Computing the LCC score for every node in the TUS dataset takes 4 seconds. For the global measure BC, since we are more interested in the score rankings rather than the scores themselves, approximating BC via sampling can significantly decrease the runtime without compromising quality.

In Figure 11(a), we examine how precision and runtime vary as we change the number of samples used for the approximate BC algorithm [36] on the TUS benchmark. Even for a small sample size (e.g., 1,000), precision stabilizes at 0.6. Notice that 1,000 samples correspond to around .5% of the nodes in the TUS graph, and it takes about 40 seconds for the algorithm to complete. The BC approximation has a complexity of \({\mathcal {O}}(sm),\) where s is the number of nodes sampled, and m the number of edges in the graph. Based on the literature and testing on our graphs, we found that sampling \(1\%\) of the nodes provides a good approximation of BC that is very consistent with the score rankings produced by the exact BC computation.

We also considered a bigger data lake to further test execution times—the NYC education open data dataset as used in \(D^4\) [71]. The bipartite graph representation of that dataset has roughly 1.5M nodes and 2.3M edges, which is an order of magnitude larger than the bipartite graph for the TUS dataset. The graph was constructed in 3.5 minutes, and the BC scores for every node were computed in 27 minutes using approximate BC on 1% of the nodes (\(\sim\)15k nodes).

To examine how runtime scales with graph size, we extracted random subgraphs¹⁵ of various sizes from the bipartite graph used for the NYC education dataset. We ran approximate BC for each graph by sampling \(1\%\) of its nodes and measured the runtime. Figure 11(b) shows that runtime increases linearly with graph size (i.e., number of edges), which is in accordance with the \({\mathcal {O}}(sm)\) complexity of the approximate BC algorithm.

In this section, we examine the computational speedup we gain by compressing our bipartite DomainNet graph before calculating its BC as described in Section 3.4. Figure 12(a) shows the accuracy of the approximate BC scores on the TUS dataset when using graph compression as we vary the percentage of nodes used for sampling as well as our mode of sampling. The y-axis measure in Figure 12(a) corresponds to Kendall’s \(\tau\) coefficient [49], which is a statistic to measure the similarity between two rankings. If two rankings are identical, then they have a Kendall’s \(\tau\) coefficient of 1, and if they are reversed, then \(\tau =-1\). Each point in the graph corresponds to a run of approximate BC with node compression and a specified percentage of nodes that were sampled using either a weighted or unweighted scheme. All nodes in the graph are then ranked in descending order by their computed approximate BC scores, and that ranking is then compared with the ranking produced by running exact BC. As seen in Figure 12(a), as the number of nodes sampled increases, the better the approximate BC becomes. Moreover, there is a clear improvement in the accuracy of the approximate BC scores when using weighted sampling, since we are more likely to choose large ident scored compressed nodes, which have larger impact in the BC rankings.

Figure 12(b) shows the amounts by which the bipartite DomainNet graphs for the SB, TUS, and NYC-EDU datasets were compressed. As seen, both the number of nodes and edges were reduced significantly across all datasets. For instance, the graph from the NYC-EDU dataset has originally \(\approx 1.47\)M nodes and was reduced to have only \(27,\!014\) nodes (i.e., the number of nodes was reduced by 98%). The amount of reduction is highly dependent on the data repetition and overlap across tables in the data lake, so the TUS dataset, which has a lot of tables that were generated by cutting larger tables horizontally and vertically, leads to compressed nodes that are generally smaller and thus the reduction factor is not as high against other datasets. Nevertheless, TUS still has a sizable 15\(\times\) reduction factor in the number of nodes in its compressed graph.

This reduction in the number of nodes and edges leads to significant computational speedups in BC. Running exact BC on the TUS takes about 150 minutes. After applying node compression to the graph, we can compute the exact BC for all nodes in only 38 seconds, a \(237\times\) speedup. A comparison of the exact BC runtimes with and without node compression is shown Table 4.

Supervised methods for semantic type detection (see Section 2.2) such as Sherlock [43] and SATO [100] can be adapted to solve our problem of identifying all homographs in a data lake. This can be done by first using their pre-trained models to predict a semantic type for each column in each table. Then a value is identified as a homograph if it is found in columns that are assigned at least two different semantic types. Notice that it is not necessary for the semantic type assignment to be accurate, as long as it is differentiable between columns that are truly semantically different, so homographs can still be identified. For example, if the value “empty” is found in two columns that are assigned the semantic types “first_name” and “plant_name,” but in ground truth their assigned types should have been “middle_name” and “plan_name,” respectively, then the value “empty” will still be correctly identified as a homograph, even though the semantic types assigned were not accurate. However, incorrect semantic type predication will have an impact in correctly grouping the meanings of the instances of the homographs. Also notice that this adaptation will only return a set of values determined as homographs or unambiguous values and will not provide a ranked list of homographs as DomainNet does.

Using the pre-trained models of Sherlock and SATO and the adaptation outlined above, we compare their homograph detection performance against DomainNet over the TUS, TUS-I, and NYC-EDU datasets. Additionally, we compare their performance in identifying the number of meanings of homographs as well as correctly clustering together the same meaning instances of a homograph. Table 5 shows the performance and runtime of Sherlock and SATO over the TUS benchmark against DomainNet at \(k=26,\!035\). DomainNet outperforms both Sherlock and SATO, but Sherlock performs quite well and has an overall higher precision (but lower recall and F1-score as well as slower performance). Sherlock is able to perform relatively well, since the TUS benchmark contains tables with columns that have semantic types over which Sherlock has a good coverage. We hypothesize that in datasets that are even more heterogeneous, Sherlock would perform considerably worse, a hypothesis that we confirmed when looking at the homograph detection results over the NYC-Education dataset shown in Table 6. One possible explanation for the very low performance results for SATO is due to the construction of the TUS benchmark. The TUS benchmark is constructed from larger tables that are sliced horizontally and vertically to construct many smaller table subsets, and SATO, unlike Sherlock, is a method that also considers row context information for its semantic type detection. The table splitting may be a cause for SATO’s worse performance (in homograph detection) over Sherlock. Note that SATO predicted that 130,859 of the 190,399 unique values of the TUS dataset were determined to belong in columns with at least two semantic types (i.e., homographs) even though there are only 26,035 homographs in ground truth. Because of this behavior as well as SATO’s slow running time, in the remainder of our experiments, we compared only against Sherlock as the best supervised semantic type detection method for homograph detection.

To test Sherlock on an even more heterogeneous dataset, we used the NYC-Education dataset. Since we do not know the homograph ground truth in the NYC-Education dataset, we selected the top-100 highest BC values identified using DomainNet and manually labeled them if they were indeed homographs. In the top-100, a total of 71 numeric values were present, which we removed to allow for a fair comparison against Sherlock, since it does not perform as well on numeric-only columns. This leaves us with 29 unique text values of which many were indeed homographs. For example, the value “Chemistry” appears both as a course as well as a title/description of a statue/landmark in a school. The value “–” appears as a null equivalent value for phone numbers, addresses, corporation names, and so on. The value “EMER” appears as a first name as well as a shorthand for emergency in project budget tables. As seen from Table 6. Sherlock’s precision is noticeably worse than in the TUS benchmark due to the presence of columns with semantic types that do fall into the 78 types Sherlock was trained on. As a consequence, the number of meanings predicted for each non-numeric value in the top-29 is also low. Similarly the quality of the clusters that group the same meaning instances of a homograph together are also low for Sherlock when compared to DomainNet.

We also tested Sherlock against DomainNet over our TUS-I benchmark, where we inject a small number of homographs over a cleaned dataset with no known homographs. More specifically, we experimented by varying the cardinality of the data values replaced by the injected homographs exactly as in Section 6.2. Figure 13 shows the performance of Sherlock vs. DomainNet, where although Sherlock achieves a slightly lower recall than DomainNet, its precision is extremely low. This is because Sherlock wrongly assigned different semantic types to many time and product code columns and thus a large number of values that were not homographs were identified as homographs.

We compare the performance of overlapping community detection algorithms over our bipartite graph to identify homographs. As discussed in Section 3.3, identifying homographs using community detection techniques can be challenging, as the number of communities can be very large in massive repositories of tables. Moreover, the communities are not disjoint, since values (specifically, homographs) can be members of multiple communities. We ran four SOTA overlapping community detection algorithms, namely, Big-Clam [94], LPANNI [63], DANMF [96], and Core Expansion [20], over the bipartite graph representation for the SB and the TUS benchmark. We used the implementations provided by cdlib [80], a popular Python library that provides implementations for multiple community detection algorithms with the aim of improving evaluation and benchmarking across them. Any cell-value node that is found in more than one community, we consider to be a homograph.

Table 7 shows the performance of the four algorithms over the SB, and Table 8 shows their performance over the TUS benchmark. As shown in Table 7 and Table 8, the overlapping community detection algorithms with the exception of core expansion [20] are unable to find a cell-value node that belongs in more than one community. Moreover, even though core expansion does find overlapping nodes, its performance is not comparable to that of DomainNet. We also experimented with using the dense cell nodes co-occurrence graph representation rather than our bipartite graph representation, but the graph becomes extremely large, and the overlapping community detection algorithms would not run in a reasonable amount of time (e.g., the cell nodes co-occurrence graph for just the synthetic benchmark has over 7 million edges).

As shown in Table 1 the number of homographs in a real data lake can be large. To further understand the impact of homographs on existing approaches, we consider the task of domain discovery and examine how knowing homographs a priori can benefit them.

We report the results of five different runs of \(D^4\) in Figure 14. The plots show the number of domains found by \(D^4\) (y-axis) as we vary the number and meanings of the injected homographs. To be fair in the comparison and to understand the impact of homographs on the domain discovery task, we use the TUS-I benchmark. We first ran \(D^4\) over the dataset without homographs and then over the same dataset with injected homographs. More specifically, we injected 50, 100, 150, and 200 homographs with 2, 4, and 6 meanings. In all the above configurations, the dataset always had 68 domains based on the ground truth. The horizontal line in Figure 14 shows that \(D^4\) returns 134 domains for TUS-I with no homographs. The difference in the number of domains based on the ground truth and \(D^4\)’s results is due to the nature of the TUS benchmark [68], as it is created from a set of large real open data tables that were randomly sliced vertically and horizontally. Consequently, in some cases the columns originating from the same table no longer share any values, causing \(D^4\) to discover more domains than there are based on ground truth.

As we increase the number and meanings of the injected homographs, \(D^4\) returns even more domains leading to lower accuracy. \(D^4\)’s output provides statistics about the maximum and the average number of domains assigned to a column. In the TUS-I with no homographs, that maximum is 2 and the average is almost 1 (i.e., 1.031), and it increases with the number of homographs. With 200 homographs, the maximum is 4 and the average is 1.04. We also ran \(D^4\) on the TUS-I with 5,000 injected homographs to simulate a dataset with a large proportion of homographs as in the TUS benchmark. The maximum domains per column is 22, and the average is 1.7 with a total of 371 domains found. The presence of homographs is negatively affecting \(D^4\) and causing it to erroneously assign larger numbers of heterogeneous domains to attributes as the number of homographs increases. Homograph discovery, therefore, is an important step that can be executed before domain discovery to improve its performance.

Although entity matching, as we discussed in Section 2, cannot be used to solve the homograph discovery problem, the presence of homographs can have an impact in the performance of even state-of-the-art transformer-based EM methods such as DITTO [59]. To show the impact of homographs, we examined some of the datasets used in DITTO’s evaluation, and in each dataset, we introduced some homographs by replacing two previously unambiguous values with a single homograph. For example, in the (Textual) Abt-Buy dataset, we can introduce a homograph for the brand name column by replacing all instances of the values “Microsoft” and “Yamaha” with the same string. We ensured that in all datasets and cases where this replacement occurred, our homographs would not change the ground truth for any pair of table rows that were used in evaluating the EM task. Moreover, only about 1% of the pairs of table rows were updated. Table 9 shows the performance of DITTO over four different datasets before and after the introduction of a few homographs. As shown, homographs can have a noticeable impact in the entity matching quality in the two highly textual datasets and thus being able to identify them in a pre-processing step can be beneficial. For instance, in the (WDC) Cameras Small dataset, 42 pairs of rows contained our injected homographs, of which 11 pairs were incorrectly predicted for the match/no match task even though they were all correctly predicted before the injection of homographs.