research-article

Synthesizing Type-Detection Logic for Rich Semantic Data Types using Open-source Code

Authors:

Yeye HeAuthors Info & Claims

SIGMOD '18: Proceedings of the 2018 International Conference on Management of Data

Pages 35 - 50

https://doi.org/10.1145/3183713.3196888

Published: 27 May 2018 Publication History

Abstract

Given a table of data, existing systems can often detect basic atomic types (e.g., strings vs. numbers) for each column. A new generation of data-analytics and data-preparation systems are starting to automatically recognize rich semantic types such as date-time, email address, etc., for such metadata can bring an array of benefits including better table understanding, improved search relevance, precise data validation, and semantic data transformation. However, existing approaches only detect a limited number of types using regular-expression-like patterns, which are often inaccurate, and cannot handle rich semantic types such as credit card and ISBN numbers that encode semantic validations (e.g., checksum).

We developed AUTOTYPE from open-source repositories like GitHub. Users only need to provide a set of positive examples for a target data type and a search keyword, our system will automatically identify relevant code, and synthesize type-detection functions using execution traces. We compiled a benchmark with 112 semantic types, out of which the proposed system can synthesize code to detect 84 such types at a high precision. Applying the synthesized type-detection logic on web table columns have also resulted in a significant increase in data types discovered compared to alternative approaches.

References

[1]

Btrace - a tracing tool for java. https://github.com/btraceio/btrace.

[2]

Calculate GS1 check digit. https://www.gs1.org/how-calculate-check-digit-manually.

[3]

Codeplex. https://www.codeplex.com/.

[4]

Enterprise version of the proprietary github. https://enterprise.github.com/home.

[5]

GitHubGist. https://gist.github.com/.

[6]

Javascript debugger statement. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/debugger.

[7]

Luhn algorithm. https://en.wikipedia.org/wiki/Luhn_algorithm.

[8]

Python built-in ast parser. https://docs.python.org/2/library/ast.html.

[9]

Ruby debugging. http://guides.rubyonrails.org/debugging_rails_applications.html.

[10]

Trifacta built-in data types. https://docs.trifacta.com/display/PE/Supported Data Types.

[11]

A. Arulselvan. A note on the set union knapsack problem. Discrete Applied Mathematics, 169:214--218, 2014.

Digital Library

[12]

S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives. Dbpedia: A nucleus for a web of open data. The semantic web, pages 722--735, 2007.

Digital Library

[13]

P. A. Bernstein. Applying model management to classical meta data problems. In CIDR, 2003.

[14]

M. J. Cafarella, A. Halevy, D. Z. Wang, E. Wu, and Y. Zhang. Webtables: exploring the power of tables on the web. Proceedings of VLDB, (1), 2008.

Digital Library

[15]

K. Chakrabarti, S. Chaudhuri, Z. Chen, K. Ganjam, Y. He, and W. Redmond. Data services leveraging bing's data assets. IEEE Data Eng. Bull., 39(3):15--28, 2016.

[16]

X. Chu, Y. He, K. Chakrabarti, and K. Ganjam. Tegra: Table extraction by global record alignment. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pages 1713--1728. ACM, 2015.

Digital Library

[17]

X. Chu, J. Morcos, I. F. Ilyas, M. Ouzzani, P. Papotti, N. Tang, and Y. Ye. Katara: A data cleaning system powered by knowledge bases and crowdsourcing. In Proceedings of SIGMOD, 2015.

Digital Library

[18]

E. Cortez, P. A. Bernstein, Y. He, and L. Novik. Annotating database schemas to help enterprise search. Proceedings of VLDB, (12), 2015.

Digital Library

[19]

R. Dhamankar, Y. Lee, A. Doan, A. Halevy, and P. Domingos. imap: discovering complex semantic matches between database schemas. In Proceedings of SIGMOD, 2004.

Digital Library

[20]

M. Hazewinkel, editor. Encyclopedia of Mathematics. Springer Science

[21]

Business Media B.V. / Kluwer Academic Publishers, 2001.

[22]

Y. He, K. Chakrabarti, T. Cheng, and T. Tylenda. Automatic discovery of attribute synonyms using query logs and table corpora. In Proceedings of WWW. International World Wide Web Conferences Steering Committee, 2016.

Digital Library

[23]

Y. He, K. Ganjam, K. Lee, Y. Wang, V. Narasayya, S. Chaudhuri, X. Chu, and Y. Zheng. Transform-Data-by-Example (TDE): Extensible Data Transformation in Excel. In SIGMOD, 2018.

Digital Library

[24]

K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422--446, 2002.

Digital Library

[25]

P. G. Kolaitis. Schema mappings, data exchange, and metadata management. In Proceedings of PODS, 2005.

Digital Library

[26]

C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. J. Reddi, and K. Hazelwood. Pin: building customized program analysis tools with dynamic instrumentation. In Acm sigplan notices, 2005.

Digital Library

[27]

C. D. Manning, P. Raghavan, and H. Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008.

[28]

C. D. Manning, P. Raghavan, and H. Schütze. Scoring, term weighting and the vector space model. Introduction to information retrieval, 100:2--4, 2008.

[29]

J. L. McCarthy. Metadata management for large statistical databases. In Proceedings of VLDB, 1982.

Digital Library

[30]

T. M. Mitchell. Machine learning., 1997.

Digital Library

[31]

J. Morcos, Z. Abedjan, I. F. Ilyas, M. Ouzzani, P. Papotti, and M. Stonebraker. Dataxformer: An interactive data transformation tool. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pages 883--888, 2015.

Digital Library

[32]

F. Naumann, C.-T. Ho, X. Tian, L. M. Haas, and N. Megiddo. Attribute classification using feature analysis. In Proceedings of ICDE, 2002.

Digital Library

[33]

V. Raman and J. M. Hellerstein. Potter's wheel: An interactive data cleaning system. In Proceedings of VLDB, 2001.

Digital Library

[34]

D. Ritze, O. Lehmberg, and C. Bizer. Matching html tables to dbpedia. In Proceedings of WIMS, page 10. ACM, 2015.

Digital Library

[35]

A. Sen. Metadata management: past, present and future. Decision Support Systems, (1), 2004.

Digital Library

[36]

Y. L. Simmhan, B. Plale, and D. Gannon. A survey of data provenance in e-science. SIGMOD Rec., 2005.

Digital Library

[37]

F. M. Suchanek, G. Kasneci, and G. Weikum. Yago: a core of semantic knowledge. In Proceedings of WWW, 2007.

Digital Library

[38]

A. Talaika, J. Biega, A. Amarilli, and F. M. Suchanek. Ibex: harvesting entities from the web using unique identifiers. In Proceedings of WebDB, 2015.

Digital Library

[39]

Y. Wang and Y. He. Synthesizing mapping relationships using table corpus. In Proceedings of SIGMOD, 2017.

Digital Library

Cited By

Li PHe YYashar DCui WGe SZhang HRifinski Fainman DZhang DChaudhuri S(2024)Table-GPT: Table Fine-tuned GPT for Diverse Table TasksProceedings of the ACM on Management of Data10.1145/36549792:3(1-28)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654979
Mior M(2023)Learning from Uncurated Regular Expressions for Semantic Type ClassificationProceedings of the 1st Workshop on Simplicity in Management of Data10.1145/3596225.3596226(1-5)Online publication date: 23-Jun-2023
https://dl.acm.org/doi/10.1145/3596225.3596226
Namysł MEsser ABehnke SKöhler J(2023)Flexible Hybrid Table Recognition and Semantic Interpretation SystemSN Computer Science10.1007/s42979-022-01659-z4:3Online publication date: 4-Mar-2023
https://dl.acm.org/doi/10.1007/s42979-022-01659-z
Show More Cited By

Index Terms

Synthesizing Type-Detection Logic for Rich Semantic Data Types using Open-source Code
1. Theory of computation
  1. Theory and algorithms for application domains
    1. Database theory
      1. Data integration
      2. Data modeling

Recommendations

Code semantic enrichment for deep code search
Abstract
Code search aims to retrieve code snippets from a large-scale codebase, where the semantics of the searched code match developers’ query intent. Code is a low-level implementation of programming intents, but query is always expressed as clear and ...
Graphical abstract

Display Omitted
Highlights
- Finding that the code semantics can be enriched by incorporating with the description of its most similar code.
- Proposing a code semantic enrichment approach named SemEnr for deep code search.
- Evaluating the performance of SemEnr ...
Learning to rank code examples for code search engines

Source code examples are used by developers to implement unfamiliar tasks by learning from existing solutions. To better support developers in finding existing solutions, code search engines are designed to locate and rank code examples relevant to user'...
Type-Indexed Data Types
MPC '02: Proceedings of the 6th International Conference on Mathematics of Program Construction

A polytypic function is a function that can be instantiated on many data types to obtain data type specific functionality. Examples of polytypic functions are the functions that can be derived in Haskell, such as show, read, and '=='. More advanced ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '18: Proceedings of the 2018 International Conference on Management of Data

May 2018

1874 pages

ISBN:9781450347037

DOI:10.1145/3183713

General Chairs:
Gautam Das
University of Texas at Arlington, USA
,
Christopher Jermaine
Rice University, USA
,
Philip Bernstein
Microsoft Research, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGMOD/PODS '18

Sponsor:

SIGMOD

SIGMOD/PODS '18: International Conference on Management of Data

June 10 - 15, 2018

TX, Houston, USA

Acceptance Rates

SIGMOD '18 Paper Acceptance Rate 90 of 461 submissions, 20%;

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
526
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)4

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li PHe YYashar DCui WGe SZhang HRifinski Fainman DZhang DChaudhuri S(2024)Table-GPT: Table Fine-tuned GPT for Diverse Table TasksProceedings of the ACM on Management of Data10.1145/36549792:3(1-28)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654979
Mior M(2023)Learning from Uncurated Regular Expressions for Semantic Type ClassificationProceedings of the 1st Workshop on Simplicity in Management of Data10.1145/3596225.3596226(1-5)Online publication date: 23-Jun-2023
https://dl.acm.org/doi/10.1145/3596225.3596226
Namysł MEsser ABehnke SKöhler J(2023)Flexible Hybrid Table Recognition and Semantic Interpretation SystemSN Computer Science10.1007/s42979-022-01659-z4:3Online publication date: 4-Mar-2023
https://dl.acm.org/doi/10.1007/s42979-022-01659-z
Piai FAtzeni PMerialdo PSrivastava D(2022)Fine-grained semantic type discovery for heterogeneous sources using clusteringThe VLDB Journal10.1007/s00778-022-00743-332:2(305-324)Online publication date: 17-May-2022
https://doi.org/10.1007/s00778-022-00743-3
Song JHe YLi GLi ZIdreos SSrivastava D(2021)Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data LakesProceedings of the 2021 International Conference on Management of Data10.1145/3448016.3457250(1678-1691)Online publication date: 9-Jun-2021
https://dl.acm.org/doi/10.1145/3448016.3457250
Chen BAbedjan ZGrundy J(2021)Interactive cross-language code retrieval with auto-encodersProceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE51524.2021.9678929(167-178)Online publication date: 15-Nov-2021
https://dl.acm.org/doi/10.1109/ASE51524.2021.9678929
Bavishi RLaddad SYoshida HPrasad MSen KGrundy J(2021)VizSmithProceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE51524.2021.9678696(129-141)Online publication date: 15-Nov-2021
https://dl.acm.org/doi/10.1109/ASE51524.2021.9678696
Jin ZHe YChauduri S(2020)Auto-transformProceedings of the VLDB Endowment10.14778/3407790.340783113:12(2368-2381)Online publication date: 14-Sep-2020
https://dl.acm.org/doi/10.14778/3407790.3407831
Zhang DHulsebos MSuhara YDemiralp ÇLi JTan W(2020)SatoProceedings of the VLDB Endowment10.14778/3407790.340779313:12(1835-1848)Online publication date: 1-Jul-2020
https://dl.acm.org/doi/10.14778/3407790.3407793
Zhao CHe Y(2019)Auto-EM: End-to-end Fuzzy Entity-Matching using Pre-trained Deep Models and Transfer LearningThe World Wide Web Conference10.1145/3308558.3313578(2413-2424)Online publication date: 13-May-2019
https://dl.acm.org/doi/10.1145/3308558.3313578
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents