Effective and Efficient Data Cleaning for Entity Matching
Authors: 
Jing Ao, Rada ChirkovaAuthors Info & Claims
Article No.: 2, Pages 1 - 7
Abstract
As a key data-integration step, entity matching (EM) identifies tuples referring to the same real-world entities in disparate data sources. In many cases, the EM quality can be improved by repairing incorrect values in the data; at the same time, it is well known that the time costs of data cleaning by human experts could be prohibitive. In this paper, we focus on the time-consuming human-in-the-loop data-cleaning problem for relational EM, by recommending to human experts a time-efficient order in which values of attributes could be cleaned in the given data. Our proposed domain-independent cleaning framework aims to save human users' time, by guiding them in cleaning the EM inputs in an attribute order that is as conducive to maximizing EM accuracy as possible within a given constraint on the time they spend on cleaning. In guiding the cleaning process, our attribute-recommendation methods discover and take advantage of information provided by the data, and also use feedback from the EM engine. Our preliminary experimental results suggest that the proposed approach leads to measurable speedup, for a variety of time constraints, in the improvement of EM accuracy over the baseline approach, in which domain experts choose the sequence in which to clean the attributes of the inputs.
References
[1]
2019. OpenRefine(GoogleRefine). http://openrefine.org
[2]
2019. Trifacta. https://www.trifacta.com/
[3]
Ahmad Assadi, Tova Milo, and Slava Novgorodov. 2018. Cleaning Data with Constraints and Experts. In Proceedings of the 21st International Workshop on the Web and Databases (WebDB'18). ACM, New York, NY, USA, 1:1--1:6.
[4]
Moria Bergman, Tova Milo, Slava Novgorodov, and Wang-Chiew Tan. 2015. Query-Oriented Data Cleaning with Oracles. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (SIGMOD '15). ACM, New York, NY, USA, 1199--1214.
[5]
Gavin Brown, Adam Pocock, Ming-Jie Zhao, and Mikel Luján. 2012. Conditional Likelihood Maximisation: A Unifying Framework for Information Theoretic Feature Selection. J. Mach. Learn. Res. 13 (Jan. 2012), 27--66. http://dl.acm.org/citation.cfm?id=2188387
[6]
Peter Christen. 2012. Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection. Springer, New York, NY, USA.
[7]
Xu Chu, Ihab F. Ilyas, Sanjay Krishnan, and Jiannan Wang. 2016. Data Cleaning: Overview and Emerging Challenges. In Proceedings of the 2016 International Conference on Management of Data (SIGMOD '16). ACM, New York, NY, USA, 2201--2206.
[8]
Sanjib Das, AnHai Doan, Paul Suganthan G. C., Chaitanya Gokhale, and Pradap Konda. 2019. The Magellan Data Repository. https://sites.google.com/site/anhaidgroup/projects/data.
[9]
AnHai Doan. 2018. Human-in-the-Loop Data Analysis: A Personal Perspective. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics (HILDA'18). ACM, New York, NY, USA, 1:1--1:6.
[10]
AnHai Doan, Adel Ardalan, Jeffrey Ballard, Sanjib Das, Yash Govind, Pradap Konda, Han Li, Sidharth Mudgal, Erik Paulson, G. C. Paul Suganthan, and Haojun Zhang. 2017. Human-in-the-Loop Challenges for Entity Matching: A Midterm Report. In Proceedings of the 2Nd Workshop on Human-In-the-Loop Data Analytics (HILDA'17). ACM, New York, NY, USA, 12:1--12:6.
[11]
AnHai Doan, Alon Halevy, and Zachary Ives. 2012. Principles of Data Integration (1st ed.). Morgan Kaufmann Publishers, San Francisco, CA, USA.
[12]
AnHai Doan, Pradap Konda, Paul Suganthan G C, Adel Ardalan, Jeffrey R Ballard, Sanjib Das, Yash Govind, Han Li, Philip Martinkus, Sidharth Mudgal, Erik Paulson, and Haojun Zhang. 2018. Toward a System Building Agenda for Data Integration (and Data Science). IEEE Data Eng. Bull. 41, 2 (2018), 35--46. http://sites.computer.org/debull/A18june/p35.pdf
[13]
Xin Luna Dong and Divesh Srivastava. 2013. Big Data Integration. In 2013 IEEE 29th International Conference on Data Engineering (ICDE). IEEE, Piscataway, NJ, USA, 1245--1248.
[14]
Xin Luna Dong and Divesh Srivastava. 2015. Big Data Integration. Morgan & Claypool Publishers, San Rafael, CA, USA.
[15]
Ahmed K Elmagarmid, Panagiotis G Ipeirotis, and Vassilios S Verykios. 2007. Duplicate Record Detection: A Survey. IEEE Transactions on Knowledge and Data Engineering 19, 1 (Jan 2007), 1--16.
[16]
Wenfei Fan and Floris Geerts. 2012. Foundations of Data Quality Management. Morgan & Claypool Publishers, San Rafael, CA, USA.
[17]
Jian He, Enzo Veltri, Donatello Santoro, Guoliang Li, Giansalvatore Mecca, Paolo Papotti, and Nan Tang. 2016. Interactive and Deterministic Data Cleaning. In Proceedings of the 2016 International Conference on Management of Data (SIGMOD '16). ACM, New York, NY, USA, 893--907.
[18]
Ihab F. Ilyas and Xu Chu. 2015. Trends in Cleaning Relational Data: Consistency and Deduplication. Now Publishers, Hanover, MA, USA.
[19]
George H. John, Ron Kohavi, and Karl Pfleger. 1994. Irrelevant Features and the Subset Selection Problem. In Machine Learning, Proceedings of the Eleventh International Conference. Morgan Kaufmann Publishers, San Francisco, CA, USA, 121--129.
[20]
Pradap Konda, Sanjib Das, Paul Suganthan G. C., AnHai Doan, Adel Ardalan, Jeffrey R. Ballard, Han Li, Fatemah Panahi, Haojun Zhang, Jeff Naughton, Shishir Prasad, Ganesh Krishnan, Rohit Deep, and Vijay Raghavendra. 2016. Magellan: Toward Building Entity Matching Management Systems. Proc. VLDB Endow. 9, 12 (Aug. 2016), 1197--1208.
[21]
Sanjay Krishnan, Daniel Haas, Michael J. Franklin, and Eugene Wu. 2016. Towards Reliable Interactive Data Cleaning: A User Survey and Recommendations. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics (HILDA '16). ACM, New York, NY, USA, 9:1--9:5.
[22]
Sanjay Krishnan, Jiannan Wang, Eugene Wu, Michael J. Franklin, and Ken Goldberg. 2016. ActiveClean: Interactive Data Cleaning for Statistical Modeling. Proc. VLDB Endow. 9, 12 (Aug. 2016), 948--959.
[23]
Guoliang Li. 2017. Human-in-the-loop Data Integration. Proc. VLDB Endow. 10, 12 (Aug. 2017), 2006--2017.
[24]
Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P. Trevino, Jiliang Tang, and Huan Liu. 2016. Feature Selection: A Data Perspective. (2016). arXiv:1601.07996 http://arxiv.org/abs/1601.07996
[25]
Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P. Trevino, Jiliang Tang, and Huan Liu. 2017. Feature Selection: A Data Perspective. ACM Comput. Surv. 50, 6 (Dec. 2017), 94:1--94:45.
[26]
Patrick Emmanuel Meyer, Colas Schretter, and Gianluca Bontempi. 2008. Information-theoretic Feature Selection in Microarray Data Using Variable Complementarity. IEEE Journal of Selected Topics in Signal Processing 2, 3 (June 2008), 261--274.
[27]
Protiva Rahman, Courtney Hebert, and Arnab Nandi. 2018. ICARUS: Minimizing Human Effort in Iterative Data Completion. Proc. VLDB Endow. 11, 13 (Sept. 2018), 2263--2276.
[28]
Jiannan Wang, Sanjay Krishnan, Michael J. Franklin, Ken Goldberg, Tim Kraska, and Tova Milo. 2014. A Sample-and-clean Frame work for Fast and Accurate Query Processing on Dirty Data. In Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data (SIGMOD '14). ACM, New York, NY, USA, 469--480.
[29]
Mohamed Yakout, Ahmed K. Elmagarmid, Jennifer Neville, Mourad Ouzzani, and Ihab F. Ilyas. 2011. Guided Data Repair. Proc. VLDB Endow. 4, 5 (Feb. 2011), 279--289.
[30]
Howard Hua Yang and John Moody. 1999. Data Visualization and Feature Selection: New Algorithms for Nongaussian Data. In Proceedings of the 12th International Conference on Neural Information Processing Systems (NIPS'99). MIT Press, Cambridge, MA, USA, 687--693. http://dl.acm.org/citation.cfm?id=3009657.3009755
[31]
Lei Yu and Huan Liu. 2004. Efficient Feature Selection via Analysis of Relevance and Redundancy. J. Mach. Learn. Res. 5 (Dec. 2004), 1205--1224. http://dl.acm.org/citation.cfm?id=1005332.1044700
Index Terms
- Effective and Efficient Data Cleaning for Entity Matching
Recommendations
Data Cleaning: Overview and Emerging Challenges
SIGMOD '16: Proceedings of the 2016 International Conference on Management of DataDetecting and repairing dirty data is one of the perennial challenges in data analytics, and failure to do so can result in inaccurate analytics and unreliable decisions. Over the past few years, there has been a surge of interest from both industry and ...
A grammar-based entity representation framework for data cleaning
SIGMOD '09: Proceedings of the 2009 ACM SIGMOD International Conference on Management of dataFundamental to data cleaning is the need to account for multiple data representations. We propose a formal framework that can be used to reason about and manipulate data representations. The framework is declarative and combines elements of a generative ...
A Comparative Study of Data Cleaning Tools
In the information era, data is crucial in decision making. Most data sets contain impurities that need to be weeded out before any meaningful decision can be made from the data. Hence, data cleaning is essential and often takes more than 80 percent ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
July 2019
67 pages
ISBN:9781450367912
DOI:10.1145/3328519
Copyright © 2019 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 05 July 2019
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
SIGMOD/PODS '19
Sponsor:
Acceptance Rates
HILDA '19 Paper Acceptance Rate 12 of 24 submissions, 50%;
Overall Acceptance Rate 28 of 56 submissions, 50%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 241Total Downloads
- Downloads (Last 12 months)19
- Downloads (Last 6 weeks)2
Reflects downloads up to 22 Sep 2024
Other Metrics
Citations
Cited By
View allView Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in