research-article

Development and user experiences of an open source data cleaning, deduplication and record linkage system

Author:

Peter ChristenAuthors Info & Claims

ACM SIGKDD Explorations Newsletter, Volume 11, Issue 1

Pages 39 - 48

https://doi.org/10.1145/1656274.1656282

Published: 16 November 2009 Publication History

Abstract

Record linkage, also known as database matching or entity resolution, is now recognised as a core step in the KDD process. Data mining projects increasingly require that information from several sources is combined before the actual mining can be conducted. Also of increasing interest is the deduplication of a single database. The objectives of record linkage and deduplication are to identify, match and merge all records that relate to the same real-world entities. Because real-world data is commonly 'dirty', data cleaning is an important first step in many deduplication, record linkage, and data mining project.

In this paper, an overview of the Febrl (Freely Extensible Biomedical Record Linkage) system is provided, and the results of a recent survey of Febrl users is discussed. Febrl includes a variety of functionalities required for data cleaning, deduplication and record linkage, and it provides a graphical user interface that facilitates its application for users who do not have programming experience.

References

[1]

A. Aizawa and K. Oyama. A fast linkage detection scheme for multi-source information integration. In International Workshop on Challenges in Web Information Retrieval and Integration (WIRI'05), pages 30--39, Tokyo, 2005.

Digital Library

[2]

R. Baxter, P. Christen, and T. Churches. A comparison of fast blocking methods for record linkage. In ACM SIGKDD'03 workshop on Data Cleaning, Record Linkage and Object Consolidation, pages 25--27, Washington DC, 2003.

[3]

P. Christen. A comparison of personal name matching: Techniques and practical issues. In Workshop on Mining Complex Data (MCD'06), held at IEEE ICDM'06, Hong Kong, 2006.

Digital Library

[4]

P. Christen. A two-step classification approach to unsupervised record linkage. In Australasian Data Mining Conference (AusDM'07), Conferences in Research and Practice in Information Technology (CRPIT), volume 70, pages 111--119, Gold Coast, Australia, 2007.

Digital Library

[5]

P. Christen. Automatic record linkage using seeded nearest neighbour and support vector machine classification. In ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD'08), pages 151--159, Las Vegas, 2008.

Digital Library

[6]

P. Christen. Automatic training example selection for scalable unsupervised record linkage. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD'08), Springer LNAI 5012, pages 511--518, Osaka, Japan, 2008.

Digital Library

[7]

P. Christen. Febrl - An open source data cleaning, deduplication and record linkage system with a graphical user interface (Demonstration Session). In ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD'08), pages 1065--1068, Las Vegas, 2008.

Digital Library

[8]

P. Christen and D. Belacic. Automated probabilistic address standardisation and verification. In Australasian Data Mining Conference (AusDM'05), pages 53--67, Sydney, 2005.

[9]

P. Christen, T. Churches, and M. Hegland. Febrl - A parallel open source data linkage system. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD'04), Springer LNAI 3056, pages 638--647, Sydney, 2004.

[10]

P. Christen and K. Goiser. Quality and complexity measures for data linkage and deduplication. In F. Guillet and H. Hamilton, editors, Quality Measures in Data Mining, volume 43 of Studies in Computational Intelligence, pages 127--151. Springer, 2007.

[11]

P. Christen and A. Pudjijono. Accurate synthetic generation of realistic personal information. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD'09), Springer LNAI 5476, pages 507--514, Bangkok, Thailand, 2009.

Digital Library

[12]

T. Churches, P. Christen, K. Lim, and J.X. Zhu. Preparation of name and address data for record linkage using hidden Markov models. BioMed Central Medical Informatics and Decision Making, 2(9), 2002.

[13]

D.E. Clark. Practical introduction to record linkage for injury research. British Medical Journal, 10(3):186--191, 2004.

[14]

W.W. Cohen, P. Ravikumar, and S. Fienberg. A comparison of string distance metrics for name-matching tasks. In Workshop on Information Integration on the Web (IIWeb'03), held at IJCAI'03, pages 73--78, Acapulco, 2003.

[15]

W.W. Cohen and J. Richman. Learning to match and cluster large high-dimensional data sets for data integration. In ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD'02), pages 475--480, Edmonton, 2002.

Digital Library

[16]

A. Elmagarmid, P. Ipeirotis, and V. Verykios. Duplicate record detection: A survey. IEEE Transactions on Knowledge and Data Engineering, 19(1):1--16, 2007.

Digital Library

[17]

I.P. Fellegi and A.B. Sunter. A theory for record linkage. Journal of the American Statistical Society, 64(328):1183--1210, 1969.

[18]

K. Goiser and P. Christen. Towards automated record linkage. In Australasian Data Mining Conference (AusDM'06), Conferences in Research and Practice in Information Technology (CRPIT), volume 61, pages 23--31, Sydney, 2006.

Digital Library

[19]

L. Gu and R. Baxter. Decision models for record linkage. In Selected Papers from AusDM, Springer LNCS 3755, pages 146--160, 2006.

Digital Library

[20]

M.A. Hernandez and S.J. Stolfo. The merge/purge problem for large databases. In ACM International Conference on Management of Data (SIGMOD'95), pages 127--138, San Jose, 1995.

Digital Library

[21]

L. Jin, C. Li, and S. Mehrotra. Efficient record linkage in large data sets. In International Conference on Database Systems for Advanced Applications (DASFAA'03), pages 137--146, Tokyo, 2003.

Digital Library

[22]

G. Williams. Data mining with Rattle and R. Togaware, Canberra, 2009. Software available at: http://rattle.togaware.com.

[23]

W. Winkler. Methods for evaluating and creating data quality. Elsevier Information Systems, 29(7):531--550, 2004.

Digital Library

[24]

W. E. Yancey. BigMatch: A program for extracting probable matches from a large file for record linkage. Technical Report RR2007/01, US Bureau of the Census, 2007.

Cited By

Yin MZheng YDuan SJiao LYang H(2023)User entity alignment method based on cross-attribute knowledge associationProceedings of the 2023 7th International Conference on Computer Science and Artificial Intelligence10.1145/3638584.3638602(53-61)Online publication date: 8-Dec-2023
https://dl.acm.org/doi/10.1145/3638584.3638602
Yu SChen THan LDemartini GSadiq S(2023)DataOps-4G: On Supporting Generalists in Data Quality DiscoveryIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.315160535:5(4668-4681)Online publication date: 1-May-2023
https://dl.acm.org/doi/10.1109/TKDE.2022.3151605
Mon Myint KNaing W(2023)Historical Census Data Linkage: Graph-based Household Matching Method2023 IEEE Conference on Computer Applications (ICCA)10.1109/ICCA51723.2023.10181546(351-356)Online publication date: 27-Feb-2023
https://doi.org/10.1109/ICCA51723.2023.10181546
Show More Cited By

Index Terms

Development and user experiences of an open source data cleaning, deduplication and record linkage system
1. Information systems
  1. Information systems applications

Recommendations

Febrl -: an open source data cleaning, deduplication and record linkage system with a graphical user interface
KDD '08: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining

Matching records that refer to the same entity across data-bases is becoming an increasingly important part of many data mining projects, as often data from multiple sources needs to be matched in order to enrich data or improve its quality. Significant ...
A note on using the F-measure for evaluating record linkage algorithms

Record linkage is the process of identifying and linking records about the same entities from one or more databases. Record linkage can be viewed as a classification problem where the aim is to decide whether a pair of records is a match (i.e. two ...
A Survey of Indexing Techniques for Scalable Record Linkage and Deduplication

Record linkage is the process of matching records from several databases that refer to the same entities. When applied on a single database, this process is known as deduplication. Increasingly, matched data are becoming important in many application ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGKDD Explorations Newsletter

ACM SIGKDD Explorations Newsletter Volume 11, Issue 1

June 2009

56 pages

ISSN:1931-0145

EISSN:1931-0153

DOI:10.1145/1656274

Issue’s Table of Contents

Copyright © 2009 Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 November 2009

Published in SIGKDD Volume 11, Issue 1

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
551
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yin MZheng YDuan SJiao LYang H(2023)User entity alignment method based on cross-attribute knowledge associationProceedings of the 2023 7th International Conference on Computer Science and Artificial Intelligence10.1145/3638584.3638602(53-61)Online publication date: 8-Dec-2023
https://dl.acm.org/doi/10.1145/3638584.3638602
Yu SChen THan LDemartini GSadiq S(2023)DataOps-4G: On Supporting Generalists in Data Quality DiscoveryIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.315160535:5(4668-4681)Online publication date: 1-May-2023
https://dl.acm.org/doi/10.1109/TKDE.2022.3151605
Mon Myint KNaing W(2023)Historical Census Data Linkage: Graph-based Household Matching Method2023 IEEE Conference on Computer Applications (ICCA)10.1109/ICCA51723.2023.10181546(351-356)Online publication date: 27-Feb-2023
https://doi.org/10.1109/ICCA51723.2023.10181546
Zhan TFang YHe W(2023)Privacy-Preserving Record Linkage for Real-World DataReal-World Evidence in Medical Product Development10.1007/978-3-031-26328-6_7(109-122)Online publication date: 20-Jan-2023
https://doi.org/10.1007/978-3-031-26328-6_7
Khan SKhan AHoque A(2022)Privacy preserved incremental record linkageJournal of Big Data10.1186/s40537-022-00655-79:1Online publication date: 14-Nov-2022
https://doi.org/10.1186/s40537-022-00655-7
Carvalho MMangaravite VPonce LCantelli LCampoi BNunes GMiranda de Paiva BLaender AGoncalves M(2022)Deduplicating Large Volumes of Data from Natural and Legal Entities in the Governmental Field2022 IEEE International Conference on Big Data (Big Data)10.1109/BigData55660.2022.10020407(2206-2213)Online publication date: 17-Dec-2022
https://doi.org/10.1109/BigData55660.2022.10020407
Hildebrandt KPanse FWilcke NRitter N(2020)Large-Scale Data Pollution with Apache SparkIEEE Transactions on Big Data10.1109/TBDATA.2016.26373786:2(396-411)Online publication date: 1-Jun-2020
https://doi.org/10.1109/TBDATA.2016.2637378
Zhao CHe Y(2019)Auto-EM: End-to-end Fuzzy Entity-Matching using Pre-trained Deep Models and Transfer LearningThe World Wide Web Conference10.1145/3308558.3313578(2413-2424)Online publication date: 13-May-2019
https://dl.acm.org/doi/10.1145/3308558.3313578
Myint KNaing W(2019)Analysis of Historical Census using Graph-based Household Matching Method2019 International Conference on Advanced Information Technologies (ICAIT)10.1109/AITC.2019.8920872(102-107)Online publication date: Nov-2019
https://doi.org/10.1109/AITC.2019.8920872
Al-janabi SJanicki R(2019)Generation and Corruption of Semi-Structured and Structured DataFrom Security to Community Detection in Social Networking Platforms10.1007/978-3-030-11286-8_7(159-169)Online publication date: 10-Apr-2019
https://doi.org/10.1007/978-3-030-11286-8_7
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents