Article

Data quality through knowledge engineering

Authors:

Tamraparni Dasu,

Gregg T. Vesonder,

Jon R. WrightAuthors Info & Claims

KDD '03: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining

Pages 705 - 710

https://doi.org/10.1145/956750.956844

Published: 24 August 2003 Publication History

Abstract

Traditionally, data quality programs have acted as a preprocessing stage to make data suitable for a data mining or analysis operation. Recently, data quality concepts have been applied to databases that support business operations such as provisioning and billing. Incorporating business rules that drive operations and their associated data processes is critically important to the success of such projects. However, there are many practical complications. For example, documentation on business rules is often meager. Rules change frequently. Domain knowledge is often fragmented across experts, and those experts do not always agree. Typically, rules have to be gathered from subject matter experts iteratively, and are discovered out of logical or procedural sequence, like a jigsaw puzzle. Our approach is to impement business rules as constraints on data in a classical expert system formalism sometimes called production rules. Our system works by allowing good data to pass through a system of constraints unchecked. Bad data violate constraints and are flagged, and then fed back after correction. Constraints are added incrementally as better understanding of the business rules is gained. We include a real-life case study.

References

[1]

M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander. LOF: Identifying density-based local outliers. In Proc. ACM SIGMOD Conf., pages 93--104, 2000.]]

Digital Library

[2]

T. Dasu. and T. Johnson. Exploratory Data Mining and Data Cleaning. John Wiley, New York, 2003.]]

Digital Library

[3]

L. D. Delwiche. and S. J. Slaughter. The little sas book-a primer, 1998. Second edition.]]

Digital Library

[4]

C. L. Forgy. Ops5 user's manual, 1981. Technical Report CMU-CS-81-135.]]

[5]

E. J. Friedman-Hill. Jess. http://herzberg.ca.sandia.gov/jess, 1997. Sandia National Laboratories.]]

[6]

J. C. Giarratano. Expert Systems: Principals and Programming. BrooksCole Publishing Co., 1998.]]

Digital Library

[7]

M. Hernandez nd S. Stolfo. Real-world data is dirty: Data cleansing and the merge/purge problem. Data Mining and Knowledge Discovery, 2(1):9--37, 1998.]]

Digital Library

[8]

T. Johnson and T. Dasu. Comparing mssive high-dimensional data sets. In Knowledge Discovery and Data Mining, pages 229--233, 1998.]]

[9]

E. Knorr and R. Ng. Algorithms for mining distance-based outliers in large datasets. In Proc. Intl. Conf. Very Large Data Bases, pages 392--403, 1998.]]

Digital Library

[10]

R. J. A. Little and D. B. Rubin. Statistical Analysis with Missing Data. Wiley, New York, 1987.]]

Digital Library

[11]

NASA. Clips. http://siliconvalleynone.com/clips/html. NASA Johnson Space Center.]]

[12]

A. Newell. Production Systems: Models of Control Structures in Visual Information Processing. New York: Academic Press, 1973.]]

[13]

R. Ramakrishnan. and P. J. S. (Ed). Constraints and Databases. Kluwer Academic, 1998.]]

Digital Library

[14]

T. Redman. Data Quality: Management and Technology. Bantam Books, New York, 1992.]]

Digital Library

[15]

T. Redman. Data Quality: The Field Guide. Digital Press (Elsevier), 2001.]]

Digital Library

[16]

J. R. Rowland. and G. T. Vesonder. The c5 user manual release 1.0, 1987.]]

[17]

J. Tukey. Exploratory Data Analysis: Addison-Wesley, Reading, 1977.]]

[18]

G. T. Vesonder. Rule-based programming in the unix system. AT&T Technical Journal, pages 69--80, January 1988.]]

Cited By

Caballero IGualo FRodríguez MPiattini M(2022)BR4DQInformation Systems10.1016/j.is.2022.102058109:COnline publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1016/j.is.2022.102058
Ngueilbaye AWang HMahamat DElgendy IJunaidu S(2021)Methods for detecting and correcting contextual data quality problemsIntelligent Data Analysis10.3233/IDA-20528225:4(763-787)Online publication date: 9-Jul-2021
https://doi.org/10.3233/IDA-205282
Arias RPerez-Yauli FGudino F(2019)A Methodological Aproach to the Determination of the Future Electrical Spatial Demand in Electric Distribution Companies in Ecuador2019 IEEE PES Innovative Smart Grid Technologies Conference - Latin America (ISGT Latin America)10.1109/ISGT-LA.2019.8895274(1-6)Online publication date: Sep-2019
https://doi.org/10.1109/ISGT-LA.2019.8895274
Show More Cited By

Index Terms

Data quality through knowledge engineering
1. Computing methodologies
  1. Machine learning

Recommendations

Rule-based data quality
CIKM '02: Proceedings of the eleventh international conference on Information and knowledge management

In the business intelligence/data warehouse user community, there is a growing confusion as to the difference between data cleansing and data quality. While many data cleansing products can help in applying data edits to name and address data, or help ...
Discovering dynamic integrity rules with a rules-based tool for data quality analyzing
CompSysTech '10: Proceedings of the 11th International Conference on Computer Systems and Technologies and Workshop for PhD Students in Computing on International Conference on Computer Systems and Technologies

Rules based approaches for data quality solutions often use business rules or integrity rules for data monitoring purpose. Integrity rules are constraints on data derived from business rules into a formal form in order to allow computerization. One of ...
Modeling decisional knowledge with the help of data quality information
CDVE'11: Proceedings of the 8th international conference on Cooperative design, visualization, and engineering

The success of the deployment of decision support systems heavily relies on the design of knowledge bases. In particular, assessing the quality of instanced data helps ensure an appropriate use of the knowledge. We present a collaborative editor for ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '03: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining

August 2003

736 pages

ISBN:1581137370

DOI:10.1145/956750

Conference Chair:
Lise Getoor
University of Maryland, College Park
,
General Chair:
Ted Senator
DARPA
,
Program Chairs:
Pedro Domingos
University of Washington
,
Christos Faloutsos
Carnegie Mellon University

Copyright © 2003 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 August 2003

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

KDD03

Sponsor:

KDD03: The Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 24 - 27, 2003

Washington, D.C.

Acceptance Rates

KDD '03 Paper Acceptance Rate 46 of 298 submissions, 15%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
1,427
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Caballero IGualo FRodríguez MPiattini M(2022)BR4DQInformation Systems10.1016/j.is.2022.102058109:COnline publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1016/j.is.2022.102058
Ngueilbaye AWang HMahamat DElgendy IJunaidu S(2021)Methods for detecting and correcting contextual data quality problemsIntelligent Data Analysis10.3233/IDA-20528225:4(763-787)Online publication date: 9-Jul-2021
https://doi.org/10.3233/IDA-205282
Arias RPerez-Yauli FGudino F(2019)A Methodological Aproach to the Determination of the Future Electrical Spatial Demand in Electric Distribution Companies in Ecuador2019 IEEE PES Innovative Smart Grid Technologies Conference - Latin America (ISGT Latin America)10.1109/ISGT-LA.2019.8895274(1-6)Online publication date: Sep-2019
https://doi.org/10.1109/ISGT-LA.2019.8895274
Looten VKong Win Chang LNeuraz ALandau-Loriot MVedie BPaul JMauge LRivet NBonifati AChatellier GBurgun ARance B(2019)What can millions of laboratory test results tell us about the temporal aspect of data quality? Study of data spanning 17 years in a clinical data warehouseComputer Methods and Programs in Biomedicine10.1016/j.cmpb.2018.12.030181:COnline publication date: 1-Nov-2019
https://dl.acm.org/doi/10.1016/j.cmpb.2018.12.030
Almeida RMaio POliveira PBarroso JDesai BAlameida ABernardino J(2016)Ontology Based Rewriting Data Cleaning OperationsProceedings of the Ninth International C* Conference on Computer Science & Software Engineering10.1145/2948992.2949007(85-88)Online publication date: 20-Jul-2016
https://dl.acm.org/doi/10.1145/2948992.2949007
Almeida RMaio POliveira PJoão B(2015)Towards Reusing Data Cleaning KnowledgeNew Contributions in Information Systems and Technologies10.1007/978-3-319-16486-1_14(143-150)Online publication date: 2015
https://doi.org/10.1007/978-3-319-16486-1_14
Al-Mudimigh AUllah Z(2011)Prevention of Dirty Data and the Role of MADAR ProjectProceedings of the 2011 UKSim 5th European Symposium on Computer Modeling and Simulation10.1109/EMS.2011.85(412-414)Online publication date: 16-Nov-2011
https://dl.acm.org/doi/10.1109/EMS.2011.85
Farinha JTrigueiros MBelo OMarcos EPapazoglou MPiattini M(2009)Using inheritance in a metadata based approach to data quality assessmentProceedings of the first international workshop on Model driven service engineering and data quality and security10.1145/1651415.1651417(1-8)Online publication date: 6-Nov-2009
https://dl.acm.org/doi/10.1145/1651415.1651417
Wakchaure AEaglin RMotlagh B(2008)A technique for the quantitative measure of data cleanliness2008 IEEE Conference on Cybernetics and Intelligent Systems10.1109/ICCIS.2008.4670930(1258-1263)Online publication date: Sep-2008
https://doi.org/10.1109/ICCIS.2008.4670930
Farinha JTrigueiros M(2007)An extensible metadata framework for data quality assessment of composite structuresProceedings of the 9th international conference on Data Warehousing and Knowledge Discovery10.5555/2391952.2391958(34-44)Online publication date: 3-Sep-2007
https://dl.acm.org/doi/10.5555/2391952.2391958
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents