Article

Automatic segmentation of text into structured records

Authors:

Vinayak Borkar,

Kaustubh Deshmukh,

Sunita SarawagiAuthors Info & Claims

SIGMOD '01: Proceedings of the 2001 ACM SIGMOD international conference on Management of data

Pages 175 - 186

https://doi.org/10.1145/375663.375682

Published: 01 May 2001 Publication History

Abstract

In this paper we present a method for automatically segmenting unformatted text records into structured elements. Several useful data sources today are human-generated as continuous text whereas convenient usage requires the data to be organized as structured records. A prime motivation is the warehouse address cleaning problem of transforming dirty addresses stored in large corporate databases as a single text field into subfields like “City” and “Street”. Existing tools rely on hand-tuned, domain-specific rule-based systems.

We describe a tool DATAMOLD that learns to automatically extract structure when seeded with a small number of training examples. The tool enhances on Hidden Markov Models (HMM) to build a powerful probabilistic model that corroborates multiple sources of information including, the sequence of elements, their length distribution, distinguishing words from the vocabulary and an optional external data dictionary. Experiments on real-life datasets yielded accuracy of 90% on Asian addresses and 99% on US addresses. In contrast, existing information extraction methods based on rule-learning techniques yielded considerably lower accuracy.

References

[1]

B. Aldelberg. Nodose: A tool for semi-automatically extracting structured and semistructured data from text documents. In SIGMOD, 1998.

Digital Library

[2]

Arnaud Sahuguet and Fabien Azavant. Building light-weight wrappers for legacy Web data-sources using W4F. In International Conference on Very Large Databases (VLDB), 1999.

Digital Library

[3]

G. Barish, Y.-S. Chen, D. DiPasquo, C. A. Knoblock, S. Minton, I. Muslea, and C. Shahabi. Theaterloc: Using information integration technology to rapidly build virtual applications. In Intl. Conf. on Data Engineering ICDE, pages 681-682, 2000.

Digital Library

[4]

D. M. Bikel, S. Miller, R. Schwartz, and R. Weischedel. Nymble: a high-performance learning name-under. In Proceedings of ANLP-97, pages 194-201, 1997.

Digital Library

[5]

M. E. Califf and R. J. Mooney. Relational learning of pattern-match rules for information extraction. In Proceedings of the Sixteenth National Conference on Artificial Intelligence (AAAI-99), pages 328-334, July 1999.

Digital Library

[6]

A. Crespo, J. Jannink, E. Neuhold, M. Rys, and R. Studer. A survey of semi-automatic extraction and transformation. http://www-db.stanford.edu/ crespo/publications/.

[7]

D. W. Embley, Y. S. Jiang, and Y.-K. Ng. Record-boundary discovery in web documents. In SIGMOD 1999, Proceedings ACM SIGMOD International Conference on Management of Data, June 1-3, 1999, Philadephia, Pennsylvania, USA, pages 467-478, 1999.

Digital Library

[8]

D. Freitag and A. McCallum. Information extraction using HMMs and shrinkage. In Papers from the AAAI-99 Workshop on Machine Learning for Information Extraction, pages 31-36, 1999.

[9]

D. Freitag and A. McCallum. Information extraction with HMM structures learned by stochastic optimization. In Proceedings of the AAAI 2000, 2000.

Digital Library

[10]

H. Galhardas. http://caravel.inria.fr/ galharda/cleaning.html.

[11]

J. Hammer, H. Garcia-Molina, J. Cho, R. Aranha, and A. Crespo. Extracting semistructure information from the web. In Workshop on mangement of semistructured data, 1997.

[12]

M. A. Hernandez and S. J. Stolfo. The merge/purge problem for large databases. In Proceedings of the ACM SIGMOD, 1995.

Digital Library

[13]

C.-N. Hsu and M.-T. Dung. Generating finite-state transducers for semistructured data extraction from the web. Information Systems Special Issue on Semistructured Data, 23(8), 1998.

Digital Library

[14]

S. Huffman. Learning information extraction patterns from examples. In S. Wermter, G. Scheler, and E. Riloff, editors, Proceedings of the 1995 IJCAI Workshop on New Approaches to Learning for Natural Language Processing., 1995.

[15]

R. Kimball. Dealing with dirty data. Intelligent Enterprise, September 1996. http://www.intelligententerprise.com/.

Digital Library

[16]

J. Kupiec. Robust part of speech tagging using a hidden Markov model. Computer Speech and Language, 6:225-242, 1992.

[17]

N. Kushmerick, D. Weld, and R. Doorenbos. Wrapper induction for information extraction. In Proceedings of IJCAI, 1997.

[18]

P.-S. Laplace. Philosophical Essays on Probabilities. Springer-Verlag, New York, 1995. Translated by A. I. Dale from the 5th French edition of 1825.

[19]

S. Lawrence, C. L. Giles, and K. Bollacker. Digital libraries and autonomous citation indexing. IEEE Computer, 32(6):67-71, 1999.

Digital Library

[20]

L. Liu, C. Pu, and W. Han. Xwrap: An xml-enabled wrapper construction system for web information sources. In International Conference on Data Engineering (ICDE), pages 611-621, 2000.

Digital Library

[21]

A. McCallum, D. Freitag, and F. Pereira". Maximum entropy markov models for information extraction and segmentation. In In proceedings of ICML-2000, 2000.

Digital Library

[22]

G. Mecca, P. Merialdo, and P. Atzeni. Araneus in the era of xml. In IEEE Data Engineering Bullettin, Special Issue on XML. IEEE, September 1999.

[23]

A. E. Monge and C. P. Elkan. The field matching problem: Algorithms and applications. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (KDD-96), 1996.

[24]

I. Muslea. Extraction patterns for information extraction tasks: A survey. In The AAAI-99 Workshop on Machine Learning for Information Extraction, 1999.

[25]

I. Muslea, S. Minton, and C. A. Knoblock. A hierarchical approach to wrapper induction. In Proceedings of the Third International Conference on Autonomous Agents, Seattle, WA, 1999.

Digital Library

[26]

L. Rabiner. A tutorial on Hidden Markov Models and selected applications in speech recognition. In Proceedings of the IEEE, 77(2), 1989.

[27]

L. Rabiner and B.-H. Juang. Fundamentals of Speech Recognition, chapter 6. Prentice-Hall, 1993.

Digital Library

[28]

K. Seymore, A. McCallum, and R. Rosenfeld. Learning Hidden Markov Model structure for information extraction. In Papers from the AAAI-99 Workshop on Machine Learning for Information Extraction, pages 37-42, 1999.

[29]

S. Soderland. Learning information extraction rules for semi-structured and free text. Machine Learning, 34, 1999.

Digital Library

Cited By

Mansurova MBarakhnin VKyrgyzbayeva MKadyrbek N(2021)Named Entity Extraction Model Based on the Random Walk Method2021 IEEE International Conference on Smart Information Systems and Technologies (SIST)10.1109/SIST50301.2021.9465992(1-6)Online publication date: 28-Apr-2021
https://doi.org/10.1109/SIST50301.2021.9465992
Mansurova MBarakhnin VKhibatkhanuly YPastushkov I(2019)Named Entity Extraction from Semi-structured Data Using Machine Learning AlgorithmsComputational Collective Intelligence10.1007/978-3-030-28374-2_6(58-69)Online publication date: 9-Aug-2019
https://doi.org/10.1007/978-3-030-28374-2_6
Ganti V(2018)Data CleaningEncyclopedia of Database Systems10.1007/978-1-4614-8265-9_592(737-741)Online publication date: 7-Dec-2018
https://doi.org/10.1007/978-1-4614-8265-9_592
Show More Cited By

Index Terms

Automatic segmentation of text into structured records

Recommendations

Automatic segmentation of text into structured records

In this paper we present a method for automatically segmenting unformatted text records into structured elements. Several useful data sources today are human-generated as continuous text whereas convenient usage requires the data to be organized as ...
Registering and searching for records in electronic records management systems

Records are increasingly becoming electronic. Both public and private organizations are more and more making information and records only available to their employees, customers and constituents on such a format. Electronic Records Management Systems (...
Linking records in dynamic world
PhD '12: Proceedings of the on SIGMOD/PODS 2012 PhD Symposium

In real-world, entities change dynamically and the changes are capture in two dimensions: time and space. For data sets that contain temporal records, where each record is associated with a time stamp and describes some aspects of a real-world entity at ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '01: Proceedings of the 2001 ACM SIGMOD international conference on Management of data

May 2001

630 pages

ISBN:1581133324

DOI:10.1145/375663

ACM SIGMOD Record Volume 30, Issue 2
June 2001
625 pages
ISSN:0163-5808
DOI:10.1145/376284
Editors:
Timos Sellis
National Technical Univ. of Athens
,
Sharad Mehrotra
Univ. of California at Irvine
Issue’s Table of Contents

Copyright © 2001 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 May 2001

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

SIGMOD/PODS01

Sponsor:

SIGMOD

SIGMOD/PODS01: ACM SIGMOD International Conference on Management of Data

May 21 - 24, 2001

California, Santa Barbara, USA

Acceptance Rates

SIGMOD '01 Paper Acceptance Rate 44 of 293 submissions, 15%;

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

156
Total Citations
View Citations
1,937
Total Downloads

Downloads (Last 12 months)30
Downloads (Last 6 weeks)4

Reflects downloads up to 31 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Mansurova MBarakhnin VKyrgyzbayeva MKadyrbek N(2021)Named Entity Extraction Model Based on the Random Walk Method2021 IEEE International Conference on Smart Information Systems and Technologies (SIST)10.1109/SIST50301.2021.9465992(1-6)Online publication date: 28-Apr-2021
https://doi.org/10.1109/SIST50301.2021.9465992
Mansurova MBarakhnin VKhibatkhanuly YPastushkov I(2019)Named Entity Extraction from Semi-structured Data Using Machine Learning AlgorithmsComputational Collective Intelligence10.1007/978-3-030-28374-2_6(58-69)Online publication date: 9-Aug-2019
https://doi.org/10.1007/978-3-030-28374-2_6
Ganti V(2018)Data CleaningEncyclopedia of Database Systems10.1007/978-1-4614-8265-9_592(737-741)Online publication date: 7-Dec-2018
https://doi.org/10.1007/978-1-4614-8265-9_592
Lu YOkada SNitta K(2017)Weibull partition models with applications to hidden semi-Markov models2017 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN.2017.7965850(162-169)Online publication date: May-2017
https://doi.org/10.1109/IJCNN.2017.7965850
Torra VTorra V(2017)Privacy Models and Disclosure Risk MeasuresData Privacy: Foundations, New Developments and the Big Data Challenge10.1007/978-3-319-57358-8_5(111-189)Online publication date: 18-May-2017
https://doi.org/10.1007/978-3-319-57358-8_5
Arora NKim JKim YGeller JChoi WLeung CPark Y(2016)Knock knockProceedings of the Sixth International Conference on Emerging Databases: Technologies, Applications, and Theory10.1145/3007818.3007828(86-89)Online publication date: 17-Oct-2016
https://dl.acm.org/doi/10.1145/3007818.3007828
Wang MHaberland VYeo AMartin AHowroyd JBishop J(2016)A Probabilistic Address Parser Using Conditional Random Fields and Stochastic Regular Grammar2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW)10.1109/ICDMW.2016.0039(225-232)Online publication date: Dec-2016
https://doi.org/10.1109/ICDMW.2016.0039
Ganti V(2016)Data CleaningEncyclopedia of Database Systems10.1007/978-1-4899-7993-3_592-2(1-4)Online publication date: 8-Dec-2016
https://doi.org/10.1007/978-1-4899-7993-3_592-2
Babatunde FOjokoh BOluwadare S(2015)Automatic Table Recognition and Extraction from Heterogeneous DocumentsJournal of Computer and Communications10.4236/jcc.2015.31200903:12(100-110)Online publication date: 2015
https://doi.org/10.4236/jcc.2015.312009
Chu XHe YChakrabarti KGanjam KSellis TDavidson SIves Z(2015)TEGRAProceedings of the 2015 ACM SIGMOD International Conference on Management of Data10.1145/2723372.2723725(1713-1728)Online publication date: 27-May-2015
https://dl.acm.org/doi/10.1145/2723372.2723725
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents