Article

Data extraction and label assignment for web databases

Authors:

Fred H. LochovskyAuthors Info & Claims

WWW '03: Proceedings of the 12th international conference on World Wide Web

Pages 187 - 196

https://doi.org/10.1145/775152.775179

Published: 20 May 2003 Publication History

Abstract

Many tools have been developed to help users query, extract and integrate data from web pages generated dynamically from databases, i.e., from the Hidden Web. A key prerequisite for such tools is to obtain the schema of the attributes of the retrieved data. In this paper, we describe a system called, DeLa, which reconstructs (part of) a "hidden" back-end web database. It does this by sending queries through HTML forms, automatically generating regular expression wrappers to extract data objects from the result pages and restoring the retrieved data into an annotated (labelled) table. The whole process needs no human involvement and proves to be fast (less than one minute for wrapper induction for each site) and accurate (over 90% correctness for data extraction and around 80% correctness for label assignment).

References

[1]

B. Adelberg. "NoDoSE - A tool for semi-automatically extracting structured and semistructured data from text documents," Proc. ACM SIGMOD Conf., 1998, 283--294.

Digital Library

[2]

R. Baumgartner, S. Flesca and G. Gottlob. "Visual web information extraction with Lixto," Proc.27th VLDB Conf., 2001, 119--128.

Digital Library

[3]

BrightPlanet Corp. "The Deep Web: Surfacing hidden value." http://www.completeplanet.com/Tutorials/DeepWeb/

[4]

D. Buttler, L. Liu and C. Pu. "A fully automated object extraction system for the World Wide Web," Proc. Intl. Conf. on Distributed Computing Systems, 2001, 361--370.

Digital Library

[5]

C.H. Chang and S.C. Lui. "IEPAD: information extraction based on pattern discovery," Proc. 10th Intl. Conf. on World Wide Web, 2001, 681--688.

Digital Library

[6]

S. Chawathe, H. Garcia-Molina, J. Hammer, K. Ireland, Y. Papakonstantinou, J. Ullman and J. Widom. "The TRIMMIS project: integration of heterogeneous information sources," Proc. IPSJ Conference, 1994, 7--18.

[7]

V. Crescenzi, G. Mecca and P. Merialdo. "ROADRUNNER: towards automatic data extraction from large web sites," Proc. 27th VLDB Conf., 2001, 109--118.

Digital Library

[8]

D. Embley, Y. Jiang and Y. K. Ng. "Record-boundary discovery in web documents," Proc. ACM SIGMOD Conf., 1999, 467--478.

Digital Library

[9]

D. Florescu, A. Y. Levy and A. O. Mendelzon. "Database techniques for the world-wide web: a survey," SIGMOD Record 27(3), 1998, 59--74.

Digital Library

[10]

D. Gusfield. Algorithms on Strings, Trees, and Sequences. Cambridge, 1997.

Digital Library

[11]

C. Hus and M. Dung. "Generating finite-state transducers for semi-structured data extraction from the web," Information Systems 23(8), 1998, 521--538.

Digital Library

[12]

T. Kirk, A. Levy, Y. Sagiv and D. Srivastava. "The Information Manifold," Proc. the AAAI Spring Symp. on Information Gathering from Heterogeneous, Distributed Environments, 1995, 85--91.

[13]

L. Liu, C. Pu and W. Han. "XWRAP: An XML-enabled wrapper construction system for web information sources," Proc. 16th Intl. Conf. on Data Engineering (ICDE), 2000, 611--621.

Digital Library

[14]

I. Muslea, S. Minton and C. Knoblock. "A hierarchical approach to wrapper induction," Proc. 3rd Intl. Conf. on Autonomous Agents, 1999, 190--197.

Digital Library

[15]

S. Raghavan and H. Garcia-Molina. "Crawling the hidden web," Proc. 27th VLDB Conf., 2001, 129--138.

Digital Library

[16]

S. Raghavan and H. Garcia-Molina. "Integrating diverse information management systems: a brief survey," IEEE Data Engineering Bulletin 24(4), 2001, 44--52.

[17]

B. Ribeiro-Neto, A. Laender and A.S. da Silva. "Extracting semi-structured data through examples," Proc. Intl. Conf. on Information and Knowledge Management, 1999, 94--101.

Digital Library

[18]

A. Sahuguet and F. Azavant. "WysiWyg web wrapper factory (W4F)," Proc. 8th World Wide Web, 1999.

[19]

J. Wang and F. Lochovsky. "Data-rich section extraction from HTML pages," Proc. 3rd Conf. on Web Information Systems Engineering, 2002, 313--322.

Digital Library

[20]

J. Wang and F. Lochovsky. "Wrapper Induction based on nested pattern discovery.", Technical Report HKUST-CS-27-02, Dept. of Computer Science, Hong Kong U. of Science and Technology, 2002 (submitted for publication). http://www.cs.ust.hk/~cswangjy/paper/tr-27-02.pdf

[21]

World Wide Web Consortium. Document Object Model Level 3 Core Specification, 2001.

[22]

World Wide Web Consortium. HTML 4.01 Specification, 1999.

Cited By

Harshavardhan TVasisht NPraneeth MDeepika NHariharan SKukreja V(2024)Web Application for Identifying Emerging Trends and Technologies using Scopus and OpenAI APIs2024 International Conference on Advances in Data Engineering and Intelligent Computing Systems (ADICS)10.1109/ADICS58448.2024.10533192(1-4)Online publication date: 18-Apr-2024
https://doi.org/10.1109/ADICS58448.2024.10533192
Malhotra DBhatia RKumar M(2023)Automated Selection of Web Form Text Field Values Based on Bayesian InferencesInternational Journal of Information Retrieval Research10.4018/IJIRR.31839913:1(1-13)Online publication date: 16-Feb-2023
https://dl.acm.org/doi/10.4018/IJIRR.318399
Acikalin UCaskurlu BSubramani K(2023)Security-Aware Database Migration PlanningConstraints10.1007/s10601-023-09351-628:3(472-505)Online publication date: 10-Aug-2023
https://doi.org/10.1007/s10601-023-09351-6
Show More Cited By

Index Terms

Data extraction and label assignment for web databases

Recommendations

A QIIIEP based domain specific hidden web crawler
ICWET '11: Proceedings of the International Conference & Workshop on Emerging Trends in Technology

For context based surfing of World Wide Web in a systematic and automatic manner, a web crawler is required. The World Wide Web consists interlinked documents and resources that are easily crawled by general web crawler, known as surface web crawler. ...
Web Data Extraction Based on Label Library
CSIE '09: Proceedings of the 2009 WRI World Congress on Computer Science and Information Engineering - Volume 05

A Web data Extraction technique based on label library is proposed for extracting information from data intensive Web pages in this paper. It eliminates conception ambiguity of the contents of Web pages with the label library, mines data regions by ...
Annotating Search Results from Web Databases

An increasing number of databases have become web accessible through HTML form-based search interfaces. The data units returned from the underlying database are usually encoded into the result pages dynamically for human browsing. For the encoded data ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '03: Proceedings of the 12th international conference on World Wide Web

May 2003

772 pages

ISBN:1581136803

DOI:10.1145/775152

Conference Chairs:
Gusztáv Hencsey
MTA SZTAKI, Hungary
,
Bebo White
Stanford Linear Accelerator Center, USA
,
Program Chairs:
Yih-Farn Robin Chen
AT&T Labs -- Research, USA
,
László Kovács
MTA SZTAKI, Hungary
,
Steve Lawrence
Google Inc., USA

Copyright © 2003 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 May 2003

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

157
Total Citations
View Citations
2,911
Total Downloads

Downloads (Last 12 months)28
Downloads (Last 6 weeks)2

Reflects downloads up to 02 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Harshavardhan TVasisht NPraneeth MDeepika NHariharan SKukreja V(2024)Web Application for Identifying Emerging Trends and Technologies using Scopus and OpenAI APIs2024 International Conference on Advances in Data Engineering and Intelligent Computing Systems (ADICS)10.1109/ADICS58448.2024.10533192(1-4)Online publication date: 18-Apr-2024
https://doi.org/10.1109/ADICS58448.2024.10533192
Malhotra DBhatia RKumar M(2023)Automated Selection of Web Form Text Field Values Based on Bayesian InferencesInternational Journal of Information Retrieval Research10.4018/IJIRR.31839913:1(1-13)Online publication date: 16-Feb-2023
https://dl.acm.org/doi/10.4018/IJIRR.318399
Acikalin UCaskurlu BSubramani K(2023)Security-Aware Database Migration PlanningConstraints10.1007/s10601-023-09351-628:3(472-505)Online publication date: 10-Aug-2023
https://doi.org/10.1007/s10601-023-09351-6
Gupta SBhatia K(2021)Design of a Parallel and Scalable Crawler for the Hidden WebInternational Journal of Information Retrieval Research10.4018/IJIRR.28961212:1(1-23)Online publication date: 15-Oct-2021
https://doi.org/10.4018/IJIRR.289612
Subramani KCaskurlu BAcikalin U(2020)Security-Aware Database Migration PlanningAlgorithmic Aspects of Cloud Computing10.1007/978-3-030-58628-7_7(103-121)Online publication date: 30-Aug-2020
https://doi.org/10.1007/978-3-030-58628-7_7
Rimal Y(2020)Machine Learning Techniques Best for Large Data PredictionOntology‐Based Information Retrieval for Healthcare Systems10.1002/9781119641391.ch11(245-256)Online publication date: 28-Jul-2020
https://doi.org/10.1002/9781119641391.ch11
Guo JCrescenzi VFurche TGrasso GGottlob G(2019)RED: Redundancy-Driven Data Extraction from Result Pages?The World Wide Web Conference10.1145/3308558.3313529(605-615)Online publication date: 13-May-2019
https://dl.acm.org/doi/10.1145/3308558.3313529
Ahmad Sabri IMan MAbu Bakar WMohd Rose A(2019)Web Data Extraction Approach for Deep Web using WEIDJProcedia Computer Science10.1016/j.procs.2019.12.124163(417-426)Online publication date: 2019
https://doi.org/10.1016/j.procs.2019.12.124
Umamageswari BKalpana R(2018)Web HarvestingThe Dark Web10.4018/978-1-5225-3163-0.ch010(199-226)Online publication date: 2018
https://doi.org/10.4018/978-1-5225-3163-0.ch010
Gupta SBhatia K(2018)Optimal Query Generation for Hidden Web Extraction Through Response AnalysisThe Dark Web10.4018/978-1-5225-3163-0.ch005(65-83)Online publication date: 2018
https://doi.org/10.4018/978-1-5225-3163-0.ch005
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten