research-article

Just can't get enough: Synthesizing Big Data

Authors:

Manuel Danisch,

Sebastian Schindler,

Hans-Arno JacobsenAuthors Info & Claims

SIGMOD '15: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data

Pages 1457 - 1462

https://doi.org/10.1145/2723372.2735378

Published: 27 May 2015 Publication History

Abstract

With the rapidly decreasing prices for storage and storage systems ever larger data sets become economical. While only few years ago only successful transactions would be recorded in sales systems, today every user interaction will be stored for ever deeper analysis and richer user modeling. This has led to the development of big data systems, which offer high scalability and novel forms of analysis. Due to the rapid development and ever increasing variety of the big data landscape, there is a pressing need for tools for testing and benchmarking.

Vendors have little options to showcase the performance of their systems but to use trivial data sets like TeraSort or WordCount. Since customers' real data is typically subject to privacy regulations and rarely can be utilized, simplistic proof-of-concepts have to be used, leaving both, customers and vendors, unclear of the target use-case performance. As a solution, we present an automatic approach to data synthetization from existing data sources. Our system enables a fully automatic generation of large amounts of complex, realistic, synthetic data.

References

[1]

A. Alexandrov, K. Tzoumas, and V. Markl. Myriad: Scalable and Expressive Data Generation. In VLDB, 2012.

Digital Library

[2]

A. Arasu, R. Kaushik, and J. Li. Data Generation Using Declarative Constraints. In SIGMOD, 2011.

Digital Library

[3]

C. Binnig, D. Kossmann, E. Lo, and M. T. Özsu. QAGen: Generating Query-aware Test Databases. In SIGMOD, 2007.

Digital Library

[4]

N. Bruno and S. Chaudhuri. Flexible Database Generators. In VLDB, pages 1097--1107, 2005.

Digital Library

[5]

B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears. Benchmarking Cloud Serving Systems with YCSB. In SoCC, pages 143--154, 2010.

Digital Library

[6]

M. Frank, M. Poess, and T. Rabl. Efficient Update Data Generation for DBMS Benchmark. In ICPE, 2012.

Digital Library

[7]

A. Ghazal, T. Rabl, M. Hu, F. Raab, M. Poess, A. Crolotte, and H.-A. Jacobsen. BigBench: Towards an industry standard benchmark for big data analytics. In SIGMOD, 2013.

Digital Library

[8]

J. Gray, P. Sundaresan, S. Englert, K. Baclawski, and P. J. Weinberger. Quickly Generating Billion-Record Synthetic Databases. In SIGMOD, pages 243--252, 1994.

Digital Library

[9]

J. E. Hoag and C. W. Thompson. A Parallel General-Purpose Synthetic Data Generator. SIGMOD Record, 36(1):19--24, 2007.

Digital Library

[10]

K. Houkjær, K. Torp, and R. Wind. Simple and Realistic Data Generation. In VLDB, pages 1243--1246, 2006.

Digital Library

[11]

P. J. Lin, B. Samadi, A. Cipolone, D. R. Jeske, S. Cox, C. Rendón, D. Holt, and R. Xiao. Development of a Synthetic Data Set Generator for Building and Testing Information Discovery Systems. In ITNG, pages 707--712, Washington, DC, USA, 2006. IEEE Computer Society.

Digital Library

[12]

E. Lo, N. Cheng, and W.-K. Hon. Generating Databases for Query Workloads. PVLDB, 3(1--2):848--859, 2010.

Digital Library

[13]

Z. Ming, C. Luo, W. Gao, R. Han, Q. Yang, L. Wang, and J. Zhan. BDGS: A Scalable Big Data Generator Suite in Big Data Benchmarking. In WBDB, 2013.

[14]

M. Poess and C. Floyd. New TPC Benchmarks for Decision Support and Web Commerce. SIGMOD Record, 29(4):64--71, 2000.

Digital Library

[15]

M. Poess, T. Rabl, M. Frank, and M. Danisch. A PDGF Implementation for TPC-H. In TPCTC, 2011.

Digital Library

[16]

M. Poess, T. Rabl, H.-A. Jacobsen, and B. Caufield. TPC-DI: The First Industry Benchmark for Data Integration. PVLDB, 13(7):1367--1378, 2014.

Digital Library

[17]

T. Rabl, M. Frank, H. M. Sergieh, and H. Kosch. A Data Generator for Cloud-Scale Benchmarking. In TPCTC, pages 41--56, 2010.

Digital Library

[18]

T. Rabl, M. Poess, M. Danisch, and H.-A. Jacobsen. Rapid Development of Data Generators Using Meta Generators in PDGF. In DBTest, 2013.

Digital Library

[19]

T. Rabl, M. Poess, H.-A. Jacobsen, P. E. O'Neil, and E. O'Neil. Variations of the Star Schema Benchmark to Test Data Skew in Database Management Systems. In ICPE, 2013.

Digital Library

[20]

E. Shen and L. Antova. Reversing Statistics for Scalable Test Databases Generation. In DBTest, 2013.

Digital Library

[21]

V. Sikka. Does the World Need a New Benchmark? http://www.saphana.com/community/blogs/blog/2013/09/16/does-the-world-need-a-new-benchmark, 2013.

[22]

J. M. Stephens and M. Poess. MUDD: a multi-dimensional data generator. In WOSP, pages 104--109, 2004.

Digital Library

[23]

Y. Tay, B. T. Dai, D. T. Wang, E. Y. Sun, Y. Lin, and Y. Lin. UpSizeR: Synthetically Scaling an Empirical Relational Database. Information Systems, 38(8):1168--1183, 2013.

Digital Library

[24]

E. Torlak. Scalable Test Data Generation from Multidimensional Models. In FSE, 2012.

Digital Library

Cited By

Hnatkowska BKimmel M(2023)Data Generation Based on Domain OntologyProceedings of the 31st International Conference on Information Systems Development10.62036/ISD.2023.16Online publication date: 2023
https://doi.org/10.62036/ISD.2023.16
Sanghi AAhmed SHaritsa J(2022)Projection-compliant database generationProceedings of the VLDB Endowment10.14778/3510397.351039815:5(998-1010)Online publication date: 18-May-2022
https://dl.acm.org/doi/10.14778/3510397.3510398
Gilad APatwa SMachanavajjhala ALi GLi ZIdreos SSrivastava D(2021)Synthesizing Linked Data Under Cardinality and Integrity ConstraintsProceedings of the 2021 International Conference on Management of Data10.1145/3448016.3457242(619-631)Online publication date: 9-Jun-2021
https://dl.acm.org/doi/10.1145/3448016.3457242
Show More Cited By

Index Terms

Just can't get enough: Synthesizing Big Data
1. Information systems
  1. Data management systems
    1. Database administration
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Data Just Right: Introduction to Large-Scale Data & Analytics
Can we analyze big data inside a DBMS?
DOLAP '13: Proceedings of the sixteenth international workshop on Data warehousing and OLAP

Relational DBMSs remain the main data management technology, despite the big data analytics and no-SQL waves. On the other hand, for data analytics in a broad sense, there are plenty of non-DBMS tools including statistical languages, matrix packages, ...
When Good-Enough is Enough: Complex Queries at Fixed Cost
BIGDATASERVICE '15: Proceedings of the 2015 IEEE First International Conference on Big Data Computing Service and Applications

Collections of time-series data appear in a wide variety of contexts. To gain insight into the underlying phenomenon (that the data represents), one must analyze the time-series data. Analysis can quickly become challenging for very large data (~...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '15: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data

May 2015

2110 pages

ISBN:9781450327589

DOI:10.1145/2723372

General Chair:
Timos Sellis
RMIT University, Australia
,
Program Chairs:
Susan B. Davidson
University of Pennsylvania, USA
,
Zack Ives
University of Pennsylvania, USA

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGMOD/PODS'15

Sponsor:

SIGMOD

SIGMOD/PODS'15: International Conference on Management of Data

May 31 - June 4, 2015

Victoria, Melbourne, Australia

Acceptance Rates

SIGMOD '15 Paper Acceptance Rate 106 of 415 submissions, 26%;

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
496
Total Downloads

Downloads (Last 12 months)32
Downloads (Last 6 weeks)3

Reflects downloads up to 23 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hnatkowska BKimmel M(2023)Data Generation Based on Domain OntologyProceedings of the 31st International Conference on Information Systems Development10.62036/ISD.2023.16Online publication date: 2023
https://doi.org/10.62036/ISD.2023.16
Sanghi AAhmed SHaritsa J(2022)Projection-compliant database generationProceedings of the VLDB Endowment10.14778/3510397.351039815:5(998-1010)Online publication date: 18-May-2022
https://dl.acm.org/doi/10.14778/3510397.3510398
Gilad APatwa SMachanavajjhala ALi GLi ZIdreos SSrivastava D(2021)Synthesizing Linked Data Under Cardinality and Integrity ConstraintsProceedings of the 2021 International Conference on Management of Data10.1145/3448016.3457242(619-631)Online publication date: 9-Jun-2021
https://dl.acm.org/doi/10.1145/3448016.3457242
Cheng K(2020)A Regular Expression-based DGL for Meaningful Synthetic Data Generation2020 IEEE International Conference on Big Data and Smart Computing (BigComp)10.1109/BigComp48618.2020.00-42(396-401)Online publication date: Feb-2020
https://doi.org/10.1109/BigComp48618.2020.00-42
Ahmadi SNassiri MRezvani M(2020)XACBench: a XACML policy benchmarkSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-020-04925-524:21(16081-16096)Online publication date: 1-Nov-2020
https://dl.acm.org/doi/10.1007/s00500-020-04925-5
Lanti DXiao GCalvanese DNgonga Ngomo AFundulaki IKrithara A(2019)VIGSemantic Web10.3233/SW-18033610:2(413-433)Online publication date: 1-Jan-2019
https://dl.acm.org/doi/10.3233/SW-180336
Mannino MAbouzied AGuimbretière FBernstein MReinecke K(2019)Is this Real?Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology10.1145/3332165.3347866(549-561)Online publication date: 17-Oct-2019
https://dl.acm.org/doi/10.1145/3332165.3347866
Sanghi ASood RSingh DHaritsa JTirthapura S(2018)HYDRAProceedings of the VLDB Endowment10.14778/3229863.323623811:12(1974-1977)Online publication date: 1-Aug-2018
https://dl.acm.org/doi/10.14778/3229863.3236238
Mărăşoiu MBlackwell ASarkar ASpott MBertini EElmqvist NWischgoll T(2016)Clarifying hypotheses by sketching dataProceedings of the Eurographics / IEEE VGTC Conference on Visualization: Short Papers10.5555/3058878.3058905(125-129)Online publication date: 6-Jun-2016
https://dl.acm.org/doi/10.5555/3058878.3058905
Zhang JTay Y(2016)DscalerProceedings of the VLDB Endowment10.14778/3007328.30073339:14(1671-1682)Online publication date: 1-Oct-2016
https://dl.acm.org/doi/10.14778/3007328.3007333
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents