research-article

Dremel: interactive analysis of web-scale datasets

Editors: Elisa Bertino, Paolo Atzeni, Kian Lee Tan, Yi Chen, Y. C. Tay Authors:

Andrey Gubarev,

Jing Jing Long,

Geoffrey Romer,

Shiva Shivakumar,

Theo VassilakisAuthors Info & Claims

Proceedings of the VLDB Endowment, Volume 3, Issue 1-2

Pages 330 - 339

https://doi.org/10.14778/1920841.1920886

Published: 01 September 2010 Publication History

Abstract

Dremel is a scalable, interactive ad-hoc query system for analysis of read-only nested data. By combining multi-level execution trees and columnar data layout, it is capable of running aggregation queries over trillion-row tables in seconds. The system scales to thousands of CPUs and petabytes of data, and has thousands of users at Google. In this paper, we describe the architecture and implementation of Dremel, and explain how it complements MapReduce-based computing. We present a novel columnar storage representation for nested records and discuss experiments on few-thousand node instances of the system.

References

[1]

D. J. Abadi, P. A. Boncz, and S. Harizopoulos. Column-Oriented Database Systems. VLDB, 2(2), 2009.

Digital Library

[2]

S. Abiteboul, R. Hull, and V. Vianu. Foundations of Databases. Addison Wesley, 1995.

Digital Library

[3]

A. Abouzeid, K. Bajda-Pawlikowski, D. J. Abadi, A. Rasin, and A. Silberschatz. HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads. VLDB, 2(1), 2009.

Digital Library

[4]

Z. Bar-Yossef, T. S. Jayram, R. Kumar, D. Sivakumar, and L. Trevisan. Counting Distinct Elements in a Data Stream. In RANDOM, pages 1--10, 2002.

Digital Library

[5]

L. A. Barroso and U. Hölzle. The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines. Morgan & Claypool Publishers, 2009.

Digital Library

[6]

R. Chaiken, B. Jenkins, P.-A. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou. SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets. VLDB, 1(2), 2008.

Digital Library

[7]

C. Chambers, A. Raniwala, F. Perry, S. Adams, R. Henry, R. Bradshaw, and N. Weizenbaum. FlumeJava: Easy, Efficient Data-Parallel Pipelines. In PLDI, 2010.

Digital Library

[8]

F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. Gruber. Bigtable: A Distributed Storage System for Structured Data. In OSDI, 2006.

Digital Library

[9]

L. S. Colby. A Recursive Algebra and Query Optimization for Nested Relations. SIGMOD Rec., 18(2), 1989.

Digital Library

[10]

G. Czajkowski. Sorting 1PB with MapReduce. Official Google Blog, Nov. 2008. At http://googleblog.blogspot.com/2008/11/sorting-1pb-with-mapreduce.html.

[11]

J. Dean. Challenges in Building Large-Scale Information Retrieval Systems: Invited Talk. In WSDM, 2009.

Digital Library

[12]

J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. In OSDI, 2004.

Digital Library

[13]

J. Dean and S. Ghemawat. MapReduce: a Flexible Data Processing Tool. Commun. ACM, 53(1), 2010.

Digital Library

[14]

S. Ghemawat, H. Gobioff, and S.-T. Leung. The Google File System. In SOSP, 2003.

Digital Library

[15]

Hadoop Apache Project. http://hadoop.apache.org.

[16]

Hive. http://wiki.apache.org/hadoop/Hive, 2009.

[17]

H. Liefke and D. Suciu. XMill: An Efficient Compressor for XML Data. In SIGMOD, 2000.

Digital Library

[18]

C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: a Not-so-Foreign Language for Data Processing. In SIGMOD, 2008.

Digital Library

[19]

P. E. O'Neil, E. J. O'Neil, S. Pal, I. Cseri, G. Schaller, and N. Westbury. ORDPATHs: Insert-Friendly XML Node Labels. In SIGMOD, 2004.

Digital Library

[20]

R. Pike, S. Dorward, R. Griesemer, and S. Quinlan. Interpreting the Data: Parallel Analysis with Sawzall. Scientific Programming, 13(4), 2005.

Digital Library

[21]

Protocol Buffers: Developer Guide. Available at http://code.google.com/apis/protocolbuffers/docs/overview.html.

[22]

M. Stonebraker, D. Abadi, D. J. DeWitt, S. Madden, E. Paulson, A. Pavlo, and A. Rasin. MapReduce and Parallel DBMSs: Friends or Foes? Commun. ACM, 53(1), 2010.

Digital Library

[23]

Y. Yu, M. Isard, D. Fetterly, M. Budiu, Ú. Erlingsson, P. K. Gunda, and J. Currey. DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language. In OSDI, 2008.

Digital Library

Cited By

Song XZheng JQian HZhao SZhang HPan XChen G(2025)In Search of a Memory-Efficient Framework for Online Cardinality EstimationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.348657137:1(392-407)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TKDE.2024.3486571
Zhao XQiao JHuang XWang CSong SWang J(2024)Apache TsFile: An IoT-Native Time Series File FormatProceedings of the VLDB Endowment10.14778/3685800.368582717:12(4064-4076)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.14778/3685800.3685827
Hansert PMichel S(2024)Partition, Don't Sort! Compression Boosters for Cloud Data Ingestion PipelinesProceedings of the VLDB Endowment10.14778/3681954.368201317:11(3456-3469)Online publication date: 30-Aug-2024
https://dl.acm.org/doi/10.14778/3681954.3682013
Show More Cited By

Recommendations

Dremel: a decade of interactive SQL analysis at web scale

Google's Dremel was one of the first systems that combined a set of architectural principles that have become a common practice in today's cloud-native analytics tools, including disaggregated storage and compute, in situ analysis, and columnar storage ...
Dremel: interactive analysis of web-scale datasets

Dremel is a scalable, interactive ad hoc query system for analysis of read-only nested data. By combining multilevel execution trees and columnar data layout, it is capable of running aggregation queries over trillion-row tables in seconds. The system ...
Dremel: Adaptive Configuration Tuning of RocksDB KV-Store
SIGMETRICS '22

LSM-tree-based key-value stores like RocksDB are widely used to support many applications. However, configuring a RocksDB instance is challenging for the following reasons: 1) RocksDB has a massive parameter space to configure; 2) there are inherent ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the VLDB Endowment

Proceedings of the VLDB Endowment Volume 3, Issue 1-2

September 2010

1658 pages

ISSN:2150-8097

Issue’s Table of Contents

Publisher

VLDB Endowment

Publication History

Published: 01 September 2010

Published in PVLDB Volume 3, Issue 1-2

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

491
Total Citations
View Citations
3,212
Total Downloads

Downloads (Last 12 months)296
Downloads (Last 6 weeks)24

Reflects downloads up to 28 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Song XZheng JQian HZhao SZhang HPan XChen G(2025)In Search of a Memory-Efficient Framework for Online Cardinality EstimationIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.348657137:1(392-407)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TKDE.2024.3486571
Zhao XQiao JHuang XWang CSong SWang J(2024)Apache TsFile: An IoT-Native Time Series File FormatProceedings of the VLDB Endowment10.14778/3685800.368582717:12(4064-4076)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.14778/3685800.3685827
Hansert PMichel S(2024)Partition, Don't Sort! Compression Boosters for Cloud Data Ingestion PipelinesProceedings of the VLDB Endowment10.14778/3681954.368201317:11(3456-3469)Online publication date: 30-Aug-2024
https://dl.acm.org/doi/10.14778/3681954.3682013
Wang H(2024)Enhancing Accuracy for Super Spreader Identification in High-Speed Data StreamsProceedings of the VLDB Endowment10.14778/3681954.368198817:11(3124-3137)Online publication date: 30-Aug-2024
https://dl.acm.org/doi/10.14778/3681954.3681988
Bergui MHourri SNajah SNikolov N(2024)Predictive modelling of MapReduce job performance in cloud environments using machine learning techniquesJournal of Big Data10.1186/s40537-024-00964-z11:1Online publication date: 23-Jul-2024
https://doi.org/10.1186/s40537-024-00964-z
Stonebraker MPavlo A(2024)What Goes Around Comes Around... And Around...ACM SIGMOD Record10.1145/3685980.368598453:2(21-37)Online publication date: 31-Jul-2024
https://dl.acm.org/doi/10.1145/3685980.3685984
Hu XTao Y(2024)Parallel Acyclic Joins: Optimal Algorithms and Cyclicity SeparationJournal of the ACM10.1145/363351271:1(1-44)Online publication date: 11-Feb-2024
https://dl.acm.org/doi/10.1145/3633512
Ramalhete PCorreia ALee IChabbi MSteuwer M(2024)Scaling Up Transactions with Slower ClocksProceedings of the 29th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming10.1145/3627535.3638472(2-16)Online publication date: 2-Mar-2024
https://dl.acm.org/doi/10.1145/3627535.3638472
Levandoski JCasto GDeng MDesai REdara PHottelier THormati AJohnson AJohnson JKurzyniec DMcVeety SRamanathan PSaxena GShanmugan VVolobuev YBarcelo PSanchez-Pi NMeliou ASudarshan S(2024)BigLake: BigQuery's Evolution toward a Multi-Cloud LakehouseCompanion of the 2024 International Conference on Management of Data10.1145/3626246.3653388(334-346)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3626246.3653388
Lamb AShen YHeres DChakraborty JKabak MHsieh LSun CBarcelo PSanchez-Pi NMeliou ASudarshan S(2024)Apache Arrow DataFusion: A Fast, Embeddable, Modular Analytic Query EngineCompanion of the 2024 International Conference on Management of Data10.1145/3626246.3653368(5-17)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3626246.3653368
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents