research-article

HTAPBench: Hybrid Transactional and Analytical Processing Benchmark

Authors:

Ricardo Vilaça,

Rui OliveiraAuthors Info & Claims

ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

Pages 293 - 304

https://doi.org/10.1145/3030207.3030228

Published: 17 April 2017 Publication History

Abstract

The increasing demand for real-time analytics requires the fusion of Transactional (OLTP) and Analytical (OLAP) systems, eschewing ETL processes and introducing a plethora of proposals for the so-called Hybrid Analytical and Transactional Processing (HTAP) systems.

Unfortunately, current benchmarking approaches are not able to comprehensively produce a unified metric from the assessment of an HTAP system. The evaluation of both engine types is done separately, leading to the use of disjoint sets of benchmarks such as TPC-C or TPC-H.

In this paper we propose a new benchmark, HTAPBench, providing a unified metric for HTAP systems geared toward the execution of constantly increasing OLAP requests limited by an admissible impact on OLTP performance. To achieve this, a load balancer within HTAPBench regulates the coexistence of OLTP and OLAP workloads, proposing a method for the generation of both new data and requests, so that OLAP requests over freshly modified data are comparable across runs.

We demonstrate the merit of our approach by validating it with different types of systems: OLTP, OLAP and HTAP; showing that the benchmark is able to highlight the differences between them, while producing queries with comparable complexity across experiments with negligible variability.

References

[1]

A. Bog, J. Krüger, and J. Schaffner. A composite benchmark for online transaction processing and operational reporting. In Advanced Management of Information for Globalized Enterprises, 2008. AMIGE 2008. IEEE Symposium on, pages 1--5. IEEE, 2008.

[2]

A. Bog, H. Plattner, and A. Zeier. A mixed transaction processing and operational reporting benchmark. Information Systems Frontiers, 13(3):321--335, July 2011.

Digital Library

[3]

A. Bog, K. Sachs, and H. Plattner. Interactive performance monitoring of a composite oltp and olap workload. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD '12, pages 645--648, New York, NY, USA, 2012. ACM.

Digital Library

[4]

A. Bog, K. Sachs, and A. Zeier. Benchmarking database design for mixed oltp and olap workloads. In Proceedings of the 2Nd ACM/SPEC International Conference on Performance Engineering, ICPE '11, pages 417--418, New York, NY, USA, 2011. ACM.

Digital Library

[5]

R. Cole, F. Funke, L. Giakoumakis, W. Guy, A. Kemper, S. Krompass, H. Kuno, R. Nambiar, T. Neumann, M. Poess, et al. The mixed workload ch-benchmark. In Proceedings of the Fourth International Workshop on Testing Database Systems, page 8. ACM, 2011.

Digital Library

[6]

S. P. Council. Storage Performance Council. 2015.

[7]

S. P. E. Council. Standard Performance Evaluation Council. 2015.

[8]

T. P. P. Council. TPC Benchmark C. 2010.

[9]

T. P. P. Council. TPC Benchmark H. 2010.

[10]

T. P. P. Council. TPC Benchmark DS. 2012.

[11]

T. P. P. Council. TPC Benchmark E. 2015.

[12]

T. P. P. Council. The Transaction Processing Performance Council. 2015.

[13]

D. E. Difallah, A. Pavlo, C. Curino, and P. Cudré-Mauroux. Oltp-bench: An extensible testbed for benchmarking relational databases. PVLDB, 7(4):277--288, 2013.

Digital Library

[14]

C. D. French. "one size fits all" database architectures do not work for dss. In Proceedings of the 1995 ACM SIGMOD International Conference on Management of Data, SIGMOD '95, pages 449--450, New York, NY, USA, 1995. ACM.

Digital Library

[15]

F. Funke, A. Kemper, and T. Neumann. Benchmarking hybrid oltp&olap database systems. In BTW, pages 390--409, 2011.

[16]

A. K. Goel, J. Pound, N. Auch, P. Bumbulis, S. MacLean, F. Färber, F. Gropengiesser, C. Mathis, T. Bodner, and W. Lehner. Towards scalable real-time analytics: An architecture for scale-out of olxp workloads. Proc. VLDB Endow., 8(12):1716--1727, Aug. 2015.

Digital Library

[17]

T. Haerder and A. Reuter. Principles of transaction-oriented database recovery. ACM Comput. Surv., 15(4):287--317, Dec. 1983.

Digital Library

[18]

R. Kimball, M. Ross, et al. The data warehouse toolkit: the complete guide to dimensional modelling. US: John Wiley & Sons, 2002.

[19]

S. S. Lightstone, T. J. Teorey, and T. Nadeau. Physical Database Design: The Database Professional's Guide to Exploiting Indexes, Views, Storage, and More. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2007.

[20]

S. Machine. Splice machine. Technical report, Splice Macine, 2015.

[21]

N. Mukherjee, S. Chavan, M. Colgan, D. Das, M. Gleeson, S. Hase, A. Holloway, H. Jin, J. Kamp, K. Kulkarni, T. Lahiri, J. Loaiza, N. Macnaughton, V. Marwah, A. Mullick, A. Witkowski, J. Yan, and M. Zait. Distributed architecture of oracle database in-memory. Proc. VLDB Endow., 8(12):1630--1641, Aug. 2015.

Digital Library

[22]

M. Nakamura, T. Tabaru, Y. Ujibashi, T. Hashida, M. Kawaba, and L. Harada. Extending postgresql to handle olxp workloads. In Innovative Computing Technology (INTECH), 2015 Fifth International Conference on, pages 40--44. IEEE, 2015.

[23]

R. O. Nambiar and M. Poess. The making of tpc-ds. In Proceedings of the 32Nd International Conference on Very Large Data Bases, VLDB '06, pages 1049--1058. VLDB Endowment, 2006.

Digital Library

[24]

NuoDB. Hybrid transaction and analytical processing with nuodb. Technical report, NuoDB, 2014.

[25]

M. Poess and C. Floyd. New tpc benchmarks for decision support and web commerce. SIGMOD Rec., 29(4):64--71, Dec. 2000.

Digital Library

[26]

J. Rossberg and R. Redler. Pro Scalable .NET 2.0 Application Designs. Expert's Voice in .Net. Apress, 2005.

[27]

VoltDB. Voltdb - whitepaper. Technical report, VoltDB, 2014.

[28]

M. Waclawiczek. HTAP - What it Is and Why it Matters. 2015.

Cited By

Zhang CLi GLv T(2024)HyBench: A New Benchmark for HTAP DatabasesProceedings of the VLDB Endowment10.14778/3641204.364120617:5(939-951)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.14778/3641204.3641206
Zhang CLi GZhang JZhang XFeng J(2024)HTAP Databases: A SurveyIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.338969336:11(6410-6429)Online publication date: Nov-2024
https://doi.org/10.1109/TKDE.2024.3389693
Weng SWang QQu LZhang RCai PQian WZhou A(2024)Lauca: A Workload Duplicator for Benchmarking Transactional Database PerformanceIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.336011636:7(3180-3194)Online publication date: Jul-2024
https://doi.org/10.1109/TKDE.2024.3360116
Show More Cited By

Index Terms

HTAPBench: Hybrid Transactional and Analytical Processing Benchmark
1. Information systems
  1. Data management systems
    1. Database administration
      1. Database performance evaluation
    2. Database management system engines
      1. Parallel and distributed DBMSs
        Relational parallel and distributed DBMSs

Recommendations

Hybrid Transactional/Analytical Processing: A Survey
SIGMOD '17: Proceedings of the 2017 ACM International Conference on Management of Data

The popularity of large-scale real-time analytics applications (real-time inventory/pricing, recommendations from mobile apps, fraud detection, risk analysis, IoT, etc.) keeps rising. These applications require distributed data management systems that ...
Adaptive HTAP through Elastic Resource Scheduling
SIGMOD '20: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

Modern Hybrid Transactional/Analytical Processing (HTAP) systems use an integrated data processing engine that performs analytics on fresh data, which are ingested from a transactional engine. HTAP systems typically consider data freshness at design ...
BatchDB: Efficient Isolated Execution of Hybrid OLTP+OLAP Workloads for Interactive Applications
SIGMOD '17: Proceedings of the 2017 ACM International Conference on Management of Data

In this paper we present BatchDB, an in-memory database engine designed for hybrid OLTP and OLAP workloads. BatchDB achieves good performance, provides a high level of data freshness, and minimizes load interaction between the transactional and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

April 2017

450 pages

ISBN:9781450344043

DOI:10.1145/3030207

General Chairs:
Walter Binder
Università della Svizzera Italiana, Switzerland
,
Vittorio Cortellessa
Università dell'Aquila, Italy
,
Program Chairs:
Anne Koziolek
Karlsruhe Institute of Technology, Germany
,
Evgenia Smirni
College of William & Mary, USA
,
Meikel Poess
Oracle Corporation, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMETRICS: ACM Special Interest Group on Measurement and Evaluation
SIGSOFT: ACM Special Interest Group on Software Engineering
SPEC: SPEC Research Group

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 April 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

ERDF - European Regional Development Fund
Portuguese funding agency, FCT - Fundacao para a Ciencia e a Tecnologia

Conference

ICPE '17

Sponsor:

ICPE '17: ACM/SPEC International Conference on Performance Engineering

April 22 - 26, 2017

L'Aquila, Italy

Acceptance Rates

ICPE '17 Paper Acceptance Rate 27 of 83 submissions, 33%;

Overall Acceptance Rate 252 of 851 submissions, 30%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
443
Total Downloads

Downloads (Last 12 months)37
Downloads (Last 6 weeks)3

Reflects downloads up to 10 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang CLi GLv T(2024)HyBench: A New Benchmark for HTAP DatabasesProceedings of the VLDB Endowment10.14778/3641204.364120617:5(939-951)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.14778/3641204.3641206
Zhang CLi GZhang JZhang XFeng J(2024)HTAP Databases: A SurveyIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.338969336:11(6410-6429)Online publication date: Nov-2024
https://doi.org/10.1109/TKDE.2024.3389693
Weng SWang QQu LZhang RCai PQian WZhou A(2024)Lauca: A Workload Duplicator for Benchmarking Transactional Database PerformanceIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.336011636:7(3180-3194)Online publication date: Jul-2024
https://doi.org/10.1109/TKDE.2024.3360116
Song HZhou WCui HPeng XLi F(2024)A survey on hybrid transactional and analytical processingThe VLDB Journal10.1007/s00778-024-00858-933:5(1485-1515)Online publication date: 4-Jun-2024
https://doi.org/10.1007/s00778-024-00858-9
Perera ROetomo BRubinstein BBorovica-Gajic R(2023)No DBA? No Regret! Multi-Armed Bandits for Index Tuning of Analytical and HTAP Workloads With Provable GuaranteesIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.327166435:12(12855-12872)Online publication date: 1-Dec-2023
https://doi.org/10.1109/TKDE.2023.3271664
Kang GChen SLi H(2023)Benchmarking HTAP databases for performance isolation and real-time analyticsBenchCouncil Transactions on Benchmarks, Standards and Evaluations10.1016/j.tbench.2023.1001223:2(100122)Online publication date: Jun-2023
https://doi.org/10.1016/j.tbench.2023.100122
Milkai EChronis YGaffney KGuo ZPatel JYu XIves ZBonifati AEl Abbadi A(2022)How Good is My HTAP System?Proceedings of the 2022 International Conference on Management of Data10.1145/3514221.3526148(1810-1824)Online publication date: 10-Jun-2022
https://dl.acm.org/doi/10.1145/3514221.3526148
Li GZhang CIves ZBonifati AEl Abbadi A(2022)HTAP Databases: What is New and What is NextProceedings of the 2022 International Conference on Management of Data10.1145/3514221.3522565(2483-2488)Online publication date: 10-Jun-2022
https://dl.acm.org/doi/10.1145/3514221.3522565
Kang GWang LGao WTang FZhan J(2022)OLxPBench: Real-time, Semantically Consistent, and Domain-specific are Essential in Benchmarking, Designing, and Implementing HTAP Systems2022 IEEE 38th International Conference on Data Engineering (ICDE)10.1109/ICDE53745.2022.00182(1822-1834)Online publication date: May-2022
https://doi.org/10.1109/ICDE53745.2022.00182
Qu LLi YZhang RChen TShu KQian WZhou A(2022)Application-Oriented Workload Generation for Transactional Database Performance Evaluation2022 IEEE 38th International Conference on Data Engineering (ICDE)10.1109/ICDE53745.2022.00036(420-432)Online publication date: May-2022
https://doi.org/10.1109/ICDE53745.2022.00036
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten