short-paper

Behavior-driven Load Testing Using Contextual Knowledge - Approach and Experiences

Authors:

Henning Schulz,

Dušan Okanović,

André van Hoorn,

Vincenzo Ferme,

Cesare PautassoAuthors Info & Claims

ICPE '19: Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering

Pages 265 - 272

https://doi.org/10.1145/3297663.3309674

Published: 04 April 2019 Publication History

Abstract

Load testing is widely considered a meaningful technique for performance quality assurance. However, empirical studies reveal that in practice, load testing is not applied systematically, due to the sound expert knowledge required to specify, implement, and execute load tests.

Our Behavior-driven Load Testing (BDLT) approach eases load test specification and execution for users with no or little expert knowledge. It allows a user to describe a load test in a template-based natural language and to rely on an automated framework to execute the test. Utilizing the system's contextual knowledge such as workload-influencing events, the framework automatically determines the workload and test configuration. We investigated the applicability of our approach in an industrial case study, where we were able to express four load test concerns using BDLT and received positive feedback from our industrial partner. They understood the BDLT definitions well and proposed further applications, such as the usage for software quality acceptance criteria.

References

[1]

Varsha Apte, T V S Viswanath, Devidas Gawali, Akhilesh Kommireddy, and Anshul Gupta. 2017. AutoPerf: Automated load testing and resource usage profiling of multi-tier internet applications. In Proc. ICPE 2017. 115--126.

Digital Library

[2]

Alberto Avritzer, Vincenzo Ferme, Andrea Janes, Barbara Russo, Henning Schulz, and André van Hoorn. 2018. A Quantitative Approach for the Assessment of Microservice Architecture Deployment Alternatives by Automated Performance Testing. In Proc. ECSA 2018. 159--174.

[3]

Maicon Bernardino, Avelino F Zorzo, and Elder M Rodrigues. 2014. Canopus: A Domain-Specific Language for Modeling Performance Testing. In Proc. ICSEA 2014. 157--167.

[4]

Cor-Paul Bezemer, Simon Eismann, Vincenzo Ferme, Johannes Grohmann, Robert Heinrich, Pooyan Jamshidi, Weiyi Shang, André van Hoorn, Monica Villavicencio, Jürgen Walter, and Felix Willnecker. 2018. How is Performance Addressed in DevOps? A Survey on Industrial Practices. In Proc. ICPE 2019 .

Digital Library

[5]

Yuhong Cai, John C. Grundy, and John G. Hosking. 2007. Synthesizing Client Load Models for Performance Engineering via Web Crawling. In Proc. ASE 2007. 353--362.

Digital Library

[6]

Maria Carla Calzarossa, Luisa Massari, and Daniele Tessera. 2016. Workload Characterization: A Survey Revisited. Comput. Surveys, Vol. 48, 3 (2016), 48:1--48:43.

Digital Library

[7]

Vincenzo Ferme and Cesare Pautasso. 2017. Towards Holistic Continuous Software Performance Assessment. In Proc. QUDOS@ICPE 2017. 159--164.

Digital Library

[8]

Vincenzo Ferme and Cesare Pautasso. 2018. A Declarative Approach for Performance Tests Execution in Continuous Software Development Environments. In Proc. ICPE 2018. 261--272.

Digital Library

[9]

Nikolas Roman Herbst, Nikolaus Huber, Samuel Kounev, and Erich Amrehn. 2013. Self-Adaptive Workload Classification and Forecasting for Proactive Resource Provisioning. In Proc. ICPE 2013. 187--198.

Digital Library

[10]

Zhen Ming Jiang and Ahmed E Hassan. 2015. A Survey on Load Testing of Large-Scale Software Systems. IEEE Trans. Soft. Eng., Vol. 41, 11 (2015), 1091--1118.

[11]

Tobias Kuhn. 2014. A survey and classification of controlled natural languages. Computational Linguistics, Vol. 40, 1 (2014), 121--170.

Digital Library

[12]

Abderrahman Matoussi and Régine Laleau. 2008. A Survey of Non-Functional Requirements in Software Development Process. Research Report TR-LACL-2008-7. LACL.

[13]

Daniel A. Menascé and Virgilio A. F. Almeida. 2002. Capacity Planning for Web Services: Metrics, Models and Methods 1st ed.). Prentice Hall, Upper Saddle River, NJ, USA.

Digital Library

[14]

Dan North. 2006. Introducing BDD. https://dannorth.net/introducing-bdd/. (2006).

[15]

Duvsan Okanoviç, André van Hoorn, Christoph Zorn, Fabian Beck, Vincenzo Ferme, and Jürgen Walter. 2019. Concern-driven Reporting of Software Performance Analysis Results. In Proc. ICPE 2019.

Digital Library

[16]

Henning Schulz, Tobias Angerstein, and André van Hoorn. 2018. Towards Automating Representative Load Testing in Continuous Software Engineering. In Proc. ICPE 2018. 123--126.

Digital Library

[17]

Henning Schulz, Dusan Okanović, André van Hoorn, Vincenzo Ferme, and Cesare Pautasso. 2019. Behavior-driven Load Testing Using Contextual Knowledge-Approach and Experiences. (Feb. 2019).

Digital Library

[18]

Ken Schwaber and Mike Beedle. 2001. Agile Software Development with Scrum. Prentice Hall PTR.

Digital Library

[19]

Mahnaz Shams, Diwakar Krishnamurthy, and Behrouz Homayoun Far. 2006. A Model-Based Approach for Testing the Performance of Web Applications. In Proc. SOQUA 2006. 54--61.

Digital Library

[20]

Christian Vögele, André van Hoorn, Eike Schulz, Wilhelm Hasselbring, and Helmut Krcmar. 2018. WESSBAS: Extraction of Probabilistic Workload Specifications for Load Testing and Performance Prediction -- a Model-Driven Approach for Session-Based Application Systems. Software and System Modeling, Vol. 17, 2 (2018), 443--477.

Digital Library

[21]

Jürgen Walter, André van Hoorn, Heiko Koziolek, Dusan Okanovic, and Samuel Kounev. 2016. Asking "What"?, Automating the "How"? - The Vision of Declarative Performance Engineering. In Proc. ICPE 2016. 91--94.

Digital Library

[22]

Yang Wang and Stefan Wagner. 2018. Combining STPA and BDD for Safety Analysis and Verification in Agile Development: A Controlled Experiment. In Proc. XP 2018. 37--53.

[23]

Matt Wynne, Aslak Hellesoy, and Steve Tooke. 2017. The Cucumber Book: Behaviour-Driven Development for Testers and Developers. O'Reilly UK Ltd.

Digital Library

Cited By

Prawira MAbdussalam FYossy E(2023)Database Implementation using Citus on Telecommunication Data2023 IEEE 9th International Conference on Computing, Engineering and Design (ICCED)10.1109/ICCED60214.2023.10425036(1-6)Online publication date: 7-Nov-2023
https://doi.org/10.1109/ICCED60214.2023.10425036
Binamungu LMaro S(2023)Behaviour driven developmentJournal of Systems and Software10.1016/j.jss.2023.111749203:COnline publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1016/j.jss.2023.111749
Avritzer ACamilli MJanes ARusso BTrubiani Cvan Hoorn A(2023)Continuous Dependability Assessment of Microservice SystemsSoftware Architecture. ECSA 2022 Tracks and Workshops10.1007/978-3-031-36889-9_11(138-147)Online publication date: 16-Jul-2023
https://doi.org/10.1007/978-3-031-36889-9_11
Show More Cited By

Index Terms

Behavior-driven Load Testing Using Contextual Knowledge - Approach and Experiences
1. Software and its engineering

Recommendations

Analytics-driven load testing: an industrial experience report on load testing of large-scale systems
ICSE-SEIP '17: Proceedings of the 39th International Conference on Software Engineering: Software Engineering in Practice Track

Assessing how large-scale software systems behave under load is essential because many problems cannot be uncovered without executing tests of large volumes of concurrent requests. Load-related problems can directly affect the customer- perceived ...
A Survey on Load Testing of Large-Scale Software Systems
Many large-scale software systems must service thousands or millions of concurrent requests. These systems must be load tested to ensure that they can function correctly under load (i.e., the rate of the incoming requests). In this paper, we survey the ...
Delivering Web service load testing as a service with a global cloud

In this paper, we present WS-TaaS, a Web services load testing platform built on a global platform PlanetLab. WS-TaaS enables load testing process to be simple, transparent, and as close as possible to the real running scenarios of the target services. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPE '19: Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering

April 2019

348 pages

ISBN:9781450362399

DOI:10.1145/3297663

General Chairs:
Varsha Apte
IIT Bombay, India
,
Antinisca Di Marco
University of L'Aquila, Italy
,
Program Chairs:
Marin Litoiu
York University, Canada
,
José Merseguer
Universidad de Zaragoza, Spain

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 April 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Conference

ICPE '19

Sponsor:

ICPE '19: Tenth ACM/SPEC International Conference on Performance Engineering

April 7 - 11, 2019

Mumbai, India

Acceptance Rates

ICPE '19 Paper Acceptance Rate 13 of 71 submissions, 18%;

Overall Acceptance Rate 252 of 851 submissions, 30%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
298
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)3

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Prawira MAbdussalam FYossy E(2023)Database Implementation using Citus on Telecommunication Data2023 IEEE 9th International Conference on Computing, Engineering and Design (ICCED)10.1109/ICCED60214.2023.10425036(1-6)Online publication date: 7-Nov-2023
https://doi.org/10.1109/ICCED60214.2023.10425036
Binamungu LMaro S(2023)Behaviour driven developmentJournal of Systems and Software10.1016/j.jss.2023.111749203:COnline publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1016/j.jss.2023.111749
Avritzer ACamilli MJanes ARusso BTrubiani Cvan Hoorn A(2023)Continuous Dependability Assessment of Microservice SystemsSoftware Architecture. ECSA 2022 Tracks and Workshops10.1007/978-3-031-36889-9_11(138-147)Online publication date: 16-Jul-2023
https://doi.org/10.1007/978-3-031-36889-9_11
Whiting EDatta S(2022)Design and Development of a Technology-Agnostic NFR Testing FrameworkProceedings of the 2022 5th International Conference on Software Engineering and Information Management10.1145/3520084.3520092(45-50)Online publication date: 21-Jan-2022
https://dl.acm.org/doi/10.1145/3520084.3520092
Schulz HOkanović Dvan Hoorn ATůma PBourcier JJiang ZBezemer CCortellessa V(2021)Context-tailored Workload Model Generation for Continuous Representative Load TestingProceedings of the ACM/SPEC International Conference on Performance Engineering10.1145/3427921.3450240(21-32)Online publication date: 9-Apr-2021
https://dl.acm.org/doi/10.1145/3427921.3450240
Moghadam MHamidi GBorg MSaadatmand MBohlin MLisper BPotena P(2021)Performance Testing Using a Smart Reinforcement Learning-Driven Test Agent2021 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC45853.2021.9504763(2385-2394)Online publication date: 28-Jun-2021
https://doi.org/10.1109/CEC45853.2021.9504763
Alnafessah AGias AWang RZhu LCasale GFilieri A(2021)Quality-Aware DevOps Research: Where Do We Stand?IEEE Access10.1109/ACCESS.2021.30648679(44476-44489)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3064867
Andry JHadiyanto Kadarisman V(2021)Comparison Elicitation Techniques in Furniture Manufacture RequirementE3S Web of Conferences10.1051/e3sconf/202131705022317(05022)Online publication date: 5-Nov-2021
https://doi.org/10.1051/e3sconf/202131705022
Moghadam MSaadatmand MBorg MBohlin MLisper B(2021)An autonomous performance testing framework using self-adaptive fuzzy reinforcement learningSoftware Quality Journal10.1007/s11219-020-09532-z30:1(127-159)Online publication date: 10-Mar-2021
https://doi.org/10.1007/s11219-020-09532-z
Gias Avan Hoorn AZhu LCasale GDüllmann TWurster MAmaral JKoziolek ATrubiani CIosup A(2020)Performance Engineering for Microservices and Serverless Applications: The RADON ApproachCompanion of the ACM/SPEC International Conference on Performance Engineering10.1145/3375555.3383120(46-49)Online publication date: 20-Apr-2020
https://dl.acm.org/doi/10.1145/3375555.3383120
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents