research-article

A Declarative Approach for Performance Tests Execution in Continuous Software Development Environments

Authors:

Vincenzo Ferme,

Cesare PautassoAuthors Info & Claims

ICPE '18: Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering

Pages 261 - 272

https://doi.org/10.1145/3184407.3184417

Published: 30 March 2018 Publication History

Abstract

Software performance testing is an important activity to ensure quality in continuous software development environments. Current performance testing approaches are mostly based on scripting languages and framework where users implement, in a procedural way, the performance tests they want to issue to the system under test. However, existing solutions lack support for explicitly declaring the performance test goals and intents. Thus, while it is possible to express how to execute a performance test, its purpose and applicability context remain implicitly described. In this work, we propose a declarative domain specific language (DSL) for software performance testing and a model-driven framework that can be programmed using the mentioned language and drive the end-to-end process of executing performance tests. Users of the DSL and the framework can specify their performance intents by relying on a powerful goal-oriented language, where standard (e.g., load tests) and more advanced (e.g., stability boundary detection, and configuration tests) performance tests can be specified starting from templates. The DSL and the framework have been designed to be integrated into a continuous software development process and validated through extensive use cases that illustrate the expressiveness of the goal-oriented language, and the powerful control it enables on the end-to-end performance test execution to determine how to reach the declared intent.

References

[1]

Varsha Apte, T V S Viswanath, Devidas Gawali, Akhilesh Kommireddy, and Anshul Gupta. 2017. AutoPerf: Automated load testing and resource usage profiling of multi-tier internet applications. In Proc. of ICPE. 115--126.

Digital Library

[2]

Maicon Bernardino, Avelino F Zorzo, and Elder M Rodrigues. 2014. Canopus: A Domain-Specific Language for Modeling Performance Testing Proc. of ICSEA. 157--167.

[3]

Andreas Brunnert and Helmut Krcmar. 2017. Continuous performance evaluation and capacity planning using resource profiles for enterprise applications. Journal of Systems and Software Vol. 123, 1 (2017), 239--262.

[4]

Andreas Brunnert, André van Hoorn, Felix Willnecker, et almbox. 2015. Performance-oriented DevOps: A Research Agenda. Technical Report. SPEC RG DevOps.

[5]

Matheus Cunha, Nabor C Mendonc ca, and Américo Sampaio. 2013. A Declarative Environment for Automatic Performance Evaluation in IaaS Clouds Proc. of CLOUD. 285--292.

Digital Library

[6]

Stefano Di Alesio, Shiva Nejati, Arnaud Gotlieb, and Lionel Briand. 2013. Stress testing of task deadlines - A constraint programming approach Proc of. ISSRE. 158--167.

[7]

Vincenzo Ferme, Ana Ivanchikj, and Cesare Pautasso. 2015. A Framework for Benchmarking BPMN 2.0 Workflow Management Systems Proc. of BPM. 251--259.

Digital Library

[8]

Vincenzo Ferme and Cesare Pautasso. 2017. Towards Holistic Continuous Software Performance Assessment Proc. of ICPE Companion. 159--164.

Digital Library

[9]

Brian Fitzgerald and Klaas-Jan Stol. 2017. Continuous software engineering: A roadmap and agenda. Journal of Systems and Software Vol. 123, 1 (2017), 176--189.

[10]

Ilias Gerostathopoulos, Tomas Bures, Sanny Schmid, Vojtech Horký, Christian Prehofer, and Petr Truma. 2016. Towards Systematic Live Experimentation in Software-Intensive Systems of Systems Proc. of ECSA. 1--7.

Digital Library

[11]

Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D Sculley. 2017. Google Vizier - A Service for Black-Box Optimization Proc. of KDD. 1487--1495.

Digital Library

[12]

Juha Itkonen, Casper Lassenius, and Eero Laukkanen. 2017. Problems, causes and solutions when adopting continuous delivery - A systematic literature review. Information and Software Technology Vol. 82, 2 (2017), 55--79.

[13]

Nicolas Michael, Nitin Ramannavar, Yixiao Shen, Sheetal Patil, and Jan-Lung Sung. 2017. CloudPerf - A Performance Test Framework for Distributed and Dynamic Multi-Tenant Environments. In Proc. of ICPE. 189--200.

Digital Library

[14]

Jean Christophe Petkovich, A Oliveira, Y Zhang, Thomas Reidemeister, and Sebastian Fischmeister. 2015. DataMill: a distributed heterogeneous infrastructure for robust experimentation. Software: Practice and Experience Vol. 46, 10 (2015), 1411--1440.

Digital Library

[15]

A Omar Portillo-Dominguez, Miao Wang, John Murphy, Damien Magoni, Nick Mitchell, Peter F Sweeney, and Erik Altman. 2014. Towards an automated approach to use expert systems in the performance testing of distributed systems. In Proc. of JAMAICA. 22--27.

Digital Library

[16]

Gerald Schermann, Dominik Schöni, Philipp Leitner, and Harald C Gall. 2016. Bifrost - Supporting Continuous Deployment with Automated Enactment of Multi-Phase Live Testing Strategies. In Proc. of Middleware. 12:1--12:14.

Digital Library

[17]

Joel Scheuner, Philipp Leitner, Jürgen Cito, and Harald C Gall. 2014. Cloud Work Bench - Infrastructure-as-Code Based Cloud Benchmarking. Proc. of CloudCom. 246--253.

Digital Library

[18]

Simon Spinner, Giuliano Casale, Fabian Brosig, and Samuel Kounev. 2015. Evaluating approaches to resource demand estimation. Performance Evaluation Vol. 92, C (2015), 51--71.

Digital Library

[19]

Jan Waller, Nils C Ehmke, and Wilhelm Hasselbring. 2015. Including Performance Benchmarks into Continuous Integration to Enable DevOps. ACM SIGSOFT Software Engineering Notes Vol. 40, 2 (2015), 1--4.

Digital Library

[20]

Jürgen Walter, André van Hoorn, Heiko Koziolek, Dusan Okanovic, and Samuel Kounev. 2016. Asking "What"?, Automating the "How"? - The Vision of Declarative Performance Engineering Proc. of ICPE. 91--94.

Digital Library

[21]

Dennis Westermann. 2014. Deriving Goal-oriented Performance Models by Systematic Experimentation. Ph.D. Dissertation. Karlsruhe Institute of Technology.

[22]

Johannes Wettinger, Uwe Breitenbücher, Michael Falkenthal, and Frank Leymann. 2016. Collaborative gathering and continuous delivery of DevOps solutions through repositories. Computer Science - Research and Development Vol. 31, 4 (2016), 1--10.

Digital Library

[23]

Liming Zhu, Len Bass, and George Champlin-Scharff. 2016. DevOps and Its Practices. IEEE Software Vol. 33, 3 (2016), 32--34.

Cited By

Almufareh MHumayun M(2023)Improving the Safety and Security of Software Systems by Mediating SAP VerificationApplied Sciences10.3390/app1301064713:1(647)Online publication date: 3-Jan-2023
https://doi.org/10.3390/app13010647
Esteves RBrito M(2023)Agnostic Performance Testing FrameworkProceedings of the 2023 9th International Conference on Computer Technology Applications10.1145/3605423.3605436(179-183)Online publication date: 10-May-2023
https://dl.acm.org/doi/10.1145/3605423.3605436
Camilli MColarusso CRusso BZimeo E(2023)Actor-Driven Decomposition of Microservices through Multi-level Scalability AssessmentACM Transactions on Software Engineering and Methodology10.1145/358356332:5(1-46)Online publication date: 15-Feb-2023
https://dl.acm.org/doi/10.1145/3583563
Show More Cited By

Index Terms

A Declarative Approach for Performance Tests Execution in Continuous Software Development Environments

Recommendations

Declarative Software Development: Distilled Tutorial
PPDP '14: Proceedings of the 16th International Symposium on Principles and Practice of Declarative Programming

Software development could be said to be declarative, if declarative programming languages were used significantly in the development of a software system. Software development could also be said to be declarative, if lightweight or heavyweight formal ...
PerfRanker: prioritization of performance regression tests for collection-intensive software
ISSTA 2017: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis

Regression performance testing is an important but time/resource-consuming phase during software development. Developers need to detect performance regressions as early as possible to reduce their negative impact and fixing cost. However, conducting ...
Search-based mutation testing to improve performance tests
GECCO '18: Proceedings of the Genetic and Evolutionary Computation Conference Companion

Performance bugs are common and can cause a significant deterioration in the behaviour of a program, leading to costly issues. To detect them and reduce their impact, performance tests are typically applied. However, there is a lack of mechanisms to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPE '18: Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering

March 2018

328 pages

ISBN:9781450350952

DOI:10.1145/3184407

General Chairs:
Katinka Wolter
Free University of Berlin, Germany
,
Will Knottenbelt
Imperial College London, UK
,
Program Chairs:
André van Hoorn
University of Stuttgart, Germany
,
Manoj Nambiar
Tata Consultancy Services, India
,
Heiko Koziolek
ABB, Germany

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 March 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Swiss National Science Foundation

Conference

ICPE '18

Sponsor:

ICPE '18: ACM/SPEC International Conference on Performance Engineering

April 9 - 13, 2018

Berlin, Germany

Acceptance Rates

Overall Acceptance Rate 252 of 851 submissions, 30%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
367
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)4

Reflects downloads up to 30 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Almufareh MHumayun M(2023)Improving the Safety and Security of Software Systems by Mediating SAP VerificationApplied Sciences10.3390/app1301064713:1(647)Online publication date: 3-Jan-2023
https://doi.org/10.3390/app13010647
Esteves RBrito M(2023)Agnostic Performance Testing FrameworkProceedings of the 2023 9th International Conference on Computer Technology Applications10.1145/3605423.3605436(179-183)Online publication date: 10-May-2023
https://dl.acm.org/doi/10.1145/3605423.3605436
Camilli MColarusso CRusso BZimeo E(2023)Actor-Driven Decomposition of Microservices through Multi-level Scalability AssessmentACM Transactions on Software Engineering and Methodology10.1145/358356332:5(1-46)Online publication date: 15-Feb-2023
https://dl.acm.org/doi/10.1145/3583563
Khan HAfsar WNazir SNoor AKundi MMaashi MAlshahrani H(2023)Revolutionizing software developmental processes by utilizing continuous software approachesThe Journal of Supercomputing10.1007/s11227-023-05818-880:7(9579-9608)Online publication date: 12-Dec-2023
https://doi.org/10.1007/s11227-023-05818-8
Camilli MGuerriero AJanes ARusso BRusso SGarrido AWong WDe Angelis GDo HNguyen B(2022)Microservices integrated performance and reliability testingProceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test10.1145/3524481.3527233(29-39)Online publication date: 17-May-2022
https://dl.acm.org/doi/10.1145/3524481.3527233
Keskin RPileggi P(2022)Making the Cloud Monitor Real-Time Adaptive2022 IEEE Cloud Summit10.1109/CloudSummit54781.2022.00017(69-74)Online publication date: Oct-2022
https://doi.org/10.1109/CloudSummit54781.2022.00017
Camilli MJanes ARusso B(2022)Automated test-based learning and verification of performance models for microservices systemsJournal of Systems and Software10.1016/j.jss.2022.111225187:COnline publication date: 12-May-2022
https://dl.acm.org/doi/10.1016/j.jss.2022.111225
Schulz HOkanović Dvan Hoorn ATůma PBourcier JJiang ZBezemer CCortellessa V(2021)Context-tailored Workload Model Generation for Continuous Representative Load TestingProceedings of the ACM/SPEC International Conference on Performance Engineering10.1145/3427921.3450240(21-32)Online publication date: 9-Apr-2021
https://dl.acm.org/doi/10.1145/3427921.3450240
Moghadam MHamidi GBorg MSaadatmand MBohlin MLisper BPotena P(2021)Performance Testing Using a Smart Reinforcement Learning-Driven Test Agent2021 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC45853.2021.9504763(2385-2394)Online publication date: 28-Jun-2021
https://doi.org/10.1109/CEC45853.2021.9504763
Moghadam MSaadatmand MBorg MBohlin MLisper B(2021)An autonomous performance testing framework using self-adaptive fuzzy reinforcement learningSoftware Quality Journal10.1007/s11219-020-09532-z30:1(127-159)Online publication date: 10-Mar-2021
https://doi.org/10.1007/s11219-020-09532-z
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten