short-paper

PerfCI: a toolchain for automated performance testing during continuous integration of Python projects

Authors:

Joshua Heneage Dawes,

Giovanni Franzoni,

Andreas Pfeiffer,

Walter BinderAuthors Info & Claims

ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

Pages 1344 - 1348

https://doi.org/10.1145/3324884.3415288

Published: 27 January 2021 Publication History

Abstract

Software performance testing is an essential quality assurance mechanism that can identify optimization opportunities. Automating this process requires strong tool support, especially in the case of Continuous Integration (CI) where tests need to run completely automatically and it is desirable to provide developers with actionable feedback. A lack of existing tools means that performance testing is normally left out of the scope of CI. In this paper, we propose a toolchain - PerfCI - to pave the way for developers to easily set up and carry out automated performance testing under CI. Our toolchain is based on allowing users to (1) specify performance testing tasks, (2) analyze unit tests on a variety of python projects ranging from scripts to full-blown flask-based web services, by extending a performance analysis framework (VyPR) and (3) evaluate performance data to get feedback on the code. We demonstrate the feasibility of our toolchain by using it on a web service running at the Compact Muon Solenoid (CMS) experiment at the world's largest particle physics laboratory --- CERN.

Package. Source code, example and documentation of PerfCI are available: https://gitlab.cern.ch/omjaved/perfci. Tool demonstration can be viewed on YouTube: https://youtu.be/RDmXMKA1v7g. We also provide the data set used in the analysis: https://gitlab.cern.ch/omjaved/perfci-dataset.

References

[1]

Ezio Bartocci, Yliès Falcone, Adrian Francalanza, and Giles Reger. 2018. Introduction to Runtime Verification. In Lectures on RV.

[2]

Grady Booch. 1990. Object-Oriented Design with Applications.

[3]

J. Burnim, S. Juvekar, and K. Sen. 2009. WISE: Automated test generation for worst-case complexity. In ICSE.

[4]

CERN. 2020. https://home.cern/science/computing/processing-what-record. (Accessed on 05/26/2020).

[5]

Jesús Mauricio Chimento, Wolfgang Ahrendt, and Gerardo Schneider. 2018. Testing Meets Static and Runtime Verification. In FormaliSE@ICSE.

[6]

CircleCI. 2020. https://circleci.com. (Accessed on 05/25/2020).

[7]

David Daly, William Brown, Henrik Ingo, Jim O'Leary, and David Bradford. 2020. The Use of Change Point Detection to Identify Software Performance Regressions in a Continuous Integration System. In ICPE.

[8]

Joshua H Dawes. 2017. A Python object-oriented framework for the CMS alignment and calibration data. In Journal of Physics: Conference Series.

[9]

Joshua Heneage Dawes, Marta Han, Giles Reger, Giovanni Franzoni, and Andreas Pfeiffer. 2019. Analysis Tools for the VyPR Framework for Python. In CHEP.

[10]

Joshua Heneage Dawes and Giles Reger. 2019. Explaining Violations of Properties in Control-Flow Temporal Logic. In RV.

[11]

Joshua Heneage Dawes, Giles Reger, Giovanni Franzoni, Andreas Pfeiffer, and Giacomo Govi. 2019. VyPR2: A Framework for Runtime Verification of Python Web Services. In TACAS.

[12]

T. Durieux, R. Abreu, M. Monperrus, T. F. Bissyandé, and L. Cruz. 2019. An Analysis of 35+ Million Jobs of Travis CI. In ICSME.

[13]

GitLab. 2020. https://docs.gitlab.com/ee/ci/. (Accessed on 05/25/2020).

[14]

Xue Han, Tingting Yu, and David Lo. 2018. PerfLearner: Learning from Bug Reports to Understand and Generate Performance Test Frames. In ASE.

[15]

Guoliang Jin, Linhai Song, Xiaoming Shi, Joel Scherpelz, and Shan Lu. 2012. Understanding and Detecting Real-World Performance Bugs. In PLDI.

[16]

C. Laaber and P. Leitner. 2018. An Evaluation of Open-Source Software Microbenchmark Suites for Continuous Performance Assessment. In 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR). 119--130.

[17]

Rashmi Mudduluru and Murali Krishna Ramanathan. 2016. Efficient Flow Profiling for Detecting Performance Bugs. In ISSTA.

[18]

Jenkins CI performance plugin. 2020. http://jenkinsci.github.io/performance-plugin/RunTests.html. (Accessed on 05/29/2020).

[19]

The Register. 2002. https://www.theregister.co.uk/2002/07/03/1901_census_site_still_down/. (Accessed on 05/25/2020).

[20]

D. G. Reichelt, S. Kühne, and W. Hasselbring. 2019. PeASS: A Tool for Identifying Performance Changes at Code Level. In ASE.

[21]

TravisCI. 2020. https://travis-ci.org/. (Accessed on 05/25/2020).

[22]

Jan Waller, Nils C. Ehmke, and Wilhelm Hasselbring. 2015. Including Performance Benchmarks into Continuous Integration to Enable DevOps. In SIGSOFT Softw. Eng. Notes.

[23]

Fiorella Zampetti, Carmine Vassallo, Sebastiano Panichella, Gerardo Canfora, Harald Gall, and Massimiliano Di Penta. 2020. An empirical characterization of bad practices in continuous integration. In ESE.

Cited By

Esteves RBrito M(2023)Agnostic Performance Testing FrameworkProceedings of the 2023 9th International Conference on Computer Technology Applications10.1145/3605423.3605436(179-183)Online publication date: 10-May-2023
https://dl.acm.org/doi/10.1145/3605423.3605436
Japke NWitzko CGrambow MBermbach DFu SCarnevale LRana OVillari M(2023)The Early Microbenchmark Catches the Bug -- Studying Performance Issues Using Micro- and Application BenchmarksProceedings of the IEEE/ACM 16th International Conference on Utility and Cloud Computing10.1145/3603166.3632128(1-10)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1145/3603166.3632128
Zhao YXiao LBondi AChen BLiu Y(2023)A Large-Scale Empirical Study of Real-Life Performance Issues in Open Source ProjectsIEEE Transactions on Software Engineering10.1109/TSE.2022.316762849:2(924-946)Online publication date: 1-Feb-2023
https://doi.org/10.1109/TSE.2022.3167628
Show More Cited By

PerfCI: a toolchain for automated performance testing during continuous integration of Python projects
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
  2. Software organization and properties
    1. Extra-functional properties

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

December 2020

1449 pages

ISBN:9781450367684

DOI:10.1145/3324884

General Chair:
John Grundy,
Program Chairs:
Claire Le Goues,
David Lo

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 January 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Short-paper

Conference

ASE '20

Sponsor:

ASE '20: 35th IEEE/ACM International Conference on Automated Software Engineering

December 21 - 25, 2020

Virtual Event, Australia

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
186
Total Downloads

Downloads (Last 12 months)53
Downloads (Last 6 weeks)2

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Esteves RBrito M(2023)Agnostic Performance Testing FrameworkProceedings of the 2023 9th International Conference on Computer Technology Applications10.1145/3605423.3605436(179-183)Online publication date: 10-May-2023
https://dl.acm.org/doi/10.1145/3605423.3605436
Japke NWitzko CGrambow MBermbach DFu SCarnevale LRana OVillari M(2023)The Early Microbenchmark Catches the Bug -- Studying Performance Issues Using Micro- and Application BenchmarksProceedings of the IEEE/ACM 16th International Conference on Utility and Cloud Computing10.1145/3603166.3632128(1-10)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1145/3603166.3632128
Zhao YXiao LBondi AChen BLiu Y(2023)A Large-Scale Empirical Study of Real-Life Performance Issues in Open Source ProjectsIEEE Transactions on Software Engineering10.1109/TSE.2022.316762849:2(924-946)Online publication date: 1-Feb-2023
https://doi.org/10.1109/TSE.2022.3167628
Tronge JChen JGrubel PRandles TDavis RWofford QAnaya SGuan Q(2023)An HPC-Container Based Continuous Integration Tool for Detecting Scaling and Performance Issues in HPC ApplicationsIEEE Transactions on Services Computing10.1109/TSC.2023.3337662(1-12)Online publication date: 2023
https://doi.org/10.1109/TSC.2023.3337662
Chen JZhang ZYu DHu H(2023)What makes a real change in software performance? An empirical study on analyzing the factors that affect the triagement of performance change pointsScience of Computer Programming10.1016/j.scico.2023.103068(103068)Online publication date: Nov-2023
https://doi.org/10.1016/j.scico.2023.103068
Sion LVan Landuyt DYskout KVerreydt SJoosen W(2023)Ctam: A Tool for Continuous Threat Analysis and ManagementCyberSecurity in a DevOps Environment10.1007/978-3-031-42212-6_7(195-223)Online publication date: 23-Aug-2023
https://doi.org/10.1007/978-3-031-42212-6_7
Chen JHu HYu DFeng DBecker SHerbst NLeitner P(2022)Characterizing and Triaging Change PointsCompanion of the 2022 ACM/SPEC International Conference on Performance Engineering10.1145/3491204.3527487(33-37)Online publication date: 14-Jul-2022
https://dl.acm.org/doi/10.1145/3491204.3527487
Chen JHu HYu D(2022)Characterizing and Detecting Methods to be Benchmarked under Performance Unit TestInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402250048632:09(1279-1305)Online publication date: 20-Aug-2022
https://doi.org/10.1142/S0218194022500486
Grambow MKovalev DLaaber CLeitner PBermbach D(2022)Using Microbenchmark Suites to Detect Application Performance ChangesIEEE Transactions on Cloud Computing10.1109/TCC.2022.3217947(1-18)Online publication date: 2022
https://doi.org/10.1109/TCC.2022.3217947
Reichelt DKühne SHasselbring W(2022)Automated Identification of Performance Changes at Code Level2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS57517.2022.00096(916-925)Online publication date: Dec-2022
https://doi.org/10.1109/QRS57517.2022.00096
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten