research-article

An Exploratory Study of the State of Practice of Performance Testing in Java-Based Open Source Projects

Authors:

Philipp Leitner,

Cor-Paul BezemerAuthors Info & Claims

ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

Pages 373 - 384

https://doi.org/10.1145/3030207.3030213

Published: 17 April 2017 Publication History

Abstract

The usage of open source (OS) software is wide-spread across many industries. While the functional quality of OS projects is considered to be similar to closed-source software, much is unknown about the quality in terms of performance. One challenge for OS developers is that, unlike for functional testing, there is a lack of accepted best practices for performance testing. To reveal the state of practice of performance testing in OS projects, we conduct an exploratory study on 111 Java-based OS projects from GitHub. We study the performance tests of these projects from five perspectives: (1) developers, (2) size, (3) test organization, (4) types of performance tests and (5) used tooling. We show that writing performance tests is not a popular task in OS projects: performance tests form only a small portion of the test suite, are rarely updated, and are usually maintained by a small group of core project developers. Further, even though many projects are aware that they need performance tests, developers appear to struggle implementing them. We argue that future performance testing frameworks should provider better support for low-friction testing, for instance via non-parameterized methods or performance test generation, as well as focus on a tight integration with standard continuous integration tooling.

References

[1]

M. Aberdour. Achieving quality in open-source software. IEEE Software, 24(1):58--64, Jan 2007.

Digital Library

[2]

J. P. S. Alcocer and A. Bergel. Tracking down performance variation against source code evolution. In Proceedings of the 11th Symposium on Dynamic Languages (DLS), pages 129--139. ACM, 2015.

Digital Library

[3]

A. Avritzer, J. Kondek, D. Liu, and E. J. Weyuker. Software performance testing based on workload characterization. In Proceedings of the 3rd International Workshop on Software and Performance (WOSP), pages 17--24. ACM, 2002.

Digital Library

[4]

S. Baltes, O. Moseler, F. Beck, and S. Diehl. Navigate, understand, communicate: How developers locate performance bugs. In 2015 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 1--10, Oct 2015.

[5]

M. Beller, G. Gousios, A. Panichella, and A. Zaidman. When, how, and why developers (do not) test in their IDEs. In Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE), pages 179--190. ACM, 2015.

Digital Library

[6]

S. M. Blackburn, R. Garner, C. Hoffmann, A. M. Khang, K. S. McKinley, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J. E. B. Moss, A. Phansalkar, D. Stefanović, T. VanDrunen, D. von Dincklage, and B. Wiedermann. The DaCapo benchmarks: Java benchmarking development and analysis. In Proceedings of the 21st ACM SIGPLAN Conference on Object-oriented Programming Systems, Languages, and Applications (OOPSLA), pages 169--190. ACM, 2006.

Digital Library

[7]

J. Cito, P. Leitner, T. Fritz, and H. C. Gall. The making of cloud applications: An empirical study on software development for the cloud. In Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE), pages 393--403. ACM, 2015.

Digital Library

[8]

A. Georges, D. Buytaert, and L. Eeckhout. Statistically rigorous java performance evaluation. In Proceedings of the 22nd ACM SIGPLAN Conference on Object-oriented Programming Systems and Applications (OOPSLA), pages 57--76. ACM, 2007.

Digital Library

[9]

X. Han and T. Yu. An empirical study on performance bugs for highly configurable software systems. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 23:1--23:10, 2016.

Digital Library

[10]

C. Heger, J. Happe, and R. Farahbod. Automated Root Cause Isolation of Performance Regressions During Software Development. In Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering (ICPE), pages 27--38, 2013.

Digital Library

[11]

I. Herraiz and A. E. Hassan. Beyond lines of code: Do we need more complexity metrics? Making software: what really works, and why we believe it, pages 125--141, 2010.

[12]

V. Horký, P. Libič, L. Marek, A. Steinhauser, and P. Tůma. Utilizing performance unit tests to increase performance awareness. In Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE), pages 289--300. ACM, 2015.

Digital Library

[13]

P. Huang, X. Ma, D. Shen, and Y. Zhou. Performance regression testing target prioritization via performance risk analysis. In Proceedings of the 36th International Conference on Software Engineering (ICSE), pages 60--71. ACM, 2014.

Digital Library

[14]

Z. M. Jiang and A. E. Hassan. A survey on load testing of large-scale software systems. IEEE Transactions on Software Engineering (TSE), 41(11):1091--1118, 2015.

[15]

G. Jin, L. Song, X. Shi, J. Scherpelz, and S. Lu. Understanding and detecting real-world performance bugs. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 77--88. ACM, 2012.

Digital Library

[16]

J. D. Long, D. Feng, and N. Cliff. Ordinal analysis of behavioral data. Handbook of psychology, 2003.

[17]

Q. Luo, D. Poshyvanyk, and M. Grechanik. Mining performance regression inducing code changes in evolving software. In Proceedings of the 13th International Conference on Mining Software Repositories (MSR), pages 25--36. ACM, 2016.

Digital Library

[18]

A. Mockus, R. T. Fielding, and J. Herbsleb. A case study of open source software development: The apache server. In Proceedings of the 22nd International Conference on Software Engineering (ICSE), pages 263--272. ACM, 2000.

Digital Library

[19]

R. Pooley. Software engineering and performance: A roadmap. In Proceedings of the Conference on The Future of Software Engineering (ICSE), pages 189--199. ACM, 2000.

Digital Library

[20]

J. Romano, J. D. Kromrey, J. Coraggio, J. Skowronek, and L. Devine. Exploring methods for evaluating group differences on the NSSE and other surveys: Are the t-test and Cohen's d indices the most appropriate choices. In Annual meeting of the Southern Association for Institutional Research, 2006.

[21]

J. P. Sandoval Alcocer, A. Bergel, and M. T. Valente. Learning from source code history to identify performance failures. In Proceedings of the 7th ACM/SPEC on International Conference on Performance Engineering (ICPE), pages 37--48, 2016.

Digital Library

[22]

C. U. Smith and L. G. Williams. Software performance engineering: A case study including performance comparison with design alternatives. IEEE Transactions on Software Engineering (TSE), 19(7):720--741, July 1993.

Digital Library

[23]

M. D. Syer, Z. M. Jiang, M. Nagappan, A. E. Hassan, M. Nasser, and P. Flora. Continuous validation of load test suites. In Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering (ICPE), pages 259--270. ACM, 2014.

Digital Library

[24]

M. L. Vasquez, C. Vendome, Q. Luo, and D. Poshyvanyk. How developers detect and fix performance bottlenecks in Android apps. In R. Koschke, J. Krinke, and M. P. Robillard, editors, Proceedings of the International Conference on Software Maintenance and Evolution (ICSME), pages 352--361. IEEE Computer Society, 2015.

[25]

E. J. Weyuker and F. I. Vokolos. Experience with performance testing of software systems: Issues, an approach, and case study. IEEE Transactions on Software Engineering (TSE), 26(12):1147--1156, 2000.

Digital Library

[26]

S. Zaman, B. Adams, and A. E. Hassan. A Qualitative Study on Performance Bugs. In Proceedings of the 9th IEEE Working Conference on Mining Software Repositories (MSR), pages 199--208. IEEE Press, 2012.

[27]

F. Zhang, A. E. Hassan, S. McIntosh, and Y. Zou. The use of summation to aggregate software metrics hinders the performance of defect prediction models. IEEE Transactions on Software Engineering (TSE), (1):1--16, 2016.

Digital Library

Cited By

Imran MCortellessa VDi Ruscio DRubei RTraini L(2024)An Empirical Study on Code Coverage of Performance TestingProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661196(48-57)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661196
Laaber CYue TAli S(2024)Evaluating Search-Based Software Microbenchmark PrioritizationIEEE Transactions on Software Engineering10.1109/TSE.2024.338083650:7(1687-1703)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3380836
Bulej LHorký VTucci MTuma PFarquet FLeopoldseder DProkopec AVieira MCardellini VDi Marco ATuma P(2023)GraalVM Compiler Benchmark Results Dataset (Data Artifact)Companion of the 2023 ACM/SPEC International Conference on Performance Engineering10.1145/3578245.3585025(65-69)Online publication date: 15-Apr-2023
https://dl.acm.org/doi/10.1145/3578245.3585025
Show More Cited By

Index Terms

An Exploratory Study of the State of Practice of Performance Testing in Java-Based Open Source Projects
1. General and reference
  1. Cross-computing tools and techniques
    1. Performance

Recommendations

Unit Testing Performance in Java Projects: Are We There Yet?
ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

Although methods and tools for unit testing of performance exist for over a decade, anecdotal evidence suggests unit testing of performance is not nearly as common as unit testing of functionality. We examine this situation in a study of GitHub projects ...
The Maintenance and Evolution of Field-Representative Performance Tests
ICSME '14: Proceedings of the 2014 IEEE International Conference on Software Maintenance and Evolution

The rise of large-scale software systems poses new challenges for the software performance engineering field. Failures in these systems are typically associated with performance issues, rather than with feature bugs. Therefore, performance testing has ...
An Empirical Study on Code Coverage of Performance Testing
EASE '24: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering

Performance testing aims to ensure the operational efficiency of software systems. However, many factors influencing the efficacy and adoption of performance tests in practice are not yet fully understood. For instance, while code coverage is widely ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

April 2017

450 pages

ISBN:9781450344043

DOI:10.1145/3030207

General Chairs:
Walter Binder
Università della Svizzera Italiana, Switzerland
,
Vittorio Cortellessa
Università dell'Aquila, Italy
,
Program Chairs:
Anne Koziolek
Karlsruhe Institute of Technology, Germany
,
Evgenia Smirni
College of William & Mary, USA
,
Meikel Poess
Oracle Corporation, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMETRICS: ACM Special Interest Group on Measurement and Evaluation
SIGSOFT: ACM Special Interest Group on Software Engineering
SPEC: SPEC Research Group

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 April 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung

Conference

ICPE '17

Sponsor:

ICPE '17: ACM/SPEC International Conference on Performance Engineering

April 22 - 26, 2017

L'Aquila, Italy

Acceptance Rates

ICPE '17 Paper Acceptance Rate 27 of 83 submissions, 33%;

Overall Acceptance Rate 252 of 851 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

42
Total Citations
View Citations
447
Total Downloads

Downloads (Last 12 months)24
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Imran MCortellessa VDi Ruscio DRubei RTraini L(2024)An Empirical Study on Code Coverage of Performance TestingProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661196(48-57)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661196
Laaber CYue TAli S(2024)Evaluating Search-Based Software Microbenchmark PrioritizationIEEE Transactions on Software Engineering10.1109/TSE.2024.338083650:7(1687-1703)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3380836
Bulej LHorký VTucci MTuma PFarquet FLeopoldseder DProkopec AVieira MCardellini VDi Marco ATuma P(2023)GraalVM Compiler Benchmark Results Dataset (Data Artifact)Companion of the 2023 ACM/SPEC International Conference on Performance Engineering10.1145/3578245.3585025(65-69)Online publication date: 15-Apr-2023
https://dl.acm.org/doi/10.1145/3578245.3585025
Khosravi Tabrizi AEzzati-Jivan NVieira MCardellini VDi Marco ATuma P(2023)Software Mining -- Investigating Correlation between Source Code Features and Michrobenchmark's Steady StateCompanion of the 2023 ACM/SPEC International Conference on Performance Engineering10.1145/3578245.3584695(107-111)Online publication date: 15-Apr-2023
https://dl.acm.org/doi/10.1145/3578245.3584695
He SKim IWang WVieira MCardellini VDi Marco ATuma P(2023)A Study of Java Microbenchmark Tail LatenciesCompanion of the 2023 ACM/SPEC International Conference on Performance Engineering10.1145/3578245.3584690(77-81)Online publication date: 15-Apr-2023
https://dl.acm.org/doi/10.1145/3578245.3584690
Zhang XChen LZou WCao YRen HWang ZLi YZhou Y(2023)ICG: A Machine Learning Benchmark Dataset and Baselines for Inline Code Comments Generation TaskInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402350054734:02(331-356)Online publication date: 20-Oct-2023
https://doi.org/10.1142/S0218194023500547
Macklon FViggiato MRomanova NBuzon CPaas DBezemer C(2023)A Taxonomy of Testable HTML5 Canvas IssuesIEEE Transactions on Software Engineering10.1109/TSE.2023.3270740(1-13)Online publication date: 2023
https://doi.org/10.1109/TSE.2023.3270740
Trubiani CPinciroli RBiaggi AFontana F(2023)Automated Detection of Software Performance Antipatterns in Java-Based ApplicationsIEEE Transactions on Software Engineering10.1109/TSE.2023.323432149:4(2873-2891)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2023.3234321
Zhang XZou WChen LLi YZhou Y(2023)Towards the Analysis and Completion of Syntactic Structure Ellipsis for Inline CommentsIEEE Transactions on Software Engineering10.1109/TSE.2022.321627949:4(2285-2302)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2022.3216279
Jangali MTang YAlexandersson NLeitner PYang JShang W(2023)Automated Generation and Evaluation of JMH Microbenchmark Suites From Unit TestsIEEE Transactions on Software Engineering10.1109/TSE.2022.318800549:4(1704-1725)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2022.3188005
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents