short-paper

Analyzing software engineering experiments: everything you always wanted to know but were afraid to ask

Author:

Sira VegasAuthors Info & Claims

ICSE '18: Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings

Pages 534 - 535

https://doi.org/10.1145/3183440.3183466

Published: 27 May 2018 Publication History

Get Access

Abstract

Experimentation is a key issue in science and engineering. But it is one of software engineering's stumbling blocks. Quite a lot of experiments are run nowadays, but it is a risky business. Software engineering has some special features, leading to some experimentation issues being conceived of differently than in other disciplines. The aim of this technical briefing is to help participants to avoid common pitfalls when analyzing the results of software engineering experiments. The technical briefing is not intended as a data analysis course, because there is already plenty of literature on this subject. It reviews several key issues that we have identified in published software engineering experiments, and addresses them based on the knowledge acquired after 19 years running experiments.

References

[1]

H. Pashler, E.J. Wagenmakers. Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence? Perspectives on Psychological Science, 7(6):528--530, 2012.

Crossref

Google Scholar

[2]

M. Shepperd, D. Bowes, T. Hall. Researcher Bias: The Use of Machine Learning in Software Defect Prediction. IEEE Transactions on Software Engineering, 40(6):603--616, 2014.

Crossref

Google Scholar

[3]

D.I.K. Sjøberg, J.E. Hannay, O. Hansen, V.B. Kampenes, A. Karahasanovic, N.K. Liborg, A.C. Rekdal. A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering, 31(9):733--753, 2005

Digital Library

Google Scholar

Cited By

View all

Malavolta IChinnappan KSwanborn SLewis GLago P(2021)Mining the ROS ecosystem for Green Architectural Tactics in Robotics and an Empirical Evaluation2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)10.1109/MSR52588.2021.00042(300-311)Online publication date: May-2021
https://doi.org/10.1109/MSR52588.2021.00042
Malavolta IChinnappan KJasmontas LGupta SSoltany KLo DMariani LMesbah A(2020)Evaluating the impact of caching on the energy consumption and performance of progressive web appsProceedings of the IEEE/ACM 7th International Conference on Mobile Software Engineering and Systems10.1145/3387905.3388593(109-119)Online publication date: 13-Jul-2020
https://dl.acm.org/doi/10.1145/3387905.3388593

Index Terms

Analyzing software engineering experiments: everything you always wanted to know but were afraid to ask
1. Software and its engineering
  1. Software creation and management

Recommendations

Analyzing software engineering experiments: everything you always wanted to know but were afraid to ask
ICSE-C '17: Proceedings of the 39th International Conference on Software Engineering Companion

Experimentation is a key issue in science and engineering. But it is one of software engineering's stumbling blocks. Quite a lot of experiments are run nowadays, but it is a risky business. Software engineering has some special features, leading to some ...
Analyzing software engineering experiments: everything you always wanted to know but were afraid to ask
ICSE '16: Proceedings of the 38th International Conference on Software Engineering Companion

Experimentation is a key issue in science and engineering. But it is one of software engineering's stumbling blocks. Quite a lot of experiments are run nowadays, but it is a risky business. Software engineering has some special features, leading to some ...
Incorrect results in software engineering experiments

Publication and researcher bias is common in software engineering experiments.Our model shows how these biases lead to a high proportion of incorrect results.Increased statistical power is a key factor to improve the trustworthiness. ContextThe ...

Comments

Information & Contributors

Information

Published In

ICSE '18: Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings

May 2018

231 pages

ISBN:9781450356633

DOI:10.1145/3183440

Conference Chair:
Michel Chaudron
Chalmers University of Technology, University of Gothenburg, Sweden
,
General Chair:
Ivica Crnkovic
Chalmers University of Technology, University of Gothenburg, Sweden
,
Program Chairs:
Marsha Chechik
University of Toronto, Canada
,
Mark Harman
Facebook and University College London, United Kingdom

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Spanish Ministry of Economy and Competitiveness

Conference

ICSE '18

Sponsor:

SIGSOFT
IEEE-CS

ICSE '18: 40th International Conference on Software Engineering

May 27 - June 3, 2018

Gothenburg, Sweden

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
85
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Malavolta IChinnappan KSwanborn SLewis GLago P(2021)Mining the ROS ecosystem for Green Architectural Tactics in Robotics and an Empirical Evaluation2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)10.1109/MSR52588.2021.00042(300-311)Online publication date: May-2021
https://doi.org/10.1109/MSR52588.2021.00042
Malavolta IChinnappan KJasmontas LGupta SSoltany KLo DMariani LMesbah A(2020)Evaluating the impact of caching on the energy consumption and performance of progressive web appsProceedings of the IEEE/ACM 7th International Conference on Mobile Software Engineering and Systems10.1145/3387905.3388593(109-119)Online publication date: 13-Jul-2020
https://dl.acm.org/doi/10.1145/3387905.3388593

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Analyzing software engineering experiments: everything you always wanted to know but were afraid to ask

Analyzing software engineering experiments: everything you always wanted to know but were afraid to ask

Incorrect results in software engineering experiments