research-article

A large-scale empirical comparison of static and dynamic test case prioritization techniques

Authors:

Denys PoshyvanykAuthors Info & Claims

FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

Pages 559 - 570

https://doi.org/10.1145/2950290.2950344

Published: 01 November 2016 Publication History

Abstract

The large body of existing research in Test Case Prioritization (TCP) techniques, can be broadly classified into two categories: dynamic techniques (that rely on run-time execution information) and static techniques (that operate directly on source and test code). Absent from this current body of work is a comprehensive study aimed at understanding and evaluating the static approaches and comparing them to dynamic approaches on a large set of projects.

In this work, we perform the first extensive study aimed at empirically evaluating four static TCP techniques comparing them with state-of-research dynamic TCP techniques at different test-case granularities (e.g., method and class-level) in terms of effectiveness, efficiency and similarity of faults detected. This study was performed on 30 real-word Java programs encompassing 431 KLoC. In terms of effectiveness, we find that the static call-graph-based technique outperforms the other static techniques at test-class level, but the topic-model-based technique performs better at test-method level. In terms of efficiency, the static call-graph-based technique is also the most efficient when compared to other static techniques. When examining the similarity of faults detected for the four static techniques compared to the four dynamic ones, we find that on average, the faults uncovered by these two groups of techniques are quite dissimilar, with the top 10% of test cases agreeing on only 25% - 30% of detected faults. This prompts further research into the severity/importance of faults uncovered by these techniques, and into the potential for combining static and dynamic information for more effective approaches.

References

[1]

Github https://github.com.

[2]

P. Ammann, M. E. Delamaro, and J. Offutt. Establishing theoretical minimal sets of mutants. In Proceedings of the 2014 IEEE International Conference on Software Testing, Verification, and Validation, ICST ’14, pages 21–30, Washington, DC, USA, 2014. IEEE Computer Society.

Digital Library

[3]

J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments? In ICSE, pages 402–411, 2005.

Digital Library

[4]

J. H. Andrews, L. C. Briand, Y. Labiche, and A. S. Namin. Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Trans. Softw. Eng., 32(8):608–624, Aug. 2006.

Digital Library

[5]

M. J. Arafeen and H. Do. Test case prioritization using requirements based clustering. In Proc. ICST, pages 312–321, 2013.

Digital Library

[6]

ASM. http://asm.ow2.org/.

[7]

C. Catal and D. Mishra. Test case prioritization: a systematic mapping study. Software Quality Journal, 21(3):445–478, 2013.

Digital Library

[8]

T. H. Chen. Studying software quality using topic models. 2013.

[9]

T. Y. Chen, H. Leung, and I. K. Mak. Adaptive random testing. In Asian Computing Science Conference, pages 320–329, 2004.

Digital Library

[10]

J. Czerwonka, R. Das, N. Nagappan, A. Tarvo, and A. Teterev. Crane: Failure prediction, change analysis and test prioritization in practice – experiences from windows. In Proceedings of the 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation, ICST ’11, pages 357–366, 2011.

Digital Library

[11]

H. Do, S. Mirarab, L. Tahvildari, and G. Rothermel. An empirical study of the effect of time constraints on the cost-benefits of regression testing. In FSE, pages 71–82, 2008.

Digital Library

[12]

H. Do and G. Rothermel. A controlled experiment assessing test case prioritization techniques via mutation faults. In ICSM, pages 411–420, 2005.

Digital Library

[13]

H. Do and G. Rothermel. An empirical study of regression testing techniques incorporating context and lifecycle factors and improved cost-benefit models. In Proc. FSE, pages 141–151, 2006.

Digital Library

[14]

H. Do and G. Rothermel. On the use of mutation faults in empirical assessments of test case prioritization techniques. TSE, 32(9):733–752, 2006.

Digital Library

[15]

H. Do, G. Rothermel, and A. Kinneer. Empirical studies of test case prioritization in a JUnit testing environment. In ISSRE, pages 113–124, 2004.

Digital Library

[16]

S. Elbaum, P. Kallakuri, A. Malishevsky, G. Rothermel, and S. Kanduri. Understanding the effects of changes on the cost-effectiveness of regression testing techniques. Software testing, verification and reliability, 13(2):65–83, 2003.

[17]

S. Elbaum, A. Malishevsky, and G. Rothermel. Prioritizing test cases for regression testing. In ISSTA, pages 102–112, 2000.

Digital Library

[18]

S. Elbaum, A. Malishevsky, and G. Rothermel. Incorporating varying test costs and fault severities into test case prioritization. In ICSE, pages 329–338, 2001.

Digital Library

[19]

S. Elbaum, A. G. Malishevsky, and G. Rothermel. Test case prioritization: A family of empirical studies. Software Engineering, IEEE Transactions on, 28(2):159–182, 2002.

Digital Library

[20]

S. Elbaum, G. Rothermel, S. Kanduri, and A. G. Malishevsky. Selecting a cost-effective test case prioritization technique. Software Quality Journal, 12(3):185–210, 2004.

Digital Library

[21]

M. G. Epitropakis, S. Yoo, M. Harman, and E. K. Burke. Empirical evaluation of pareto efficient multi-objective regression test case prioritisation. In Proceedings of the 2015 International Symposium on Software Testing and Analysis, pages 234–245. ACM, 2015.

Digital Library

[22]

D. Hao, L. Zhang, L. Zhang, G. Rothermel, and H. Mei. A unified test case prioritization approach. TOSEM, 10:1–31, 2014.

Digital Library

[23]

C. Henard, M. Papadakis, M. Harman, Y. Jia, and Y. L. Traon. Comparing white-box and black-box test prioritization. In Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, page to appear, New York, NY, USA, 2016. ACM.

Digital Library

[24]

C. Henard, M. Papadakis, G. Perrouin, J. Klein, P. Heymans, and Y. L. Traon. Bypassing the combinatorial explosion: Using similarity to generate and prioritize t-wise test configurations for software product lines. IEEE Transactions on Software Engineering, 40(7):650–670, July 2014.

Digital Library

[25]

M. M. Islam, A. Marchetto, A. Susi, and G. Scanniello. A multi-objective technique to prioritize test cases based on latent semantic indexing. In 2012 16th European Conference on Software Maintenance and Reengineering (CSMR), pages 21–30, 2012.

Digital Library

[26]

JDT. http://www.eclipse.org/jdt/.

[27]

B. Jiang and W. Chan. Bypassing code coverage approximation limitations via effective input-based randomized test case prioritization. In Proc. COMPSAC, pages 190–199, 2013.

Digital Library

[28]

B. Jiang, Z. Zhang, W. K. Chan, and T. H. Tse. Adaptive random test case prioritization. In ASE, pages 257–266, 2009.

Digital Library

[29]

R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser. Are mutants a valid substitute for real faults in software testing? In Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014, pages 654–665, New York, NY, USA, 2014. ACM.

Digital Library

[30]

G. M. Kapfhammer and M. L. Soffa. Using coverage effectiveness to evaluate test suite prioritizations. In Proceedings of the 1st ACM international workshop on Empirical assessment of software engineering languages and technologies: held in conjunction with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE) 2007, pages 19–20. ACM, 2007.

Digital Library

[31]

J. Kasurinen, O. Taipale, and K. Smolander. Test case selection and prioritization: Risk-based or design-based? In Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, page 10. ACM, 2010.

Digital Library

[32]

R. Kavitha and N. Sureshkumar. Test case prioritization for regression testing based on severity of fault. International Journal on Computer Science and Engineering, 2(5):1462–1466, 2010.

[33]

B. Korel, L. H. Tahat, and M. Harman. Test prioritization using system models. In Software Maintenance, 2005. ICSM’05. Proceedings of the 21st IEEE International Conference on, pages 559–568, Sept 2005.

Digital Library

[34]

LDA. https://cran.r-project.org/web/packages/lda/.

[35]

Y. Ledru, A. Petrenko, S. Boroday, and N. Mandran. Prioritizing test cases with string distances. Automated Software Engineering, 19(1):65–95, 2012.

Digital Library

[36]

Z. Li, M. Harman, and R. Hierons. Search algorithms for regression test case prioritisation. TSE, 33(4):225–237, 2007.

Digital Library

[37]

Y. Lu, Y. Lou, S. Cheng, L. Zhang, D. Hao, Y. Zhou, and L. Zhang. How does regression test prioritization perform in real-world software evolution? In In Proc. ICSE’16, page to appear, 2016.

Digital Library

[38]

Q. Luo, K. Moran, and D. Poshyvanyk. Fse’16 online appendix. http://www.cs.wm.edu/semeru/data/FSE16-TCPSTUDY/.

[39]

Mallet. http://mallet.cs.umass.edu/.

[40]

H. Mei, D. Hao, L. Zhang, L. Zhang, J. Zhou, and G. Rothermel. A static approach to prioritizing junit test cases. IEEE Trans. Softw. Eng., 38(6):1258–1275, Nov. 2012.

Digital Library

[41]

C. D. Nguyen, A. Marchetto, and P. Tonella. Test case prioritization for audit testing of evolving web services using information retrieval techniques. In Proc. ICWS, pages 636–643, 2011.

Digital Library

[42]

PIT. http://pitest.org/.

[43]

X. Qu, M. B. Cohen, and G. Rothermel. Configuration-aware regression testing: an empirical study of sampling and prioritization. In Proceedings of the 2008 international symposium on Software testing and analysis, pages 75–86. ACM, 2008.

Digital Library

[44]

G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Test case prioritization: an empirical study. In ICSM, pages 179–188, 1999.

Digital Library

[45]

G. Rothermel, R. J. Untch, C. Chu, and M. J. Harrold. Prioritizing test cases for regression testing. TSE, 27(10):929–948, 2001.

Digital Library

[46]

R. K. Saha, L. Zhang, S. Khurshid, and D. E. Perry. An information retrieval approach for regression test prioritization based on program changes. In Proc. ICSE, page to appear, 2015.

Digital Library

[47]

A. M. Smith and G. M. Kapfhammer. An empirical study of incorporating cost into test suite reduction and prioritization. In Proceedings of the 2009 ACM symposium on Applied Computing, pages 461–467. ACM, 2009.

Digital Library

[48]

A. Srivastava and J. Thiagarajan. Effectively prioritizing tests in development environment. In Proceedings of the 2002 ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA ’02, pages 97–106, 2002.

Digital Library

[49]

T. H. test. https://en.wikipedia.org/wiki/Tukey

[50]

S. W. Thomas, H. Hemmati, A. E. Hassan, and D. Blostein. Static test case prioritization using topic models. EMSE, 19(1):182–212, 2014.

Digital Library

[51]

P. Tonella, P. Avesani, and A. Susi. Using the case-based ranking methodology for test case prioritization. In Proc. ICSM, pages 123–133, 2006.

Digital Library

[52]

S. Varun Kumar and M. Kumar. Test case prioritization using fault severity. IJCST, 1(1), 2010.

[53]

WALA. https://github.com/wala/WALA.

[54]

K. R. Walcott, M. L. Soffa, G. M. Kapfhammer, and R. S. Roos. Time aware test suite prioritization. In ISSTA, pages 1–11, 2006.

Digital Library

[55]

S. Wang, D. Buchmann, S. Ali, A. Gotlieb, D. Pradhan, and M. Liaaen. Multi-objective test prioritization in software product line testing: an industrial case study. In Proceedings of the 18th International Software Product Line Conference-Volume 1, pages 32–41. ACM, 2014.

Digital Library

[56]

D. Xu and J. Ding. Prioritizing state-based aspect tests. In Proc. ICST, pages 265–274, 2010.

Digital Library

[57]

S. Yoo and M. Harman. Regression testing minimization, selection and prioritization: a survey. STVR, 22(2):67–120, 2012.

Digital Library

[58]

D. You, Z. Chen, B. Xu, B. Luo, and C. Zhang. An empirical study on the effectiveness of time-aware test case prioritization techniques. In Proceedings of the 2011 ACM Symposium on Applied Computing, pages 1451–1456. ACM, 2011.

Digital Library

[59]

L. Zhang, D. Hao, L. Zhang, G. Rothermel, and H. Mei. Bridging the gap between the total and additional test-case prioritization strategies. In ICSE, pages 192–201, 2013.

Digital Library

[60]

L. Zhang, M. Kim, and S. Khurshid. Localizing failure-inducing program edits based on spectrum information. In Software Maintenance (ICSM), 2011 27th IEEE International Conference on, pages 23–32. IEEE, 2011.

Digital Library

[61]

L. Zhang, D. Marinov, and S. Khurshid. Faster mutation testing inspired by test prioritization and reduction. In Proceedings of the 2013 International Symposium on Software Testing and Analysis, ISSTA 2013, pages 235–245, New York, NY, USA, 2013. ACM.

Digital Library

[62]

L. Zhang, J. Zhou, D. Hao, L. Zhang, and H. Mei. Prioritizing JUnit test cases in absence of coverage information. In ICSM, pages 19–28, 2009.

Digital Library

Cited By

Li CBaz AShi AFilkov VRay BZhou M(2024)Reducing Test Runtime by Transforming Test FixturesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695541(1757-1769)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695541
Baz AHuang MShi AFilkov VRay BZhou M(2024)Prioritizing Tests for Improved RuntimeProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695298(2273-2278)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695298
Laaber CYue TAli S(2024)Evaluating Search-Based Software Microbenchmark PrioritizationIEEE Transactions on Software Engineering10.1109/TSE.2024.338083650:7(1687-1703)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3380836
Show More Cited By

Index Terms

A large-scale empirical comparison of static and dynamic test case prioritization techniques
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques

Regression testing is an important activity in the software life cycle, but it can also be very expensive. To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure,...
A Static Approach to Prioritizing JUnit Test Cases

Test case prioritization is used in regression testing to schedule the execution order of test cases so as to expose faults earlier in testing. Over the past few years, many test case prioritization techniques have been proposed in the literature. Most ...
Static test case prioritization using topic models

Software development teams use test suites to test changes to their source code. In many situations, the test suites are so large that executing every test for every source code change is infeasible, due to time and resource constraints. Development ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

November 2016

1156 pages

ISBN:9781450342186

DOI:10.1145/2950290

General Chair:
Thomas Zimmermann
Microsoft Research, USA
,
Program Chairs:
Jane Cleland-Huang
University of Notre Dame, USA
,
Zhendong Su
University of California at Davis, USA

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

FSE'16

Sponsor:

SIGSOFT

FSE'16: 24nd ACM SIGSOFT International Symposium on the Foundations of Software Engineering

November 13 - 18, 2016

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 17 of 128 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

44
Total Citations
View Citations
629
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)5

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li CBaz AShi AFilkov VRay BZhou M(2024)Reducing Test Runtime by Transforming Test FixturesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695541(1757-1769)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695541
Baz AHuang MShi AFilkov VRay BZhou M(2024)Prioritizing Tests for Improved RuntimeProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695298(2273-2278)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695298
Laaber CYue TAli S(2024)Evaluating Search-Based Software Microbenchmark PrioritizationIEEE Transactions on Software Engineering10.1109/TSE.2024.338083650:7(1687-1703)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3380836
Baz AGligoric MShi A(2024)Impact of JVM Configurations on Test Runtime2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58944.2024.00032(249-261)Online publication date: 6-Oct-2024
https://doi.org/10.1109/ICSME58944.2024.00032
Shu THe ZYin XDing ZZhou M(2024)Model-based diversity-driven learn-to-rank test case prioritizationExpert Systems with Applications10.1016/j.eswa.2024.124768255(124768)Online publication date: Dec-2024
https://doi.org/10.1016/j.eswa.2024.124768
Fallahzadeh ERigby PAdams B(2024)Contrasting test selection, prioritization, and batch testing at scaleEmpirical Software Engineering10.1007/s10664-024-10589-830:1Online publication date: 29-Nov-2024
https://doi.org/10.1007/s10664-024-10589-8
Li YDang XMa LKlein JBissyandé T(2024)Prioritizing test cases for deep learning-based video classifiersEmpirical Software Engineering10.1007/s10664-024-10520-129:5Online publication date: 22-Jul-2024
https://doi.org/10.1007/s10664-024-10520-1
Chen ZChen JWang WZhou JWang MChen XZhou SWang J(2023)Exploring Better Black-Box Test Case Prioritization via Log AnalysisACM Transactions on Software Engineering and Methodology10.1145/356993232:3(1-32)Online publication date: 26-Apr-2023
https://dl.acm.org/doi/10.1145/3569932
Laiq MAli NBörstler JEngström E(2023)A data-driven approach for understanding invalid bug reports: An industrial case studyInformation and Software Technology10.1016/j.infsof.2023.107305164(107305)Online publication date: Dec-2023
https://doi.org/10.1016/j.infsof.2023.107305
Roman AMnich MHryszko J(2023)Empirical Verification of TQED - A New Test Design Heuristic TechniqueTesting Software and Systems10.1007/978-3-031-43240-8_6(80-96)Online publication date: 19-Sep-2023
https://doi.org/10.1007/978-3-031-43240-8_6
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents