skip to main content
10.1109/SBST.2019.00011acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article

Towards automated test case generation maturity

Published: 27 May 2019 Publication History

Abstract

This short paper reports our observations after six editions of the JUnitContest that benchmarks automated unit test generation tools for Java programs. We discuss our experience and depict the current state-of-the-art and identify potential future research directions. We advocate the use of benchmark as a standard practice to enhance maturity and foster adoption by the industry of automated test case generation tools.

References

[1]
D. Gelperin and B. Hetzel, "The growth of software testing," Commun. ACM, vol. 31, no. 6, pp. 687--695, Jun. 1988. {Online}. Available
[2]
N. Alshahwan, X. Gao, M. Harman, Y. Jia, K. Mao, A. Mols, T. Tei, and I. Zorin, "Deploying Search Based Software Engineering with Sapienz at Facebook," in Search-Based Software Engineering. SSBSE 2018., ser. LNCS, vol. 11036. Springer, 2018.
[3]
V. Garousi and M. V. Mäntylä, "When and what to automate in software testing? a multi-vocal literature review," Information & Software Technology, vol. 76, pp. 92--117, 2016.
[4]
J. L. Henning, "Spec cpu2000: measuring cpu performance in the new millennium," Computer, vol. 33, no. 7, pp. 28--35, July 2000.
[5]
A. Rorissa, "Image retrieval: Benchmarking visual information indexing and retrieval systems," Bulletin of the American Society for Information Science and Technology, vol. 33, no. 3, pp. 15--17, 2007. {Online}. Available:
[6]
S. E. Sim, R. C. Holt, and S. Easterbrook, "On using a benchmark to evaluate c++ extractors," in Proceedings 10th International Workshop on Program Comprehension, June 2002, pp. 114--123.
[7]
S. E. Sim, S. Easterbrook, and R. C. Holt, "Using benchmarking to advance research: A challenge to software engineering," in Proceedings of the 25th International Conference on Software Engineering, ser. ICSE '03. Washington, DC, USA: IEEE Computer Society, 2003, pp. 74--83. {Online}. Available: http://dl.acm.org/citation.cfm?id=776816.776826
[8]
R. Just, D. Jalali, and M. D. Ernst, "Defects4J: a database of existing faults to enable controlled testing studies for Java programs," in Proceedings of the 2014 International Symposium on Software Testing and Analysis - ISSTA 2014. ACM Press, 2014, pp. 437--440.
[9]
G. Fraser and A. Arcuri, "A Large-Scale Evaluation of Automated Unit Test Generation Using EvoSuite," ACM Transactions on Software Engineering and Methodology, vol. 24, no. 2, pp. 1--42, dec 2014.
[10]
M. D'Ambros, M. Lanza, and R. Robbes, "Evaluating defect prediction approaches: a benchmark and an extensive comparison," Empirical Software Engineering, vol. 17, no. 4, pp. 531--577, Aug 2012. {Online}. Available
[11]
U. Rueda Molina, F. Kifetew, and A. Panichella, "Java unit testing tool competition: Sixth round," in Proceedings of the 11th International Workshop on Search-Based Software Testing, ser. SBST '18. New York, NY, USA: ACM, 2018, pp. 22--29. {Online}. Available
[12]
O. Pastor and V. Pelechano, The Conceptual Model Is The Code. Why Not? Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 153--159.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SBST '19: Proceedings of the 12th International Workshop on Search-Based Software Testing
May 2019
41 pages

Sponsors

Publisher

IEEE Press

Publication History

Published: 27 May 2019

Check for updates

Author Tags

  1. automation
  2. benchmarking
  3. challenges
  4. industry adoption
  5. maturation
  6. quality

Qualifiers

  • Research-article

Conference

ICSE '19
Sponsor:

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 74
    Total Downloads
  • Downloads (Last 12 months)4
  • Downloads (Last 6 weeks)0
Reflects downloads up to 06 Jan 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media