research-article

Tests from traces: automated unit test extraction for R

Authors:

Filip Křikava,

Jan VitekAuthors Info & Claims

ISSTA 2018: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis

Pages 232 - 241

https://doi.org/10.1145/3213846.3213863

Published: 12 July 2018 Publication History

Abstract

Unit tests are labor-intensive to write and maintain. This paper looks into how well unit tests for a target software package can be extracted from the execution traces of client code. Our objective is to reduce the effort involved in creating test suites while minimizing the number and size of individual tests, and maximizing coverage. To evaluate the viability of our approach, we select a challenging target for automated test extraction, namely R, a programming language that is popular for data science applications. The challenges presented by R are its extreme dynamism, coerciveness, and lack of types. This combination decrease the efficacy of traditional test extraction techniques. We present Genthat, a tool developed over the last couple of years to non-invasively record execution traces of R programs and extract unit tests from those traces. We have carried out an evaluation on 1,545 packages comprising 1.7M lines of R code. The tests extracted by Genthat improved code coverage from the original rather low value of 267,496 lines to 700,918 lines. The running time of the generated tests is 1.9 times faster than the code they came from

References

[1]

Chandrasekhar Boyapati, Sarfraz Khurshid, and Darko Marinov. 2002. Korat: Automated Testing Based on Java Predicates. In International Symposium on Software Testing and Analysis (ISSTA).

Digital Library

[2]

Koen Claessen and John Hughes. 2000. QuickCheck: a lightweight tool for random testing of Haskell programs. In International Conference on Functional Programming (ICFP).

Digital Library

[3]

Christoph Csallner and Yannis Smaragdakis. 2004. JCrasher: An Automatic Robustness Tester for Java. Softw. Pract. Exper. 34, 11 (2004).

Digital Library

[4]

1002/spe.602

[5]

Sebastian G. Elbaum, Hui Nee Chin, Matthew B. Dwyer, and Jonathan Dokulil. 2006. Carving differential unit test cases from system test cases. In International Symposium on Foundations of Software Engineering (FSE). 1181775.1181806

Digital Library

[6]

Michael Ernst, Sai Zhang, David Saff, and Yingyi Bu. 2011. Combined Static and Dynamic Automated Test Generation. In International Symposium on Software Testing and Analysis (ISSTA).

Digital Library

[7]

Hojun Jaygarl, Sunghun Kim, Tao Xie, and Carl Chang. 2010. OCAT: object capture-based automated testing. In nternational Symposium on Software Testing and Analysis (ISSTA).

Digital Library

[8]

Shrinivas Joshi and Alessandro Orso. 2007. SCARPE: A Technique and Tool for Selective Capture and Replay of Program Executions. In International Conference on Software Maintenance (ICSM).

[9]

Uwe Ligges. {n. d.}. 20 Years of CRAN (Video on Channel9. In Keynote at UseR!

[10]

Floréal Morandat, Brandon Hill, Leo Osvald, and Jan Vitek. 2012. Evaluating the Design of the R Language: Objects and Functions for Data Analysis. In European Conference on Object-Oriented Programming (ECOOP). 978-3-642-31057-7_6

Digital Library

[11]

R Core Team. 2017. R Internals 3.4.3.

[12]

R Core Team. 2017. R Language 3.4.3. 90019-7

[13]

R Core Team. 2017. Writing R Extensions 3.4.3. cropsci1998.0011183X003800020005x

[14]

Gregor Richards, Andreas Gal, Brendan Eich, and Jan Vitek. 2011. Automated Construction of JavaScript Benchmarks. In Conference on Object-Oriented Programming Systems, Languages and Applications (OOPSLA). 1145/2048066.2048119

Digital Library

[15]

Thomas Rooney. 2015. Automated Test Generation through Runtime Execution Trace Analysis. Master’s thesis. Imparial College London.

[16]

David Saff, Shay Artzi, Jeff H. Perkins, and Michael D. Ernst. 2005. Automatic test factoring for Java. In International Conference on Automated Software Engineering (ASE).

Digital Library

[17]

Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: A Concolic Unit Testing Engine for C. In European Software Engineering Conference and International Symposium on Foundations of Software Engineering (ESEC/FSE).

Digital Library

[18]

O. Tange. 2011. GNU Parallel–The Command-Line Power Tool. ;login: The USENIX Magazine 42, 47 (2011).

[19]

O. Tange. 2018. GNU Parallel 2018.

Digital Library

[20]

Suresh Thummalapenta, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. 2009. MSeqGen: Object-oriented Unit-test Generation via Mining Source Code. In European Software Engineering Conference and Symposium on The Foundations of Software Engineering (ESEC/FSE). 1595696.1595725

Digital Library

[21]

Hadley Wickham. 2014. Advanced R (1 edition ed.). Chapman and Hall/CRC.

[22]

Shin Yoo and Mark Harman. 2012. Regression testing minimization, selection and prioritization: a survey. Software Testing, Verification and Reliability 22, 2 (2012), 67–120.

Cited By

Gambi AGouni HBerreiter DTymofyeyev VFazzini M(2023)Action-Based Test Carving for Android Apps2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW58534.2023.00032(107-116)Online publication date: May-2023
https://doi.org/10.1109/ICSTW58534.2023.00032
Gambi ALi MFazzini M(2023)Artisan: An Action-Based Test Carving Tool for Android Apps2023 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58846.2023.00076(580-585)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICSME58846.2023.00076
Schittekat IAbdi MDemeyer S(2022)Can We Increase the Test-coverage in Libraries using Dependent Projects’ Test-suites?Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering10.1145/3530019.3535309(294-298)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3530019.3535309
Show More Cited By

Index Terms

Tests from traces: automated unit test extraction for R
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Two-dimensional Search Space for Extracting Broadside Tests from Functional Test Sequences
Testing for delay faults after chip manufacturing is critical to correct chip operation. Tests for delay faults are applied using scan chains that provide access to internal memory elements. As a result, a circuit may operate under non-functional ...
Unit Tests Reloaded: Parameterized Unit Testing with Symbolic Execution

Unit tests are popular. However, it's an art to write them in a way that specifies a program's behavior well, and it's laborious to write enough of them to have confidence in an implementation's correctness. Symbolic execution techniques can help ...
Extracting Inline Tests from Unit Tests
ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

We recently proposed inline tests for validating individual program statements; they allow developers to provide test inputs, expected outputs, and test oracles immediately after a target statement. But, existing code can have many target statements. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2018: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis

July 2018

379 pages

ISBN:9781450356992

DOI:10.1145/3213846

General Chair:
Frank Tip
Northeastern University, USA
,
Program Chair:
Eric Bodden
University of Paderborn, Germany / Fraunhofer IEM, Germany

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 July 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Funding Sources

H2020 European Research Council

Conference

ISSTA '18

Sponsor:

SIGSOFT

ISSTA '18: International Symposium on Software Testing and Analysis

July 16 - 21, 2018

Amsterdam, Netherlands

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
272
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)2

Reflects downloads up to 28 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gambi AGouni HBerreiter DTymofyeyev VFazzini M(2023)Action-Based Test Carving for Android Apps2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW58534.2023.00032(107-116)Online publication date: May-2023
https://doi.org/10.1109/ICSTW58534.2023.00032
Gambi ALi MFazzini M(2023)Artisan: An Action-Based Test Carving Tool for Android Apps2023 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58846.2023.00076(580-585)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICSME58846.2023.00076
Schittekat IAbdi MDemeyer S(2022)Can We Increase the Test-coverage in Libraries using Dependent Projects’ Test-suites?Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering10.1145/3530019.3535309(294-298)Online publication date: 13-Jun-2022
https://dl.acm.org/doi/10.1145/3530019.3535309
Tiwari DZhang LMonperrus MBaudry B(2022)Production Monitoring to Improve Test SuitesIEEE Transactions on Reliability10.1109/TR.2021.310131871:3(1381-1397)Online publication date: Oct-2022
https://doi.org/10.1109/TR.2021.3101318
Abdi MDemeyer S(2022)Test Transplantation through Dynamic Test Slicing2022 IEEE 22nd International Working Conference on Source Code Analysis and Manipulation (SCAM)10.1109/SCAM55253.2022.00009(35-39)Online publication date: Oct-2022
https://doi.org/10.1109/SCAM55253.2022.00009
Saieva AKaiser G(2022)Update with careJournal of Systems and Software10.1016/j.jss.2022.111381191:COnline publication date: 1-Sep-2022
https://dl.acm.org/doi/10.1016/j.jss.2022.111381
Goel ADonat-Bouillud PKřikava FKirsch CVitek J(2021)What we eval in the shadows: a large-scale study of eval in R programsProceedings of the ACM on Programming Languages10.1145/34855025:OOPSLA(1-23)Online publication date: 15-Oct-2021
https://dl.acm.org/doi/10.1145/3485502
Bluemke IMalanowska A(2021)Software Testing Effort Estimation and Related ProblemsACM Computing Surveys10.1145/344269454:3(1-38)Online publication date: 17-Apr-2021
https://dl.acm.org/doi/10.1145/3442694
Vidoni M(2021)Evaluating Unit Testing Practices in R PackagesProceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00136(1523-1534)Online publication date: 22-May-2021
https://dl.acm.org/doi/10.1109/ICSE43902.2021.00136
Goel AVitek J(2019)On the design, implementation, and use of laziness in RProceedings of the ACM on Programming Languages10.1145/33605793:OOPSLA(1-27)Online publication date: 10-Oct-2019
https://dl.acm.org/doi/10.1145/3360579
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten