Article

DART: directed automated random testing

Authors:

Patrice Godefroid,

Koushik SenAuthors Info & Claims

PLDI '05: Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation

Pages 213 - 223

https://doi.org/10.1145/1065010.1065036

Published: 12 June 2005 Publication History

Abstract

We present a new tool, named DART, for automatically testing software that combines three main techniques: (1) automated extraction of the interface of a program with its external environment using static source-code parsing; (2) automatic generation of a test driver for this interface that performs random testing to simulate the most general environment the program can operate in; and (3) dynamic analysis of how the program behaves under random testing and automatic generation of new test inputs to direct systematically the execution along alternative program paths. Together, these three techniques constitute Directed Automated Random Testing, or DART for short. The main strength of DART is thus that testing can be performed completely automatically on any program that compiles -- there is no need to write any test driver or harness code. During testing, DART detects standard errors such as program crashes, assertion violations, and non-termination. Preliminary experiments to unit test several examples of C programs are very encouraging.

References

[1]

R. Alur, P. Cerny, G. Gupta, P. Madhusudan, W. Nam, and A. Srivastava. Synthesis of Interface Specifications for Java Classes. In Proceedings of POPL'05 (32nd ACM Symposium on Principles of Programming Languages), Long Beach, January 2005.]]

Digital Library

[2]

T. Ball and S. Rajamani. The SLAM Toolkit. In Proceedings of CAV'2001 (13th Conference on Computer Aided Verification), volume 2102 of Lecture Notes in Computer Science, pages 260--264, Paris, July 2001. Springer-Verlag.]]

Digital Library

[3]

D. Beyer, A. J. Chlipala, T. A. Henzinger, R. Jhala, and R. Majumdar. Generating Test from Counterexamples. In Proceedings of ICSE'2004 (26th International Conference on Software Engineering). ACM, May 2004.]]

Digital Library

[4]

D. Bird and C. Munoz. Automatic Generation of Random Self-Checking Test Cases. IBM Systems Journal, 22(3):229--245, 1983.]]

Digital Library

[5]

C. Boyapati, S. Khurshid, and D. Marinov. Korat: Automated testing based on Java predicates. In Proceedings of ISSTA'2002 (International Symposium on Software Testing and Analysis), pages 123--133, 2002.]]

Digital Library

[6]

W. Bush, J. Pincus, and D. Sielaff. A static analyzer for finding dynamic programming errors. Software Practice and Experience, 30(7):775--802, 2000.]]

Digital Library

[7]

K. Claessen and J. Hughes. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. In Proceedings of ICFP'2000, 2000.]]

Digital Library

[8]

C. Colby, P. Godefroid, and L. J. Jagadeesan. Automatically Closing Open Reactive Programs. In Proceedings of PLDI'98 (1998 ACM SIGPLAN Conference on Programming Language Design and Implementation), pages 345--357, Montreal, June 1998. ACM Press.]]

Digital Library

[9]

C. Csallner and Y. Smaragdakis. Check'n Crash: Combining Static Checking and Testing. In Proceedings of ICSE'2005 (27th International Conference on Software Engineering). ACM, May 2005.]]

Digital Library

[10]

J. Edvardsson. A Survey on Automatic Test Data Generation. In Proceedings of the 2nd Conference on Computer Science and Engineering, pages 21--28, Linkoping, October 1999.]]

[11]

J. E. Forrester and B. P. Miller. An Empirical Study of the Robustness of Windows NT Applications Using Random Testing. In Proceedings of the 4th USENIX Windows System Symposium, Seattle, August 2000.]]

Digital Library

[12]

P. Godefroid. Model Checking for Programming Languages using VeriSoft. In Proceedings of POPL'97 (24th ACM Symposium on Principles of Programming Languages), pages 174--186, Paris, January 1997.]]

Digital Library

[13]

P. Godefroid and S. Khurshid. Exploring Very Large State Spaces Using Genetic Algorithms. In Proceedings of TACAS'2002 (8th Conference on Tools and Algorithms for the Construction and Analysis of Systems), Grenoble, April 2002.]]

Digital Library

[14]

S. Gulwani and G. C. Necula. Precise Interprocedural Analysis using Random Interpretation. In To appear in Proceedings of POPL'05 (32nd ACM Symposium on Principles of Programming Languages), Long Beach, January 2005.]]

Digital Library

[15]

N. Gupta, A. P. Mathur, and M. L. Soffa. Generating test data for branch coverage. In Proceedings of the 15th IEEE International Conference on Automated Software Engineering, pages 219--227, September 2000.]]

Digital Library

[16]

S. Hallem, B. Chelf, Y. Xie, and D. Engler. A System and Language for Building System-Specific Static Analyses. In Proceedings of PLDI'02 (2002 ACM SIGPLAN Conference on Programming Language Design and Implementation), pages 69--82, 2002.]]

Digital Library

[17]

R. Hastings and B. Joyce. Purify: Fast Detection of Memory Leaks and Access Errors. In Proceedings of the Usenix Winter 1992 technical Conference, pages 125--138, Berkeley, January 1992.]]

[18]

T. Henzinger, R. Jhala, R. Majumdar, and G. Sutre. Lazy Abstraction. In Proceedings of the 29th ACM Symposium on Principles of Programming Languages, pages 58--70, Portland, January 2002.]]

Digital Library

[19]

S. Johnson. Lint, a C program checker, 1978. Unix Programmer's Manual, AT&T Bell Laboratories.]]

[20]

Junit. web page: http://www.junit.org/.]]

[21]

J. C. King. Symbolic Execution and Program Testing. Communications of the ACM, 19(7):385--394, 1976.]]

Digital Library

[22]

Klocwork. web page: http://klocwork.com/index.asp.]]

[23]

B. Korel. A dynamic Approach of Test Data Generation. In IEEE Conference on Software Maintenance, pages 311--317, San Diego, November 1990.]]

[24]

G. Lowe. An Attack on the Needham-Schroeder Public-Key Authentication Protocol. Information Processing Letters, 1995.]]

Digital Library

[25]

G. Lowe. Breaking and Fixing the Needham-Schroeder Public-Key Protocol using FDR. In Proceedings of TACAS'1996 ((Second International Workshop on Tools and Algorithms for the Construction and Analysis of Systems), volume 1055 of Lecture Notes in Computer Science, pages 147--166. Springer-Verlag, 1996.]]

Digital Library

[26]

lp_solve. web page: http://groups.yahoo.com/group/lp_solve/.]]

[27]

G. J. Myers. The Art of Software Testing. Wiley, 1979.]]

Digital Library

[28]

G. C. Necula, S. McPeak, S. P. Rahul, and W. Weimer. CIL: Intermediate Language and Tools for Analysis and transformation of C Programs. In Proceedings of Conference on compiler Construction, pages 213--228, 2002.]]

Digital Library

[29]

G. C. Necula, S. McPeak, and W. Weimer. CCured: Type-Safe Retrofitting of Legacy Code. In Proceedings of POPL'02 (29th ACM Symposium on Principles of Programming Languages), pages 128--139, Portland, January 2002.]]

Digital Library

[30]

R. Needham and M. Schroeder. Using Encryption for Authentication in Large Networks of Computers. Communications of the ACM, 21(12):993--999, 1978.]]

Digital Library

[31]

The economic impacts of inadequate infrastructure for software testing. National Institute of Standards and technology, Planning Report 02-3, May 2002.]]

[32]

J. Offut and J. Hayes. A Semantic Model of Program Faults. In Proceedings of ISSTA'96 (International Symposium on Software Testing and Analysis), pages 195--200, San Diego, January 1996.]]

Digital Library

[33]

Polyspace. web page: http://www.polyspace.com.]]

[34]

D. Saff and M. D. Ernst. Continuous testing in Eclipse. In Proceedings of 2nd Eclipse Technology Exchange Workshop(eTX), Barcelona, March 2004.]]

[35]

S. D. Stoller. Domain Partitioning for Open Reactive Programs. In Proceedings of ACM SIGSOFT ISSTA'02 (International Symposium on Software Testing and Analysis), 200.]]

Digital Library

[36]

W. Visser, C. Pasareanu, and S. Khurshid. Test Input Generation with Java PathFinder. In Proceedings of ACM SIGSOFT ISSTA'04 (International Symposium on Software Testing and Analysis), Boston, July 2004.]]

Digital Library

[37]

J. Whaley, M. C. Martin, and M. S. Lam. Automatic Extraction of Object-Oriented Component Interfaces. In Proceedings of ACM SIGSOFT ISSTA'02 (International Symposium on Software Testing and Analysis), 2002.]]

Digital Library

[38]

T. Xie, D. Marinov, W. Schulte, and D. Notkin. Symstra: A framework for generating object-oriented unit tests using symbolic execution. In Proceedings of TACAS'05 (11th Conference on Tools and Algorithms for the Construction and Analysis of Systems), volume 3440 of LNCS, pages 365--381. Springer, 2005.]]

Digital Library

Cited By

Verbeek FSefat MFu ZRavindran B(2025)On Extending Incorrectness Logic with Backwards ReasoningProceedings of the ACM on Programming Languages10.1145/37048509:POPL(391-415)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3704850
Wang ZChen JZheng PZhang YZhang WZheng Z(2025)Unity is Strength: Enhancing Precision in Reentrancy Vulnerability Detection of Smart Contract Analysis ToolsIEEE Transactions on Software Engineering10.1109/TSE.2024.342732151:1(1-13)Online publication date: Jan-2025
https://doi.org/10.1109/TSE.2024.3427321
Singh ASarita YMendis CSingh G(2025)ConstraintFlow: A Declarative DSL for Easy Development of DNN CertifiersStatic Analysis10.1007/978-3-031-74776-2_16(407-424)Online publication date: 21-Jan-2025
https://doi.org/10.1007/978-3-031-74776-2_16
Show More Cited By

Index Terms

DART: directed automated random testing

Recommendations

DART: directed automated random testing
Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation

We present a new tool, named DART, for automatically testing software that combines three main techniques: (1) automated extraction of the interface of a program with its external environment using static source-code parsing; (2) automatic generation of ...
The Effectiveness of T-Way Test Data Generation
SAFECOMP '08: Proceedings of the 27th international conference on Computer Safety, Reliability, and Security

This paper reports the results of a study comparing the effectiveness of automatically generated tests constructed using random and <em>t</em>-way combinatorial techniques on safety related industrial code using mutation adequacy criteria. A reference ...
Prioritizing random combinatorial test suites
SAC '17: Proceedings of the Symposium on Applied Computing

The behaviour of a system under test can be influenced by several factors, such as system configurations, user inputs, and so on. It has also been observed that many failures are caused by only a small number of factors. Combinatorial testing aims at ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

PLDI '05: Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation

June 2005

338 pages

ISBN:1595930566

DOI:10.1145/1065010

General Chair:
Vivek Sarkar
IBM Research
,
Program Chair:
Mary Hall
USC/ISI

ACM SIGPLAN Notices Volume 40, Issue 6
Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation
June 2005
325 pages
ISSN:0362-1340
EISSN:1558-1160
DOI:10.1145/1064978
Issue’s Table of Contents

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 June 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

PLDI05

Sponsor:

PLDI05: ACM SIGPLAN Conference on Programming Language Design and Implementation 2005

June 12 - 15, 2005

IL, Chicago, USA

Acceptance Rates

Overall Acceptance Rate 406 of 2,067 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2,160
Total Citations
View Citations
9,083
Total Downloads

Downloads (Last 12 months)412
Downloads (Last 6 weeks)45

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Verbeek FSefat MFu ZRavindran B(2025)On Extending Incorrectness Logic with Backwards ReasoningProceedings of the ACM on Programming Languages10.1145/37048509:POPL(391-415)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3704850
Wang ZChen JZheng PZhang YZhang WZheng Z(2025)Unity is Strength: Enhancing Precision in Reentrancy Vulnerability Detection of Smart Contract Analysis ToolsIEEE Transactions on Software Engineering10.1109/TSE.2024.342732151:1(1-13)Online publication date: Jan-2025
https://doi.org/10.1109/TSE.2024.3427321
Singh ASarita YMendis CSingh G(2025)ConstraintFlow: A Declarative DSL for Easy Development of DNN CertifiersStatic Analysis10.1007/978-3-031-74776-2_16(407-424)Online publication date: 21-Jan-2025
https://doi.org/10.1007/978-3-031-74776-2_16
Christodorescu MGanapathy V(2025)Dynamic AnalysisEncyclopedia of Cryptography, Security and Privacy10.1007/978-3-030-71522-9_836(710-712)Online publication date: 8-Jan-2025
https://doi.org/10.1007/978-3-030-71522-9_836
Liu CMera AKirda EXu MLu LBalzarotti DXu W(2024)CO3Proceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699213(5591-5608)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699213
Qi ZHu JXiao ZYin HBalzarotti DXu W(2024)SYMFITProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3698924(415-431)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3698924
Iyer RArgyraki KCandea GGavrilovska ATerry D(2024)Automatically reasoning about how systems code uses the CPU cacheProceedings of the 18th USENIX Conference on Operating Systems Design and Implementation10.5555/3691938.3691969(581-598)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.5555/3691938.3691969
Singha RMondal RBeckett RKakarla SMillstein TVarghese GVanbever LZhang I(2024)MESSIProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691881(1009-1023)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691881
Distefano DMarescotti MAhs CCela SSampaio GGrigore RHajdu AKapus TMao KSuzanne TFilkov VRay BZhou M(2024)Enhancing Compositional Static Analysis with Dynamic AnalysisProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695599(2121-2129)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695599
Wang ZLiu KLi GJin ZFilkov VRay BZhou M(2024)HITS: High-coverage LLM-based Unit Test Generation via Method SlicingProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695501(1258-1268)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695501
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten