article

DART: directed automated random testing

Authors:

Patrice Godefroid,

Koushik SenAuthors Info & Claims

ACM SIGPLAN Notices, Volume 40, Issue 6

Pages 213 - 223

https://doi.org/10.1145/1064978.1065036

Published: 12 June 2005 Publication History

Abstract

We present a new tool, named DART, for automatically testing software that combines three main techniques: (1) automated extraction of the interface of a program with its external environment using static source-code parsing; (2) automatic generation of a test driver for this interface that performs random testing to simulate the most general environment the program can operate in; and (3) dynamic analysis of how the program behaves under random testing and automatic generation of new test inputs to direct systematically the execution along alternative program paths. Together, these three techniques constitute Directed Automated Random Testing, or DART for short. The main strength of DART is thus that testing can be performed completely automatically on any program that compiles -- there is no need to write any test driver or harness code. During testing, DART detects standard errors such as program crashes, assertion violations, and non-termination. Preliminary experiments to unit test several examples of C programs are very encouraging.

References

[1]

R. Alur, P. Cerny, G. Gupta, P. Madhusudan, W. Nam, and A. Srivastava. Synthesis of Interface Specifications for Java Classes. In Proceedings of POPL'05 (32nd ACM Symposium on Principles of Programming Languages), Long Beach, January 2005.]]

Digital Library

[2]

T. Ball and S. Rajamani. The SLAM Toolkit. In Proceedings of CAV'2001 (13th Conference on Computer Aided Verification), volume 2102 of Lecture Notes in Computer Science, pages 260--264, Paris, July 2001. Springer-Verlag.]]

Digital Library

[3]

D. Beyer, A. J. Chlipala, T. A. Henzinger, R. Jhala, and R. Majumdar. Generating Test from Counterexamples. In Proceedings of ICSE'2004 (26th International Conference on Software Engineering). ACM, May 2004.]]

Digital Library

[4]

D. Bird and C. Munoz. Automatic Generation of Random Self-Checking Test Cases. IBM Systems Journal, 22(3):229--245, 1983.]]

Digital Library

[5]

C. Boyapati, S. Khurshid, and D. Marinov. Korat: Automated testing based on Java predicates. In Proceedings of ISSTA'2002 (International Symposium on Software Testing and Analysis), pages 123--133, 2002.]]

Digital Library

[6]

W. Bush, J. Pincus, and D. Sielaff. A static analyzer for finding dynamic programming errors. Software Practice and Experience, 30(7):775--802, 2000.]]

Digital Library

[7]

K. Claessen and J. Hughes. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. In Proceedings of ICFP'2000, 2000.]]

Digital Library

[8]

C. Colby, P. Godefroid, and L. J. Jagadeesan. Automatically Closing Open Reactive Programs. In Proceedings of PLDI'98 (1998 ACM SIGPLAN Conference on Programming Language Design and Implementation), pages 345--357, Montreal, June 1998. ACM Press.]]

Digital Library

[9]

C. Csallner and Y. Smaragdakis. Check'n Crash: Combining Static Checking and Testing. In Proceedings of ICSE'2005 (27th International Conference on Software Engineering). ACM, May 2005.]]

Digital Library

[10]

J. Edvardsson. A Survey on Automatic Test Data Generation. In Proceedings of the 2nd Conference on Computer Science and Engineering, pages 21--28, Linkoping, October 1999.]]

[11]

J. E. Forrester and B. P. Miller. An Empirical Study of the Robustness of Windows NT Applications Using Random Testing. In Proceedings of the 4th USENIX Windows System Symposium, Seattle, August 2000.]]

Digital Library

[12]

P. Godefroid. Model Checking for Programming Languages using VeriSoft. In Proceedings of POPL'97 (24th ACM Symposium on Principles of Programming Languages), pages 174--186, Paris, January 1997.]]

Digital Library

[13]

P. Godefroid and S. Khurshid. Exploring Very Large State Spaces Using Genetic Algorithms. In Proceedings of TACAS'2002 (8th Conference on Tools and Algorithms for the Construction and Analysis of Systems), Grenoble, April 2002.]]

Digital Library

[14]

S. Gulwani and G. C. Necula. Precise Interprocedural Analysis using Random Interpretation. In To appear in Proceedings of POPL'05 (32nd ACM Symposium on Principles of Programming Languages), Long Beach, January 2005.]]

Digital Library

[15]

N. Gupta, A. P. Mathur, and M. L. Soffa. Generating test data for branch coverage. In Proceedings of the 15th IEEE International Conference on Automated Software Engineering, pages 219--227, September 2000.]]

Digital Library

[16]

S. Hallem, B. Chelf, Y. Xie, and D. Engler. A System and Language for Building System-Specific Static Analyses. In Proceedings of PLDI'02 (2002 ACM SIGPLAN Conference on Programming Language Design and Implementation), pages 69--82, 2002.]]

Digital Library

[17]

R. Hastings and B. Joyce. Purify: Fast Detection of Memory Leaks and Access Errors. In Proceedings of the Usenix Winter 1992 technical Conference, pages 125--138, Berkeley, January 1992.]]

[18]

T. Henzinger, R. Jhala, R. Majumdar, and G. Sutre. Lazy Abstraction. In Proceedings of the 29th ACM Symposium on Principles of Programming Languages, pages 58--70, Portland, January 2002.]]

Digital Library

[19]

S. Johnson. Lint, a C program checker, 1978. Unix Programmer's Manual, AT&T Bell Laboratories.]]

[20]

Junit. web page: http://www.junit.org/.]]

[21]

J. C. King. Symbolic Execution and Program Testing. Communications of the ACM, 19(7):385--394, 1976.]]

Digital Library

[22]

Klocwork. web page: http://klocwork.com/index.asp.]]

[23]

B. Korel. A dynamic Approach of Test Data Generation. In IEEE Conference on Software Maintenance, pages 311--317, San Diego, November 1990.]]

[24]

G. Lowe. An Attack on the Needham-Schroeder Public-Key Authentication Protocol. Information Processing Letters, 1995.]]

Digital Library

[25]

G. Lowe. Breaking and Fixing the Needham-Schroeder Public-Key Protocol using FDR. In Proceedings of TACAS'1996 ((Second International Workshop on Tools and Algorithms for the Construction and Analysis of Systems), volume 1055 of Lecture Notes in Computer Science, pages 147--166. Springer-Verlag, 1996.]]

Digital Library

[26]

lp_solve. web page: http://groups.yahoo.com/group/lp_solve/.]]

[27]

G. J. Myers. The Art of Software Testing. Wiley, 1979.]]

Digital Library

[28]

G. C. Necula, S. McPeak, S. P. Rahul, and W. Weimer. CIL: Intermediate Language and Tools for Analysis and transformation of C Programs. In Proceedings of Conference on compiler Construction, pages 213--228, 2002.]]

Digital Library

[29]

G. C. Necula, S. McPeak, and W. Weimer. CCured: Type-Safe Retrofitting of Legacy Code. In Proceedings of POPL'02 (29th ACM Symposium on Principles of Programming Languages), pages 128--139, Portland, January 2002.]]

Digital Library

[30]

R. Needham and M. Schroeder. Using Encryption for Authentication in Large Networks of Computers. Communications of the ACM, 21(12):993--999, 1978.]]

Digital Library

[31]

The economic impacts of inadequate infrastructure for software testing. National Institute of Standards and technology, Planning Report 02-3, May 2002.]]

[32]

J. Offut and J. Hayes. A Semantic Model of Program Faults. In Proceedings of ISSTA'96 (International Symposium on Software Testing and Analysis), pages 195--200, San Diego, January 1996.]]

Digital Library

[33]

Polyspace. web page: http://www.polyspace.com.]]

[34]

D. Saff and M. D. Ernst. Continuous testing in Eclipse. In Proceedings of 2nd Eclipse Technology Exchange Workshop(eTX), Barcelona, March 2004.]]

[35]

S. D. Stoller. Domain Partitioning for Open Reactive Programs. In Proceedings of ACM SIGSOFT ISSTA'02 (International Symposium on Software Testing and Analysis), 200.]]

Digital Library

[36]

W. Visser, C. Pasareanu, and S. Khurshid. Test Input Generation with Java PathFinder. In Proceedings of ACM SIGSOFT ISSTA'04 (International Symposium on Software Testing and Analysis), Boston, July 2004.]]

Digital Library

[37]

J. Whaley, M. C. Martin, and M. S. Lam. Automatic Extraction of Object-Oriented Component Interfaces. In Proceedings of ACM SIGSOFT ISSTA'02 (International Symposium on Software Testing and Analysis), 2002.]]

Digital Library

[38]

T. Xie, D. Marinov, W. Schulte, and D. Notkin. Symstra: A framework for generating object-oriented unit tests using symbolic execution. In Proceedings of TACAS'05 (11th Conference on Tools and Algorithms for the Construction and Analysis of Systems), volume 3440 of LNCS, pages 365--381. Springer, 2005.]]

Digital Library

Cited By

Luo JZhao J(2025)Formalization of Quantum Intermediate Representations for code safetyJournal of Systems and Software10.1016/j.jss.2024.112236219(112236)Online publication date: Jan-2025
https://doi.org/10.1016/j.jss.2024.112236
Winterer DSu Z(2024)Validating SMT Solvers for Correctness and Performance via Grammar-Based EnumerationProceedings of the ACM on Programming Languages10.1145/36897958:OOPSLA2(2378-2401)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689795
Qian KSmith SStride BWeng SWu K(2024)Semantic-Type-Guided Bug FindingProceedings of the ACM on Programming Languages10.1145/36897888:OOPSLA2(2183-2210)Online publication date: 8-Oct-2024
https://doi.org/10.1145/3689788
Show More Cited By

Index Terms

DART: directed automated random testing

Recommendations

DART: directed automated random testing
PLDI '05: Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation

We present a new tool, named DART, for automatically testing software that combines three main techniques: (1) automated extraction of the interface of a program with its external environment using static source-code parsing; (2) automatic generation of ...
The Effectiveness of T-Way Test Data Generation
SAFECOMP '08: Proceedings of the 27th international conference on Computer Safety, Reliability, and Security

This paper reports the results of a study comparing the effectiveness of automatically generated tests constructed using random and <em>t</em>-way combinatorial techniques on safety related industrial code using mutation adequacy criteria. A reference ...
Prioritizing random combinatorial test suites
SAC '17: Proceedings of the Symposium on Applied Computing

The behaviour of a system under test can be influenced by several factors, such as system configurations, user inputs, and so on. It has also been observed that many failures are caused by only a small number of factors. Combinatorial testing aims at ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGPLAN Notices

ACM SIGPLAN Notices Volume 40, Issue 6

Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation

June 2005

325 pages

ISSN:0362-1340

EISSN:1558-1160

DOI:10.1145/1064978

Issue’s Table of Contents

PLDI '05: Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation
June 2005
338 pages
ISBN:1595930566
DOI:10.1145/1065010
General Chair:
Vivek Sarkar
IBM Research
,
Program Chair:
Mary Hall
USC/ISI

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 June 2005

Published in SIGPLAN Volume 40, Issue 6

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2,151
Total Citations
View Citations
9,057
Total Downloads

Downloads (Last 12 months)386
Downloads (Last 6 weeks)35

Reflects downloads up to 04 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Luo JZhao J(2025)Formalization of Quantum Intermediate Representations for code safetyJournal of Systems and Software10.1016/j.jss.2024.112236219(112236)Online publication date: Jan-2025
https://doi.org/10.1016/j.jss.2024.112236
Winterer DSu Z(2024)Validating SMT Solvers for Correctness and Performance via Grammar-Based EnumerationProceedings of the ACM on Programming Languages10.1145/36897958:OOPSLA2(2378-2401)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689795
Qian KSmith SStride BWeng SWu K(2024)Semantic-Type-Guided Bug FindingProceedings of the ACM on Programming Languages10.1145/36897888:OOPSLA2(2183-2210)Online publication date: 8-Oct-2024
https://doi.org/10.1145/3689788
Correnson ANießen TFinkbeiner BWeissenbacher G(2024)Finding ∀∃ Hyperbugs using Symbolic ExecutionProceedings of the ACM on Programming Languages10.1145/36897618:OOPSLA2(1420-1445)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689761
Kuliamin V(2024)A Survey of Software Dynamic Analysis MethodsProgramming and Computing Software10.1134/S036176882401007950:1(90-114)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1134/S0361768824010079
Hull MPednekar VMurray HRoy NTung ERoutray SGuerin CChen JWang ZLee SRoozbahani MChau D(2024)VG: Automatic Grading of D3 VisualizationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332718130:1(617-627)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3327181
Dinesh SParthasarathy MFletcher C(2024)ConjunCT: Learning Inductive Invariants to Prove Unbounded Instruction Safety Against Microarchitectural Timing Attacks2024 IEEE Symposium on Security and Privacy (SP)10.1109/SP54263.2024.00180(3735-3753)Online publication date: 19-May-2024
https://doi.org/10.1109/SP54263.2024.00180
Haltermann JJakobs MRichter CWehrheim H(2024)Parallel Program Analysis on Path RangesScience of Computer Programming10.1016/j.scico.2024.103154(103154)Online publication date: May-2024
https://doi.org/10.1016/j.scico.2024.103154
Semujju SLiu FHuang HXiang YYan XHao Z(2024)Knowledge transfer based many-objective approach for finding bugs in multi-path loopsComplex & Intelligent Systems10.1007/s40747-023-01323-wOnline publication date: 24-Jan-2024
https://doi.org/10.1007/s40747-023-01323-w
Zheng ZLyu Y(2023)STSearch: State Tracing-based Search Heuristics for RTL Validation2023 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE56975.2023.10136970(1-6)Online publication date: Apr-2023
https://doi.org/10.23919/DATE56975.2023.10136970
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents