research-article

Generating Software Test Data by Evolution

Authors:

M. A. SchatzAuthors Info & Claims

IEEE Transactions on Software Engineering, Volume 27, Issue 12

Pages 1085 - 1110

https://doi.org/10.1109/32.988709

Published: 01 December 2001 Publication History

Abstract

This paper discusses the use of genetic algorithms (GAs) for automatic software test data generation. This research extends previous work on dynamic test data generation where the problem of test data generation is reduced to one of minimizing a function. In our work, the function is minimized by using one of two genetic algorithms in place of the local minimization techniques used in earlier research. We describe the implementation of our GA-based system and examine the effectiveness of this approach on a number of programs, one of which is significantly larger than those for which results have previously been reported in the literature. We also examine the effect of program complexity on the test data generation problem by executing our system on a number of synthetic programs that have varying complexities.

References

[1]

W. Miller and D.L. Spooner, “Automatic Generation of Floating Point Test Data,” IEEE Trans. Software Eng., vol. 2, no. 3, pp. 223–226, Sept. 1976.

Digital Library

[2]

B. Korel, “Automated Software Test Data Generation,” IEEE Trans. Software Eng., vol. 16, no. 8, pp. 870–879, Aug. 1990.

Digital Library

[3]

P. Frankl D. Hamlet B. Littlewood and L. Strigini, “Choosing a Testing Method to Deliver Reliability,” Proc. 19th Int'l Conf. Software Eng. (ICSE '97), pp. 68–78, May 1997.

Digital Library

[4]

R. Ferguson and B. Korel, “The Chaning Approach for Software Test Data Generation,” ACM Trans. Software Eng. Methodology, vol. 5, no. 1, pp. 63–86, Jan. 1996.

Digital Library

[5]

M.J. Gallagher and V.L. Narasimhan, “Adtest: A Test Data Generation Suite for Ada Software Systems,” IEEE Trans. Software Eng., vol. 23, no. 8, pp. 473–484, Aug. 1997.

Digital Library

[6]

J.H. Holland, Adaption in Natural and Artificial Systems. Ann Arbor, Mich.: Univ. of Michigan Press, 1975.

Digital Library

[7]

S. Kirkpatrick C.D. Gellat Jr. and M.P. Vecchi, “Optimization by Simulated Annealing,” Science, vol. 220, no. 4, 598, pp. 671–680, May 1983.

[8]

F. Glover, “Tabu Search Part I, II,” ORSA J. Computing, vol. 1, no. 3, pp. 190–206, 1989.

[9]

J. Horgan S. London and M. Lyu, “Achieving Software Quality with Testing Coverage Measures,” Computer, vol. 27, no. 9, pp. 60–69, Sept. 1994.

Digital Library

[10]

J. Chilenski and S. Miller, “Applicability of Modified Condition/Decision Coverage to Software Testing,” Software Eng. J., pp. 193–200, Sept. 1994.

[11]

R. DeMillo and A. Mathur, “On the Uses of Software Artifacts to Evaluate the Effectiveness of Mutation Analysis for Detecting Errors in Production Software,” Technical Report SERC-TR-92-P, Purdue Univ., 1992.

[12]

R. Hamlet and R. Taylor, “Partition Testing Does Not Inspire Confidence,” IEEE Trans. Software Eng., vol. 16, no. 12, pp. 1402–1411, Dec. 1990.

Digital Library

[13]

E.J. Weyuker and B. Jeng, “Analyzing Partition Testing Strategies,” IEEE Trans. Software Eng., vol. 17, no. 7, pp. 703–711, July 1991.

Digital Library

[14]

E.J. Weyuker, “Axiomatizing Software Test Adequacy,” IEEE Trans. Software Eng., vol. 12, no. 12, pp. 1128–1137, Dec. 1986.

Digital Library

[15]

P.G. Frankl and E.J. Weyuker, “A Formal Analysis of the Fault-Detecting Ability of Testing Methods,” IEEE Trans. Software Eng., vol. 19, no. 3, pp. 202–213, Mar. 1993.

Digital Library

[16]

B. Korel, “Automated Test Data Generation for Programs with Procedures,” Proc. Int'l Symp. Software Testing and Analysis, pp. 209–215, 1996.

Digital Library

[17]

L.A. Clarke, “A System to Generate Test Data Symbolically and Execute Programs,” IEEE Trans. Software Eng., vol. 2, no. 3, pp. 215–222, Sept. 1976.

Digital Library

[18]

C.V. Ramamoorty S.F. Ho and W.T. Chen, “On the Automated Generation of Program Test Data,” IEEE Trans. Software Eng., vol. 2, no. 4, pp. 293–300, Dec. 1976.

[19]

J. Offutt, “An Integrated Automatic Test Data Generation System,” J. Systems Integration, vol. 1, pp. 391–409, 1991.

[20]

W.H. Deason D.B. Brown K-H. Chang and J.H. Cross II, “A Rule-Based Software Test Data Generator,” IEEE Trans. Knowledge and Data Eng., vol. 3, no. 1, pp. 108–117, Mar. 1991.

Digital Library

[21]

K.H. Chang J.H. Cross II W.H. Carlisle and S.-S. Liao, “A Performance Evaluation of Heuristics-Based Test Case Generation Methods for Software Branch Coverage,” Int'l J. Software Eng. and Knowledge Eng., vol. 6, no. 4, pp. 585–608, 1966.

[22]

N. Tracey J. Clark and K. Mander, “The Way Forward for Unifying Dynamic Test-Case Generation: The Optimisation-Based Approach,” Proc. Int'l Workshop Dependable Computing and Its Applications (DCIA), pp. 169–180, Jan. 1998.

[23]

N. Tracey J. Clark and K. Mander, “Automated Program Flaw Finding Using Simulated Annealing,” Proc. Int'l Symp. Software Testing and Analysis, Software Eng. Notes, pp. 73–81, Mar. 1998.

Digital Library

[24]

H. Tracey J. Clark K. Mander and J. McDermid, “An Automated Framework for Structural Test-Data Generation,” Proc. Automated Software Eng. '98, pp. 285–288, 1998.

Digital Library

[25]

C.C. Michael G.E. McGraw and M.A. Schatz, “Genetic Algorithms for Dynamic Test Data Generation,” Proc. Automated Software Eng. '97, pp. 307–308, 1997.

Digital Library

[26]

C.C. Michael G.E. McGraw and M.A. Schatz, “Opportunism and Diversity in Automated Software Test Data Generation,” Proc. Automated Software Eng. '98, pp. 136–146, 1998.

Digital Library

[27]

A.C. Schultz J.C. Grefenstette and K.A. DeJong, “Test and Evaluation by Genetic Algorithms,” IEEE Expert, pp. 9–14, Oct. 1993.

Digital Library

[28]

W.H. Press B.P. Flannery S.A. Teukolsky and W.T. Vetterling, Numerical Recipes in C. New York: Cambridge Univ. Press, 1991.

[29]

J. Skorin-Kapov, “Tabu Search Applied to the Quadratic Assignment Problem,” ORSA J. Computing, vol. 2, no. 1, pp. 33–41, Winter 1990.

[30]

D. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning. Reading, Mass.: Addison-Wesley, 1989.

Digital Library

[31]

M. Mitchell, An Introduction to Genetic Algorithms. Cambridge, Mass.: MIT Press, 1996.

Digital Library

[32]

Foundations of Genetic Algorithms. G. Rawlins, ed. San Mateo, Calif.: Morgan Kaufmann, 1991.

[33]

R. Storn, “On the Usage of Differential Evolution for Function Optimization,” Proc. North Am. Fuzzy Information Prossessing Soc., (NAFIPS '96), pp. 519–523, June 1996.

[34]

S.K. Park and K.W. Miller, “Random Number Generators: Good Ones are Hard to Find,” Comm. ACM, vol. 31, no. 10, pp. 1192–1201, Oct. 1988.

Digital Library

[35]

R.A. DeMillo and A.J. Offutt, “Experimental Results from an Automatic Test Case Generator,” ACM Trans. Software Eng. Methodology, vol. 2, no. 1, pp. 215–222, Jan. 1993.

Digital Library

[36]

J.D. Musa, “Operational Profiles in Software Engineering,” IEEE Software, vol. 10, no. 2, pp. 14–332, 1993.

Digital Library

Cited By

Nguyen Tung LBinh Duong NLe KHung P(2024)Automated test data generation and stubbing method for C/C++ embedded projectsAutomated Software Engineering10.1007/s10515-024-00449-631:2Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1007/s10515-024-00449-6
Ahsan FAnwer F(2024)A systematic literature review on software security testing using metaheuristicsAutomated Software Engineering10.1007/s10515-024-00433-031:2Online publication date: 23-May-2024
https://dl.acm.org/doi/10.1007/s10515-024-00433-0
First EBrun YDwyer MDamian DZeller A(2022)Diversity-driven automated formal verificationProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510138(749-761)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510138
Show More Cited By

Index Terms

Generating Software Test Data by Evolution

Recommendations

Genetic algorithms for dynamic test data generation
ASE '97: Proceedings of the 12th international conference on Automated software engineering (formerly: KBSE)

In software testing, it is often desirable to find test inputs that exercise specific program features. To find these inputs by hand is extremely time-consuming, especially when the software is complex. Therefore, numerous attempts have been made to ...
Optimisation of software testing using Genetic Algorithm

Software testing is meant to increase confidence in the correctness of software. Test data generation is one of the key issues in software testing. A properly generated test suite may not only locate the errors in a software system, but also help in ...
GA-based multiple paths test data generator

Developers have learned over time that software testing costs a considerable amount of a software project budget. Hence, software quality managers have been looking for solutions to reduce testing costs and time. Considering path coverage as the test ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Software Engineering

IEEE Transactions on Software Engineering Volume 27, Issue 12

December 2001

100 pages

ISSN:0098-5589

Issue’s Table of Contents

Copyright © Copyright © 2001 IEEE. All Rights Reserved.

Publisher

IEEE Press

Publication History

Published: 01 December 2001

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

115
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Nguyen Tung LBinh Duong NLe KHung P(2024)Automated test data generation and stubbing method for C/C++ embedded projectsAutomated Software Engineering10.1007/s10515-024-00449-631:2Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1007/s10515-024-00449-6
Ahsan FAnwer F(2024)A systematic literature review on software security testing using metaheuristicsAutomated Software Engineering10.1007/s10515-024-00433-031:2Online publication date: 23-May-2024
https://dl.acm.org/doi/10.1007/s10515-024-00433-0
First EBrun YDwyer MDamian DZeller A(2022)Diversity-driven automated formal verificationProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510138(749-761)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510138
Goschen JBosman AGruner S(2022)Genetic Micro-Programs for Automated Software Testing with Large Path Coverage2022 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC55065.2022.9870310(1-8)Online publication date: 18-Jul-2022
https://dl.acm.org/doi/10.1109/CEC55065.2022.9870310
Benito-Parejo MMerayo M(2021)Using Genetic Algorithms To Select Test Cases For Finite State Machines With Timeouts2021 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC45853.2021.9504764(2403-2410)Online publication date: 28-Jun-2021
https://dl.acm.org/doi/10.1109/CEC45853.2021.9504764
Anjum MRyan C(2021)Seeding Grammars in Grammatical Evolution to Improve Search-Based Software TestingSN Computer Science10.1007/s42979-021-00631-72:4Online publication date: 19-May-2021
https://dl.acm.org/doi/10.1007/s42979-021-00631-7
First EBrun YGuha A(2020)TacTok: semantics-aware proof synthesisProceedings of the ACM on Programming Languages10.1145/34282994:OOPSLA(1-31)Online publication date: 13-Nov-2020
https://dl.acm.org/doi/10.1145/3428299
Anjum MRyan CCoello Coello C(2020)Scalability analysis of grammatical evolution based test data generationProceedings of the 2020 Genetic and Evolutionary Computation Conference10.1145/3377930.3390167(1213-1221)Online publication date: 25-Jun-2020
https://dl.acm.org/doi/10.1145/3377930.3390167
Benito-Parejo MMerayo M(2020)An evolutionary algorithm for selection of test cases2020 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC48606.2020.9185773(1-8)Online publication date: 19-Jul-2020
https://dl.acm.org/doi/10.1109/CEC48606.2020.9185773
Anjum MRyan C(2020)Seeding Grammars in Grammatical Evolution to Improve Search Based Software TestingGenetic Programming10.1007/978-3-030-44094-7_2(18-34)Online publication date: 15-Apr-2020
https://dl.acm.org/doi/10.1007/978-3-030-44094-7_2
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents