skip to main content
survey

Using Genetic Algorithms in Test Data Generation: A Critical Systematic Mapping

Published: 22 May 2018 Publication History

Abstract

Software testing activities account for a considerable portion of systems development cost and, for this reason, many studies have sought to automate these activities. Test data generation has a high cost reduction potential (especially for complex domain systems), since it can decrease human effort. Although several studies have been published about this subject, articles of reviews covering this topic usually focus only on specific domains. This article presents a systematic mapping aiming at providing a broad, albeit critical, overview of the literature in the topic of test data generation using genetic algorithms. The selected studies were categorized by software testing technique (structural, functional, or mutation testing) for which test data were generated and according to the most significantly adapted genetic algorithms aspects. The most used evaluation metrics and software testing techniques were identified. The results showed that genetic algorithms have been successfully applied to simple test data generation, but are rarely used to generate complex test data such as images, videos, sounds, and 3D (three-dimensional) models. From these results, we discuss some challenges and opportunities for research in this area.

References

[1]
ACM Digital Library. 2017. Retrieved December 2016 from http://dl.acm.org.
[2]
Wasif Afzal, Richard Torkar, and Robert Feldt. 2009. A systematic review of search-based testing for non-functional system properties. Inform. Softw. Technol. 51, 6 (2009), 957--976.
[3]
Moataz A. Ahmed and Fakhreldin Ali. 2015. Multiple-path testing for cross site scripting using genetic algorithms. J. Syst. Archit. (2015).
[4]
Moataz A. Ahmed and Irman Hermadi. 2008. GA-based multiple paths test data generator. Comput. Oper. Res. 35, 10 (2008), 3107--3124. Part Special Issue: Search-based Software Engineering.
[5]
Nassima Aleb and Samir Kechid. 2013. Automatic test data generation using a genetic algorithm. In International Conference on Computational Science and Its Applications. Springer, 574--586.
[6]
Stefano Di Alesio, Lionel C. Briand, Shiva Nejati, and Arnaud Gotlieb. 2015. Combining genetic algorithms and constraint programming to support stress testing of task deadlines. ACM Trans. Softw. Engi. Methodol. 25, 1 (2015), 4.
[7]
Aldeida Aleti and Lars Grunske. 2015. Test data generation with a Kalman filter-based adaptive genetic algorithm. J. Syst. Softw. 103 (2015), 343--352.
[8]
Aldeida Aleti and Irene Moser. 2016. A systematic literature review of adaptive parameter control methods for evolutionary algorithms. ACM Comput. Surv. 49, 3 (Oct. 2016), 56:1--56:35.
[9]
S. Ali, C. Briand, H. Hemmati, and R. K. Panesar-Walawege. 2010. A systematic review of the application and empirical investigation of search-based test case generation. IEEE Trans. Softw. Eng. 36, 6 (2010), 742--762.
[10]
Mohammad Alshraideh, Basel A. Mahafzah, and Saleh Al-Sharaeh. 2011. A multiple-population genetic algorithm for branch coverage test data generation. Softw. Qual. J. 19, 3 (2011), 489--513.
[11]
I. Alsmadi. 2010. Using genetic algorithms for test case generation and selection optimization. In Proceedings of the 2010 23rd Canadian Conference on Electrical and Computer Engineering (CCECE’10). 1--4.
[12]
Anuja Arora and Madhavi Sinha. 2014. State based test case generation using VCL-GA. In Proceedings of the 2014 International Conference on Issues and Challenges in Intelligent Computing Techniques (ICICT’14). IEEE, 661--665.
[13]
K. Betts and M. Petty. 2016. Automated search-based robustness testing for autonomous vehicle software. Model. Simul. Eng. 2016 (2016).
[14]
A. Bouchachia. 2007. An immune genetic algorithm for software test data generation. In Proceedings of the 7th International Conference on Hybrid Intelligent Systems, 2007. (HIS’07). 84--89.
[15]
Cristian Cadar and Koushik Sen. 2013. Symbolic execution for software testing: Three decades later. Commun. ACM 56, 2 (Feb. 2013), 82--90.
[16]
Yang Cao, Chunhua Hu, and Luming Li. 2009. Search-based multi-paths test data generation for structure-oriented testing. In Proceedings of the 1st ACM/SIGEVO Summit on Genetic and Evolutionary Computation (GEC’09). ACM, New York, NY, 25--32.
[17]
Y. Chen and Y. Zhong. 2008. Automatic path-oriented test data generation using a multi-population genetic algorithm. In Proceedings of the 4th International Conference on Natural Computation, 2008 (ICNC’08), Vol. 1. 566--570.
[18]
Y. Chen and Y. Zhong. 2009. Experimental study on GA-based path-oriented test data generation using branch distance function. In Proceedings of the 3rd International Symposium on Intelligent Information Technology Application, 2009. (IITA’09), Vol. 1. 216--219.
[19]
M. Dave and R. Agrawal. 2015. Search based techniques and mutation analysis in automatic test case generation: A survey. In Proceedings of the 2015 IEEE International Advance Computing Conference (IACC’15). 795--799.
[20]
A. Deepak and P. Samuel. 2009. An evolutionary multi population approach for test data generation. In Proceedings of the World Congress on Nature Biologically Inspired Computing, 2009 (NaBIC’09). 1451--1456.
[21]
Karnig Derderian, Mercedes G. Merayo, Robert M. Hierons, and Manuel Núñez. 2011. A case study on the use of genetic algorithms to generate test cases for temporal systems. In International Work-Conference on Artificial Neural Networks. Springer, 396--403.
[22]
A. El-Serafy, G. El-Sayed, C. Salama, and A. Wahba. 2015. Enhanced genetic algorithm for MC/DC test data generation. In Proceedings of the 2015 International Symposium on Innovations in Intelligent SysTems and Applications (INISTA’15). 1--8.
[23]
X. Fan, F. Y. Yang, W. Zheng, and Q. J. Liang. 2015. Test data generation with a hybrid genetic tabu search algorithm for decision coverage criteria. Proc. Sci. 12-13-September-2015 (2015).
[24]
Marten Fischer and Ralf Tönjes. 2012. Generating test data for black-box testing using genetic algorithms. In Proceedings of 2012 IEEE 17th International Conference on Emerging Technologies & Factory Automation (ETFA’12). IEEE, 1--6.
[25]
D. Garg and P. Garg. 2015. Basis path testing using SGA & HGA with ExLB fitness function, J. Mathew and A.K. Singh (Eds.). Proced. Comput. Sci. 70 (2015), 593--602.
[26]
Ahmed S. Ghiduk and Moheb R. Girgis. 2010. Using genetic algorithms and dominance concepts for generating reduced test data. Informatica 34, 3 (2010).
[27]
A. S. Ghiduk, M. J. Harrold, and M. R. Girgis. 2007. Using genetic algorithms to aid test-data generation for data-flow coverage. In Proceedings of the 14th Asia-Pacific Software Engineering Conference, 2007 (APSEC’07). 41--48.
[28]
David E. Goldberg. 1989. Genetic Algorithms in Search, Optimization and Machine Learning (13th ed.). Addison-Wesley Publishing Company.
[29]
Dunwei Gong, Gongjie Zhang, Xiangjuan Yao, and Fanlin Meng. 2017. Mutant reduction based on dominance relation for weak mutation testing. Inform. Softw. Technol. 81 (2017), 82--96.
[30]
Dunwei Gong, Wanqiu Zhang, and Xiangjuan Yao. 2011. Evolutionary generation of test data for many paths coverage based on grouping. J. Syst. Softw. 84, 12 (2011), 2222--2233.
[31]
N. Gupta and M. Rohil. 2013. Improving GA based automated test data generation technique for object oriented software. In Proceedings of the, 2013 IEEE 3rd International Advance Computing Conference (IACC’13). 249--253.
[32]
L. T. M. Hanh, N. T. Binh, and K. T. Tung. 2014. Applying the meta-heuristic algorithms for mutation-based test data generation for simulink models. ACM International Conference Proceeding Series December 4--5, 2014, 102--109.
[33]
L. T. M. Hanh, N. T. Binh, and K. T. Tung. 2016. A novel fitness function of metaheuristic algorithms for test data generation for simulink models based on mutation analysis. J. Syst. Softw. 120 (2016), 17--30.
[34]
Irman Hermadi and Moataz A. Ahmed. 2003. Genetic algorithm-based test data generator. In Proceedings of the 2003 Congress on Evolutionary Computation, 2003 (CEC’03), Vol. 1. IEEE, 85--91.
[35]
J. H. Holland. 1975. Adaptation in Natural and Artificial Systems. MIT Press.
[36]
S. Huang, M. B. Cohen, and A. M. Memon. 2010. Repairing GUI test suites using a genetic algorithm. In Proceedings of the 2010 3rd International Conference on Software Testing, Verification and Validation (ICST’10). 245--254.
[37]
IEEE Xplore Digital Library. 2017. IEEE xplore digital library. Retrieved December 2016 from http://ieeexplore.ieee.org.
[38]
H. Ji and J. Sun. 2012. Automatic test data generation based on SA-QGA. In Proceedings of the 2012 IEEE/ACIS 11th International Conference on Computer and Information Science (ICIS). 611--615.
[39]
Rong Jin, Shujuan Jiang, and Hongchang Zhang. 2011. Generation of test data based on genetic algorithms and program dependence analysis. In Proceedings of the 2011 IEEE International Conference on Cyber Technology in Automation, Control, and Intelligent Systems (CYBER’11). IEEE, 116--121.
[40]
R. Khan and M. Amjad. 2015. Automatic test case generation for unit software testing using genetic algorithm and mutation analysis. In 2015 IEEE UP Section Conference on Electrical Computer and Electronics (UPCON). 1--5.
[41]
R. Khan, M. Amjad, and A. K. Srivastava. 2016. Optimization of automatic generated test cases for path testing using genetic algorithm. In Proceedings of the 2016 2nd International Conference on Computational Intelligence Communication Technology (CICT’16). 32--36.
[42]
Juhi Khandelwal and Pradeep Tomar. 2015. Approach for automated test data generation for path testing in aspect-oriented programs using genetic algorithm. In Proceedings of the 2015 International Conference on Computing, Communication & Automation (ICCCA’15). IEEE, 854--858.
[43]
Fitsum M. Kifetew, Annibale Panichella, Andrea De Lucia, Rocco Oliveto, and Paolo Tonella. 2013. Orthogonal exploration of the search space in evolutionary test case generation. In Proceedings of the 2013 International Symposium on Software Testing and Analysis (ISSTA’13). ACM, New York, NY, 257--267.
[44]
B. Kitchenham and S. Charters. 2007. Guidelines for performing systematic literature reviews in software engineering. Version 2.3. EBSE Technical Report EBSE-2007-01, Keele University and Durham University.
[45]
C. Koleejan, B. Xue, and M. Zhang. 2015. Code coverage optimisation in genetic algorithms and particle swarm optimisation for automatic software test data generation. Proceedings of the 2015 IEEE Congress on Evolutionary Computation (CEC’15), 1204--1211.
[46]
Kiran Lakhotia, Mark Harman, and Phil McMinn. 2007. A multi-objective approach to search-based test data generation. In Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation (GECCO’07). ACM, New York, NY, 1098--1105.
[47]
Raluca Lefticaru and Florentin Ipate. 2007. Automatic state-based test generation using genetic algorithms. In Synasc, Vol. 7. 188--195.
[48]
Wang Lijuan, Zhai Yue, and Hou Hongfeng. 2012. Genetic algorithms and its application in software test data generation. In Proceedings of the 2012 International Conference on Computer Science and Electronics Engineering (ICCSEE’12), Vol. 2. IEEE, 617--620.
[49]
Peng Lin, Xiaolu Bao, Zhiyong Shu, Xiaojuan Wang, and Jingmin Liu. 2012. Test case generation based on adaptive genetic algorithm. In Proceedings of the 2012 International Conference on Quality, Reliability, Risk, Maintenance, and Safety Engineering (ICQR2MSE’12). IEEE, 863--866.
[50]
Dan Liu, Xuejun Wang, and Jianmin Wang. 2013. Automatic test case generation based on genetic algorithm. J. Theor. Appl. Inform. Technol. 48, 1 (2013), 411--416.
[51]
T. Manikumar, A. John Sanjeev Kumar, and R. Maruthamuthu. 2016. Automated test data generation for branch testing using incremental genetic algorithm. Sadhana—Acad. Proc. Eng. Sci. 41, 9 (2016), 959--976. SAPSE
[52]
M. Mann, O. P. Sangwan, P. Tomar, and S. Singh. 2016. Automatic goal-oriented test data generation using a genetic algorithm and simulated annealing. In Proceedings of the 2016 6th International Conference—Cloud System and Big Data Engineering (Confluence). 83--87.
[53]
J. Mayan and T. Ravi. 2014. Test case optimization using hybrid search technique. ACM International Conference Proceeding Series October 10--11, 2014.
[54]
James Miller, Marek Reformat, and Howard Zhang. 2006. Automatic test data generation using genetic algorithm and program dependence graphs. Inform. Softw. Technol. 48, 7 (2006), 586--605.
[55]
Melanie Mitchell. 1999. An Introduction to Genetic Algorithms (5th ed.). MIT Press, Cambridge, Massachusetts.
[56]
Glenford J. Myers. 1979. The Art of Software Testing. Wiley.
[57]
Alberto Núñez, Mercedes G. Merayo, Robert M. Hierons, and Manuel Núñez. 2013. Using genetic algorithms to generate test sequences for complex timed systems. Soft Comput. 17, 2 (2013), 301--315.
[58]
Ankur Pachauri and Gursaran Srivasatava. 2015. Towards a parallel approach for test data generation for branch coverage with genetic algorithm using the extended path prefix strategy. In Proceedings of the 2015 2nd International Conference on Computing for Sustainable Global Development (INDIACom’15). IEEE, 1786--1792.
[59]
M. Parthiban and M. R. Sumalatha. 2013. GASE—An input domain reduction and branch coverage system based on genetic algorithm and symbolic execution. In Proceedings of the 2013 International Conference on Information Communication and Embedded Systems (ICICES’13). 429--433.
[60]
Xuan Peng and Lu Lu. 2011. User-session-based automatic test case generation using GA. Int. J. Phys. Sci. 6, 13 (2011), 3232--3245.
[61]
G. H. L. Pinto and S. R. Vergilio. 2010. A multi-objective genetic algorithm to test data generation. In 2010 22nd IEEE International Conference on Tools with Artificial Intelligence (ICTAI’10), Vol. 1. 129--134.
[62]
Shweta Rani and Bharti Suri. 2015. An approach for test data generation based on genetic algorithm and delete mutation operators. In 2015 2nd International Conference on Advances in Computing and Communication Engineering (ICACCE’15). IEEE, 714--718.
[63]
Science Direct. 2017. Science direct. Retrieved December 2016 from http://www.sciencedirect.com.
[64]
Scopus. 2017. Scopus. Retrieved December 2016 from http://www.scopus.com.
[65]
N. Sharma, A. Pasala, and R. Kommineni. 2012. Generation of character test input data using GA for functional testing. In Proceedings of the 2012 19th Asia-Pacific Software Engineering Conference (APSEC’12), Vol. 2. 87--94.
[66]
Liu Shimin and Wang Zhangang. 2011. Genetic algorithm and its application in the path-oriented test data automatic generation. Proced. Eng. 15 (2011), 1186--1190. CEIS 2011.
[67]
B. Shuai, H. Li, L. Zhang, Q. Zhang, and C. Tang. 2015. Software vulnerability detection based on code coverage and test cost. Proceedings—2015 11th International Conference on Computational Intelligence and Security (CIS’15). 317--321.
[68]
Lucas Serpa Silva and Maarten van Someren. 2010. Evolutionary testing of object-oriented software. In Proceedings of the 2010 ACM Symposium on Applied Computing (SAC’10). ACM, New York, NY, 1126--1130.
[69]
M. Soltani, A. Panichella, and A. Van Deursen. 2016. Evolutionary testing for crash reproduction. Proceedings—9th International Workshop on Search-Based Software Testing (SBST’16), 1--4.
[70]
Ian Sommerville. 2007. Software Engineering (8th ed.). Addison-Wesley, São Paulo.
[71]
J. H. Sun and S. J. Jiang. 2010. An approach to automatic generating test data for multi-path coverage by genetic algorithm. In Proceedings of the 2010 6th International Conference on Natural Computation (ICNC’10), Vol. 3. 1533--1536.
[72]
X. Tan, Cheng Longxin, and Xu Xiumei. 2009. Test data generation using annealing immune genetic algorithm. In Proceedings of the 5th International Joint Conference on INC, IMS and IDC, 2009 (NCM’09). IEEE, 344--348.
[73]
Tian Tian and Dunwei Gong. 2016. Test data generation for path coverage of message-passing parallel programs based on co-evolutionary genetic algorithms. Autom. Softw. Eng. 23, 3 (Sept. 2016), 469--500.
[74]
Paolo Tonella. 2004. Evolutionary testing of classes. SIGSOFT Softw. Eng. Notes 29, 4 (July 2004), 119--128.
[75]
Sapna Varshney and Monica Mehrotra. 2016. Search-based test data generator for data-flow dependencies using dominance concepts, branch distance and elitism. Arabian J. Sci. Eng. 41, 3 (2016), 853--881.
[76]
Chandra Prakash Vudatha, Sateesh Nalliboena, Sastry K. R. Jammalamadaka, Bala Krishna Kamesh Duvvuri, and L. S. S. Reddy. 2011. Automated generation of test cases from output domain of an embedded system using genetic algorithms. In Proceedings of the 2011 3rd International Conference on Electronics Computer Technology (ICECT’11), Vol. 5. IEEE, 216--220.
[77]
Wang Xibo and Su Na. 2011. Automatic test data generation for path testing using genetic algorithms. In Proceedings of the 2011 3rd International Conference on Measuring Technology and Mechatronics Automation, Vol. 1. IEEE, 596--599.
[78]
S. Yang, T. Man, J. Xu, F. Zeng, and K. Li. 2016. RGA: A lightweight and effective regeneration genetic algorithm for coverage-oriented software test data generation. Inform. Softw. Technol. 76 (2016), 19--30.
[79]
Xiangjuan Yao and Dunwei Gong. 2014. Genetic algorithm-based test data generation for multiple paths via individual sharing. Comput. Intell. Neurosci. 2014 (2014), 29.
[80]
Xiangjuan Yao, Dunwei Gong, and Wenliang Wang. 2015a. Test data generation for multiple paths based on local evolution. Chinese J. Electron. 24, 1 (2015), 46--51.
[81]
X. Yao, D. Gong, and G. Zhang. 2015b. Constrained multi-objective test data generation based on set evolution. IET Softw. 9, 4 (2015), 103--108.
[82]
Na Zhang, Biao Wu, and Xiaoan Bao. 2015. Automatic generation of test cases based on multi-population genetic algorithm. Technology 10, 6 (2015).
[83]
Wanqiu Zhang, Dunwei Gong, Xiangjuan Yao, and Yan Zhang. 2010. Evolutionary generation of test data for many paths coverage. In Proceedings of the 2010 Chinese Control and Decision Conference. 230--235.
[84]
Y. Zhang and D. Gong. 2010. Evolutionary generation of test data for multiple paths coverage with faults detection. In 2010 IEEE 5th International Conference on Bio-Inspired Computing: Theories and Applications (BIC-TA’10). 406--410.
[85]
Xiaofei Zhou, Ruilian Zhao, and Feng You. 2014. EFSM-based test data generation with multi-population genetic algorithm. In Proceedings of the 2014 5th IEEE International Conference on Software Engineering and Service Science (ICSESS’14). IEEE, 925--928.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Computing Surveys
ACM Computing Surveys  Volume 51, Issue 2
March 2019
748 pages
ISSN:0360-0300
EISSN:1557-7341
DOI:10.1145/3186333
  • Editor:
  • Sartaj Sahni
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 May 2018
Accepted: 01 January 2018
Revised: 01 January 2018
Received: 01 January 2017
Published in CSUR Volume 51, Issue 2

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Test data generation
  2. evolutionary test
  3. genetic algorithms
  4. scoping study
  5. software testing
  6. systematic mapping
  7. test case generation

Qualifiers

  • Survey
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)66
  • Downloads (Last 6 weeks)6
Reflects downloads up to 09 Nov 2024

Other Metrics

Citations

Cited By

View all

View Options

Get Access

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media