research-article

Footprints of fitness functions in search-based software testing

Authors:

Carlos Oliveira,

Mohamed AbdelrazekAuthors Info & Claims

GECCO '19: Proceedings of the Genetic and Evolutionary Computation Conference

Pages 1399 - 1407

https://doi.org/10.1145/3321707.3321880

Published: 13 July 2019 Publication History

Abstract

Testing is technically and economically crucial for ensuring software quality. One of the most challenging testing tasks is to create test suites that will reveal potential defects in software. However, as the size and complexity of software systems increase, the task becomes more labour-intensive and manual test data generation becomes infeasible. To address this issue, researchers have proposed different approaches to automate the process of generating test data using search techniques; an area that is known as Search-Based Software Testing (SBST).

SBST methods require a fitness function to guide the search to promising areas of the solution space. Over the years, a plethora of fitness functions have been proposed. Some methods use control information, others focus on goals. Deciding on what fitness function to use is not easy, as it depends on the software system under test. This work investigates the impact of software features on the effectiveness of different fitness functions. We propose the Mapping the Effectiveness of Test Automation (META) Framework which analyses the footprint of different fitness functions and creates a decision tree that enables the selection of the appropriate function based on software features.

References

[1]

Aldeida Aleti and Lars Grunske. 2015. Test data generation with a Kalman filter-based adaptive genetic algorithm. Journal of Systems and Software 103 (2015), 343 -- 352. Special Issue.

Digital Library

[2]

Aldeida Aleti and Irene Moser. 2011. Predictive Parameter Control. In Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation. ACM, 561--568.

Digital Library

[3]

Aldeida Aleti and Irene Moser. 2013. Entropy-based Adaptive Range Parameter Control for Evolutionary Algorithms. In Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation. ACM, 1501--1508.

Digital Library

[4]

Aldeida Aleti, Irene Moser, and Lars Grunske. 2016. Analysing the fitness landscape of search-based software testing problems. Automated Software Engineering (2016), 1--19.

Digital Library

[5]

Aldeida Aleti, Irene Moser, Indika Meedeniya, and Lars Grunske. 2014. Choosing the appropriate forecasting model for predictive parameter control. Evolutionary computation 22, 2 (2014), 319--349.

Digital Library

[6]

Aldeida Aleti, Irene Moser, and Sanaz Mostaghim. 2012. Adaptive range parameter control. In 2012 IEEE Congress on Evolutionary Computation. IEEE, 1--8.

[7]

André Baresel, Harmen Sthamer, and Michael Schmidt. 2002. Fitness Function Design To Improve Evolutionary Structural Testing. In Genetic and Evolutionary Computation Conference, Vol. 2. 1329--1336.

[8]

David L. Bird and Carlos Urias Munoz. 1983. Automatic generation of random self-checking test cases. IBM systems journal 22, 3 (1983), 229--245.

[9]

B Borah and DK Bhattacharyya. 2004. An improved sampling-based DBSCAN for large spatial databases. In Intelligent Sensing and Information Processing, 2004. Proceedings of International Conference on. IEEE, 92--96.

[10]

Jon Edvardsson. 1999. A survey on automatic test data generation. In Proceedings of the 2nd Conference on Computer Science and Engineering. 21--28.

[11]

Roger Ferguson and Bogdan Korel. 1995. Software test data generation using the chaining approach. In International Test Conference. IEEE, 703--709.

Digital Library

[12]

Gordon Fraser and Andrea Arcuri. 2013. Whole test suite generation. Software Engineering, IEEE Transactions on 39, 2 (2013), 276--291.

Digital Library

[13]

Gordon Fraser and Andrea Arcuri. 2014. A large-scale evaluation of automated unit test generation using EvoSuite. ACM Transactions on Software Engineering and Methodology 24, 2 (2014), 8.

Digital Library

[14]

Gordon Fraser and Andrea Arcuri. 2014. A Large-Scale Evaluation of Automated Unit Test Generation Using EvoSuite. ACM Transactions on Software Engineering and Methodology 24, 2, Article 8 (2014), 8:1--8:42 pages.

Digital Library

[15]

Matthew J Gallagher and V Lakshmi Narasimhan. 1997. Adtest: A test data generation suite for ada software systems. Software Engineering, IEEE Transactions on 23, 8 (1997), 473--484.

Digital Library

[16]

Kamran Ghani and John A Clark. 2009. Automatic test data generation for multiple condition and MCDC coverage. In Software Engineering Advances, International Conference on. IEEE, 152--157.

Digital Library

[17]

Marius Gheorghita, Irene Moser, and Aldeida Aleti. 2013. Characterising fitness landscapes using predictive local search. In Proceedings of the 15th annual conference companion on genetic and evolutionary computation. ACM, 67--68.

Digital Library

[18]

Mark Harman and Phil McMinn. 2010. A theoretical and empirical study of search-based testing: Local, global, and hybrid search. Software Engineering, IEEE Transactions on 36, 2 (2010), 226--247.

Digital Library

[19]

Yue Jia and Mark Harman. 2008. Constructing subtle faults using higher order mutation testing. In Source Code Analysis and Manipulation, International Working Conference on. IEEE, 249--258.

[20]

Bogdan Korel. 1990. Automated software test data generation. Software Engineering, IEEE Transactions on 16, 8 (1990), 870--879.

Digital Library

[21]

Bogdan Korel. 1992. Dynamic method for software test data generation. Software Testing, Verification and Reliability 2, 4 (1992), 203--213.

[22]

Bogdan Korel. 1996. Automated test data generation for programs with procedures. In ACM SIGSOFT Software Engineering Notes, Vol. 21. ACM, 209--215.

Digital Library

[23]

Nashat Mansour and Miran Salame. 2004. Data generation for path testing. Software Quality Journal 12, 2 (2004), 121--136.

Digital Library

[24]

Phil McMinn. 2004. Search-based software test data generation: A survey. Software Testing Verification and Reliability 14, 2 (2004), 105--156.

[25]

Webb Miller and David L. Spooner. 1976. Automatic generation of floating-point test data. IEEE Transactions on Software Engineering 2, 3 (1976), 223.

Digital Library

[26]

I Moser, M Gheorghita, and A Aleti. 2016. Identifying Features of Fitness Landscapes and Relating Them to Problem Difficulty. Evolutionary computation (2016).

Digital Library

[27]

Irene Moser, Marius Gheorghita, and Aldeida Aleti. 2016. Investigating the correlation between indicators of predictive diagnostic optimisation and search result quality. Information Sciences 372 (2016), 162--180.

Digital Library

[28]

Mario A. Muñoz, Laura Villanova, Davaatseren Baatar, and Kate Smith-Miles. 2018. Instance spaces for machine learning classification. Machine Learning 107, 1 (2018), 109--147.

Digital Library

[29]

Carlos Oliveira, Aldeida Aleti, Lars Grunske, and Kate Smith-Miles. 2018. Mapping the Effectiveness of Automated Test Suite Generation Techniques. IEEE Transactions on Reliability 99 (2018), 1--15.

[30]

Alessandro Orso and Gregg Rothermel. 2014. Software testing: a research travelogue (2000--2014). In Future of Software Engineering. ACM, 117--132.

Digital Library

[31]

Roy P Pargas, Mary Jean Harrold, and Robert R Peck. 1999. Test-data generation using genetic algorithms. Software Testing Verification and Reliability 9, 4 (1999), 263--282.

[32]

Kemal Polat and Salih Güneş. 2009. A novel hybrid intelligent method based on C4. 5 decision tree classifier and one-against-all approach for multi-class classification problems. Expert Systems with Applications 36, 2 (2009), 1587--1592.

Digital Library

[33]

D. M. W. Powers. 2011. Evaluation: From precision, recall and f-measure to roc., informedness, markedness & correlation. Journal of Machine Learning Technologies 2, 1 (2011), 37--63.

[34]

Peter Puschner and Roman Nossal. 1998. Testing the results of static worst-case execution-time analysis. In Real-Time Systems Symposium. IEEE, 134--143.

Digital Library

[35]

José Miguel Rojas, Mattia Vivanti, Andrea Arcuri, and Gordon Fraser. 2017. A detailed investigation of the effectiveness of whole test suite generation. Empirical Software Engineering 22, 2 (2017), 852--893.

Digital Library

[36]

Marc Roper. 1997. Computer aided software testing using genetic algorithms. (1997).

[37]

Sina Shamshiri, José Miguel Rojas, Gordon Fraser, and Phil McMinn. 2015. Random or Genetic Algorithm Search for Object-Oriented Test Suite Generation?. In Genetic and Evolutionary Computation Conference. ACM, 1367--1374.

Digital Library

[38]

Kate A. Smith-Miles, Davaatseren Baatar, Brendan Wreford, and Rhyd Lewis. 2014. Towards objective measures of algorithm performance across instance space. 45 (2014), 12--24.

Digital Library

[39]

Pascale Thevenod-Fosse and Helene Waeselynck. 1993. STATEMATE applied to statistical software testing. In ACM SIGSOFT Software Engineering Notes, Vol. 18. ACM, 99--109.

Digital Library

[40]

Nigel Tracey, John Clark, Keith Mander, and John McDermid. 1998. An automated framework for structural test-data generation. In Automated Software Engineering. IEEE, 285--288.

Digital Library

[41]

Nigel James Tracey. 2000. A search-based automated test-data generation framework for safety-critical software. Ph.D. Dissertation. Citeseer.

[42]

Jeffrey Voas, Larry Morell, and Keith Miller. 1991. Predicting where faults can hide from testing. Software 8, 2 (1991), 41--48.

Digital Library

[43]

Alison Watkins and Ellen M Hufnagel. 2006. Evolutionary test data generation: a comparison of fitness functions. Software: Practice and Experience 36, 1 (2006), 95--116.

Digital Library

[44]

Alison Lachut Watkins. 1995. The automatic generation of test data using genetic algorithms. In Software Quality Conference, Vol. 2. 300--309.

[45]

Joachim Wegener, André Baresel, and Harmen Sthamer. 2001. Evolutionary test environment for automatic structural testing. Information and Software Technology 43, 14 (2001), 841--854.

[46]

Joachim Wegener, Harmen Sthamer, Bryan F Jones, and David E Eyres. 1997. Testing real-time systems using genetic algorithms. Software Quality Journal 6, 2 (1997), 127--135.

Digital Library

[47]

David H Wolpert and William G Macready. 1997. No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation 1, 1 (1997), 67--82.

Digital Library

[48]

S Xanthakis, C Ellis, C Skourlas, A Le Gall, S Katsikas, and K Karapoulios. 1992. Application of genetic algorithms to software testing. In International Conference on Software Engineering and its Applications. 625--636.

[49]

Man Xiao, Mohamed El-Attar, Marek Reformat, and James Miller. 2007. Empirical evaluation of optimization algorithms when used in goal-oriented automated test data generation techniques. Empirical Software Engineering 12, 2 (2007), 183--239.

Digital Library

Cited By

Jian Jia Tan AYong Chong CAleti A(2023)Extended Abstract of E-SC4R: Explaining Software Clustering for Remodularisation2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER56733.2023.00098(846-847)Online publication date: Mar-2023
https://doi.org/10.1109/SANER56733.2023.00098
Aleti A(2023)Software Testing of Generative AI Systems: Challenges and Opportunities2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE)10.1109/ICSE-FoSE59343.2023.00009(4-14)Online publication date: 14-May-2023
https://doi.org/10.1109/ICSE-FoSE59343.2023.00009
Aleti AMartinez M(2021)E-APR: Mapping the effectiveness of automated program repair techniquesEmpirical Software Engineering10.1007/s10664-021-09989-x26:5Online publication date: 13-Jul-2021
https://doi.org/10.1007/s10664-021-09989-x
Show More Cited By

Index Terms

Footprints of fitness functions in search-based software testing
1. Computing methodologies
  1. Artificial intelligence
    1. Search methodologies

Recommendations

Analysing the fitness landscape of search-based software testing problems

Search-based software testing automatically derives test inputs for a software system with the goal of improving various criteria, such as branch coverage. In many cases, evolutionary algorithms are implemented to find near-optimal test suites for ...
Test Suite Optimization Using Chaotic Firefly Algorithm in Software Testing

Software testing is time consuming and a costly activity. Effective generation of test cases is necessary in order to perform rigorous testing. There exist various techniques for effective test case generation. These techniques are based on various test ...
A dynamic fitness function for search based software testing
GECCO '18: Proceedings of the Genetic and Evolutionary Computation Conference Companion

Search Based Software Testing (SBST) formulates testing as an optimization problem, hence some search algorithms can be used to tackle it. The goal of SBST is to improve various test adequacy criteria. There are different types of coverage criteria, and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '19: Proceedings of the Genetic and Evolutionary Computation Conference

July 2019

1545 pages

ISBN:9781450361118

DOI:10.1145/3321707

Editor:
Manuel López-Ibáñez
University of Manchester, UK
,
General Chairs:
Anne Auger
Inria and Ecole Polytechnique, France
,
Thomas Stützle
IRIDIA, Université libre de Bruxelles Belgium

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

GECCO '19

Sponsor:

SIGEVO

GECCO '19: Genetic and Evolutionary Computation Conference

July 13 - 17, 2019

Prague, Czech Republic

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
174
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)1

Reflects downloads up to 07 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Jian Jia Tan AYong Chong CAleti A(2023)Extended Abstract of E-SC4R: Explaining Software Clustering for Remodularisation2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER56733.2023.00098(846-847)Online publication date: Mar-2023
https://doi.org/10.1109/SANER56733.2023.00098
Aleti A(2023)Software Testing of Generative AI Systems: Challenges and Opportunities2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE)10.1109/ICSE-FoSE59343.2023.00009(4-14)Online publication date: 14-May-2023
https://doi.org/10.1109/ICSE-FoSE59343.2023.00009
Aleti AMartinez M(2021)E-APR: Mapping the effectiveness of automated program repair techniquesEmpirical Software Engineering10.1007/s10664-021-09989-x26:5Online publication date: 13-Jul-2021
https://doi.org/10.1007/s10664-021-09989-x
Aleti A(2021)On the Effectiveness of SBSE TechniquesSearch-Based Software Engineering10.1007/978-3-030-88106-1_1(3-6)Online publication date: 29-Sep-2021
https://doi.org/10.1007/978-3-030-88106-1_1
Perera AAleti ABöhme MTurhan BGrundy JLe Goues CLo D(2020)Defect prediction guided search-based software testingProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering10.1145/3324884.3416612(448-460)Online publication date: 21-Dec-2020
https://dl.acm.org/doi/10.1145/3324884.3416612

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents