MulTi-Wise Sampling: Trading Uniform T-Wise Feature Interaction Coverage for Smaller Samples
Pages 47 - 53
Abstract
Ensuring the functional safety of highly configurable systems often requires testing representative subsets of all possible configurations. The ratio of covered t-wise feature interactions is a common criterion for determining whether a subset of configurations is representative and capable of finding faults. Existing t-wise sampling algorithms cover t-wise feature interactions for all features equally, resulting in extensive sampling times and large sample sizes. In this paper, we introduce a novel approach to t-wise feature interaction sampling, relating the necessity of equal coverage across all t-wise feature interactions, called MulTi-Wise Sampling. MulTi-Wise Sampling prioritizes between subsets of critical and non-critical features, considering higher t-values for subsets of critical features when generating a sample. We evaluate MulTi-Wise Sampling using subject systems from real-world applications, including BusyBox, Soletta, Fiasco, and uCLibc-ng. Our results show that MulTi-Wise Sampling provides a flexible mechanism to reduce the number of configurations for testing, while the performance of the algorithm is still on par with state-of-the-art sampling algorithms.
References
[1]
Mustafa Al-Hajjaji, Sebastian Krieter, Thomas Thüm, Malte Lochau, and Gunter Saake. 2016. IncLing: Efficient Product-line Testing Using Incremental Pairwise Sampling. In Proceedings of the 2016 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences(GPCE 2016). New York, NY, USA, 144–155. https://doi.org/10.1145/2993236.2993253
[2]
Mustafa Al-Hajjaji, Thomas Thüm, Jens Meinicke, Malte Lochau, and Gunter Saake. 2014. Similarity-based prioritization in software product-line testing. In Proceedings of the 18th International Software Product Line Conference - Volume 1 (Florence, Italy) (SPLC ’14). Association for Computing Machinery, New York, NY, USA, 197–206. https://doi.org/10.1145/2648511.2648532
[3]
Ståle Amland. 2000. Risk-based testing:: Risk analysis fundamentals and metrics for software testing including a financial application case study. Journal of Systems and Software 53, 3 (Sept. 2000), 287–295. https://doi.org/10.1016/S0164-1212(00)00019-4
[4]
Paul Ammann and Jeff Offutt. 2016. Introduction to software testing. Cambridge University Press.
[5]
Eduard Baranov and Axel Legay. 2022. Baital: an adaptive weighted sampling platform for configurable systems. In Proceedings of the 26th ACM International Systems and Software Product Line Conference - Volume B(SPLC ’22, Vol. B). Association for Computing Machinery, New York, NY, USA, 46–49. https://doi.org/10.1145/3503229.3547030
[6]
Eduard Baranov, Axel Legay, and Kuldeep S. Meel. 2020. Baital: An Adaptive Weighted Sampling Approach for Improved t-Wise Coverage. Association for Computing Machinery, New York, NY, USA, 1114–1126. https://doi.org/10.1145/3368089.3409744
[7]
Andrea Bombarda, Silvia Bonfanti, and Angelo Gargantini. 2023. On the Reuse of Existing Configurations for Testing Evolving Feature Models. In Proceedings of the 27th ACM International Systems and Software Product Line Conference - Volume B (Tokyo, Japan) (SPLC ’23). Association for Computing Machinery, New York, NY, USA, 67–76. https://doi.org/10.1145/3579028.3609017
[8]
Vasek Chvatal. 1979. A Greedy Heuristic for the Set-Covering Problem. Mathematics of operations research 4, 3 (Aug. 1979), 233–235. https://doi.org/10.1287/moor.4.3.233
[9]
Myra B. Cohen, Matthew B. Dwyer, and Jiangfan Shi. 2006. Coverage and adequacy in software product line testing. In Proceedings of the ISSTA 2006 workshop on Role of software architecture for testing and analysis(ROSATEA ’06). Association for Computing Machinery, New York, NY, USA, 53–63. https://doi.org/10.1145/1147249.1147257
[10]
Myra B. Cohen, Matthew B. Dwyer, and Jiangfan Shi. 2007. Interaction testing of highly-configurable systems in the presence of constraints. In Proceedings of the 2007 international symposium on Software testing and analysis(ISSTA ’07). Association for Computing Machinery, New York, NY, USA, 129–139. https://doi.org/10.1145/1273463.1273482
[11]
Myra B. Cohen, Matthew B. Dwyer, and Jiangfan Shi. 2008. Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach. IEEE Transactions on Software Engineering 34, 5 (2008), 633–650. https://doi.org/10.1109/TSE.2008.50
[12]
Ivan do Carmo Machado, John D. McGregor, Yguaratã Cerqueira Cavalcanti, and Eduardo Santana de Almeida. 2014. On strategies for testing software product lines: A systematic literature review. Information and Software Technology 56, 10 (2014), 1183–1199. https://doi.org/10.1016/j.infsof.2014.04.002
[13]
Emelie Engström and Per Runeson. 2011. Software product line testing – A systematic mapping study. Information and Software Technology 53, 1 (2011), 2–13. https://doi.org/10.1016/j.infsof.2010.05.011
[14]
Alireza Ensan, Ebrahim Bagheri, Mohsen Asadi, Dragan Gasevic, and Yevgen Biletskiy. 2011. Goal-Oriented Test Case Selection and Prioritization for Product Line Feature Models. In 2011 Eighth International Conference on Information Technology: New Generations. 291–298. https://doi.org/10.1109/ITNG.2011.58
[15]
Axel Halin, Alexandre Nuttinck, Mathieu Acher, Xavier Devroey, Gilles Perrouin, and Benoit Baudry. 2019. Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack. Empirical Software Engineering 24, 2 (April 2019), 674–717. https://doi.org/10.1007/s10664-018-9635-4
[16]
Evelyn Nicole Haslinger, Roberto E. Lopez-Herrejon, and Alexander Egyed. 2013. Using feature model knowledge to speed up the generation of covering arrays. In Proceedings of the 7th International Workshop on Variability Modelling of Software-Intensive Systems(VaMoS ’13). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/2430502.2430524
[17]
Marc Hentze, Tobias Pett, Chico Sundermann, Sebastian Krieter, Thomas Thüm, and Ina Schaefer. 2022. Generic Solution-Space Sampling for Multi-domain Product Lines. In Proceedings of the 21st ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences(GPCE 2022). Association for Computing Machinery, New York, NY, USA, 135–147. https://doi.org/10.1145/3564719.3568695
[18]
Ruben Heradio, David Fernandez-Amoros, José A. Galindo, David Benavides, and Don Batory. 2022. Uniform and scalable sampling of highly configurable systems. Empirical Software Engineering 27, 2 (Jan. 2022), 44. https://doi.org/10.1007/s10664-021-10102-5
[19]
Tobias Heß, Tim Jannik Schmidt, Lukas Ostheimer, Sebastian Krieter, and Thomas Thüm. 2024. UnWise: High T-Wise Coverage from Uniform Sampling. In Proceedings of the 18th International Working Conference on Variability Modelling of Software-Intensive Systems(VaMoS ’24). Association for Computing Machinery, New York, NY, USA, 37–45. https://doi.org/10.1145/3634713.3634716
[20]
Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. 2012. An algorithm for generating t-wise covering arrays from large feature models. In Proceedings of the 16th International Software Product Line Conference - Volume 1(SPLC ’12). Association for Computing Machinery, New York, NY, USA, 46–55. https://doi.org/10.1145/2362536.2362547
[21]
Martin Fagereng Johansen, Øystein Haugen, Franck Fleurey, Anne Grete Eldegard, and Torbjørn Syversen. 2012. Generating Better Partial Covering Arrays by Modeling Weights on Sub-Product Lines. In Model Driven Engineering Languages and Systems, Robert B. France, Jürgen Kazmeier, Ruth Breu, and Colin Atkinson (Eds.). Springer, Berlin, Heidelberg, 269–284. https://doi.org/10.1007/978-3-642-33666-9_18
[22]
Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. 2011. Properties of Realistic Feature Models Make Combinatorial Testing of Product Lines Feasible. In Model Driven Engineering Languages and Systems, Jon Whittle, Tony Clark, and Thomas Kühne (Eds.). Springer, Berlin, Heidelberg, 638–652. https://doi.org/10.1007/978-3-642-24485-8_47
[23]
Chang Hwan Peter Kim, Don S. Batory, and Sarfraz Khurshid. 2011. Reducing combinatorics in testing product lines. In Proceedings of the tenth international conference on Aspect-oriented software development(AOSD ’11). Association for Computing Machinery, New York, NY, USA, 57–68. https://doi.org/10.1145/1960275.1960284
[24]
Chang Hwan Peter Kim, Eric Bodden, Don Batory, and Sarfraz Khurshid. 2010. Reducing Configurations to Monitor in a Software Product Line. In Runtime Verification, Howard Barringer, Ylies Falcone, Bernd Finkbeiner, Klaus Havelund, Insup Lee, Gordon Pace, Grigore Roşu, Oleg Sokolsky, and Nikolai Tillmann (Eds.). Springer, Berlin, Heidelberg, 285–299. https://doi.org/10.1007/978-3-642-16612-9_22
[25]
Matthias Kowal, Sandro Schulze, and Ina Schaefer. 2013. Towards efficient SPL testing by variant reduction. In Proceedings of the 4th international workshop on Variability & composition(VariComp ’13). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/2451617.2451619
[26]
Sebastian Krieter. 2020. Large-scale T-wise interaction sampling using YASA. In Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A - Volume A(SPLC ’20). Association for Computing Machinery, New York, NY, USA, 1–4. https://doi.org/10.1145/3382025.3414989
[27]
Sebastian Krieter, Thomas Thüm, Sandro Schulze, Gunter Saake, and Thomas Leich. 2020. YASA: yet another sampling algorithm. In Proceedings of the 14th International Working Conference on Variability Modelling of Software-Intensive Systems(VaMoS ’20). Association for Computing Machinery, New York, NY, USA, Article 4, 10 pages. https://doi.org/10.1145/3377024.3377042
[28]
D Richard Kuhn, Raghu N Kacker, and Yu Lei. 2013. Introduction to combinatorial testing. CRC press.
[29]
Remo Lachmann, Simon Beddig, Sascha Lity, Sandro Schulze, and Ina Schaefer. 2017. Risk-based integration testing of software product lines. In Proceedings of the 11th International Workshop on Variability Modelling of Software-Intensive Systems(VaMoS ’17). Association for Computing Machinery, New York, NY, USA, 52–59. https://doi.org/10.1145/3023956.3023958
[30]
Jihyun Lee, Sungwon Kang, and Danhyung Lee. 2012. A survey on software product line testing. In Proceedings of the 16th International Software Product Line Conference - Volume 1(SPLC ’12). Association for Computing Machinery, New York, NY, USA, 31–40. https://doi.org/10.1145/2362536.2362545
[31]
Roberto E. Lopez-Herrejon, Stefan Fischer, Rudolf Ramler, and Aalexander Egyed. 2015. A first systematic mapping study on combinatorial interaction testing for software product lines. In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 1–10. https://doi.org/10.1109/ICSTW.2015.7107435
[32]
Chuan Luo, Binqi Sun, Bo Qiao, Junjie Chen, Hongyu Zhang, Jinkun Lin, Qingwei Lin, and Dongmei Zhang. 2021. LS-sampling: an effective local search based sampling approach for achieving high t-wise coverage. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering(ESEC/FSE 2021). Association for Computing Machinery, New York, NY, USA, 1081–1092. https://doi.org/10.1145/3468264.3468622
[33]
Dusica Marijan, Arnaud Gotlieb, Sagar Sen, and Aymeric Hervieu. 2013. Practical pairwise testing for software product lines. In Proceedings of the 17th International Software Product Line Conference(SPLC ’13). Association for Computing Machinery, New York, NY, USA, 227–235. https://doi.org/10.1145/2491627.2491646
[34]
John D. McGregor. 2010. Testing a Software Product Line. In Testing Techniques in Software Engineering, Paulo Borba, Ana Cavalcanti, Augusto Sampaio, and Jim Woodcook (Eds.). Springer, Berlin, Heidelberg, 104–140. https://doi.org/10.1007/978-3-642-14335-9_4
[35]
Flávio Medeiros, Christian Kästner, Márcio Ribeiro, Sarah Nadi, and Rohit Gheyi. 2015. The Love/Hate Relationship with the C Preprocessor: An Interview Study. In 29th European Conference on Object-Oriented Programming (ECOOP 2015). Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, 495–518.
[36]
Flávio Medeiros, Christian Kästner, Márcio Ribeiro, Rohit Gheyi, and Sven Apel. 2016. A comparison of 10 sampling algorithms for configurable systems. In Proceedings of the 38th International Conference on Software Engineering(ICSE ’16). Association for Computing Machinery, New York, NY, USA, 643–654. https://doi.org/10.1145/2884781.2884793
[37]
Bartosz Michalik and Danny Weyns. 2011. Towards a Solution for Change Impact Analysis of Software Product Line Products. In 2011 Ninth Working IEEE/IFIP Conference on Software Architecture. 290–293. https://doi.org/10.1109/WICSA.2011.45
[38]
Daniel-Jesus Munoz, Jeho Oh, Mónica Pinto, Lidia Fuentes, and Don Batory. 2019. Uniform Random Sampling Product Configurations of Feature Models That Have Numerical Features. In Proceedings of the 23rd International Systems and Software Product Line Conference - Volume A(SPLC ’19). Association for Computing Machinery, New York, NY, USA, 289–301. https://doi.org/10.1145/3336294.3336297
[39]
Jeho Oh, Don Batory, Margaret Myers, and Norbert Siegmund. 2017. Finding Near-Optimal Configurations in Product Lines by Random Sampling. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering(ESEC/FSE 2017). Association for Computing Machinery, New York, NY, USA, 61–71. https://doi.org/10.1145/3106237.3106273
[40]
Jeho Oh, Paul Gazzillo, and Don Batory. 2019. t-wise Coverage by Uniform Sampling. In Proceedings of the 23rd International Systems and Software Product Line Conference - Volume A(SPLC ’19). Association for Computing Machinery, New York, NY, USA, 84–87. https://doi.org/10.1145/3336294.3342359
[41]
Gilles Perrouin, Sagar Sen, Jacques Klein, Benoit Baudry, and Yves le Traon. 2010. Automated and Scalable T-wise Test Case Generation Strategies for Software Product Lines. In Verification and Validation 2010 Third International Conference on Software Testing. 459–468. https://doi.org/10.1109/ICST.2010.43 ISSN: 2159-4848.
[42]
Tobias Pett, Domenik Eichhorn, and Ina Schaefer. 2020. Risk-based compatibility analysis in automotive systems engineering. In Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings(MODELS ’20). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3417990.3421263
[43]
Tobias Pett, Tobias Heß, Sebastian Krieter, Thomas Thüm, and Ina Schaefer. 2023. Continuous T-Wise Coverage. In Proceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A (Tokyo, Japan) (SPLC ’23, Vol. A). Association for Computing Machinery, New York, NY, USA, 87–98. https://doi.org/10.1145/3579027.3608980
[44]
Jiangfan Shi, Myra B. Cohen, and Matthew B. Dwyer. 2012. Integration Testing of Software Product Lines Using Compositional Symbolic Execution. In Fundamental Approaches to Software Engineering, Juan de Lara and Andrea Zisman (Eds.). Springer, Berlin, Heidelberg, 270–284. https://doi.org/10.1007/978-3-642-28872-2_19
[45]
Reinhard Tartler, Daniel Lohmann, Christian Dietrich, Christoph Egger, and Julio Sincero. 2011. Configuration coverage in the analysis of large-scale system software. In Proceedings of the 6th Workshop on Programming Languages and Operating Systems(PLOS ’11). Association for Computing Machinery, New York, NY, USA, 1–5. https://doi.org/10.1145/2039239.2039242
[46]
Mahsa Varshosaz, Mustafa Al-Hajjaji, Thomas Thüm, Tobias Runge, Mohammad Reza Mousavi, and Ina Schaefer. 2018. A Classification of Product Sampling for Software Product Lines. In Proceedings of the 22nd International Systems and Software Product Line Conference - Volume 1(SPLC ’18). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3233027.3233035
Index Terms
- MulTi-Wise Sampling: Trading Uniform T-Wise Feature Interaction Coverage for Smaller Samples
Recommendations
UnWise: High T-Wise Coverage from Uniform Sampling
VaMoS '24: Proceedings of the 18th International Working Conference on Variability Modelling of Software-Intensive SystemsConfiguration spaces of industrial product lines are typically too large to be tested exhaustively. Therefore, testing in practice is often carried out on samples, sets of configurations which satisfy the requirements of the testing scenario. For t-wise ...
Continuous T-Wise Coverage
SPLC '23: Proceedings of the 27th ACM International Systems and Software Product Line Conference - Volume AQuality assurance for highly configurable systems uses t-wise feature interaction coverage as a metric to measure the quality of selected samples for testing. Achieving t-wise feature interaction coverage requires testing many configurations, often ...
Baital: an adaptive weighted sampling approach for improved t-wise coverage
ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software EngineeringThe rise of highly configurable complex software and its widespread usage requires design of efficient testing methodology. t-wise coverage is a leading metric to measure the quality of the testing suite and the underlying test generation engine. While ...
Comments
Information & Contributors
Information
Published In
September 2024
103 pages
ISBN:9798400705939
DOI:10.1145/3646548
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 02 September 2024
Check for updates
Badges
Author Tags
Qualifiers
- Short-paper
- Research
- Refereed limited
Funding Sources
- DFG, German Research Foundation
- Ministry of Science, Research and Arts of the Federal State of Baden-Württemberg
Conference
SPLC '24
Sponsor:
SPLC '24: 28th ACM International Systems and Software Product Line Conference
September 2 - 6, 2024
Dommeldange, Luxembourg
Acceptance Rates
Overall Acceptance Rate 167 of 463 submissions, 36%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 78Total Downloads
- Downloads (Last 12 months)78
- Downloads (Last 6 weeks)26
Reflects downloads up to 06 Jan 2025
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in