Combinatorial Transition Testing in Dynamically Adaptive Systems

Due to the large number of possible interactions and transitions among features in dynamically adaptive systems, testing such systems poses significant challenges. To verify that such systems behave correctly, the technique of combinatorial interaction testing (CIT) can be used to create concise test suites covering all valid pairs of features of such systems. However, while CIT claims to find all errors caused by two features, we show that it does not cover certain errors occurring only for specific transitions between two features. To address this issue we study in depth the complementary technique of Combinatorial Transition Testing (CTT). From an initial generation algorithm that combines both interaction and transition coverage, we propose an optimised version that reduces the size of generated test suites by ∼ 30%, reconfiguration cost by ∼ 27% and drastically stabilises these results. After multiple generations, the standard deviation on the sizes of generated test suites is reduced by ∼ 81%. Based on a comprehensive analysis over a large number of feature models, we also conclude that the size of CTT-generated test suites is linearly correlated to CIT-generated ones and that combinatorial transition testing also grows logarithmically in the number of features.