research-article

The impact of source test case selection on the effectiveness of metamorphic testing

Authors:

Arlinta Christy Barus,

Tsong Yueh Chen,

Heinz W. SchmidtAuthors Info & Claims

MET '16: Proceedings of the 1st International Workshop on Metamorphic Testing

Pages 5 - 11

https://doi.org/10.1145/2896971.2896977

Published: 14 May 2016 Publication History

Abstract

Metamorphic Testing (MT) aims to alleviate the oracle problem. In MT, testers define metamorphic relations (MRs) which are used to generate new test cases (referred to as follow-up test cases) from the available test cases (referred to as source test cases). Both source and follow-up test cases are executed and their outputs are verified against the relevant MRs, of which any violation implies that the software under test is faulty. So far, the research on the effectiveness of MT has been focused on the selection of better MRs (that is, MRs that are more likely to be violated). In addition to MR selection, the source and follow-up test cases may also affect the effectiveness of MT. Since follow-up test cases are defined by the source test cases and MRs, selection of source test cases will then affect the effectiveness of MT. However, in existing MT studies, random testing is commonly adopted as the test case selection strategy for source test cases. This study aims to investigate the impact of source test cases on the effectiveness of MT. Since Adaptive Random Testing (ART) has been developed as an enhancement to Random Testing (RT), this study will focus on comparing the performance of RT and ART as source test case selection strategies on the effectiveness of MT. Experiment results show that ART outperforms RT on enhancing the effectiveness of MT.

References

[1]

A. C. Barus. Testing Heuristic Method, Software Engineering Techniques, Lecture Notes in Computer Science volume 4980, 2011, pp. 246--260.

Digital Library

[2]

A. C. Barus. An In-depth Study of Adaptive Random Testing for Testing Program with Complex Input Types, Ph.D. dissertation, Swinburne University of Technology, 2010.

[3]

E. J. Weyuker, On testing non-testable programs, The Computer Journal, vol. 25, No. 4, 1982, pp. 465--470.

[4]

T. Y. Chen, S. Cheung, and S. Yiu, Metamorphic testing: a new approach for generating next test cases, Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong, Tech. Rep. HKUST-CS98-01, 1998.

[5]

T. Y. Chen, H. Leung, and I. K. Mak, 2004. Adaptive Random Testing, In Proceedings of the 9th Asian Computing Science Conference, ser. Lecture Notes in Computer Science, vol. 3321, 2004, pp. 320--329.

Digital Library

[6]

W. K. Chan, S. C. Cheung, and K. R. P. H. Leung. A Metamorphic Testing Approach for Online Testing of Service-Oriented Software Applications, a Special Issue on Service Engineering of International Journal of Web Services Research, vol. 4, No. 2, 2007, pp. 60--80.

[7]

T. Y. Chen, D. H. Huang, T. H. Tse, and Z. Q. Zhou. Case studies on the selection of useful relations in metamorphic testing, in Proceedings of the 4th lbero-American Symposium on Software Engineering and Knowledge Engineer-ing (JIISIC). Polytechnic University of Madrid, 2004, pp. 569--583.

[8]

T. Y. Chen, T. H. Tse, and Z. Q. Zhou. Semi-proving: an integrated method based on global symbolic evaluation and metamorphic testing, in Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). ACM Press, New York, 2002, pp. 191--195.

Digital Library

[9]

T. Y. Chen, T. H. Tse, and Y. T. Yu, Proportional sampling strategy: A compendium and some insights, Information and Software Technology, vol. 58, No. 1, 2001, pp. 65--81.

Digital Library

[10]

F.-C. Kuo, On adaptive random testing, Ph.D. dissertation, Swinburne University of Technology, 2006.

[11]

P. E. Ammann and J. C. Knight, Data diversity: an approach to software fault tolerance, IEEE Transactions on Computers, vol. 37, No. 4, 1998, pp. 418--425.

Digital Library

[12]

P. G. Bishop, The variation of software survival times for diffrent operational input profiles, in FTSC-23. Digitest of Papers, the Twenty-Third International Symposium on Fault-Tolerant Computing. IEEE Computer Society Press, 1993, pp. 98--107.

[13]

R. Merkel, Analysis and enhancements of adaptive random testing, Ph.D. dissertation, Swinburne University of Technology, 2005.

[14]

M. Hutchins, H. Foster, T. Goradia, and T. Ostrand, Experiments on the effectiveness of dataflow and controlfow-based test adequacy criteria, in Proceedings of International Conference of Software Engineering, 2004, pp. 191--200.

Digital Library

[15]

G. Rothermel, E. Sebastian, H. Do, and A. Kinneer, Software-artifact infrastructure repository, {Online}. Available: http://sir.unl.edu.

[16]

S. G. Elbaum, A. G. Malishevsky, and G. Rothermel, Prioritizing test cases for regression testing, in In Proceedings of the 2000 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2000), ACM SIGSOFT Software Engineering Notes, 2000, pp. 102--112.

Digital Library

[17]

T. J. Ostrand and M. J. Balcer, The category-partition method for specifying and generating functional tests, Communications of the ACM, vol. 31, No. 6, June 1988.

Digital Library

[18]

X. Xie, W. E. Wong, T. Y. Chen, B. W. Xu, Metamorphic Slice: An Application in Spectrum-based Fault Localization, Information and Software Technology, 2013, vol. 55, No. 5, pp. 866--879.

Digital Library

[19]

T. Y. Chen, F.-C. Kuo, R. Merkel, On the statistical properties of testing effectivenes measures, Journal of Systems and Software, vol. 79, No. 5, 2006, pp. 591--601.

Digital Library

[20]

A. C. Barus, T. Y. Chen, F.-C. Kuo, H. Liu, R. Merkel, G. Rothermel, A Novel Linear-Order Algorithm for Adaptive Random Testing of Programs with Non-Numeric Inputs, Technical Report TR-UNL-CSE-2014-0004, University of Nebraska Lincoln, US, 2014.

[21]

H. Liu, F.-C. Kuo, D. Towey, T. Y. Chen, How Effectively Does Metamorphic Testing Alleviate the Oracle Problem?, IEEE Transactions on Software Engineering, vol. 40, No. 1, January 2014.

Digital Library

[22]

Z. Q. Zhou, S. Zhang, M. Hagenbuchner, T. H. Tse, F. -C. Kuo, T. Y. Chen, Automated functional testing of online search services, Software Testing, Verification and Reliability, vol. 22, No. 4, 2012, pp. 221--243.

Digital Library

[23]

Y. Cao, Z. Q. Zhou, T. Y. Chen, On the Correlation between the Effectiveness of Metamorphic Relations and Dissimilarities of Test Case Executions, in Proceedings of the 13th International Conference of Quality Software, 2013, pp. 153--162.

Digital Library

[24]

The GNU Project, Grep home page, accessed 2015-11-01 {Online}, Available: http://www.gnu.org/software/grep/manual/grep.html.

[25]

K.-P. Chan, T. Y. Chen, and D. Towey, Forgetting test cases, In Proceedings of the 30th Annual International Computer Software and Applications Conference, COMPSAC 06, 2006, pp 485--494.

Digital Library

Cited By

Bayati Chaleshtari NMarquer YPastore FBriand L(2024)AIM: Automated Input Set Minimization for Metamorphic Security TestingIEEE Transactions on Software Engineering10.1109/TSE.2024.3488525(1-31)Online publication date: 2024
https://doi.org/10.1109/TSE.2024.3488525
Hong TQian LFang JWang TZhu ZQin XNai WYang Z(2024)Metamorphic relationship generation based on Faure Sequence Initialized Slime Mould Algorithm2024 IEEE 14th International Conference on Electronics Information and Emergency Communication (ICEIEC)10.1109/ICEIEC61773.2024.10561718(235-239)Online publication date: 24-May-2024
https://doi.org/10.1109/ICEIEC61773.2024.10561718
Ying ZTowey DChen TZhou Z(2024)MT-PART: Metamorphic-Testing-Based Adaptive Random Testing Through Partitioning2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC61105.2024.00158(1184-1193)Online publication date: 2-Jul-2024
https://doi.org/10.1109/COMPSAC61105.2024.00158
Show More Cited By

Recommendations

Feedback-Directed Metamorphic Testing
Over the past decade, metamorphic testing has gained rapidly increasing attention from both academia and industry, particularly thanks to its high efficacy on revealing real-life software faults in a wide variety of application domains. On the basis of a ...
Metamorphic Testing: A Review of Challenges and Opportunities

Metamorphic testing is an approach to both test case generation and test result verification. A central element is a set of metamorphic relations, which are necessary properties of the target function or algorithm in relation to multiple inputs and ...
Fault detection effectiveness of source test case generation strategies for metamorphic testing
MET '18: Proceedings of the 3rd International Workshop on Metamorphic Testing

Metamorphic testing is a well known approach to tackle the oracle problem in software testing. This technique requires the use of source test cases that serve as seeds for the generation of follow-up test cases. Systematic design of test cases is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MET '16: Proceedings of the 1st International Workshop on Metamorphic Testing

May 2016

59 pages

ISBN:9781450341639

DOI:10.1145/2896971

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS\DATC: IEEE Computer Society
TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '16

Sponsor:

ACM
SIGSOFT
IEEE-CS\DATC
TCSE

ICSE '16: 38th International Conference on Software Engineering

May 14 - 22, 2016

Texas, Austin

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
189
Total Downloads

Downloads (Last 12 months)26
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bayati Chaleshtari NMarquer YPastore FBriand L(2024)AIM: Automated Input Set Minimization for Metamorphic Security TestingIEEE Transactions on Software Engineering10.1109/TSE.2024.3488525(1-31)Online publication date: 2024
https://doi.org/10.1109/TSE.2024.3488525
Hong TQian LFang JWang TZhu ZQin XNai WYang Z(2024)Metamorphic relationship generation based on Faure Sequence Initialized Slime Mould Algorithm2024 IEEE 14th International Conference on Electronics Information and Emergency Communication (ICEIEC)10.1109/ICEIEC61773.2024.10561718(235-239)Online publication date: 24-May-2024
https://doi.org/10.1109/ICEIEC61773.2024.10561718
Ying ZTowey DChen TZhou Z(2024)MT-PART: Metamorphic-Testing-Based Adaptive Random Testing Through Partitioning2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC61105.2024.00158(1184-1193)Online publication date: 2-Jul-2024
https://doi.org/10.1109/COMPSAC61105.2024.00158
Ying ZTowey DGraham Bellotti AQuan Zhou Z(2024)MRGS‐ART: Metamorphic Relation and Group Selection Based on Adaptive Random TestingSoftware Testing, Verification and Reliability10.1002/stvr.190835:1Online publication date: 27-Nov-2024
https://doi.org/10.1002/stvr.1908
Sun CDai HLiu HChen T(2023)Feedback-Directed Metamorphic TestingACM Transactions on Software Engineering and Methodology10.1145/353331432:1(1-34)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3533314
Saha PIzurieta CKanewala U(2023)A Test Suite Minimization Technique for Testing Numerical Programs2023 IEEE/ACIS 21st International Conference on Software Engineering Research, Management and Applications (SERA)10.1109/SERA57763.2023.10197757(61-68)Online publication date: 23-May-2023
https://doi.org/10.1109/SERA57763.2023.10197757
Jafari FNadeem AZaman Q(2022)Evaluation of Metamorphic Testing for Edge Detection in MRI Brain DiagnosticsApplied Sciences10.3390/app1217868412:17(8684)Online publication date: 30-Aug-2022
https://doi.org/10.3390/app12178684
Ayerdi JArrieta APobee EArratibel M(2022)Multi-Objective Metamorphic Test Case Selection: an Industrial Case Study (Practical Experience Report)2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE55969.2022.00058(541-552)Online publication date: Oct-2022
https://doi.org/10.1109/ISSRE55969.2022.00058
Ying ZBellotti ATowey DChen TZhou Z(2022)Using Metamorphic Relation Violation Regions to Support a Simulation Framework for the Process of Metamorphic Testing2022 IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC54236.2022.00274(1722-1727)Online publication date: Jun-2022
https://doi.org/10.1109/COMPSAC54236.2022.00274
Sun CLiu BFu ALiu YLiu H(2022)Path-directed source test case generation and prioritization in metamorphic testingJournal of Systems and Software10.1016/j.jss.2021.111091183(111091)Online publication date: Jan-2022
https://doi.org/10.1016/j.jss.2021.111091
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents