research-article

Mart: a mutant generation tool for LLVM

Authors:

Thierry Titcheu Chekam,

Mike Papadakis,

Yves Le TraonAuthors Info & Claims

ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Pages 1080 - 1084

https://doi.org/10.1145/3338906.3341180

Published: 12 August 2019 Publication History

Abstract

Program mutation makes small syntactic alterations to programs' code in order to artificially create faulty programs (mutants). Mutants creation (generation) tools are often characterized by their mutation operators and the way they create and represent the mutants. This paper presents Mart, a mutants generation tool, for LLVM bitcode, that supports the fine-grained definition of mutation operators (as matching rule - replacing pattern pair; uses 816 defined pairs by default) and the restriction of the code parts to mutate. New operators are implemented in Mart by implementing their matching rules and replacing patterns. Mart also implements in-memory Trivial Compiler Equivalence to eliminate equivalent and duplicate mutants during mutants generation. Mart generates mutant code as separated mutant files, meta-mutants file, weak mutation and mutant coverage instrumented files. Mart is publicly available (https://github.com/thierry-tct/mart). Mart has been applied to generate mutants for several research experiments and generated more than 4,000,000 mutants.

References

[1]

Marcel Böhme and Abhik Roychoudhury. 2014. CoREBench: studying complexity of regression errors. In International Symposium on Software Testing and Analysis, ISSTA ’14, San Jose, CA, USA - July 21 - 26, 2014. 105–115. 2610384.2628058

Digital Library

[2]

Timothy Alan Budd. 1980. Mutation Analysis of Program Test Data. phdthesis. Yale University, New Haven, Connecticut.

[3]

Thierry Titcheu Chekam, Mike Papadakis, Yves Le Traon, and Mark Harman. 2017. An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption. In ICSE 2017.

Digital Library

[4]

597–608.

[5]

A. Denisov and S. Pankevich. 2018. Mull It Over: Mutation Testing Based on LLVM. In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 25–31.

[6]

Farah Hariri and August Shi. 2018. SRCIROR: A Toolset for Mutation Testing of C Source Code and LLVM Intermediate Representation. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE 2018). ACM, New York, NY, USA, 860–863.

Digital Library

[7]

René Just, Bob Kurtz, and Paul Ammann. 2017. Inferring mutant utility from program context. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, Santa Barbara, CA, USA, July 10 - 14, 2017.

Digital Library

[8]

Chris Lattner and Vikram Adve. 2004. LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation. In Proceedings of the International Symposium on Code Generation and Optimization: Feedback-directed and Runtime Optimization (CGO ’04). IEEE Computer Society, Washington, DC, USA, 75–. http://dl.acm.org/citation.cfm?id=977395.977673

Digital Library

[9]

Thomas Laurent, Mike Papadakis, Marinos Kintis, Christopher Henard, Yves Le Traon, and Anthony Ventresque. 2017. Assessing and Improving the Mutation Testing Practice of PIT. In ICST 2017. 430–435.

[10]

A. Jefferson Offutt, Ammei Lee, Gregg Rothermel, Roland H. Untch, and Christian Zapf. 1996. An Experimental Determination of Sufficient Mutant Operators. ACM Trans. on Soft. Eng. & Meth. 5, 2 (April 1996), 99–118.

Digital Library

[11]

Mike Papadakis, Yue Jia, Mark Harman, and Yves Le Traon. 2015. Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique. In ICSE 2015. 936–946.

Digital Library

[12]

Mike Papadakis, Thierry Titcheu Chekam, and Yves Le Traon. 2018. Mutant Quality Indicators. In the 13th International Workshop on Mutation Analysis (Mutation 2018).

[13]

Goran Petrovic and Marko Ivankovic. 2018. State of Mutation Testing at Google. In 40th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice Track, ICSE-SEIP 2018, May 27 - 3 June 2018, Gothenburg, Sweden.

Digital Library

[14]

Shin Hwei Tan, Jooyong Yi, Yulis, Sergey Mechtaev, and Abhik Roychoudhury. 2017. Codeflaws: a programming competition benchmark for evaluating automated program repair tools. In ICSE 2017. 180–182.

Digital Library

[15]

Thierry Titcheu Chekam, Mike Papadakis, Tegawendé Bissyandé, Yves Le Traon, and Koushik Sen. 2018. Selecting Fault Revealing Mutants. arXiv preprint arXiv:1803.07901v3 (2018).

Cited By

Yoon JCha S(2024)FeatMaker: Automated Feature Engineering for Search Strategy of Symbolic ExecutionProceedings of the ACM on Software Engineering10.1145/36608151:FSE(2447-2468)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660815
Ojdanic MPapadakis MHarman MChandra SBlincoe KTonella P(2023)Keeping Mutation Test Suites Consistent and Relevant with Long-Standing MutantsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613089(2067-2071)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613089
Viganò ECornejo OPastore FBriand L(2023)Data-Driven Mutation Analysis for Cyber-Physical SystemsIEEE Transactions on Software Engineering10.1109/TSE.2022.321304149:4(2182-2201)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2022.3213041
Show More Cited By

Index Terms

Mart: a mutant generation tool for LLVM
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

The Effectiveness of T-Way Test Data Generation
SAFECOMP '08: Proceedings of the 27th international conference on Computer Safety, Reliability, and Security

This paper reports the results of a study comparing the effectiveness of automatically generated tests constructed using random and <em>t</em>-way combinatorial techniques on safety related industrial code using mutation adequacy criteria. A reference ...
Combining bug detection and test case generation
FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

Detecting bugs in software is an important software engineering activity. Static bug finding tools can assist in detecting bugs automatically, but they suffer from high false positive rates. Automatic test generation tools can generate test cases which ...
A New Mutant Generation Algorithm Based on Basic Path Coverage for Mutant Reduction
Structured Object-Oriented Formal Language and Method
Abstract
Mutation testing is a fault-based testing technique that can be used to measure the adequacy of a test set, but its application usually incurs a high cost due to the necessity of generating and executing a great number of mutants. How to reduce ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

August 2019

1264 pages

ISBN:9781450355728

DOI:10.1145/3338906

General Chairs:
Marlon Dumas
University of Tartu, Estonia
,
Dietmar Pfahl
University of Tartu, Estonia
,
Program Chairs:
Sven Apel
Saarland University, Germany
,
Alessandra Russo
Imperial College, UK

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 August 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ESEC/FSE '19

Sponsor:

SIGSOFT

ESEC/FSE '19: 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

August 26 - 30, 2019

Tallinn, Estonia

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
262
Total Downloads

Downloads (Last 12 months)28
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yoon JCha S(2024)FeatMaker: Automated Feature Engineering for Search Strategy of Symbolic ExecutionProceedings of the ACM on Software Engineering10.1145/36608151:FSE(2447-2468)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660815
Ojdanic MPapadakis MHarman MChandra SBlincoe KTonella P(2023)Keeping Mutation Test Suites Consistent and Relevant with Long-Standing MutantsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613089(2067-2071)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613089
Viganò ECornejo OPastore FBriand L(2023)Data-Driven Mutation Analysis for Cyber-Physical SystemsIEEE Transactions on Software Engineering10.1109/TSE.2022.321304149:4(2182-2201)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2022.3213041
Garg AOjdanic MDegiovanni RChekam TPapadakis MLe Traon Y(2023)Cerebro: Static Subsuming Mutant SelectionIEEE Transactions on Software Engineering10.1109/TSE.2022.314051049:1(24-43)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TSE.2022.3140510
Lévai ZMcMinn P(2023)Batching Non-Conflicting Mutations for Efficient, Safe, Parallel Mutation Analysis in Rust2023 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST57152.2023.00014(49-59)Online publication date: Apr-2023
https://doi.org/10.1109/ICST57152.2023.00014
Viganò ECornejo OPastore FBriand LGrundy J(2023)DaMAT: A Data-Driven Mutation Analysis ToolProceedings of the 45th International Conference on Software Engineering: Companion Proceedings10.1109/ICSE-Companion58688.2023.00047(165-169)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE-Companion58688.2023.00047
Vercammen SDemeyer SBorg M(2023)F-ASTMut mutation optimisations techniques using the Clang front-endSoftware Impacts10.1016/j.simpa.2023.10050016(100500)Online publication date: May-2023
https://doi.org/10.1016/j.simpa.2023.100500
Rostami TJalili S(2023)FrMi: Fault-revealing Mutant Identification using killability severityInformation and Software Technology10.1016/j.infsof.2023.107307164(107307)Online publication date: Dec-2023
https://doi.org/10.1016/j.infsof.2023.107307
Vercammen SDemeyer SBorg MPettersson NHedin G(2023)Mutation testing optimisations using the Clang front‐endSoftware Testing, Verification and Reliability10.1002/stvr.1865Online publication date: 17-Oct-2023
https://doi.org/10.1002/stvr.1865
Ojdanić MRoychoudhury ACadar CKim M(2022)Change-aware mutation testing for evolving systemsProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3558911(1785-1789)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3558911
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten