Evaluation of spectrum based fault localization tools
Authors: 
Archana, 
Ashutosh AgarwalAuthors Info & Claims
Archana, Ashutosh AgarwalAuthors Info & ClaimsArticle No.: 19, Pages 1 - 5
Abstract
Software Fault localization (SFL) is the first step in any program debugging process. For more than three decades, researchers have aggressively studied, evaluated, and proposed numerous automatic SFL techniques spanning across various families of methods such as spectrum-based, slice-based, mutation-based, etc. Another facet contributed by researchers is the practical implementation of the above techniques in the form of open-source tools, IDE plugins, extensions, etc. Examples include (but are not limited to) GZoltzar, Jaguar, and iFL4Eclipse.
Previous research has established the metrics and threshold values for the adoption of SFL techniques in real-life software development. Several attempts have been made to evaluate automatic fault repair tools, and Information Retrieval (IR) based fault localization tools. Whilst Spectrum Based Fault Localization (SBFL) remains the most contributed family of SFL methods, no studies have been found which evaluate the existing SBFL tools.
This paper presents a comparative theoretical assessment of selected SBFL tools by understanding the developmental dynamics involved in implementing them and establishing results that would guide the same in the future. Our research steps can briefly be summarized as systematic collection and filtering of SBFL tools and their research papers, developing a historical timeline for the same, and comparative theoretical analysis through the lens of software engineering.
We theoretically determined that there is a lack of rigorous testing for the scalability and correctness of SBFL tools. While some tools are extensible with respect to the underlying algorithm used for computation, none provide flexibility in choosing the coverage collection framework. While Open-source tools have been more successful, there is a general lack in maintenance and development post initial publication. A natural progression of this work is a large-scale empirical assessment of the SBFL tools.
References
[1]
[n.d.]. CharmFL. https://sed-szeged.github.io/SpectrumBasedFaultLocalization/
[2]
Shayan A Akbar and Avinash C Kak. 2020. A large-scale comparative evaluation of IR-based tools for bug localization. In Proceedings of the 17th International Conference on Mining Software Repositories. 21–31.
[3]
Aaron Ang, Alexandre Perez, Arie Van Deursen, and Rui Abreu. 2017. Revisiting the Practical Use of Automated Software Fault Localization Techniques. In 2017 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). 175–182. https://doi.org/10.1109/ISSREW.2017.68
[4]
Gergo Balogh, Ferenc Horváth, and Árpád Beszédes. 2019. Poster: Aiding Java Developers with Interactive Fault Localization in Eclipse IDE. In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST). 371–374. https://doi.org/10.1109/ICST.2019.00045
[5]
Marcel Böhme, Ezekiel O. Soremekun, Sudipta Chattopadhyay, Emamurho Ugherughe, and Andreas Zeller. 2017. Where is the Bug and How is It Fixed? An Experiment with Practitioners. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (Paderborn, Germany) (ESEC/FSE 2017). Association for Computing Machinery, New York, NY, USA, 117–128. https://doi.org/10.1145/3106237.3106255
[6]
José Campos, André Riboira, Alexandre Perez, and Rui Abreu. 2012. Gzoltar: an eclipse plug-in for testing and debugging. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering. 378–381.
[7]
Cheng Chen and Nan Wang. 2016. UnitFL: A fault localization tool integrated with unit test. In 2016 5th International Conference on Computer Science and Network Technology (ICCSNT). 136–142. https://doi.org/10.1109/ICCSNT.2016.8070135
[8]
Higor Amario de Souza, Marcos Lordello Chaim, and Fabio Kon. 2016. Spectrum-based Software Fault Localization: A Survey of Techniques, Advances, and Challenges. CoRR abs/1607.04347(2016). arxiv:1607.04347http://arxiv.org/abs/1607.04347
[9]
Thomas Durieux, Fernanda Madeiral, Matias Martinez, and Rui Abreu. 2019. Empirical review of Java program repair tools: A large-scale experiment on 2,141 bugs and 23,551 repair attempts. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 302–313.
[10]
Mojdeh Golagha, Abu Mohammed Raisuddin, Lennart Mittag, Dominik Hellhake, and Alexander Pretschner. 2018. Aletheia: A Failure Diagnosis Toolchain. In 2018 IEEE/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion). 13–16.
[11]
Carlos Gouveia, José Campos, and Rui Abreu. 2013. Using HTML5 visualizations in software fault localization. In 2013 First IEEE Working Conference on Software Visualization (VISSOFT). 1–10. https://doi.org/10.1109/VISSOFT.2013.6650539
[12]
GZoltar. [n.d.]. GZoltar/gzoltar: GZoltar - Java Library for Automatic Debugging http://www.gzoltar.com. https://github.com/GZoltar/gzoltar
[13]
Dan Hao, Lingming Zhang, Lu Zhang, Jiasu Sun, and Hong Mei. 2009. VIDA: Visual interactive debugging. In 2009 IEEE 31st International Conference on Software Engineering. IEEE, 583–586.
[14]
Ferenc Horváth, Árpád Beszédes, Béla Vancsics, Gergő Balogh, László Vidács, and Tibor Gyimóthy. 2020. Experiments with Interactive Fault Localization Using Simulated and Real Users. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). 290–300. https://doi.org/10.1109/ICSME46990.2020.00036
[15]
Ferenc Horváth, Victor Schnepper Lacerda, Árpád Beszédes, László Vidács, and Tibor Gyimóthy. 2019. A New Interactive Fault Localization Method with Context Aware User Feedback. In 2019 IEEE 1st International Workshop on Intelligent Bug Fixing (IBF). 23–28. https://doi.org/10.1109/IBF.2019.8665415
[16]
InteractiveFaultLocalization. [n.d.]. InteractiveFaultLocalization/iFL4Eclipse: iFL for Eclipse is an Eclipse plug-in for supporting interactive fault localization for Java projects developed in this environment.https://github.com/InteractiveFaultLocalization/iFL4Eclipse
[17]
James A. Jones and Mary Jean Harrold. 2005. Empirical Evaluation of the Tarantula Automatic Fault-Localization Technique. In Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering (Long Beach, CA, USA) (ASE ’05). Association for Computing Machinery, New York, NY, USA, 273–282. https://doi.org/10.1145/1101908.1101949
[18]
Amy J. Ko and Brad A. Myers. 2009. Finding Causes of Program Output with the Java Whyline. Association for Computing Machinery, New York, NY, USA, 1569–1578. https://doi.org/10.1145/1518701.1518942
[19]
Pavneet Singh Kochhar, Xin Xia, David Lo, and Shanping Li. 2016. Practitioners’ expectations on automated fault localization. In Proceedings of the 25th International Symposium on Software Testing and Analysis. 165–176.
[20]
Sangmin Park, Richard W. Vuduc, and Mary Jean Harrold. 2010. Falcon: fault localization in concurrent programs. In 2010 ACM/IEEE 32nd International Conference on Software Engineering, Vol. 1. 245–254. https://doi.org/10.1145/1806799.1806838
[21]
Chris Parnin and Alessandro Orso. 2011. Are automated debugging techniques actually helping programmers?. In Proceedings of the 2011 international symposium on software testing and analysis. 199–209.
[22]
Spencer Pearson, José Campos, René Just, Gordon Fraser, Rui Abreu, Michael D Ernst, Deric Pang, and Benjamin Keller. 2016. Evaluating & improving fault localization techniques. University of Washington Department of Computer Science and Engineering, Seattle, WA, USA, Tech. Rep. UW-CSE-16-08-03 (2016), 27.
[23]
Henrique L. Ribeiro, Roberto P. A. de Araujo, Marcos L. Chaim, Higor A. de Souza, and Fabio Kon. 2018. Jaguar: A Spectrum-Based Fault Localization Tool for Real-World Software. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST). 404–409. https://doi.org/10.1109/ICST.2018.00048
[24]
Henrique L. Ribeiro, P. A. Roberto de Araujo, Marcos L. Chaim, Higor A. de Souza, and Fabio Kon. 2019. Evaluating data-flow coverage in spectrum-based fault localization. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 1–11. https://doi.org/10.1109/ESEM.2019.8870182
[25]
Saeg. [n.d.]. saeg/jaguar: JAva code coveraGe faUlt locAlization Rank. https://github.com/saeg/jaguar
[26]
Qusay Idrees Sarhan, Attila Szatmari, Rajmond Toth, and Arpad Beszedes. 2021. CharmFL: A Fault Localization Tool for Python. arxiv:2108.12377 [cs.SE]
[27]
George Thompson and Allison K. Sullivan. 2020. ProFL: A Fault Localization Framework for Prolog. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (Virtual Event, USA) (ISSTA 2020). Association for Computing Machinery, New York, NY, USA, 561–564. https://doi.org/10.1145/3395363.3404367
[28]
Nan Wang, Zheng Zheng, Zhenyu Zhang, and Cheng Chen. 2015. FLAVS: A Fault Localization Add-In for Visual Studio. In 2015 IEEE/ACM 1st International Workshop on Complex Faults and Failures in Large Software Systems (COUFLESS). 1–6. https://doi.org/10.1109/COUFLESS.2015.8
[29]
W Eric Wong and Vidroha Debroy. 2009. A survey of software fault localization. Department of Computer Science, University of Texas at Dallas, Tech. Rep. UTDCS-45 9(2009).
[30]
Abubakar Zakari, Sai Peck Lee, Khubaib Amjad Alam, and Rodina Ahmad. 2019. Software fault localisation: a systematic mapping study. IET Software 13, 1 (2019), 60–74.
Index Terms
- Evaluation of spectrum based fault localization tools
Index terms have been assigned to the content through auto-classification.
Recommendations
Boosting spectrum-based fault localization using PageRank
ISSTA 2017: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and AnalysisManual debugging is notoriously tedious and time consuming. Therefore, various automated fault localization techniques have been proposed to help with manual debugging. Among the existing fault localization techniques, spectrum-based fault localization ...
Fault centrality: boosting spectrum-based fault localization via local influence calculation
AbstractSpectrum-Based Fault Localization (SBFL) is a widely investigated heuristic approach and a lightweight but efficient technique. Recently, to discover useful latent information for accelerating fault localization, more attention has been paid to ...
A fault localization approach based on fault propagation context
Abstract Context:Spectrum-based fault localization (SBFL) performs statistical analysis on the coverage information of failed or passed test cases. It provides the programmer with a guide for fault localization by generating a ...
Comments
Information & Contributors
Information
Published In
February 2022
235 pages
ISBN:9781450396189
DOI:10.1145/3511430
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 24 February 2022
Check for updates
Author Tags
Qualifiers
- Short-paper
- Research
- Refereed limited
Conference
ISEC 2022
ISEC 2022: 15th Innovations in Software Engineering Conference
February 24 - 26, 2022
Gandhinagar, India
Acceptance Rates
Overall Acceptance Rate 76 of 315 submissions, 24%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 167Total Downloads
- Downloads (Last 12 months)15
- Downloads (Last 6 weeks)1
Reflects downloads up to 23 Dec 2024
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format