research-article

Evaluation of Open-Source IDE Plugins for Detecting Security Vulnerabilities

Authors:

Magnus Melseth KarlsenAuthors Info & Claims

EASE '19: Proceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering

Pages 200 - 209

https://doi.org/10.1145/3319008.3319011

Published: 15 April 2019 Publication History

Abstract

Securing information systems has become a high priority as our reliance on them increases. Global multi-billion dollar companies have their critical information regularly exposed, costing them money and impairing their users' privacy. To defend against security breaches, IDE-integrated plugins to detect and remove security vulnerabilities in the first place are being used more frequently. More information about these plugins is needed in order to improve the state of the art within the field. Five open-source IDE plugins which can identify and report vulnerabilities are evaluated. We evaluate and compare how many categories of vulnerabilities the plugins can detect, how well the plugins detect the vulnerabilities, and how user-friendly the output of the plugin is to the developers. Our results show that certain vulnerabilities such as injection and broken access control are vastly covered by most plugins, while others have been completely ignored. A discrepancy between the claimed and actually confirmed coverage of the plugins is discovered, underlining the importance of this research. High false positive rate and obvious limitations in usability show that more work is needed before these plugins can be widely used and relied upon in a corporate setting.

References

[1]

Aniqua Z. Baset and Tamara Denning. 2017. IDE Plugins for Detecting Input-Validation Vulnerabilities. In 2017 IEEE Security and Privacy Workshops (SPW). 143--146.

[2]

Bernhard J. Berger. 2013. lapse-plus. (2013). https://github.com/bergerbd/lapse-plus/

[3]

Thomas Charest, Nick Rodgers, and Yan Wu. 2016. Comparison of Static Analysis Tools for Java Using the Juliet Test Suite. In Proceedings of the 11th International Conference on Cyber Warfare and Security, ICCWS 2016. 431--438.

[4]

Maria Christakis and Christian Bird. 2016. What developers want and need from program analysis: an empirical study. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering - ASE 2016. ACM Press, New York, New York, USA, 332--343.

Digital Library

[5]

Aurelien Delaitre, Bertrand Stivalet, Paul E. Black, Vadim Okun, Athos Ribeiro, and Terry S. Cohen. 2018. SATE V Report: Ten Years of Static Analysis Tool Expositions. Technical Report. National Institute of Standards and Technology.

[6]

Find Security Bugs. 2018. Find Security Bugs - The SpotBugs plugin for security audits of Java web applications. (2018). https://find-sec-bugs.github.io

[7]

Sarah Heckman and Laurie Williams. 2008. On establishing a benchmark for evaluating static analysis alert prioritization and classification techniques. In Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement - ESEM '08. ACM Press, New York, New York, USA, 41.

Digital Library

[8]

David Hovemeyer and William Pugh. 2004. Finding Bugs is Easy. SIGPLAN Not. 39, 12 (2004), 92--106.

Digital Library

[9]

Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. 2013. Why don't software developers use static analysis tools to find bugs?. In Proceedings of the 2013 International Conference on Software Engineering. IEEE Press, San Francisco, CA, USA, 672--681. https://dl.acm.org/citation.cfm?id=2486877

Digital Library

[10]

Benjamin Livshits. 2006. Stanford SecuriBench Micro. (2006). https://suif.stanford.edu/

[11]

MITRE. 2018. CWE VIEW: Research Concepts. (2018). https://cwe.mitre.org/data/definitions/1000.html

[12]

NIST. 2017. Test Suites. (2017). https://samate.nist.gov/SRD/testsuite.php

[13]

NSA. 2012. Juliet Test Suite v1.2 for Java User Guide. (2012).

[14]

OWASP. 2016. OWASP ASIDE Project. (2016). https://www.owasp.org/index.php/OWASP_ASIDE_Project

[15]

OWASP. 2017. OWASP LAPSE Project. (2017). https://www.owasp.org/index.php/OWASP_LAPSE_Project

[16]

OWASP. 2017. OWASP Top 10 - 2017 The Ten Most Critical Web Application Security Risks. (2017).

[17]

OWASP. 2018. OWASP Benchmark Project. (2018). https://www.owasp.org/index.php/Benchmark

[18]

OWASP. 2018. OWASP WebGoat Project. (2018). https://www.owasp.org/index.php/Category:OWASP_WebGoat_Project

[19]

Tosin D. Oyetoyan, Bisera Milosheska, Mari Grini, and Daniela S. Cruzes. 2018. Myths and Facts About Static Application Security Testing Tools: An Action Research at Telenor Digital. In Agile Processes in Software Engineering and Extreme Programming. Springer International Publishing, 86--103.

[20]

T. J. Robertson, Shrinu Prabhakararao, Margaret Burnett, Curtis Cook, Joseph R. Ruthruff, Laura Beckwith, and Amit Phalgune. 2004. Impact of interruption style on end-user debugging. In Proceedings of the 2004 conference on Human factors in computing systems - CHI '04, Vol. 6. ACM Press, New York, New York, USA, 287--294.

Digital Library

[21]

Nick Rutar, Christian B. Almazan, and Jeffrey S. Foster. 2004. A Comparison of Bug Finding Tools for Java. 15th International Symposium on Software Reliability Engineering (2004), 245--256.

Digital Library

[22]

Caitlin Sadowski, Jeffrey van Gogh, Ciera Jaspan, Emma Soderberg, and Collin Winter. 2015. Tricorder: Building a Program Analysis Ecosystem. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. IEEE, 598--608.

Digital Library

[23]

Luciano Sampaio. 2016. TCM_Plugin. (2016). https://github.com/lsampaioweb/TCM_Plugin

[24]

Luciano Sampaio and Alessandro Garcia. 2016. Exploring context-sensitive data flow analysis for early vulnerability detection. Journal of Systems and Software 113 (2016), 337--361.

Digital Library

[25]

SpotBugs Team. 2018. SpotBugs Eclipse plugin. (2018). https://marketplace.eclipse.org/content/spotbugs-eclipse-plugin

[26]

TIOBE. 2018. TIOBE Index for November 2018. (2018). https://www.tiobe.com/tiobe-index/

[27]

Jing Xie, Bill Chu, Heather R. Lipford, and John T. Melton. 2011. ASIDE: IDE Support for Web Application Security. In Proceedings of the 27th Annual Computer Security Applications Conference (ACSAC '11). ACM, New York, NY, USA, 267--276.

Digital Library

[28]

Jun Zhu. 2013. ASIDE-Education. (2013). https://github.com/JunZhuSecurity/ASIDE-Education

Cited By

Esposito MFalaschi VFalessi D(2024)An Extensive Comparison of Static Application Security Testing ToolsProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661199(69-78)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661199
Liu JLiu SChen JDig DBryksin TGolubev YBezzubov A(2024)I3DE: An IDE for Inspecting Inconsistencies in PL/SQL CodeProceedings of the 1st ACM/IEEE Workshop on Integrated Development Environments10.1145/3643796.3648461(74-75)Online publication date: 20-Apr-2024
https://dl.acm.org/doi/10.1145/3643796.3648461
Alqaradaghi MKozsik T(2024)Comprehensive Evaluation of Static Analysis Tools for Their Performance in Finding Vulnerabilities in Java CodeIEEE Access10.1109/ACCESS.2024.338995512(55824-55842)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3389955
Show More Cited By

Recommendations

A transformer-based IDE plugin for vulnerability detection
ASE '22: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering

Automatic vulnerability detection is of paramount importance to promote the security of an application and should be exercised at the earliest stages within the software development life cycle (SDLC) to reduce the risk of exposure. Despite the ...
Capturing and Exploiting IDE Interactions
Onward! 2014: Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software

Integrated development environments (IDEs) dominate the production and maintenance of software. Developers interact intensively with their IDEs while working. These interactions reflect a developer's thought process and work habits. By capturing and ...
Detecting Blind Cross-Site Scripting Attacks Using Machine Learning
SPML '18: Proceedings of the 2018 International Conference on Signal Processing and Machine Learning

Cross-site scripting (XSS) is a scripting attack targeting web applications by injecting malicious scripts into web pages. Blind XSS is a subset of stored XSS, where an attacker blindly deploys malicious payloads in web pages that are stored in a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

EASE '19: Proceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering

April 2019

345 pages

ISBN:9781450371452

DOI:10.1145/3319008

Program Chairs:
Shaukat Ali,
Vahid Garousi

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

IT University of Copenhagen

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 April 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

EASE '19

EASE '19: Evaluation and Assessment in Software Engineering

April 15 - 17, 2019

Copenhagen, Denmark

Acceptance Rates

EASE '19 Paper Acceptance Rate 20 of 73 submissions, 27%;

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
469
Total Downloads

Downloads (Last 12 months)52
Downloads (Last 6 weeks)3

Reflects downloads up to 16 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Esposito MFalaschi VFalessi D(2024)An Extensive Comparison of Static Application Security Testing ToolsProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661199(69-78)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661199
Liu JLiu SChen JDig DBryksin TGolubev YBezzubov A(2024)I3DE: An IDE for Inspecting Inconsistencies in PL/SQL CodeProceedings of the 1st ACM/IEEE Workshop on Integrated Development Environments10.1145/3643796.3648461(74-75)Online publication date: 20-Apr-2024
https://dl.acm.org/doi/10.1145/3643796.3648461
Alqaradaghi MKozsik T(2024)Comprehensive Evaluation of Static Analysis Tools for Their Performance in Finding Vulnerabilities in Java CodeIEEE Access10.1109/ACCESS.2024.338995512(55824-55842)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3389955
Saroar SAhmed WOnagh ENayebi M(2024)GitHub marketplace for automation and innovation in software productionInformation and Software Technology10.1016/j.infsof.2024.107522175(107522)Online publication date: Nov-2024
https://doi.org/10.1016/j.infsof.2024.107522
Li KChen SFan LFeng RLiu HLiu CLiu YChen YChandra SBlincoe KTonella P(2023)Comparison and Evaluation on Static Application Security Testing (SAST) Tools for JavaProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616262(921-933)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616262
Miltenberger MArzt SHolzinger PNäumann J(2023)Benchmarking the BenchmarksProceedings of the 2023 ACM Asia Conference on Computer and Communications Security10.1145/3579856.3582830(387-400)Online publication date: 10-Jul-2023
https://dl.acm.org/doi/10.1145/3579856.3582830
Anis Al Hilmi MRaswa Robiyanto ROranova Siahaan DPuspaningrum ASusanti Samosir H(2023)A Static IDE Plugin to Detect Security Hotspot for Laravel Framework Based Web Application2023 IEEE International Conference on Data and Software Engineering (ICoDSE)10.1109/ICoDSE59534.2023.10291941(1-6)Online publication date: 7-Sep-2023
https://doi.org/10.1109/ICoDSE59534.2023.10291941
Grünewald E(2022)Cloud Native Privacy Engineering through DevPrivOpsPrivacy and Identity Management. Between Data Protection and Security10.1007/978-3-030-99100-5_10(122-141)Online publication date: 31-Mar-2022
https://doi.org/10.1007/978-3-030-99100-5_10
Beba SKarlsen MLi JZhang B(2021)Critical Understanding of Security Vulnerability Detection Plugin Evaluation Reports2021 28th Asia-Pacific Software Engineering Conference (APSEC)10.1109/APSEC53868.2021.00035(275-284)Online publication date: Dec-2021
https://doi.org/10.1109/APSEC53868.2021.00035
Javed YArian QAlenezi M(2021)SecurityGuard: An Automated Secure Coding FrameworkIntelligent Technologies and Applications10.1007/978-3-030-71711-7_25(303-310)Online publication date: 15-Mar-2021
https://doi.org/10.1007/978-3-030-71711-7_25
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents