research-article

Empirical review of automated analysis tools on 47,587 Ethereum smart contracts

Authors:

Thomas Durieux,

João F. Ferreira,

Pedro CruzAuthors Info & Claims

ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

Pages 530 - 541

https://doi.org/10.1145/3377811.3380364

Published: 01 October 2020 Publication History

Abstract

Over the last few years, there has been substantial research on automated analysis, testing, and debugging of Ethereum smart contracts. However, it is not trivial to compare and reproduce that research. To address this, we present an empirical evaluation of 9 state-of-the-art automated analysis tools using two new datasets: i) a dataset of 69 annotated vulnerable smart contracts that can be used to evaluate the precision of analysis tools; and ii) a dataset with all the smart contracts in the Ethereum Blockchain that have Solidity source code available on Etherscan (a total of 47,518 contracts). The datasets are part of SmartBugs, a new extendable execution framework that we created to facilitate the integration and comparison between multiple analysis tools and the analysis of Ethereum smart contracts. We used SmartBugs to execute the 9 automated analysis tools on the two datasets. In total, we ran 428,337 analyses that took approximately 564 days and 3 hours, being the largest experimental setup to date both in the number of tools and in execution time. We found that only 42% of the vulnerabilities from our annotated dataset are detected by all the tools, with the tool Mythril having the higher accuracy (27%). When considering the largest dataset, we observed that 97% of contracts are tagged as vulnerable, thus suggesting a considerable number of false positives. Indeed, only a small number of vulnerabilities (and of only two categories) were detected simultaneously by four or more tools.

References

[1]

Elvira Albert, Pablo Gordillo, Benjamin Livshits, Albert Rubio, and Ilya Sergey. 2018. EthIR: A Framework for High-Level Analysis of Ethereum Bytecode. In Automated Technology for Verification and Analysis, Shuvendu K. Lahiri and Chao Wang (Eds.). Springer International Publishing, Cham, 513--520.

[2]

Shaun Azzopardi, Joshua Ellul, and Gordon J. Pace. 2018. Monitoring Smart Contracts: ContractLarva and Open Challenges Beyond. In Runtime Verification, Christian Colombo and Martin Leucker (Eds.). Springer International Publishing, Cham, 113--137.

[3]

Karthikeyan Bhargavan, Antoine Delignat-Lavaud, Cédric Fournet, Anitha Gollamudi, Georges Gonthier, Nadim Kobeissi, Natalia Kulatova, Aseem Rastogi, Thomas Sibut-Pinote, Nikhil Swamy, et al. 2016. Formal verification of smart contracts: Short paper. In Proceedings of the 2016 ACM Workshop on Programming Languages and Analysis for Security. ACM, New York, NY, USA, 91--96.

Digital Library

[4]

Lexi Brent, Anton Jurisevic, Michael Kong, Eric Liu, Francois Gauthier, Vincent Gramoli, Ralph Holz, and Bernhard Scholz. 2018. Vandal: A scalable security analysis framework for smart contracts. arXiv:1809.03981

[5]

Vitalik Buterin et al. 2013. Ethereum white paper. GitHub repository 1, GitHub (2013), 22--23.

[6]

Jialiang Chang, Bo Gao, Hao Xiao, Jun Sun, and Zijiang Yang. 2018. sCompile: Critical path identification and analysis for smart contracts. arXiv:1808.00624

[7]

Huashan Chen, Marcus Pendleton, Laurent Njilla, and Shouhuai Xu. 2019. A Survey on Ethereum Systems Security: Vulnerabilities, Attacks and Defenses. arXiv:1908.04507

[8]

Jiachi Chen, Xin Xia, David Lo, John Grundy, Daniel Xiapu Luo, and Ting Chen. 2019. Domain Specific Code Smells in Smart Contracts. arXiv:arXiv:1905.01467

[9]

Ting Chen, Xiaoqi Li, Xiapu Luo, and Xiaosong Zhang. 2017. Under-optimized smart contracts devour your money. In 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, Klagenfurt, Austria, 442--446.

[10]

Blockchain Company. 2018. Solhydra. https://github.com/BlockChainCompany/solhydra.

[11]

Phil Daian. 2016. Analysis of the DAO exploit. http://hackingdistributed.com/2016/06/18/analysis-of-the-dao-exploit/.

[12]

M. di Angelo and G. Salzer. 2019. A Survey of Tools for Analyzing Ethereum Smart Contracts. In 2019 IEEE International Conference on Decentralized Applications and Infrastructures (DAPPCON). IEEE, Newark, CA, USA, USA, 69--78.

[13]

Thomas Durieux, João F. Ferreira, Rui Abreu, and Pedro Cruz. 2019. SmartBugs execution results. https://github.com/smartbugs/smartbugs-results.

[14]

Thomas Durieux, João F. Ferreira, Rui Abreu, and Pedro Cruz. 2019. SmartBugs repository. https://github.com/smartbugs/smartbugs.

[15]

Thomas Durieux, João F. Ferreira, Rui Abreu, and Pedro Cruz. 2019. SmartBugs Wild dataset. https://github.com/smartbugs/smartbugs-wild.

[16]

Josselin Feist, Gustavo Greico, and Alex Groce. 2019. Slither: A Static Analysis Framework for Smart Contracts. In Proceedings of the 2Nd International Workshop on Emerging Trends in Software Engineering for Blockchain (WETSEB '19). IEEE Press, Piscataway, NJ, USA, 8--15.

Digital Library

[17]

Neville Grech, Michael Kong, Anton Jurisevic, Lexi Brent, Bernhard Scholz, and Yannis Smaragdakis. 2018. Madmax: Surviving out-of-gas conditions in ethereum smart contracts. Proceedings of the ACM on Programming Languages 2, OOPSLA (2018), 116.

Digital Library

[18]

Ilya Grishchenko, Matteo Maffei, and Clara Schneidewind. 2018. A Semantic Framework for the Security Analysis of Ethereum Smart Contracts. In Principles of Security and Trust, Lujo Bauer and Ralf Küsters (Eds.). Springer International Publishing, Cham, 243--269.

[19]

Ilya Grishchenko, Matteo Maffei, and Clara Schneidewind. 2018. A Semantic Framework for the Security Analysis of Ethereum Smart Contracts. In Principles of Security and Trust, Lujo Bauer and Ralf Küsters (Eds.). Springer International Publishing, Cham, 243--269.

[20]

Peter Hegedus. 2019. Towards analyzing the complexity landscape of solidity based ethereum smart contracts. Technologies 7, 1 (2019), 6.

[21]

Everett Hildenbrandt, Manasvi Saxena, Nishant Rodrigues, Xiaoran Zhu, Philip Daian, Dwight Guth, Brandon Moore, Daejun Park, Yi Zhang, Andrei Stefanescu, et al. 2018. KEVM: A complete formal semantics of the ethereum virtual machine. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF). IEEE, Oxford, UK, 204--217.

[22]

Sukrit Kalra, Seep Goel, Mohan Dhawan, and Subodh Sharma. 2018. ZEUS: Analyzing Safety of Smart Contracts. In 25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18--21, 2018. NDSS, San Diego, California, USA, 1--15.

[23]

Johannes Krupp and Christian Rossow. 2018. teEther: Gnawing at Ethereum to Automatically Exploit Smart Contracts. In 27th USENIX Security Symposium (USENIX Security 18). USENIX Association, Baltimore, MD, 1317--1333. https://www.usenix.org/conference/usenixsecurity18/presentation/krupp

Digital Library

[24]

Shuvendu K Lahiri, Shuo Chen, Yuepeng Wang, and Isil Dillig. 2018. Formal Specification and Verification of Smart Contracts for Azure Blockchain. arXiv:1812.08829

[25]

Chao Liu, Han Liu, Zhao Cao, Zhong Chen, Bangdao Chen, and Bill Roscoe. 2018. Reguard: finding reentrancy bugs in smart contracts. In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings. ACM, New York, NY, USA, 65--68.

Digital Library

[26]

Han Liu, Chao Liu, Wenqi Zhao, Yu Jiang, and Jiaguang Sun. 2018. S-gram: towards semantic-aware security auditing for ethereum smart contracts. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. ACM, New York, NY, USA, 814--819.

Digital Library

[27]

Loi Luu, Duc-Hiep Chu, Hrishi Olickel, Prateek Saxena, and Aquinas Hobor. 2016. Making smart contracts smarter. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security. ACM, New York, NY, USA, 254--269.

Digital Library

[28]

Anastasia Mavridou and Aron Laszka. 2018. Tool Demonstration: FSolidM for Designing Secure Ethereum Smart Contracts. In Principles of Security and Trust, Lujo Bauer and Ralf Küsters (Eds.). Springer International Publishing, Cham, 270--277.

[29]

Evgeny Medvedev. 2018. Ethereum in BigQuery: a Public Dataset for smart contract analytics. https://cloud.google.com/blog/products/data-analytics/ethereum-bigquery-public-dataset-smart-contract-analytics.

[30]

Mark Mossberg, Felipe Manzano, Eric Hennenfent, Alex Groce, Gustavo Grieco, Josselin Feist, Trent Brunson, and Artem Dinaburg. 2019. Manticore: A User-Friendly Symbolic Execution Framework for Binaries and Smart Contracts. arXiv:1907.03890

[31]

Bernhard Mueller. 2018. Smashing ethereum smart contracts for fun and real profit. In 9th Annual HITB Security Conference (HITBSecConf). HITB, Amsterdam, Netherlands, 54.

[32]

Ivica Nikolić, Aashish Kolluri, Ilya Sergey, Prateek Saxena, and Aquinas Hobor. 2018. Finding the greedy, prodigal, and suicidal contracts at scale. In Proceedings of the 34th Annual Computer Security Applications Conference. ACM, New York, NY, USA, 653--663.

Digital Library

[33]

Robert Norvill, Beltran Borja Fiz Pontiveros, Radu State, and Andrea Cullen. 2018. Visual emulation for Ethereum's virtual machine. In NOMS 2018--2018 IEEE/IFIP Network Operations and Management Symposium. IEEE, Taipei, Taiwan, 1--4.

Digital Library

[34]

Reza M. Parizi, Ali Dehghantanha, Kim-Kwang Raymond Choo, and Amritraj Singh. 2018. Empirical Vulnerability Analysis of Automated Smart Contracts Security Testing on Blockchains. In Proceedings of the 28th Annual International Conference on Computer Science and Software Engineering (CASCON '18). IBM Corp., Riverton, NJ, USA, 103--113. http://dl.acm.org/citation.cfm?id=3291291.3291303

Digital Library

[35]

Daniel Perez and Benjamin Livshits. 2019. Smart Contract Vulnerabilities: Does Anyone Care? arXiv:1902.06710

[36]

Andrea Pinna, Simona Ibba, Gavina Baralla, Roberto Tonelli, and Michele Marchesi. 2019. A Massive Analysis of Ethereum Smart Contracts Empirical Study and Code Metrics. IEEE Access 7 (2019), 78194--78213.

[37]

Matt Suiche. 2017. The $280M Ethereum's Parity bug. https://blog.comae.io/the-280m-ethereums-bug-f28e5de43513.

[38]

Matt Suiche. 2017. Porosity: A decompiler for blockchain-based smart contracts bytecode. DEF con 25 (2017), 11.

[39]

Sergei Tikhomirov, Ekaterina Voskresenskaya, Ivan Ivanitskiy, Ramil Takhaviev, Evgeny Marchenko, and Yaroslav Alexandrov. 2018. Smartcheck: Static analysis of ethereum smart contracts. In 2018 IEEE/ACM 1st International Workshop on Emerging Trends in Software Engineering for Blockchain (WETSEB). IEEE, Gothenburg, Sweden, Sweden, 9--16.

Digital Library

[40]

Christof Ferreira Torres, Julian Schütte, et al. 2018. Osiris: Hunting for integer bugs in ethereum smart contracts. In Proceedings of the 34th Annual Computer Security Applications Conference. ACM, New York, NY, USA, 664--676.

Digital Library

[41]

Christof Ferreira Torres, Mathis Steichen, and Radu State. 2019. The Art of The Scam: Demystifying Honeypots in Ethereum Smart Contracts. In 28th USENIX Security Symposium (USENIX Security 19). USENIX Association, Santa Clara, CA, 1591--1607. https://www.usenix.org/conference/usenixsecurity19/presentation/ferreira

[42]

Petar Tsankov, Andrei Dan, Dana Drachsler-Cohen, Arthur Gervais, Florian Buenzli, and Martin Vechev. 2018. Securify: Practical security analysis of smart contracts. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. ACM, New York, NY, USA, 67--82.

Digital Library

[43]

E. Zhou, S. Hua, B. Pi, J. Sun, Y. Nomura, K. Yamashita, and H. Kurihara. 2018. Security Assurance for Smart Contract. In 2018 9th IFIP International Conference on New Technologies, Mobility and Security (NTMS). IEEE, Paris, France, 1--5.

[44]

Yi Zhou, Deepak Kumar, Surya Bakshi, Joshua Mason, Andrew Miller, and Michael Bailey. 2018. Erays: Reverse Engineering Ethereum's Opaque Smart Contracts. In 27th USENIX Security Symposium (USENIX Security 18). USENIX Association, Baltimore, MD, 1371--1385. https://www.usenix.org/conference/usenixsecurity18/presentation/zhou

Cited By

Wang ZGuoming LXu HYou SMa HWang H(2024)Deep learning-based methodology for vulnerability detection in smart contractsPeerJ Computer Science10.7717/peerj-cs.232010(e2320)Online publication date: 26-Sep-2024
https://doi.org/10.7717/peerj-cs.2320
Pishdar MBahaghighat MKumar RXin Q(2024)Major vulnerabilities in Ethereum smart contracts: Investigation and statistical analysisEAI Endorsed Transactions on Internet of Things10.4108/eetiot.512011Online publication date: 18-Dec-2024
https://doi.org/10.4108/eetiot.5120
Wu HPeng YHe YLu S(2024)EDSCVD: Enhanced Dual-Channel Smart Contract Vulnerability Detection MethodSymmetry10.3390/sym1610138116:10(1381)Online publication date: 17-Oct-2024
https://doi.org/10.3390/sym16101381
Show More Cited By

Index Terms

Empirical review of automated analysis tools on 47,587 Ethereum smart contracts
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

SmartBugs: a framework to analyze solidity smart contracts
ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

Over the last few years, there has been substantial research on automated analysis, testing, and debugging of Ethereum smart contracts. However, it is not trivial to compare and reproduce that research. To address this, we present SmartBugs, an ...
Empirical vulnerability analysis of automated smart contracts security testing on blockchains
CASCON '18: Proceedings of the 28th Annual International Conference on Computer Science and Software Engineering

The emerging blockchain technology supports decentralized computing paradigm shift and is a rapidly approaching phenomenon. While blockchain is thought primarily as the basis of Bitcoin, its application has grown far beyond cryptocurrencies due to the ...
Evolution of automated weakness detection in Ethereum bytecode: a comprehensive study
Abstract
Blockchain programs (also known as smart contracts) manage valuable assets like cryptocurrencies and tokens, and implement protocols in domains like decentralized finance (DeFi) and supply-chain management. These types of applications require a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

June 2020

1640 pages

ISBN:9781450371216

DOI:10.1145/3377811

General Chairs:
Gregg Rothermel
North Carolina State University
,
Doo-Hwan Bae
KAIST, South Korea

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

KIISE: Korean Institute of Information Scientists and Engineers
IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Funding Sources

Fundação para a Ciência e a Tecnologia
Horizon 2020

Conference

ICSE '20

Sponsor:

SIGSOFT

ICSE '20: 42nd International Conference on Software Engineering

June 27 - July 19, 2020

Seoul, South Korea

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

195
Total Citations
View Citations
1,810
Total Downloads

Downloads (Last 12 months)407
Downloads (Last 6 weeks)51

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang ZGuoming LXu HYou SMa HWang H(2024)Deep learning-based methodology for vulnerability detection in smart contractsPeerJ Computer Science10.7717/peerj-cs.232010(e2320)Online publication date: 26-Sep-2024
https://doi.org/10.7717/peerj-cs.2320
Pishdar MBahaghighat MKumar RXin Q(2024)Major vulnerabilities in Ethereum smart contracts: Investigation and statistical analysisEAI Endorsed Transactions on Internet of Things10.4108/eetiot.512011Online publication date: 18-Dec-2024
https://doi.org/10.4108/eetiot.5120
Wu HPeng YHe YLu S(2024)EDSCVD: Enhanced Dual-Channel Smart Contract Vulnerability Detection MethodSymmetry10.3390/sym1610138116:10(1381)Online publication date: 17-Oct-2024
https://doi.org/10.3390/sym16101381
Bennouk KAit Aali NEl Bouzekri El Idrissi YSebai BFaroukhi AMahouachi D(2024)A Comprehensive Review and Assessment of Cybersecurity Vulnerability Detection MethodologiesJournal of Cybersecurity and Privacy10.3390/jcp40400404:4(853-908)Online publication date: 7-Oct-2024
https://doi.org/10.3390/jcp4040040
Bani-Hani RShatnawi AAl-Yahya L(2024)Vulnerability Detection and Classification of Ethereum Smart Contracts Using Deep LearningFuture Internet10.3390/fi1609032116:9(321)Online publication date: 4-Sep-2024
https://doi.org/10.3390/fi16090321
Kiani RSheng V(2024)Ethereum Smart Contract Vulnerability Detection and Machine Learning-Driven Solutions: A Systematic Literature ReviewElectronics10.3390/electronics1312229513:12(2295)Online publication date: 12-Jun-2024
https://doi.org/10.3390/electronics13122295
Xu JWang TLv MChen TZhu TJi B(2024)MVD-HG: multigranularity smart contract vulnerability detection method based on heterogeneous graphsCybersecurity10.1186/s42400-024-00245-57:1Online publication date: 11-Oct-2024
https://doi.org/10.1186/s42400-024-00245-5
Wei ZSun JZhang ZZhang XYang XZhu L(2024)Survey on Quality Assurance of Smart ContractsACM Computing Surveys10.1145/369586457:2(1-36)Online publication date: 10-Oct-2024
https://dl.acm.org/doi/10.1145/3695864
Li WLi XLi ZZhang YFilkov VRay BZhou M(2024)COBRA: Interaction-Aware Bytecode-Level Vulnerability Detector for Smart ContractsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695601(1358-1369)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695601
Wu YXie XPeng CLiu DWu HFan MLiu TWang HFilkov VRay BZhou M(2024)AdvSCanner: Generating Adversarial Smart Contracts to Exploit Reentrancy Vulnerabilities Using LLM and Static AnalysisProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695482(1019-1031)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695482
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents