research-article

How effective are smart contract analysis tools? evaluating smart contract static analysis tools using bug injection

Authors:

Karthik PattabiramanAuthors Info & Claims

ISSTA 2020: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis

Pages 415 - 427

https://doi.org/10.1145/3395363.3397385

Published: 18 July 2020 Publication History

Abstract

Security attacks targeting smart contracts have been on the rise, which have led to financial loss and erosion of trust. Therefore, it is important to enable developers to discover security vulnerabilities in smart contracts before deployment. A number of static analysis tools have been developed for finding security bugs in smart contracts. However, despite the numerous bug-finding tools, there is no systematic approach to evaluate the proposed tools and gauge their effectiveness. This paper proposes SolidiFI, an automated and systematic approach for evaluating smart contracts’ static analysis tools. SolidiFI is based on injecting bugs (i.e., code defects) into all potential locations in a smart contract to introduce targeted security vulnerabilities. SolidiFI then checks the generated buggy contract using the static analysis tools, and identifies the bugs that the tools are unable to detect (false-negatives) along with identifying the bugs reported as false-positives. SolidiFI is used to evaluate six widely-used static analysis tools, namely, Oyente, Securify, Mythril, SmartCheck, Manticore and Slither, using a set of 50 contracts injected by 9369 distinct bugs. It finds several instances of bugs that are not detected by the evaluated tools despite their claims of being able to detect such bugs, and all the tools report many false positives.

References

[1]

2016. Analysis of the DAO exploit. https://hackingdistributed.com/ 2016 /06/18/ analysis-of-the-dao-exploit

[2]

2017. History of Ethereum Security Vulnerabilities, Hacks, and Their Fixes. https://applicature.com/blog/blockchain-technology/ history-ofethereum-security-vulnerabilities-hacks-and-their-fixes

[3]

2017. The parity wallet breach. https://bitcoinexchangeguide.com/parity-walletbreach

[4]

2017. Remix-Solidity IDE. http://remix.ethereum.org

[5]

2018. eth-mutants. https://github.com/federicobond/eth-mutants

[6]

2018. New batchOverflow Bug in Multiple ERC20 Smart Contracts (CVE2018-10299). https://medium.com/@peckshield/alert-new-batchoverflow-bugin-multiple-erc20-smart-contracts-cve-2018-10299-511067db6536

[7]

2020. CVE-2018-10299 Detail. https://nvd.nist.gov/vuln/detail/CVE-2018-10299

[8]

2020. INFURA. https://infura.io

[9]

2020. MetaMask. https://metamask.io

[10]

2020. solidity-security-blog. https://github.com/sigp/solidity-security-blog

[11]

Sefa Akca, Ajitha Rajan, and Chao Peng. 2019. SolAnalyser: A Framework for Analysing and Testing Smart Contracts. In 2019 26th Asia-Pacific Software Engineering Conference (APSEC). IEEE, 482-489.

[12]

Sidney Amani, Myriam Bégel, Maksym Bortin, and Mark Staples. 2018. Towards verifying ethereum smart contract bytecode in Isabelle/HOL. In Proceedings of the 7th ACM SIGPLAN International Conference on Certified Programs and Proofs. ACM, 66-77.

Digital Library

[13]

Nicola Atzei, Massimo Bartoletti, and Tiziana Cimoli. 2017. A survey of attacks on ethereum smart contracts (sok). In Principles of Security and Trust. Springer, 164-186.

[14]

Karthikeyan Bhargavan, Antoine Delignat-Lavaud, Cédric Fournet, Anitha Gollamudi, Georges Gonthier, Nadim Kobeissi, Natalia Kulatova, Aseem Rastogi, Thomas Sibut-Pinote, Nikhil Swamy, et al. 2016. Formal verification of smart contracts: Short paper. In Proceedings of the 2016 ACM Workshop on Programming Languages and Analysis for Security. ACM, 91-96.

Digital Library

[15]

Richard Bonett, Kaushal Kafle, Kevin Moran, Adwait Nadkarni, and Denys Poshyvanyk. 2018. Discovering flaws in security-focused static analysis tools for Android using systematic mutation. In 27th {USENIX} Security Symposium ({USENIX} Security 18). 1263-1280.

[16]

Lexi Brent, Anton Jurisevic, Michael Kong, Eric Liu, Francois Gauthier, Vincent Gramoli, Ralph Holz, and Bernhard Scholz. 2018. Vandal: A Scalable Security Analysis Framework for Smart Contracts. arXiv preprint arXiv: 1809. 03981 ( 2018 ).

[17]

Vitalik Buterin. 2014. Ethereum: A next-generation smart contract and decentralized application platform. URL https://github. com/ethereum/wiki/wiki/% 5BEnglish% 5D-White-Paper 7 ( 2014 ).

[18]

WK Chan and Bo Jiang. 2018. Fuse: An Architecture for Smart Contract Fuzz Testing Service. In 2018 25th Asia-Pacific Software Engineering Conference (APSEC). IEEE, 707-708.

[19]

Christopher D Clack, Vikram A Bakshi, and Lee Braine. 2016. Smart contract templates: foundations, design landscape and research directions. arXiv preprint arXiv:1608.00771 ( 2016 ).

[20]

Crytic. [n.d.]. Echdina. https://github.com/crytic/echidna

[21]

Chris Dannen. 2017. Introducing Ethereum and Solidity: Foundations of Cryptocurrency and Blockchain Programming for Beginners. Springer.

[22]

Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An Eficient SMT Solver. In Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS'08/ETAPS'08). 337-340.

[23]

Brendan Dolan-Gavitt, Patrick Hulin, Engin Kirda, Tim Leek, Andrea Mambretti, Wil Robertson, Frederick Ulrich, and Ryan Whelan. 2016. Lava: Large-scale automated vulnerability addition. In 2016 IEEE Symposium on Security and Privacy (SP). IEEE, 110-121.

[24]

Thomas Durieux, João F Ferreira, Rui Abreu, and Pedro Cruz. 2019. Empirical Review of Automated Analysis Tools on 47,587 Ethereum Smart Contracts. arXiv preprint arXiv: 1910. 10601 ( 2019 ).

[25]

Etherscan. [n.d.]. Etherscan. https://etherscan.io

[26]

Josselin Feist, Gustavo Grieco, and Alex Groce. 2019. Slither: a static analysis framework for smart contracts. In 2019 IEEE/ACM 2nd International Workshop on Emerging Trends in Software Engineering for Blockchain (WETSEB). IEEE, 8-15.

Digital Library

[27]

Yu Feng, Emina Torlak, and Rastislav Bodík. 2019. Precise Attack Synthesis for Smart Contracts. CoRR abs/ 1902.06067 ( 2019 ). arXiv: 1902.06067 http://arxiv.org/ abs/ 1902.06067

[28]

Ilya Grishchenko, Matteo Mafei, and Clara Schneidewind. 2018. A Semantic Framework for the Security Analysis of Ethereum smart contracts. In International Conference on Principles of Security and Trust. Springer, 243-269.

[29]

Everett Hildenbrandt, Manasvi Saxena, Xiaoran Zhu, Nishant Rodrigues, Philip Daian, Dwight Guth, and Grigore Rosu. 2017. Kevm: A complete semantics of the ethereum virtual machine. Technical Report.

[30]

Yoichi Hirai. 2017. Defining the ethereum virtual machine for interactive theorem provers. In International Conference on Financial Cryptography and Data Security. Springer, 520-535.

[31]

Bo Jiang, Ye Liu, and WK Chan. 2018. Contractfuzzer: Fuzzing smart contracts for vulnerability detection. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. ACM, 259-269.

Digital Library

[32]

Johannes Krupp and Christian Rossow. 2018. teether: Gnawing at ethereum to automatically exploit smart contracts. In 27th {USENIX} Security Symposium ({USENIX} Security 18 ). {USENIX Association}, 1317-1333.

[33]

Loi Luu, Duc-Hiep Chu, Hrishi Olickel, Prateek Saxena, and Aquinas Hobor. 2016. Making smart contracts smarter. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. ACM, 254-269.

Digital Library

[34]

Florian Mathieu and Ryno Mathee. 2017. Blocktix: decentralized event hosting and ticket distribution network. https://www.cryptoground.com/storage/files/ 1527588859-blocktix-wp-draft.pdf

[35]

Mark Mossberg, Felipe Manzano, Eric Hennenfent, Alex Groce, Gustavo Grieco, Josselin Feist, Trent Brunson, and Artem Dinaburg. 2019. Manticore: A userfriendly symbolic execution framework for binaries and smart contracts. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1186-1189.

Digital Library

[36]

Bernhard Mueller. 2018. Smashing ethereum smart contracts for fun and real profit. HITB SECCONF Amsterdam ( 2018 ).

[37]

Reza M Parizi, Ali Dehghantanha, Kim-Kwang Raymond Choo, and Amritraj Singh. 2018. Empirical vulnerability analysis of automated smart contracts security testing on blockchains. In Proceedings of the 28th Annual International Conference on Computer Science and Software Engineering. IBM Corp., 103-113.

Digital Library

[38]

Chao Peng, Sefa Akca, and Ajitha Rajan. 2019. SIF: A Framework for Solidity Contract Instrumentation and Analysis. In 2019 26th Asia-Pacific Software Engineering Conference (APSEC). IEEE, 466-473.

[39]

Daniel Perez and Benjamin Livshits. 2019. Smart Contract Vulnerabilities: Does Anyone Care? arXiv preprint arXiv: 1902. 06710 ( 2019 ).

[40]

Jannik Pewny and Thorsten Holz. 2016. EvilCoder: automated bug insertion. In Proceedings of the 32nd Annual Conference on Computer Security Applications. ACM, 214-225.

Digital Library

[41]

Ferdian Thung, David Lo, Lingxiao Jiang, Foyzur Rahman, Premkumar T Devanbu, et al. 2012. To what extent could we detect field defects? an empirical study of false negatives in static bug finding tools. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering. ACM, 50-59.

Digital Library

[42]

Sergei Tikhomirov, Ekaterina Voskresenskaya, Ivan Ivanitskiy, Ramil Takhaviev, Evgeny Marchenko, and Yaroslav Alexandrov. 2018. SmartCheck: Static Analysis of Ethereum Smart Contracts. ( 2018 ).

[43]

Petar Tsankov, Andrei Dan, Dana Drachsler-Cohen, Arthur Gervais, Florian Buenzli, and Martin Vechev. 2018. Securify: Practical security analysis of smart contracts. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. 67-82.

Digital Library

[44]

Xingya Wang, Haoran Wu, Weisong Sun, and Yuan Zhao. 2019. Towards Generating Cost-Efective Test-Suite for Ethereum Smart Contract. In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 549-553.

[45]

Haoran Wu, Xingya Wang, Jiehui Xu, Weiqin Zou, Lingming Zhang, and Zhenyu Chen. 2019. Mutation testing for ethereum smart contract. arXiv preprint arXiv: 1908. 03707 ( 2019 ).

Cited By

Bennouk KAit Aali NEl Bouzekri El Idrissi YSebai BFaroukhi AMahouachi D(2024)A Comprehensive Review and Assessment of Cybersecurity Vulnerability Detection MethodologiesJournal of Cybersecurity and Privacy10.3390/jcp40400404:4(853-908)Online publication date: 7-Oct-2024
https://doi.org/10.3390/jcp4040040
Alsunaidi SAljamaan HHammoudeh M(2024)MultiTagging: A Vulnerable Smart Contract Labeling and Evaluation FrameworkElectronics10.3390/electronics1323461613:23(4616)Online publication date: 22-Nov-2024
https://doi.org/10.3390/electronics13234616
Ge XFang CLi XSun WWu DZhai JLin SZhao ZLiu YChen Z(2024)Machine Learning for Actionable Warning Identification: A Comprehensive SurveyACM Computing Surveys10.1145/369635257:2(1-35)Online publication date: 19-Sep-2024
https://dl.acm.org/doi/10.1145/3696352
Show More Cited By

Index Terms

How effective are smart contract analysis tools? evaluating smart contract static analysis tools using bug injection
1. Security and privacy
  1. Software and application security

Recommendations

Towards Effective Static Analysis Approaches for Security Vulnerabilities in Smart Contracts
ASE '22: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering

The growth in the popularity of smart contracts has been accompanied by a rise in security attacks targeting smart contracts, which have led to financial losses of millions of dollars and erosion of trust. To enable developers discover vulnerabilities ...
Contractsentry: a static analysis tool for smart contract vulnerability detection
Abstract
Frequent smart contract security incidents pose a threat to the credibility of the Ethereum platform, making smart contract vulnerability detection a focal point of concern. Previous research has proposed vulnerability detection methods in smart ...
Empirical review of automated analysis tools on 47,587 Ethereum smart contracts
ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

Over the last few years, there has been substantial research on automated analysis, testing, and debugging of Ethereum smart contracts. However, it is not trivial to compare and reproduce that research. To address this, we present an empirical evaluation ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2020: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis

July 2020

591 pages

ISBN:9781450380089

DOI:10.1145/3395363

General Chair:
Sarfraz Khurshid
University of Texas at Austin, USA
,
Program Chair:
Corina S. Păsăreanu
Carnegie Mellon University Silicon Valley / NASA Ames Research Center, USA

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 July 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Conference

ISSTA '20

Sponsor:

SIGSOFT

ISSTA '20: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis

July 18 - 22, 2020

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

111
Total Citations
View Citations
1,597
Total Downloads

Downloads (Last 12 months)321
Downloads (Last 6 weeks)49

Reflects downloads up to 22 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bennouk KAit Aali NEl Bouzekri El Idrissi YSebai BFaroukhi AMahouachi D(2024)A Comprehensive Review and Assessment of Cybersecurity Vulnerability Detection MethodologiesJournal of Cybersecurity and Privacy10.3390/jcp40400404:4(853-908)Online publication date: 7-Oct-2024
https://doi.org/10.3390/jcp4040040
Alsunaidi SAljamaan HHammoudeh M(2024)MultiTagging: A Vulnerable Smart Contract Labeling and Evaluation FrameworkElectronics10.3390/electronics1323461613:23(4616)Online publication date: 22-Nov-2024
https://doi.org/10.3390/electronics13234616
Ge XFang CLi XSun WWu DZhai JLin SZhao ZLiu YChen Z(2024)Machine Learning for Actionable Warning Identification: A Comprehensive SurveyACM Computing Surveys10.1145/369635257:2(1-35)Online publication date: 19-Sep-2024
https://dl.acm.org/doi/10.1145/3696352
Wei ZSun JZhang ZZhang XYang XZhu L(2024)Survey on Quality Assurance of Smart ContractsACM Computing Surveys10.1145/369586457:2(1-36)Online publication date: 10-Oct-2024
https://dl.acm.org/doi/10.1145/3695864
Wu YXie XPeng CLiu DWu HFan MLiu TWang HFilkov VRay BZhou M(2024)AdvSCanner: Generating Adversarial Smart Contracts to Exploit Reentrancy Vulnerabilities Using LLM and Static AnalysisProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695482(1019-1031)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695482
Eshghie MArtho CFilkov VRay BZhou M(2024)Oracle-Guided Vulnerability Diversity and Exploit Synthesis of Smart Contracts Using LLMsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695292(2240-2248)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695292
Kado CYanai NCruz JYamashita KOkamura S(2024)Empirical Study of Impact of Solidity Compiler Updates on Vulnerabilities in Ethereum Smart ContractsDistributed Ledger Technologies: Research and Practice10.1145/3688812Online publication date: 22-Aug-2024
https://doi.org/10.1145/3688812
Li KXue YChen SLiu HSun KHu MWang HLiu YChen Y(2024)Static Application Security Testing (SAST) Tools for Smart Contracts: How Far Are We?Proceedings of the ACM on Software Engineering10.1145/36607721:FSE(1447-1470)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660772
Zheng PSu BLuo XChen TZhang NZheng ZChristakis MPradel M(2024)LENT-SSE: Leveraging Executed and Near Transactions for Speculative Symbolic Execution of Smart ContractsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680303(566-577)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680303
Gao CYang WYe JXue YSun J(2024)sGuard+: Machine Learning Guided Rule-Based Automated Vulnerability Repair on Smart ContractsACM Transactions on Software Engineering and Methodology10.1145/364184633:5(1-55)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3641846
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents