research-article

Characterizing and Detecting WebAssembly Runtime Bugs

Authors:

Xuanzhe LiuAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology, Volume 33, Issue 2

Article No.: 37, Pages 1 - 29

https://doi.org/10.1145/3624743

Published: 21 December 2023 Publication History

Abstract

WebAssembly (abbreviated WASM) has emerged as a promising language of the Web and also been used for a wide spectrum of software applications such as mobile applications and desktop applications. These applications, named WASM applications, commonly run in WASM runtimes. Bugs in WASM runtimes are frequently reported by developers and cause the crash of WASM applications. However, these bugs have not been well studied. To fill in the knowledge gap, we present a systematic study to characterize and detect bugs in WASM runtimes. We first harvest a dataset of 311 real-world bugs from hundreds of related posts on GitHub. Based on the collected high-quality bug reports, we distill 31 bug categories of WASM runtimes and summarize their common fix strategies. Furthermore, we develop a pattern-based bug detection framework to automatically detect bugs in WASM runtimes. We apply the detection framework to seven popular WASM runtimes and successfully uncover 60 bugs that have never been reported previously, among which 13 have been confirmed and 9 have been fixed by runtime developers.

References

[1]

Nicolas Falliere. 2018. Reverse Engineering WebAssembly. https://www.pnfsoftware.com/reversing-wasm.pdf

[2]

Wasmer. 2019. wasmer issue 830. https://github.com/wasmerio/wasmer/issues/830

[3]

Bytecode Alliance. 2020. WAMR issue 1144. https://github.com/bytecodealliance/wasm-micro-runtime/issues/1144

[4]

Wasmer. 2020. wasmer issue 1263. https://github.com/wasmerio/wasmer/issues/1263

[5]

Bytecode Alliance. 2020. wasmtime issue 2347. https://github.com/bytecodealliance/wasmtime/issues/2347

[6]

Wasmer. 2021. wasmer-go issue 244. https://github.com/wasmerio/wasmer-go/issues/244

[7]

Wasmer. 2021. wasmer issue 1640. https://github.com/wasmerio/wasmer/issues/1640

[8]

Wasmer. 2021. wasmer issue 2028. https://github.com/wasmerio/wasmer/issues/2028

[9]

Wasmer. 2021. wasmer issue 2187. https://github.com/wasmerio/wasmer/issues/2187

[10]

Wasmer. 2021. wasmer issue 2215. https://github.com/wasmerio/wasmer/issues/2215.

[11]

Wasmer. 2021. wasmer issue 2297. https://github.com/wasmerio/wasmer/issues/2297

[12]

Bytecode Alliance. 2021. wasmtime issue 2826. https://github.com/bytecodealliance/wasmtime/issues/2826

[13]

Bytecode Alliance. 2021. wasmtime issue 2883. https://github.com/bytecodealliance/wasmtime/issues/2883

[14]

Bytecode Alliance. 2021. wasmtime issue 3173. https://github.com/bytecodealliance/wasmtime/issues/3173

[15]

Bytecode Alliance. 2021. wasmtime issue 3337. https://github.com/bytecodealliance/wasmtime/issues/3337

[16]

Chris Fallin. 2022. Cranelift Doc. https://hacks.mozilla.org/2020/10/a-new-backend-for-cranelift-part-1-instruction-selection/

[17]

Bytecode Alliance. 2022. Cranelift IR Doc. https://github.com/bytecodealliance/wasmtime/blob/main/cranelift/docs/ir.md

[18]

EOSIO. 2022. EOS VM - A low-latency, high performance and extensible WebAssembly engine.https://github.com/EOSIO/eos

[19]

ewasm. 2022. hera - An ewasm (revision 4) virtual machine implemented in C++ conforming to EVMC ABIv9.https://github.com/ewasm/hera

[20]

WebAssembly Community Group. 2022. WABT: The WebAssembly Binary Toolkit. https://github.com/WebAssembly/wabt

[21]

Bytecode Alliance. 2022. WAMR issue 1282. https://github.com/bytecodealliance/wasm-micro-runtime/issues/1282

[22]

Bytecode Alliance. 2022. WAMR issue 1289. https://github.com/bytecodealliance/wasm-micro-runtime/issues/1289

[23]

WebAssembly Community Group. 2022. WASI link. https://wasi.dev/

[24]

WebAssembly Community Group. 2022. Wasm non web usage. https://webassembly.org/docs/non-web/

[25]

Timothy McCallum. 2022. Wasm runtime architecture. https://medium.com/wasm/webassembly-wasm-runtimes-522bcc7478fd

[26]

Wasm3 Labs. 2022. wasm3 - The fastest WebAssembly interpreter, and the most universal runtime.https://github.com/wasm3/wasm3

[27]

Wasm3 Labs. 2022. wasm3 bug fix commit.https://github.com/wasm3/wasm3/commits/fbbacefeaf28e019244bbfa281fc4dea3dbdedc9

[28]

Wasm3 Labs. 2022. wasm3 issue 351. https://github.com/wasm3/wasm3/issues/351

[29]

Wasm3 Labs. 2022. wasm3 issue 355. https://github.com/wasm3/wasm3/issues/355

[30]

Cloud Native Computing Foundation. 2022. WasmEdge bug fix commit.https://github.com/WasmEdge/WasmEdge/commit/4103613ff57341af346f7ff82bd0beb47e798474

[31]

Cloud Native Computing Foundation. 2022. WasmEdge issue 1711. https://github.com/WasmEdge/WasmEdge/issues/1711

[32]

Cloud Native Computing Foundation. 2022. WasmEdge Runtime. https://github.com/WasmEdge/WasmEdge

[33]

Wasmer. 2022. wasmer - A fast and secure WebAssembly runtime. https://github.com/wasmerio/wasmer

[34]

Wasmer. 2022. wasmer-go - A complete and mature WebAssembly runtime for Go based on Wasmer.https://github.com/wasmerio/wasmer-go

[35]

Wasmer. 2022. wasmer-python - A complete and mature WebAssembly runtime for Python based on Wasmer.https://github.com/wasmerio/wasmer-python

[36]

Bytecode Alliance. 2022. wasmtime - A standalone runtime for WebAssembly. https://github.com/bytecodealliance/wasmtime

[37]

Bytecode Alliance. 2022. wasmtime issue 4170. https://github.com/bytecodealliance/wasmtime/issues/4170

[38]

MDN Web Docs Community. 2022. Wat file. https://developer.mozilla.org/en-US/docs/WebAssembly/Text_format_to_wasm

[39]

WAVM. 2022. WAVM - A WebAssembly virtual machine, designed for use in non-browser applications.https://github.com/WAVM/WAVM

[40]

Bytecode Alliance. 2022. WebAssembly Micro Runtime. https://github.com/bytecodealliance/wasm-micro-runtime

[41]

Lin Clark. 2022. WebAssembly system interface Doc. https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/

[42]

WebAssembly Community Group. 2022. WebAssmebly Doc. https://webassembly.org/

[43]

Emscripten community. 2023. Emscripten compiler. https://emscripten.org/

[44]

Yixuan Zhang. 2023. Supplemental materials. https://github.com/bnmcxlzd/TOSEM2023_Complementary_materials

[45]

Emad Aghajani, Csaba Nagy, Olga Lucero Vega-Márquez, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, and Michele Lanza. 2019. Software documentation issues unveiled. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE ’19). IEEE, 1199–1210.

Digital Library

[46]

Rafael Belchior, André Vasconcelos, Sérgio Guerreiro, and Miguel Correia. 2021. A survey on blockchain interoperability: Past, present, and future trends. ACM Computing Surveys (CSUR) 54, 8 (2021), 1–41.

Digital Library

[47]

Stefanie Beyer, Christian Macho, Massimiliano Di Penta, and Martin Pinzger. 2018. Automatically classifying posts into question categories on stack overflow. In 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC ’18). IEEE, 211–21110.

Digital Library

[48]

Shrenik Bhansali, Ahmet Aris, Abbas Acar, Harun Oz, and A. Selcuk Uluagac. 2022. A first look at code obfuscation for WebAssembly. In Proceedings of the 15th ACM Conference on Security and Privacy in Wireless and Mobile Networks. 140–145.

Digital Library

[49]

Zhenpeng Chen, Huihan Yao, Yiling Lou, Yanbin Cao, Yuanqiang Liu, Haoyu Wang, and Xuanzhe Liu. 2021. An empirical study on deployment faults of deep learning based mobile applications. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE ’21). IEEE, 674–685.

Digital Library

[50]

Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20, 1 (1960), 37–46.

[51]

Anthony Di Franco, Hui Guo, and Cindy Rubio-González. 2017. A comprehensive study of real-world numerical bug characteristics. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE ’17). IEEE, 509–519.

[52]

Zhen Yu Ding and Claire Le Goues. 2021. An empirical study of OSS-Fuzz bugs. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR ’21). IEEE, 131–142.

[53]

Phani Kishore Gadepalli, Sean McBride, Gregor Peach, Ludmila Cherkasova, and Gabriel Parmer. 2020. Sledge: A serverless-first, light-weight WASM runtime for the edge. In Proceedings of the 21st International Middleware Conference. 265–279.

Digital Library

[54]

Phani Kishore Gadepalli, Gregor Peach, Ludmila Cherkasova, Rob Aitken, and Gabriel Parmer. 2019. Challenges and opportunities for efficient serverless computing at the edge. In 2019 38th Symposium on Reliable Distributed Systems (SRDS ’19). IEEE, 261–2615.

[55]

Andreas Haas, Andreas Rossberg, Derek L. Schuff, Ben L. Titzer, Michael Holman, Dan Gohman, Luke Wagner, Alon Zakai, and J. F. Bastien. 2017. Bringing the web up to speed with WebAssembly. In Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation. 185–200.

Digital Library

[56]

David Herrera, Hangfen Chen, Erick Lavoie, and Laurie Hendren. 2018. WebAssembly and JavaScript challenge: Numerical program performance using modern browser technologies and devices. University of McGill, Montreal: QC, Technical Report SABLE-TR-2018-2.

[57]

Aaron Hilbig, Daniel Lehmann, and Michael Pradel. 2021. An empirical study of real-world WebAssembly binaries: Security, languages, use cases. In Proceedings of the Web Conference 2021. 2696–2708.

Digital Library

[58]

Eric Holk. 2018. Schism: A self-hosting scheme to WebAssembly compiler. In Proceedings of the Scheme and Functional.

[59]

Abhinav Jangda, Bobby Powers, Emery D. Berger, and Arjun Guha. 2019. Not so fast: Analyzing the performance of WebAssembly vs. native code. In 2019 USENIX Annual Technical Conference (USENIX ATC ’19). 107–120.

[60]

Daniel Lehmann, Johannes Kinder, and Michael Pradel. 2020. Everything old is new again: Binary security of WebAssembly. In 29th USENIX Security Symposium (USENIX Security ’20). 217–234.

[61]

Daniel Lehmann and Michael Pradel. 2022. Finding the dwarf: Recovering precise types from WebAssembly binaries. In Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation. 410–425.

Digital Library

[62]

Shan Lu, Soyeon Park, Eunsoo Seo, and Yuanyuan Zhou. 2008. Learning from mistakes: A comprehensive study on real world concurrency bug characteristics. In Proceedings of the 13th International Conference on Architectural Support for Programming Languages and Operating Systems. 329–339.

Digital Library

[63]

Niko Mäkitalo, Tommi Mikkonen, Cesare Pautasso, Victor Bankowski, Paulius Daubaris, Risto Mikkola, and Oleg Beletski. 2021. WebAssembly modules as lightweight containers for liquid IoT applications. In International Conference on Web Engineering. Springer, 328–336.

Digital Library

[64]

Brian McFadden, Tyler Lukasiewicz, Jeff Dileo, and Justin Engler. 2018. Security chasms of WASM. NCC Group Whitepaper.

[65]

Pankaj Mendki. 2020. Evaluating WebAssembly enabled serverless approach for edge computing. In 2020 IEEE Cloud Summit. IEEE, 161–166.

[66]

Jämes Ménétrey, Marcelo Pasin, Pascal Felber, and Valerio Schiavoni. 2021. Twine: An embedded trusted runtime for WebAssembly. In 2021 IEEE 37th International Conference on Data Engineering (ICDE ’21). IEEE, 205–216.

[67]

Matteo Paltenghi and Michael Pradel. 2022. Bugs in quantum computing platforms: An empirical study. Proceedings of the ACM on Programming Languages 6, OOPSLA1 (2022), 1–27.

Digital Library

[68]

Alan Romano, Xinyue Liu, Yonghwi Kwon, and Weihang Wang. 2021. An empirical study of bugs in WebAssembly compilers. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE ’21). IEEE, 42–54.

Digital Library

[69]

Alan Romano and Weihang Wang. 2020. WASim: Understanding WebAssembly applications through classification. In 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE ’20). IEEE, 1321–1325.

Digital Library

[70]

Carolyn B. Seaman. 1999. Qualitative methods in empirical studies of software engineering. IEEE Transactions on Software Engineering 25, 4 (1999), 557–572.

Digital Library

[71]

Quentin Stiévenart, David W. Binkley, and Coen De Roover. 2022. Static stack-preserving intra-procedural slicing of WebAssembly binaries. In 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE ’22). IEEE, 2031–2042.

Digital Library

[72]

Weihang Wang. 2021. Empowering web applications with WebAssembly: Are we there yet?. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE ’21). IEEE, 1301–1305.

Digital Library

[73]

Ziyuan Wang, Dexin Bu, Aiyue Sun, Shanyi Gou, Yong Wang, and Lin Chen. 2022. An empirical study on bugs in python interpreters. IEEE Transactions on Reliability 1,1 (2022). DOI:

Digital Library

[74]

Elliott Wen and Gerald Weber. 2020. Wasmachine: Bring IoT up to speed with a WebAssembly OS. In 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops ’20). IEEE, 1–4.

[75]

Jinfeng Wen, Zhenpeng Chen, Yi Liu, Yiling Lou, Yun Ma, Gang Huang, Xin Jin, and Xuanzhe Liu. 2021. An empirical study on challenges of application development in serverless computing. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 416–428.

Digital Library

[76]

Tianyi Zhang, Cuiyun Gao, Lei Ma, Michael Lyu, and Miryung Kim. 2019. An empirical study of common challenges in developing deep learning applications. In 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE ’19). IEEE, 104–115.

[77]

Xiuhong Zhang. 2020. WebAssembly Principles and Core Technologies. China Machine Press.

[78]

Yuhao Zhang, Yifan Chen, Shing-Chi Cheung, Yingfei Xiong, and Lu Zhang. 2018. An empirical study on TensorFlow program bugs. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. 129–140.

Digital Library

[79]

Zhide Zhou, Zhilei Ren, Guojun Gao, and He Jiang. 2021. An empirical study of optimization bugs in GCC and LLVM. Journal of Systems and Software 174 (2021), 110884.

Cited By

Zhao WZeng RZhou YChristakis MPradel M(2024)Wapplique: Testing WebAssembly Runtime via Execution Context-Aware Bytecode MutationProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680340(1035-1047)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680340
Ray P(2023)An Overview of WebAssembly for IoT: Background, Tools, State-of-the-Art, Challenges, and Future DirectionsFuture Internet10.3390/fi1508027515:8(275)Online publication date: 18-Aug-2023
https://doi.org/10.3390/fi15080275
Subramanyan BHollum AMazzeo GFicke MVaydia D(2023)Enabling Trusted TEE-as-a-Service Models with Privacy Preserving Automatons2023 IEEE International Conference on Cloud Computing Technology and Science (CloudCom)10.1109/CloudCom59040.2023.00048(252-260)Online publication date: 4-Dec-2023
https://doi.org/10.1109/CloudCom59040.2023.00048
Show More Cited By

Index Terms

Characterizing and Detecting WebAssembly Runtime Bugs
1. General and reference
  1. Cross-computing tools and techniques
    1. Empirical studies
2. Software and its engineering
  1. Software creation and management

Recommendations

Characterizing duplicate bugs: Perceptions of practitioners and an empirical analysis
Abstract
Bug handling is an essential part of the software development process. Ideally, in a bug‐tracking system, bugs are reported, fixed, verified, and closed. In some cases, bugs have to be reopened mostly due to an incorrect fix. However, instead of ...

Our study investigates the difference between unique and duplicate bugs and further categorizes duplicates into Master‐Unresolved (Category‐1) and Missed‐Reopen (Category‐2) bugs. Through investigation of bug reports, we found that duplicates are up to ...
WASMaker: Differential Testing of WebAssembly Runtimes via Semantic-Aware Binary Generation
ISSTA 2024: Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis

A fundamental component of the Wasm ecosystem is the Wasm runtime, as it directly impacts whether Wasm applications can be executed as expected. Bugs in Wasm runtimes are frequently reported, so the research community has made a few attempts to design ...
Insights into WebAssembly: compilation performance and shared code caching in Node.js
CASCON '20: Proceedings of the 30th Annual International Conference on Computer Science and Software Engineering

Alongside JavaScript, V8 and Node.js have become essential components of contemporary web and cloud applications. With the addition of WebAssembly to the web, developers finally have a fast platform for performance-critical code. However, this addition ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 33, Issue 2

February 2024

947 pages

EISSN:1557-7392

DOI:10.1145/3618077

Editor:
Mauro Pezzè
USI Universitá della Svizzera italiana and SIT Schaffhausen Institute of Technology, Switzerland

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 December 2023

Online AM: 20 September 2023

Accepted: 16 August 2023

Revised: 07 August 2023

Received: 20 January 2023

Published in TOSEM Volume 33, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Key R&D Program of China
National Natural Science Foundation of China
Beijing Outstanding Young Scientist Program
Center for Data Space Technology and System, Peking University
ERC Advanced Grant
Hong Kong RGC Project

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
691
Total Downloads

Downloads (Last 12 months)574
Downloads (Last 6 weeks)60

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhao WZeng RZhou YChristakis MPradel M(2024)Wapplique: Testing WebAssembly Runtime via Execution Context-Aware Bytecode MutationProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680340(1035-1047)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680340
Ray P(2023)An Overview of WebAssembly for IoT: Background, Tools, State-of-the-Art, Challenges, and Future DirectionsFuture Internet10.3390/fi1508027515:8(275)Online publication date: 18-Aug-2023
https://doi.org/10.3390/fi15080275
Subramanyan BHollum AMazzeo GFicke MVaydia D(2023)Enabling Trusted TEE-as-a-Service Models with Privacy Preserving Automatons2023 IEEE International Conference on Cloud Computing Technology and Science (CloudCom)10.1109/CloudCom59040.2023.00048(252-260)Online publication date: 4-Dec-2023
https://doi.org/10.1109/CloudCom59040.2023.00048
Zheng WHua B(2023)A Comprehensive Study of Bugs in Embedded WebAssembly Virtual Machines2023 3rd International Conference on Computer Science, Electronic Information Engineering and Intelligent Control Technology (CEI)10.1109/CEI60616.2023.10528174(901-907)Online publication date: 15-Dec-2023
https://doi.org/10.1109/CEI60616.2023.10528174
Zhou SJiang MChen WZhou HWang HLuo X(2023)WADIFF: A Differential Testing Framework for WebAssembly Runtimes2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00188(939-950)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASE56229.2023.00188

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents