research-article

Open access

Random testing for C and C++ compilers with YARPGen

Authors:

Vsevolod Livinskii,

Dmitry Babokin,

John RegehrAuthors Info & Claims

Proceedings of the ACM on Programming Languages, Volume 4, Issue OOPSLA

Article No.: 196, Pages 1 - 25

https://doi.org/10.1145/3428264

Published: 13 November 2020 Publication History

Abstract

Compilers should not crash and they should not miscompile applications. Random testing is an effective method for finding compiler bugs that have escaped other kinds of testing. This paper presents Yet Another Random Program Generator (YARPGen), a random test-case generator for C and C++ that we used to find and report more than 220 bugs in GCC, LLVM, and the Intel® C++ Compiler. Our research contributions include a method for generating expressive programs that avoid undefined behavior without using dynamic checks, and generation policies, a mechanism for increasing diversity of generated code and for triggering more optimizations. Generation policies decrease the testing time to find hard-to-trigger compiler bugs and, for the kinds of scalar optimizations YARPGen was designed to stress-test, increase the number of times these optimizations are applied by the compiler by an average of 20% for LLVM and 40% for GCC. We also created tools for automating most of the common tasks related to compiler fuzzing; these tools are also useful for fuzzers other than ours.

Supplementary Material

Auxiliary Presentation Video (oopsla20main-p292-p-video.mp4)

This is a presentation video of my talk at OOPSLA 2020 on our paper about compiler random testing. This paper describes the C and C++ fuzzer, called YARPGen, that is designed to test scalar optimizations. It can produce code that is free from Undefined Behavior and target optimizations explicitly. This tool helped us to find more than 220 bugs in GCC, LLVM, and Intel® C++ Compiler.

Download
129.65 MB

References

[1]

Domenico Amalfitano, Nicola Amatucci, Anna Rita Fasolino, Porfirio Tramontana, Emily Kowalczyk, and Atif M. Memon. 2015. Exploiting the Saturation Efect in Automatic Random Testing of Android Applications. In Proceedings of the Second ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft '15). IEEE Press, 33-43.

[2]

Gergö Barany. 2018a. Finding Missed Compiler Optimizations by Diferential Testing. In Proceedings of the 27th International Conference on Compiler Construction (CC 2018 ). Association for Computing Machinery, New York, NY, USA, 82-92. https://doi.org/10.1145/3178372.3179521

Digital Library

[3]

Gergö Barany. 2018b. Liveness-Driven Random Program Generation. In Logic-Based Program Synthesis and Transformation, Fabio Fioravanti and John P. Gallagher (Eds.). Springer International Publishing, Cham, 112-127.

[4]

Osbert Bastani, Rahul Sharma, Alex Aiken, and Percy Liang. 2017. Synthesizing Program Input Grammars. In Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2017 ). Association for Computing Machinery, New York, NY, USA, 95-110. https://doi.org/10.1145/3062341.3062349

Digital Library

[5]

Matteo Biagiola, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2019. Diversity-Based Web Test Generation. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2019 ). Association for Computing Machinery, New York, NY, USA, 142-153. https: //doi.org/10.1145/3338906.3338970

Digital Library

[6]

Marcel Böhme and Brandon Falk. 2020. Fuzzing: On the Exponential Cost of Vulnerability Discovery. In Proceedings of the 2020 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software. 11.

Digital Library

[7]

Colin J Burgess and M Saidi. 1996. The automatic generation of test cases for optimizing Fortran compilers. Information and Software Technology 38, 2 ( 1996 ), 111-119.

[8]

Junjie Chen, Jibesh Patra, Michael Pradel, Yingfei Xiong, Hongyu Zhang, Dan Hao, and Lu Zhang. 2020. A Survey of Compiler Testing. ACM Comput. Surv. 53, 1, Article 4 ( Feb. 2020 ), 36 pages. https://doi.org/10.1145/3363562

Digital Library

[9]

Tsong Yueh Chen, Fei-Ching Kuo, Robert G. Merkel, and T. H. Tse. 2010. Adaptive Random Testing: The ART of Test Case Diversity. J. Syst. Softw. 83, 1 (Jan. 2010 ), 60-66. https://doi.org/10.1016/j.jss. 2009. 02.022

Digital Library

[10]

Chris Cummins, Pavlos Petoumenos, Alastair Murray, and Hugh Leather. 2018. Compiler Fuzzing through Deep Learning. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2018 ). Association for Computing Machinery, New York, NY, USA, 95-105. https://doi.org/10.1145/3213846.3213848

Digital Library

[11]

Pascal Cuoq, Florent Kirchner, Nikolai Kosmatov, Virgile Prevosto, Julien Signoles, and Boris Yakobowski. 2012. Frama-C: A Software Analysis Perspective. In Proceedings of the 10th International Conference on Software Engineering and Formal Methods (SEFM'12). Springer-Verlag, Berlin, Heidelberg, 233-247. https://doi.org/10.1007/978-3-642-33826-7_16

Digital Library

[12]

K. Dewey, J. Roesch, and B. Hardekopf. 2015. Fuzzing the Rust Typechecker Using CLP (T). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). 482-493.

[13]

Eric Eide and John Regehr. 2008. Volatiles are miscompiled, and what to do about it. In Proceedings of the 8th ACM international conference on Embedded software. 255-264.

Digital Library

[14]

Alex Groce, Chaoqiang Zhang, Eric Eide, Yang Chen, and John Regehr. 2012. Swarm Testing. In Proceedings of the 2012 International Symposium on Software Testing and Analysis (ISSTA 2012 ). Association for Computing Machinery, New York, NY, USA, 78-88. https://doi.org/10.1145/2338965.2336763

Digital Library

[15]

Atsushi Hashimoto and Nagisa Ishiura. 2016. Detecting arithmetic optimization opportunities for C compilers by randomly generated equivalent programs. IPSJ Transactions on System LSI Design Methodology 9 ( 2016 ), 21-29.

[16]

R. Huang, W. Sun, Y. Xu, H. Chen, D. Towey, and X. Xia. 2019. A Survey on Adaptive Random Testing. IEEE Transactions on Software Engineering ( 2019 ).

[17]

International Organization for Standardization 2011. ISO/IEC 9899: 201x: Programming Languages-C. International Organization for Standardization. http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf.

[18]

International Organization for Standardization 2012. ISO/IEC N3337: Working Draft, Standard for Programming Language C++. International Organization for Standardization. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2012/n3337.pdf.

[19]

Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler Validation via Equivalence modulo Inputs. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '14). Association for Computing Machinery, New York, NY, USA, 216-226. https://doi.org/10.1145/2594291.2594334

Digital Library

[20]

Vu Le, Chengnian Sun, and Zhendong Su. 2015. Finding Deep Compiler Bugs via Guided Stochastic Program Mutation. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA 2015 ). Association for Computing Machinery, New York, NY, USA, 386-399. https://doi.org/10. 1145/2814270.2814319

Digital Library

[21]

Christian Lindig. 2005. Random Testing of C Calling Conventions. In Proceedings of the Sixth International Symposium on Automated Analysis-Driven Debugging (AADEBUG'05). Association for Computing Machinery, New York, NY, USA, 3-12. https://doi.org/10.1145/1085130.1085132

Digital Library

[22]

Xiao Liu, Xiaoting Li, Rupesh Prajapati, and Dinghao Wu. 2019. Deepfuzz: Automatic generation of syntax valid c programs for fuzz testing. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 1044-1051.

Digital Library

[23]

Michaël Marcozzi, Qiyi Tang, Alastair Donaldson, and Cristian Cadar. 2019. A Systematic Impact Study for Fuzzer-Found Compiler Bugs. arXiv preprint arXiv: 1902. 09334 ( 2019 ).

[24]

William M. McKeeman. 1998. Diferential Testing for Software. Digital Technical Journal 10, 1 (Dec. 1998 ), 100-107.

[25]

Eriko Nagai, Hironobu Awazu, Nagisa Ishiura, and Naoya Takeda. 2012. Random testing of C compilers targeting arithmetic optimization. In Workshop on Synthesis And System Integration of Mixed Information Technologies (SASIMI 2012 ). 48-53.

[26]

Eriko Nagai, Atsushi Hashimoto, and Nagisa Ishiura. 2013. Scaling up size and number of expressions in random testing of arithmetic optimization of C compilers. In Workshop on Synthesis And System Integration of Mixed Information Technologies (SASIMI 2013 ). 88-93.

[27]

Eriko Nagai, Atsushi Hashimoto, and Nagisa Ishiura. 2014. Reinforcing random testing of arithmetic optimization of C compilers by scaling up size and number of expressions. IPSJ Transactions on System LSI Design Methodology 7 ( 2014 ), 91-100.

[28]

K. Nakamura and N. Ishiura. 2016. Random testing of C compilers based on test program generation by equivalence transformation. In 2016 IEEE Asia Pacific Conference on Circuits and Systems (APCCAS). 676-679.

[29]

Georg Ofenbeck, Tiark Rompf, and Markus Püschel. 2016. RandIR: Diferential Testing for Embedded Compilers. In Proceedings of the 2016 7th ACM SIGPLAN Symposium on Scala (SCALA 2016 ). Association for Computing Machinery, New York, NY, USA, 21-30. https://doi.org/10.1145/2998392.2998397

Digital Library

[30]

Jibesh Patra and Michael Pradel. 2016. Learning to fuzz: Application-independent fuzz testing with probabilistic, generative models of input data. TU Darmstadt, Department of Computer Science, Tech. Rep. TUD-CS-2016-14664 ( 2016 ).

[31]

John Regehr, Yang Chen, Pascal Cuoq, Eric Eide, Chucky Ellison, and Xuejun Yang. 2012. Test-Case Reduction for C Compiler Bugs. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '12). Association for Computing Machinery, New York, NY, USA, 335-346. https://doi.org/10.1145/2254064.2254104

Digital Library

[32]

Richard L. Sauder. 1962. A General Test Data Generator for COBOL. In Proceedings of the May 1-3, 1962, Spring Joint Computer Conference (AIEE-IRE ' 62 (Spring)). Association for Computing Machinery, New York, NY, USA, 317-323. https://doi.org/10.1145/1460833.1460869

Digital Library

[33]

Chengnian Sun, Vu Le, and Zhendong Su. 2016. Finding Compiler Bugs via Live Code Mutation. In Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA 2016 ). Association for Computing Machinery, New York, NY, USA, 849-863. https://doi.org/10.1145/2983990.2984038

Digital Library

[34]

David B. Whalley. 1994. Automatic Isolation of Compiler Errors. ACM Trans. Program. Lang. Syst. 16, 5 (Sept. 1994 ), 1648-1659. https://doi.org/10.1145/186025.186103

Digital Library

[35]

Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and Understanding Bugs in C Compilers. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '11). Association for Computing Machinery, New York, NY, USA, 283-294. https://doi.org/10.1145/1993498.1993532

Digital Library

Cited By

Wang QJung R(2024)Rustlantis: Randomized Differential Testing of the Rust CompilerProceedings of the ACM on Programming Languages10.1145/36897808:OOPSLA2(1955-1981)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689780
Yang CDeng YLu RYao JLiu JJabbarvand RZhang L(2024)WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language ModelsProceedings of the ACM on Programming Languages10.1145/36897368:OOPSLA2(709-735)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689736
Zhang CSu Z(2024)SMT2Test: From SMT Formulas to Effective Test CasesProceedings of the ACM on Programming Languages10.1145/36897198:OOPSLA2(222-245)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689719
Show More Cited By

Index Terms

Random testing for C and C++ compilers with YARPGen
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools
    1. Compilers
      1. Source code generation

Recommendations

Fuzzing Loop Optimizations in Compilers for C++ and Data-Parallel Languages

Compilers are part of the foundation upon which software systems are built; they need to be as correct as possible. This paper is about stress-testing loop optimizers; it presents a major reimplementation of Yet Another Random Program Generator (...
Finding and understanding bugs in C compilers
PLDI '11: Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation

Compilers should be correct. To improve the quality of C compilers, we created Csmith, a randomized test-case generation tool, and spent three years using it to find compiler bugs. During this period we reported more than 325 previously unknown bugs to ...
Finding and understanding bugs in C compilers
PLDI '11

Compilers should be correct. To improve the quality of C compilers, we created Csmith, a randomized test-case generation tool, and spent three years using it to find compiler bugs. During this period we reported more than 325 previously unknown bugs to ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Programming Languages

Proceedings of the ACM on Programming Languages Volume 4, Issue OOPSLA

November 2020

3108 pages

EISSN:2475-1421

DOI:10.1145/3436718

Issue’s Table of Contents

Copyright © 2020 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 November 2020

Published in PACMPL Volume 4, Issue OOPSLA

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Distinguished Paper

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

58
Total Citations
View Citations
3,461
Total Downloads

Downloads (Last 12 months)886
Downloads (Last 6 weeks)109

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang QJung R(2024)Rustlantis: Randomized Differential Testing of the Rust CompilerProceedings of the ACM on Programming Languages10.1145/36897808:OOPSLA2(1955-1981)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689780
Yang CDeng YLu RYao JLiu JJabbarvand RZhang L(2024)WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language ModelsProceedings of the ACM on Programming Languages10.1145/36897368:OOPSLA2(709-735)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689736
Zhang CSu Z(2024)SMT2Test: From SMT Formulas to Effective Test CasesProceedings of the ACM on Programming Languages10.1145/36897198:OOPSLA2(222-245)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689719
Schwarcz FBerlakovich FBarany GMössenböck HBöhme MNoller YSzekeres L(2024)LOOL: Low-Overhead, Optimization-Log-Guided Compiler Fuzzing (Registered Report)Proceedings of the 3rd ACM International Fuzzing Workshop10.1145/3678722.3685533(42-51)Online publication date: 13-Sep-2024
https://dl.acm.org/doi/10.1145/3678722.3685533
Zeng JTang YZhang JLi WHuang Z(2024)Detecting Optimizing Compiler Bugs via History-Driven Test Program MutationProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3671387(145-154)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3671387
Georgescu COlsthoorn MDerakhshanfar PAkhin MPanichella Ad'Amorim M(2024)Evolutionary Generative Fuzzing for Differential Testing of the Kotlin CompilerCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663864(197-207)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663864
Li STheodoridis TSu Z(2024)Boosting Compiler Testing by Injecting Real-World CodeProceedings of the ACM on Programming Languages10.1145/36563868:PLDI(223-245)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3656386
Kim HKim SLee JCha SChristakis MPradel M(2024)AsFuzzer: Differential Testing of Assemblers with Error-Driven Grammar InferenceProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680345(1099-1111)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680345
Zhang MTian YXu ZDong YTan SSun CChristakis MPradel M(2024)LPR: Large Language Models-Aided Program ReductionProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3652126(261-273)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3652126
Jin SLi YChen LShi G(2024)SSFuzz:Generating syntactic and semantic seeds for RISC-V processorsProceedings of the Great Lakes Symposium on VLSI 202410.1145/3649476.3658712(421-426)Online publication date: 12-Jun-2024
https://dl.acm.org/doi/10.1145/3649476.3658712
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents