research-article

Precise concolic unit testing of C programs using extended units and symbolic alarm filtering

Authors:

Moonzoo KimAuthors Info & Claims

ICSE '18: Proceedings of the 40th International Conference on Software Engineering

Pages 315 - 326

https://doi.org/10.1145/3180155.3180253

Published: 27 May 2018 Publication History

Abstract

Automated unit testing reduces manual effort to write unit test drivers/stubs and generate unit test inputs. However, automatically generated unit test drivers/stubs raise false alarms because they often over-approximate real contexts of a target function f and allow infeasible executions of f. To solve this problem, we have developed a concolic unit testing technique CONBRIO. To provide realistic context to f, it constructs an extended unit of f that consists of f and closely relevant functions to f. Also, CONBRIO filters out a false alarm by checking feasibility of a corresponding symbolic execution path with regard to f's symbolic calling contexts obtained by combining symbolic execution paths of f's closely related predecessor functions.

In the experiments on the crash bugs of 15 real-world C programs, CONBRIO shows both high bug detection ability (i.e. 91.0% of the target bugs detected) and high precision (i.e. a true to false alarm ratio is 1:4.5). Also, CONBRIO detects 14 new bugs in 9 target C programs studied in papers on crash bug detection techniques.

References

[1]

CROWN: Concolic testing for Real-wOrld softWare aNalysis. http://github.com/swtv-kaist/CROWN. Accessed: 2018-02-14.

[2]

Shay Artzi, Adam Kiezun, Julian Dolby, Frank Tip, Danny Dig, Amit Paradkar, and Michael D. Ernst. 2008. Finding Bugs in Dynamic Web Applications. In Proceedings of the 2008 International Symposium on Software Testing and Analysis (ISSTA '08). ACM, New York, NY, USA, 261--272.

Digital Library

[3]

Thanassis Avgerinos, Alexandre Rebert, Sang Kil Cha, and David Brumley. 2014. Enhancing Symbolic Execution with Veritesting. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). ACM, New York, NY, USA, 1083--1094.

Digital Library

[4]

Radu Banabic, George Candea, and Rachid Guerraoui. 2014. Finding Trojan Message Vulnerabilities in Distributed Systems. In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '14). ACM, New York, NY, USA, 113--126.

Digital Library

[5]

Luciano Baresi, Pier Luca Lanzi, and Matteo Miraz. 2010. TestFul: An Evolutionary Test Approach for Java. In Proceedings of the 2010 Third International Conference on Software Testing, Verification and Validation (ICST '10). IEEE Computer Society, Washington, DC, USA, 185--194.

Digital Library

[6]

Ella Bounimova, Patrice Godefroid, and David Molnar. 2013. Billions and Billions of Constraints: Whitebox Fuzz Testing in Production. In Proceedings of the 2013 International Conference on Software Engineering (ICSE '13). IEEE Press, Piscataway, NJ, USA, 122--131.

Digital Library

[7]

Cristian Cadar, Daniel Dunbar, and Dawson Engler. 2008. KLEE: Unassisted and Automatic Generation of High-coverage Tests for Complex Systems Programs. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation (OSDI'08). USENIX Association, Berkeley, CA, USA, 209--224.

Digital Library

[8]

Sang Kil Cha, Maverick Woo, and David Brumley. 2015. Program-Adaptive Mutational Fuzzing. In 2015 IEEE Symposium on Security and Privacy. 725--741.

Digital Library

[9]

Arindam Chakrabarti and Patrice Godefroid. 2006. Software Partitioning for Effective Automated Unit Testing. In Proceedings of the 6th International Conference on Embedded Software (EMSOFT '06). ACM, New York, NY, USA, 262--271.

Digital Library

[10]

Christoph Csallner and Yannis Smaragdakis. 2004. JCrasher: An Automatic Robustness Tester for Java. Software Practical Experience 34, 11 (Sept. 2004), 1025--1050.

Digital Library

[11]

Hyunsook Do, Sebastian Elbaum, and Gregg Rothermel. 2005. Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and Its Potential Impact. Empirical Software Engineering 10, 4 (Oct. 2005), 405--435.

Digital Library

[12]

Sebastian Elbaum, Hui Nee Chin, Matthew B. Dwyer, and Matthew Jorde. 2009. Carving and Replaying Differential Unit Test Cases from System Test Cases. IEEE Transactions on Software Engineering 35, 1 (Jan 2009), 29--45.

Digital Library

[13]

Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: Automatic Test Suite Generation for Object-oriented Software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE '11). ACM, New York, NY, USA, 416--419.

Digital Library

[14]

Gordon Fraser and Andrea Arcuri. 2015. 1600 Faults in 100 Projects: Automatically Finding Faults While Achieving High Coverage with EvoSuite. Empirical Software Engineering 20, 3 (June 2015), 611--639.

Digital Library

[15]

Pranav Garg, Franjo Ivancic, Gogul Balakrishnan, Naoto Maeda, and Aarti Gupta. 2013. Feedback-directed Unit Test Generation for C/C++ Using Concolic Execution. In Proceedings of the 2013 International Conference on Software Engineering (ICSE '13). IEEE Press, Piscataway, NJ, USA, 132--141.

Digital Library

[16]

Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed Automated Random Testing. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '05). ACM, New York, NY, USA, 213--223.

Digital Library

[17]

Patrice Godefroid, Michael Y Levin, and David A Molnar. 2008. Automated Whitebox Fuzz Testing. In Proceedings of the 2008 Network and Distributed System Symposium, Vol. 8. 151--166.

[18]

Denis Gopan, Evan Driscoll, Ducson Nguyen, Dimitri Naydich, Alexey Loginov, and David Melski. 2015. Data-delineation in Software Binaries and Its Application to Buffer-overrun Discovery. In Proceedings of the 37th International Conference on Software Engineering - Volume 1 (ICSE '15). IEEE Press, Piscataway, NJ, USA, 145--155. http://dl.acm.org/citation.cfm?id=2818754.2818775

Digital Library

[19]

Florian Gross, Gordon Fraser, and Andreas Zeller. 2012. Search-based System Testing: High Coverage, No False Alarms. In Proceedings of the 2012 International Symposium on Software Testing and Analysis (ISSTA 2012). ACM, New York, NY, USA, 67--77.

Digital Library

[20]

Shin Hong, Byeongcheol Lee, Taehoon Kwak, Yiru Jeon, Bongsuk Ko, Yunho Kim, and Moonzoo Kim. 2015. Mutation-Based Fault Localization for Real-World Multilingual Programs (T). In Proceedings of the 201530th IEEE/ACM International Conference on Automated Software Engineering (ASE) (ASE '15). IEEE Computer Society, Washington, DC, USA, 464--475.

Digital Library

[21]

Hojun Jaygarl, Sunghun Kim, Tao Xie, and Carl K. Chang. 2010. OCAT: Object Capture-based Automated Testing. In Proceedings of the 19th International Symposium on Software Testing and Analysis (ISSTA '10). ACM, New York, NY, USA, 159--170.

Digital Library

[22]

Sarfraz Khurshid, Corina S. Păsăreanu, and Willem Visser. 2003. Generalized Symbolic Execution for Model Checking and Testing. In Proceedings of the 9th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS '03). Springer-Verlag, Berlin, Heidelberg, 553--568.

Digital Library

[23]

Moonzoo Kim, Yunho Kim, and Yunja Choi. 2012. Concolic testing of the multi-sector read operation for flash storage platform software. Formal Aspects of Computing 24, 3 (01 May 2012), 355--374.

[24]

Moonzoo Kim, Yunho Kim, and Yoonkyu Jang. 2012. Industrial Application of Concolic Testing on Embedded Software: Case Studies. In Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation (ICST '12). IEEE Computer Society, Washington, DC, USA, 390--399.

Digital Library

[25]

Moonzoo Kim, Yunho Kim, and Gregg Rothermel. 2012. A Scalable Distributed Concolic Testing Approach: An Empirical Evaluation. In 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation. 340--349.

Digital Library

[26]

Yunho Kim, Shin Hong, Bongseok Ko, Duy Loc Phan, and Moonzoo Kim. 2018. Invasive Software Testing: Mutating Target Programs to Diversify Test Exploration for High Test Coverage. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation.

[27]

Yunho Kim, Moonzoo Kim, YoungJoo Kim, and Yoonkyu Jang. 2012. Industrial Application of Concolic Testing Approach: A Case Study on Libexif by Using CREST-BV and KLEE. In Proceedings of the 34th International Conference on Software Engineering (ICSE '12). IEEE Press, Piscataway, NJ, USA, 1143--1152.

Digital Library

[28]

Yunho Kim, Youil Kim, Taeksu Kim, Gunwoo Lee, Yoonkyu Jang, and Moonzoo Kim. 2013. Automated Unit Testing of Large Industrial Embedded Software Using Concolic Testing. In Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering (ASE'13). IEEE Press, Piscataway, NJ, USA, 519--528.

Digital Library

[29]

Chris Lattner and Vikram Adve. 2004. LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation. In Proceedings of the International Symposium on Code Generation and Optimization: Feedback-directed and Runtime Optimization (CGO '04). IEEE Computer Society, Washington, DC, USA, 75--.

[30]

Wei Le and Shannon D. Pattison. 2014. Patch Verification via Multiversion Interprocedural Control Flow Graphs. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). ACM, New York, NY, USA, 1047--1058.

Digital Library

[31]

David Molnar, Xue Cong Li, and David A. Wagner. 2009. Dynamic Test Generation to Find Integer Bugs in x86 Binary Linux Programs. In Proceedings of the 18th Conference on USENIX Security Symposium (SSYM'09). USENIX Association, Berkeley, CA, USA, 67--82.

Digital Library

[32]

Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-Directed Random Test Generation. In Proceedings of the 29th International Conference on Software Engineering (ICSE '07). IEEE Computer Society, Washington, DC, USA, 75--84.

Digital Library

[33]

Yongbae Park, Shin Hong, Moonzoo Kim, Dongju Lee, and Junhee Cho. 2015. Systematic Testing of Reactive Software with Non-deterministic Events: A Case Study on LG Electric Oven. In Proceedings of the 37th International Conference on Software Engineering - Volume 2 (ICSE '15). IEEE Press, Piscataway, NJ, USA, 29--38.

Digital Library

[34]

Benny Pasternak, Shmuel Tyszberowicz, and Amiram Yehudai. 2009. GenUTest: a unit test and mock aspect generation tool. International Journal on Software Tools for Technology Transfer 11, 4 (03 Sep 2009), 273.

Digital Library

[35]

Corina S. Păsăreanu, Peter C. Mehlitz, David H. Bushnell, Karen Gundy-Burlet, Michael Lowry, Suzette Person, and Mark Pape. 2008. Combining Unit-level Symbolic Execution and System-level Concrete Execution for Testing NASA Software. In Proceedings of the 2008 International Symposium on Software Testing and Analysis (ISSTA '08). ACM, New York, NY, USA, 15--26.

Digital Library

[36]

Fernando Magno Quintao Pereira, Raphael Ernani Rodrigues, and Victor Hugo Sperle Campos. 2013. A Fast and Low-overhead Technique to Secure Programs Against Integer Overflows. In Proceedings of the 2013 IEEE/ACM International Symposium on Code Generation and Optimization (CGO) (CGO '13). IEEE Computer Society, Washington, DC, USA, 1--11.

Digital Library

[37]

David A. Ramos and Dawson Engler. 2015. Under-constrained Symbolic Execution: Correctness Checking for Real Code. In Proceedings of the 24th USENIX Conference on Security Symposium (SEC'15). USENIX Association, Berkeley, CA, USA, 49--64.

Digital Library

[38]

Alexandre Rebert, Sang Kil Cha, Thanassis Avgerinos, Jonathan Foote, David Warren, Gustavo Grieco, and David Brumley. 2014. Optimizing Seed Selection for Fuzzing. In 23rd USENIX Security Symposium (USENIX Security 14). USENIX Association, San Diego, CA, 861--875. https://www.usenix.org/conference/usenixsecurity14/technical-sessions/presentation/rebert

Digital Library

[39]

Raimondas Sasnauskas, Olaf Landsiedel, Muhammad Hamad Alizai, Carsten Weise, Stefan Kowalewski, and Klaus Wehrle. 2010. KleeNet: Discovering Insidious Interaction Bugs in Wireless Sensor Networks Before Deployment. In Proceedings of the 9th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN '10). ACM, New York, NY, USA, 186--196.

Digital Library

[40]

Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: A Concolic Unit Testing Engine for C. In Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE-13). ACM, New York, NY, USA, 263--272.

Digital Library

[41]

spec2006. The SPEC CPU 2006 Benchmark Suite. https://www.spec.org/cpu2006/.

[42]

Suresh Thummalapenta, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. 2009. MSeqGen: Object-oriented Unit-test Generation via Mining Source Code. In Proceedings of the the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering (ESEC/FSE '09). ACM, New York, NY, USA, 193--202.

Digital Library

[43]

Nikolai Tillmann and Jonathan De Halleux. 2008. Pex: White Box Test Generation for .NET. In Proceedings of the 2Nd International Conference on Tests and Proofs (TAP'08). Springer-Verlag, Berlin, Heidelberg, 134--153.

Digital Library

[44]

Aaron Tomb, Guillaume Brat, and Willem Visser. 2007. Variably Interprocedural Program Analysis for Runtime Error Detection. In Proceedings of the 2007 International Symposium on Software Testing and Analysis (ISSTA '07). ACM, New York, NY, USA, 97--107.

Digital Library

[45]

Willem Visser, Corina S. Păsăreanu, and Sarfraz Khurshid. 2004. Test Input Generation with Java PathFinder. In Proceedings of the 2004 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA '04). ACM, New York, NY, USA, 97--107.

Digital Library

[46]

Jonas Wagner, Volodymyr Kuznetsov, George Candea, and Johannes Kinder. 2015. High System-Code Security with Low Overhead. In 2015 IEEE Symposium on Security and Privacy. 866--879.

Digital Library

[47]

Xiaofei Xie, Yang Liu, Wei Le, Xiaohong Li, and Hongxu Chen. 2015. S-looper: Automatic Summarization for Multipath String Loops. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA 2015). ACM, New York, NY, USA, 188--198.

Digital Library

[48]

Fabian Yamaguchi, Alwin Maier, Hugo Gascon, and Konrad Rieck. 2015. Automatic Inference of Search Patterns for Taint-Style Vulnerabilities. In 2015 IEEE Symposium on Security and Privacy. 797--812.

Digital Library

Cited By

Vinzenz NOka DVinzenz NOka D(2024)Applying Concolic Testing to the Automotive DomainSAE Technical Paper Series10.4271/2024-01-2802Online publication date: 16-Apr-2024
https://doi.org/10.4271/2024-01-2802
Kim YChoi Y(2024)PBE-Based Selective Abstraction and Refinement for Efficient Property Falsification of Embedded SoftwareProceedings of the ACM on Software Engineering10.1145/36437401:FSE(293-315)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643740
Liu MHuang XHe WXie YZhang JJing XChen ZMa Y(2024)Research artifacts in software engineering publications: Status and trendsJournal of Systems and Software10.1016/j.jss.2024.112032213(112032)Online publication date: Jul-2024
https://doi.org/10.1016/j.jss.2024.112032
Show More Cited By

Index Terms

Precise concolic unit testing of C programs using extended units and symbolic alarm filtering
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

TDCT: Target-Driven Concolic Testing Using Extended Units by Calculating Function Relevance
Algorithms and Architectures for Parallel Processing
Abstract
Concolic unit testing is able to perform comprehensive analysis with a small function of the program. However, due to the following disadvantages, it cannot be widely and effectively applied to test the whole program. One is that it includes many ...
Automated unit testing of large industrial embedded software using concolic testing
ASE '13: Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering

Current testing practice in industry is often ineffective and slow to detect bugs, since most projects utilize manually generated test cases. Concolic testing alleviates this problem by automatically generating test cases that achieve high coverage. ...
Grey-box concolic testing on binary code
ICSE '19: Proceedings of the 41st International Conference on Software Engineering

We present grey-box concolic testing, a novel path-based test case generation method that combines the best of both white-box and grey-box fuzzing. At a high level, our technique systematically explores execution paths of a program under test as in white-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '18: Proceedings of the 40th International Conference on Software Engineering

May 2018

1307 pages

ISBN:9781450356381

DOI:10.1145/3180155

Conference Chair:
Michel Chaudron
Chalmers University of Technology, University of Gothenburg, Sweden
,
General Chair:
Ivica Crnkovic
Chalmers University of Technology, University of Gothenburg, Sweden
,
Program Chairs:
Marsha Chechik
University of Toronto, Canada
,
Mark Harman
Facebook and University College London, United Kingdom

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

Ministry of Science and ICT (MSIT)
Ministry of Education

Conference

ICSE '18

Sponsor:

SIGSOFT
IEEE-CS

ICSE '18: 40th International Conference on Software Engineering

May 27 - June 3, 2018

Gothenburg, Sweden

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
361
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Vinzenz NOka DVinzenz NOka D(2024)Applying Concolic Testing to the Automotive DomainSAE Technical Paper Series10.4271/2024-01-2802Online publication date: 16-Apr-2024
https://doi.org/10.4271/2024-01-2802
Kim YChoi Y(2024)PBE-Based Selective Abstraction and Refinement for Efficient Property Falsification of Embedded SoftwareProceedings of the ACM on Software Engineering10.1145/36437401:FSE(293-315)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643740
Liu MHuang XHe WXie YZhang JJing XChen ZMa Y(2024)Research artifacts in software engineering publications: Status and trendsJournal of Systems and Software10.1016/j.jss.2024.112032213(112032)Online publication date: Jul-2024
https://doi.org/10.1016/j.jss.2024.112032
Nguyen Tung LBinh Duong NLe KHung P(2024)Automated test data generation and stubbing method for C/C++ embedded projectsAutomated Software Engineering10.1007/s10515-024-00449-631:2Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1007/s10515-024-00449-6
Lee AAriq IKim YKim M(2022)POWER: Program Option-Aware Fuzzer for High Bug Detection Ability2022 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST53961.2022.00032(220-231)Online publication date: Apr-2022
https://doi.org/10.1109/ICST53961.2022.00032
Iwendi CRehman SJaved AKhan SSrivastava G(2021)Sustainable Security for the Internet of Things Using Artificial Intelligence ArchitecturesACM Transactions on Internet Technology10.1145/344861421:3(1-22)Online publication date: 16-Jun-2021
https://dl.acm.org/doi/10.1145/3448614
Fan MWang WYu AMeng D(2021)TDCT: Target-Driven Concolic Testing Using Extended Units by Calculating Function RelevanceAlgorithms and Architectures for Parallel Processing10.1007/978-3-030-95391-1_13(196-213)Online publication date: 3-Dec-2021
https://dl.acm.org/doi/10.1007/978-3-030-95391-1_13
Kim YMun SYoo SKim M(2019)Precise Learn-to-Rank Fault Localization Using Dynamic and Static Features of Target ProgramsACM Transactions on Software Engineering and Methodology10.1145/334562828:4(1-34)Online publication date: 9-Oct-2019
https://dl.acm.org/doi/10.1145/3345628
Cha SOh HDumas MPfahl DApel SRusso A(2019)Concolic testing with adaptively changing search heuristicsProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338964(235-245)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338964
Kim YHong SKim MDumas MPfahl DApel SRusso A(2019)Target-driven compositional concolic testing with function summary refinement for effective bug detectionProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338934(16-26)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338934
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents