research-article

Concolic testing for high test coverage and reduced human effort in automotive industry

Authors:

Moonzoo KimAuthors Info & Claims

ICSE-SEIP '19: Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice

Pages 151 - 160

https://doi.org/10.1109/ICSE-SEIP.2019.00024

Published: 27 May 2019 Publication History

Abstract

The importance of automotive software has been rapidly increasing because software now controls many components in motor vehicles such as window controller, smart-key system, and tire pressure monitoring system. Consequently, the automotive industry spends a large amount of human effort testing automotive software and is interested in automated software testing techniques that can ensure high-quality automotive software with reduced human effort.

In this paper, we report our industrial experience applying concolic testing to automotive software developed by Hyundai Mobis. We have developed an automated testing framework MAIST that automatically generates the test driver, stubs, and test inputs to a target task by applying concolic testing. As a result, MAIST has achieved 90.5% branch coverage and 77.8% MC/DC coverage on the integrated body unit (IBU) software. Furthermore, it reduced the cost of IBU coverage testing by reducing the manual testing effort for coverage testing by 53.3%.

References

[1]

M. Broy, "Challenges in automotive software engineering," in International Conference on Software Engineering (ICSE). New York, NY, USA: ACM, 2006, pp. 33--42.

Digital Library

[2]

M. Broy, I. H. Kruger, A. Pretschner, and C. Salzmann, "Engineering automotive software," Proceedings of the IEEE, vol. 95, no. 2, pp. 356--373, Feb 2007.

[3]

F. Falcini, G. Lami, and A. M. Costanza, "Deep learning in automotive software," IEEE Software, vol. 34, no. 3, pp. 56--63, May 2017.

Digital Library

[4]

M. Traub, A. Maier, and K. L. Barbehön, "Future automotive architecture and the impact of it trends," IEEE Software, vol. 34, no. 3, pp. 27--32, May 2017.

Digital Library

[5]

K. Sen, D. Marinov, and G. Agha, "CUTE: A concolic unit testing engine for C," in European Software Engineering Conference/Foundations of Software Engineering (ESEC/FSE), 2005, pp. 263--272.

Digital Library

[6]

N. Tillmann and W. Schulte, "Parameterized unit tests," in European Software Engineering Conference/Foundations of Software Engineering (ESEC/FSE), 2005.

Digital Library

[7]

M. Kim, Y. Kim, and Y. Choi, "Concolic testing of the multi-sector read operation for flash storage platform software," Formal Aspects of Computing, vol. 24, no. 3, pp. 355--374, May 2012.

[8]

M. Kim, Y. Kim, and Y. Jang, "Industrial application of concolic testing on embedded software: Case studies," in IEEE International Conference on Software Testing, Verification and Validation (ICST), 2012, pp. 390--399.

Digital Library

[9]

Y. Kim, M. Kim, Y. Kim, and Y. Jang, "Industrial application of concolic testing approach: A case study on libexif by using CREST-BV and KLEE," in International Conference on Software Engineering (ICSE), 2012, pp. 1143--1152.

Digital Library

[10]

Y. Kim, Y. Kim, T. Kim, G. Lee, Y. Jang, and M. Kim, "Automated unit testing of large industrial embedded software using concolic testing," in Automated Software Engineering (ASE), 2013, pp. 519--528.

Digital Library

[11]

"Hyundai mobis increases ai use for improved efficiency," http://www.koreaherald.com/view.php?ud=20180722000186, accessed: 2018-10-10.

[12]

"Hyundai mobis taps ai for car software," https://www.koreatimes.co.kr/www/tech/2018/07/419_252629.html, accessed: 2018-10-01.

[13]

"Hyundai mobis introduces ai-based software verification system," http://www.businesskorea.co.kr/news/articleView.html?idxno=23830, accessed: 2018-10-01.

[14]

Y. Kim, Y. Choi, and M. Kim, "Precise concolic unit testing of c programs using extended units and symbolic alarm filtering," in International Conference on Software Engineering (ICSE), 2018, pp. 315--326.

Digital Library

[15]

"Open hub, the open source network," https://www.openhub.net/.

[16]

C. Lattner and V. Adve, "LLVM: A compilation framework for lifelong program analysis & transformation," in Proceedings of the International Symposium on Code Generation and Optimization: Feedback-directed and Runtime Optimization, ser. CGO '04, 2004.

Digital Library

[17]

G. Necula, S. McPeak, S. Rahul, and W. Weimer, "CIL: Intermediate language and tools for analysis and transformation of c programs," in Compiler Construction (CC), 2002.

[18]

J. Burnim, "CREST - automatic test generation tool for C," http://code.google.com/p/crest/.

[19]

C. Cadar, D. Dunbar, and D. Engler, "KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs," in Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI'08, 2008, pp. 209--224.

Digital Library

[20]

N. Williams, B. Marre, P. Mouy, and M. Roger, "Pathcrawler: automatic generation of path tests by combining static and dynamic analysis," in EDCC, 2005.

Digital Library

[21]

R. Baldoni, E. Coppa, D. C. D'elia, C. Demetrescu, and I. Finocchi, "A survey of symbolic execution techniques," ACM Comput. Surv., vol. 51, no. 3, pp. 50:1--50:39, May 2018.

Digital Library

[22]

J. Burnim and K. Sen, "Heuristics for scalable dynamic test generation," EECS Department, University of California, Berkeley, Tech. Rep. UCB/EECS-2008-123, Sep 2008.

[23]

"Testwell CTC++," https://www.verifysoft.com/en_ctcpp.html, accessed: 2018-10-01.

[24]

N. Tillmann and J. De Halleux, "Pex: White box test generation for .NET," in Proceedings of the 2Nd International Conference on Tests and Proofs, ser. TAP'08. Berlin, Heidelberg: Springer-Verlag, 2008, pp. 134--153.

Digital Library

[25]

L. Ciortea, C. Zamfir, S. Bucur, V. Chipounov, and G. Candea, "Cloud9: a software testing service," ACM SIGOPS Operating Systems Review, vol. 43, no. 4, pp. 5--10, 2010.

Digital Library

[26]

K. Jayaraman, D. Harvison, V. Ganesh, and A. Kiezun, "jFuzz: A concolic whitebox fuzzer for Java," in NFM, 2009.

[27]

C. S. Pǎsǎreanu, P. C. Mehlitz, D. H. Bushnell, K. Gundy-Burlet, M. Lowry, S. Person, and M. Pape, "Combining unit-level symbolic execution and system-level concrete execution for testing NASA software," in Proceedings of the 2008 International Symposium on Software Testing and Analysis, 2008, pp. 15--26.

Digital Library

[28]

W. Visser, K. Havelund, G. Brat, and S. Park, "Model checking programs," in ASE, Sep. 2000.

Digital Library

[29]

P. Godefroid, N. Klarlund, and K. Sen, "DART: Directed automated random testing," in Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, 2005, pp. 213--223.

Digital Library

[30]

M. Kim, Y. Kim, and G. Rothermel, "A scalable distributed concolic testing approach: An empirical evaluation," in IEEE International Conference on Software Testing, Verification and Validation (ICST), April 2012, pp. 340--349.

Digital Library

[31]

K. Sen and G. Agha, "CUTE and jCUTE: Concolic unit testing and explicit path model-checking tools," in CAV, 2006.

Digital Library

[32]

"Catg: Concolic testing engine for java," https://github.com/ksen007/janala2, accessed: 2018-10-01.

[33]

D. A. Ramos and D. Engler, "Under-constrained symbolic execution: Correctness checking for real code," in Proceedings of the 24th USENIX Conference on Security Symposium, ser. SEC'15, 2015, pp. 49--64.

Digital Library

[34]

S. Khurshid, C. S. Păsăreanu, and W. Visser, "Generalized symbolic execution for model checking and testing," in Proceedings of the 9th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, ser. TACAS'03, 2003, pp. 553--568.

Digital Library

[35]

A. Chakrabarti and P. Godefroid, "Software partitioning for effective automated unit testing," in Proceedings of the 6th International Conference on Embedded Software, ser. EMSOFT '06. New York, NY, USA: ACM, 2006, pp. 262--271.

Digital Library

[36]

T. Kim, J. Park, I. Kulida, and Y. Jang, "Concolic testing framework for industrial embedded software," in 2014 21st Asia-Pacific Software Engineering Conference, vol. 2, Dec 2014, pp. 7--10.

[37]

C. Zhang, Y. Yan, H. Zhou, Y. Yao, K. Wu, T. Su, W. Miao, and G. Pu, "Smartunit: Empirical evaluations for automated unit testing of embedded software in industry," in Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, 2018, pp. 296--305.

Digital Library

[38]

P. Godefroid, A. Kiezun, and M. Y. Levin, "Grammar-based white-box fuzzing," in Programming Language Design and Implementation (PLDI), 2008.

Digital Library

[39]

"Microsoft security risk detection," https://www.microsoft.com/en-us/security-risk-detection/.

[40]

S. Bardin and P. Herrmann, "Structural testing of executables," in International Conference on Software Testing, Verification and Validation (ICST), 2008.

Digital Library

[41]

S. Bardin, "OSMOSE: automatic structural testing of executables," Journal of Software Testing, Verification, and Reliability (STVR), vol. 21, no. 1, pp. 29--54, 2011.

Digital Library

[42]

A. Franzén, A. Cimatti, A. Nadel, R. Sebastiani, and J. Shalev, "Applying smt in symbolic execution of microcode," in Formal Methods in Computer Aided Design, Oct 2010, pp. 121--128.

[43]

G. Li, I. Ghosh, and S. P. Rajan, "Klover: A symbolic execution and automatic test generation tool for c++ programs," in Computer Aided Verification, G. Gopalakrishnan and S. Qadeer, Eds., 2011, pp. 609--615.

Digital Library

[44]

M. Kim and Y. Kim, "Concolic testing of the multi-sector read operation for flash memory file system," in Brazilian Symposium on Formal Methods, 2009.

Digital Library

[45]

Y. Park, S. Hong, M. Kim, D. Lee, and J. Cho, "Systematic testing of reactive software with non-deterministic events: A case study on lg electric oven," in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol. 2, May 2015, pp. 29--38.

Digital Library

Cited By

Nguyen Tung LBinh Duong NLe KHung P(2024)Automated test data generation and stubbing method for C/C++ embedded projectsAutomated Software Engineering10.1007/s10515-024-00449-631:2Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1007/s10515-024-00449-6
Tung LTran HLe KHung P(2022)An automated test data generation method for void pointers and function pointers in C/C++ libraries and embedded projectsInformation and Software Technology10.1016/j.infsof.2022.106821145:COnline publication date: 9-May-2022
https://dl.acm.org/doi/10.1016/j.infsof.2022.106821
Kim YMun SYoo SKim M(2019)Precise Learn-to-Rank Fault Localization Using Dynamic and Static Features of Target ProgramsACM Transactions on Software Engineering and Methodology10.1145/334562828:4(1-34)Online publication date: 9-Oct-2019
https://dl.acm.org/doi/10.1145/3345628
Show More Cited By

Recommendations

Enhancing Test Cases generated by Concolic Testing
ISEC '19: Proceedings of the 12th Innovations in Software Engineering Conference (formerly known as India Software Engineering Conference)

Concolic testing tools can automatically generate test cases. However, there is a scope to increase the coverage achieved. Pair-wise testing is a prominent black-box testing method. We propose to combine ideas from these two techniques to enhance the ...
Grey-box concolic testing on binary code
ICSE '19: Proceedings of the 41st International Conference on Software Engineering

We present grey-box concolic testing, a novel path-based test case generation method that combines the best of both white-box and grey-box fuzzing. At a high level, our technique systematically explores execution paths of a program under test as in white-...
Micro-scale Concolic Testing Framework for Automated Test Data Generation Based on Path Coverage
ICSE-Companion '24: Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings

Automated test data generation based on path coverage (ATDG-PC) is an essential task in software testing. However, existing concolic testing approaches typically employ single static analyses or search-based algorithms for different paths without ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE-SEIP '19: Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice

May 2019

339 pages

Conference Chairs:
Helen Sharp
The Open University, UK
,
Mike Whalen
Amazon Inc.

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

IEEE Press

Publication History

Published: 27 May 2019

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '19

Sponsor:

SIGSOFT
IEEE-CS

ICSE '19: 41st International Conference on Software Engineering

May 27, 2019

Quebec, Montreal, Canada

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
98
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Nguyen Tung LBinh Duong NLe KHung P(2024)Automated test data generation and stubbing method for C/C++ embedded projectsAutomated Software Engineering10.1007/s10515-024-00449-631:2Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1007/s10515-024-00449-6
Tung LTran HLe KHung P(2022)An automated test data generation method for void pointers and function pointers in C/C++ libraries and embedded projectsInformation and Software Technology10.1016/j.infsof.2022.106821145:COnline publication date: 9-May-2022
https://dl.acm.org/doi/10.1016/j.infsof.2022.106821
Kim YMun SYoo SKim M(2019)Precise Learn-to-Rank Fault Localization Using Dynamic and Static Features of Target ProgramsACM Transactions on Software Engineering and Methodology10.1145/334562828:4(1-34)Online publication date: 9-Oct-2019
https://dl.acm.org/doi/10.1145/3345628
Kim YHong SKim MDumas MPfahl DApel SRusso A(2019)Target-driven compositional concolic testing with function summary refinement for effective bug detectionProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338934(16-26)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338934
Ahmadi RJahed KDingel JZimmermann TLawall JMarinov D(2019)mCUTEProceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE.2019.00132(1182-1185)Online publication date: 10-Nov-2019
https://dl.acm.org/doi/10.1109/ASE.2019.00132

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents