research-article

Smartunit: empirical evaluations for automated unit testing of embedded software in industry

Authors:

Geguang PuAuthors Info & Claims

ICSE-SEIP '18: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice

Pages 296 - 305

https://doi.org/10.1145/3183519.3183554

Published: 27 May 2018 Publication History

Abstract

In this paper, we aim at the automated unit coverage-based testing for embedded software. To achieve the goal, by analyzing the industrial requirements and our previous work on automated unit testing tool CAUT, we rebuild a new tool, SmartUnit, to solve the engineering requirements that take place in our partner companies. SmartUnit is a dynamic symbolic execution implementation, which supports statement, branch, boundary value and MC/DC coverage.

SmartUnit has been used to test more than one million lines of code in real projects. For confidentiality motives, we select three in-house real projects for the empirical evaluations. We also carry out our evaluations on two open source database projects, SQLite and PostgreSQL, to test the scalability of our tool since the scale of the embedded software project is mostly not large, 5K-50K lines of code on average. From our experimental results, in general, more than 90% of functions in commercial embedded software achieve 100% statement, branch, MC/DC coverage, more than 80% of functions in SQLite achieve 100% MC/DC coverage, and more than 60% of functions in PostgreSQL achieve 100% MC/DC coverage. Moreover, SmartUnit is able to find the runtime exceptions at the unit testing level. We also have reported exceptions like array index out of bounds and divided-by-zero in SQLite. Furthermore, we analyze the reasons of low coverage in automated unit testing in our setting and give a survey on the situation of manual unit testing with respect to automated unit testing in industry.

References

[1]

M Moein Almasi, Hadi Hemmati, Gordon Fraser, Andrea Arcuri, and Jānis Benefelds. 2017. An industrial evaluation of unit test generation: Finding real faults in a financial application. In Proceedings of the 39th International Conference on Software Engineering: Software Engineering in Practice Track. IEEE Press, 263--272.

Digital Library

[2]

Saswat Anand, Corina Păsăreanu, and Willem Visser. 2007. JPF-SE: A symbolic execution extension to java pathfinder. Tools and Algorithms for the Construction and Analysis of Systems (2007), 134--138.

Digital Library

[3]

IEEE Standards Association et al. 1990. Standard glossary of software engineering terminology. IEEE Std (1990), 610--12.

[4]

Clark Barrett. 2013. fiDecision Procedures: An Algorithmic Point of View,fi by Daniel Kroening and Ofer Strichman, Springer-Verlag, 2008. Journal of Automated Reasoning 51, 4(2013), 453--456.

Digital Library

[5]

Clark Barrett, Christopher L Conway, Morgan Deters, Liana Hadarean, Dejan Jovanović, Tim King, Andrew Reynolds, and Cesare Tinelli. 2011. Cvc4. In International Conference on Computer Aided Verification. Springer, 171--177.

Digital Library

[6]

Jacob Burnim and Koushik Sen. 2008. Heuristics for scalable dynamic test generation. In Automated Software Engineering, 2008. ASE 2008. 23rd IEEE/ACM International Conference on. IEEE, 443--446.

Digital Library

[7]

Cristian Cadar, Daniel Dunbar, Dawson R Engler, et al. 2008. KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. In OSDI, Vol. 8. 209--224.

Digital Library

[8]

Cristian Cadar, Vijay Ganesh, Peter M Pawlowski, David L Dill, and Dawson R Engler. 2008. EXE: automatically generating inputs of death. ACM Transactions on Information and System Security (TISSEC) 12, 2 (2008), 10.

Digital Library

[9]

Cristian Cadar, Patrice Godefroid, Sarfraz Khurshid, Corina S Păsăreanu, Koushik Sen, Nikolai Tillmann, and Willem Visser. 2011. Symbolic execution for software testing in practice: preliminary assessment. In Proceedings of the 33rd International Conference on Software Engineering. ACM, 1066--1071.

Digital Library

[10]

Cristian Cadar and Koushik Sen. 2013. Symbolic execution for software testing: three decades later. Commun. ACM 56, 2 (2013), 82--90.

Digital Library

[11]

Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An efficient SMT solver. Tools and Algorithms for the Construction and Analysis of Systems (2008), 337--340.

Digital Library

[12]

Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. ACM, 416--419.

Digital Library

[13]

Gordon Fraser and Andrea Arcuri. 2014. A large-scale evaluation of automated unit test generation using evosuite. ACM Transactions on Software Engineering and Methodology (TOSEM) 24, 2 (2014), 8.

Digital Library

[14]

Vijay Ganesh and David L Dill. 2007. A decision procedure for bit-vectors and arrays. In CAV, Vol. 4590. Springer, 519--531.

Digital Library

[15]

Pranav Garg, Franjo Ivancic, Gogul Balakrishnan, Naoto Maeda, and Aarti Gupta. 2013. Feedback-directed unit test generation for C/C++ using concolic execution. In Proceedings of the 2013 International Conference on Software Engineering. IEEE Press, 132--141.

Digital Library

[16]

Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: directed automated random testing. In ACM Sigplan Notices, Vol. 40. ACM, 213--223.

Digital Library

[17]

Patrice Godefroid, Michael Y Levin, and David Molnar. 2012. SAGE: whitebox fuzzing for security testing. Queue 10, 1 (2012), 20.

Digital Library

[18]

Patrice Godefroid, Michael Y Levin, David A Molnar, et al. 2008. Automated whitebox fuzz testing. In NDSS, Vol. 8. 151--166.

[19]

Yunho Kim, Moonzoo Kim, and Yoonkyu Jang. 2011. Concolic testing on embedded software-case studies on mobile platform programs. In European Software Engineering Conference/Foundations of Software Engineering (ESEC/FSE) Industrial Track, Vol. 29. 30.

[20]

Yunho Kim, Moonzoo Kim, YoungJoo Kim, and Yoonkyu Jang. 2012. Industrial application of concolic testing approach: A case study on libexif by using CREST-BV and KLEE. In Software Engineering (ICSE), 2012 34th International Conference on. IEEE, 1143--1152.

Digital Library

[21]

James C King. 1976. Symbolic execution and program testing. Commun. ACM 19, 7 (1976), 385--394.

Digital Library

[22]

Guodong Li, Indradeep Ghosh, and Sreeranga Rajan. 2011. KLOVER: A symbolic execution and automatic test generation tool for C++ programs. In Computer Aided Verification. Springer, 609--615.

Digital Library

[23]

Kin-Keung Ma, Khoo Yit Phang, Jeffrey Foster, and Michael Hicks. 2011. Directed symbolic execution. Static Analysis (2011), 95--111.

Digital Library

[24]

Rupak Majumdar and Ru-Gang Xu. 2009. Reducing Test Inputs Using Information Partitions. In CAV, Vol. 9. Springer, 555--569.

Digital Library

[25]

Weikai Miao, Geguang Pu, Yinbo Yao, Ting Su, Danzhu Bao, Yang Liu, Shuohao Chen, and Kunpeng Xiong. 2016. Automated Requirements Validation for ATP Software via Specification Review and Testing. In Formal Methods and Software Engineering - 18th International Conference on Formal Engineering Methods, ICFEM 2016, Tokyo, Japan, November 14--18, 2016, Proceedings. 26--40.

[26]

Carlos Pacheco and Michael D Ernst. 2007. Randoop: feedback-directed random testing for Java. In Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion. ACM, 815--816.

Digital Library

[27]

Brian Robinson, Michael D Ernst, Jeff H Perkins, Vinay Augustine, and Nuo Li. 2011. Scaling up automated test generation: Automatically generating maintainable regression unit tests for programs. In Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering. IEEE Computer Society, 23--32.

Digital Library

[28]

Per Runeson. 2006. A survey of unit testing practices. IEEE software 23, 4 (2006), 22--29.

Digital Library

[29]

Koushik Sen. 2006. Scalable automated methods for dynamic program analysis. Technical Report.

[30]

Koushik Sen and Gul Agha. 2006. CUTE and jCUTE: Concolic unit testing and explicit path model-checking tools. In CAV, Vol. 6. Springer, 419--423.

Digital Library

[31]

Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: a concolic unit testing engine for C. In ACM SIGSOFT Software Engineering Notes, Vol. 30. ACM, 263--272.

Digital Library

[32]

Sina Shamshiri, Rene Just, Jose Miguel Rojas, Gordon Fraser, Phil McMinn, and Andrea Arcuri. 2015. Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges (t). In Automated Software Engineering (ASE), 2015 30th IEEE/ACM International Conference on. IEEE, 201--211.

Digital Library

[33]

Ting Su, Zhoulai Fu, Geguang Pu, Jifeng He, and Zhendong Su. 2015. Combining symbolic execution and model checking for data flow testing. In Proceedings of the 37th International Conference on Software Engineering-Volume 1. IEEE Press, 654--665.

Digital Library

[34]

Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su. 2017. Guided, Stochastic Model-based GUI Testing of Android Apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2017). ACM, New York, NY, USA, 245--256.

Digital Library

[35]

Ting Su, Geguang Pu, Bin Fang, Jifeng He, Jun Yan, Siyuan Jiang, and Jianjun Zhao. 2014. Automated coverage-driven test data generation using dynamic symbolic execution. In Software Security and Reliability, 2014 Eighth International Conference on. IEEE, 98--107.

Digital Library

[36]

Ting Su, Geguang Pu, Weikai Miao, Jifeng He, and Zhendong Su. 2016. Automated coverage-driven testing: combining symbolic execution and model checking. SCIENCE CHINA Information Sciences 59, 9 (2016), 98101.

[37]

Ting Su, Ke Wu, Weikai Miao, Geguang Pu, Jifeng He, Yuting Chen, and Zhendong Su. 2017. A Survey on Data-Flow Testing. ACM Comput. Surv. 50, 1, Article 5 (March 2017), 35 pages.

Digital Library

[38]

Nikolai Tillmann and Jonathan De Halleux. 2008. Pex-white box test generation for. net. Tests and Proofs (2008), 134--153.

Digital Library

[39]

Nikolai Tillmann and Wolfram Schulte. 2005. Parameterized unit tests with unit meister. In ACM SIGSOFT Software Engineering Notes, Vol. 30. ACM, 241--244.

Digital Library

[40]

Susumu Tokumoto, Tadahiro Uehara, Kazuki Munakata, Haruyuki Ishida, Toru Eguchi, and Masafumi Baba. 2012. Enhancing symbolic execution to test the compatibility of re-engineered industrial software. In Software Engineering Conference (APSEC), 2012 19th Asia-Pacific, Vol. 1. IEEE, 314--317.

Digital Library

[41]

Gina Venolia, Robert DeLine, and Thomas LaToza. 2005. Software development at microsoft observed. Microsoft Research, TR (2005).

[42]

Tao Xie and David Notkin. 2003. Tool-assisted unit test selection based on operational violations. In Automated Software Engineering, 2003. Proceedings. 18th IEEE International Conference on. IEEE, 40--48.

Digital Library

[43]

Sai Zhang, David Saff, Yingyi Bu, and Michael D Ernst. 2011. Combined static and dynamic automated test generation. In Proceedings of the 2011 International Symposium on Software Testing and Analysis. ACM, 353--363.

Digital Library

[44]

Wujie Zheng, Qirun Zhang, Michael Lyu, and Tao Xie. 2010. Random unit-test generation with MUT-aware sequence recommendation. In Proceedings of the IEEE/ACM international conference on Automated software engineering. ACM, 293--296.

Digital Library

[45]

Hong Zhu, Patrick AV Hall, and John HR May. 1997. Software unit test coverage and adequacy. Acm computing surveys (csur) 29, 4 (1997), 366--427.

Digital Library

Cited By

Haider SShabbir HIqbal MAhmad SArif S(2024)Fuzzy Based Expert System For Test Case Generation On Web Graphical User Interface For Usability Test ImprovementBulletin of Business and Economics (BBE)10.61506/01.0041913:2(990-998)Online publication date: 1-Jun-2024
https://doi.org/10.61506/01.00419
Ayaz ZRehman HArshid NAli RIqbal MNoor MAhmad S(2024)Expert System for Web Graphical User Interface Test Case Generation for Usability TestBulletin of Business and Economics (BBE)10.61506/01.0022813:1Online publication date: 25-Mar-2024
https://doi.org/10.61506/01.00228
Shi JChen YLi QHuang YYang YZhao M(2024)Automated Test Cases Generator for IEC 61131-3 Structured Text Based Dynamic Symbolic ExecutionIEEE Transactions on Computers10.1109/TC.2024.335128573:4(1048-1059)Online publication date: 8-Jan-2024
https://dl.acm.org/doi/10.1109/TC.2024.3351285
Show More Cited By

Index Terms

Smartunit: empirical evaluations for automated unit testing of embedded software in industry
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Empirical software validation
      2. Software defect analysis
        Software testing and debugging
  2. Software organization and properties
    1. Software functional properties
      1. Formal methods
        Dynamic analysis

Recommendations

Evaluating Automated Unit Testing in Sulu
ICST '08: Proceedings of the 2008 International Conference on Software Testing, Verification, and Validation

Sulu is a programming language designed with automated unit testing specifically in mind. One aim of Sulu is to demonstrate how automated software testing can be more integrated into current software development processes. Sulu's runtime and tools ...
CUT: automatic unit testing in the cloud
ISSTA 2017: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis

Unit tests can be significantly sped up by running them in parallel over distributed execution environments, such as the cloud. However, manually setting up such environments and configuring the testing frameworks to effectively use them is cumbersome ...
Experience report: how is dynamic symbolic execution different from manual testing? a study on KLEE
ISSTA 2015: Proceedings of the 2015 International Symposium on Software Testing and Analysis

Software testing has been the major approach to software quality assurance for decades, but it typically involves intensive manual efforts. To reduce manual efforts, researchers have proposed numerous approaches to automate test-case generation, which ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE-SEIP '18: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice

May 2018

336 pages

ISBN:9781450356596

DOI:10.1145/3183519

Conference Chairs:
Frances Paulisch
Siemens Healthineers, Germany
,
Jan Bosch
Chalmers University of Technology

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '18

Sponsor:

SIGSOFT
IEEE-CS

ICSE '18: 40th International Conference on Software Engineering

May 27 - June 3, 2018

Gothenburg, Sweden

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
209
Total Downloads

Downloads (Last 12 months)22
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Haider SShabbir HIqbal MAhmad SArif S(2024)Fuzzy Based Expert System For Test Case Generation On Web Graphical User Interface For Usability Test ImprovementBulletin of Business and Economics (BBE)10.61506/01.0041913:2(990-998)Online publication date: 1-Jun-2024
https://doi.org/10.61506/01.00419
Ayaz ZRehman HArshid NAli RIqbal MNoor MAhmad S(2024)Expert System for Web Graphical User Interface Test Case Generation for Usability TestBulletin of Business and Economics (BBE)10.61506/01.0022813:1Online publication date: 25-Mar-2024
https://doi.org/10.61506/01.00228
Shi JChen YLi QHuang YYang YZhao M(2024)Automated Test Cases Generator for IEC 61131-3 Structured Text Based Dynamic Symbolic ExecutionIEEE Transactions on Computers10.1109/TC.2024.335128573:4(1048-1059)Online publication date: 8-Jan-2024
https://dl.acm.org/doi/10.1109/TC.2024.3351285
Su TZhang CYan YFan LLiu YFu ZSu Z(2023)Towards Efficient Data-Flow Test Data GenerationTheories of Programming and Formal Methods10.1007/978-3-031-40436-8_10(257-293)Online publication date: 8-Sep-2023
https://doi.org/10.1007/978-3-031-40436-8_10
Tzoref-Brill RSinha SNassar AGoldin VKermany H(2022)TackleTest: A Tool for Amplifying Test Generation via Type-Based Combinatorial Coverage2022 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST53961.2022.00050(444-455)Online publication date: Apr-2022
https://doi.org/10.1109/ICST53961.2022.00050
He WMao XSu THuang YShi J(2021)Data Flow Testing for PLC Programs via Dynamic Symbolic Execution2021 28th Asia-Pacific Software Engineering Conference (APSEC)10.1109/APSEC53868.2021.00023(152-160)Online publication date: Dec-2021
https://doi.org/10.1109/APSEC53868.2021.00023
Feng JMiao WZheng HHuang YLi JWang ZSu TGu BPu GYang MHe JDevanbu PCohen MZimmermann T(2020)FREPA: an automated and formal approach to requirement modeling and analysis in aircraft control domainProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3368089.3417047(1376-1386)Online publication date: 8-Nov-2020
https://dl.acm.org/doi/10.1145/3368089.3417047
Ivannikov A(2020)Emulators – Digital System Simulation on the Architecture LevelRecent Research in Control Engineering and Decision Making10.1007/978-3-030-65283-8_18(209-222)Online publication date: 2-Dec-2020
https://doi.org/10.1007/978-3-030-65283-8_18
Hao LShi JSu THuang Y(2019)Automated Test Generation for IEC 61131-3 ST Programs via Dynamic Symbolic Execution2019 International Symposium on Theoretical Aspects of Software Engineering (TASE)10.1109/TASE.2019.00004(200-207)Online publication date: Jul-2019
https://doi.org/10.1109/TASE.2019.00004
Ahishakiye FJaksic SStolz VLange FSchmitz MThoma D(2019)Non-Intrusive MC/DC Measurement Based on Traces2019 International Symposium on Theoretical Aspects of Software Engineering (TASE)10.1109/TASE.2019.00-15(86-92)Online publication date: Jul-2019
https://doi.org/10.1109/TASE.2019.00-15

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents