research-article

Inner oracles: input-specific assertions on internal states

Authors:

Tian LanAuthors Info & Claims

ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

Pages 902 - 905

https://doi.org/10.1145/2786805.2803204

Published: 30 August 2015 Publication History

Abstract

Traditional test oracles are defined on the outputs of test executions, and cannot assert internal states of executions. Traditional assertions are common to all test execution, and are usually more difficult to construct than on oracle for one test input. In this paper we propose the concept of inner oracles, which are assertions on internal states that are specific to one test input. We first motivate the necessity of inner oracles, and then show that it can be implemented easily using the available programming mechanisms. Next, we report two initial empirical studies on inner oracles, showing that inner oracles have a significant impact on both the fault-detection capability of tests and the performance of test suite reduction. Finally, we highlight the implications of inner oracles on several research and practical problems.

References

[1]

E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo. The oracle problem in software testing: A survey. IEEE TSE, 41(5):507–525, 2014.

[2]

K. Beck. Test-driven development: by example. Addison-Wesley Professional, 2003.

Digital Library

[3]

H. Do, S. Elbaum, and G. Rothermel. Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. ESE, 10(4):405–435, 2005.

Digital Library

[4]

C. Fang, Z. Chen, and B. Xu. Comparing logic coverage criteria on test case prioritization. SCIENCE CHINA Information Sciences, 55(12):2826–2840, 2012.

[5]

R. G. Hamlet. Testing programs with the aid of a compiler. IEEE TSE, 3(4):279–290, 1977.

Digital Library

[6]

D. Hao, L. Zhang, M. Liu, H. Li, and J. Sun. Test-data generation guided by static defect detection. JCST, 24(2):284–293, 2009.

Digital Library

[7]

D. Hao, L. Zhang, L. Zhang, G. Rothermel, and H. Mei. A unified test-case prioritization approach. ACM TOSEM, 24(2):1–31, 2014.

Digital Library

[8]

W. E. Howden. Weak mutation testing and completeness of test sets. IEEE TSE, 8(4):371–379, 1982.

Digital Library

[9]

G. Kiczales, J. Lamping, A. Mendhekar, C. Maeda, C. Lopes, J.-M. Loingtier, and J. Irwin. Aspect-oriented programming. Springer, 1997.

[10]

P. McMinn. Search-based software test data generation: A survey. STVR, 14(2):105–156, 2004.

Digital Library

[11]

H. Mei, D. Hao, L. Zhang, L. Zhang, J. Zhou, and G. Rothermel. A static approach to prioritizing JUnit test cases. IEEE TSE, 38(6):1258–1275, 2012.

Digital Library

[12]

L. Pinto, S. Sinha, and A. Orso. Understanding myths and realities of test-suite evolution. In Proc. FSE, pages 1–11, 2012.

Digital Library

[13]

D. Schuler and A. Zeller. Javalanche: Efficient mutation testing for Java. In Proc. FSE, pages 297–298, 2009.

Digital Library

[14]

M. Staats, M. W. Whalen, and M. P. Heimdahl. Programs, tests, and oracles: The foundations of testing revisited. In Proc. ICSE, pages 391–400, 2011.

Digital Library

[15]

J. M. Voas. PIE: A dynamic failure-based technique. IEEE TSE, 18(8):717–727, 1992.

Digital Library

[16]

T. Xie. Augmenting automatically generated unit-test suites with regression oracle checking. In Proc. ECOOP, pages 380–403. Springer, 2006.

Digital Library

[17]

J. Zhang, J. Chen, D. Hao, Y. Xiong, B. Xie, L. Zhang, and H. Mei. Search-based inference of polynomial metamorphic relations. In ASE, pages 701–712, 2014.

Digital Library

[18]

L. Zhang, D. Marinov, L. Zhang, and S. Khurshid. An empirical study of junit test-suite reduction. In Proc. ISSRE, pages 170–179, 2011.

Digital Library

[19]

H. Zhong, L. Zhang, and H. Mei. An experimental study of four typical test suite reduction techniques. IST, 50(6):534–546, 2008.

Digital Library

[20]

H. Zhu. A formal analysis of the subsume relation between software test adequacy criteria. IEEE TSE, 22(4):248–255, 1996.

Digital Library

Cited By

Du HPalepu VJones JRoychoudhury APaiva AAbreu RStorey M(2024)Ripples of a Mutation — An Empirical Study of Propagation Effects in Mutation TestingProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639179(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639179
Jahangirova GBultan TSen K(2017)Oracle problem in software testingProceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3092703.3098235(444-447)Online publication date: 10-Jul-2017
https://dl.acm.org/doi/10.1145/3092703.3098235
Chen JBai YHao DZhang LZhang LXie BMei HLo DApel SKhurshid S(2016)Supporting oracle construction via static analysisProceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering10.1145/2970276.2970366(178-189)Online publication date: 25-Aug-2016
https://dl.acm.org/doi/10.1145/2970276.2970366
Show More Cited By

Index Terms

Inner oracles: input-specific assertions on internal states
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Assertions are strongly correlated with test suite effectiveness
ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

Code coverage is a popular test adequacy criterion in practice. Code coverage, however, remains controversial as there is a lack of coherent empirical evidence for its relation with test suite effectiveness. More recently, test suite size has been ...
Mutation-driven generation of unit tests and oracles
ISSTA '10: Proceedings of the 19th international symposium on Software testing and analysis

To assess the quality of test suites, mutation analysis seeds artificial defects (mutations) into programs; a non-detected mutation indicates a weakness in the test suite. We present an automated approach to generate unit tests that detect these ...
Mutation-Driven Generation of Unit Tests and Oracles

To assess the quality of test suites, mutation analysis seeds artificial defects (mutations) into programs; a nondetected mutation indicates a weakness in the test suite. We present an automated approach to generate unit tests that detect these ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

August 2015

1068 pages

ISBN:9781450336758

DOI:10.1145/2786805

General Chair:
Elisabetta Di Nitto
Politecnico di Milano, Italy
,
Program Chairs:
Mark Harman
University College London, UK
,
Patrick Heymans
University of Namur, Belgium

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 August 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

High-Tech Research and Development Program of Chin
National Natural Science Foundation of China

Conference

ESEC/FSE'15

Sponsor:

SIGSOFT

ESEC/FSE'15: Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering

August 30 - September 4, 2015

Bergamo, Italy

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
230
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)0

Reflects downloads up to 07 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Du HPalepu VJones JRoychoudhury APaiva AAbreu RStorey M(2024)Ripples of a Mutation — An Empirical Study of Propagation Effects in Mutation TestingProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639179(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639179
Jahangirova GBultan TSen K(2017)Oracle problem in software testingProceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3092703.3098235(444-447)Online publication date: 10-Jul-2017
https://dl.acm.org/doi/10.1145/3092703.3098235
Chen JBai YHao DZhang LZhang LXie BMei HLo DApel SKhurshid S(2016)Supporting oracle construction via static analysisProceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering10.1145/2970276.2970366(178-189)Online publication date: 25-Aug-2016
https://dl.acm.org/doi/10.1145/2970276.2970366
Patrick MCraig ACunniffe NParry MGilligan CZeller ARoychoudhury A(2016)Testing stochastic software using pseudo-oraclesProceedings of the 25th International Symposium on Software Testing and Analysis10.1145/2931037.2931063(235-246)Online publication date: 18-Jul-2016
https://dl.acm.org/doi/10.1145/2931037.2931063
Cheng YWang MXiong YHao DZhang L(2016)Empirical Evaluation of Test Coverage for Functional Programs2016 IEEE International Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST.2016.8(255-265)Online publication date: Apr-2016
https://doi.org/10.1109/ICST.2016.8

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents