article

Branching processes for QuickCheck generators

Authors:

Agustín Mista,

Alejandro Russo,

John HughesAuthors Info & Claims

ACM SIGPLAN Notices, Volume 53, Issue 7

Pages 1 - 13

https://doi.org/10.1145/3299711.3242747

Published: 17 September 2018 Publication History

Abstract

In QuickCheck (or, more generally, random testing), it is challenging to control random data generators' distributions---specially when it comes to user-defined algebraic data types (ADT). In this paper, we adapt results from an area of mathematics known as branching processes, and show how they help to analytically predict (at compile-time) the expected number of generated constructors, even in the presence of mutually recursive or composite ADTs. Using our probabilistic formulas, we design heuristics capable of automatically adjusting probabilities in order to synthesize generators which distributions are aligned with users' demands. We provide a Haskell implementation of our mechanism in a tool called DRaGeN and perform case studies with real-world applications. When generating random values, our synthesized QuickCheck generators show improvements in code coverage when compared with those automatically derived by state-of-the-art tools.

References

[1]

A. Arcuri and L. Briand. 2011. Adaptive Random Testing: An Illusion of Effectiveness?. In Proc. of the International Symposium on Software Testing and Analysis (ISSTA ’11). ACM.

Digital Library

[2]

B. Arkin, S. Stender, and G. McGraw. 2005. Software penetration testing. IEEE Security Privacy (2005).

Digital Library

[3]

T. Arts, J. Hughes, U. Norell, and H. Svensson. 2015. Testing AUTOSAR software with QuickCheck. In In Proc. of IEEE International Conference on Software Testing, Verification and Validation, ICST Workshops.

[4]

M. Bendkowski, K. Grygiel, and P. Tarau. 2017. Boltzmann Samplers for Closed Simply-Typed Lambda Terms. In In Proc. of International Symposium on Practical Aspects of Declarative Languages. ACM.

[5]

F.T. Chan, T.Y. Chen, I.K. Mak, and Y.T. Yu. 1996. Proportional sampling strategy: guidelines for software testing practitioners. Information and Software Technology 38, 12 (1996), 775 – 782.

[6]

T. Y. Chen, H. Leung, and I. K. Mak. 2005. Adaptive Random Testing. In Advances in Computer Science - ASIAN 2004. Higher-Level Decision Making, Michael J. Maher (Ed.). Springer Berlin Heidelberg.

Digital Library

[7]

I. Ciupa, A. Leitner, M. Oriol, and B. Meyer. 2008. ARTOO: adaptive random testing for object-oriented software. In Proc. of International Conference on Software Engineering. ACM/IEEE.

Digital Library

[8]

K. Claessen, J. Duregård, and M. H. Palka. 2014. Generating Constrained Random Data with Uniform Distribution. In Proc. of the Functional and Logic Programming FLOPS.

[9]

K. Claessen and J. Hughes. 2000. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. In Proc. of the ACM SIGPLAN International Conference on Functional Programming (ICFP).

Digital Library

[10]

P. Duchon, P. Flajolet, G. Louchard, and G. Schaeffer. 2004. Boltzmann Samplers for the Random Generation of Combinatorial Structures. Combinatorics, Probability and Computing. 13 (2004).

Digital Library

[11]

J. Duregård, P. Jansson, and M. Wang. 2012. Feat: Functional enumeration of algebraic types. In Proc. of the ACM SIGPLAN Symposium on Haskell.

Digital Library

[12]

R. Feldt and S. Poulding. 2013. Finding test data with specific properties via metaheuristic search. In Proc. of International Symp. on Software Reliability Engineering (ISSRE). IEEE.

[13]

G. Grieco, M. Ceresa, and P. Buiras. 2016. QuickFuzz: An automatic random fuzzer for common file formats. In Proc. of the International Symposium on Haskell. ACM.

Digital Library

[14]

G. Grieco, M. Ceresa, A. Mista, and P. Buiras. 2017. QuickFuzz testing for fun and profit. Journal of Systems and Software 134, Supp. C (2017).

Digital Library

[15]

P. Haccou, P. Jagers, and V. Vatutin. 2005. Branching processes. Variation, growth, and extinction of populations. Cambridge University Press.

[16]

J. Hughes, C. Pierce B, T. Arts, and U. Norell. 2016. Mysteries of DropBox: Property-Based Testing of a Distributed Synchronization Service. In Proc. of the Int. Conference on Software Testing, Verification and Validation, ICST.

[17]

J. Hughes, U. Norell, N. Smallbone, and T. Arts. 2016. Find more bugs with QuickCheck!. In Proc. of the International Workshop on Automation of Software Test, AST@ICSE.

Digital Library

[18]

L. Lampropoulos, D. Gallois-Wong, C. Hritcu, J. Hughes, B. C. Pierce, and L. Xia. 2017. Beginner’s luck: a language for property-based generators. In Proc. of the ACM SIGPLAN Symposium on Principles of Programming Languages, POPL.

Digital Library

[19]

L. Lampropoulos, Z. Paraskevopoulou, and B. C. Pierce. 2017. Generating Good Generators for Inductive Relations. In Proc. ACM on Programming Languages 2, POPL, Article 45 (2017).

Digital Library

[20]

M. Zalewski. 2010. American Fuzzy Lop: a security-oriented fuzzer. http://lcamtuf.coredump.cx/afl/ . (2010).

[21]

C. McBride and R. Paterson. 2008. Applicative Programming with Effects. Journal of Functional Programming 18, 1 (Jan. 2008).

Digital Library

[22]

J. Midtgaard, M. N. Justesen, P. Kasting, F. Nielson, and H. R. Nielson. 2017. Effect-driven QuickChecking of compilers. In Proceedings of the ACM on Programming Languages, Volume 1 ICFP (2017).

Digital Library

[23]

A. Mista, A. Russo, and J. Hughes. 2018. Branching Processes for QuickCheck Generators (extended version). https://bitbucket.org/ agustinmista/dragen/downloads/full-paper.pdf . (2018).

[24]

N. Mitchell. 2007. Deriving Generic Functions by Example. In Proc. of the 1st York Doctoral Syposium. Tech. Report YCS-2007-421, Department of Computer Science, University of York, UK, 55–62.

[25]

M. Pałka, K. Claessen, A. Russo, and J. Hughes. 2011. Testing and Optimising Compiler by Generating Random Lambda Terms. In The IEEE/ACM International Workshop on Automation of Software Test (AST 2011).

Digital Library

[26]

S. M. Poulding and R. Feldt. 2017. Automated Random Testing in Multiple Dispatch Languages. IEEE International Conference on Software Testing, Verification and Validation (ICST) (2017).

[27]

C. Runciman, M. Naylor, and F. Lindblad. 2008. Smallcheck and Lazy Smallcheck: automatic exhaustive testing for small values. In Proc. of the ACM SIGPLAN Symposium on Haskell.

Digital Library

[28]

T. Sheard and Simon L. Peyton Jones. 2002. Template meta-programming for Haskell. SIGPLAN Notices 37, 12 (2002), 60–75.

Digital Library

[29]

H. W. Watson and F. Galton. 1875. On the probability of the extinction of families. The Journal of the Anthropological Institute of Great Britain and Ireland (1875).

Cited By

Zago NBraga AMello BFeitosa S(2023)Usando Esquema GraphQL para Geração de Consultas de Forma AleatóriaProceedings of the XXVII Brazilian Symposium on Programming Languages10.1145/3624309.3624321(17-24)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3624309.3624321
Arts TSvensson HBenac Earle CFredlund L(2023)Testing feature‐rich blockchainsSoftware: Practice and Experience10.1002/spe.318353:5(1144-1173)Online publication date: 16-Jan-2023
https://doi.org/10.1002/spe.3183
Li FWang MHao D(2022)Bridging the Gap between Different Programming Paradigms in Coverage-based Fault LocalizationProceedings of the 13th Asia-Pacific Symposium on Internetware10.1145/3545258.3545272(75-84)Online publication date: 11-Jun-2022
https://dl.acm.org/doi/10.1145/3545258.3545272
Show More Cited By

Index Terms

Branching processes for QuickCheck generators
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Branching processes for QuickCheck generators
Haskell 2018: Proceedings of the 11th ACM SIGPLAN International Symposium on Haskell

In QuickCheck (or, more generally, random testing), it is challenging to control random data generators' distributions---specially when it comes to user-defined algebraic data types (ADT). In this paper, we adapt results from an area of mathematics ...
Testing Erlang data types with quviq quickcheck
ERLANG '08: Proceedings of the 7th ACM SIGPLAN workshop on ERLANG

When creating software, data types are the basic bricks. Most of the time a programmer will use data types defined in library modules, therefore being tested by many users over many years. But sometimes, the appropriate data type is unavailable in the ...
Splittable pseudorandom number generators using cryptographic hashing
Haskell '13

We propose a new splittable pseudorandom number generator (PRNG) based on a cryptographic hash function. Splittable PRNGs, in contrast to linear PRNGs, allow the creation of two (seemingly) independent generators from a given random number generator. ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGPLAN Notices

ACM SIGPLAN Notices Volume 53, Issue 7

Haskell '18

July 2018

185 pages

ISSN:0362-1340

EISSN:1558-1160

DOI:10.1145/3299711

Editor:
Nicolas Wu
University of Bristol

Issue’s Table of Contents

Haskell 2018: Proceedings of the 11th ACM SIGPLAN International Symposium on Haskell
September 2018
185 pages
ISBN:9781450358354
DOI:10.1145/3242744
Program Chair:
Nicolas Wu

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 September 2018

Published in SIGPLAN Volume 53, Issue 7

Check for updates

Author Tags

Qualifiers

Article

Funding Sources

Swedish Foundation for Strategic Research
Vetenskaprådet

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
215
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zago NBraga AMello BFeitosa S(2023)Usando Esquema GraphQL para Geração de Consultas de Forma AleatóriaProceedings of the XXVII Brazilian Symposium on Programming Languages10.1145/3624309.3624321(17-24)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3624309.3624321
Arts TSvensson HBenac Earle CFredlund L(2023)Testing feature‐rich blockchainsSoftware: Practice and Experience10.1002/spe.318353:5(1144-1173)Online publication date: 16-Jan-2023
https://doi.org/10.1002/spe.3183
Li FWang MHao D(2022)Bridging the Gap between Different Programming Paradigms in Coverage-based Fault LocalizationProceedings of the 13th Asia-Pacific Symposium on Internetware10.1145/3545258.3545272(75-84)Online publication date: 11-Jun-2022
https://dl.acm.org/doi/10.1145/3545258.3545272
Mista AJuristo N(2021)MutagenProceedings of the 43rd International Conference on Software Engineering: Companion Proceedings10.1109/ICSE-Companion52605.2021.00053(120-122)Online publication date: 25-May-2021
https://dl.acm.org/doi/10.1109/ICSE-Companion52605.2021.00053
Kobayashi NLago UGrellois CBouyer P(2019)On the termination problem for probabilistic higher-order recursive programsProceedings of the 34th Annual ACM/IEEE Symposium on Logic in Computer Science10.5555/3470152.3470177(1-14)Online publication date: 24-Jun-2019
https://dl.acm.org/doi/10.5555/3470152.3470177
Goldstein HTao JHatfield-Dodds ZPierce BHead A(2024)Tyche: Making Sense of PBT EffectivenessProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676407(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676407
Mista ARusso A(2023)MUTAGEN: Reliable Coverage-Guided, Property-Based Testing using Exhaustive Mutations2023 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST57152.2023.00025(176-187)Online publication date: Apr-2023
https://doi.org/10.1109/ICST57152.2023.00025
Goldstein HHughes JLampropoulos LPierce B(2021)Do Judge a Test by its CoverProgramming Languages and Systems10.1007/978-3-030-72019-3_10(264-291)Online publication date: 23-Mar-2021
https://doi.org/10.1007/978-3-030-72019-3_10
Reddy SLemieux CPadhye RSen KRothermel GBae D(2020)Quickly generating diverse valid test inputs with reinforcement learningProceedings of the ACM/IEEE 42nd International Conference on Software Engineering10.1145/3377811.3380399(1410-1421)Online publication date: 27-Jun-2020
https://dl.acm.org/doi/10.1145/3377811.3380399
Mista ARusso A(2019)Deriving compositional random generatorsProceedings of the 31st Symposium on Implementation and Application of Functional Languages10.1145/3412932.3412943(1-12)Online publication date: 25-Sep-2019
https://dl.acm.org/doi/10.1145/3412932.3412943
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents