Tyche: In Situ Analysis of Random Testing Effectiveness

Property-based testing (PBT) frameworks, like the Hypothesis [6] library in Python, allow users to specify their program's behavior and check that the program behaves according to that specification. To write a property in Hypothesis, the user simply writes a function that performs some operation on the system under test: if this function raises an exception, the property is judged to have failed; otherwise the property passes. Properties succinctly and declaratively capture aspects of program correctness, with users specifying what (e.g., make sure this function maintains a particular invariant) to test but not exactly how to test it (e.g., check that invariant with , , specific inputs).

Unfortunately, in practice this declarative approach presents a rather leaky abstraction—programmers cannot always rely on the system to figure out how best to test a given property. For example, many properties come with preconditions: if the input data does not fit a certain form, the test will be discarded and the property will pass trivially. In such cases, Hypothesis can struggle to find bugs. Most PBT frameworks have some way to mark a test as “discarded,” and they print a warning if too many tests are thrown away, but these messages are easy to miss. Worse, even a generator that exclusively produces inputs that satisfy the property's precondition might still fail to find bugs because its distribution is not interesting or varied enough. If the user is not paying attention to these subtleties, they may incorrectly conclude that their code is bug-free.

There is a fundamental interface design problem here. While modern testing interfaces succeed well at communicating test failures, they fail to communicate what it means when testing “succeeds”. Developers need interactive tools that allow them to analyze and understand the effectiveness of passing properties, improve test distributions that miss the mark, and gain well-founded confidence that their final test set of properties covers the cases they care about.

Our solution is a tool called Tyche. After a brief discussion of related work (§2), we describe the following contributions:

• We present Tyche, an editor extension that provides in-situ visibility into PBT effectiveness, with affordances for inspecting input distributions, exploring input examples, and evaluating line coverage. (§3)
  
• We demonstrate Tyche on a common PBT case study and show that it can help users improve their testing. It gives feedback about inadequate testing and responds in real time as the user updates their testing code. (§4)
  
• We show that Tyche also works on real-world code bases “out of the box,” applying it in the context of a significant Python project pulled as-is from GitHub. (§5)
  

We conclude with a brief discussion of future work (§6).

Property-based testing was popularized by QuickCheck [3] in Haskell. Since then, similar libraries have appeared in most major programming languages. Hypothesis is one of the most widely used.

The vast majority of PBT frameworks work from the command line. Hypothesis, for example provides , which runs property-based tests and prints results to the terminal, and it integrates with

 [5], so properties can be checked as part of an existing test suite. In both cases, but the latter especially, information can get lost: test output is often hidden or de-emphasized when tests pass.

The adjacent “fuzzing” community has some more interesting approaches to testing feedback. For example, AFL++ [4] includes a sophisticated TUI with feedback on code coverage and other fuzzing statistics over time.

Tyche an extension to the Visual Studio Code (VSCode) editor that provides a new paradigm for interacting with and testing properties. (See the screenshot in Figure 1.) The user invokes Tyche by clicking on a button that the extension places above each property declaration in the user's currently open file. When clicked, the extension runs the property: if the test fails Tyche displays the failing example; but, more interestingly, if it passes Tyche displays a wealth of information about the testing process that the user can use to decide if the property is being tested properly. The current version of Tyche integrates with Hypothesis.

We demonstrate Tyche’s core features on a common PBT example: binary search trees (BSTs). BSTs are a useful example because they are well understood but nontrivial to test. We show how Tyche helps the user to notice and address three common difficulties that might arise when testing an unfamiliar data structure.

The first such issue is the need for preconditions: BSTs require their data to be ordered, but a naïve data generator will not produce very many ordered trees. Tyche alerts the user of this issue with a chart that indicates the proportion of the values that pass the BST precondition, and the user can dig deeper to see that the few values that do pass the precondition are trivial trees.

After fixing this issue by writing a better generator, the user might still not be happy. Generators sometimes require tuning to ensure that they produce a wide and interesting distribution of inputs. Tyche is helpful again; this time, the user can explore the live-updating bar charts to gauge which changes to the generator result in the kind of data distribution they are looking for.

Finally, once the inputs to the property look good, the user can use Tyche to validate that the property gets good code coverage over the system under test. If they find that certain lines are uncovered, they can either continue to tune the generator or add further properties that exercise the un-covered code.

We also demonstrate Tyche in the real world. We chose to use bidict [2], a popular Python library for bidirectional mappings that is used extensively in industry, to demonstrate our tool. The library already uses Hypothesis, but has not been adapted at all to work with Tyche. By adding just two lines of code—one to import the tyche library and one specify a feature extractor—we can immediately jump in and start analyzing the properties as they run.

We are excited to share this demo with the UIST community and get feedback on our interface design. This project is still early stage: while the current version of Tyche already seems useful, we hope a future version can really change the way that users think about and interact with PBT.

We plan to add a number of new affordances to Tyche.

Debugging Test Failures. The current version of Tyche simply prints an error message if the property fails. For the moment this is fine, as we are currently concerned with validating passing tests, but moving forward we hope to include tools for debugging failing tests as well. In particular, we would like to integrate our tool with Hypothesis's “explain mode.” Tyche could include Tarantula-style [7] coverage feedback and present counterexamples to users in a way that lets them easily manipulate and explore them.

Bidirectional Distribution Tuning. Tyche visualizations already aid users when tuning their random data generators by providing live feedback of how changes impact the distribution. But what if the user could simply click and drag a Tyche bar chart to change their generator's distribution? This kind of bidirectional editing may be possible with the help of other ongoing work in the PBT space, and it may make PBT much more accessible to novices who do not have experience writing data generators by hand.

However, beyond affordances, our biggest goal moving forward is to put this interface in front of real users. The current design is informed by our own use of PBT and PBT tools, but developers who use Hypothesis in their day-to-day work will likely have different ideas about which parts of Tyche are most useful and which parts need to change.

We plan to collaborate with the maintainers of Hypothesis to recruit for and carry out an observational user study, evaluating the design of Tyche and looking for opportunities to make it better. The exact parameters of the study is still up in the air, and feedback will be welcome!