Falsify: Hypothesis-Inspired Shrinking for Haskell (2023)

I've found in practice that shrinking to get the "smallest amount of detail" is often unhelpful.

Suppose I have a function which takes four string parameters, and I have a bug which means it crashes if the third is empty.

I'd rather see this in the failure report:

("ldiuhuh!skdfh", "nd#lkgjdflkgdfg", "", "dc9ofugdl ifugidlugfoidufog")

than this:

("", "", "", "")

Really? Your examples seem the opposite. I am left immediately thinking, "hm, is it failing on a '!', some sort of shell issue? Or is it truncating the string on '#', maybe? Or wait, there's a space in the third one, that looks pretty dangerous, as well as noticeably longer so there could be a length issue..." As opposed to the shrunk version where I immediately think, "uh oh: one of them is not handling an empty input correctly." Also, way easier to read, copy-paste, and type.

Their point is that in the unshrunk example the “special” value stands out.

I guess if we were even more clever we could get to something more like (…, …, "", …).

The special value doesn't stand out, though. All three examples I gave were what I thought skimming his comment before my brain caught up to his caveat about an empty third argument. The empty string looked like it was by far the most harmless part... Whereas if they are all empty strings, then by definition the empty string stands out as the most suspicious possible part.

The Hypothesis explain phase [1][2] does this!

  fails_on_empty_third_arg(
      a = "",  # or any other generated value
      b = "",  # or any other generated value
      c = "",  
      d = "",  # or any other generated value
  )

[1] https://hypothesis.readthedocs.io/en/latest/reference/api.ht...

[2] https://github.com/HypothesisWorks/hypothesis/pull/3555

> I'd rather see this in the failure report:

> ("ldiuhuh!skdfh", "nd#lkgjdflkgdfg", "", "dc9ofugdl ifugidlugfoidufog")

I would prefer LazySmallcheck's result, which would be the following:

    (_, _, "", _)

Where `_` indicates that part of the input wasn't evaluated.

> As opposed to the shrunk version where I immediately think, "uh oh: one of them is not handling an empty input correctly."

I agree that non-empty strings are worse, but unfortunately `("", "", "", "")` wouldn't only make me think of empty strings; e.g. I'd wonder whether duplicate/equal values are the problem.

A minimal reproducing example cannot guarantee that you'll correctly diagnose a bug just by looking at the example (because multiple potential bugs could cause the same example to fail) but it can guarantee that when you step through the code to understand what's happening, you won't have to deal with huge amounts of irrelevant data.

Maybe an alternative shrinking procedure could directly minimize the number of instructions that need to be executed to hit a failure...

(Author of falsify here.) You are absolutely correct that the empty string isn't always the best counter-example. The goal of shrinking is to shrink to the _simplest_ possible value (this is true for all approaches to shrinking). What constitutes "simple" is very much domain specific. It would certainly be possible to write a generator that would shrink to, say, "foo", as the canonical "simplest" example of a simple string. Indeed, since we are working in a lazy language, you could (with a bit of effort) shrink to `undefined` if the other arguments are not used at all.

I agree it can be domain-specific, but I think it's more common than not that empty containers, and the number zero, are corner cases rather than typical values.

So I think it would be a decent quality-of-life improvement to make generators of the sort you suggest easily available, and have the tutorial docs use them from the start.