←back to thread

Falsify: Hypothesis-Inspired Shrinking for Haskell (2023)

(www.well-typed.com)
90 points birdculture | 1 comments | 20 Apr 25 19:41 UTC | HN request time: 0.224s | source
Show context
mjw1007 ◴[20 Apr 25 22:51 UTC] No.43747119[source]▶
I've found in practice that shrinking to get the "smallest amount of detail" is often unhelpful.

Suppose I have a function which takes four string parameters, and I have a bug which means it crashes if the third is empty.

I'd rather see this in the failure report:

("ldiuhuh!skdfh", "nd#lkgjdflkgdfg", "", "dc9ofugdl ifugidlugfoidufog")

than this:

("", "", "", "")

replies(4): >>43747380 #>>43748889 #>>43749509 #>>43749569 #
gwern ◴[20 Apr 25 23:43 UTC] No.43747380[source]▶
Really? Your examples seem the opposite. I am left immediately thinking, "hm, is it failing on a '!', some sort of shell issue? Or is it truncating the string on '#', maybe? Or wait, there's a space in the third one, that looks pretty dangerous, as well as noticeably longer so there could be a length issue..." As opposed to the shrunk version where I immediately think, "uh oh: one of them is not handling an empty input correctly." Also, way easier to read, copy-paste, and type.
replies(3): >>43747816 #>>43749072 #>>43749633 #
dullcrisp ◴[21 Apr 25 01:27 UTC] No.43747816[source]▶
Their point is that in the unshrunk example the “special” value stands out.

I guess if we were even more clever we could get to something more like (…, …, "", …).

replies(2): >>43747837 #>>43748112 #
1. tybug ◴[21 Apr 25 02:44 UTC] No.43748112[source]▶
The Hypothesis explain phase [1][2] does this!

  fails_on_empty_third_arg(
      a = "",  # or any other generated value
      b = "",  # or any other generated value
      c = "",  
      d = "",  # or any other generated value
  )
[1] https://hypothesis.readthedocs.io/en/latest/reference/api.ht...

[2] https://github.com/HypothesisWorks/hypothesis/pull/3555