←back to thread

Delete tests

(andre.arko.net)
125 points mooreds | 1 comments | | HN request time: 0s | source
Show context
dcminter ◴[] No.45071880[source]
How about you fix the flakey tests?

The tests I'd delete are the ones that just test that the code is written in a particular way instead of testing the expected behaviour of thr code.

replies(3): >>45071959 #>>45072007 #>>45072191 #
Shank ◴[] No.45072191[source]
> How about you fix the flakey tests?

Often times a flakey test is not flakey because it was well-written and something else strange is failing. Often times the test reveals something about the system that is somewhat non-deterministic, but not non-deterministic in a detrimental way. When you have multiple levels of abstraction and parallelization and interdependent behavior, fixing a single test becomes a time consuming process that is difficult to work with (because it's flakey, you can't always replicate the failure).

If a test fails in CI and the traceback is unclear, many people will re-run once and let it continue to flake. Obvious flakes around time and other dependencies are much easier to spot and fix, so they are. It's only the weird ones that lead to pain and regret.

replies(2): >>45073163 #>>45077959 #
1. Izkata ◴[] No.45077959[source]
My favorite flakey test where nothing was actually wrong with the code or test: The system used some of the same settings between development and CI, including the memcached server. The test would fail if one of the devs happened to be using their development site within 15 minutes of the next CI run, because the code would retrieve a nonexistent object from the cache and fail with a really strange error.