←back to thread

War story: the hardest bug I ever debugged

(www.clientserver.dev)
462 points jakevoytko | 1 comments | 24 Mar 25 14:37 UTC | HN request time: 0.001s | source
Show context
BobbyTables2 ◴[27 Mar 25 03:27 UTC] No.43490119[source]▶
Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone.

Though abs() returning negative numbers is hilarious.. “You had one job…”

To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added.

I’m not just talking about concurrency issues either…

The kind of bug where a reproduction attempt takes a week, not parallelizable due to HW constraints, and logging instrumentation makes it go away or fail differently.

2 days is cute though.

replies(13): >>43490149 #>>43490287 #>>43490459 #>>43490557 #>>43491079 #>>43491823 #>>43492539 #>>43492555 #>>43492647 #>>43493115 #>>43493245 #>>43493811 #>>43497018 #
1. Adverblessly ◴[27 Mar 25 19:14 UTC] No.43497018[source]▶
> To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added.

A favourite of mine was a bug (specifically, a stack corruption) that I only managed to see under instrumentation. After a lot of debugging turns out that the bug was in the instrumentation software itself, which generated invalid assembly under certain conditions (calling one of its own functions with 5 parameters even though it takes only 4). Resolved by upgrading to their latest version.