←back to thread

War story: the hardest bug I ever debugged

(www.clientserver.dev)
462 points jakevoytko | 1 comments | 24 Mar 25 14:37 UTC | HN request time: 0.357s | source
Show context
BobbyTables2 ◴[27 Mar 25 03:27 UTC] No.43490119[source]▶
Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone.

Though abs() returning negative numbers is hilarious.. “You had one job…”

To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added.

I’m not just talking about concurrency issues either…

The kind of bug where a reproduction attempt takes a week, not parallelizable due to HW constraints, and logging instrumentation makes it go away or fail differently.

2 days is cute though.

replies(13): >>43490149 #>>43490287 #>>43490459 #>>43490557 #>>43491079 #>>43491823 #>>43492539 #>>43492555 #>>43492647 #>>43493115 #>>43493245 #>>43493811 #>>43497018 #
1. sesm ◴[27 Mar 25 13:10 UTC] No.43493245[source]▶
For stuff like this we used in-memory ring buffer logger that printed the logs on request. And it didn't save the strings, just necessary data bits and a pointer to formatting function. Writing to this logger didn't affect any timings.