←back to thread

War story: the hardest bug I ever debugged

(www.clientserver.dev)
462 points jakevoytko | 1 comments | 24 Mar 25 14:37 UTC | HN request time: 0.218s | source
Show context
aetimmes ◴[27 Mar 25 14:24 UTC] No.43493994[source]▶
(disclaimer: I know OP IRL.)

I'm seeing a lot of comments saying "only 2 days? must not have been that bad of a bug". Some thoughts here:

At my current day job, our postmortem template asks "Where did we get lucky?" In this instance, the author definitely got lucky that they were working at Google where 1) there were enough users to generate this Heisenbug consistently and 2) that they had direct access to Chrome devs.

Additionally - the author (and his team) triaged, root caused and remediated a JS compiler bug in 2 days. The sheer amount of complexity involved in trying to narrow down where in the browser code this could all be going wrong is staggering. Consider that the reason it took him "only" two days is because he is very, _very_ good at what he does.

replies(5): >>43494924 #>>43495048 #>>43495849 #>>43496185 #>>43497031 #
1. lesuorac ◴[27 Mar 25 17:30 UTC] No.43495849[source]▶
> In this instance, the author definitely got lucky that they were working at Google where 1) there were enough users to generate this Heisenbug consistently and 2) that they had direct access to Chrome devs.

I'm not sure this is really luck.

The fix is to just not use Math.abs. If they didn't work at Google they still would've done the same debugging and used the same fix. Working at Google probably harmed them as once they discovered Math.abs didn't work correctly they could've just immediately used `> 0` instead of asking the chrome team about it.

There's nothing lucky about slowly adding printf statements until you understand what the computer is actually doing; that's just good work.