War story: the hardest bug I ever debugged

I wish I could recall the details better but this was 20+ years ago now. In college I had an internship working at Bose, doing QA on firmware in a new multi CD changer addon to their flagship stereo. We were provided discs of music tracks with various characteristics. And had to listen to them over and over and over and over and over and over, running through test cases provided by QA management as we did. But also doing random ad-hoc testing once we finished the required tests on a given build.

At one point I found a bug where if you hit a sequence of buttons on the remote at a very specific time--I want to say it was "next track" twice right as a new track started--the whole device would crash and reboot. This was a show stopper; people would hit the roof if their $500 stereo crashed from hitting "next". Similar to the article, the engineering lead on the product cleared his schedule to reproduce, find, and fix the issue. He did explain what was going on at the time, but the specifics are lost to me.

Overall the work was incredibly boring. I heard the same few tracks so many times I literally started to hear them in my dreams. So it was cool to find a novel, highest severity bug by coloring outside the lines of the testcases. I felt great for finding the problem! I think the lead lost 20% of his hair in the course of fixing it, lol.

I haven't had QA as a job title in a long time but that job did teach me some important lessons about how to test outside the happy path, and how to write a reproducible and helpful bug report for the dev team. Shoutout to all the extremely underpaid and unappreciated QA folks out there. It sucks that the discipline doesn't get more respect.

That is great QAing. It also speaks to why QA should be a real role in more orgs, rather than a shrinking discipline. Engineers LOVE LOVE LOVE to test the happy path.

It's not even malice/laziness, it's their entire interpretation of the problem/requirements drives their implementation which then drives their testing. It's like asking restaurants to self-certify they are up to food safety codes.

If you do not follow the happy path something will break 100% of the time. That's why engineers always follow the happy path. Some engineers even think that anything outside the happy path is an exception and not even worth investigating. These engineers only thrives if the users are unable to switch to another product. Only competition will lead to better products.

My favorite happy path developer.. and he was by far 10x worse than any engineer I worked with at this, did the following:

Spec: allow the internal BI tool to send scheduled reports to the user

Implementation: the server required the desktop front end of said user to have been opened that day for the scheduled reports to work, even though the server side was sending the mails

Why this was hilariously bad - the only reason to have this feature is for when the user is out of office / away from desk for an extended period, precisely when they may not have opened their desktop UI for the day.

One of my favorite examples of how an engineer can get the entire premise of the problem wrong.

In the end he had taken so long and was so intransigent that desktop support team found it easier to schedule the desktop UIs to auto-open in windows scheduler every day such that the whole Rube Goldberg scheduled reports would work.

You found a 1× engineer; the worst engineer that can keep the job.

Over time we actually found him to be more of a -2x engineer, but thats another story

edit: reminded me of the old joke

A programmer gets sent to the store by his wife. His wife says, “Get a gallon of milk, and if they have eggs, get a dozen.”

The programmer returns home with 12 gallons of milk and says, “They had eggs.”