Gettiers in software engineering (2019)

I wasn’t aware there was a term for this or that this was not common knowledge - for me I refer to them as “if I fix this, it will break EVERYTHING” cases that come up in my particular line of work frequently, and my peers generally tend to understand as well. Cause/effect in complex symptoms is of course itself complex, which is why the first thing I typically do in any environment is set up metrics and monitoring. If you have no idea what is going on at a granular level, you’ll quickly jump to bad conclusions and waste a lot of time aka $.

This is horrifying, and needs a trigger warning lol. It gave me a sense of panic to read it. It’s always bad when you get so lost in the codebase that it’s just a dark forest of hidden horrors.

When this kind of thing tries to surface, it’s a warning that you need to 10x your understanding of the problem space you are adjacent to.

I guess I’ve worked in a lot of ancient legacy systems that develop over multiple decades - there’s always haunted forests and swaths of arcane or forgotten knowledge. One time I inherited a kubernetes cluster in an account no one knew how to access and when finally hacking into it discovered troves of crypto mining malware shit. It had been serving prod traffic quietly untouched for years. This kind of thing is crazy common, I find disentangling these types of projects to be fun, personally, depending on how much agency I have. But I’m not really a software developer.

I’ve come across (possibly written) code that upon close examination seems to only work accidentally — that there are real failures which are somehow hidden by behavior of other systems.

The classic and oft heard “How did this ever work?”

The surest way to get yourself into a mess like this is to assume that a sufficiently complex codebase can be deeply understood in the first place.

By all means you can gain a lot by making things easier to understand, but only in service of shortcuts while developing or debugging. But this kind of understanding is not the foundation your application can safely stand on. You need detailed visibility into what the system is genuinely doing, and our mushy brains do a poor job of emulating any codebase, no matter how elegant.

I think this stuff is really funny when I find it and I have a whole list of funniest bugs like this I have found. Particularly when I get into dealing with proxies and reponse/error handling between backend systems and frontend clients - sometimes the middle layer has been silently handling errors forever, in a way no one understood, or the client code has adapted to them in a way where fixing it will break things badly - big systems naturally evolve in this way and can take a long time to ever come to a head. When it does, that’s when I try to provide consulting, lol.

In at least a few cases I can think of, the answer was almost definitely "it actually never did work, we just didn't notice how it was broken in this case".

Many years ago I was grading my students’ C programs when I found a program that worked without global variables or passing parameters. Instead, every function had the same variables declared in the same order.

Yeah, if you get over a thousand lines of code you need to be building and documenting it in a way that makes it intelligible in a modular way.

FP can be good for that but I often find that people get so carried away with the pure notion of functional code that they forget to make it obvious in its design. Way, way too much “clever” functional code out there.

The data structures are the key for many things, but a lot of software is all about handling side effects, where basically everything you touch is an input or an output with real world, interrelated global state.

That’s where correctly compartmentalising those state relationships and ample asserts or fail-soft/safe code practices become key. And properly descriptive variable names and naming conventions, with sparse but deep comments where it wasn’t possible to write the code to be self documented by its obvious nature.