Distributed systems programming has stalled

Distributed systems are difficult to reason about.

Computer hardware today is very powerful.

There is a yo-yo process in our industry over the last 50 years between centralization and distribution. We necessarily distribute when we hit the limits of what centralization can accomplish because in general centralization is easier to reason about.

When we hit those junctures, there's a flush of effort into distributed systems. The last major example of this I can think of was the 2000-2010 period, when MapReduce, "NoSQL" databases, Google's massive arrays of supposedly identical commodity grey boxes (not the case anymore), the High Scalability blog, etc. were the flavour of the time.

But then, frankly, mass adoption of SSDs, much more powerful computers, etc. made a lot of those things less necessary. The stuff that most people are doing doesn't require a high level of distributed systems sophistication.

Distributed systems are an interesting intellectual puzzle. But they should be a means to an end not an end in themselves.

I did my MSc in Distributed Systems and it was always funny (to me) to ask a super simple question when someone was presenting distributed system performance metrics that they'd captured to compare how a system scaled across multiple systems: how long does it take your laptop to process the same dataset? No one ever seemed to have that data.

And then the (in)famous COST paper came out and validated the question I'd been asking for years: https://www.usenix.org/system/files/conference/hotos15/hotos...

Wow I love that.

Many people in our profession didn't seem to really notice when the number of IOPS on predominant storage media went from under 200 to well over 100,000 in a matter of just a few years.

I remember evaluating and using clusters of stuff like Cassandra back in the late 00s because it just wasn't possible to push enough data to disk to keep up with traffic on a single machine. It's such an insanely different scenario now.

When you frame the problem that way, unnecessary complexity seems like part of a healthy solution path. /h

Companies get reliability benefits from slack, but creative people abhor wasted slack. Some basic business strategy/wisdom for maintaining/managing creative slack is needed.

10 to 15 years ago, you could argue, however implausibly, that hardware constraints meant vertical scaling was impossible, and you were forced to adopt a distributed architecture. Subsequent improvement in hardware performance, means that in 2025, vertical scaling is perfect acceptable in nearly all areas, relegating distributed architecture to the most niche and marginal applications. The type of applications that the vast majority of businesses will never encounter.