←back to thread

1070 points dondraper36 | 3 comments | | HN request time: 0.427s | source
Show context
codingwagie ◴[] No.45069135[source]
I think this works in simple domains. After working in big tech for a while, I am still shocked by the required complexity. Even the simplest business problem may take a year to solve, and constantly break due to the astounding number of edge cases and scale.

Anyone proclaiming simplicity just hasnt worked at scale. Even rewrites that have a decade old code base to be inspired from, often fail due to the sheer amount of things to consider.

A classic, Chesterton's Fence:

"There exists in such a case a certain institution or law; let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, “I don’t see the use of this; let us clear it away.” To which the more intelligent type of reformer will do well to answer: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.”"

replies(44): >>45069141 #>>45069264 #>>45069348 #>>45069467 #>>45069470 #>>45069871 #>>45069911 #>>45069939 #>>45069969 #>>45070101 #>>45070127 #>>45070134 #>>45070480 #>>45070530 #>>45070586 #>>45070809 #>>45070968 #>>45070992 #>>45071431 #>>45071743 #>>45071971 #>>45072367 #>>45072414 #>>45072570 #>>45072634 #>>45072779 #>>45072875 #>>45072899 #>>45073114 #>>45073174 #>>45073183 #>>45073201 #>>45073291 #>>45073317 #>>45073516 #>>45073758 #>>45073768 #>>45073810 #>>45073812 #>>45073942 #>>45073964 #>>45074264 #>>45074642 #>>45080346 #
1. patmcc ◴[] No.45069911[source]
The problem with this is no one can agree about what "at scale" means.

Like yes, everyone knows that if you want to index the whole internet and have tens of thousands of searches a second there are unique challenges and you need some crazy complexity. But if you have a system that has 10 transactions a second...you probably don't. The simple thing will probably work just fine. And the vast majority of systems will never get that busy.

Computers are fast now! One powerful server (with a second powerful server, just in case) can do a lot.

replies(2): >>45070278 #>>45070592 #
2. rednafi ◴[] No.45070278[source]
Yep, vertical scaling goes a long way. But it’s not compute where the bottleneck for scale lies, rather in the resiliency & availability.

So although a single server goes a long way, to hit that sweet 99.999 SLA, people horizontally scales way before hitting the maximum compute capacity of a singe machine. HA makes everything way more difficult to operate and reason about.

3. hansvm ◴[] No.45070592[source]
Yeah, we do 100k ML inferences per second. It's not a single server, but the architecture isn't much more complicated than that.

With today's computers, indexing the entire internet and serving 100k QPS also isn't really that demanding architecturally. The vast majority of current implementation complexity exists for reasons other than necessity.