←back to thread

1226 points bishopsmother | 2 comments | | HN request time: 0.001s | source
Show context
throwawaaarrgh ◴[] No.35051550[source]
I've been doing reliability stuff for near two decades. The one thing I am sure of is there is no way to just engineer your way to reliability. That is to say, no person, no matter how smart, can just invent some whizbang engineering thing and suddenly you have reliability.

Reliability is a thing that grows, like a plant. You start out with a new system or piece of software. It's fragile, small, weak. It is threatened by competing things and literal bugs and weather and the soil it's grown in and more. It needs constant care. Over time it grows stronger, and can eventually fend for itself pretty well. Sometimes you get lucky and it just grows fine by itself. And sometimes 50 different things conspire to kill it. But you have to be there monitoring it, finding the problems, learning how to prevent them. Every garden is a little different.

It doesn't matter what a company like Fly does technology wise. It takes time and care and churning. Eventually they will be reliable. But the initial process takes a while. And every new piece of tech they throw in is another plant in the garden.

So the good news is, they can become really reliable. But the bad news is, it doesn't come fast, and the more new plants they put in the ground, the more concerns there are to address before the garden is self sustaining.

replies(7): >>35051647 #>>35052736 #>>35052993 #>>35053029 #>>35053323 #>>35056046 #>>35056972 #
IshKebab ◴[] No.35052736[source]
Maybe so but there are definitely technology choices that have vastly different "initial reliability". What would you expect to be more reliable - a bash script or a Rust program?

I'm not that familiar with setting up global network infrastructure but I imagine there are similar choices that can vastly affect initial reliability.

replies(1): >>35052975 #
capableweb ◴[] No.35052975[source]
I'd say it depends more on the person rather than the technology.

A master in bash will build more reliable API (in bash no less!) than a beginner in Rust, simply because of experience and knowing their way around the tools they're using. Newer/different technologies won't simply solve a problem unless the person has some sort of domain knowledge of said problem.

replies(1): >>35054704 #
1. throwawaaarrgh ◴[] No.35054704[source]
I agree. Like, you could build two houses: one out of sticks, the other brick. Depending on who's building it, if they've not built a house out of brick before, it's very easy to make a mistake. Versus someone who's been building stick houses forever will get it right the first time.

Also, brand new software in general is like a new hybrid plant. How does it behave in this environment compared to other plants? Does it attract more bugs? Does it need different care? We don't know yet; it's new.

And even for an old well known plant, if the gardener hasn't gotten to know it yet, it's easy to make a mistake with its care. But a well known plant with a gardener who's grown it before is the most likely to work without issue.

replies(1): >>35061281 #
2. IshKebab ◴[] No.35061281[source]
> Depending on who's building it, if they've not built a house out of brick before, it's very easy to make a mistake. Versus someone who's been building stick houses forever will get it right the first time.

Decent analogy. And of course if you have people of vaguely similar skill levels then the brick house is going to be way more robust. Which was my point.