←back to thread

287 points shadaj | 1 comments | | HN request time: 0.473s | source
1. Karrot_Kream ◴[] No.43197379[source]
When I was graduating from my Masters (failed PhD :) this overview of various programming models is generally how I thought of things.

I've been writing distributed code now in industry for a long time and in practice, having worked at a some pretty high-scale tech companies over the years, most shops tend to favor static-location style models. As the post states, it's due largely to control and performance. Scaling external-distribution systems has been difficult everywhere I've seen it tried and usually ends up creating a few knowledgeable owners of a system with high bus-factor. Scaling tends to work fine until it doesn't and these discontinuous, sharp edges are very very painful as they're hard to predict and allocate resourcing for.

Are external-distribution systems dead ends then? Even if they can achieve high theoretical performance, operation of these systems tends to be very difficult. Another problem I find with external-distribution systems is that there's a lot of hidden complexity in just connecting, reading, and writing to them. So you want to talk to a distributed relational DB, okay, but are you using a threaded concurrency model or an async concurrency model? You probably want a connection pool so that TCP HOL blocking doesn't tank your throughput. But if you're using threads, how do you map your threads to the connections in the pool? The pool itself represents a bottleneck as well. How do you monitor the status of this pool? Tools like Istio strive to standardize this a little bit but fundamentally we're working with 3 domains here just to write to the external-distribution system itself: the runtime/language's concurrency model, the underlying RPC stack, and the ingress point for the external-distribution system.

Does anyone have strong stories of scaling an external-distribution system that worked well? I'd be very curious. I agree that progress here has stalled significantly. But I find myself designing big distributed architecture after big distributed architecture continuing to use my deep experience of architecting these systems to build static-location systems because if I'm already dealing with scaling pains and cross-domain concerns, I may as well rip off the band-aid and be explicit about crossing execution domains.