←back to thread

Ask HN: What's your go-to message queue in 2025?

66 points enether | 3 comments | 15 May 25 11:43 UTC | HN request time: 0.872s | source

The space is confusing to say the least.

Message queues are usually a core part of any distributed architecture, and the options are endless: Kafka, RabbitMQ, NATS, Redis Streams, SQS, ZeroMQ... and then there's the “just use Postgres” camp for simpler use cases.

I’m trying to make sense of the tradeoffs between:

- async fire-and-forget pub/sub vs. sync RPC-like point to point communication

- simple FIFO vs. priority queues and delay queues

- intelligent brokers (e.g. RabbitMQ, NATS with filters) vs. minimal brokers (e.g. Kafka’s client-driven model)

There's also a fair amount of ideology/emotional attachment - some folks root for underdogs written in their favorite programming language, others reflexively dismiss anything that's not "enterprise-grade". And of course, vendors are always in the mix trying to steer the conversation toward their own solution.

If you’ve built a production system in the last few years:

1. What queue did you choose?

2. What didn't work out?

3. Where did you regret adding complexity?

4. And if you stuck with a DB-based queue — did it scale?

I’d love to hear war stories, regrets, and opinions.

1. matt_s ◴[18 May 25 14:22 UTC] No.44021616[source]▶
Google PubSub is what we use as our message queue, mostly for communicating change data capture via messages to other internal systems. Its typically being consumed by some job system polling on an interval and then doing CRUD to sync changes.

Its not very complex and feels like we're running a lot of compute resources to just sync data between systems. Admittedly there isn't good separation of concerns so there is overlap that requires data syncs.

I've been looking at things like kafka, etc. thinking there might be some magic there that makes us use less compute or makes data syncs a little easier to deal with but wonder what scale of data throughput is a tipping point where a service like that is really needed. If it turns out its just a different service but same timeliness of data sync and similar compute resources I struggle with what benefits might be provided.

I'd love for almost like a levels.fyi style site where people could anonymously report things like this for the tech stacks being used, throughput of data, amount of compute in play, and ratings/comments on their overall solution ("would do again", "don't recommend", "overkill", "resume filler"). It feels much like other areas of technology where a use case comes out of a huge company and RDD (resume driven development) takes hold and now there are people out there doing the equivalent of souping up a 1997 honda accord like its a racecar but its only driving grandma to her appointments.

replies(1): >>44022698 #
2. mdaniel ◴[18 May 25 16:58 UTC] No.44022698[source]▶
I love hearing "watch out for" stories, because I feel that allows me to be extra vigilant about some aspect when running my own PoC

That said, my suspicion about any such aggregation project like that is that context is everything and trying to capture "this sucks" for all the input criteria which produced that outcome is going to be a wall of text that few will write and even fewer will read (ahem, LLM "tl;dr it for me" aside)

replies(1): >>44022735 #
3. mdaniel ◴[18 May 25 17:03 UTC] No.44022735[source]▶
Replying to myself so it can be downvoted separately, but in this mythical new AI gonna take our jobs world, who exactly is supposed to carry out the spikes required to know if technology X is a good fit for problem Y in company Z's culture? Vibe Kakfa into place and yolo?