Postgres. Doing ~ 70k messages/second average. Nothing huge but don’t need anything dedicated yet.
replies(3):
Message queues are usually a core part of any distributed architecture, and the options are endless: Kafka, RabbitMQ, NATS, Redis Streams, SQS, ZeroMQ... and then there's the “just use Postgres” camp for simpler use cases.
I’m trying to make sense of the tradeoffs between:
- async fire-and-forget pub/sub vs. sync RPC-like point to point communication
- simple FIFO vs. priority queues and delay queues
- intelligent brokers (e.g. RabbitMQ, NATS with filters) vs. minimal brokers (e.g. Kafka’s client-driven model)
There's also a fair amount of ideology/emotional attachment - some folks root for underdogs written in their favorite programming language, others reflexively dismiss anything that's not "enterprise-grade". And of course, vendors are always in the mix trying to steer the conversation toward their own solution.
If you’ve built a production system in the last few years:
1. What queue did you choose?
2. What didn't work out?
3. Where did you regret adding complexity?
4. And if you stuck with a DB-based queue — did it scale?
I’d love to hear war stories, regrets, and opinions.
So when selecting a message that isn’t started. It also looks for in progress ones that have been going longer than the timeout.
The update sets status, start time, and attempt counter.
If attempt counter equals 3 when the update happens, it sets the message to failed. The return looks at the stats sees failed and raises a notification.
Then if it’s a fix like correcting data or something I just reset the state to have it reprocess.
Never needed to track workers or cleanup jobs etc.
That row indicates you are the one processing the data and no one else should. When reading, abort the read if someone else wrote that row first.
When you are finished processing, hold the lock and update the row you added before to indicate processing is complete.
The timestamp can be used to timeout the request.