←back to thread

Anything can be a message queue if you use it wrongly enough (2023)

(xeiaso.net)
183 points crescit_eundo | 1 comments | 28 Aug 25 15:14 UTC | HN request time: 0s | source
Show context
adamcharnock ◴[28 Aug 25 18:12 UTC] No.45055206[source]▶
I had a developer colleague a while back who was toying with an idea that would require emitting and consuming a _lot_ of messages. I think it was somewhere on the order of 10k-100k/second. He was looking at some pretty expensive solutions IIRC.

I asked if the messages were all under 1.5kb, he said yes. I asked if at-most-one delivery was ok, he said yes. So I proposed he just grab a router and fire messages through it as UDP packets, then use BGP/ECMP to balance the packets between receivers. Add some queues on the router, then just let the receivers pick up the packets as fast as they could. You'd need some kind of feedback to manage back pressure, but ¯\_(ツ)_/¯

A fairly cheap way to achieve 1M+ messages per second.

I never got the chance to flesh-out the idea fully, but the simplicity of it tickled me. Maybe it would have worked, maybe not.

replies(1): >>45055387 #
HeyLaughingBoy ◴[28 Aug 25 18:29 UTC] No.45055387[source]▶
Isn't a fundamental property of a queue that it's FIFO?

UDP message delivery order is not guaranteed. Hell, UDP delivery itself is not guaranteed (although IME, messages don't usually get dropped unless they cross subnets).

replies(3): >>45055437 #>>45055745 #>>45081133 #
1. tomjen3 ◴[31 Aug 25 07:26 UTC] No.45081133[source]▶
Even Azures native queue systems can guarantee that, because it gets hard to scale. You can only have one processor for any given queue, because otherwise one might take so long to process a message that others have finished processing others in the meantime.

But given OPs need for at-most-once, this doesn't really seem to matter in the first place.