←back to thread

Postgres LISTEN/NOTIFY does not scale

(www.recall.ai)
570 points davidgu | 5 comments | 07 Jul 25 14:05 UTC | HN request time: 0.983s | source
Show context
sorentwo ◴[10 Jul 25 20:55 UTC] No.44525509[source]▶
Postgres LISTEN/NOTIFY was a consistent pain point for Oban (background job processing framework for Elixir) for a while. The payload size limitations and connection pooler issues alone would cause subtle breakage.

It was particularly ironic because Elixir has a fantastic distribution and pubsub story thanks to distributed Erlang. That’s much more commonly used in apps now compared to 5 or so years ago when 40-50% of apps didn’t weren’t clustered. Thanks to the rise of platforms like Fly that made it easier, and the decline of Heroku that made it nearly impossible.

replies(3): >>44525640 #>>44526115 #>>44535609 #
1. cpursley ◴[10 Jul 25 21:09 UTC] No.44525640[source]▶
How did you resolve this? Did you consider listening to the WAL?
replies(2): >>44525760 #>>44525833 #
2. sorentwo ◴[10 Jul 25 21:21 UTC] No.44525760[source]▶
We have Postgres based pubsub, but encourage people to use a distributed Erlang based notifier instead whenever possible. Another important change was removing insert triggers, partially for the exact reasons mentioned in this post.
replies(1): >>44527140 #
3. parthdesai ◴[10 Jul 25 21:30 UTC] No.44525833[source]▶
Distributed Erlang if application is clustered, redis if it is not.

Source: Dev at one of the companies that hit this issue with Oban

4. MuffinFlavored ◴[11 Jul 25 00:16 UTC] No.44527140[source]▶
> Another important change was removing insert triggers, partially for the exact reasons mentioned in this post.

What did you replace them with instead?

replies(1): >>44527560 #
5. sorentwo ◴[11 Jul 25 01:28 UTC] No.44527560{3}[source]▶
In app notifications, which can be disabled. Our triggers were only used to get subsecond job dispatching though.