←back to thread

"You Don't Need Kafka, Just Use Postgres" Considered Harmful

(www.morling.dev)
39 points ingve | 5 comments | 02 Nov 25 20:37 UTC | HN request time: 0s | source
Show context
cultofmetatron ◴[02 Nov 25 22:15 UTC] No.45793910[source]▶
Id argue that if you are in the position where you legitimately NEED kafka, you hopefully also know what you're doing. You're outside the audience for the "just use postres" crowd. That said, if you're in a startup with a few thousand users, just use postgres is still solid advice.
replies(1): >>45793967 #
threatofrain ◴[02 Nov 25 22:24 UTC] No.45793967[source]▶
If you need some kind of event streaming system there are other choices which have less dev ops burden, such as just using any particular cloud's proprietary or managed offerings. I've seen two companies on NATS so far, I'm trying it out myself for size as well.

There are plenty of choices between PSQL & Kafka. It's not like you take one step north and you're in the "oh no you better know what you're doing" territory.

replies(1): >>45794284 #
strken ◴[02 Nov 25 23:17 UTC] No.45794284[source]▶
The problem with taking one step north and leaving the border of Postgres is what you lose, not the direct ops burden.

Postgres land is a comfy place filled with transactions across all your data at once, one backup solution that you (hopefully) have had running for months or years and has been thoroughly tested, and ACID compliance. You have a single host, probably, which means that you are neither Available nor Partion-tolerant, but at least you are Consistent.

The moment you expand beyond a single database host you now have a distributed system, and woe unto you if you don't understand what that means.

replies(3): >>45794354 #>>45794363 #>>45796327 #
threatofrain ◴[02 Nov 25 23:32 UTC] No.45794363[source]▶
If you wanted such simplicity then nothing is stopping you from running single-node NATS or even just Redis. You always had all the simplicity and consistency you wanted.
replies(1): >>45794498 #
1. strken ◴[02 Nov 25 23:55 UTC] No.45794498[source]▶
The problem is that now you use Postgres for 95% of your system, and also Redis or NATS, which means you lose the ability to atomically commit changes to your database and send a message in one transaction.

You can work around this, but to the best of my knowledge you can't have consistency (between your existing Postgres database and your separate queue or event log) and simplicity.

replies(2): >>45796999 #>>45803218 #
2. hbogert ◴[03 Nov 25 08:29 UTC] No.45796999[source]▶
Indeed, only eventual consistency. The article approaches this subject and mentions the use of the outbox pattern and/or using tools like Debezium.
3. threatofrain ◴[03 Nov 25 19:22 UTC] No.45803218[source]▶
One of the secret traps of Kafka is that it tried to be a collection of nice primitives without that much built on top. So the Fortune 500 story of Kafka is actually one of everyone building on top of Kafka internally with a bigger team. If you weren't prepared to do a little building on top of Kafka you might find the ecosystem a little empty.

But with PSQL you have even less built on top for a use case it wasn't meant for. Now you are in the "you better know what you're doing" territory.

replies(1): >>45805608 #
4. strken ◴[03 Nov 25 23:12 UTC] No.45805608[source]▶
My experience has been that a SQL database is an easier foundation to build on top of than a distributed event streaming log until you run into performance issues that can't be solved by understanding your database better and/or scaling vertically, and that companies which run into performance issues can afford to migrate to other technologies and/or pay for a good ops team, while companies which started using message queues for systems handling 5 requests per second often struggle to make their system sound and get features out the door.
replies(1): >>45807162 #
5. threatofrain ◴[04 Nov 25 03:25 UTC] No.45807162{3}[source]▶
You're still talking about performance. I'm talking about how much you have to build.

When people use Redis/whatever, are you surprised that speed is just a side benefit and ergonomics is what they're looking for? Those are the same ergonomics you'd have to rebuild, as opposed to not build at all because it's already there.

If you want to rebuild Redis in PSQL you are very much in the "you better know what you're doing" territory, and you're also very much in the "are you sure this is your core business value" territory as you rediscover how much nice stuff was packed into the Redis ecosystem. And how am I supposed to uncover the surface or ergonomics of your Redis replacement?