"You Don't Need Kafka, Just Use Postgres" Considered Harmful

The problem with taking one step north and leaving the border of Postgres is what you lose, not the direct ops burden.

Postgres land is a comfy place filled with transactions across all your data at once, one backup solution that you (hopefully) have had running for months or years and has been thoroughly tested, and ACID compliance. You have a single host, probably, which means that you are neither Available nor Partion-tolerant, but at least you are Consistent.

The moment you expand beyond a single database host you now have a distributed system, and woe unto you if you don't understand what that means.

well said. I've been working on my startup. We are profitable in part because I spend my time focused on building new features and improving our reliability instead of chasing after all the idiosyncratic bugs that come with distributed systems.

If you wanted such simplicity then nothing is stopping you from running single-node NATS or even just Redis. You always had all the simplicity and consistency you wanted.

The problem is that now you use Postgres for 95% of your system, and also Redis or NATS, which means you lose the ability to atomically commit changes to your database and send a message in one transaction.

You can work around this, but to the best of my knowledge you can't have consistency (between your existing Postgres database and your separate queue or event log) and simplicity.

Current "one host" options are in ridiculous territory - 256 core CPUs with terabytes of RAM an storage in 100 GB/s range. A decade ago that much needed a few racks.

Indeed, only eventual consistency. The article approaches this subject and mentions the use of the outbox pattern and/or using tools like Debezium.

One of the secret traps of Kafka is that it tried to be a collection of nice primitives without that much built on top. So the Fortune 500 story of Kafka is actually one of everyone building on top of Kafka internally with a bigger team. If you weren't prepared to do a little building on top of Kafka you might find the ecosystem a little empty.

But with PSQL you have even less built on top for a use case it wasn't meant for. Now you are in the "you better know what you're doing" territory.

My experience has been that a SQL database is an easier foundation to build on top of than a distributed event streaming log until you run into performance issues that can't be solved by understanding your database better and/or scaling vertically, and that companies which run into performance issues can afford to migrate to other technologies and/or pay for a good ops team, while companies which started using message queues for systems handling 5 requests per second often struggle to make their system sound and get features out the door.

You're still talking about performance. I'm talking about how much you have to build.

When people use Redis/whatever, are you surprised that speed is just a side benefit and ergonomics is what they're looking for? Those are the same ergonomics you'd have to rebuild, as opposed to not build at all because it's already there.

If you want to rebuild Redis in PSQL you are very much in the "you better know what you're doing" territory, and you're also very much in the "are you sure this is your core business value" territory as you rediscover how much nice stuff was packed into the Redis ecosystem. And how am I supposed to uncover the surface or ergonomics of your Redis replacement?