How does this compare to https://github.com/Query-farm/tributary ?
replies(2):
We leverage DuckDB as the stream processing engine, which gives SQLFlow the ability to process 10's of thousands of messages a second using ~250MiB of memory!
DuckDB also supports a rich ecosystem of sinks and connectors!
https://sql-flow.com/docs/category/tutorials/
https://github.com/turbolytics/sql-flow
We were tired of running JVM's for simple stream processing, and also of bespoke one off stream processors
I would love your feedback, criticisms and/or experiences!
Thank you
Based on the tributary documentation, I understand that tributary embeds kafka consumers into duckdb. This makes duckdb the main process that you run to perform consumption. I think that this makes creating stream processing POCs very accessible. It looks like it is quite easy to start streaming data into duckdb. What I don't see is a full story around Devops, operations, testing, configuration as code etc.
SQLFlow is a service that embeds DuckDB as the storage and processing brains. Because of this, we're able to offer metrics, testing utilities, pipelines as code, and all the other DevOps utilities that are necessary to run a huge number of streaming instances 24x7. SQLFlow was created as a tool that I wish I had to for simple stream processing in production in high availability contexts :)