Transactional Object Storage?

Pretty cool! Do you have any ideas already about how to make it work with S3, considering it doesn't support If- headers?

S3 recently added basic matching support (https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3..., https://docs.aws.amazon.com/AmazonS3/latest/userguide/condit...).

They don't have the full suite of GCS's capabilities (https://cloud.google.com/storage/docs/request-preconditions#...) but it's something.

I think it's now much easier to achieve than a year ago. The critical one is conditional writes on new objects, because otherwise you can't safely create transaction logs in the presence of timeouts. This is not enough though.

My approach on S3 would be to ensure to modify the ETag of an object whenever other transactions looking at it must be blocked. This makes it easier to use conditional reads (https://docs.aws.amazon.com/AmazonS3/latest/userguide/condit...) on COPY or GET operations.

For write, I would use PUT on a temporary staging area and then conditional COPY + DELETE afterward. This is certainly slower than GCS, but I think it should work.

Locking without modifying the object is the part that needs some optimization though.

And I see more possibilities now that https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3... is available. It will get easier and easier to build serverless data lakes, streaming, queues.

Not a full solution, but seeing the OP seeks to be a key-value store (versus full RDBMS? despite the comparisons with Spanner and Postgres?), important to weigh how Rockset (also mainly KV store) dealt with S3-backed caching at scale:

  * https://rockset.com/blog/separate-compute-storage-rocksdb/

  * https://github.com/rockset/rocksdb-cloud

Keep in mind Rockset is definitely a bit biased towards vector search use cases.

Nice, thanks for the reference!

BTW, the comparison was only to give an idea about isolation levels, it wasn't meant to be a feature-to-feature comparison.

Perhaps I didn't make it prominent enough, but at some point I say that many SQL databases have key-value stores at their core, and implement a SQL layer on top (e.g. https://www.cockroachlabs.com/docs/v22.1/architecture/overvi...).

Basically SQL can be a feature added later to a solid KV store as a base.

You can do it without using an append only logs https://github.com/endpointservices/mps3

However it will be much simpler with the new conditional writes