S3 as a Git remote and LFS server

I'm surprised they just punt on concurrent updates [0] instead of locking with something like dynamodb, like terraform does.

[0] https://github.com/awslabs/git-remote-s3?tab=readme-ov-file#...

I thank goodness I have access to a non-stupid Terraform state provider[1] so I've never tried that S3+dynamodb setup but, if I understand the situation correctly, introducing Yet Another AWS Service ™ into this mix would mandate that callers also be given a `dynamo:WriteSomething` IAM perm, which is actually different from S3 in that in S3 one can -- at their discretion -- set the policies on the bucket such that it would work without any explicit caller IAM

1: https://docs.gitlab.com/ee/user/infrastructure/iac/terraform...

Google Cloud Storage is good enough to implement locks all by itself: https://reddit.com/r/golang/comments/t52d4f/gmutex_a_global_...

Doesn't S3 provide primitives to do the same? At least since moving to strong read-after-write consistency?

PS: I wrote the above package. Happy to answer questions about it.

S3 recently got conditional writes and you can use do locking entirely in S3 - I don't think they are using this though. Must be too recent an addition.

Conditional write is just added to S3 2 month ago: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3...

Unfortunately this functionality is much more limited in S3 as you can only use `If-None-Match: *` to prevent overwrites. https://docs.aws.amazon.com/AmazonS3/latest/userguide/condit...

GCS also allows for conditional overwrites using `If-Match: <etag>` which means you can do optimistic concurrency control. https://cloud.google.com/storage/docs/request-preconditions

Yeah, it might still be possible to implement a mutex based on just the existence of an object, but it'll be harder to add expiration/liveness which I find essential.