←back to thread

Docker Model Runner

(www.docker.com)
100 points kordlessagain | 2 comments | 10 Apr 25 14:10 UTC | HN request time: 0.652s | source
Show context
jpgvm ◴[14 Apr 25 03:45 UTC] No.43677869[source]▶
I'm looking at using OCI at $DAY_JOB for model distribution for fleets of machines also so it's good to see it's getting some traction elsewhere.

OCI has some benefits over other systems, namely that tiered caching/pull-through is already pretty battle-tested as is signing etc, beating more naive distribution methods for reliability, performance and trust.

If combined with eStargz or zstd::chunked it's also pretty nice for distributed systems as long as you can slice things up into files in such a way that not every machine needs to pull the full model weights.

Failing that there are P2P distribution mechanisms for OCI (Dragonfly etc) that can lessen the burden without resorting to DIY on Bittorrent or similar.

replies(2): >>43681299 #>>43682381 #
1. wofo ◴[14 Apr 25 15:29 UTC] No.43682381[source]▶
I've been pretty disappointed with eStargz performance, though... Do you have any numbers you can share? All over the internet people refer to numbers from 10 years ago, from workloads that don't seem realistic at all. In my experiments it didn't provide a significant enough speedup.

(I ended up developing an alternative pull mechanism, which is described in https://outerbounds.com/blog/faster-cloud-compute though note that the article is a bit light on the technical details)

replies(1): >>43684956 #
2. jpgvm ◴[14 Apr 25 19:09 UTC] No.43684956[source]▶
In our case some machines would need to access less than 1% of the image size but being able to have an image with the entire model weights as a single artifact is an important feature in and of itself. In our specific scenario even if eStargz would be slow by filesystem standards it's competing with network transfer anyway so if it's the same order of magnitude as rsync that will do.

I don't have any perf numbers I can share but I can say we see ~30% compression with eStargz which is already a small win atleast heh.