←back to thread

Docker Model Runner

(www.docker.com)
100 points kordlessagain | 2 comments | 10 Apr 25 14:10 UTC | HN request time: 0.743s | source
Show context
jpgvm ◴[14 Apr 25 03:45 UTC] No.43677869[source]▶
I'm looking at using OCI at $DAY_JOB for model distribution for fleets of machines also so it's good to see it's getting some traction elsewhere.

OCI has some benefits over other systems, namely that tiered caching/pull-through is already pretty battle-tested as is signing etc, beating more naive distribution methods for reliability, performance and trust.

If combined with eStargz or zstd::chunked it's also pretty nice for distributed systems as long as you can slice things up into files in such a way that not every machine needs to pull the full model weights.

Failing that there are P2P distribution mechanisms for OCI (Dragonfly etc) that can lessen the burden without resorting to DIY on Bittorrent or similar.

replies(2): >>43681299 #>>43682381 #
remram ◴[14 Apr 25 13:53 UTC] No.43681299[source]▶
Kubernetes added "image volumes" so this will probably become more and more common: https://kubernetes.io/blog/2024/08/16/kubernetes-1-31-image-...
replies(2): >>43681894 #>>43684892 #
1. mdaniel ◴[14 Apr 25 14:46 UTC] No.43681894[source]▶
Damn, that's handy. I now wonder how much trouble making a CSI driver that does this would be for backporting to the 1.2x clusters (since I don't think that kubernetes does backports for anything)
replies(1): >>43684907 #
2. jpgvm ◴[14 Apr 25 19:06 UTC] No.43684907[source]▶
Not too hard. If you happen to be on CRI-O this has been implemented for a bit but if you are like us and on containerd then you need the new 2.1 beta release. That does most of the heavy lifting, implementing a CSI driver that mounted these as PVs wouldn't be super hard I don't think and you could borrow liberally from the volume source implementation.