Docker Model Runner

(www.docker.com)

100 points kordlessagain | 1 comments | 10 Apr 25 14:10 UTC | HN request time: 0s | source

Show context

jpgvm ◴[14 Apr 25 03:45 UTC] No.43677869[source]▶

I'm looking at using OCI at $DAY_JOB for model distribution for fleets of machines also so it's good to see it's getting some traction elsewhere.

OCI has some benefits over other systems, namely that tiered caching/pull-through is already pretty battle-tested as is signing etc, beating more naive distribution methods for reliability, performance and trust.

If combined with eStargz or zstd::chunked it's also pretty nice for distributed systems as long as you can slice things up into files in such a way that not every machine needs to pull the full model weights.

Failing that there are P2P distribution mechanisms for OCI (Dragonfly etc) that can lessen the burden without resorting to DIY on Bittorrent or similar.

replies(2): >>43681299 #>>43682381 #

remram ◴[14 Apr 25 13:53 UTC] No.43681299[source]▶

>>43677869 #

Kubernetes added "image volumes" so this will probably become more and more common: https://kubernetes.io/blog/2024/08/16/kubernetes-1-31-image-...

replies(2): >>43681894 #>>43684892 #

1. jpgvm ◴[14 Apr 25 19:04 UTC] No.43684892[source]▶

>>43681299 #

That is exactly the feature we are using, right now you need to be on a beta release of containerd but before long it should be pretty widespread. In combination with lazy pull (eStargz) it's a pretty compelling implementation.

↑