←back to thread

Docker Model Runner

(www.docker.com)
100 points kordlessagain | 1 comments | 10 Apr 25 14:10 UTC | HN request time: 0.22s | source
Show context
Havoc ◴[14 Apr 25 10:48 UTC] No.43679917[source]▶
Can’t say I'm a fan of packaging models as docker images. Feels forced - a solution in search of a problem.

The existing stack - a server and model file - works just fine. There doesn’t seem to be a need to jam an abstraction layer in there. The core problem docker solves just isn’t there

replies(2): >>43680346 #>>43690314 #
gardnr ◴[14 Apr 25 11:59 UTC] No.43680346[source]▶
> GPU acceleration on Apple silicon

There is at least one benefit. I'd be interested to see what their security model is.

replies(1): >>43681655 #
cmiles74 ◴[14 Apr 25 14:21 UTC] No.43681655[source]▶
Is this really a Docker feature, though? llama.cpp provides acceleration on Apple hardware, I guess you could create a Docker image with llama.cpp and an LLLM model and have mostly this feature.
replies(1): >>43690349 #
1. kiview ◴[15 Apr 25 08:31 UTC] No.43690349[source]▶
Unfortunately not, since the container won't have access to the Apple silicon GPU. That's why in our architecture, we have to run llama.cpp as a host process and wire it up with the rest of the Docker Desktop architecture, to make it easily accessible from containers.