"By deploying this implementation locally, it translates to a cost of $0.20/1M output tokens"
Is that just the cost of electricity, or does it include the cost of the GPUs spread out over their predicted lifetime?
replies(3):
Maybe the cost of renting?
That's silly, but the idea that "local" is not the opposite of remote is even sillier.
Lots of people were advocating for running their k8s on bare metal servers to maximize the performance of their containers
Now wherever that's applied to your conversation... I've no clue, too little context ( ｡ ŏ ﹏ ŏ )
I a Java app running on Linux bare metal?
Bare metal in the context of running software is a technical term with a clear meaning that hasn't become contested like "AI" or "Crypto" - and that meaning is that the software is running directly on the hardware.
As k8s isn't virtualization, processes spawned by its orchestrator are still running on bare metal. It's the whole reason why containers are more efficient compared to virtual machines
Of course, a process running inside Kubernetes Pod, on a baremetal node will show up in `top` if I run it on the node directly. In such terms, it is running directly on hardware.
But when I deploy this Pod, I'm not interacting with the OS in any way. I'm interacting with Kubernetes apiserver, telling it what to run, not really caring about the operating system underneath. In such terms, the application is running "in k8s".
It also does not always "clearly" have this new meaning. Somebody who is used to running programs directly (with no intermediate OS) on hardware might not understand what you're saying, or might ask you to clarify, and you probably shouldn't feel put upon by a totally understandable misinterpretation.
edit: Especially when you keep repeating "directly on hardware" when you mean "not on a VM." VMs also run on hardware. You're saying that you're only running on one OS instead an OS in your OS.