"By deploying this implementation locally, it translates to a cost of $0.20/1M output tokens"
Is that just the cost of electricity, or does it include the cost of the GPUs spread out over their predicted lifetime?
replies(3):
Is that just the cost of electricity, or does it include the cost of the GPUs spread out over their predicted lifetime?
Maybe the cost of renting?
That's silly, but the idea that "local" is not the opposite of remote is even sillier.
Lots of people were advocating for running their k8s on bare metal servers to maximize the performance of their containers
Now wherever that's applied to your conversation... I've no clue, too little context ( ｡ ŏ ﹏ ŏ )
Bare metal in the context of running software is a technical term with a clear meaning that hasn't become contested like "AI" or "Crypto" - and that meaning is that the software is running directly on the hardware.
As k8s isn't virtualization, processes spawned by its orchestrator are still running on bare metal. It's the whole reason why containers are more efficient compared to virtual machines