←back to thread

Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system

(www.tomshardware.com)
521 points hd4 | 1 comments | 20 Oct 25 12:31 UTC | HN request time: 0.211s | source
Show context
checker659 ◴[20 Oct 25 16:39 UTC] No.45645894[source]▶
They are working with tiny models. Not sure how well it'd scale to bigger models (if at all).
replies(1): >>45646108 #
1. CaptainOfCoit ◴[20 Oct 25 16:51 UTC] No.45646108[source]▶
They're all LLMs, so no, not tiny, but not exactly huge either:

> Our current deployment runs in a cross-region cluster comprising 213 H20 GPUs, serving twenty-eight 1.8–7B models (TP=1) and nineteen 32–72B models (TP=4).