Running a 180B parameter LLM on a single Apple M2 Ultra

Even a linear growth rate of average RAM capacity would obviate the need to run current SOTA LLMs remotely in short order.

Historically average RAM has grown far faster than linear, and there really hasn't been anything pressing manufacturers to push the envelope here in the past few years... until now.

It could be that LLM model sizes keep increasing such that we continue to require cloud consumption, but I suspect the sizes will not increase as quickly as hardware for inference.

Given how useful GPT-4 is already. Maybe one more iteration would unlock the vast majority of practical use cases.

I think people will be surprised that consumers ultimately end up benefitting far more from LLMs than the providers. There's not going to be much moat or differentiation to defend margins... more of a race to the bottom on pricing

I can't wait for my phone to have something like 512Gb-1TB of RAM to run some really interesting models locally :D

You can buy 768GB of DDR3 and an Ivy Bridge Xeon E5 to put it in for a total of around $500, most of which is the memory. (The CPUs wouldn't be fast for a model that size though.)

I'd be impressed if you fit that into a phone.