Running a 180B parameter LLM on a single Apple M2 Ultra

Historically average RAM has grown far faster than linear, and there really hasn't been anything pressing manufacturers to push the envelope here in the past few years... until now.

It could be that LLM model sizes keep increasing such that we continue to require cloud consumption, but I suspect the sizes will not increase as quickly as hardware for inference.

Given how useful GPT-4 is already. Maybe one more iteration would unlock the vast majority of practical use cases.

I think people will be surprised that consumers ultimately end up benefitting far more from LLMs than the providers. There's not going to be much moat or differentiation to defend margins... more of a race to the bottom on pricing

Unless I'm misunderstanding, doesn't OpenAI have a very vested interest to keep making their products so good/so complex/so large that consumer hobbyists can't just `git clone` an alternative that's 95% as good running locally?

Here’s another way to think about it. Why does ISA matter in CPUs? There are minor issues around efficiencies of various kinds, but the real advantage of any mainstream ISA is, in part, the availability of tooling (hence this was a correct and heavy early focus for the RISCV effort) but also a lot of ecosystem things you don’t see: for example, Intel and Arm have truly mammoth test and verification suites that represent thousands++ of man years of investment.

OpenAI almost certainly has a massive invisible accumulated value at this point.

The actual models themselves are the output in the same way that a packaged CPU is the output. How you got there matters almost as much or more.

Honestly the answer is that it mostly doesn't.

An ISA isn't viable without tooling, but that's why it's the first thing they all get. The only ISA with any significant moat is x86, and that's because there is so much legacy closed source software for it that people still need but would have to be emulated on any other architecture. And even that only works as long as x86 processors are competitive; if they fell behind then customers would just eat the emulation overhead on something else.

Other ISAs don't even have that. Would anybody actually be surprised if RISC-V took a huge chunk of out ARM's market share in the not too distant future?