The AMD Radeon Instinct MI300A's Giant Memory Subsystem

I've often thought that one of the places AMD could distinguish itself from NVIDIA is bringing significantly higher amounts of VRAM (or memory systems that are as performant as what we currently know as VRAM) to the consumer space.

A card with a fraction of the FLOPS of cutting-edge graphics cards (and ideally proportionally less power consumption), but with 64-128GB VRAM-equivalent, would be a gamechanger for letting people experiment with large multi-modal models, and seriously incentivize researchers to build the next generation of tensor abstraction libraries for both CUDA and ROCm/HIP. And for gaming, you could break new grounds on high-resolution textures. AMD would be back in the game.

Of course, if it's not real VRAM, it needs to be at least somewhat close on the latency and bandwidth front, so let's pop on over and see what's happening in this article...

> An Infinity Cache hit has a load-to-use latency of over 140 ns. Even DRAM on the AMD Ryzen 9 7950X3D shows less latency. Missing Infinity Cache of course drives latency up even higher, to a staggering 227 ns. HBM stands for High Bandwidth Memory, not low latency memory, and it shows.

Welp. Guess my wish isn't coming true today.

> AMD would be back in the game.

The market for prosumer cards with high VRAM and low FLOPS would be negligibly small. The data center market is massive on one end and the gaming market is big on the other. Casual consumers who just want a lot of VRAM are such a small minority of people that it doesn’t matter to the bottom line.

It also wouldn’t be financially advantageous to divert RAM chips away from data center production. We don’t have a surplus of chips waiting to be installed, so building out high VRAM but affordable cards would only take away from higher margin products in the datacenter space.

> The market for prosumer cards with high VRAM and low FLOPS would be negligibly small. The data center market is massive on one end and the gaming market is big on the other. Casual consumers who just want a lot of VRAM are such a small minority of people that it doesn’t matter to the bottom line.

I'm sure this is also what AMD is thinking, and it's also why they will never catch up to NVidia in ecosystem and software support.

It's not for the casual consumers, and it's not supposed to make money directly! You want these high VRAM SKUs to attract enthusiast and researchers. I have read a staggering amount of research papers where the authors used some random consumer NVidia GPU. Do you know how many I've read which used AMD GPUs? Big fat ZERO! You want to incentivize these people to use your hardware? You want to get devs to support your platform? Give them a unique value proposition that the competition won't.

I'm currently waiting for the 5090 to be available, and I'm going to buy two of them. If AMD would have released a GPU at a fair price, with reasonable performance and double the VRAM that NVidia offers, do you know what would I do? I would buy two AMD cards instead, port my software to it, and contribute PRs to any upstream software that I use so that it works with these cards. But alas, here we are.