The AMD Radeon Instinct MI300A's Giant Memory Subsystem

I've often thought that one of the places AMD could distinguish itself from NVIDIA is bringing significantly higher amounts of VRAM (or memory systems that are as performant as what we currently know as VRAM) to the consumer space.

A card with a fraction of the FLOPS of cutting-edge graphics cards (and ideally proportionally less power consumption), but with 64-128GB VRAM-equivalent, would be a gamechanger for letting people experiment with large multi-modal models, and seriously incentivize researchers to build the next generation of tensor abstraction libraries for both CUDA and ROCm/HIP. And for gaming, you could break new grounds on high-resolution textures. AMD would be back in the game.

Of course, if it's not real VRAM, it needs to be at least somewhat close on the latency and bandwidth front, so let's pop on over and see what's happening in this article...

> An Infinity Cache hit has a load-to-use latency of over 140 ns. Even DRAM on the AMD Ryzen 9 7950X3D shows less latency. Missing Infinity Cache of course drives latency up even higher, to a staggering 227 ns. HBM stands for High Bandwidth Memory, not low latency memory, and it shows.

Welp. Guess my wish isn't coming true today.

> AMD would be back in the game.

The market for prosumer cards with high VRAM and low FLOPS would be negligibly small. The data center market is massive on one end and the gaming market is big on the other. Casual consumers who just want a lot of VRAM are such a small minority of people that it doesn’t matter to the bottom line.

It also wouldn’t be financially advantageous to divert RAM chips away from data center production. We don’t have a surplus of chips waiting to be installed, so building out high VRAM but affordable cards would only take away from higher margin products in the datacenter space.