The AMD Radeon Instinct MI300A's Giant Memory Subsystem

I've often thought that one of the places AMD could distinguish itself from NVIDIA is bringing significantly higher amounts of VRAM (or memory systems that are as performant as what we currently know as VRAM) to the consumer space.

A card with a fraction of the FLOPS of cutting-edge graphics cards (and ideally proportionally less power consumption), but with 64-128GB VRAM-equivalent, would be a gamechanger for letting people experiment with large multi-modal models, and seriously incentivize researchers to build the next generation of tensor abstraction libraries for both CUDA and ROCm/HIP. And for gaming, you could break new grounds on high-resolution textures. AMD would be back in the game.

Of course, if it's not real VRAM, it needs to be at least somewhat close on the latency and bandwidth front, so let's pop on over and see what's happening in this article...

> An Infinity Cache hit has a load-to-use latency of over 140 ns. Even DRAM on the AMD Ryzen 9 7950X3D shows less latency. Missing Infinity Cache of course drives latency up even higher, to a staggering 227 ns. HBM stands for High Bandwidth Memory, not low latency memory, and it shows.

Welp. Guess my wish isn't coming true today.

I wholeheartedly agree. Nvidia is intentionally suppressing the amount of memory on their consumer GPUs to prevent data centers from using consumer cards rather than their far more expensive counterparts. The fact that they used to offer the 3060 with 12GB, but have now pushed the pricing higher and limited many cards to 8GB is a testament to the fact they are. I don’t need giga-TOPS with 8-16gb of memory, I’d be perfectly happy with half that speed but with 64gb of memory or more. Even slower memory would be fine. I don’t need 1000t/s, but being able to load a reasonable intelligent model even at 50t/s would be great.