←back to thread

Evaluating the Infinity Cache in AMD Strix Halo

(chipsandcheese.com)
141 points zdw | 1 comments | 22 Oct 25 04:20 UTC | HN request time: 0.229s | source
Show context
andrewstuart ◴[22 Oct 25 05:17 UTC] No.45665124[source]▶
Despite this APU being deeply interesting to people who want to do local AI, anecdotally I hear that it’s hard to get models to run on it.

Why would AMD not have focused everything it possibly has on demonstrating and documenting and fixing and showing and smoothing the path for AI on their systems?

Why does AMD come across as so generally clueless when it comes to giving developers what they want, compared to Nvidia?

AMD should do whatever it takes to avoid these sort of situations:

https://youtu.be/cF4fx4T3Voc?si=wVmYmWVIya4DQ8Ut

replies(10): >>45665138 #>>45665148 #>>45665186 #>>45665215 #>>45665736 #>>45665755 #>>45665858 #>>45665962 #>>45667229 #>>45671834 #
green7ea ◴[22 Oct 25 07:42 UTC] No.45665962[source]▶
I have a Strix Halo based HP ZBook G1A and it's been pretty easy getting local models to run on it. Training small LLMs on it has been a bit harder but doable as well. Mind you, I 'only' have 64 GB with mine.

Under Linux, getting LM Studio to work using the Vulkan backend was trivial. Llama.cpp was a bit more involved. ROCm worked surprisingly well with Arch — I would credit the package maintainers. The only hard part was sorting out Python packaging for PyTorch (use local packages with system's ROCm).

I wouldn't say it's perfect but it's definitely not as bad as it used to be. I think the biggest downside is the difference in environment when you use this as a dev machine and then run the models on NVIDIA hardware for prod.

replies(1): >>45667687 #
ctas ◴[22 Oct 25 11:51 UTC] No.45667687[source]▶
Can you share a bit more on the small LLMs you've trained? I'm interested in the applicability of current consumer hardware for local training and finetuning.
replies(1): >>45668150 #
1. green7ea ◴[22 Oct 25 12:39 UTC] No.45668150[source]▶
I'm not the AI expert in the company but one of my colleagues creates image segmentation models for our specific use case. I've been able to run the PyTorch training code on my computer without any issues. These are smaller models that are destined to run on Jetson boards so they're limited compared to larger LLMs.

edit: just to be clear, I can't train anything competitive with even the smallest LLMs.