←back to thread

AI PCs Aren't Good at AI: The CPU Beats the NPU

(github.com)
486 points dbreunig | 2 comments | 16 Oct 24 19:44 UTC | HN request time: 0.001s | source
Show context
jsheard ◴[16 Oct 24 20:16 UTC] No.41863390[source]▶
These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. I can't find a die analysis of the Snapdragon X which isolates the NPU specifically but AMDs equivalent with the same ~50 TOPS performance target can be seen here, and takes up about as much area as three high performance CPU cores:

https://www.techpowerup.com/325035/amd-strix-point-silicon-p...

replies(4): >>41863880 #>>41863905 #>>41864412 #>>41865466 #
JohnFen ◴[16 Oct 24 22:19 UTC] No.41864412[source]▶
> These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much.

This has been my thinking. Today you have to go out of your way to buy a system with an NPU, so I don't have any. But tomorrow, will they just be included by default? That seems like a waste for those of us who aren't going to be running models. I wonder what other uses they could be put to?

replies(6): >>41864427 #>>41864488 #>>41864879 #>>41865208 #>>41865384 #>>41870713 #
crazygringo ◴[16 Oct 24 23:17 UTC] No.41864879[source]▶
Aren't they used for speech recognition -- for dictation? Also for FaceID.

They're useful for more things than just LLM's.

replies(1): >>41866451 #
JohnFen ◴[17 Oct 24 04:38 UTC] No.41866451[source]▶
Yes, but I'm not interested in those sorts of uses. I'm wondering what else an NPU could be used for. I don't know what an NPU actually is at a technical level, so I'm ignorant of the possibilities.
replies(1): >>41867894 #
1. ItsBob ◴[17 Oct 24 09:35 UTC] No.41867894{3}[source]▶
I'm probably about to show my ignorance here (I'm not neck-deep in the AI space but I am a software architect...) but are they not just dedicated matrix multiplication engines (plus some other AI stuff)? So instead of asking the CPU to do the math, you have a dedicated area that does it instead... well, that's my understanding of it.

As to why, I think it's along the lines of this: the CPU does 100 things, one of those is AI acceleration. Let's take the AI acceleration and give it its own space instead so we can keep the power down a bit, add some specialization, and leave the CPU to do other stuff.

Again, I'm coming at this from a high-level as if explaining it to my ageing parents.

replies(1): >>41868347 #
2. JohnFen ◴[17 Oct 24 10:55 UTC] No.41868347[source]▶
Yes, that's my understanding as well. What I meant is that I don't know the fine details. My ignorance is purely because I don't actually have a machine that has an NPU, so I haven't bothered to study up on them.