←back to thread

AI PCs Aren't Good at AI: The CPU Beats the NPU

(github.com)
486 points dbreunig | 2 comments | 16 Oct 24 19:44 UTC | HN request time: 0.421s | source
Show context
dmitrygr ◴[16 Oct 24 20:11 UTC] No.41863335[source]▶
In general MAC unit utilization tends to be low for transformers, but 1.3% seems pretty bad. I wonder if they fucked up the memory interface for the NPU. All the MACs in the world are useless if you cannot feed them.
replies(2): >>41863438 #>>41863595 #
Hizonner ◴[16 Oct 24 20:37 UTC] No.41863595[source]▶
It's a tablet. It probably has like one DDR channel. It's not so much that they "fucked it up" as that they knowingly built a grossly unbalanced system so they could report a pointless number.
replies(1): >>41863634 #
dmitrygr ◴[16 Oct 24 20:40 UTC] No.41863634[source]▶
Well, no. If the CPU can hit better numbers on the same model then the bandwidth from the DDR IS there. Probably the NPU does not attach to the proper cache level, or just has a very thin pipe to it
replies(1): >>41863711 #
1. Hizonner ◴[16 Oct 24 20:47 UTC] No.41863711[source]▶
The CPU is only about twice as good as the NPU, though (four times as good on one test). The NPU is being advertised as capable of 45 trillion operations per second, and he's getting 1.3 percent of that.

So, OK, yeah, I concede that the NPU may have even worse access to memory than the CPU, but the bottom line is that neither one of them has anything close to what it needs to to actually delivering anything like the marketing headline performance number on any realistic workload.

I bet a lot of people have bought those things after seeing "45 TOPS", thinking that they'd be able to usefully run transformers the size of main memory, and that's not happening on CPU or NPU.

replies(1): >>41863731 #
2. dmitrygr ◴[16 Oct 24 20:50 UTC] No.41863731[source]▶
Yup, sad all round. We are in agreement.