AI PCs Aren't Good at AI: The CPU Beats the NPU

(github.com)

Show context

jsheard ◴[16 Oct 24 20:16 UTC] No.41863390[source]▶

These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. I can't find a die analysis of the Snapdragon X which isolates the NPU specifically but AMDs equivalent with the same ~50 TOPS performance target can be seen here, and takes up about as much area as three high performance CPU cores:

https://www.techpowerup.com/325035/amd-strix-point-silicon-p...

replies(4): >>41863880 #>>41863905 #>>41864412 #>>41865466 #

JohnFen ◴[16 Oct 24 22:19 UTC] No.41864412[source]▶

>>41863390 #

> These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much.

This has been my thinking. Today you have to go out of your way to buy a system with an NPU, so I don't have any. But tomorrow, will they just be included by default? That seems like a waste for those of us who aren't going to be running models. I wonder what other uses they could be put to?

replies(6): >>41864427 #>>41864488 #>>41864879 #>>41865208 #>>41865384 #>>41870713 #

1. heavyset_go ◴[17 Oct 24 00:16 UTC] No.41865208[source]▶

>>41864412 #

The idea is that your OS and apps will integrate ML models, so you will be running models whether you know it or not.

replies(1): >>41866421 #

2. JohnFen ◴[17 Oct 24 04:28 UTC] No.41866421[source]▶

>>41865208 (TP) #

I'm confident that I'll be able to know and control whether or not my Linux and BSD machines will be using ML models.

replies(2): >>41866482 #>>41875239 #

3. hollerith ◴[17 Oct 24 04:45 UTC] No.41866482[source]▶

>>41866421 #

--and whether anyone is using your interactions with your computer to train a model.

replies(1): >>41875252 #

4. heavyset_go ◴[18 Oct 24 00:31 UTC] No.41875239[source]▶

>>41866421 #

I agree with the premise as a Linux user myself, but if you're using any JetBrains products, or Zoom, you're running models on the client-side. I suspect small models will continue to creep into apps. Even Firefox ships ML models in the browser.

5. heavyset_go ◴[18 Oct 24 00:32 UTC] No.41875252{3}[source]▶

>>41866482 #

Luckily, while NPUs do nothing about data exfiltration, they're a poor solution for training models. Your data is still going to get sucked up to the mothership, but offloading training to your machine hopefully won't happen.

replies(1): >>41879416 #

6. hollerith ◴[18 Oct 24 13:43 UTC] No.41879416{4}[source]▶

>>41875252 #

Yes, when I was writing my comment, I was imagining my user-interaction data getting sucked up to data centers.

↑