Most active commenters
  • defrost(4)

←back to thread

AI PCs Aren't Good at AI: The CPU Beats the NPU

(github.com)
486 points dbreunig | 11 comments | 16 Oct 24 19:44 UTC | HN request time: 1.716s | source | bottom
Show context
isusmelj ◴[16 Oct 24 20:23 UTC] No.41863460[source]▶
I think the results show that just in general the compute is not used well. That the CPU took 8.4ms and GPU took 3.2ms shows a very small gap. I'd expect more like 10x - 20x difference here. I'd assume that the onnxruntime might be the issue. I think some hardware vendors just release the compute units without shipping proper support yet. Let's see how fast that will change.

Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption. To focus on speed you'd need to get rid of the memory bottleneck. Then you end up designing your own ASIC with it's own memory. The NPUs we see in most devices are part of the SoC around the CPU to offload AI computations. It would be interesting to run this benchmark in a infinite loop for the three devices (CPU, NPU, GPU) and measure power consumption. I'd expect the NPU to be lowest and also best in terms of "ops/watt"

replies(8): >>41863552 #>>41863639 #>>41864898 #>>41864928 #>>41864933 #>>41866594 #>>41869485 #>>41870575 #
AlexandrB ◴[16 Oct 24 20:33 UTC] No.41863552[source]▶
> Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption.

I have a sneaking suspicion that the real real reason for an NPU is marketing. "Oh look, NVDA is worth $3.3T - let's make sure we stick some AI stuff in our products too."

replies(8): >>41863644 #>>41863654 #>>41865529 #>>41865968 #>>41866150 #>>41866423 #>>41867045 #>>41870116 #
itishappy ◴[16 Oct 24 20:42 UTC] No.41863654[source]▶
I assume you're both right. I'm sure NPUs exist to fill a very real niche, but I'm also sure they're being shoehorned in everywhere regardless of product fit because "AI big right now."
replies(2): >>41864463 #>>41865770 #
brookst ◴[17 Oct 24 02:06 UTC] No.41865770[source]▶
The shoehorning only works if there is buyer demand.

As a company, if customers are willing to pay a premium for a NPU, or if they are unwilling to buy a product without one, it is not your place to say “hey we don’t really believe in the AI hype so we’re going to sell products people don’t want to prove a point”

replies(3): >>41865911 #>>41865951 #>>41866019 #
Spooky23 ◴[17 Oct 24 02:56 UTC] No.41866019[source]▶
Apple will have a completely AI capable product line in 18 months, with the major platforms basically done.

Microsoft is built around the broken Intel tick/tick model of incremental improvement — they are stuck with OEM shitware that will take years to flush out of the channel. That means for AI, they are stuck with cloud based OpenAI, where NVIDIA has them by the balls and the hyperscalers are all fighting for GPU.

Apple will deliver local AI features as software (the hardware is “free”) at a much higher margin - while Office 365 AI is like $400+ a year per user.

You’ll have people getting iPhones to get AI assisted emails or whatever Apple does that is useful.

replies(6): >>41866402 #>>41866405 #>>41866461 #>>41866768 #>>41875505 #>>41885273 #
1. hakfoo ◴[17 Oct 24 04:41 UTC] No.41866461[source]▶
We're still looking for "that is useful".

The stuff they've been trying to sell AI to the public with is increasingly looking as absurd as every 1978 "you'll store your recipes on the home computer" argument.

AI text became a Human Centipede story: Start with a coherent 10-word sentence, let AI balloon it into five pages of flowery nonsense, send it to someone else, who has their AI smash it back down to 10 meaningful words.

Coding assistance, even as spicy autocorrect, is often a net negative as you have to plow through hallucinations and weird guesses as to what you want but lack the tools to explain to it.

Image generation is already heading rapidly into cringe territory, in part due to some very public social media operations. I can imagine your kids' kids in 2040 finding out they generated AI images in the 2020s and looking at them with the same embarrassment you'd see if they dug out your high-school emo fursona.

There might well be some more "closed-loop" AI applications that make sense. But are they going to be running on every desktop in the world? Or are they going to be mostly used in datacentres and purpose-built embedded devices?

I also wonder how well some of the models and techniques scale down. I know Microsoft pushed a minimum spec to promote a machine as Copilot-ready, but that seems like it's going to be "Vista Basic Ready" redux as people try to run tools designed for datacentres full of Quadro cards, or at least high-end GPUs, on their $299 HP laptop.

replies(3): >>41866517 #>>41869224 #>>41870131 #
2. jjmarr ◴[17 Oct 24 04:51 UTC] No.41866517[source]▶
Cringe emo girls are trendy now because the nostalgia cycle is hitting the early 2000s. Your kid would be impressed if you told them you were a goth gf. It's not hard to imagine the same will happen with primitive AIs in the 40s.
replies(1): >>41866595 #
3. defrost ◴[17 Oct 24 05:09 UTC] No.41866595[source]▶
Early 2000's ??

Bela Lugosi Died in 1979, and Peter Murphy was onto his next band by 1984.

By 2000 Goth was fully a distant dot in the rear view mirror for the OG's

    In 2002, Murphy released *Dust* with Turkish-Canadian composer and producer Mercan Dede, which utilizes traditional Turkish instrumentation and songwriting, abandoning Murphy's previous pop and rock incarnations, and juxtaposing elements from progressive rock, trance, classical music, and Middle Eastern music, coupled with Dede's trademark atmospheric electronics.
https://www.youtube.com/watch?v=Yy9h2q_dr9k

https://en.wikipedia.org/wiki/Bauhaus_(band)

replies(2): >>41866683 #>>41866897 #
4. djur ◴[17 Oct 24 05:31 UTC] No.41866683{3}[source]▶
I'm not sure what "gothic music existed in the 1980s" is meant to indicate as a response to "goths existed in the early 2000s as a cultural archetype".
replies(1): >>41866722 #
5. defrost ◴[17 Oct 24 05:39 UTC] No.41866722{4}[source]▶
That Goths in 2000's were at best second wave nostalgia cycle of Goths from the 1980s.

That people recalling Goths in that period should beware of thinking that was a source and not an echo.

In 2006 Noel Fielding's Richmond Felicity Avenal was a basement dwelling leftover from many years past.

replies(1): >>41866888 #
6. bee_rider ◴[17 Oct 24 06:14 UTC] No.41866888{5}[source]▶
True Goth died our way before any of that. They totally sold out when the sacked Rome, the gold went to their heads and everything since then has been nostalgia.
replies(1): >>41866910 #
7. carlob ◴[17 Oct 24 06:17 UTC] No.41866897{3}[source]▶
There was a submission here a few months ago about the various incarnations of goth starting from the late Roman empire.

https://www.the-hinternet.com/p/the-goths

replies(1): >>41866940 #
8. defrost ◴[17 Oct 24 06:19 UTC] No.41866910{6}[source]▶
That was just the faux life Westside Visigoths .. what'd you expect?

#Ostrogoth #TwueGoth

9. defrost ◴[17 Oct 24 06:25 UTC] No.41866940{4}[source]▶
Was there? This one: https://news.ycombinator.com/item?id=41232761 ?

Nice: https://www.youtube.com/watch?v=VZvSqgn_Zf4

10. Spooky23 ◴[17 Oct 24 12:54 UTC] No.41869224[source]▶
The product isn’t released, so I don’t think we know what is or isn’t good.

People are clearly finding LLM tech useful, and we’re barely scratching the surface.

11. HelloNurse ◴[17 Oct 24 14:42 UTC] No.41870131[source]▶
I expect this sort of thing to go out of fashion and/or be regulated after "AI" causes some large life loss, e.g. starting a war or designing a collapsing building.