←back to thread

AI PCs Aren't Good at AI: The CPU Beats the NPU

(github.com)
488 points dbreunig | 6 comments | 16 Oct 24 19:44 UTC | HN request time: 0.932s | source | bottom
Show context
isusmelj ◴[16 Oct 24 20:23 UTC] No.41863460[source]▶
I think the results show that just in general the compute is not used well. That the CPU took 8.4ms and GPU took 3.2ms shows a very small gap. I'd expect more like 10x - 20x difference here. I'd assume that the onnxruntime might be the issue. I think some hardware vendors just release the compute units without shipping proper support yet. Let's see how fast that will change.

Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption. To focus on speed you'd need to get rid of the memory bottleneck. Then you end up designing your own ASIC with it's own memory. The NPUs we see in most devices are part of the SoC around the CPU to offload AI computations. It would be interesting to run this benchmark in a infinite loop for the three devices (CPU, NPU, GPU) and measure power consumption. I'd expect the NPU to be lowest and also best in terms of "ops/watt"

replies(8): >>41863552 #>>41863639 #>>41864898 #>>41864928 #>>41864933 #>>41866594 #>>41869485 #>>41870575 #
spookie ◴[16 Oct 24 23:28 UTC] No.41864933[source]▶
I've been building an app in pure C using onnxruntime, and it outperforms a comparable one done with python by a substancial amount. There are many other gains to be made.

(In the end python just calls C, but it's pretty interesting how much performance is lost)

replies(1): >>41867439 #
dacryn ◴[17 Oct 24 08:09 UTC] No.41867439[source]▶
agree there, but then again using ort in Rust is faster again.

You cannot compare python with a onxx executor.

I don't know what you used in Python, but if it's pytorch or similar, those are built with flexibility in mind, for optimal performance you want to export those to onxx and use whatever executor that is optimized for your env. onxxruntime is one of them, but definitely not the only one, and given it's from Microsoft, some prefer to avoid it and choose among the many free alternatives.

replies(1): >>41867667 #
rerdavies ◴[17 Oct 24 08:53 UTC] No.41867667[source]▶
Why would the two not be entirely comparable? PyTorch may be slower at building the models; but once the model is compiled and loaded on the NPU, there's just not a whole lot of Python involved anymore. A few hundred CPU cycles to push the input data using python; a few hundred CPU cycles to receive the results using python. And everything in-between gets executed on the NPU.
replies(1): >>41868158 #
1. noduerme ◴[17 Oct 24 10:21 UTC] No.41868158[source]▶
I really wish Python wasn't the language controlling all the C code. You need a controller, in a scripting language that's easy to modify, but it's a rather hideous choice. It would be like choosing to build the world's largest social network in PHP or something. lol.
replies(3): >>41868287 #>>41873886 #>>41921156 #
2. robertlagrant ◴[17 Oct 24 10:46 UTC] No.41868287[source]▶
> it's a rather hideous choice

Why?

3. johnny22 ◴[17 Oct 24 21:16 UTC] No.41873886[source]▶
isn't that the case? Which then became a dialect of php with a custom interpreter (and then compiler) as they scaled.
replies(1): >>41876745 #
4. noduerme ◴[18 Oct 24 06:11 UTC] No.41876745[source]▶
Yes, that was the case. I was being sarcastic. Zuck wrote facebook in PHP and spent millions of dollars then writing a custom interpreter to let his janky code run slightly faster than normal.
replies(1): >>41921175 #
5. rerdavies ◴[23 Oct 24 02:27 UTC] No.41921156[source]▶
Be careful for what you wish for.

I've just spent a week writing Neural net code in C++, so i have direct insight into what a C++ implementation might look like.

Much as I dislike python, and having to deal with endless runtime errors when your various inputs and outputs are mismatched, the inconvenience pales in comparison with having to wade through three pages of error messages that a C++ compiler generates when you have a single mismatched templated Eigen matrix with the incorrect dimension. The "Required from here" message you are actually interested in is typically the 3rd or fourth "Required from here", amongst a massive stack of cascading errors, each of which wraps to about 4 lines when displayed. You know what I mean. Sometimes you don't get a "Required from here" at all, which is horrifying. And it's infuriating to find and parse the template arguments of classes involved.

Debugging Python runtime errors is kind of horrible, and rubs me the wrong way on principle. But it is sweetness and light compared to debugging C++ compile-time error messages, which are unimaginably horrible.

The project: converting a C++ Neural Amp Modeller LV2 plugin to use fixed-size matrices (Eigen::Matrix<float,N,M>) instead of dynamic matrixes (Eigen::MatrixXf) to see if doing so would improve performance. (It does. Signficantly). So a substantial and realistic experiment in doing ML work in C++. Not directly comparable to working in Pytorch, but directly analogous in that it involves hooking up high-level ML constructs, like Conv1D, LayerT, WaveNet ML chunks.

6. rerdavies ◴[23 Oct 24 02:31 UTC] No.41921175{3}[source]▶
Zuck's obvious mistake: he should have written the PHP compiler to precompile chunks of GPU code that would be the only code that actually runs when serving web pages. </sarcasm>

Facebook isn't really a comparable problems, because ALL of the performance-critical code in PyTorch does runs on a GPU.