←back to thread

AI engineers claim new algorithm reduces AI power consumption by 95%

(www.tomshardware.com)
352 points ferriswil | 2 comments | 19 Oct 24 18:03 UTC | HN request time: 0s | source
Show context
didgetmaster ◴[19 Oct 24 21:59 UTC] No.41891092[source]▶
Maybe I am just a natural skeptic, but whenever I see a headline that says 'method x reduces y by z%'; but when you read the text it instead says that optimizing some step 'could potentially reduce y by up to z%'; I am suspicious.

Why not publish some actual benchmarks that prove your claim in even a few special cases?

replies(4): >>41891148 #>>41891162 #>>41891234 #>>41891545 #
andrewstuart ◴[19 Oct 24 23:14 UTC] No.41891545[source]▶
https://github.com/microsoft/BitNet

"The first release of bitnet.cpp is to support inference on CPUs. bitnet.cpp achieves speedups of 1.37x to 5.07x on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by 55.4% to 70.0%, further boosting overall efficiency. On x86 CPUs, speedups range from 2.37x to 6.17x with energy reductions between 71.9% to 82.2%. Furthermore, bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5-7 tokens per second), significantly enhancing the potential for running LLMs on local devices. More details will be provided soon."

replies(1): >>41891612 #
jdiez17 ◴[19 Oct 24 23:24 UTC] No.41891612[source]▶
Damn. Seems almost too good to be true. Let’s see where this goes in two weeks.
replies(1): >>41891735 #
1. andrewstuart ◴[19 Oct 24 23:49 UTC] No.41891735[source]▶
Intel and AMD will be extremely happy.

Nvidia will be very unhappy.

replies(1): >>41892139 #
2. l11r ◴[20 Oct 24 01:18 UTC] No.41892139[source]▶
Their GPU will still be needed to do training. As far as I understand this will improve only interference performance and efficiency.