Microsoft BitNet: inference framework for 1-bit LLMs

Can anyone help me understand how this works without special bitnet precision-specific hardware? Is special hardware unnecessary? Maybe it just doesn't reach the full bitnet potential without it? Or maybe it does, with some fancy tricks? Thanks!

I haven't checked this one out yet, but a common trick is using combinations of instructions and data invariants allowing you to work in "lanes".

The easiest example is xor, which can trivially be interpreted as either xoring one large integer or xoring a vector of smaller integers.

Take a look at the SWAR example here [0] as a pretty common/easy example of that technique being good for something in the real world.

Dedicated hardware is almost always better, but you can still get major improvements with a little elbow grease.

[0] https://nimrod.blog/posts/algorithms-behind-popcount/

This is extremely easy to implement in-FPGA.

While fancy hardware would make it faster, what you are comparing it to is a bunch of floating point and large number multiplication. I believe in this case they just use a look up table:

If one value is 0, it is 0.

If the signs are different, it is -1.

If the signs are the same, it is 1.

I’m sure those can be done with relatively few instructions using far less power hungry hardware.

The major benefit would be its significant decrease in memory consumption, rather than the compute itself. The major bottleneck of the current LLM infra is typically memory bandwidth and that's the reason why those chip industries are going crazy on HBM. Surely compute optimization helps but this is useful even without any hardware changes.

Inference speeds go brrrr as well.