BitNet b1.58 2B4T Technical Report

I built it at home this morning and tried it, perhaps my expectations were high but I wasn't terribly impressed. I asked it for a list of ten types of data I might show on a home info display panel. It gave me three. I clarified that I wanted ten, it gave me six. Every request after that just returned the same six things.

I know it's not chatGPT4 but I've tried other very small models that run on CPU only and had better results

> I've tried other very small models that run on CPU only and had better results

Maybe you can you share some comparative examples?

sure, here's my conversation with BitNet b1.58 2B4T

https://pastebin.com/ZZ1tADvp

here's the same prompt given to smollm2:135m

https://pastebin.com/SZCL5WkC

The quality of the second results are not fantastic. The data isn't public, and it repeats itself mentioning income a few times. I don't think I would use either of these models for accurate data but I was surprised at the truncated results from bitnet

Smollm2:360M returned better quality results, no repetition, but it did suggest things which didn't fit the brief exactly (public data given location only)

https://pastebin.com/PRFqnqVF

Edit:

I tried the same query on the live demo site and got much better results. Maybe something went wrong on my end?

You were using bitnet.cpp?

Yes

This is a technology demo, not a model you'd want to use. Because Bitnet models are only average 1.58 bits per weight you'd expect to need the model to be much larger than your fp8/fp16 counterparts in terms of parameter count. Plus this is only a 2 billion parameter model in the first place, even fp16 2B parameter models generally perform pretty poorly.

Ok that's fair. I still think something was up with my build though, the online demo worked far better than my local build