←back to thread

BitNet b1.58 2B4T Technical Report

(arxiv.org)
111 points galeos | 3 comments | 17 Apr 25 07:27 UTC | HN request time: 1.049s | source
Show context
nopelynopington ◴[17 Apr 25 12:57 UTC] No.43716144[source]▶
I built it at home this morning and tried it, perhaps my expectations were high but I wasn't terribly impressed. I asked it for a list of ten types of data I might show on a home info display panel. It gave me three. I clarified that I wanted ten, it gave me six. Every request after that just returned the same six things.

I know it's not chatGPT4 but I've tried other very small models that run on CPU only and had better results

replies(2): >>43716331 #>>43720674 #
ashirviskas ◴[17 Apr 25 13:13 UTC] No.43716331[source]▶
> I've tried other very small models that run on CPU only and had better results

Maybe you can you share some comparative examples?

replies(1): >>43716768 #
1. nopelynopington ◴[17 Apr 25 13:45 UTC] No.43716768[source]▶
sure, here's my conversation with BitNet b1.58 2B4T

https://pastebin.com/ZZ1tADvp

here's the same prompt given to smollm2:135m

https://pastebin.com/SZCL5WkC

The quality of the second results are not fantastic. The data isn't public, and it repeats itself mentioning income a few times. I don't think I would use either of these models for accurate data but I was surprised at the truncated results from bitnet

Smollm2:360M returned better quality results, no repetition, but it did suggest things which didn't fit the brief exactly (public data given location only)

https://pastebin.com/PRFqnqVF

Edit:

I tried the same query on the live demo site and got much better results. Maybe something went wrong on my end?

replies(1): >>43717694 #
2. sroussey ◴[17 Apr 25 14:47 UTC] No.43717694[source]▶
You were using bitnet.cpp?
replies(1): >>43717900 #
3. nopelynopington ◴[17 Apr 25 15:00 UTC] No.43717900[source]▶
Yes