←back to thread

262 points rain1 | 1 comments | | HN request time: 0.221s | source
Show context
dale_glass ◴[] No.44442315[source]
How big are those in terms of size on disk and VRAM size?

Something like 1.61B just doesn't mean much to me since I don't know much about the guts of LLMs. But I'm curious about how that translates to computer hardware -- what specs would I need to run these? What could I run now, what would require spending some money, and what I might hope to be able to run in a decade?

replies(3): >>44442353 #>>44442714 #>>44450773 #
mjburgess ◴[] No.44442353[source]
At 1byte/param that's 1.6GB (f8), at 2 bytes (f16) that's 2.3GB -- but there's other space costs beyond loading the parameters for the GPU. So a rule of thumb is ~4x parameter count. So round up, 2B -> 2*4 = 8GB VRAM
replies(1): >>44452812 #
1. 1718627440 ◴[] No.44452812[source]
That sounds about the size of a modern browser (aka. any Electron et al. application)