How large are large language models?

Less a technical comment and more just a mind-blown comment, but I still can’t get over just how much data is compressed into and available in these downloadable models. Yesterday I was on a plane with no WiFi, but had gemma3:12b downloaded through Ollama. Was playing around with it and showing my kids, and we fired history questions at it, questions about recent video games, and some animal fact questions. It wasn’t perfect, but holy cow the breadth of information that is embedded in an 8.1 GB file is incredible! Lossy, sure, but a pretty amazing way of compressing all of human knowledge into something incredibly contained.

Intelligence is compression some say

How well does that apply to robotics or animal intelligence? Manipulating the real world is more fundamental to human intelligence than compressing text.

Under the predictive coding model (and I'm sure some others), animal intelligence is also compression. The idea is that the early layers of the brain minimize how surprising incoming sensory signals are, so the later layers only have to work with truly entropic signal. But it has non-compression-based intelligence within those more abstract layers.