Dummy's Guide to Modern LLM Sampling

No, it's done to turn an uncreative model into a creative model. This idea that sampling isn't that important or is some violation of the bitter lesson is exactly why I had to call out the whole academic field as having a giant blindspot for this kind of research in our oral presentation at ICLR!

Top n sigma has been around since mid 2024, min_p around since 2023 and we are still waiting for these innovations to be integrated outside of open source stuff (i.e. outside of HF/vllm). It's being done slowly on purpose by API providers because they don't want to deal with the risk of models being "too creative" (also high temp likely breaks their watermarking)

One other thing - making models aware of their own sampling settings is super easy if you just feed it back to the model every token or generation (say, using structured generation). Models can control their own sampling settings and thus "have access to its internal states" with just a tiny bit of extra programming (the model can write that code for you now lol)

I guess variance is a better word for this. Creativity is a pretty loose term, for example most people will describe R1 as creative in RP/stories for its tendency to derail everything in an unhinged way, but it still lacks variance like every other modern model (kill the reasoning chain and look at logprobs to get what I mean). The bitter lesson is not some threshold and can't be violated, it describes a curve of diminishing returns. As long as you're on the easy part, it's fine.

But the bigger problem is that the concepts are expressed before they're decoded into the output distribution. You can steer them to a degree by hacking the autoregressive transport, but if the model itself learned that this concept corresponds to that particular concept, not a set of concepts (and RL tends to do exactly that), fixing it with sampling is usually hard to impossible, you'll just lose accuracy/make it dumber as you basically force out-of-distribution outputs.

How is it not a violation of the bitter lesson? You're trying to correct the model after the fact using human logic, where the bitter lesson would want you to just train a better model.

Not that I think that goes against your point -- I think it's rather a problem with the bitter lesson.

> No, it's done to turn an uncreative model into a creative model. This idea that sampling isn't that important or is some violation of the bitter lesson is exactly why I had to call out the whole academic field as having a giant blindspot for this kind of research in our oral presentation at ICLR!

I see this sentiment a lot, there's even people that swear by samplers like XTC (which sounds counter intuitive af) but it's always on "creative" tasks. On math tasks, with a clear correct/incorrect answer, none of the "creative" samplers come on top, not even min_p (except for crazy temperatures, and even there the overall accuracy is still lower than normal temps w/ normal sampling)...

The main problem is that "creativity" is such a subjective measure that it's hard to score properly.

I think "crazy" temperatures start around 100, not 2-3 as folks commonly claim in the literature.

You're right in general on this post, but I think you underestimate how many coomers/erp folks there are and how much they use LLMs. XTC was made for them to give some notion of slop removal. It's probably not quite as good at that task as the antislop sampler (from Sam Peach, EQ bench creator) - but I find XTC to be quite good at adding "spice" to outputs.

re: difficulty to measure "creativity" is especially true - especially around the difficulty of scoring it! We have some nitpickers of our own whispering into our ears about this. You don't happen to be at Stanford do you? IFYKYK...

The primary argument for why it's not a violation is that the heuristic is (almost) free. LLM designed samplers are and will probably be better - but in order to start the recursive self-improvement engine a few free heuristics will be needed.

The bitter lesson critique is that the human designed heuristics were not free, and harmed the notion of "letting the computer figure it out" by slowing down training. High temp sampling is very important for half-way decent synthetic data generation and thus enabling "letting the computer figure it out" for natural language. Better sampling is the only way to make high temperature generations coherent.