This is probably better phrased as "LLMs may not provide consistent answers due to changing data and built-in randomness."
Barring rare(?) GPU race conditions, LLMs produce the same output given the same inputs.
Are these LLMs in the room with us?
Not a single LLM available as a SaaS is deterministic.
As for other models: I've only run ollama locally, and it, too, provided different answers for the same question five minutes apart
Edit/update: not a single LLM available as a SaaS's output is deterministic, especially when used from a UI. Pointing out that you could probably run a tightly controlled model in a tightly controlled environment to achieve deterministic output is very extremely irrelevant when describing output of grok in situations when the user has no control over it
The SaaS APIs are sometimes nondeterministic due to caching strategies and load balancing between experts on MoE models. However, if you took that model and executed it in single user environment, it could also be done deterministically.
I'm now wondering, would it be desirable to have deterministic outputs on an LLM?
How do we also turn off all the intermediate layers in between that we don't know about like "always rant about white genocide in South Africa" or "crash when user mentions David Meyer"?
Again, are those environments in the room with us?
In the context of the article, is the model executed in such an environment? Do we even know anything about the environment, randomness, sampling and anything in between or have any control over it (see e.g https://news.ycombinator.com/item?id=44528930)?
Gemini Flash has deterministic outputs, assuming you're referring to temperature 0 (obviously). Gemini Pro seems to be deterministic within the same kernel (?) but is likely switching between a few different kernels back and forth, depending on the batch or some other internal grouping.
> but is likely switching between a few different kernels back and forth, depending on the batch or some other internal grouping.
So you're literally saying it's non-deterministic
That is, it really is important in practical use because it's impossible to talk about stuff like in the original article without being able to consistently reproduce results.
Also, in almost all situations you really do want deterministic output (remember how "do what I want and what is expected" was an important property of computer systems? Good times)
> The only reason it was mentioned in the article is because the author is basically reverse engineering a particular model.
The author is attempting reverse engineering the model, the randomness and the temperature, the system prompts and the training set, and all the possible layers added by xAI in between, and still getting a non-deterministic output.
HN: no-no-no, you don't understand, it's 100% deterministic and it doesn't matter
And even if you do set it to zero, you never know what changes to the layers and layers of wrappers and system prompts you will run into on any given day resulting in "on this day we crash for certain input, and on other days we don't": https://www.techdirt.com/2024/12/03/the-curious-case-of-chat...