←back to thread

Olmo 3: Charting a path through the model flow to lead open-source AI

(allenai.org)
361 points mseri | 2 comments | 21 Nov 25 06:50 UTC | HN request time: 0.001s | source
Show context
Y_Y ◴[21 Nov 25 09:59 UTC] No.46002975[source]▶
I asked it if giraffes were kosher to eat and it told me:

> Giraffes are not kosher because they do not chew their cud, even though they have split hooves. Both requirements must be satisfied for an animal to be permissible.

HN will have removed the extraneous emojis.

This is at odds with my interpretation of giraffe anatomy and behaviour and of Talmudic law.

Luckily old sycophant GPT5.1 agrees with me:

> Yes. They have split hooves and chew cud, so they meet the anatomical criteria. Ritual slaughter is technically feasible though impractical.

replies(3): >>46004171 #>>46005088 #>>46006063 #
embedding-shape ◴[21 Nov 25 13:10 UTC] No.46004171[source]▶
How many times did you retry (so it's not just up to chance), what were the parameters, specifically for temperature and top_p?
replies(2): >>46005252 #>>46005308 #
latexr ◴[21 Nov 25 15:05 UTC] No.46005252[source]▶
> How many times did you retry (so it's not just up to chance)

If you don’t know the answer to a question, retrying multiple times only serves to amplify your bias, you have no basis to know the answer is correct.

replies(3): >>46005264 #>>46005329 #>>46005903 #
1. zamadatix ◴[21 Nov 25 15:15 UTC] No.46005329[source]▶
If you retry until it gives the answer you want then it only serves to amplify your bias. If you retry and see how often it agrees with itself then it serves to show there is no confidence in an answer all around.

It's a bit of a crutch for LLMs lacking the ability to just say "I'm not sure" because doing so is against how they are rewarded in training.

replies(1): >>46005793 #
2. oivey ◴[21 Nov 25 16:10 UTC] No.46005793[source]▶
You’re still likely to just amplify your own bias if you don’t do some basic experimental controls like having some preselected criteria on how many retries you’re going to do or how many agreeing trials are statistically significant.