←back to thread

OpenAI o3 and o4-mini

(openai.com)
555 points maheshrijal | 2 comments | 16 Apr 25 17:01 UTC | HN request time: 0s | source
Show context
jcynix ◴[16 Apr 25 17:34 UTC] No.43708160[source]▶
To plan a visit to a dark sky place, I used duck.ai (Duckduckgo's experimental AI chat feature) to ask five different AIs on what date the new moon will happen in August 2025.

GPT-4o mini: The new moon in August 2025 will occur on August 12.

Llama 3.3 70B: The new moon in August 2025 is expected to occur on August 16, 2025.

Claude 3 Haiku: The new moon in August 2025 will occur on August 23, 2025.

o3-mini: Based on astronomical calculations, the new moon in August 2025 is expected to occur on August 7, 2025 (UTC). [...]

Mistral Small 3: To determine the date of the new moon in August 2025, we can use astronomical data or a reliable astronomical calendar. As of my last update in October 2023, I don't have real-time data access, but I can guide you on how to find this information. [...]

I got different answers, mostly wrong. My calendars (both paper and app versions) show me 23. august as the date.

And btw, when I asked those AIs which entries in a robots.text file would block most Chinese search engines, one of them (Claude) told me that it can't tell because that might be discriminatory: "I apologize, but I do not feel comfortable providing recommendations about how to block specific search engines in a robots.txt file. That could be seen as attempting to circumvent or manipulate search engine policies, which goes against my principles."

replies(5): >>43708189 #>>43708210 #>>43709121 #>>43709510 #>>43710432 #
WhatIsDukkha ◴[16 Apr 25 17:38 UTC] No.43708210[source]▶
I would never ask any of these questions of an LLM (and I use and rely on LLMs multiple times a day), this is a job for a computer.

I would also never ask a coworker for this precise number either.

replies(4): >>43708844 #>>43709680 #>>43709731 #>>43712171 #
1. jcynix ◴[16 Apr 25 20:02 UTC] No.43709731[source]▶
My reasoning for the plain question was: as people start to replace search engines by AI chat, I thought that asking "plain" questions to see how trustworthy the answers might be, would be a good test. Because plain folks will ask plain questions and won't think about the subtle details. They would not expect a "precise number" either, i.e. not 23:06 PDT, but would like to know if this weekend would be fine for a trip or the previous or next weekend would be better to book a "dark sky" tour.

And, BTW, I thought that LLMs are computers too ;-0

replies(1): >>43711041 #
2. WhatIsDukkha ◴[16 Apr 25 22:28 UTC] No.43711041[source]▶
I think its much better to help people learn that an LLM is "not" a computer (even if it technically is).

Thinking its a computer makes you do dumb things with them that they simply have never done a good job with.

Build intuitions about what they do well and intuitions about what they don't do well and help others learn the same things.

Don't encourage people to have poor ideas about how they work, it makes things worse.

Would you ask an LLM a phone number? If it doesn't use a function call the answer is simply not worth having.