←back to thread

Recent AI model progress feels mostly like bullshit

(www.lesswrong.com)
579 points paulpauper | 10 comments | 06 Apr 25 18:01 UTC | HN request time: 1.324s | source | bottom
Show context
aerhardt ◴[06 Apr 25 19:30 UTC] No.43604214[source]▶
My mom told me yesterday that Paul Newman had massive problems with alcohol. I was somewhat skeptical, so this morning I asked ChatGPT a very simple question:

"Is Paul Newman known for having had problems with alcohol?"

All of the models up to o3-mini-high told me he had no known problems. Here's o3-mini-high's response:

"Paul Newman is not widely known for having had problems with alcohol. While he portrayed characters who sometimes dealt with personal struggles on screen, his personal life and public image were more focused on his celebrated acting career, philanthropic work, and passion for auto racing rather than any issues with alcohol. There is no substantial or widely reported evidence in reputable biographies or interviews that indicates he struggled with alcohol abuse."

There is plenty of evidence online that he struggled a lot with alcohol, including testimony from his long-time wife Joanne Woodward.

I sent my mom the ChatGPT reply and in five minutes she found an authoritative source to back her argument [1].

I use ChatGPT for many tasks every day, but I couldn't fathom that it would get so wrong something so simple.

Lesson(s) learned... Including not doubting my mother's movie trivia knowledge.

[1] https://www.newyorker.com/magazine/2022/10/24/who-paul-newma...

replies(27): >>43604240 #>>43604254 #>>43604266 #>>43604352 #>>43604411 #>>43604434 #>>43604445 #>>43604447 #>>43604474 #>>43605109 #>>43605148 #>>43605609 #>>43605734 #>>43605773 #>>43605938 #>>43605941 #>>43606141 #>>43606176 #>>43606197 #>>43606455 #>>43606465 #>>43606551 #>>43606632 #>>43606774 #>>43606870 #>>43606938 #>>43607090 #
1. blitzar ◴[06 Apr 25 19:53 UTC] No.43604411[source]▶
Does the as yet unwritten prequel of Idiocracy tell the tale of when we started asking Ai chat bots for facts and this was the point of no return for humanity?
replies(4): >>43604479 #>>43604639 #>>43606171 #>>43606697 #
2. aerhardt ◴[06 Apr 25 20:01 UTC] No.43604479[source]▶
Can you blame the users for asking it, when everyone is selling that as a key defining feature?

I use it for asking - often very niche - questions on advanced probability and simulation modeling, and it often gets those right - why those and not a simple verifiable fact about one of the most popular actors in history?

I don’t know about Idiocracy, but something that I have read specific warnings about is that people will often blame the user for any of the tool’s misgivings.

3. pclmulqdq ◴[06 Apr 25 20:20 UTC] No.43604639[source]▶
It turns out there's huge demand for un-monetized web search.
replies(2): >>43605041 #>>43605053 #
4. leereeves ◴[06 Apr 25 21:19 UTC] No.43605041[source]▶
I like that it's unmonetized, of course, but that's not why I use AI. I use AI because it's better at search. When I can't remember the right keywords to find something, or when the keywords aren't unique, I frequently find that web search doesn't return what I need and AI does.

It's impressive how often AI returns the right answer to vague questions. (not always though)

replies(1): >>43605052 #
5. pclmulqdq ◴[06 Apr 25 21:20 UTC] No.43605052{3}[source]▶
Google used to return the right answer to vague questions until it decided to return the most lucrative answer to vague questions instead.
replies(1): >>43606034 #
6. spudlyo ◴[06 Apr 25 21:20 UTC] No.43605053[source]▶
Soon sadly, there will be a huge demand for un-monetized LLMs. Enshitification is coming.
replies(1): >>43606177 #
7. HPsquared ◴[06 Apr 25 23:48 UTC] No.43606034{4}[source]▶
Fortunately there is a lot of competition in the LLM space.

Edit: and, more importantly, plenty of people willing to pay a subscription for good quality.

8. MarcelOlsz ◴[07 Apr 25 00:10 UTC] No.43606171[source]▶
The chatgpt'd tariffs marks this epoch perfectly.
9. ◴[07 Apr 25 00:12 UTC] No.43606177{3}[source]▶
10. jmount ◴[07 Apr 25 01:47 UTC] No.43606697[source]▶
Some prior works that work as prequels include C.M. Kornbluth's "The Marching Morons" and "The Little Black Bag."