←back to thread

The "confident idiot" problem: Why AI needs hard rules, not vibe checks

(steerlabs.substack.com)
323 points steerlabs | 7 comments | 04 Dec 25 20:48 UTC | HN request time: 0s | source | bottom
Show context
keiferski ◴[08 Dec 25 13:46 UTC] No.46192154[source]▶
The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning.

LLMs don't do this. Instead, every question is immediately responded to with extreme confidence with a paragraph or more of text. I know you can minimize this by configuring the settings on your account, but to me it just highlights how it's not operating in a way remotely similar to the human-human one I mentioned above. I constantly find myself saying, "No, I meant [concept] in this way, not that way," and then getting annoyed at the robot because it's masquerading as a human.

replies(37): >>46192230 #>>46192268 #>>46192346 #>>46192427 #>>46192525 #>>46192574 #>>46192631 #>>46192754 #>>46192800 #>>46192900 #>>46193063 #>>46193161 #>>46193374 #>>46193376 #>>46193470 #>>46193656 #>>46193908 #>>46194231 #>>46194299 #>>46194388 #>>46194411 #>>46194483 #>>46194761 #>>46195048 #>>46195085 #>>46195309 #>>46195615 #>>46195656 #>>46195759 #>>46195794 #>>46195918 #>>46195981 #>>46196365 #>>46196372 #>>46196588 #>>46197200 #>>46198030 #
ryandrake ◴[08 Dec 25 15:48 UTC] No.46193656[source]▶
LLMs all behave as if they are semi-competent (yet eager, ambitious, and career-minded) interns or administrative assistants, working for a powerful CEO-founder. All sycophancy, confidence and positive energy. "You're absolutely right!" "Here's the answer you are looking for!" "Let me do that for you immediately!" "Here is everything I know about what you just mentioned." Never admitting a mistake unless you directly point it out, and then all sorry-this and apologize-that and "here's the actual answer!" It's exactly the kind of personality you always see bubbling up into the orbit of a rich and powerful tech CEO.

No surprise that these products are all dreamt up by powerful tech CEOs who are used to all of their human interactions being with servile people-pleasers. I bet each and every one of them are subtly or overtly shaped by feedback from executives about how they should respond to conversation.

replies(12): >>46193679 #>>46193872 #>>46193884 #>>46194322 #>>46195018 #>>46195066 #>>46195075 #>>46195385 #>>46196040 #>>46196762 #>>46196779 #>>46213184 #
1. code_for_monkey ◴[08 Dec 25 15:49 UTC] No.46193679[source]▶
thats the audience! Incompetent CEOS!
replies(2): >>46194458 #>>46195703 #
2. LogicFailsMe ◴[08 Dec 25 16:43 UTC] No.46194458[source]▶
Nearly every woman I know who is an English as a second language speaker is leaning hard into these things currently to make their prose sound more natural. And that has segued into them being treated almost as a confidant or a friend.

As flawed as they are currently, I remain astounded that people think they will never improve and that people don't want a plastic pal who's fun to be with(tm).

I find them frustrating personally, but then I ask them deep technical questions on obscure subjects and I get science fiction in return.

replies(1): >>46195236 #
3. pessimizer ◴[08 Dec 25 17:40 UTC] No.46195236[source]▶
> I get science fiction in return.

And once this garbage is in your context, it's polluting everything that comes after. If they don't know, I need them to shut up. But they don't know when they don't know. They don't know shit.

replies(2): >>46195554 #>>46196523 #
4. LogicFailsMe ◴[08 Dec 25 18:04 UTC] No.46195554{3}[source]▶
I am reminded of AI summaries and Microsoft Copilot. All push low value. But I separate that from the underlying potential of the technology. And I wish we heard more from deep domain experts like Karpathy and less from influencer dilettantes like Dylan Patel about where this is going.
5. bwahah4 ◴[08 Dec 25 18:16 UTC] No.46195703[source]▶
As an EE working in engineering 30 years, I ran out of fingers and toes 29 years ago trying to count the number of asocial, incompetent programmer Dark Triads who can only relate to the world through esoteric semantics unrelated to engineering problems right in front of them.

"To add two numbers I must first simulate the universe." types that created a bespoke DSL for every problem. Software engineering is a field full of educated idiots.

Programmers really need to stop patting themselves on the back. Same old biology with the same old faults. Programmers are subjected to the same old physics as everyone else.

6. thfuran ◴[08 Dec 25 19:28 UTC] No.46196523{3}[source]▶
I want to query a bayesian ontology, not a Markov chain with delusions of grandeur.
replies(1): >>46197771 #
7. TeMPOraL ◴[08 Dec 25 21:18 UTC] No.46197771{4}[source]▶
Alas, computation costs energy, so you get what you can afford.

Also one thing I thought LLMs did already is kill the misguided idea of applying prescriptive, formal categorization to the real world.