AI coding and the peanut butter and jelly problem

This is actually no different than for humans once you get past the familiar. It's like the famous project management tree story: https://pmac-agpc.ca/project-management-tree-swing-story

If anything, LLMs have surprised at much better they are than humans in understanding instructions for text based activities. But they are MUCH worse than humans when it comes to creating images/videos.

> If anything, LLMs have surprised at much better they are than humans in understanding instructions for text based activities.

That's demonstrateably false, as proven by both OpenAI's own research [1] and endless independent studies by now.

What is fascinating is how some people cling on false ideas about what LLM is and isnt.

Its a recurring fallacy that's bound to get it's own name any time soon.

1: https://news.ycombinator.com/item?id=43155825

You’re comparing an LLM to expert programmers. Compare an LLM on the same task versus the average college student. And try it for a math problem. A poetry problem. Ask it a more complex question about history or to do an analysis of an essay you wrote.

Put it this way — I’m going to give you a text based question to solve and you have a choice to get another human to solve it (randomly selected from adults in the US) or ChatGPT, and both will be given 30 minutes to read and solve the problem — which would you choose?

> Put it this way — I’m going to give you a text based question to solve and you have a choice to get another human to solve it (randomly selected from adults in the US) or ChatGPT, and both will be given 30 minutes to read and solve the problem — which would you choose?

You wouldn't randomly selected an arbitrary adult from the USA to do a brain surgery on you, so this argument is rabulistic.

I would chose a random person from my company that was hired to work in that domain to solve problems in that domain. Yes, regardless of the position. Accountant in the domain, yes. Office organizer in the domain, yes. Essentially anyone in the domain, yes. No offense, but by restricting the selection to the general human population you're setting a low bar for LLMs here.

If the bar is for LLMs to replace domain experts about four years after introduction then yes, they are failing miserably.

But if you were to go back to 2020 and ask if your take a random human over a the state of the art AI to answer a text question you’d take the random human every time except for arithmetic (and you’d have to write it in math notation and not plain English).

And if you were to ask AI experts when would you chose an AI they’d say at least not for a decade or two, if ever.

I wasn't talking about how impressive AI systems are, or how far they've come. I was talking about the fact that any random human with any experience in a specific field -- even though they are not a domain expert -- is going to do better than an LLM. Or, human common sense >>>> what LLMs are doing.

We will have to agree to disagree about your fundamental point.