←back to thread

GPT-4 and professional benchmarks: the wrong answer to the wrong question

(aisnakeoil.substack.com)
340 points agomez314 | 2 comments | 21 Mar 23 13:12 UTC | HN request time: 0.001s | source
Show context
jvanderbot ◴[21 Mar 23 13:38 UTC] No.35245898[source]▶
Memorization is absolutely the most valuable part of GPT, for me. I can get natural language responses to documentation, basic scripting / sysadmin, and API questions much more easily than searching other ways.

While this is an academic interest point, and rightly tamps down on hype around replacing humans, it doesn't dissuade what I think are most peoples' basic use case: "I don't know or don't remember how to do X, can you show me?"

This is finally a good enough "knowledge reference engine" that I can see being useful to those very people it is over hyped to replace.

replies(6): >>35245958 #>>35245959 #>>35245985 #>>35246065 #>>35246167 #>>35252251 #
vidarh ◴[21 Mar 23 13:42 UTC] No.35245958[source]▶
And asking higher level questions than what you'd otherwise look up. E.g. I've had ChatGPT write forms, write API calls, put together skeletons for all kinds of things that I can easily verify and fix when it gets details from but that are time consuming to do manually. I've held back and been sceptical but I'm at the point where I'm preparing to integrate models all over the place because there are plenty of places where you can add sufficient checks that doing mostly ok much of the time is sufficient to already provide substantial time savings.
replies(1): >>35246018 #
zer00eyz ◴[21 Mar 23 13:47 UTC] No.35246018[source]▶
> I've held back and been sceptical but I'm at the point where I'm preparing to integrate models all over the place because there are plenty of places where you can add sufficient checks that doing mostly ok much of the time is sufficient to already provide substantial time savings.

Im an old engineer.

Simply put NO.

If you don't understand it don't check it in. You are just getting code to cut and paste at a higher frequency and volume. At some point in time the fire will be burning around you and you won't have the tools to deal with it.

Nothing about mostly, much and sufficient ever ends well when it has been done in the name of saving time.

replies(7): >>35246026 #>>35246079 #>>35246149 #>>35246308 #>>35248566 #>>35249906 #>>35257939 #
1. simonw ◴[21 Mar 23 13:51 UTC] No.35246079[source]▶
"You are just getting code to cut and paste at a higher frequency and volume" genuinely sounds like the key value proposition of ChatGPT for coding to me.

I treat its output like I would treat a PR from a brand new apprentice engineer on my team: review it carefully, provide some feedback and iterate a few times, accept with tests.

replies(1): >>35251224 #
2. vidarh ◴[21 Mar 23 19:24 UTC] No.35251224[source]▶
Exactly. And my point in the first place was that it's most useful for those kinds of tasks you might hand to an apprentice where the apprentice might go away, spend a lot of time doing research and distill it down to some code that is simple, likely not all that great, but saves me time.

E.g. some tasks I've used it for recently:

* Giving me an outline of a JMAP client so I can pull down stuff from my e-mail to feed to GPT.

* Giving me an outline of an OpenAPI client.

* Giving me an index page and a layout for a website, including a simple starting point for the CSS that did a reset and added basic styling for the nav bar, forms and "hero "sections.

* Giving me an outline of a Stripe API integration.

* Writing a simple DNS server.

* Writing a simple web server capable of running Sinatra apps via Rack.

None of these were complex code that'd hide obscure bugs. None were big chunks of code. All of them were simple code that was always going to have big, gaping holes and sub-optimal choices that'd need to be addressed, but that was fine because they were scaffolding that saved me starting from scratch (and the last two were not intended to turn into anything, but just exploring what it could do)

That's where the biggest savings are for me, because if I asked it to generate particularly complex stuff, I'd end up spending ages getting comfortable it'd done it right and verifying it. But the simple but tedious stuff is something it's great for.