←back to thread

GPT-4 and professional benchmarks: the wrong answer to the wrong question

(aisnakeoil.substack.com)
340 points agomez314 | 2 comments | 21 Mar 23 13:12 UTC | HN request time: 0s | source
Show context
helsinkiandrew ◴[21 Mar 23 13:48 UTC] No.35246038[source]▶
> This strongly suggests that the model is able to memorize solutions from its training set

I'm not sure why this is a problem - surely in systems like chatGPT we want the specifics that was in the training set not a generalization. It's not learning/reasoning from the training data its 'cleverly regurgitating' things its seen.

replies(1): >>35246131 #
1. jryb ◴[21 Mar 23 13:55 UTC] No.35246131[source]▶
For some applications, yes, but it comes at the cost of not knowing how powerful ChatGPT really is. So the claim from OpenAI that ChatGPT 4 can pass the bar exam are deceptive since it will likely fail any future bar exam.
replies(1): >>35246254 #
2. helsinkiandrew ◴[21 Mar 23 14:04 UTC] No.35246254[source]▶
Agreed, but apart from the novelty factor I'm not sure what the practical use is of ChatGPT passing the bar exam. In fact I think it's a good thing that it can't do well in any future tests as its likely to be used by the unscrupulous