←back to thread

GPT-4 and professional benchmarks: the wrong answer to the wrong question

(aisnakeoil.substack.com)
340 points agomez314 | 1 comments | 21 Mar 23 13:12 UTC | HN request time: 0.212s | source
Show context
jrochkind1 ◴[21 Mar 23 14:23 UTC] No.35246560[source]▶
Just realized that this is yet another example of the category of "imperfect measurement" that HN likes to talk about so much -- Goodhart's law and such.

The bar exam is a proxy for actually being a competent lawyer. But it's an imperfect proxy. It seems obvious that it will be possible for a computer program to target getting good on the bar exam, and be good at the bar exam without being a competent lawyer. It may also be possible for a human to? But hard enough that it's still a reasonable proxy when it's humans?

replies(2): >>35246726 #>>35247002 #
1. thwayunion ◴[21 Mar 23 14:31 UTC] No.35246726[source]▶
More-over, in most states, passing that bar isn't enough to practice.

In most states, the bar exam is just one component. Humans also need to pass several demanding courses in law school, and to get into law school they need to get a BA/BS degree, which again requires passing several demanding courses and writing various essays.

That's just to be allowed to practice law. Getting an actual job as a lawyer often means internships of one form or another, and then spending some time at the beginning of one's career as a de facto apprentice to a seasoned lawyer.

These sorts of exams play a very particular role in evaluation of humans.