←back to thread

579 points paulpauper | 5 comments | | HN request time: 0.609s | source
Show context
InkCanon ◴[] No.43604503[source]
The biggest story in AI was released a few weeks ago but was given little attention: on the recent USAMO, SOTA models scored on average 5% (IIRC, it was some abysmal number). This is despite them supposedly having gotten 50%, 60% etc performance on IMO questions. This massively suggests AI models simply remember the past results, instead of actually solving these questions. I'm incredibly surprised no one mentions this, but it's ridiculous that these companies never tell us what (if any) efforts have been made to remove test data (IMO, ICPC, etc) from train data.
replies(18): >>43604865 #>>43604962 #>>43605147 #>>43605224 #>>43605451 #>>43606419 #>>43607255 #>>43607532 #>>43607825 #>>43608628 #>>43609068 #>>43609232 #>>43610244 #>>43610557 #>>43610890 #>>43612243 #>>43646840 #>>43658014 #
TrackerFF ◴[] No.43609068[source]
What would the average human score be?

I.e. if you randomly sampled N humans to take those tests.

replies(1): >>43609102 #
1. sanxiyn ◴[] No.43609102[source]
The average human score on USAMO (let alone IMO) is zero, of course. Source: I won medals at Korean Mathematical Olympiad.
replies(3): >>43609193 #>>43610920 #>>43612359 #
2. vintermann ◴[] No.43609193[source]
Average, hmmm?
3. lordgrenville ◴[] No.43610920[source]
I am hesitant to correct a math Olympian, but don't you mean the median?
replies(1): >>43621718 #
4. hyperbovine ◴[] No.43612359[source]
This is a disappointing answer from an MO alum. Pick a quantile, any quantile...
5. nhinck3 ◴[] No.43621718[source]
Average is fine.