←back to thread

AI agent benchmarks are broken

(ddkang.substack.com)
181 points neehao | 5 comments | 11 Jul 25 13:06 UTC | HN request time: 1.03s | source
1. xnx ◴[11 Jul 25 13:33 UTC] No.44531958[source]▶
All benchmarks are flawed. Some benchmarks are useful.
replies(1): >>44532081 #
2. yifanl ◴[11 Jul 25 13:44 UTC] No.44532081[source]▶
Here's a third sentence fragment: These benchmarks are not.
replies(2): >>44532272 #>>44534649 #
3. suddenlybananas ◴[11 Jul 25 14:02 UTC] No.44532272[source]▶
It's nearly a haiku!
replies(1): >>44533605 #
4. layer8 ◴[11 Jul 25 15:52 UTC] No.44533605{3}[source]▶

  All benchmarks are flawed.
  Not all benchmarks are useless.
  But these benchmarks are.
5. lcnPylGDnU4H9OF ◴[11 Jul 25 17:10 UTC] No.44534649[source]▶
Just want to nit: none of those are sentence fragments, they are complete thoughts with a subject and a predicate. Yours kinda comes close to being a fragment but it really just omits what "are not" (the predicate) is referring to, which is included in prior context.

For example, a fragment with a missing predicate.