←back to thread

Hyperfine: A command-line benchmarking tool

(github.com)
238 points hundredwatt | 2 comments | 18 Nov 24 21:47 UTC | HN request time: 0.421s | source
Show context
mmastrac ◴[19 Nov 24 02:36 UTC] No.42179600[source]▶
Hyperfine is a great tool but when I was using it at Deno to benchmark startup time there was a lot of weirdness around the operating system apparently caching inodes of executables.

If you are looking at shaving sub 20ms numbers, be aware you may need to pull tricks on macos especially to get real numbers.

replies(3): >>42180391 #>>42182037 #>>42182986 #
sharkdp ◴[19 Nov 24 10:46 UTC] No.42182037[source]▶
Caching is something that you almost always have to be aware of when benchmarking command line applications, even if the application itself has no caching behavior. Please see https://github.com/sharkdp/hyperfine?tab=readme-ov-file#warm... on how to run either warm-cache benchmarks or cold-cache benchmarks.
replies(1): >>42184549 #
1. mmastrac ◴[19 Nov 24 15:31 UTC] No.42184549[source]▶
I'm fully aware but it's not a problem that warmup runs fix. An executable freshly compiled will always benchmark differently than one that has "cooled off" on macos, regardless of warmup runs.

I've tried to understand what the issue is (played with resigning executables etc) but it's literally something about the inode of the executable itself. Most likely part of the OSX security system.

replies(1): >>42185911 #
2. renewiltord ◴[19 Nov 24 17:23 UTC] No.42185911[source]▶
Interesting. I've encountered this obviously on first run (because of the security checking it does on novel executables) but didn't realize this expired. Probably because I usually attribute it to a recompilation. Thanks.