Hyperfine: A command-line benchmarking tool

Hyperfine is a great tool but when I was using it at Deno to benchmark startup time there was a lot of weirdness around the operating system apparently caching inodes of executables.

If you are looking at shaving sub 20ms numbers, be aware you may need to pull tricks on macos especially to get real numbers.

I've found pretty good results with the System Trace template in xcode instruments. You can also stack instruments, for example combining the file inspector with a virtual memory inspector.

I've run into some memory corruption with it sometimes, though, so be wary of that. Emerge tools has an alternative for iOS at least, maybe one day they'll port it to mac.

I never tried xcode instruments. Is the UX good for this kind of tool?

Caching is something that you almost always have to be aware of when benchmarking command line applications, even if the application itself has no caching behavior. Please see https://github.com/sharkdp/hyperfine?tab=readme-ov-file#warm... on how to run either warm-cache benchmarks or cold-cache benchmarks.

Not being able to rely on numbers to 20ms is pretty poor. That’s longer than a frame in a video game.

Windows has microsecond precision counters (see QueryPerformanceCounter and friends)

I'm fully aware but it's not a problem that warmup runs fix. An executable freshly compiled will always benchmark differently than one that has "cooled off" on macos, regardless of warmup runs.

I've tried to understand what the issue is (played with resigning executables etc) but it's literally something about the inode of the executable itself. Most likely part of the OSX security system.

Interesting. I've encountered this obviously on first run (because of the security checking it does on novel executables) but didn't realize this expired. Probably because I usually attribute it to a recompilation. Thanks.

System trace is pretty comprehensive, as long as it's something in there and you don't need a crazy strong profiler and are okay with occasional hangs in the profiler UI, it's pretty good