←back to thread

Hyperfine: A command-line benchmarking tool

(github.com)
238 points hundredwatt | 5 comments | 18 Nov 24 21:47 UTC | HN request time: 0.001s | source
Show context
forrestthewoods ◴[19 Nov 24 06:37 UTC] No.42180588[source]▶
Hyperfine is hyper frustrating because it only works with really really fine microsecond level benchmarks. Once you get into the millisecond range it’s worthless.
replies(2): >>42180660 #>>42182084 #
anotherhue ◴[19 Nov 24 06:50 UTC] No.42180660[source]▶
It spawns a new process each time right? I would think that would but a cap on how accurate it can get.

For my purposes I use it all the time though, quick and easy sanity-check.

replies(2): >>42180722 #>>42180749 #
oguz-ismail ◴[19 Nov 24 07:09 UTC] No.42180749[source]▶
It spawns a new shell for each run and subtracts the average shell startup time from final results. Too much noise
replies(1): >>42180880 #
PhilipRoman ◴[19 Nov 24 07:40 UTC] No.42180880[source]▶
The shell can be disabled, leaving just fork+exec
replies(1): >>42182044 #
1. sharkdp ◴[19 Nov 24 10:48 UTC] No.42182044{3}[source]▶
Yes. If you don't make use of shell builtins/syntax, you can use hyperfine's `--shell=none`/`-N` option to disable the intermediate shell.
replies(1): >>42182636 #
2. oguz-ismail ◴[19 Nov 24 12:19 UTC] No.42182636[source]▶
You still need to quote the command though. `hyperfine -N ls "$dir"' won't work, you need `hyperfine -N "ls ${dir@Q}"' or something. It'd be better if you could specify commands like with `find -exec'.
replies(1): >>42182728 #
3. PhilipRoman ◴[19 Nov 24 12:32 UTC] No.42182728[source]▶
Oh that sucks, I really hate when programs impose useless shell parsing instead of letting the user give an argument vector natively.
replies(1): >>42183105 #
4. sharkdp ◴[19 Nov 24 13:20 UTC] No.42183105{3}[source]▶
I don't think it's useless. You can use hyperfine to run multiple benchmarks at the same time, to get a comparison between multiple tools. So if you want it to work without quotes, you need to (1) come up with a way to separate commands and (2) come up with a way to distinguish hyperfine arguments from command arguments. It's doable, but it's also not a great UX if you have to write something like

    hyperfine -N -- ls "$dir" \; my_ls "$dir"
replies(1): >>42183430 #
5. oguz-ismail ◴[19 Nov 24 13:53 UTC] No.42183430{4}[source]▶
> not a great UX

Looks fine to me. Obviously it's too late to undo that mistake, but a new flag to enable new behavior wouldn't hurt anyone.