Scala 3 slowed us down?

For me the main takeaway of this is that you want to have automated performance tests in place combined with insights into flamegraphs by default. And especially for these kind of major language upgrade changes.

What are folks using for perf testing on JVM these days?

jmh is what I've always used for small benchmarks.

For production systems I use flight recordings (jfrs). To analyze I use java mission control.

For OOME problems I use a heap dump and eclipse memory analysis tool.

For microbenchmarks, I use JMH. But I tend to try and avoid doing those.

async-profiler

Benchmarking requires a bit of different setup than the rest of the testing, especially if you want down to the ms timings.

We have continous benchmarking of one of our tools, it's written in C++, and to get "same" results everytime we launch it on the same machine. This is far from ideal, but otherwise there be either noisy neighbours, pesky host (if it's vm), etc. etc.

One idea that we thought was what if we can run the same test on the same machine several times, and check older/newer code (or ideally through switches), and this could work for some codepaths, but not for really continous checkins.

Just wondering what folks do. I can assume what, but there is always something hidden, not well known.

I use jmh for micro benchmarks on any code we know is sensitive and to highlight performance differences between different implementations. (Usually keep them around but not run on CI as an archive of what we tried.)

Then we do benchmarking of the whole Java app in the container running async-profiler into pyroscope. We created a test harness for this that spins up and mocks any dependencies based on api subscription data and contracts and simulates performance.

This whole mechanism is generalised and only requires teams that create individual apps to work with contract driven testing for the test harness to function. During and after a benchmark we also verify whether other non functionals still work as required, i.e. whether tracing is still linked to the right requests etc. This works for almost any language that we use.

I agree for measuring latency differences you want similar setups. However, by running two versions of the app concurrently on the same machine they both get impacted more or less the same by noisy neighbours. Moreover, by inspecting the flamegraph you can, manually, see these large shifts of time allocation quickly. For automatic comparison you can of course use the raw data.

In addition you can look at total cpu seconds used, memory allocation on kernel level, and specifically for the jvm at the GC metrics and allocation rate. If these numbers change significantly then you know you need to have a look.

We do run this benchmark comparison in most nightly builds and find regressions this way.

Good points there - Thanks @spockz!

https://en.wikipedia.org/wiki/Hardware_performance_counter can help with noisy neighbors. I am still getting into this.

Yes, that can help with detecting how much cpu was actually used during the run. But it doesn’t influence benchmark results. Not sure how exactly to use it for doing subsequent runs and comparing final performance. Then this needs to be extrapolated to final performance in production.

Yeah, what you want to know is which change caused the slowdown, or maybe improved the performance and reasonable metric behind it (for example frame-rate for a game, or something like this).