←back to thread

44 points rundef | 5 comments | | HN request time: 0s | source
Show context
Veserv ◴[] No.46200616[source]
Why even bother with sampling profilers in Python? You can do full function traces for literally all of your code in production at ~1-10% overhead with efficient instrumentation.
replies(1): >>46200820 #
1. hansvm ◴[] No.46200820[source]
That depends on the code you're profiling. Even good line profilers can add 2-5x overhead on programs not optimized for them, and you're in a bit of a pickle because the programs least optimized for line profiling are those which are already "optimized" (fast results for a given task when written in Python).
replies(1): >>46201000 #
2. Veserv ◴[] No.46201000[source]
It does not, those are just very inefficient tracing profilers. You can literally trace C programs in 10-30% overhead. For Python you should only accept low single-digit overhead on average with 10% overhead only in degenerate cases with large numbers of tiny functions [1]. Anything more means your tracer is inefficient.

[1] https://functiontrace.com/

replies(2): >>46201136 #>>46202946 #
3. hansvm ◴[] No.46201136[source]
That's... intriguing. I just tested out functiontrace and saw 20-30% overhead. I didn't expect it to be anywhere near that low.
4. ajb ◴[] No.46202946[source]
Interesting, but "FunctionTrace is opensourced under the Prosperity Public License 3.0 license."

"This license allows you to use and share this software for noncommercial purposes for free and to try this software for commercial purposes for thirty days."

This is not an open source license. "Open Source" is a trademarked term meaning without restrictions of this kind; it is not a generic term meaning "source accessible".

You can also just use perf, but it does require an extra package from the python build (which uv frustratingly doesn't supply)

replies(1): >>46209336 #
5. Veserv ◴[] No.46209336{3}[source]
perf is a sampling profiler, not a function tracing profiler, so that fails the criteria I presented.

I used FunctionTrace as a example and evidence for my position that tracing Python is low overhead with proper design to bypass claims like: “You can not make it that low overhead or someone would have done it already, thus proving the negative.” I am not the author or in any way related to it, so you can bring that up with them.