←back to thread

259 points rbanffy | 2 comments | | HN request time: 0s | source
Show context
pansa2 ◴[] No.44004148[source]
Does removal of the GIL have any other effects on multi-threaded Python code (other than allowing it to run in parallel)?

My understanding is that the GIL has lasted this long not because multi-threaded Python depends on it, but because removing it:

- Complicates the implementation of the interpreter

- Complicates C extensions, and

- Causes single-threaded code to run slower

Multi-threaded Python code already has to assume that it can be pre-empted on the boundary between any two bytecode instructions. Does free-threaded Python provide the same guarantees, or does it require multi-threaded Python to be written differently, e.g. to use additional locks?

replies(4): >>44004334 #>>44004386 #>>44007874 #>>44011862 #
rfoo ◴[] No.44004334[source]
> Does free-threaded Python provide the same guarantees

Mostly. Some of the "can be pre-empted on the boundary between any two bytecode instructions" bugs are really hard to hit without free-threading, though. And without free-threading people don't use as much threading stuff. So by nature it exposes more bugs.

Now, my rants:

> have any other effects on multi-threaded Python code

It stops people from using multi-process workarounds. Hence, it simplifies user-code. IMO totally worth it to make the interpreter more complex.

> Complicates C extensions

The alternative (sub-interpreters) complicates C extensions more than free-threading and the top one most important C extension in the entire ecosystem, numpy, stated that they can't and they don't want to support sub-interpreters. On contrary, they already support free-threading today and are actively sorting out remaining bugs.

> Causes single-threaded code to run slower

That's the trade-off. Personally I think a single digit percentage slow-down of single-threaded code worth it.

replies(2): >>44005969 #>>44006545 #
celeritascelery ◴[] No.44005969[source]
> That's the trade-off. Personally I think a single digit percentage slow-down of single-threaded code worth it.

Maybe. I would expect that 99% of python code going forward will still be single threaded. You just don’t need that extra complexity for most code. So I would expect that python code as a whole will have worse performance, even though a handful of applications will get faster.

replies(3): >>44006258 #>>44006881 #>>44007872 #
pphysch ◴[] No.44006258[source]
But the bar to parallelizing code gets much lower, in theory. Your serial code got 5% slower but has a direct path to being 50% faster.

And if there's a good free-threaded HTTP server implementation, the RPS of "Python code as a whole" could increase dramatically.

replies(2): >>44006825 #>>44007196 #
fjasdfas ◴[] No.44007196[source]
You can do multiple processes with SO_REUSEPORT.

free-threaded makes sense if you need shared state.

replies(1): >>44009777 #
1. pphysch ◴[] No.44009777[source]
Any webserver that wants to cache and reuse content cares about shared state, but usually has to outsource that to a shared in-memory database because the language can't support it.
replies(1): >>44011868 #
2. monkeyelite ◴[] No.44011868[source]
And most web servers already need in memory databases for other things. And it’s a great design principle - use sharp focused tools.