←back to thread

The first year of free-threaded Python

(labs.quansight.org)
258 points rbanffy | 2 comments | 16 May 25 09:42 UTC | HN request time: 0.447s | source
Show context
sgarland ◴[16 May 25 12:56 UTC] No.44004897[source]▶
> Instead, many reach for multiprocessing, but spawning processes is expensive

Agreed.

> and communicating across processes often requires making expensive copies of data

SharedMemory [0] exists. Never understood why this isn’t used more frequently. There’s even a ShareableList which does exactly what it sounds like, and is awesome.

[0]: https://docs.python.org/3/library/multiprocessing.shared_mem...

replies(7): >>44004956 #>>44005006 #>>44006103 #>>44006145 #>>44006664 #>>44006670 #>>44007267 #
ogrisel ◴[16 May 25 13:05 UTC] No.44005006[source]▶
You cannot share arbitrarily structured objects in the `ShareableList`, only atomic scalars and bytes / strings.

If you want to share structured Python objects between instances, you have to pay the cost of `pickle.dump/pickle.dump` (CPU overhead for interprocess communication) + the memory cost of replicated objects in the processes.

replies(3): >>44006004 #>>44008341 #>>44010473 #
1. notpushkin ◴[16 May 25 18:14 UTC] No.44008341[source]▶
We need a dataclass-like interface on top of a ShareableList.
replies(1): >>44010745 #
2. notpushkin ◴[16 May 25 23:35 UTC] No.44010745[source]▶
Actually, ShareableList feels like a tuple really (as it’s impossible to change its length). If we could mix ShareableList and collections.namedtuple together, it would get us 90% there (99.9% if we use typing.NamedTuple). Unfortunately, I can’t decipher either one [1, 2] from the first glance – maybe if I get some more sleep?

[1]: https://github.com/python/cpython/blob/3.13/Lib/collections/...

[2]: https://github.com/python/cpython/blob/3.13/Lib/typing.py#L2...