←back to thread

The first year of free-threaded Python

(labs.quansight.org)
253 points rbanffy | 2 comments | 16 May 25 09:42 UTC | HN request time: 0.001s | source
Show context
sgarland ◴[16 May 25 12:56 UTC] No.44004897[source]▶
> Instead, many reach for multiprocessing, but spawning processes is expensive

Agreed.

> and communicating across processes often requires making expensive copies of data

SharedMemory [0] exists. Never understood why this isn’t used more frequently. There’s even a ShareableList which does exactly what it sounds like, and is awesome.

[0]: https://docs.python.org/3/library/multiprocessing.shared_mem...

replies(7): >>44004956 #>>44005006 #>>44006103 #>>44006145 #>>44006664 #>>44006670 #>>44007267 #
ogrisel ◴[16 May 25 13:05 UTC] No.44005006[source]▶
You cannot share arbitrarily structured objects in the `ShareableList`, only atomic scalars and bytes / strings.

If you want to share structured Python objects between instances, you have to pay the cost of `pickle.dump/pickle.dump` (CPU overhead for interprocess communication) + the memory cost of replicated objects in the processes.

replies(3): >>44006004 #>>44008341 #>>44010473 #
tomrod ◴[16 May 25 14:34 UTC] No.44006004[source]▶
I can fit a lot of json into bytes/strings though?
replies(4): >>44006041 #>>44006052 #>>44007146 #>>44008154 #
1. frollogaston ◴[16 May 25 16:12 UTC] No.44007146[source]▶
If all your state is already json-serializable, yeah. But that's just as expensive as copying if not more, hence what cjbgkagh said about flatbuffers.
replies(1): >>44010306 #
2. frollogaston ◴[16 May 25 22:25 UTC] No.44010306[source]▶
oh nvm, that doesn't solve this either