←back to thread

The first year of free-threaded Python

(labs.quansight.org)
253 points rbanffy | 1 comments | 16 May 25 09:42 UTC | HN request time: 0s | source
Show context
sgarland ◴[16 May 25 12:56 UTC] No.44004897[source]▶
> Instead, many reach for multiprocessing, but spawning processes is expensive

Agreed.

> and communicating across processes often requires making expensive copies of data

SharedMemory [0] exists. Never understood why this isn’t used more frequently. There’s even a ShareableList which does exactly what it sounds like, and is awesome.

[0]: https://docs.python.org/3/library/multiprocessing.shared_mem...

replies(7): >>44004956 #>>44005006 #>>44006103 #>>44006145 #>>44006664 #>>44006670 #>>44007267 #
ogrisel ◴[16 May 25 13:05 UTC] No.44005006[source]▶
You cannot share arbitrarily structured objects in the `ShareableList`, only atomic scalars and bytes / strings.

If you want to share structured Python objects between instances, you have to pay the cost of `pickle.dump/pickle.dump` (CPU overhead for interprocess communication) + the memory cost of replicated objects in the processes.

replies(3): >>44006004 #>>44008341 #>>44010473 #
tomrod ◴[16 May 25 14:34 UTC] No.44006004[source]▶
I can fit a lot of json into bytes/strings though?
replies(4): >>44006041 #>>44006052 #>>44007146 #>>44008154 #
cjbgkagh ◴[16 May 25 14:37 UTC] No.44006041[source]▶
Perhaps flatbuffers would be better?
replies(2): >>44006072 #>>44007279 #
tomrod ◴[16 May 25 14:40 UTC] No.44006072[source]▶
I love learning from folks on HN -- thanks! Will check it out.
replies(1): >>44008294 #
notpushkin ◴[16 May 25 18:09 UTC] No.44008294[source]▶
Take a look at https://capnproto.org/ as well, while at it.

Neither solve the copying problem, though.

replies(2): >>44010278 #>>44010304 #
1. frollogaston ◴[16 May 25 22:25 UTC] No.44010304[source]▶
Ah, I forgot capnproto doesn't let you edit a serialized proto in-memory, it's read-only. In theory this should be possible as long as you're not changing the length of anything, but I'm not surprised such trickery is unsupported.

So this doesn't seem like a versatile solution for sharing data structs between two Python processes. You're gonna have to reserialize the whole thing if one side wants to edit, which is basically copying.