Most active commenters
  • tomrod(4)
  • frollogaston(3)

←back to thread

The first year of free-threaded Python

(labs.quansight.org)
291 points rbanffy | 13 comments | 16 May 25 09:42 UTC | HN request time: 1.179s | source | bottom
Show context
sgarland ◴[16 May 25 12:56 UTC] No.44004897[source]▶
> Instead, many reach for multiprocessing, but spawning processes is expensive

Agreed.

> and communicating across processes often requires making expensive copies of data

SharedMemory [0] exists. Never understood why this isn’t used more frequently. There’s even a ShareableList which does exactly what it sounds like, and is awesome.

[0]: https://docs.python.org/3/library/multiprocessing.shared_mem...

replies(8): >>44004956 #>>44005006 #>>44006103 #>>44006145 #>>44006664 #>>44006670 #>>44007267 #>>44013159 #
ogrisel ◴[16 May 25 13:05 UTC] No.44005006[source]▶
You cannot share arbitrarily structured objects in the `ShareableList`, only atomic scalars and bytes / strings.

If you want to share structured Python objects between instances, you have to pay the cost of `pickle.dump/pickle.dump` (CPU overhead for interprocess communication) + the memory cost of replicated objects in the processes.

replies(3): >>44006004 #>>44008341 #>>44010473 #
1. tomrod ◴[16 May 25 14:34 UTC] No.44006004[source]▶
I can fit a lot of json into bytes/strings though?
replies(4): >>44006041 #>>44006052 #>>44007146 #>>44008154 #
2. cjbgkagh ◴[16 May 25 14:37 UTC] No.44006041[source]▶
Perhaps flatbuffers would be better?
replies(2): >>44006072 #>>44007279 #
3. vlovich123 ◴[16 May 25 14:38 UTC] No.44006052[source]▶
That’s even worse than pickle.
replies(1): >>44006078 #
4. tomrod ◴[16 May 25 14:40 UTC] No.44006072[source]▶
I love learning from folks on HN -- thanks! Will check it out.
replies(1): >>44008294 #
5. tomrod ◴[16 May 25 14:41 UTC] No.44006078[source]▶
pickle pickles to pickle binary, yeah? So can stream that too with an io Buffer :D
6. frollogaston ◴[16 May 25 16:12 UTC] No.44007146[source]▶
If all your state is already json-serializable, yeah. But that's just as expensive as copying if not more, hence what cjbgkagh said about flatbuffers.
replies(1): >>44010306 #
7. tinix ◴[16 May 25 16:25 UTC] No.44007279[source]▶
let me introduce you to quickle.
8. reliabilityguy ◴[16 May 25 17:55 UTC] No.44008154[source]▶
What’s the point? The whole idea is to share an object, and not to serialize them whether it’s json, pickle, or whatever.
replies(1): >>44008726 #
9. notpushkin ◴[16 May 25 18:09 UTC] No.44008294{3}[source]▶
Take a look at https://capnproto.org/ as well, while at it.

Neither solve the copying problem, though.

replies(2): >>44010278 #>>44010304 #
10. tomrod ◴[16 May 25 18:55 UTC] No.44008726[source]▶
I mean, the answer to this is pretty straightforward -- because we can, not because we should :)
11. ◴[16 May 25 22:20 UTC] No.44010278{4}[source]▶
12. frollogaston ◴[16 May 25 22:25 UTC] No.44010304{4}[source]▶
Ah, I forgot capnproto doesn't let you edit a serialized proto in-memory, it's read-only. In theory this should be possible as long as you're not changing the length of anything, but I'm not surprised such trickery is unsupported.

So this doesn't seem like a versatile solution for sharing data structs between two Python processes. You're gonna have to reserialize the whole thing if one side wants to edit, which is basically copying.

13. frollogaston ◴[16 May 25 22:25 UTC] No.44010306[source]▶
oh nvm, that doesn't solve this either