←back to thread

The first year of free-threaded Python

(labs.quansight.org)
258 points rbanffy | 1 comments | 16 May 25 09:42 UTC | HN request time: 0s | source
Show context
sgarland ◴[16 May 25 12:56 UTC] No.44004897[source]▶
> Instead, many reach for multiprocessing, but spawning processes is expensive

Agreed.

> and communicating across processes often requires making expensive copies of data

SharedMemory [0] exists. Never understood why this isn’t used more frequently. There’s even a ShareableList which does exactly what it sounds like, and is awesome.

[0]: https://docs.python.org/3/library/multiprocessing.shared_mem...

replies(7): >>44004956 #>>44005006 #>>44006103 #>>44006145 #>>44006664 #>>44006670 #>>44007267 #
chubot ◴[16 May 25 14:46 UTC] No.44006145[source]▶
Spawning processes generally takes much less than 1 ms on Unix

Spawning a PYTHON interpreter process might take 30 ms to 300 ms before you get to main(), depending on the number of imports

It's 1 to 2 orders of magnitude difference, so it's worth being precise

This is a fallacy with say CGI. A CGI in C, Rust, or Go works perfectly well.

e.g. sqlite.org runs with a process PER REQUEST - https://news.ycombinator.com/item?id=3036124

replies(8): >>44006287 #>>44007950 #>>44008877 #>>44009754 #>>44009755 #>>44009805 #>>44010011 #>>44012318 #
charleshn ◴[16 May 25 17:31 UTC] No.44007950{3}[source]▶
> Spawning processes generally takes much less than 1 ms on Unix

It depends on whether one uses clone, fork, posix_spawn etc.

Fork can take a while depending on the size of the address space, number of VMAs etc.

replies(2): >>44009524 #>>44009676 #
1. knome ◴[16 May 25 20:45 UTC] No.44009676{4}[source]▶
for glibc and linux, fork just calls clone. as does posix_spawn, using the flag CLONE_VFORK.