←back to thread

The first year of free-threaded Python

(labs.quansight.org)
258 points rbanffy | 4 comments | 16 May 25 09:42 UTC | HN request time: 0s | source
Show context
sgarland ◴[16 May 25 12:56 UTC] No.44004897[source]▶
> Instead, many reach for multiprocessing, but spawning processes is expensive

Agreed.

> and communicating across processes often requires making expensive copies of data

SharedMemory [0] exists. Never understood why this isn’t used more frequently. There’s even a ShareableList which does exactly what it sounds like, and is awesome.

[0]: https://docs.python.org/3/library/multiprocessing.shared_mem...

replies(7): >>44004956 #>>44005006 #>>44006103 #>>44006145 #>>44006664 #>>44006670 #>>44007267 #
1. tinix ◴[16 May 25 16:24 UTC] No.44007267[source]▶
shared memory only works on dedicated hardware.

if you're running in something like AWS fargate, there is no shared memory. have to use the network and file system which adds a lot of latency, way more than spawning a process.

copying processes through fork is a whole different problem.

green threads and an actor model will get you much further in my experience.

replies(2): >>44008492 #>>44010425 #
2. bradleybuda ◴[16 May 25 18:31 UTC] No.44008492[source]▶
Fargate is just a container runtime. You can fork processes and share memory like you can in any other Linux environment. You may not want to (because you are running many cheap / small containers) but if your Fargate containers are running 0.25 vCPUs then you probably don't want traditional multiprocessing or multithreading...
replies(1): >>44010396 #
3. tinix ◴[16 May 25 22:37 UTC] No.44010396[source]▶
Go try it and report back.

Fargate isn't just ECS and plain containers.

You cannot use shared memory in fargate, there is literally no /dev/shm.

See "sharedMemorySize" here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...

> If you're using tasks that use the Fargate launch type, the sharedMemorySize parameter isn't supported.

4. sgarland ◴[16 May 25 22:42 UTC] No.44010425[source]▶
Well don’t use Fargate, there’s your problem. Run programs on actual servers, not magical serverless bullshit.