←back to thread

127 points warothia | 3 comments | | HN request time: 0s | source
Show context
openasocket ◴[] No.42167039[source]
Cool! Looks like is used dynamic loading. Which is certainly workable. One downside is that while dynamic loading is well-supported, dynamic unloading is generally not. Meaning if you are using dynamic loading and re-loading for hot updates the server process will start to accumulate memory. Not a huge issue of the server process is periodically restarted, but can potentially cause weird bugs.

You might be able to make something more robust by forking and loading the modules in a separate process. Then do something fancy with shared memory between the main process and the module processes. But I haven’t looked into this much

replies(3): >>42167215 #>>42167890 #>>42169759 #
warothia ◴[] No.42167215[source]
Oh really interesting, will look into it! Was afraid it would leak memory.
replies(1): >>42171330 #
1. mst ◴[] No.42171330[source]
You might not necessarily need even shared memory - it's possible to pass a file descriptor over a socket on all modern unices (and albeit with a completely different API also on Win32) so your control process could do an accept(), maybe read the headers in there, then talk to module processes to determine what the desired approach is, and then hand over the http socket so the module process can do whatever it needs to with it.

When I imagine how I'd use this in my head the imagined design rapidly gets much more complicated than what you're currently doing, and I'm not at all arguing that that complexity is necessarily worth it ... but there's also all sorts of cool+weird things you could implement that way that would be exceedingly tricky otherwise, so I figured I'd point it out anyway :D

(the example of code that does the relevant magic to get fds across a socket that immediately springs to mind is https://fastapi.metacpan.org/source/MLEHMANN/IO-FDPass-1.3/F... - yes, warning, it's inside a perl extension, but I see no reason that would impede you borrowing the C parts if it was useful ;)

replies(1): >>42171580 #
2. warothia ◴[] No.42171580[source]
Initially I did use fork a lot to allow each handler in its own “process” mainly because I wanted to isolate it as well in a container fashion. However the overhead became to much at the start when I just wanted it to work. Will be looking into it again!
replies(1): >>42171628 #
3. mst ◴[] No.42171628[source]
I think some stuff will be happier in the "director" process and some happier in its own "worker" process - and fd passing is the magic trick that lets you accept() in the former and still do the bulk of the work in an appropriate instance of the latter.

I 100% get the 'overhead' part - but at some point hopefully you'll have enough other stuff already running that the 'fun' factor of enabling that will win out :D