Why HN was slow and how Rtm fixed it

> when [MzScheme] wants to find a thread to run, it asks the O/S kernel about each socket in turn to see if any input is ready

They've never heard of select()? </snark>

But really, is there some reason that it's hard to collect up all the fds at once or something?

Read C10K? Both select() and poll() have this problem internally. You have to use one of the more advanced techniques available if you really want to scale. epoll(), kqueue() or friends.

Rtm's phrasing implies that MzScheme is making a syscall for each socket in turn, so it sounds like it isn't even using the basic select() or poll().

Based on RTM's description, it sounds like even select() would be a huge improvement. HN needs to solve the C100 problem before worrying about C10K.

I imagine mzscheme is using select under the hood. But I'm surprised they're not using poll. There's something up with the racket page, so I'm not able to download the mzscheme source just now (via http://arclanguage.org/install), but that's one thing I've noticed in Python web server implementations. Side-stepping epoll, libevent and all that for a second, there's a tendency to use select (which uses an array of file descriptors; 0 to highest file descriptor) instead of poll (which isn't encumbered the same way).

I think in part there is a tendency, among server developers, correctly, to fear anything that looks like a busy wait (e.g., with the name poll). But really poll is just as asynchronous as select in this context (I don't know about FreeBSD's implementation -- but Linux puts to sleep wait queues the same way, afaik). It just doesn't suffer from the crazy indexing scheme of select....

At any rate, I didn't get a chance to finish probing the internals of what mzscheme uses. But if there's a way to substitute poll for select, it can often alleviate those issues of 900 requests queue up and you eventually have an fd with a value of 1024 or greater -- even though you may not have 1024 actual concurrent requests....

Though others feel free to correct me if I'm wrong. I only comment because I came across a similar issue recently. This link may be useful too:

http://www.makelinux.net/ldd3/chp-6-sect-3.shtml

ETA. i finally got a copy of the most recent racket source (though probably not the one rtm and pg are using). but if anyone is curious, browse racket/src/network.c. the source version for mac uses a bunch of selects (e.g., for tcp-accept). replacing with poll might help.... the max number of FDs per login session is often 1024 by default. so you might want to bump that up if it's not already. and consider using poll.... just an idea.

I thought the same exact thing when I read rtm's description. I thought to myself select()/poll() is much more efficient, and it sounds like that's not what's implemented underneath.

poll() is slightly better than select(), because you only have to iterate over the file descriptors that were passed, rather than from 0 to nfds.

It doesn't hurt that its interface is far more pleasant to use than select's, either.