Node uses a thread pool also. From Ryan's JSConf slides:
"Blocking (or possibly blocking) system calls are executed in the
thread pool. Signal handlers and thread pool callbacks are marshaled back into
the main thread via a pipe."
The point is that it has a concept of a main thread. You need to get away from that, and rather have a pool of threads ready to dispatch events, not just a pool for blocking calls to avoid the main thread blocking.
Why? Everything that runs in the main thread is Real Work to compute the response. It makes more sense to run multiple node.js processes. With a thread pool for all events, you have a lock in the critical path between accept()ing a connection and handing the fd off to a worker thread, and all worker threads pay the GC penalty. With multiple processes, you have one less lock hot spot, and the processes do their GCs concurrently.
What makes you think that a thread pool dispatching events would use a lock? No modern efficient thread pool implementation dispatching work items is serialized by a single lock, to my knowledge.
What you ought to have is multiple async accept calls outstanding, and as they complete they are entered into work queues. Worker threads (i.e. the thread pool) pulling work of those queues should steal work from other queues when their own queue is empty.
Pulling items off a queue, either by its associated worker thread or another worker thread stealing its work should be lock-free.
As I mentioned elsewhere, GC is not necessarily (or even often) the most efficient approach for request / response style servers. A more optimal approach is a heap associated with the request which can be freed in a single go when the request is done with, with all allocations associated with that request (i.e. that don't need to persist between requests) coming out of that heap.
You can even design a GC around this principle: have one GC heap per worker thread, and collect it after every request has been processed. There should be little or no roots for this heap associated with the worker itself, which should be (very) low down in its call stack after it is done with the request. If you have write barriers for any mutations to inter-request (shared) state, you can trace those to find out which bits of the worker thread's GC heap you need to keep (copy out). Then you can simply zero the GC heap and reset the free pointer. You can make your write barriers smart so that they are associated with that worker's heap, so you don't have to wander all over the shared heap looking for roots.
"What makes you think that a thread pool dispatching events would use a lock? No modern efficient thread pool implementation dispatching work items is serialized by a single lock, to my knowledge."
Are you talking about thread-safe queues implemented using atomic instructions? Those aren't free - how do you think they're implemented at the hardware level? The main advantage of atomic instructions over locks is removing the possibility of waiting on a preempted thread holding the lock (lock-freedom). They also have lower overhead than making system calls to provide locking. But the equivalent mutual exclusion logic (and contention penalties) just get moved down to the chipset level - now instead of waiting on other threads, you're waiting on other cores/CPUs.
"What you ought to have is multiple async accept calls outstanding, and as they complete they are entered into work queues. Worker threads (i.e. the thread pool) pulling work of those queues should steal work from other queues when their own queue is empty.
See e.g. http://www.bluebytesoftware.com/blog/2008/09/17/BuildingACus...
That article is horrible. Please do not follow the author's advice.
Besides the fact that the code deadlocks (see the first comment on the article), it's also easy to see that if one thread starts generating all the work the "solution" is going to become a single global queue (the "local work stealing queue" of that thread), with all the other threads looping through the global queue and then all each other's queues just to reach there!
The one good thing about that article is that the throughput gains on the toy benchmark (which doesn't deadlock or spawn tasks non-uniformly) nicely illustrate my point about the expense of contention even if using atomic instructions.
What the code in the article attempts to do is alleviate contention by partitioning the tasks among several queues. The problem is that if the work is not distributed uniformly among the queues, some threads will be left idle. The way to overcome that is to fake a global queue by having some way to synchronize the partitioned queues. The optimal solution to how to do this depends not only on the particular system you're running on, but also the pattern of work spawning by the application. And all this depends on being deadlock-free (something not managed by the article)!
Why would you ever go through something so horrible for an HTTP server? Multiple node.js processes are much simpler and more efficient.
I don't think you actually read what I wrote, or if you did, you willfully misunderstood it.
"The problem is that if the work is not distributed uniformly among the queues, some threads will be left idle" - this is why you use work-stealing queues! The very nature of work stealing queues is that the worker threads aren't left idle - they steal work from other threads' queues.
And CGI is not anything like the GC I talked about - if you have a process per request, where are you going to put your shared state?
But much of this discussion is besides the point. Don't forget, the OS scheduler is at its heart an event dispatcher when there are more runnable threads than CPU cores. The thread stack is little different than context provided to a triggered event. You want to have the same number of runnable threads as CPU cores in order to avoid the kernel cost of a context switch. You can do that by having multiple single-threaded processes, or multiple threads in a single process. While neither choice of partitioning affects the degree to which you can use an eventing style to serve requests, one - the separate process model - makes it much harder to share state. And therein lies the reason why I believe that optimal performance lies in threads, rather than processes. There are other good reasons for using processes instead - but it will be at some cost to efficiency.
"Blocking (or possibly blocking) system calls are executed in the thread pool. Signal handlers and thread pool callbacks are marshaled back into the main thread via a pipe."