Currently we have a single select(or epoll) loop in daemon.c that lists for all readable and writable sockets, then passes events off to the thread pool for processing.
We listen for writable sockets if a write attempt returns incomplete. There's a pair of mutexes and condition variables used to synch up here between the writing threads and the listener thread. It's quite a lot of lock overhead. As far as I can tell the main reason we do this is so that we can stop a writer thread on demand instead of having it just block forever in write().
We could make the listener's job a lot easier if we only have it listen for readable sockets, and make each writer thread do its own poll. It would need to poll on two descriptors - the one it's waiting to write on, and a pipe used by the listener to terminate the poll. That pipe could be signalled by e.g. the listener writing a byte to it; all writer threads could poll for its read status. (One question here - if multiple threads are polling the same descriptor, do they all receive the wakeup event?)
Howard Chu writes:
We listen for writable sockets if a write attempt returns incomplete. There's a pair of mutexes and condition variables used to synch up here between the writing threads and the listener thread. It's quite a lot of lock overhead. As far as I can tell the main reason we do this is so that we can stop a writer thread on demand instead of having it just block forever in write().
slapd seems to use non-blocking socket descriptors if it can, so it's rather that write() to a full socket would otherwise do a busy loop write()ing 0 bytes until there was room.
I wonder if slapd can use blocking sockets? Then it could shutdown() the descriptor to force a blocked write() call to terminate. Unless there are times when one would want to write something else to the socket instead - some TLS magic, maybe...
connection_init() does seem to believe slapd can deal with blocking sockets, since it tests for fcntl NONBLOCK failure but proceeds instead of aborting the connection.
Le 1/7/13 5:35 PM, Hallvard Breien Furuseth a écrit :
Howard Chu writes:
We listen for writable sockets if a write attempt returns incomplete. There's a pair of mutexes and condition variables used to synch up here between the writing threads and the listener thread. It's quite a lot of lock overhead. As far as I can tell the main reason we do this is so that we can stop a writer thread on demand instead of having it just block forever in write().
slapd seems to use non-blocking socket descriptors if it can, so it's rather that write() to a full socket would otherwise do a busy loop write()ing 0 bytes until there was room.
If you try to write to a full socket, you'll get a 0 as a result to a write attempt, and then, you will have to enqueue the data and set the write_op type so that epoll_wait wakes up when the socket is ready.
When the socket is ready for write, the thread will check the queue, and will try to empty it. If it succeeds, the write_op is unset and you are done. Otherwise, the socket is bloked again, and you enter in a new wait. Of course, never forget to unset the write_op flag, otherwise the epoll_wait() will exit immediately, and you'll get a busy loop again...
Hallvard Breien Furuseth wrote:
Howard Chu writes:
We listen for writable sockets if a write attempt returns incomplete. There's a pair of mutexes and condition variables used to synch up here between the writing threads and the listener thread. It's quite a lot of lock overhead. As far as I can tell the main reason we do this is so that we can stop a writer thread on demand instead of having it just block forever in write().
slapd seems to use non-blocking socket descriptors if it can, so it's rather that write() to a full socket would otherwise do a busy loop write()ing 0 bytes until there was room.
My point was that it would have been simpler all around to use a blocking descriptor and let write() sit and wait, except for the fact that we need the ability to stop it from waiting.
Of course we also want it to be non-blocking when we read, though we could use ioctl(FIONREAD) instead.
I wonder if slapd can use blocking sockets? Then it could shutdown() the descriptor to force a blocked write() call to terminate. Unless there are times when one would want to write something else to the socket instead - some TLS magic, maybe...
connection_init() does seem to believe slapd can deal with blocking sockets, since it tests for fcntl NONBLOCK failure but proceeds instead of aborting the connection.
Le 1/7/13 3:00 PM, Howard Chu a écrit : (One question here - if multiple threads are polling the same descriptor, do they all receive the wakeup event?)
AFAIK, yes.
-
Emmanuel Lécharny -
Hallvard Breien Furuseth -
Howard Chu