Emmanuel Lecharny wrote:
On 7/31/10 3:28 AM, Howard Chu wrote:
> Revisiting an old thread...
> I'm definitely seeing our current listener running out of steam on
> servers with more than 12 or so cores, and some work in this direction
> will definitely help. First it would be a good idea to classify all of
> the tasks that the current listener manages, before deciding how to
> divide them among multiple threads.
> The listener is responsible for many events right now:
> signal/shutdown processing
> idle timeout checks
> write timeout checks
> runqueue scheduling
> listener socket events
> threadpool pauses
> client socket read events
> client socket write events
> Splitting the client socket handling across multiple threads will
> bring the greatest improvement in scalability. Just need to check our
> thinking and make sure the remaining division of labor still makes sense.
> There are two cases that annoy me in our current design - why don't we
> just dedicate a thread to each listener socket, and let it block on
> accept() ? That would eliminate a bit of churn in the current select()
I'm not actually convinced that accept() traffic is a significant portion of
our load, so maybe that part can be left as-is.
> Likewise, why don't we just let writer threads block in
> instead of having them ask the listener to listen for writability on
> their socket? Or, if we're using non-blocking sockets, why don't we
> let the writer threads block in their own select call, instead of
> relying on the central thread to do the select and re-dispatch?
If your writer threads block until the socket is writeable, how many
threads will you need to have if many client never read the data ? You
might quickly not have any remaining writing threads available... The
select() is supposed to tell you when a socket is ready to accept more
write, then it's time to select a writing thread to push data into this
socket. You just need a pool of writing thread, and when a writing
thread has pushed data into the socket, then it becomes available for
the next write.
Unfortunately, while what you're describing is perfectly sane, that's not the
way things work right now. Our writer state isn't self-contained, so we can't
just package it up and put it on a queue somewhere. Right now, a single thread
is occupied for the duration of a single operation. If it blocks on a write,
it's stuck until the write timeout expires. Obviously it would be nicer if we
had a saner structure here, but we don't...
> The first, obvious answer is this: when threads are blocked in
> calls like accept(), we can't simply wake them up again for shutdown
> events or other situations. I believe the obvious fix here is to use
> select() in each thread, waiting for both the target fd and the
> wake_sds fd which is written to whenever a signal is caught. Off the
> top of my head I'm not sure, when several threads are selecting on the
> same fd, if they all receive a Readable event or if only one of them
> will. Anyone know?
Why would you have more than one select() ? Wouldn't it be better
have one thread processing the select() and dispatching the operation to
a pool of threads ?
That's what we have right now, and as far as I can see it's a bottleneck that
prevents us from utilizing more than 12 cores. (I could be wrong, and the
bottleneck might actually be in the thread pool manager. I haven't got precise
enough measurements yet to know for sure.)
Here's the situation: suppose you have thousands of clients connected and
active. Even if you have CPUs to spare, the number of connections you can
acknowledge and dispatch is limited by the speed of the single thread that's
processing select(). Even if all it does is walk thru the list of active
descriptors and dispatch a job to the thread pool for each one, it's only
possible to dispatch a fixed number of ops/second, no matter how many other
CPUs there are.
Right now on a 24 core server I'm seeing 48,000 searches/second and 50% CPU
utilization. Adding more clients only seems to increase the overall latency,
but CPU usage and throughput don't increase any further.
Also in the test I'm looking at, some of the issues I pointed to above aren't
even happening. E.g., none of the operations are blocking on write. It's all
about handling read events, nothing else. (So again, maybe it's OK to leave
the current write behavior as-is.)
As I've noted in the past, I can get double the throughput by running two
slapds concurrently, so it's not a question of network or I/O resources. Just
that a single listener thread (and/or a single mutex controlling the thread
pool) will only scale so far, and no further.
All in all, what costs CPU consumption in a server is most certainly
processing of incoming and outgoing requests, not the processing of the
select() operation, no ?
In relative costs, sure, but eventually even the small costs add up.
Im' maybe a bit tainted by Java, but it's really based on the
mechanisms under the hood...
Heh. Indeed. Concurrency issues are the same, no matter what language you use
to dress them up.
One other realization I had is that the current design makes it very easy to
build slapd as either threaded or non-threaded. Pushing too much activity into
other threads would break the non-threaded builds. But at this point, with
even cellphones going dual-core, I have to wonder how important it is to
maintain compatibility for non-threaded slapd builds. ??
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/