Hi!
I'm starting to see how overlays works to write one. But I've got a doubt and I'm asking to be sure: Is the overlay stack called within each thread ? If so, I think it can't block an entire thread pool, right ?
My concern is about performance. The overlay I'm writing will only take place in some backend data modifications but it can be quite slow in terms of OpenLDAP connection handling, because it will be communicating via IPC with another process... If this overlay blocks only one connection, it's ok. But if it's blocks an entire thread pool, I'll have to figure out another solution.
Thanks a lot in advance Lucas Brasilino
Hi!
I'm starting to see how overlays works to write one. But I've got a doubt and I'm asking to be sure: Is the overlay stack called within each thread ? If so, I think it can't block an entire thread pool, right ?
My concern is about performance. The overlay I'm writing will only take place in some backend data modifications but it can be quite slow in terms of OpenLDAP connection handling, because it will be communicating via IPC with another process... If this overlay blocks only one connection, it's ok. But if it's blocks an entire thread pool, I'll have to figure out another solution.
The overlay stack is called within the execution of an operation. As such, it only affects the thread that executes that operation, unless your code does something nasty like locking global mutexes. This implies that it is *not* entirely blocking a connection: a connection can simultaneously spawn multiple operations. It is not clear from your message how your overlay can affect OpenLDAP connection handling, as this is not usually exposed in overlay handling.
p.
Hi!
The overlay stack is called within the execution of an operation. As such, it only affects the thread that executes that operation, unless your code does something nasty like locking global mutexes. This implies that it is *not* entirely blocking a connection: a connection can simultaneously spawn multiple operations. It is not clear from your message how your overlay can affect OpenLDAP connection handling, as this is not usually exposed in overlay handling.
I'm not aware about details of OpenLDAP multi-threading model, so I asked. :)
Imagine *if* each thread accepts one connection at time and it does it's operations in a 'serialized' manner: accept connection from client => fills up accordingly structures => pass data to overlay => pass data to backend => backend does it's job => pass data back to overlay => connection response to client. In this case, if overlay processing takes a long time, the connection from *that* and only *that* client will be delayed.
By your answer I think each thread uses a event loop to perform each operation, is that right ?
I'll not touch any thread resources like mutexes, some global variable and so on. It will not that evil :)
thanks a lot! Lucas Brasilino
Hi!
The overlay stack is called within the execution of an operation. As such, it only affects the thread that executes that operation, unless your code does something nasty like locking global mutexes. This implies that it is *not* entirely blocking a connection: a connection can simultaneously spawn multiple operations. It is not clear from your message how your overlay can affect OpenLDAP connection handling, as this is not usually exposed in overlay handling.
I'm not aware about details of OpenLDAP multi-threading model, so I asked. :)
Imagine *if* each thread accepts one connection at time and it does it's operations in a 'serialized' manner: accept connection from client => fills up accordingly structures => pass data to overlay => pass data to backend => backend does it's job => pass data back to overlay => connection response to client. In this case, if overlay processing takes a long time, the connection from *that* and only *that* client will be delayed.
By your answer I think each thread uses a event loop to perform each operation, is that right ?
Now I understand a bit more of your question. OpenLDAP does not work as I presume you imagined. There is one thread that listens for requests. As soon as a request is received, an operation structure is created, and queued for execution. As soon as one thread is available from a thread pool, the first operation in the queue is executed. So threads *do not* accept connections (a connection is a persistent channel between the client and the server; a connection is not related to a thread). A specific thread accepts requests (a request is mapped onto an operation), and each operation is executed by a different thread.
If your overlay takes a lot of time, in principle it will only affect its operation. If the client is performing multiple operations concurrently, only that operation will be delayed. Of course, if all operations use your overlay, and all operations of your overlay take a lot of time, at some point all the threads of the pool will be busy. Further requests will be accepted by the listener thread, but they will be queued until one thread of the pool is available.
If your overlay (for any reason) takes a long time to process an operation, but does not intensively use system resources (e.g. because it is contacting some remote system, and *that* system is taking long to respond), you may want to increase the size of the thread pool (the "threads" directive). In general, this is bad, but it may be good in special cases like this.
I'll not touch any thread resources like mutexes, some global variable and so on. It will not that evil :)
p.
Hi
Now I understand a bit more of your question. OpenLDAP does not work as I presume you imagined. There is one thread that listens for requests. As soon as a request is received, an operation structure is created, and queued for execution. As soon as one thread is available from a thread pool, the first operation in the queue is executed. So threads *do not* accept connections (a connection is a persistent channel between the client and the server; a connection is not related to a thread). A specific thread accepts requests (a request is mapped onto an operation), and each operation is executed by a different thread.
Thanks for your explanation. But, is this operation structure also points (or stores) connection related data like the binding DN and it's credentials (password) ? If so, do you can point me in which structure member it is stored ? The overlay has to have those informations...
If your overlay takes a lot of time, in principle it will only affect its operation. If the client is performing multiple operations concurrently, only that operation will be delayed. Of course, if all operations use your overlay, and all operations of your overlay take a lot of time, at some point all the threads of the pool will be busy. Further requests will be accepted by the listener thread, but they will be queued until one thread of the pool is available.
Great! So the overall performance of OpenLDAP won't be compromised, once the overlay will only act over modifications operations of one attribute (userPassword).
Regards Lucas Brasilino
Hi
Now I understand a bit more of your question. OpenLDAP does not work as I presume you imagined. There is one thread that listens for requests. As soon as a request is received, an operation structure is created, and queued for execution. As soon as one thread is available from a thread pool, the first operation in the queue is executed. So threads *do not* accept connections (a connection is a persistent channel between the client and the server; a connection is not related to a thread). A specific thread accepts requests (a request is mapped onto an operation), and each operation is executed by a different thread.
Thanks for your explanation. But, is this operation structure also points (or stores) connection related data like the binding DN and it's credentials (password) ? If so, do you can point me in which structure member it is stored ? The overlay has to have those informations...
The Operaton structure points to the Connection structure (op->o_conn, which is actually a macro that expands to op->o_hdr->oh_conn); it also contains copies of data that could be modified. For example, the DN the connection is bound as is in op->o_dn (pretty) and op->o_ndn (normalized). It is a copy, because the operation could modify it (e.g. via the proxied authorization control). The original value is in op->o_conn->c_ndn. Of course credentials are not stored; actually, slapd should have no notion of how the client bound after the bind succeeds, except for what type of mechanism was used, and the related ssf.
If your overlay takes a lot of time, in principle it will only affect its operation. If the client is performing multiple operations concurrently, only that operation will be delayed. Of course, if all operations use your overlay, and all operations of your overlay take a lot of time, at some point all the threads of the pool will be busy. Further requests will be accepted by the listener thread, but they will be queued until one thread of the pool is available.
Great! So the overall performance of OpenLDAP won't be compromised, once the overlay will only act over modifications operations of one attribute (userPassword).
Your overlay can access the password during a simple bind operation. It is in op->orb_cred.
p.
openldap-technical@openldap.org
-
Lucas Brasilino -
masarati＠aero.polimi.it