Heya,
Could anyone recommend a quick and dirty way in which to load test an OpenLDAP service to get an indication as to the potential capacity for the platform to handle bind requests?
I've used the python-ldap library to simulate other varieties of interactions successfully, but when it comes to binds, each interaction seems to generate a substantial amount of traffic behind the scenes, so suspect that *things* are happening that is artificially limiting the bind rate/s.
I've seen some links to people using JMeter to perform this sort of load testing, but thought there may be other (quicker to implement) tools also?
Thanks in advance,
Tim wrote:
I've used the python-ldap library to simulate other varieties of interactions successfully, but when it comes to binds, each interaction seems to generate a substantial amount of traffic behind the scenes, so suspect that *things* are happening that is artificially limiting the bind rate/s.
python-ldap itself is a pretty thin wrapper on top of libldap. Especially if you're using LDAPObject.simple_bind() or LDAPObject.simple_bind_s() [1] there is definitely no "traffic behind the scenes".
So if you have overhead on the client side I suspect your own Python code adds this.
Ciao, Michael.
[1] https://www.python-ldap.org/doc/html/ldap.html#ldap.LDAPObject.simple_bind
Use ldclt from 389 project.
#!/bin/sh ldclt \ -h localhost \ -a 1000 \ -p 389 \ -e bindeach,bindonly \ -D "uid=testXXXXXXXX,ou=people,dc=foo,dc=bar" \ -w foobar123 \ -e randombinddn,randombinddnlow=50000000,randombinddnhigh=51000000 \ -n 48
This is by far the simplest and highest performance load testing client. With the above test, I managed to achieve 14,400 binds per second. With two instances of the above test running simultaneously, I managed to achieve 12,000 binds per second for each instance -> 24,000 binds per second with OpenLDAP 2.4.44 and 2.4.45 both compiled from source.
On Wed, Nov 1, 2017 at 9:33 PM, Michael Ströder michael@stroeder.com wrote:
Tim wrote:
I've used the python-ldap library to simulate other varieties of interactions successfully, but when it comes to binds, each interaction seems to generate a substantial amount of traffic behind the scenes, so suspect that *things* are happening that is artificially limiting the bind rate/s.
python-ldap itself is a pretty thin wrapper on top of libldap. Especially if you're using LDAPObject.simple_bind() or LDAPObject.simple_bind_s() [1] there is definitely no "traffic behind the scenes".
So if you have overhead on the client side I suspect your own Python code adds this.
Ciao, Michael.
[1] https://www.python-ldap.org/doc/html/ldap.html#ldap.LDAPObject.simple_bind
Use ldclt from 389 project.
Thank you! This seems to be giving the best results so far (and thanks for the example as the man page isn't that accessible to someone coming at it cold!).
My python isn't great and trying to effectively implement an async test client, using python-ldap, was obviously causing artificial problems.
Initial attempts at using ldap-python were capping out at 200 binds per second - using ldclt I'm getting over 1000/s, but I'm now seeing a problem with the threads dieing fairly early on as its losing connectivity to the LDAP service.
$ cat ldaptest ldclt -h ldapserver.lab -p 389 \ -e "bindeach,bindonly" \ -a 1000 \ -n 10 \ -D "cn=testXXXXXXX,dc=test,dc=lab" \ -w foobar \ -e "randombinddn,randombinddnlow=0000001,randombinddnhigh=0015000"
$ ./ldaptest ldclt version 4.23 ldclt[9895]: Starting at Fri Nov 3 11:49:24 2017
ldclt[9895]: Average rate: 1225.00/thr (1225.00/sec), total: 12250 ldclt[9895]: Average rate: 1322.80/thr (1322.80/sec), total: 13228 ldclt[9895]: T000: Cannot ldap_simple_bind_s (cn=test0004533,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server) ldclt[9895]: T003: Cannot ldap_simple_bind_s (cn=test0004880,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server) ldclt[9895]: T002: Cannot ldap_simple_bind_s (cn=test0000180,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server) ldclt[9895]: T002: thread is dead. ldclt[9895]: T008: Cannot ldap_simple_bind_s (cn=test0004142,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server) ldclt[9895]: T000: thread is dead. ldclt[9895]: T005: Cannot ldap_simple_bind_s (cn=test0000056,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server) ldclt[9895]: T005: thread is dead. ldclt[9895]: T008: thread is dead. ldclt[9895]: T003: thread is dead. ldclt[9895]: T001: Cannot ldap_simple_bind_s (cn=test0003193,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server) ldclt[9895]: T001: thread is dead. ldclt[9895]: T009: Cannot ldap_simple_bind_s (cn=test0003687,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server) ldclt[9895]: T009: thread is dead. ldclt[9895]: T006: Cannot ldap_simple_bind_s (cn=test0001082,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server) ldclt[9895]: T006: thread is dead. ldclt[9895]: T004: Cannot ldap_simple_bind_s (cn=test0004994,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server) ldclt[9895]: T004: thread is dead. ldclt[9895]: T007: Cannot ldap_simple_bind_s (cn=test0003764,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server) ldclt[9895]: T007: thread is dead. ldclt[9895]: Average rate: 275.40/thr ( 275.40/sec), total: 2754 ldclt[9895]: Average rate: 0.00/thr ( 0.00/sec), total: 0 ldclt[9895]: All threads are dead - exit. ldclt[9895]: T000: pendingNb=0 ldclt[9895]: T001: pendingNb=0 ldclt[9895]: T002: pendingNb=0 ldclt[9895]: T003: pendingNb=0 ldclt[9895]: T004: pendingNb=0 ldclt[9895]: T005: pendingNb=0 ldclt[9895]: T006: pendingNb=0 ldclt[9895]: T007: pendingNb=0 ldclt[9895]: T008: pendingNb=0 ldclt[9895]: T009: pendingNb=0 ldclt[9895]: Global total pending operations: 0 ldclt[9895]: Global average rate: 2823.20/thr (705.80/sec), total: 28232 ldclt[9895]: Global number times "no activity" reports: never ldclt[9895]: Global number of dead threads: 10 ldclt[9895]: Global error -1 (Can't contact LDAP server) occurs 10 times ldclt[9895]: Ending at Fri Nov 3 11:50:04 2017 ldclt[9895]: Exit status 4 - Cannot bind.
I'm sure a better client would simply handle the reconnection, but I'm still a bit concerned as to why this behaviour is happening.
If I run the same test on a second test host while the server is dropping the connections, it has the same problem establishing new sessions - so assume that it's a buffer/limit that's being exceeded on the server?
Within the server logs I'm seeing:
local4[22124]: daemon: read active on 12 local4[22124]: connection_read(12): input error=-2 id=156683, closing. local4[22124]: connection_closing: readying conn=156683 sd=12 for close local4[22124]: connection_close: deferring conn=156683 sd=12 local4[22124]: connection_resched: attempting closing conn=156683 sd=12 local4[22124]: daemon: removing 12
Which old mailing list posts seem to suggest are caused by the client not closing the connection properly?
But in this case, I have to assume the client isn't trying to close it down, and something is happening to cause the connection to be interrupted?
Does anyone have any suggestions where to look/tweak?
In the spirit of randomly changing things to see if it makes a difference, so far I've tweaked the following:
olcThreads: 512 olcTimeLimit: unlimited olcSizeLimit: unlimited olcSockbufMaxIncoming: 262143 olcSockbufMaxIncomingAuth: 16777215
.. and bumped up some of slapds limits.
# cat /proc/$(pidof slapd)/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 256977 256977 processes Max open files 40960 40960 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 256977 256977 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
Cheers
On Wed, 2017-11-01 at 20:33 +0100, Michael Ströder wrote:
Tim wrote:
I've used the python-ldap library to simulate other varieties of interactions successfully, but when it comes to binds, each interaction seems to generate a substantial amount of traffic behind the scenes, so suspect that *things* are happening that is artificially limiting the bind rate/s.
python-ldap itself is a pretty thin wrapper on top of libldap. Especially if you're using LDAPObject.simple_bind() or LDAPObject.simple_bind_s() [1] there is definitely no "traffic behind the scenes".
So if you have overhead on the client side I suspect your own Python code adds this.
python-ldap is very thin, but it does have a global mutex that can prevent you "really hitting" the ldap server you are testing.
For our tests we have a tool called ldclt that we use for load tests (389 ds base ships with it), and it can do bind tests with connection, or "just binds" over the same connection to stress the different parts. It would work just the same pointed at openldap I'm sure.
Alternately, look at slamd, or the tools Howard suggested,
Thanks!
[1] https://www.python-ldap.org/doc/html/ldap.html#ldap.LDAPObject.simple _bind
William Brown wrote:
On Wed, 2017-11-01 at 20:33 +0100, Michael Ströder wrote:
Tim wrote:
I've used the python-ldap library to simulate other varieties of interactions successfully, but when it comes to binds, each interaction seems to generate a substantial amount of traffic behind the scenes, so suspect that *things* are happening that is artificially limiting the bind rate/s.>>
python-ldap itself is a pretty thin wrapper on top of libldap. Especially if you're using LDAPObject.simple_bind() or LDAPObject.simple_bind_s() [1] there is definitely no "traffic behind the scenes".
So if you have overhead on the client side I suspect your own Python code adds this.
python-ldap is very thin, but it does have a global mutex that can prevent you "really hitting" the ldap server you are testing.
Yes, you're right. But not sure whether you really hit the GIL limit since python-ldap releases GIL whenever it calls libldap functions. And of course when running a multi-threaded client each thread should have its own LDAPObject instance. (I assume here that Python is built with thread support and python-ldap was built against libldap_r. Otherwise all calls into libldap (without _r) are serialized with a global lock.)
Ciao, Michael.
On Thu, 2017-11-02 at 09:08 +0100, Michael Ströder wrote:
William Brown wrote:
On Wed, 2017-11-01 at 20:33 +0100, Michael Ströder wrote:
Tim wrote:
I've used the python-ldap library to simulate other varieties of interactions successfully, but when it comes to binds, each interaction seems to generate a substantial amount of traffic behind the scenes, so suspect that *things* are happening that is artificially limiting the bind rate/s.>>
python-ldap itself is a pretty thin wrapper on top of libldap. Especially if you're using LDAPObject.simple_bind() or LDAPObject.simple_bind_s() [1] there is definitely no "traffic behind the scenes".
So if you have overhead on the client side I suspect your own Python code adds this.
python-ldap is very thin, but it does have a global mutex that can prevent you "really hitting" the ldap server you are testing.
Yes, you're right. But not sure whether you really hit the GIL limit since python-ldap releases GIL whenever it calls libldap functions. And of course when running a multi-threaded client each thread should have its own LDAPObject instance. (I assume here that Python is built with thread support and python- ldap was built against libldap_r. Otherwise all calls into libldap (without _r) are serialized with a global lock.)
Yeah, the GIL isn't the issue, it's the global lock. You need to start multiple separate python interpreters to really generate this load. We have a python load test, but you have to start about 16 instances of it to really stress a server.
I've always wondered what the purpose of the ldap lock was, but that's a topic for it's own thread I think :)
Ciao, Michael.
William Brown wrote:
On Thu, 2017-11-02 at 09:08 +0100, Michael Ströder wrote:
Yes, you're right. But not sure whether you really hit the GIL limit since python-ldap releases GIL whenever it calls libldap functions. And of course when running a multi-threaded client each thread should have its own LDAPObject instance. (I assume here that Python is built with thread support and python- ldap was built against libldap_r. Otherwise all calls into libldap (without _r) are serialized with a global lock.)>
Yeah, the GIL isn't the issue, it's the global lock. You need to start multiple separate python interpreters to really generate this load. We have a python load test, but you have to start about 16 instances of it to really stress a server.
I've always wondered what the purpose of the ldap lock was, but that's a topic for it's own thread I think :)
In case of python-ldap linked against libldap_r the global lock only serializes calls into ldap_initialize(). So if you're using a separate persistent connection per thread it should never hit this lock again.
Hmm, to be on-topic here I'm looking at ldap_initialize(3) again:
Note: the first call into the LDAP library also initial- izes the global options for the library. As such the first call should be single-threaded or otherwise pro- tected to insure that only one call is active. It is rec- ommended that ldap_get_option() or ldap_set_option() be used in the program's main thread before any additional threads are created. See ldap_get_option(3).
@OpenLDAP developers: Does that mean it would be sufficient to have *one* serialized call into libldap (e.g. *one* call of ldap_get_option(3) when importing module "ldap") and then use no global lock afterwards?
Ciao, Michael.
openldap-technical@openldap.org
-
Michael Ströder -
MJ J -
Tim -
William Brown