> Use ldclt from 389 project.

Thank you!  This seems to be giving the best results so far (and thanks for the example as the man page isn't that accessible to someone coming at it cold!).

My python isn't great and trying to effectively implement an async test client, using python-ldap, was obviously causing artificial problems.

Initial attempts at using ldap-python were capping out at 200 binds per second - using ldclt I'm getting over 1000/s, but I'm now seeing a problem with the threads dieing fairly early on as its losing connectivity to the LDAP service.

$ cat ldaptest
ldclt -h ldapserver.lab -p 389 \
      -e "bindeach,bindonly"  \
      -a 1000 \
      -n 10 \
      -D "cn=testXXXXXXX,dc=test,dc=lab" \
      -w foobar \
      -e "randombinddn,randombinddnlow=0000001,randombinddnhigh=0015000"

$ ./ldaptest
ldclt version 4.23
ldclt[9895]: Starting at Fri Nov  3 11:49:24 2017

ldclt[9895]: Average rate: 1225.00/thr  (1225.00/sec), total:  12250
ldclt[9895]: Average rate: 1322.80/thr  (1322.80/sec), total:  13228
ldclt[9895]: T000: Cannot ldap_simple_bind_s (cn=test0004533,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server)
ldclt[9895]: T003: Cannot ldap_simple_bind_s (cn=test0004880,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server)
ldclt[9895]: T002: Cannot ldap_simple_bind_s (cn=test0000180,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server)
ldclt[9895]: T002: thread is dead.
ldclt[9895]: T008: Cannot ldap_simple_bind_s (cn=test0004142,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server)
ldclt[9895]: T000: thread is dead.
ldclt[9895]: T005: Cannot ldap_simple_bind_s (cn=test0000056,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server)
ldclt[9895]: T005: thread is dead.
ldclt[9895]: T008: thread is dead.
ldclt[9895]: T003: thread is dead.
ldclt[9895]: T001: Cannot ldap_simple_bind_s (cn=test0003193,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server)
ldclt[9895]: T001: thread is dead.
ldclt[9895]: T009: Cannot ldap_simple_bind_s (cn=test0003687,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server)
ldclt[9895]: T009: thread is dead.
ldclt[9895]: T006: Cannot ldap_simple_bind_s (cn=test0001082,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server)
ldclt[9895]: T006: thread is dead.
ldclt[9895]: T004: Cannot ldap_simple_bind_s (cn=test0004994,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server)
ldclt[9895]: T004: thread is dead.
ldclt[9895]: T007: Cannot ldap_simple_bind_s (cn=test0003764,dc=test,dc=lab, foobar), error=-1 (Can't contact LDAP server)
ldclt[9895]: T007: thread is dead.
ldclt[9895]: Average rate:  275.40/thr  ( 275.40/sec), total:   2754
ldclt[9895]: Average rate:    0.00/thr  (   0.00/sec), total:      0
ldclt[9895]: All threads are dead - exit.
ldclt[9895]: T000: pendingNb=0
ldclt[9895]: T001: pendingNb=0
ldclt[9895]: T002: pendingNb=0
ldclt[9895]: T003: pendingNb=0
ldclt[9895]: T004: pendingNb=0
ldclt[9895]: T005: pendingNb=0
ldclt[9895]: T006: pendingNb=0
ldclt[9895]: T007: pendingNb=0
ldclt[9895]: T008: pendingNb=0
ldclt[9895]: T009: pendingNb=0
ldclt[9895]: Global total pending operations: 0
ldclt[9895]: Global average rate: 2823.20/thr  (705.80/sec), total:  28232
ldclt[9895]: Global number times "no activity" reports: never
ldclt[9895]: Global number of dead threads: 10
ldclt[9895]: Global error -1 (Can't contact LDAP server) occurs    10 times
ldclt[9895]: Ending at Fri Nov  3 11:50:04 2017
ldclt[9895]: Exit status 4 - Cannot bind.


I'm sure a better client would simply handle the reconnection, but I'm still a bit concerned as to why this behaviour is happening.

If I run the same test on a second test host while the server is dropping the connections, it has the same problem establishing new sessions - so assume that it's a buffer/limit that's being exceeded on the server?

Within the server logs I'm seeing:

local4[22124]: daemon: read active on 12
local4[22124]: connection_read(12): input error=-2 id=156683, closing.
local4[22124]: connection_closing: readying conn=156683 sd=12 for close
local4[22124]: connection_close: deferring conn=156683 sd=12
local4[22124]: connection_resched: attempting closing conn=156683 sd=12
local4[22124]: daemon: removing 12

Which old mailing list posts seem to suggest are caused by the client not closing the connection properly?  

But in this case, I have to assume the client isn't trying to close it down, and something is happening to cause the connection to be interrupted?

Does anyone have any suggestions where to look/tweak?

In the spirit of randomly changing things to see if it makes a difference, so far I've tweaked the following:

olcThreads: 512
olcTimeLimit: unlimited
olcSizeLimit: unlimited
olcSockbufMaxIncoming: 262143
olcSockbufMaxIncomingAuth: 16777215

.. and bumped up some of slapds limits.

# cat /proc/$(pidof slapd)/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             256977               256977               processes
Max open files            40960                40960                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       256977               256977               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us



Cheers

--
Tim
tim@yetanother.net