Hi,
Was load testing against openldap (slapd-meta) where we were trying to submit 1000 concurrent bind requests. This was done using the JMeter load testing tool. In every run approximately only 420-430 bind requests are successful and remaining 500+ bind requests fail with "Communication Exception" error. We also tried with 500 bind requests and the result was somewhat same - 420 odd successful and 70-80 failed with communication exception. We have openldap 2.4.56 installed on a RHEL 8.1 host with 32GB memory and 8CPU. We have configured max open files for the slapd process to be 65K .
Our requirement is to support 6000 concurrent requests at minimum. Is there any configuration that can be tweaked to support this amount of concurrent load ? I do see "olcConcurrency" & "olcThreads" set to 0 and 16 respectively. Should we be tweaking these to support the required load, if yes what should be the values for those ? Please assist. Thank you for your help !
shekhar.shrinivasan@gmail.com wrote:
Hi,
Was load testing against openldap (slapd-meta) where we were trying to submit 1000 concurrent bind requests. This was done using the JMeter load testing tool. In every run approximately only 420-430 bind requests are successful and remaining 500+ bind requests fail with "Communication Exception" error. We also tried with 500 bind requests and the result was somewhat same - 420 odd successful and 70-80 failed with communication exception.
What does a "Communication exception" actually mean? Connection refused, connection timed out, what?
We've tested tens of thousands of concurrent connections with slapd. It is unlikely to be the problem here.
We have openldap 2.4.56 installed on a RHEL 8.1 host with 32GB memory and 8CPU. We have configured max open files for the slapd process to be 65K .
Our requirement is to support 6000 concurrent requests at minimum. Is there any configuration that can be tweaked to support this amount of concurrent load ? I do see "olcConcurrency" & "olcThreads" set to 0 and 16 respectively. Should we be tweaking these to support the required load, if yes what should be the values for those ? Please assist. Thank you for your help !
Thanks for your response. It’s Connection Timeout. Some additional info - i just went ahead and for sake of trying updated the olcthread value to 1750 and post that was able to get 1650 concurrent connections working concurrently consistently, but anything beyond that again randomly threw connection timeout errors.
Message sent from a mobile device, please excuse typos and brevity.
On Sep 13, 2021, at 9:28 PM, Howard Chu hyc@symas.com wrote:
shekhar.shrinivasan@gmail.com wrote:
Hi,
Was load testing against openldap (slapd-meta) where we were trying to submit 1000 concurrent bind requests. This was done using the JMeter load testing tool. In every run approximately only 420-430 bind requests are successful and remaining 500+ bind requests fail with "Communication Exception" error. We also tried with 500 bind requests and the result was somewhat same - 420 odd successful and 70-80 failed with communication exception.
What does a "Communication exception" actually mean? Connection refused, connection timed out, what?
We've tested tens of thousands of concurrent connections with slapd. It is unlikely to be the problem here.
We have openldap 2.4.56 installed on a RHEL 8.1 host with 32GB memory and 8CPU. We have configured max open files for the slapd process to be 65K .
Our requirement is to support 6000 concurrent requests at minimum. Is there any configuration that can be tweaked to support this amount of concurrent load ? I do see "olcConcurrency" & "olcThreads" set to 0 and 16 respectively. Should we be tweaking these to support the required load, if yes what should be the values for those ? Please assist. Thank you for your help !
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Shekhar S wrote:
Thanks for your response. It’s Connection Timeout. Some additional info - i just went ahead and for sake of trying updated the olcthread value to 1750 and post that was able to get 1650 concurrent connections working concurrently consistently, but anything beyond that again randomly threw connection timeout errors.
Since you're using back-meta, the limit is most likely on the target LDAP server.
One more thing you can try is to increase olcListenerThreads (no more than the number of CPUs). But since raising olcThreads affects your result, clearly the remote server is keeping slapd's threads tied up waiting for responses.
You can also use back-asyncmeta in OpenLDAP 2.5 for better behavior without needing thousands of slapd threads, but the end result will be the same - it can only answer as fast as the remote server and no faster.
Message sent from a mobile device, please excuse typos and brevity.
On Sep 13, 2021, at 9:28 PM, Howard Chu hyc@symas.com wrote:
shekhar.shrinivasan@gmail.com wrote:
Hi,
Was load testing against openldap (slapd-meta) where we were trying to submit 1000 concurrent bind requests. This was done using the JMeter load testing tool. In every run approximately only 420-430 bind requests are successful and remaining 500+ bind requests fail with "Communication Exception" error. We also tried with 500 bind requests and the result was somewhat same - 420 odd successful and 70-80 failed with communication exception.
What does a "Communication exception" actually mean? Connection refused, connection timed out, what?
We've tested tens of thousands of concurrent connections with slapd. It is unlikely to be the problem here.
We have openldap 2.4.56 installed on a RHEL 8.1 host with 32GB memory and 8CPU. We have configured max open files for the slapd process to be 65K .
Our requirement is to support 6000 concurrent requests at minimum. Is there any configuration that can be tweaked to support this amount of concurrent load ? I do see "olcConcurrency" & "olcThreads" set to 0 and 16 respectively. Should we be tweaking these to support the required load, if yes what should be the values for those ? Please assist. Thank you for your help !
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Thank you for your prompt response and suggestion, Howard. I did try your suggestion and increased the olcListenerThreads (based on number of CPUs) but that did not help. Got the same connection timeout errors when I tried to go beyond 1650 concurrent requests.
Although, I did see something and wanted to run by you - In the "libraries/libldap_r/tpool.c" code file I see that the max threads was defined to "LDAP_MAXTHR 1024". So does that mean the limit on the number of threads for slapd process is 1024 ?
After coming across this setting we changed the number of threads in our load test tool to 1024 for each node and ran that script 100 times in a loop and we were able to successfully complete 204,800 bind requests across 2 nodes in 40 minutes.
If we want to bump up the max thread number further to say 4096 can we do that by just updating the "LDAP_MAXTHR" accordingly ? Will it affect something else ? Thank you for your help as always !
shekhar.shrinivasan@gmail.com wrote:
Thank you for your prompt response and suggestion, Howard. I did try your suggestion and increased the olcListenerThreads (based on number of CPUs) but that did not help. Got the same connection timeout errors when I tried to go beyond 1650 concurrent requests.
Although, I did see something and wanted to run by you - In the "libraries/libldap_r/tpool.c" code file I see that the max threads was defined to "LDAP_MAXTHR 1024". So does that mean the limit on the number of threads for slapd process is 1024 ?
Yes
After coming across this setting we changed the number of threads in our load test tool to 1024 for each node and ran that script 100 times in a loop and we were able to successfully complete 204,800 bind requests across 2 nodes in 40 minutes.
If we want to bump up the max thread number further to say 4096 can we do that by just updating the "LDAP_MAXTHR" accordingly ?
Yes
Will it affect something else ? Thank you for your help as always !
The larger value will use more memory and may cause a slight slowdown in looking up thread-local variables. Probably not significant.
shekhar.shrinivasan@gmail.com schrieb am 14.09.2021 um 11:56 in Nachricht
20210914095659.5262.94107@hypatia.openldap.org:
Thank you for your prompt response and suggestion, Howard. I did try your suggestion and increased the olcListenerThreads (based on number of CPUs) but that did not help. Got the same connection timeout errors when I tried to go beyond 1650 concurrent requests.
Although, I did see something and wanted to run by you - In the "libraries/libldap_r/tpool.c" code file I see that the max threads was defined to "LDAP_MAXTHR 1024". So does that mean the limit on the number of threads for slapd process is 1024 ?
After coming across this setting we changed the number of threads in our load test tool to 1024 for each node and ran that script 100 times in a loop and we were able to successfully complete 204,800 bind requests across 2 nodes in 40 minutes.
If we want to bump up the max thread number further to say 4096 can we do that by just updating the "LDAP_MAXTHR" accordingly ? Will it affect something else ? Thank you for your help as always !
Some years ago I had developed two versions of a test program on a 32-bit laptop; one was using processes, the other was using threads. Amazingly I could run more processes than threads, because I ran out of virtual address space (each thread allocates a stack). At that time reducing the default thread stack size to the actual needs fixed the problem. Today with 64 bits such a problem (running out of virtual address space) should be no issue. But still there might be oher limits you can hit.
Regards, Ulrich
openldap-technical@openldap.org
-
Howard Chu -
Shekhar S -
shekhar.shrinivasan＠gmail.com
-
Ulrich Windl