I have an ou with 3.2M users. Doing a simple search of 1 attribute with a scope of 1 and a base of that flat ou is taking 6.2 Seconds. In a replica database, I have attempted to remove all other indexes but the attribute I am searching for and it still is taking over 6 seconds. Is that to be expected?
Apr 11 10:37:49 slapd[25081]: conn=1000 op=20 SRCH base="ou=FlatOU" scope=1 deref=3 filter="(attr=login102)" Apr 11 10:37:49 slapd[25081]: conn=1000 op=20 SRCH attr=objectClass Apr 11 10:37:55 slapd[25081]: conn=1000 op=20 SEARCH RESULT tag=101 err=0 qtime=0.000025 etime=6.267130 nentries=1 text=
Only Idex: olcDbIndex: attr pres,eq
After settting the index, I stopped ldap and ran slapindex
CPU:
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 8 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz Stepping: 0 CPU MHz: 2992.968 BogoMIPS: 5985.93 Hypervisor vendor: VMware Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 25344K NUMA node0 CPU(s): 0-7 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd rsb_ctxsw ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec arat pku ospke md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities
24 GB of RAM with 10 GB Free
Thanks for any suggestions:
Bradley Gill
of 1 attribute with a scope of 1 and a base of that flat ou is taking 6.2 seconds. Is that to be expected?
Hey Brad,
I have found that the response time on a flat branch relates directly and proportionally to nentries in that branch. And I am sure that this is the case with other vendor LDAPs, too, FWIW (inc AD and Azure-DS).
Chris
Thanks Chris, But does that mean that I am to expect a 6.6 second search time for 3.2M entries. While that seems like a large number, I think LDAP should handle it faster than that. That isn’t usable. I am hoping I am missing a tuning parameter somewhere.
Brad
From: Christopher Paul chris.paul@rexconsulting.net Sent: Tuesday, April 11, 2023 3:51 PM To: openldap-technical@openldap.org; Bradley T Gill bgill@aep.com Subject: [EXTERNAL] RE: Slow Search?
of 1 attribute with a scope of 1 and a base of that flat ou is taking 6. 2 seconds. Is that to be expected? Hey Brad, I have found that the response time on a flat branch relates directly and proportionally to nentries in that branch. And
of 1 attribute with a scope of 1 and a base of that flat ou is taking 6.2 seconds. Is that to be expected?
Hey Brad,
I have found that the response time on a flat branch relates directly and proportionally to nentries in that branch. And I am sure that this is the case with other vendor LDAPs, too, FWIW (inc AD and Azure-DS).
Chris
Maybe the answer is to create ou’s for the beginning characters and assign them accordingly then have the the system programmatically set the base when searching? But it’s searching on an attribute an not a DN. This is a short term solution that doesn’t much flexibility though.
From: Bradley T Gill bgill@aep.com Sent: Tuesday, April 11, 2023 3:58 PM To: Christopher Paul chris.paul@rexconsulting.net; openldap-technical@openldap.org Subject: [EXTERNAL] RE: Slow Search?
Thanks Chris, But does that mean that I am to expect a 6. 6 second search time for 3. 2M entries. While that seems like a large number, I think LDAP should handle it faster than that. That isn’t usable. I am hoping I am missing a tuning parameter
Thanks Chris, But does that mean that I am to expect a 6.6 second search time for 3.2M entries. While that seems like a large number, I think LDAP should handle it faster than that. That isn’t usable. I am hoping I am missing a tuning parameter somewhere.
Brad
From: Christopher Paul <chris.paul@rexconsulting.netmailto:chris.paul@rexconsulting.net> Sent: Tuesday, April 11, 2023 3:51 PM To: openldap-technical@openldap.orgmailto:openldap-technical@openldap.org; Bradley T Gill <bgill@aep.commailto:bgill@aep.com> Subject: [EXTERNAL] RE: Slow Search?
of 1 attribute with a scope of 1 and a base of that flat ou is taking 6. 2 seconds. Is that to be expected? Hey Brad, I have found that the response time on a flat branch relates directly and proportionally to nentries in that branch. And
of 1 attribute with a scope of 1 and a base of that flat ou is taking 6.2 seconds. Is that to be expected?
Hey Brad,
I have found that the response time on a flat branch relates directly and proportionally to nentries in that branch. And I am sure that this is the case with other vendor LDAPs, too, FWIW (inc AD and Azure-DS).
Chris
--On Tuesday, April 11, 2023 3:56 PM +0000 Bradley T Gill bgill@aep.com wrote:
I have an ou with 3.2M users. Doing a simple search of 1 attribute with a scope of 1 and a base of that flat ou is taking 6.2 Seconds. In a replica database, I have attempted to remove all other indexes but the attribute I am searching for and it still is taking over 6 seconds. Is that to be expected?
Apr 11 10:37:49 slapd[25081]: conn=1000 op=20 SRCH base="ou=FlatOU" scope=1 deref=3 filter="(attr=login102)"
Apr 11 10:37:49 slapd[25081]: conn=1000 op=20 SRCH attr=objectClass
Apr 11 10:37:55 slapd[25081]: conn=1000 op=20 SEARCH RESULT tag=101 err=0 qtime=0.000025 etime=6.267130 nentries=1 text=
That shows 6 seconds to return a single entry. Most likely this was after slapd was freshly started and the database not yet in memory. That's not a valid way to measure the response time.
After you start slapd, you'll want to do a query across the entire DIT to ensure it's loaded, and *then* start testing how long it takes to get a response.
Also, I suggest 1.1 instead of "objectClass", i.e. something like:
ldapsearch ... -b "root of DIT" -s sub "(objectClass=*)" 1.1
I'd also note it is mandatory to index objectClass eq, so if you haven't done that you've failed the first operational requirement. I'd also note that "pres" indices are almost always not desired, see the current documentation for a discussion on it. Additionally, you've failed to note what version of OpenLDAP you're using.
--Quanah
Quanah, Thanks for the response! I added ObjectClass eq and changed Attr to Attr eq (removed pres) and my search time is 10000 faster! Etime is now 0.00066
Thanks!
Bradley Gill
From: Quanah Gibson-Mount quanah@fast-mail.org Sent: Tuesday, April 11, 2023 5:05 PM To: Bradley T Gill bgill@aep.com; openldap-technical@openldap.org Subject: [EXTERNAL] Re: Slow Search?
--On Tuesday, April 11, 2023 3: 56 PM +0000 Bradley T Gill <bgill@ aep. com> wrote: > > > I have an ou with 3. 2M users. Doing a simple search of 1 attribute with > a scope of 1 and a base of that flat ou is taking 6. 2 Seconds.
--On Tuesday, April 11, 2023 3:56 PM +0000 Bradley T Gill <bgill@aep.commailto:bgill@aep.com>
wrote:
I have an ou with 3.2M users. Doing a simple search of 1 attribute with
a scope of 1 and a base of that flat ou is taking 6.2 Seconds. In a
replica database, I have attempted to remove all other indexes but the
attribute I am searching for and it still is taking over 6 seconds. Is
that to be expected?
Apr 11 10:37:49 slapd[25081]: conn=1000 op=20 SRCH base="ou=FlatOU"
scope=1 deref=3 filter="(attr=login102)"
Apr 11 10:37:49 slapd[25081]: conn=1000 op=20 SRCH attr=objectClass
Apr 11 10:37:55 slapd[25081]: conn=1000 op=20 SEARCH RESULT tag=101 err=0
qtime=0.000025 etime=6.267130 nentries=1 text=
That shows 6 seconds to return a single entry. Most likely this was after
slapd was freshly started and the database not yet in memory. That's not a
valid way to measure the response time.
After you start slapd, you'll want to do a query across the entire DIT to
ensure it's loaded, and *then* start testing how long it takes to get a
response.
Also, I suggest 1.1 instead of "objectClass", i.e. something like:
ldapsearch ... -b "root of DIT" -s sub "(objectClass=*)" 1.1
I'd also note it is mandatory to index objectClass eq, so if you haven't
done that you've failed the first operational requirement. I'd also note
that "pres" indices are almost always not desired, see the current
documentation for a discussion on it. Additionally, you've failed to note
what version of OpenLDAP you're using.
--Quanah
openldap-technical@openldap.org
-
Bradley T Gill -
Christopher Paul -
Quanah Gibson-Mount