Hi,
Considering the following assumptions;
- OpenLDAP version 2.4.51 - attributes objectClass and abc are indexed based on equality - the EQUALITY of attribute abc is based on distinguishedNameMatch - The database contains roughly 2 million entries - 2 entries have defined the attribute abc with a dn value cn=foo,dc=bar and objectClass=someClass - 2 entries have defined the attribute abc with a dn value cn=bar,dc=baz and objectClass=someClass
Now, the issue started with really slow search performance using objectClass=someClass & abc=cn=foo,dc=bar as filter criteria. Debugging a while seems to indicate that the objectClass filter returns roughly 2 million entries as candidates. Now, one would expect that the second filter would return only the 2 potential candidates from the abc index, or a subset of the whole database but this is not the case. The second filter also returns nearly the whole database entries as potential candidates and causes really slow query performance. Interestingly, this only occurs when attribute abc has value cn=foo,dc=bar, but for some reason for the entry having attribute abc with value cn=bar,dc=baz the query returns immediately. In both cases, the actual entries matching the search return immediately but for the problematic search "(&(objectClass=someClass)(abc=cn=foo,dc=bar))", the completion of the search takes a long time (around 15 seconds to be precise).
The issue started suddenly and wasn't a degradation of query performance over time.
Few things I have tried
- Rebuilt the whole database again - Reindex the existing database again - Testing with bdb and mdb as backends - Increased cache sizes for bdb to hold the whole database in cache - For bdb adjust the page size of the indexes according to suggestion by db_tuner - Change the order of the filters
None of these made any difference. At the moment, there does not seem to be any good options to try. Any ideas or help would be greatly appreciated!
chrichardso27@gmail.com wrote:
Hi,
Considering the following assumptions;
- OpenLDAP version 2.4.51
- attributes objectClass and abc are indexed based on equality
- the EQUALITY of attribute abc is based on distinguishedNameMatch
- The database contains roughly 2 million entries
- 2 entries have defined the attribute abc with a dn value cn=foo,dc=bar and objectClass=someClass
- 2 entries have defined the attribute abc with a dn value cn=bar,dc=baz and objectClass=someClass
Now, the issue started with really slow search performance using objectClass=someClass & abc=cn=foo,dc=bar as filter criteria. Debugging a while seems to indicate that the objectClass filter returns roughly 2 million entries as candidates.
If the objectclass filter returns so many candidates, it's not really doing much good. What is the result using only the (abc=DN) filter?
Howard Chu wrote:
chrichardso27(a)gmail.com wrote:
Hi,
Considering the following assumptions;
- OpenLDAP version 2.4.51
- attributes objectClass and abc are indexed based on equality
- the EQUALITY of attribute abc is based on distinguishedNameMatch
- The database contains roughly 2 million entries
- 2 entries have defined the attribute abc with a dn value cn=foo,dc=bar and
objectClass=someClass
- 2 entries have defined the attribute abc with a dn value cn=bar,dc=baz and
objectClass=someClass
Now, the issue started with really slow search performance using objectClass=someClass & abc=cn=foo,dc=bar as filter criteria. Debugging a while seems to indicate that the objectClass filter returns roughly 2 million entries as candidates.
If the objectclass filter returns so many candidates, it's not really doing much good. What is the result using only the (abc=DN) filter?
Search with filter "(abc=cn=foo,dc=bar)" returns close to the amount of entries in the database (2M) as candidates, and is somewhat equally slow than "(&(objectClass=someClass)(abc=cn=foo,dc=bar))", around 15 seconds.
However, search with filter "(abc=cn=bar,dc=baz)" returns a subset of the index of abc and performs reasonably fast (1-2 seconds).
This is rather weird and I have no clue on what might be causing the issue.
--On Sunday, August 30, 2020 4:48 PM +0000 chrichardso27@gmail.com wrote:
Search with filter "(abc=cn=foo,dc=bar)" returns close to the amount of entries in the database (2M) as candidates, and is somewhat equally slow than "(&(objectClass=someClass)(abc=cn=foo,dc=bar))", around 15 seconds.
However, search with filter "(abc=cn=bar,dc=baz)" returns a subset of the index of abc and performs reasonably fast (1-2 seconds).
This is rather weird and I have no clue on what might be causing the issue.
Can you provide:
a) The actual queries (filter, base, etc)
b) The schema definition of the attribute in question.
Thanks, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Quanah Gibson-Mount wrote:
--On Sunday, August 30, 2020 4:48 PM +0000 chrichardso27(a)gmail.com wrote:
Search with filter "(abc=cn=foo,dc=bar)" returns close to the amount of entries in the database (2M) as candidates, and is somewhat equally slow than "(&(objectClass=someClass)(abc=cn=foo,dc=bar))", around 15 seconds.
However, search with filter "(abc=cn=bar,dc=baz)" returns a subset of the index of abc and performs reasonably fast (1-2 seconds).
This is rather weird and I have no clue on what might be causing the issue.
Can you provide:
a) The actual queries (filter, base, etc)
Using ldapsearch
ldapsearch \ -L \ -x \ -H ldap://localhost:1389 \ -D "cn=admin,cn=directory,dc=example,dc=org" \ -w secret \ -b cn=directory,dc=example,dc=org \ "(&(objectClass=someClass)(abc=cn=foo,dc=bar,cn=server,ou=system,cn=directory,dc=example,dc=org))"
Note that in this specific case, Rebuilding a database from scratch with a different data does not reveal to problem. I have a specific LDIF exported using slapcat (which I sadly cannot share) with which I can reproduce the problem each time.
One thing that I have not mentioned previously is that we have a two node system where to other node is a failover node but is continuously kept up to date using syncrepl.
I'm just doing random guesses here as I'm running out of ideas:
- Could this be a syncrepl issue - Could there be a bug how OpenLDAP decides the candidates when processing the respective index
Anyways, again, any ideas would be greatly appreciated! Few more things I have tried during these few days:
- Restore backup prior the incident (works of course but not a solution at this point as the root cause remains to be unknown) - With bdb, use linearindex (didn't fix the issue) - With bdb, set dbpagesize for the index files (didn't succeed as the slapadd runs out of memory for some reason...)
b) The schema definition of the attribute in question.
attributetype ( 1.3.6.1.4.1.14761.1.26 NAME 'abc' DESC 'A description' EQUALITY distinguishedNameMatch SYNTAX '1.3.6.1.4.1.1466.115.121.1.12' )
Thanks, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
chrichardso27@gmail.com wrote:
Quanah Gibson-Mount wrote:
--On Sunday, August 30, 2020 4:48 PM +0000 chrichardso27(a)gmail.com wrote:
Search with filter "(abc=cn=foo,dc=bar)" returns close to the amount of entries in the database (2M) as candidates, and is somewhat equally slow than "(&(objectClass=someClass)(abc=cn=foo,dc=bar))", around 15 seconds.
However, search with filter "(abc=cn=bar,dc=baz)" returns a subset of the index of abc and performs reasonably fast (1-2 seconds).
This is rather weird and I have no clue on what might be causing the issue.
Can you provide:
a) The actual queries (filter, base, etc)
Have you looked at the slapd debug output with FILTER output enabled?
Using ldapsearch
ldapsearch \ -L \ -x \ -H ldap://localhost:1389 \ -D "cn=admin,cn=directory,dc=example,dc=org" \ -w secret \ -b cn=directory,dc=example,dc=org \ "(&(objectClass=someClass)(abc=cn=foo,dc=bar,cn=server,ou=system,cn=directory,dc=example,dc=org))"
Note that in this specific case, Rebuilding a database from scratch with a different data does not reveal to problem. I have a specific LDIF exported using slapcat (which I sadly cannot share) with which I can reproduce the problem each time.
One thing that I have not mentioned previously is that we have a two node system where to other node is a failover node but is continuously kept up to date using syncrepl.
I'm just doing random guesses here as I'm running out of ideas:
- Could this be a syncrepl issue
- Could there be a bug how OpenLDAP decides the candidates when processing the respective index
Anyways, again, any ideas would be greatly appreciated! Few more things I have tried during these few days:
- Restore backup prior the incident (works of course but not a solution at this point as the root cause remains to be unknown)
- With bdb, use linearindex (didn't fix the issue)
- With bdb, set dbpagesize for the index files (didn't succeed as the slapadd runs out of memory for some reason...)
b) The schema definition of the attribute in question.
attributetype ( 1.3.6.1.4.1.14761.1.26 NAME 'abc' DESC 'A description' EQUALITY distinguishedNameMatch SYNTAX '1.3.6.1.4.1.1466.115.121.1.12' )
Howard Chu wrote:
chrichardso27(a)gmail.com wrote:
Quanah Gibson-Mount wrote:
--On Sunday, August 30, 2020 4:48 PM +0000 chrichardso27(a)gmail.com wrote:
Search with filter "(abc=cn=foo,dc=bar)" returns close to the amount of entries in the database (2M) as candidates, and is somewhat equally slow than "(&(objectClass=someClass)(abc=cn=foo,dc=bar))", around 15
seconds.
However, search with filter "(abc=cn=bar,dc=baz)" returns a subset of
the
index of abc and performs reasonably fast (1-2 seconds).
This is rather weird and I have no clue on what might be causing the issue.
Can you provide:
a) The actual queries (filter, base, etc)
Have you looked at the slapd debug output with FILTER output enabled?
Debug log using mdb (which experiences the same poor usage of index)
begin get_filter AND begin get_filter_list begin get_filter EQUALITY end get_filter 0 begin get_filter EQUALITY end get_filter 0 end get_filter_list end get_filter 0 => mdb_filter_candidates OR => mdb_list_candidates 0xa1 => mdb_filter_candidates EQUALITY <= mdb_filter_candidates: id=0 first=0 last=0 => mdb_filter_candidates AND => mdb_list_candidates 0xa0 => mdb_filter_candidates EQUALITY <= mdb_filter_candidates: id=-1 first=2 last=2051388 => mdb_filter_candidates EQUALITY <= mdb_filter_candidates: id=-1 first=39 last=2051956 <= mdb_list_candidates: id=-1 first=39 last=2051388 <= mdb_filter_candidates: id=-1 first=39 last=2051388 <= mdb_list_candidates: id=-1 first=39 last=2051388 <= mdb_filter_candidates: id=-1 first=39 last=2051388 => test_filter AND => test_filter_and => test_filter EQUALITY <= test_filter 6 => test_filter EQUALITY <= test_filter 6 <= test_filter_and 6 <= test_filter 6 => test_filter AND => test_filter_and => test_filter EQUALITY <= test_filter 5 <= test_filter_and 5 <= test_filter 5 => test_filter AND => test_filter_and => test_filter EQUALITY
And this repeats on and on.
Furthermore, with -d 1 and -d 32 I get:
=> mdb_equality_candidates (objectClass) => key_read <= mdb_index_read 2051387 candidates <= mdb_equality_candidates: id=-1, first=2, last=2051388 <= mdb_filter_candidates: id=-1 first=2 last=2051388 => mdb_filter_candidates EQUALITY => mdb_equality_candidates (abc) => key_read <= mdb_index_read 2051918 candidates
Again, almost all entries in the database are considered as candidates for the problematic attribute. However, this is not the case as there are only few entries having that attribute specified.
Using ldapsearch
ldapsearch \ -L \ -x \ -H ldap://localhost:1389 \ -D "cn=admin,cn=directory,dc=example,dc=org" \ -w secret \ -b cn=directory,dc=example,dc=org \
"(&(objectClass=someClass)(abc=cn=foo,dc=bar,cn=server,ou=system,cn=directory,dc=example,dc=org))"
Note that in this specific case, Rebuilding a database from scratch with a different data does not reveal to problem. I have a specific LDIF exported using slapcat (which I sadly cannot share) with which I can reproduce the problem each time.
One thing that I have not mentioned previously is that we have a two node system where to other node is a failover node but is continuously kept up to date using syncrepl.
I'm just doing random guesses here as I'm running out of ideas:
- Could this be a syncrepl issue
- Could there be a bug how OpenLDAP decides the candidates when processing the respective
index
Anyways, again, any ideas would be greatly appreciated! Few more things I have tried during these few days:
- Restore backup prior the incident (works of course but not a solution at this point as
the root cause remains to be unknown)
- With bdb, use linearindex (didn't fix the issue)
- With bdb, set dbpagesize for the index files (didn't succeed as the slapadd runs
out of memory for some reason...)
b) The schema definition of the attribute in question.
attributetype ( 1.3.6.1.4.1.14761.1.26 NAME 'abc' DESC 'A description' EQUALITY distinguishedNameMatch SYNTAX '1.3.6.1.4.1.1466.115.121.1.12' )
--On Tuesday, September 1, 2020 11:18 AM +0000 chrichardso27@gmail.com wrote:
"(&(objectClass=someClass)(abc=cn=foo,dc=bar,cn=server,ou=system,cn=direc tory,dc=example,dc=org))"
foo and bar are the actual real values? dc=example,dc=org is your domain?
b) The schema definition of the attribute in question.
attributetype ( 1.3.6.1.4.1.14761.1.26 NAME 'abc' DESC 'A description' EQUALITY distinguishedNameMatch SYNTAX '1.3.6.1.4.1.1466.115.121.1.12' )
abc is your actual attribute name?
And don't waste time with back-bdb, it's deprecated and removed from 2.5+.
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Quanah Gibson-Mount wrote:
--On Tuesday, September 1, 2020 11:18 AM +0000 chrichardso27(a)gmail.com wrote:
"(&(objectClass=someClass)(abc=cn=foo,dc=bar,cn=server,ou=system,cn=direc tory,dc=example,dc=org))"
foo and bar are the actual real values? dc=example,dc=org is your domain?
The depth is the same, the values are different from the actual that we use. I cannot share the actual values without disclosing internal details.
b) The schema definition of the attribute in question.
attributetype ( 1.3.6.1.4.1.14761.1.26 NAME 'abc' DESC 'A description' EQUALITY distinguishedNameMatch SYNTAX '1.3.6.1.4.1.1466.115.121.1.12' )
abc is your actual attribute name?
Same as above, I cannot disclose the actual values but to give it a bit more meaning, lets say that the attribute name could be for example "userSystemDN".
And don't waste time with back-bdb, it's deprecated and removed from 2.5+.
This is known. However, this happens to be a critical issue at the moment with BDB. Futhermore, I can reproduce the problem with MDB but the search filter execution time drops to around 2 seconds. Still, in a high volume system, this is not acceptable. Other values in this same attribute perform in tens of milliseconds. The one specific value (which I cannot disclose here) is extremely slow compared to similar data, just with different DN value.
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
chrichardso27@gmail.com wrote:
Quanah Gibson-Mount wrote:
--On Tuesday, September 1, 2020 11:18 AM +0000 chrichardso27(a)gmail.com wrote:
"(&(objectClass=someClass)(abc=cn=foo,dc=bar,cn=server,ou=system,cn=direc tory,dc=example,dc=org))"
foo and bar are the actual real values? dc=example,dc=org is your domain?
The depth is the same, the values are different from the actual that we use. I cannot share the actual values without disclosing internal details.
There's not a lot we can do without being able to reproduce the issue and see what's going on.
You could try starting with a mostly empty DB, adding just the offending value, looking at the filter debug output for a lookup on that, and see if it looks sensible first.
is the attribute abc=foo actually unique, or are there multiple occurrences of it in the DB?
--On Wednesday, September 2, 2020 8:37 PM +0100 Howard Chu hyc@symas.com wrote:
The depth is the same, the values are different from the actual that we use. I cannot share the actual values without disclosing internal details.
There's not a lot we can do without being able to reproduce the issue and see what's going on.
You could try starting with a mostly empty DB, adding just the offending value, looking at the filter debug output for a lookup on that, and see if it looks sensible first.
is the attribute abc=foo actually unique, or are there multiple occurrences of it in the DB?
I would optionally note that Symas does provide support contracts for OpenLDAP and has NDAs if this is a mission critical problem for your company. As Howard notes, without a reproduction case, it's virtually impossible for us to help here.
One option may be to see if you can reproduce the same problem after renaming the attribute to something you can disclose, combined with values that are the same exact length but are simillarly obfuscated to see if you can reproduce the issue that way. If so, you should be able to share those details.
Looking at the open bugs, I wonder if it is https://bugs.openldap.org/show_bug.cgi?id=7743
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
First I apologize for posting a non-technical question / follow up to this list, however I can speak for the high value add that having official support for OpenLDAP that the Symas team offers. Like most folks on this list, we have a great deal of in house expertise on many software stacks including directory services in general. With that said, the Symas team members are great to work with, literally write the software that we are asking questions about and as a result have many years of experience and expertise that is easy to reach for. IMO the support that the Symas org puts forward is a model that vendors should be striving for.
Best, Aaron
-----Original Message----- From: Quanah Gibson-Mount quanah@symas.com Sent: Wednesday, September 2, 2020 2:45 PM To: Howard Chu hyc@symas.com; chrichardso27@gmail.com; openldap-technical@openldap.org Subject: Re: Index seems to return wrong amount of candidate causing really poor search performance
Warning: This email is from outside the company. Be careful clicking links or attachments.
--On Wednesday, September 2, 2020 8:37 PM +0100 Howard Chu hyc@symas.com wrote:
The depth is the same, the values are different from the actual that we use. I cannot share the actual values without disclosing internal details.
There's not a lot we can do without being able to reproduce the issue and see what's going on.
You could try starting with a mostly empty DB, adding just the offending value, looking at the filter debug output for a lookup on that, and see if it looks sensible first.
is the attribute abc=foo actually unique, or are there multiple occurrences of it in the DB?
I would optionally note that Symas does provide support contracts for OpenLDAP and has NDAs if this is a mission critical problem for your company. As Howard notes, without a reproduction case, it's virtually impossible for us to help here.
One option may be to see if you can reproduce the same problem after renaming the attribute to something you can disclose, combined with values that are the same exact length but are simillarly obfuscated to see if you can reproduce the issue that way. If so, you should be able to share those details.
Looking at the open bugs, I wonder if it is <https://urldefense.com/v3/__https://bugs.openldap.org/show_bug.cgi?id=7743__... >
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <https://urldefense.com/v3/__http://www.symas.com__;!!PIfy-9xbww!UGx4_veAjIOc... >
---------------------------------------------------------------------- The information contained in this message may be privileged, confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify your representative immediately and delete this message from your computer. Thank you.
Coming back to this after few months as the problem remained as unsolved for us. We may have found a way forward.
The feature added into OpenLDAP 2.5 to control the size of IDLs seems to address our performance issue. The specific change is this: https://bugs.openldap.org/show_bug.cgi?id=8977.
As per the documentation
5.2.6.1. olcBkMdbIdlExp <exponent>
Specify a power of 2 for the maximum size of an index slot. The default is 16, yielding a maximum slot size of 2^16 or 65536. The specified value must be in the range of 16-31.
This setting helps with the case where certain search filters are slow to return results due to an index slot having collapsed to a range value. This occurs when the number of candidate entries that match the filter for the index slot exceed the configured slot size.
If this setting is decreased on a server with existing MDB databases, each db will immediately need its indices to be rebuilt while slapd is offline with the "slapindex -q -t" command.
If this setting is increased on a server with existing MDB databases, each db will need its indices rebuilt to take advantage of the change for indices that have already been converted to ranges.
As we migrated to MDB and defined bigger size for the index slot we can now see that the index does not collapse anymore and performance remains stable.
We will still conduct more testing to verify that this is definitely the case.
Hello,
We have experienced a similar issue. Unfortunately we too have a confidential dataset we cannot share.
Our dataset started slowing down at about 1.7m entries. With slapcat/slapadd tools I can reliably reproduce the issue:
- first I add about 1.7m entries with slapadd to an empty mdb database - I test with a search operation that returns a single entry and it performs as expected, about 0.06s - next I add a single entry with slapadd - after this the exactly same search operation that returns a single entry slows down significantly, about 1.37s
With the new idlexp parameter set to for example value 17 this slow down issue no longer happens with our dataset.
I don't understand idlexp parameter well enough. My fear is that idlexp tuning is not actually fixing this issue, instead the issue is simply postponed.
Any ideas or help would be greatly appreciated!
Thanks, Petteri
--On Monday, August 16, 2021 9:07 PM +0000 petteri.stenius@ubisecure.com wrote:
Hello,
We have experienced a similar issue. Unfortunately we too have a confidential dataset we cannot share.
Our dataset started slowing down at about 1.7m entries. With slapcat/slapadd tools I can reliably reproduce the issue:
- first I add about 1.7m entries with slapadd to an empty mdb database
- I test with a search operation that returns a single entry and it
performs as expected, about 0.06s - next I add a single entry with slapadd
- after this the exactly same search operation that returns a single
entry slows down significantly, about 1.37s
With the new idlexp parameter set to for example value 17 this slow down issue no longer happens with our dataset.
I don't understand idlexp parameter well enough. My fear is that idlexp tuning is not actually fixing this issue, instead the issue is simply postponed.
If setting idlexp fixed it, then the issue was that one additional entry caused the index to collapse to a range with the default idlexp value, which is why changing the setting had an effect since it would stop it from being a range at the higher idlexp value.
I added documentation as to what the idlexp command does to the admin guide for OpenLDAP 2.5.6. You may want to read it, it applies to OpenLDAP 2.4 as well.
https://www.openldap.org/doc/admin25/slapdconf2.html#MDB%20Database%20Directives
Section 5.2.6.1
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Thank you for your quick response.
If idlexp is the accepted solution then I'd like to understand how to choose correct value for idlexp?
I have quickly tested with most values. With my quick tests I could not find any significant impact on performance. Setting idlexp to maximum 31 did however cause slapd to crash with segmentation fault on my system.
Thanks, Petteri ________________________________ From: Quanah Gibson-Mount quanah@symas.com Sent: Monday, August 16, 2021 23:19 To: Petteri Stenius Petteri.Stenius@ubisecure.com; openldap-technical@openldap.org openldap-technical@openldap.org Subject: Re: Index seems to return wrong amount of candidate causing really poor search performance
--On Monday, August 16, 2021 9:07 PM +0000 petteri.stenius@ubisecure.com wrote:
Hello,
We have experienced a similar issue. Unfortunately we too have a confidential dataset we cannot share.
Our dataset started slowing down at about 1.7m entries. With slapcat/slapadd tools I can reliably reproduce the issue:
- first I add about 1.7m entries with slapadd to an empty mdb database
- I test with a search operation that returns a single entry and it
performs as expected, about 0.06s - next I add a single entry with slapadd
- after this the exactly same search operation that returns a single
entry slows down significantly, about 1.37s
With the new idlexp parameter set to for example value 17 this slow down issue no longer happens with our dataset.
I don't understand idlexp parameter well enough. My fear is that idlexp tuning is not actually fixing this issue, instead the issue is simply postponed.
If setting idlexp fixed it, then the issue was that one additional entry caused the index to collapse to a range with the default idlexp value, which is why changing the setting had an effect since it would stop it from being a range at the higher idlexp value.
I added documentation as to what the idlexp command does to the admin guide for OpenLDAP 2.5.6. You may want to read it, it applies to OpenLDAP 2.4 as well.
https://www.openldap.org/doc/admin25/slapdconf2.html#MDB%20Database%20Directives
Section 5.2.6.1
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
--On Monday, August 16, 2021 10:00 PM +0000 Petteri Stenius Petteri.Stenius@ubisecure.com wrote:
Thank you for your quick response.
If idlexp is the accepted solution then I'd like to understand how to choose correct value for idlexp?
I have quickly tested with most values. With my quick tests I could not find any significant impact on performance. Setting idlexp to maximum 31 did however cause slapd to crash with segmentation fault on my system.
The appropriate value for any environment is entirely dependent on that environment's indexing and attribute value distribution for those indexed attributes. You generally want the minimum value for idlexp that allows searches to function without performance problems. Increasing the idlexp size increases slapd memory usage. Keep in mind that every increase in the idlexp value increases the index slot range by a power of 2. The largest value I've ever needed for a massively large db with wide value distributions was 22.
"acl" level logging of slapd will show the candidate result sets for a query. For example on a db of approximately 6.4 million entries:
5f9496ad mdb_idl_fetch_key: [d393480d] 5f9496ad <= mdb_index_read 6463387 candidates 5f9496ad => key_read 5f9496ad mdb_idl_fetch_key: [9d5c550b] 5f9496ad <= mdb_index_read 6463379 candidates 5f9496ad => key_read 5f9496ad mdb_idl_fetch_key: [37ef4d9a] 5f9496ad <= mdb_index_read 6463379 candidates 5f9496ad => key_read 5f9496ad mdb_idl_fetch_key: [e5e52605] 5f9496ad <= mdb_index_read 42313 candidates 5f9496ad => key_read 5f9496ad mdb_idl_fetch_key: [8990480b] 5f9496ad <= mdb_index_read 24146 candidates 5f9496ad => key_read 5f9496ad mdb_idl_fetch_key: [7de12f45] 5f9496ad <= mdb_index_read 3217 candidates
In the above set (which was for a substring search) there were 6 different breakdowns of the substring queried from the index. 3 of them were ranges (6463387 candidates). 3 of them were limited matches (42313, 24146, 3217). The results are pulled from the smallest candidate set (3217) which is a quick lookup.
Alternately, here's what the same search looked like for a slightly larger substring value that was always slow:
5f9495b3 mdb_idl_fetch_key: [6a48da6f] 5f9495b3 <= mdb_index_read 6463377 candidates 5f9495b3 => key_read 5f9495b3 mdb_idl_fetch_key: [de44adf5] 5f9495b3 <= mdb_index_read 6463399 candidates 5f9495b3 => key_read 5f9495b3 mdb_idl_fetch_key: [174b5285] 5f9495b3 <= mdb_index_read 6463385 candidates 5f9495b3 => key_read 5f9495b3 mdb_idl_fetch_key: [9e5c550b] 5f9495b3 <= mdb_index_read 6462721 candidates 5f9495b3 => key_read 5f9495b3 mdb_idl_fetch_key: [9b5c550b] 5f9495b3 <= mdb_index_read 6463280 candidates 5f9495b3 => key_read 5f9495b3 mdb_idl_fetch_key: [d393480d] 5f9495b3 <= mdb_index_read 6463387 candidates
You can see the 6 candidate sets are essentially "all entries" which is why it is so slow. After adjusting the idlexp value to 17, I got instead:
5f94a42e mdb_idl_fetch_key: [6a48da6f] 5f94a42e <= mdb_index_read 906885 candidates 5f94a42e => key_read 5f94a42e mdb_idl_fetch_key: [de44adf5] 5f94a42e <= mdb_index_read 6463399 candidates 5f94a42e => key_read 5f94a42e mdb_idl_fetch_key: [174b5285] 5f94a42e <= mdb_index_read 415219 candidates 5f94a42e => key_read 5f94a42e mdb_idl_fetch_key: [9e5c550b] 5f94a42e <= mdb_index_read 99550 candidates 5f94a42e => key_read 5f94a42e mdb_idl_fetch_key: [9b5c550b] 5f94a42e <= mdb_index_read 293028 candidates 5f94a42e => key_read 5f94a42e mdb_idl_fetch_key: [d393480d] 5f94a42e <= mdb_index_read 6463387 candidates
So now we can see there are 4 candidate sets that are smaller than "all entries":
906,885 415,219 99,550 293,028
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Thank you. This information is very helpful.
I think you need logging level "trace" (not "acl") to see the "mdb_index_read NN candidates" log entries.
Thanks, Petteri ________________________________ From: Quanah Gibson-Mount quanah@symas.com Sent: Tuesday, August 17, 2021 0:20 To: Petteri Stenius Petteri.Stenius@ubisecure.com; openldap-technical@openldap.org openldap-technical@openldap.org Subject: Re: Index seems to return wrong amount of candidate causing really poor search performance
--On Monday, August 16, 2021 10:00 PM +0000 Petteri Stenius Petteri.Stenius@ubisecure.com wrote:
Thank you for your quick response.
If idlexp is the accepted solution then I'd like to understand how to choose correct value for idlexp?
I have quickly tested with most values. With my quick tests I could not find any significant impact on performance. Setting idlexp to maximum 31 did however cause slapd to crash with segmentation fault on my system.
The appropriate value for any environment is entirely dependent on that environment's indexing and attribute value distribution for those indexed attributes. You generally want the minimum value for idlexp that allows searches to function without performance problems. Increasing the idlexp size increases slapd memory usage. Keep in mind that every increase in the idlexp value increases the index slot range by a power of 2. The largest value I've ever needed for a massively large db with wide value distributions was 22.
"acl" level logging of slapd will show the candidate result sets for a query. For example on a db of approximately 6.4 million entries:
5f9496ad mdb_idl_fetch_key: [d393480d] 5f9496ad <= mdb_index_read 6463387 candidates 5f9496ad => key_read 5f9496ad mdb_idl_fetch_key: [9d5c550b] 5f9496ad <= mdb_index_read 6463379 candidates 5f9496ad => key_read 5f9496ad mdb_idl_fetch_key: [37ef4d9a] 5f9496ad <= mdb_index_read 6463379 candidates 5f9496ad => key_read 5f9496ad mdb_idl_fetch_key: [e5e52605] 5f9496ad <= mdb_index_read 42313 candidates 5f9496ad => key_read 5f9496ad mdb_idl_fetch_key: [8990480b] 5f9496ad <= mdb_index_read 24146 candidates 5f9496ad => key_read 5f9496ad mdb_idl_fetch_key: [7de12f45] 5f9496ad <= mdb_index_read 3217 candidates
In the above set (which was for a substring search) there were 6 different breakdowns of the substring queried from the index. 3 of them were ranges (6463387 candidates). 3 of them were limited matches (42313, 24146, 3217). The results are pulled from the smallest candidate set (3217) which is a quick lookup.
Alternately, here's what the same search looked like for a slightly larger substring value that was always slow:
5f9495b3 mdb_idl_fetch_key: [6a48da6f] 5f9495b3 <= mdb_index_read 6463377 candidates 5f9495b3 => key_read 5f9495b3 mdb_idl_fetch_key: [de44adf5] 5f9495b3 <= mdb_index_read 6463399 candidates 5f9495b3 => key_read 5f9495b3 mdb_idl_fetch_key: [174b5285] 5f9495b3 <= mdb_index_read 6463385 candidates 5f9495b3 => key_read 5f9495b3 mdb_idl_fetch_key: [9e5c550b] 5f9495b3 <= mdb_index_read 6462721 candidates 5f9495b3 => key_read 5f9495b3 mdb_idl_fetch_key: [9b5c550b] 5f9495b3 <= mdb_index_read 6463280 candidates 5f9495b3 => key_read 5f9495b3 mdb_idl_fetch_key: [d393480d] 5f9495b3 <= mdb_index_read 6463387 candidates
You can see the 6 candidate sets are essentially "all entries" which is why it is so slow. After adjusting the idlexp value to 17, I got instead:
5f94a42e mdb_idl_fetch_key: [6a48da6f] 5f94a42e <= mdb_index_read 906885 candidates 5f94a42e => key_read 5f94a42e mdb_idl_fetch_key: [de44adf5] 5f94a42e <= mdb_index_read 6463399 candidates 5f94a42e => key_read 5f94a42e mdb_idl_fetch_key: [174b5285] 5f94a42e <= mdb_index_read 415219 candidates 5f94a42e => key_read 5f94a42e mdb_idl_fetch_key: [9e5c550b] 5f94a42e <= mdb_index_read 99550 candidates 5f94a42e => key_read 5f94a42e mdb_idl_fetch_key: [9b5c550b] 5f94a42e <= mdb_index_read 293028 candidates 5f94a42e => key_read 5f94a42e mdb_idl_fetch_key: [d393480d] 5f94a42e <= mdb_index_read 6463387 candidates
So now we can see there are 4 candidate sets that are smaller than "all entries":
906,885 415,219 99,550 293,028
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
--On Tuesday, August 17, 2021 11:08 AM +0000 Petteri Stenius Petteri.Stenius@ubisecure.com wrote:
Thank you. This information is very helpful.
I think you need logging level "trace" (not "acl") to see the "mdb_index_read NN candidates" log entries.
Yes, you are correct, trace indeed. :)
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
openldap-technical@openldap.org
-
Bliss, Aaron -
chrichardso27＠gmail.com -
Howard Chu -
Petteri Stenius -
petteri.stenius＠ubisecure.com
-
Quanah Gibson-Mount