2:03 p.m.
Hi,
Considering the following assumptions;
- OpenLDAP version 2.4.51
- attributes objectClass and abc are indexed based on equality
- the EQUALITY of attribute abc is based on distinguishedNameMatch
- The database contains roughly 2 million entries
- 2 entries have defined the attribute abc with a dn value cn=foo,dc=bar and
objectClass=someClass
- 2 entries have defined the attribute abc with a dn value cn=bar,dc=baz and
objectClass=someClass
Now, the issue started with really slow search performance using objectClass=someClass
& abc=cn=foo,dc=bar as filter criteria. Debugging a while seems to indicate that the
objectClass filter returns roughly 2 million entries as candidates. Now, one would expect
that the second filter would return only the 2 potential candidates from the abc index, or
a subset of the whole database but this is not the case. The second filter also returns
nearly the whole database entries as potential candidates and causes really slow query
performance. Interestingly, this only occurs when attribute abc has value cn=foo,dc=bar,
but for some reason for the entry having attribute abc with value cn=bar,dc=baz the query
returns immediately. In both cases, the actual entries matching the search return
immediately but for the problematic search
"(&(objectClass=someClass)(abc=cn=foo,dc=bar))", the completion of the
search takes a long time (around 15 seconds to be precise).
The issue started suddenly and wasn't a degradation of query performance over time.
Few things I have tried
- Rebuilt the whole database again
- Reindex the existing database again
- Testing with bdb and mdb as backends
- Increased cache sizes for bdb to hold the whole database in cache
- For bdb adjust the page size of the indexes according to suggestion by db_tuner
- Change the order of the filters
None of these made any difference. At the moment, there does not seem to be any good
options to try. Any ideas or help would be greatly appreciated!
4:18 a.m.
chrichardso27(a)gmail.com wrote:
Hi,
Considering the following assumptions;
- OpenLDAP version 2.4.51
- attributes objectClass and abc are indexed based on equality
- the EQUALITY of attribute abc is based on distinguishedNameMatch
- The database contains roughly 2 million entries
- 2 entries have defined the attribute abc with a dn value cn=foo,dc=bar and
objectClass=someClass
- 2 entries have defined the attribute abc with a dn value cn=bar,dc=baz and
objectClass=someClass
Now, the issue started with really slow search performance using objectClass=someClass
& abc=cn=foo,dc=bar as filter criteria. Debugging a while seems to indicate that the
objectClass filter returns roughly 2 million entries as candidates.
If the objectclass filter returns so many candidates, it's not really doing much good.
What is the
result using only the (abc=DN) filter?
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
8:48 a.m.
Howard Chu wrote:
chrichardso27(a)gmail.com wrote:
> Hi,
>
> Considering the following assumptions;
>
> - OpenLDAP version 2.4.51
> - attributes objectClass and abc are indexed based on equality
> - the EQUALITY of attribute abc is based on distinguishedNameMatch
> - The database contains roughly 2 million entries
> - 2 entries have defined the attribute abc with a dn value cn=foo,dc=bar and
> objectClass=someClass
> - 2 entries have defined the attribute abc with a dn value cn=bar,dc=baz and
> objectClass=someClass
>
> Now, the issue started with really slow search performance using
objectClass=someClass
> & abc=cn=foo,dc=bar as filter criteria. Debugging a while seems to indicate that
the
> objectClass filter returns roughly 2 million entries as candidates.
If the objectclass filter returns so many candidates, it's not really doing much
good.
What is the
result using only the (abc=DN) filter?
Search with filter "(abc=cn=foo,dc=bar)" returns close to the amount of entries
in the database (2M) as candidates, and is somewhat equally slow than
"(&(objectClass=someClass)(abc=cn=foo,dc=bar))", around 15 seconds.
However, search with filter "(abc=cn=bar,dc=baz)" returns a subset of the index
of abc and performs reasonably fast (1-2 seconds).
This is rather weird and I have no clue on what might be causing the issue.
2:26 p.m.
--On Sunday, August 30, 2020 4:48 PM +0000 chrichardso27(a)gmail.com wrote:
Search with filter "(abc=cn=foo,dc=bar)" returns close to
the amount of
entries in the database (2M) as candidates, and is somewhat equally slow
than "(&(objectClass=someClass)(abc=cn=foo,dc=bar))", around 15 seconds.
However, search with filter "(abc=cn=bar,dc=baz)" returns a subset of the
index of abc and performs reasonably fast (1-2 seconds).
This is rather weird and I have no clue on what might be causing the
issue.
Can you provide:
a) The actual queries (filter, base, etc)
b) The schema definition of the attribute in question.
Thanks,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
3:18 a.m.
Quanah Gibson-Mount wrote:
--On Sunday, August 30, 2020 4:48 PM +0000 chrichardso27(a)gmail.com
wrote:
> Search with filter "(abc=cn=foo,dc=bar)" returns close to
> the amount of
> entries in the database (2M) as candidates, and is somewhat equally slow
> than "(&(objectClass=someClass)(abc=cn=foo,dc=bar))", around 15
seconds.
>
> However, search with filter "(abc=cn=bar,dc=baz)" returns a subset of
the
> index of abc and performs reasonably fast (1-2 seconds).
>
> This is rather weird and I have no clue on what might be causing the
> issue.
Can you provide:
a) The actual queries (filter, base, etc)
Using ldapsearch
ldapsearch \
-L \
-x \
-H ldap://localhost:1389 \
-D "cn=admin,cn=directory,dc=example,dc=org" \
-w secret \
-b cn=directory,dc=example,dc=org \
"(&(objectClass=someClass)(abc=cn=foo,dc=bar,cn=server,ou=system,cn=directory,dc=example,dc=org))"
Note that in this specific case, Rebuilding a database from scratch with a different data
does not reveal to problem. I have a specific LDIF exported using slapcat (which I sadly
cannot share) with which I can reproduce the problem each time.
One thing that I have not mentioned previously is that we have a two node system where to
other node is a failover node but is continuously kept up to date using syncrepl.
I'm just doing random guesses here as I'm running out of ideas:
- Could this be a syncrepl issue
- Could there be a bug how OpenLDAP decides the candidates when processing the respective
index
Anyways, again, any ideas would be greatly appreciated! Few more things I have tried
during these few days:
- Restore backup prior the incident (works of course but not a solution at this point as
the root cause remains to be unknown)
- With bdb, use linearindex (didn't fix the issue)
- With bdb, set dbpagesize for the index files (didn't succeed as the slapadd runs out
of memory for some reason...)
b) The schema definition of the attribute in question.
attributetype (
1.3.6.1.4.1.14761.1.26
NAME 'abc'
DESC 'A description'
EQUALITY distinguishedNameMatch
SYNTAX '1.3.6.1.4.1.1466.115.121.1.12' )
>
> Thanks,
> Quanah
>
> --
>
> Quanah Gibson-Mount
> Product Architect
> Symas Corporation
> Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
> <http://www.symas.com>
11:40 a.m.
chrichardso27(a)gmail.com wrote:
Quanah Gibson-Mount wrote:
> --On Sunday, August 30, 2020 4:48 PM +0000 chrichardso27(a)gmail.com wrote:
>
>> Search with filter "(abc=cn=foo,dc=bar)" returns close to
>> the amount of
>> entries in the database (2M) as candidates, and is somewhat equally slow
>> than "(&(objectClass=someClass)(abc=cn=foo,dc=bar))", around 15
seconds.
>>
>> However, search with filter "(abc=cn=bar,dc=baz)" returns a subset of
the
>> index of abc and performs reasonably fast (1-2 seconds).
>>
>> This is rather weird and I have no clue on what might be causing the
>> issue.
> Can you provide:
>
> a) The actual queries (filter, base, etc)
Have you looked at the slapd debug output with FILTER output enabled?
Using ldapsearch
ldapsearch \
-L \
-x \
-H ldap://localhost:1389 \
-D "cn=admin,cn=directory,dc=example,dc=org" \
-w secret \
-b cn=directory,dc=example,dc=org \
"(&(objectClass=someClass)(abc=cn=foo,dc=bar,cn=server,ou=system,cn=directory,dc=example,dc=org))"
Note that in this specific case, Rebuilding a database from scratch with a different data
does not reveal to problem. I have a specific LDIF exported using slapcat (which I sadly
cannot share) with which I can reproduce the problem each time.
One thing that I have not mentioned previously is that we have a two node system where to
other node is a failover node but is continuously kept up to date using syncrepl.
I'm just doing random guesses here as I'm running out of ideas:
- Could this be a syncrepl issue
- Could there be a bug how OpenLDAP decides the candidates when processing the respective
index
Anyways, again, any ideas would be greatly appreciated! Few more things I have tried
during these few days:
- Restore backup prior the incident (works of course but not a solution at this point as
the root cause remains to be unknown)
- With bdb, use linearindex (didn't fix the issue)
- With bdb, set dbpagesize for the index files (didn't succeed as the slapadd runs
out of memory for some reason...)
>
> b) The schema definition of the attribute in question.
attributetype (
1.3.6.1.4.1.14761.1.26
NAME 'abc'
DESC 'A description'
EQUALITY distinguishedNameMatch
SYNTAX '1.3.6.1.4.1.1466.115.121.1.12' )
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
11:17 a.m.
Howard Chu wrote:
chrichardso27(a)gmail.com wrote:
> Quanah Gibson-Mount wrote:
> > --On Sunday, August 30, 2020 4:48 PM +0000 chrichardso27(a)gmail.com wrote:
> >
> >> Search with filter "(abc=cn=foo,dc=bar)" returns close to
> >> the amount of
> >> entries in the database (2M) as candidates, and is somewhat equally slow
> >> than "(&(objectClass=someClass)(abc=cn=foo,dc=bar))", around
15
> seconds.
> >>
> >> However, search with filter "(abc=cn=bar,dc=baz)" returns a
subset of
> the
> >> index of abc and performs reasonably fast (1-2 seconds).
> >>
> >> This is rather weird and I have no clue on what might be causing the
> >> issue.
> > Can you provide:
> >
> > a) The actual queries (filter, base, etc)
Have you looked at the slapd debug output with FILTER output enabled?
Debug log using mdb (which experiences the same poor usage of index)
begin get_filter
AND
begin get_filter_list
begin get_filter
EQUALITY
end get_filter 0
begin get_filter
EQUALITY
end get_filter 0
end get_filter_list
end get_filter 0
=> mdb_filter_candidates
OR
=> mdb_list_candidates 0xa1
=> mdb_filter_candidates
EQUALITY
<= mdb_filter_candidates: id=0 first=0 last=0
=> mdb_filter_candidates
AND
=> mdb_list_candidates 0xa0
=> mdb_filter_candidates
EQUALITY
<= mdb_filter_candidates: id=-1 first=2 last=2051388
=> mdb_filter_candidates
EQUALITY
<= mdb_filter_candidates: id=-1 first=39 last=2051956
<= mdb_list_candidates: id=-1 first=39 last=2051388
<= mdb_filter_candidates: id=-1 first=39 last=2051388
<= mdb_list_candidates: id=-1 first=39 last=2051388
<= mdb_filter_candidates: id=-1 first=39 last=2051388
=> test_filter
AND
=> test_filter_and
=> test_filter
EQUALITY
<= test_filter 6
=> test_filter
EQUALITY
<= test_filter 6
<= test_filter_and 6
<= test_filter 6
=> test_filter
AND
=> test_filter_and
=> test_filter
EQUALITY
<= test_filter 5
<= test_filter_and 5
<= test_filter 5
=> test_filter
AND
=> test_filter_and
=> test_filter
EQUALITY
And this repeats on and on.
Furthermore, with -d 1 and -d 32 I get:
=> mdb_equality_candidates (objectClass)
=> key_read
<= mdb_index_read 2051387 candidates
<= mdb_equality_candidates: id=-1, first=2, last=2051388
<= mdb_filter_candidates: id=-1 first=2 last=2051388
=> mdb_filter_candidates
EQUALITY
=> mdb_equality_candidates (abc)
=> key_read
<= mdb_index_read 2051918 candidates
Again, almost all entries in the database are considered as candidates for the problematic
attribute. However, this is not the case as there are only few entries having that
attribute specified.
> >
> > Using ldapsearch
> >
> > ldapsearch \
> > -L \
> > -x \
> > -H ldap://localhost:1389 \
> > -D "cn=admin,cn=directory,dc=example,dc=org" \
> > -w secret \
> > -b cn=directory,dc=example,dc=org \
> >
> >
"(&(objectClass=someClass)(abc=cn=foo,dc=bar,cn=server,ou=system,cn=directory,dc=example,dc=org))"
> >
> > Note that in this specific case, Rebuilding a database from scratch with a
different data
> > does not reveal to problem. I have a specific LDIF exported using slapcat (which
I sadly
> > cannot share) with which I can reproduce the problem each time.
> >
> > One thing that I have not mentioned previously is that we have a two node
system where to
> > other node is a failover node but is continuously kept up to date using
syncrepl.
> >
> > I'm just doing random guesses here as I'm running out of ideas:
> >
> > - Could this be a syncrepl issue
> > - Could there be a bug how OpenLDAP decides the candidates when processing the
respective
> > index
> >
> > Anyways, again, any ideas would be greatly appreciated! Few more things I have
tried
> > during these few days:
> >
> > - Restore backup prior the incident (works of course but not a solution at this
point as
> > the root cause remains to be unknown)
> > - With bdb, use linearindex (didn't fix the issue)
> > - With bdb, set dbpagesize for the index files (didn't succeed as the
slapadd runs
> > out of memory for some reason...)
> >
> > >
> > > b) The schema definition of the attribute in question.
> >
> > attributetype (
> > 1.3.6.1.4.1.14761.1.26
> > NAME 'abc'
> > DESC 'A description'
> > EQUALITY distinguishedNameMatch
> > SYNTAX '1.3.6.1.4.1.1466.115.121.1.12' )
noon
--On Tuesday, September 1, 2020 11:18 AM +0000 chrichardso27(a)gmail.com
wrote:
"(&(objectClass=someClass)(abc=cn=foo,dc=bar,cn=server,ou=system,cn=direc
tory,dc=example,dc=org))"
foo and bar are the actual real values? dc=example,dc=org is your domain?
>
> b) The schema definition of the attribute in question.
attributetype (
1.3.6.1.4.1.14761.1.26
NAME 'abc'
DESC 'A description'
EQUALITY distinguishedNameMatch
SYNTAX '1.3.6.1.4.1.1466.115.121.1.12' )
abc is your actual attribute name?
And don't waste time with back-bdb, it's deprecated and removed from 2.5+.
Regards,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
11:08 a.m.
Quanah Gibson-Mount wrote:
--On Tuesday, September 1, 2020 11:18 AM +0000
chrichardso27(a)gmail.com
wrote:
>
> "(&(objectClass=someClass)(abc=cn=foo,dc=bar,cn=server,ou=system,cn=direc
> tory,dc=example,dc=org))"
foo and bar are the actual real values? dc=example,dc=org is your domain?
The depth is the same, the values are different from the actual that we use. I cannot
share the actual values without disclosing internal details.
> >
> > b) The schema definition of the attribute in question.
>
> attributetype (
> 1.3.6.1.4.1.14761.1.26
> NAME 'abc'
> DESC 'A description'
> EQUALITY distinguishedNameMatch
> SYNTAX '1.3.6.1.4.1.1466.115.121.1.12' )
abc is your actual attribute name?
Same as above, I cannot disclose the actual values but to give it a bit more meaning, lets
say that the attribute name could be for example "userSystemDN".
And don't waste time with back-bdb, it's deprecated and removed from 2.5+.
This is known. However, this happens to be a critical issue at the moment with BDB.
Futhermore, I can reproduce the problem with MDB but the search filter execution time
drops to around 2 seconds. Still, in a high volume system, this is not acceptable. Other
values in this same attribute perform in tens of milliseconds. The one specific value
(which I cannot disclose here) is extremely slow compared to similar data, just with
different DN value.
>
> Regards,
> Quanah
>
> --
>
> Quanah Gibson-Mount
> Product Architect
> Symas Corporation
> Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
> <http://www.symas.com>
11:37 a.m.
chrichardso27(a)gmail.com wrote:
Quanah Gibson-Mount wrote:
> --On Tuesday, September 1, 2020 11:18 AM +0000 chrichardso27(a)gmail.com
> wrote:
>
>>
>>
"(&(objectClass=someClass)(abc=cn=foo,dc=bar,cn=server,ou=system,cn=direc
>> tory,dc=example,dc=org))"
>
> foo and bar are the actual real values? dc=example,dc=org is your domain?
The depth is the same, the values are different from the actual that we use. I cannot
share the actual values without disclosing internal details.
There's not a lot we can do without being able to reproduce the issue and see
what's going on.
You could try starting with a mostly empty DB, adding just the offending value, looking at
the
filter debug output for a lookup on that, and see if it looks sensible first.
is the attribute abc=foo actually unique, or are there multiple occurrences of it in the
DB?
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
11:44 a.m.
--On Wednesday, September 2, 2020 8:37 PM +0100 Howard Chu <hyc(a)symas.com>
wrote:
> The depth is the same, the values are different from the actual
that we
> use. I cannot share the actual values without disclosing internal
> details.
There's not a lot we can do without being able to reproduce the issue and
see what's going on.
You could try starting with a mostly empty DB, adding just the offending
value, looking at the filter debug output for a lookup on that, and see
if it looks sensible first.
is the attribute abc=foo actually unique, or are there multiple
occurrences of it in the DB?
I would optionally note that Symas does provide support contracts for
OpenLDAP and has NDAs if this is a mission critical problem for your
company. As Howard notes, without a reproduction case, it's virtually
impossible for us to help here.
One option may be to see if you can reproduce the same problem after
renaming the attribute to something you can disclose, combined with values
that are the same exact length but are simillarly obfuscated to see if you
can reproduce the issue that way. If so, you should be able to share those
details.
Looking at the open bugs, I wonder if it is
<https://bugs.openldap.org/show_bug.cgi?id=7743>
Regards,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
1:36 p.m.
First I apologize for posting a non-technical question / follow up to this list, however I
can speak for the high value add that having official support for OpenLDAP that the Symas
team offers. Like most folks on this list, we have a great deal of in house expertise on
many software stacks including directory services in general. With that said, the Symas
team members are great to work with, literally write the software that we are asking
questions about and as a result have many years of experience and expertise that is easy
to reach for. IMO the support that the Symas org puts forward is a model that vendors
should be striving for.
Best,
Aaron
-----Original Message-----
From: Quanah Gibson-Mount <quanah(a)symas.com>
Sent: Wednesday, September 2, 2020 2:45 PM
To: Howard Chu <hyc(a)symas.com>; chrichardso27(a)gmail.com;
openldap-technical(a)openldap.org
Subject: Re: Index seems to return wrong amount of candidate causing really poor search
performance
Warning: This email is from outside the company. Be careful clicking links or
attachments.
--On Wednesday, September 2, 2020 8:37 PM +0100 Howard Chu <hyc(a)symas.com>
wrote:
> The depth is the same, the values are different from the actual
that
> we use. I cannot share the actual values without disclosing internal
> details.
There's not a lot we can do without being able to reproduce the issue
and see what's going on.
You could try starting with a mostly empty DB, adding just the
offending value, looking at the filter debug output for a lookup on
that, and see if it looks sensible first.
is the attribute abc=foo actually unique, or are there multiple
occurrences of it in the DB?
I would optionally note that Symas does provide support contracts for OpenLDAP and has
NDAs if this is a mission critical problem for your company. As Howard notes, without a
reproduction case, it's virtually impossible for us to help here.
One option may be to see if you can reproduce the same problem after renaming the
attribute to something you can disclose, combined with values that are the same exact
length but are simillarly obfuscated to see if you can reproduce the issue that way. If
so, you should be able to share those details.
Looking at the open bugs, I wonder if it is
<https://urldefense.com/v3/__https://bugs.openldap.org/show_bug.cgi?id=774...
>
Regards,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<https://urldefense.com/v3/__http://www.symas.com__;!!PIfy-9xbww!UGx4_veAj...
>
----------------------------------------------------------------------
The information contained in this message may be privileged, confidential and protected
from disclosure. If the reader of this message is not the intended recipient, or an
employee or agent responsible for delivering this message to the intended recipient, you
are hereby notified that any dissemination, distribution or copying of this communication
is strictly prohibited. If you have received this communication in error, please notify
your representative immediately and delete this message from your computer. Thank you.
5:07 a.m.
Coming back to this after few months as the problem remained as unsolved for us. We may
have found a way forward.
The feature added into OpenLDAP 2.5 to control the size of IDLs seems to address our
performance issue. The specific change is this:
https://bugs.openldap.org/show_bug.cgi?id=8977.
As per the documentation
5.2.6.1. olcBkMdbIdlExp <exponent>
Specify a power of 2 for the maximum size of an index slot. The default is 16, yielding a
maximum slot size of 2^16 or 65536. The specified value must be in the range of 16-31.
This setting helps with the case where certain search filters are slow to return results
due to an index slot having collapsed to a range value. This occurs when the number of
candidate entries that match the filter for the index slot exceed the configured slot
size.
If this setting is decreased on a server with existing MDB databases, each db will
immediately need its indices to be rebuilt while slapd is offline with the "slapindex
-q -t" command.
If this setting is increased on a server with existing MDB databases, each db will need
its indices rebuilt to take advantage of the change for indices that have already been
converted to ranges.
As we migrated to MDB and defined bigger size for the index slot we can now see that the
index does not collapse anymore and performance remains stable.
We will still conduct more testing to verify that this is definitely the case.
1:07 p.m.
Hello,
We have experienced a similar issue. Unfortunately we too have a confidential dataset we
cannot share.
Our dataset started slowing down at about 1.7m entries. With slapcat/slapadd tools I can
reliably reproduce the issue:
- first I add about 1.7m entries with slapadd to an empty mdb database
- I test with a search operation that returns a single entry and it performs as expected,
about 0.06s
- next I add a single entry with slapadd
- after this the exactly same search operation that returns a single entry slows down
significantly, about 1.37s
With the new idlexp parameter set to for example value 17 this slow down issue no longer
happens with our dataset.
I don't understand idlexp parameter well enough. My fear is that idlexp tuning is not
actually fixing this issue, instead the issue is simply postponed.
Any ideas or help would be greatly appreciated!
Thanks,
Petteri
1:19 p.m.
--On Monday, August 16, 2021 9:07 PM +0000 petteri.stenius(a)ubisecure.com
wrote:
Hello,
We have experienced a similar issue. Unfortunately we too have a
confidential dataset we cannot share.
Our dataset started slowing down at about 1.7m entries. With
slapcat/slapadd tools I can reliably reproduce the issue:
- first I add about 1.7m entries with slapadd to an empty mdb database
- I test with a search operation that returns a single entry and it
performs as expected, about 0.06s - next I add a single entry with slapadd
- after this the exactly same search operation that returns a single
entry slows down significantly, about 1.37s
With the new idlexp parameter set to for example value 17 this slow down
issue no longer happens with our dataset.
I don't understand idlexp parameter well enough. My fear is that idlexp
tuning is not actually fixing this issue, instead the issue is simply
postponed.
If setting idlexp fixed it, then the issue was that one additional entry
caused the index to collapse to a range with the default idlexp value,
which is why changing the setting had an effect since it would stop it from
being a range at the higher idlexp value.
I added documentation as to what the idlexp command does to the admin guide
for OpenLDAP 2.5.6. You may want to read it, it applies to OpenLDAP 2.4 as
well.
<https://www.openldap.org/doc/admin25/slapdconf2.html#MDB%20Database%20Dir...
Section 5.2.6.1
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
2 p.m.
Thank you for your quick response.
If idlexp is the accepted solution then I'd like to understand how to choose correct
value for idlexp?
I have quickly tested with most values. With my quick tests I could not find any
significant impact on performance. Setting idlexp to maximum 31 did however cause slapd to
crash with segmentation fault on my system.
Thanks,
Petteri
________________________________
From: Quanah Gibson-Mount <quanah(a)symas.com>
Sent: Monday, August 16, 2021 23:19
To: Petteri Stenius <Petteri.Stenius(a)ubisecure.com>; openldap-technical(a)openldap.org
<openldap-technical(a)openldap.org>
Subject: Re: Index seems to return wrong amount of candidate causing really poor search
performance
--On Monday, August 16, 2021 9:07 PM +0000 petteri.stenius(a)ubisecure.com
wrote:
Hello,
We have experienced a similar issue. Unfortunately we too have a
confidential dataset we cannot share.
Our dataset started slowing down at about 1.7m entries. With
slapcat/slapadd tools I can reliably reproduce the issue:
- first I add about 1.7m entries with slapadd to an empty mdb database
- I test with a search operation that returns a single entry and it
performs as expected, about 0.06s - next I add a single entry with slapadd
- after this the exactly same search operation that returns a single
entry slows down significantly, about 1.37s
With the new idlexp parameter set to for example value 17 this slow down
issue no longer happens with our dataset.
I don't understand idlexp parameter well enough. My fear is that idlexp
tuning is not actually fixing this issue, instead the issue is simply
postponed.
If setting idlexp fixed it, then the issue was that one additional entry
caused the index to collapse to a range with the default idlexp value,
which is why changing the setting had an effect since it would stop it from
being a range at the higher idlexp value.
I added documentation as to what the idlexp command does to the admin guide
for OpenLDAP 2.5.6. You may want to read it, it applies to OpenLDAP 2.4 as
well.
<https://www.openldap.org/doc/admin25/slapdconf2.html#MDB%20Database%20Dir...
Section 5.2.6.1
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
2:20 p.m.
--On Monday, August 16, 2021 10:00 PM +0000 Petteri Stenius
<Petteri.Stenius(a)ubisecure.com> wrote:
Thank you for your quick response.
If idlexp is the accepted solution then I'd like to understand how to
choose correct value for idlexp?
I have quickly tested with most values. With my quick tests I could not
find any significant impact on performance. Setting idlexp to maximum 31
did however cause slapd to crash with segmentation fault on my system.
The appropriate value for any environment is entirely dependent on that
environment's indexing and attribute value distribution for those indexed
attributes. You generally want the minimum value for idlexp that allows
searches to function without performance problems. Increasing the idlexp
size increases slapd memory usage. Keep in mind that every increase in the
idlexp value increases the index slot range by a power of 2. The largest
value I've ever needed for a massively large db with wide value
distributions was 22.
"acl" level logging of slapd will show the candidate result sets for a
query. For example on a db of approximately 6.4 million entries:
5f9496ad mdb_idl_fetch_key: [d393480d]
5f9496ad <= mdb_index_read 6463387 candidates
5f9496ad => key_read
5f9496ad mdb_idl_fetch_key: [9d5c550b]
5f9496ad <= mdb_index_read 6463379 candidates
5f9496ad => key_read
5f9496ad mdb_idl_fetch_key: [37ef4d9a]
5f9496ad <= mdb_index_read 6463379 candidates
5f9496ad => key_read
5f9496ad mdb_idl_fetch_key: [e5e52605]
5f9496ad <= mdb_index_read 42313 candidates
5f9496ad => key_read
5f9496ad mdb_idl_fetch_key: [8990480b]
5f9496ad <= mdb_index_read 24146 candidates
5f9496ad => key_read
5f9496ad mdb_idl_fetch_key: [7de12f45]
5f9496ad <= mdb_index_read 3217 candidates
In the above set (which was for a substring search) there were 6 different
breakdowns of the substring queried from the index. 3 of them were ranges
(6463387 candidates). 3 of them were limited matches (42313, 24146, 3217).
The results are pulled from the smallest candidate set (3217) which is a
quick lookup.
Alternately, here's what the same search looked like for a slightly larger
substring value that was always slow:
5f9495b3 mdb_idl_fetch_key: [6a48da6f]
5f9495b3 <= mdb_index_read 6463377 candidates
5f9495b3 => key_read
5f9495b3 mdb_idl_fetch_key: [de44adf5]
5f9495b3 <= mdb_index_read 6463399 candidates
5f9495b3 => key_read
5f9495b3 mdb_idl_fetch_key: [174b5285]
5f9495b3 <= mdb_index_read 6463385 candidates
5f9495b3 => key_read
5f9495b3 mdb_idl_fetch_key: [9e5c550b]
5f9495b3 <= mdb_index_read 6462721 candidates
5f9495b3 => key_read
5f9495b3 mdb_idl_fetch_key: [9b5c550b]
5f9495b3 <= mdb_index_read 6463280 candidates
5f9495b3 => key_read
5f9495b3 mdb_idl_fetch_key: [d393480d]
5f9495b3 <= mdb_index_read 6463387 candidates
You can see the 6 candidate sets are essentially "all entries" which is why
it is so slow. After adjusting the idlexp value to 17, I got instead:
5f94a42e mdb_idl_fetch_key: [6a48da6f]
5f94a42e <= mdb_index_read 906885 candidates
5f94a42e => key_read
5f94a42e mdb_idl_fetch_key: [de44adf5]
5f94a42e <= mdb_index_read 6463399 candidates
5f94a42e => key_read
5f94a42e mdb_idl_fetch_key: [174b5285]
5f94a42e <= mdb_index_read 415219 candidates
5f94a42e => key_read
5f94a42e mdb_idl_fetch_key: [9e5c550b]
5f94a42e <= mdb_index_read 99550 candidates
5f94a42e => key_read
5f94a42e mdb_idl_fetch_key: [9b5c550b]
5f94a42e <= mdb_index_read 293028 candidates
5f94a42e => key_read
5f94a42e mdb_idl_fetch_key: [d393480d]
5f94a42e <= mdb_index_read 6463387 candidates
So now we can see there are 4 candidate sets that are smaller than "all
entries":
906,885
415,219
99,550
293,028
Regards,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
3:08 a.m.
Thank you. This information is very helpful.
I think you need logging level "trace" (not "acl") to see the
"mdb_index_read NN candidates" log entries.
Thanks,
Petteri
________________________________
From: Quanah Gibson-Mount <quanah(a)symas.com>
Sent: Tuesday, August 17, 2021 0:20
To: Petteri Stenius <Petteri.Stenius(a)ubisecure.com>; openldap-technical(a)openldap.org
<openldap-technical(a)openldap.org>
Subject: Re: Index seems to return wrong amount of candidate causing really poor search
performance
--On Monday, August 16, 2021 10:00 PM +0000 Petteri Stenius
<Petteri.Stenius(a)ubisecure.com> wrote:
Thank you for your quick response.
If idlexp is the accepted solution then I'd like to understand how to
choose correct value for idlexp?
I have quickly tested with most values. With my quick tests I could not
find any significant impact on performance. Setting idlexp to maximum 31
did however cause slapd to crash with segmentation fault on my system.
The appropriate value for any environment is entirely dependent on that
environment's indexing and attribute value distribution for those indexed
attributes. You generally want the minimum value for idlexp that allows
searches to function without performance problems. Increasing the idlexp
size increases slapd memory usage. Keep in mind that every increase in the
idlexp value increases the index slot range by a power of 2. The largest
value I've ever needed for a massively large db with wide value
distributions was 22.
"acl" level logging of slapd will show the candidate result sets for a
query. For example on a db of approximately 6.4 million entries:
5f9496ad mdb_idl_fetch_key: [d393480d]
5f9496ad <= mdb_index_read 6463387 candidates
5f9496ad => key_read
5f9496ad mdb_idl_fetch_key: [9d5c550b]
5f9496ad <= mdb_index_read 6463379 candidates
5f9496ad => key_read
5f9496ad mdb_idl_fetch_key: [37ef4d9a]
5f9496ad <= mdb_index_read 6463379 candidates
5f9496ad => key_read
5f9496ad mdb_idl_fetch_key: [e5e52605]
5f9496ad <= mdb_index_read 42313 candidates
5f9496ad => key_read
5f9496ad mdb_idl_fetch_key: [8990480b]
5f9496ad <= mdb_index_read 24146 candidates
5f9496ad => key_read
5f9496ad mdb_idl_fetch_key: [7de12f45]
5f9496ad <= mdb_index_read 3217 candidates
In the above set (which was for a substring search) there were 6 different
breakdowns of the substring queried from the index. 3 of them were ranges
(6463387 candidates). 3 of them were limited matches (42313, 24146, 3217).
The results are pulled from the smallest candidate set (3217) which is a
quick lookup.
Alternately, here's what the same search looked like for a slightly larger
substring value that was always slow:
5f9495b3 mdb_idl_fetch_key: [6a48da6f]
5f9495b3 <= mdb_index_read 6463377 candidates
5f9495b3 => key_read
5f9495b3 mdb_idl_fetch_key: [de44adf5]
5f9495b3 <= mdb_index_read 6463399 candidates
5f9495b3 => key_read
5f9495b3 mdb_idl_fetch_key: [174b5285]
5f9495b3 <= mdb_index_read 6463385 candidates
5f9495b3 => key_read
5f9495b3 mdb_idl_fetch_key: [9e5c550b]
5f9495b3 <= mdb_index_read 6462721 candidates
5f9495b3 => key_read
5f9495b3 mdb_idl_fetch_key: [9b5c550b]
5f9495b3 <= mdb_index_read 6463280 candidates
5f9495b3 => key_read
5f9495b3 mdb_idl_fetch_key: [d393480d]
5f9495b3 <= mdb_index_read 6463387 candidates
You can see the 6 candidate sets are essentially "all entries" which is why
it is so slow. After adjusting the idlexp value to 17, I got instead:
5f94a42e mdb_idl_fetch_key: [6a48da6f]
5f94a42e <= mdb_index_read 906885 candidates
5f94a42e => key_read
5f94a42e mdb_idl_fetch_key: [de44adf5]
5f94a42e <= mdb_index_read 6463399 candidates
5f94a42e => key_read
5f94a42e mdb_idl_fetch_key: [174b5285]
5f94a42e <= mdb_index_read 415219 candidates
5f94a42e => key_read
5f94a42e mdb_idl_fetch_key: [9e5c550b]
5f94a42e <= mdb_index_read 99550 candidates
5f94a42e => key_read
5f94a42e mdb_idl_fetch_key: [9b5c550b]
5f94a42e <= mdb_index_read 293028 candidates
5f94a42e => key_read
5f94a42e mdb_idl_fetch_key: [d393480d]
5f94a42e <= mdb_index_read 6463387 candidates
So now we can see there are 4 candidate sets that are smaller than "all
entries":
906,885
415,219
99,550
293,028
Regards,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
8:28 a.m.
--On Tuesday, August 17, 2021 11:08 AM +0000 Petteri Stenius
<Petteri.Stenius(a)ubisecure.com> wrote:
Thank you. This information is very helpful.
I think you need logging level "trace" (not "acl") to see the
"mdb_index_read NN candidates" log entries.
Yes, you are correct, trace indeed. :)
Regards,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
656
days inactive
1010
days old
openldap-technical@openldap.org
18 comments
6 participants
participants (6)
-
Bliss, Aaron -
chrichardso27＠gmail.com -
Howard Chu -
Petteri Stenius -
petteri.stenius＠ubisecure.com
-
Quanah Gibson-Mount