Thank you for your quick response.

If idlexp is the accepted solution then I'd like to understand how to choose correct value for idlexp? 

I have quickly tested with most values. With my quick tests I could not find any significant impact on performance. Setting idlexp to maximum 31 did however cause slapd to crash with segmentation fault on my system. 

Thanks,
Petteri
From: Quanah Gibson-Mount <quanah@symas.com>
Sent: Monday, August 16, 2021 23:19
To: Petteri Stenius <Petteri.Stenius@ubisecure.com>; openldap-technical@openldap.org <openldap-technical@openldap.org>
Subject: Re: Index seems to return wrong amount of candidate causing really poor search performance
 


--On Monday, August 16, 2021 9:07 PM +0000 petteri.stenius@ubisecure.com
wrote:

> Hello,
>
> We have experienced a similar issue. Unfortunately we too have a
> confidential dataset we cannot share.
>
> Our dataset started slowing down at about 1.7m entries. With
> slapcat/slapadd tools I can reliably reproduce the issue:
>
> - first I add about 1.7m entries with slapadd to an empty mdb database
> - I test with a search operation that returns a single entry and it
> performs as expected, about 0.06s - next I add a single entry with slapadd
> - after this the exactly same search operation that returns a single
> entry slows down significantly, about 1.37s
>
> With the new idlexp parameter set to for example value 17 this slow down
> issue no longer happens with our dataset.
>
> I don't understand idlexp parameter well enough. My fear is that idlexp
> tuning is not actually fixing this issue, instead the issue is simply
> postponed.

If setting idlexp fixed it, then the issue was that one additional entry
caused the index to collapse to a range with the default idlexp value,
which is why changing the setting had an effect since it would stop it from
being a range at the higher idlexp value.

I added documentation as to what the idlexp command does to the admin guide
for OpenLDAP 2.5.6.  You may want to read it, it applies to OpenLDAP 2.4 as
well.

<https://www.openldap.org/doc/admin25/slapdconf2.html#MDB%20Database%20Directives>

Section 5.2.6.1

--Quanah


--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>