--On Monday, August 16, 2021 10:00 PM +0000 Petteri Stenius
<Petteri.Stenius@ubisecure.com> wrote:

>
> Thank you for your quick response.
>
>
> If idlexp is the accepted solution then I'd like to understand how to
> choose correct value for idlexp?
>
>
> I have quickly tested with most values. With my quick tests I could not
> find any significant impact on performance. Setting idlexp to maximum 31
> did however cause slapd to crash with segmentation fault on my system.

The appropriate value for any environment is entirely dependent on that
environment's indexing and attribute value distribution for those indexed
attributes.  You generally want the minimum value for idlexp that allows
searches to function without performance problems.  Increasing the idlexp
size increases slapd memory usage.  Keep in mind that every increase in the
idlexp value increases the index slot range by a power of 2.  The largest
value I've ever needed for a massively large db with wide value
distributions was 22.

"acl" level logging of slapd will show the candidate result sets for a
query.  For example on a db of approximately 6.4 million entries:

5f9496ad mdb_idl_fetch_key: [d393480d]
5f9496ad <= mdb_index_read 6463387 candidates
5f9496ad => key_read
5f9496ad mdb_idl_fetch_key: [9d5c550b]
5f9496ad <= mdb_index_read 6463379 candidates
5f9496ad => key_read
5f9496ad mdb_idl_fetch_key: [37ef4d9a]
5f9496ad <= mdb_index_read 6463379 candidates
5f9496ad => key_read
5f9496ad mdb_idl_fetch_key: [e5e52605]
5f9496ad <= mdb_index_read 42313 candidates
5f9496ad => key_read
5f9496ad mdb_idl_fetch_key: [8990480b]
5f9496ad <= mdb_index_read 24146 candidates
5f9496ad => key_read
5f9496ad mdb_idl_fetch_key: [7de12f45]
5f9496ad <= mdb_index_read 3217 candidates

In the above set (which was for a substring search) there were 6 different
breakdowns of the substring queried from the index.  3 of them were ranges
(6463387 candidates).  3 of them were limited matches (42313, 24146, 3217).
The results are pulled from the smallest candidate set (3217) which is a
quick lookup.


Alternately, here's what the same search looked like for a slightly larger
substring value that was always slow:

5f9495b3 mdb_idl_fetch_key: [6a48da6f]
5f9495b3 <= mdb_index_read 6463377 candidates
5f9495b3 => key_read
5f9495b3 mdb_idl_fetch_key: [de44adf5]
5f9495b3 <= mdb_index_read 6463399 candidates
5f9495b3 => key_read
5f9495b3 mdb_idl_fetch_key: [174b5285]
5f9495b3 <= mdb_index_read 6463385 candidates
5f9495b3 => key_read
5f9495b3 mdb_idl_fetch_key: [9e5c550b]
5f9495b3 <= mdb_index_read 6462721 candidates
5f9495b3 => key_read
5f9495b3 mdb_idl_fetch_key: [9b5c550b]
5f9495b3 <= mdb_index_read 6463280 candidates
5f9495b3 => key_read
5f9495b3 mdb_idl_fetch_key: [d393480d]
5f9495b3 <= mdb_index_read 6463387 candidates

You can see the 6 candidate sets are essentially "all entries" which is why
it is so slow.  After adjusting the idlexp value to 17, I got instead:

5f94a42e mdb_idl_fetch_key: [6a48da6f]
5f94a42e <= mdb_index_read 906885 candidates
5f94a42e => key_read
5f94a42e mdb_idl_fetch_key: [de44adf5]
5f94a42e <= mdb_index_read 6463399 candidates
5f94a42e => key_read
5f94a42e mdb_idl_fetch_key: [174b5285]
5f94a42e <= mdb_index_read 415219 candidates
5f94a42e => key_read
5f94a42e mdb_idl_fetch_key: [9e5c550b]
5f94a42e <= mdb_index_read 99550 candidates
5f94a42e => key_read
5f94a42e mdb_idl_fetch_key: [9b5c550b]
5f94a42e <= mdb_index_read 293028 candidates
5f94a42e => key_read
5f94a42e mdb_idl_fetch_key: [d393480d]
5f94a42e <= mdb_index_read 6463387 candidates

So now we can see there are 4 candidate sets that are smaller than "all
entries":

906,885
415,219
99,550
293,028


Regards,
Quanah

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>