Soisik Froger wrote:
On 01/09/2021 17:17, Quanah Gibson-Mount wrote:
>
>
> --On Wednesday, September 1, 2021 2:07 PM +0100 Mark Cairney
> <Mark.Cairney(a)ed.ac.uk> wrote:
>
>> Hi,
>>
>> I've been out the LDAP loop for a bit but the recent discussion of the
>> memberof overlay on 2.5 piqued my curiosity. Having upgraded a Dev box,
>> removed the memberof elements from the database and replaced the
>> memberof overlay with dynlist the queries appear to work as expected but
>> are both a) slow and b) heavily CPU-intensive on the LDAP server.
>
>
> As an aside, I would note that you appear to be indexing "pres"
> unnecessarily. Please read
> <
https://www.openldap.org/doc/admin25/tuning.html#Presence%20indexing>
>
> If the group object is large you may be having slow searches due to indices
> being collapsed to a range. You would need to run the search with trace
> logging to determine if that's the case as was recently discussed on the
> list.
>
> Regards,
> Quanah
>
Hi,
We are also observing very long query time involving memberOf when using dynlist
migrating a 2.4 OpenLDAP docker install to 2.5.11. We have followed community
recommendation and replaced the deprecated memberOf overlay with dynlist to calculate
memberOf attributes, and removed the memberof elements from the data
before importing it.
Using dynlist, we have noticed an important drop of performance in queries involving the
memberOf attributes on (relatively) large database (about 36000 users
and 9000 groups dispatched in 150 branches) compared to 2.4: Queries on memberOf
attributes that used to be processed in 0.200 seconds now takes about 6-7
seconds. In the example below, the search base is a branch with a relatively small number
of users (about 350) and the targeted group contains 260 users:
time ldapsearch -x -h 172.17.0.2 -D "cn=admin,dc=domain,dc=com" -w
"secret" -b "ou=people,dc=branch,dc=domain,dc=com"
"(memberof=cn=dev,ou=groups,dc=branch,dc=domain,dc=com)" uid
real 0m6,358s
user 0m0,024s
sys 0m0,020s
If the group used in memberOf has 1200 members, the query takes over 30 seconds. These
results are obtained using an OpenLDAP running in a simple docker
container on a host with 8CPU/16GB RAM.
When using the memberOf overlay instead of dynlist to calculate memberOf, the performance
are back to what is used to be in 2.4 (< 0.200 second for the above
ldapsearch).
We believe our config to be very standard: member attribute is indexed and dynlist is
loaded and configured as below
dn: olcOverlay={3}dynlist,olcDatabase={1}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcDynListConfig
olcOverlay: {3}dynlist
olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames
Are this performance issues an expected side-effect of switching to dynlist - as the
memberOf attributes are now dynamically calculated while the memberOf
overlay used to writes these attributes - or something is more likely wrong with our
install or could be tuned to improve these performances?
That's just the cost of dynamically calculating these attributes, yes.
It's possible we can make some other optimizations to the overlay to reduce the amount
of work in calculating memberOf if you're only
using it in a filter and not returning all of its values. Looks like it could be a fairly
significant refactoring to achieve that though.
Some organizations have applications that relies on memberOf queries to be fast, and
expects to keep these performances when upgrading to 2.5. From what we
observed, we can only advise them to stick with the legacy memberOf overlay if they need
these queries to stay fast, or rethink their approach to avoid querying
memberOf on large databases.
Makes sense.
Regards,
--
-- Howard Chu
CTO, Symas Corp.
http://www.symas.com
Director, Highland Sun
http://highlandsun.com/hyc/
Chief Architect, OpenLDAP
http://www.openldap.org/project/