Re: dynlist vs memberof performance issues

On 01/09/2021 17:17, Quanah Gibson-Mount wrote: > > > --On Wednesday, September 1, 2021 2:07 PM +0100 Mark Cairney > <Mark.Cairney(a)ed.ac.uk&gt; wrote: > >> Hi, >> >> I've been out the LDAP loop for a bit but the recent discussion of the >> memberof overlay on 2.5 piqued my curiosity. Having upgraded a Dev box, >> removed the memberof elements from the database and replaced the >> memberof overlay with dynlist the queries appear to work as expected but >> are both a) slow and b) heavily CPU-intensive on the LDAP server. > > > As an aside, I would note that you appear to be indexing "pres" > unnecessarily.  Please read > <https://www.openldap.org/doc/admin25/tuning.html#Presence%20indexing> > > If the group object is large you may be having slow searches due to indices > being collapsed to a range.  You would need to run the search with trace > logging to determine if that's the case as was recently discussed on the > list. > > Regards, > Quanah > Hi, We are also observing very long query time involving memberOf when using dynlist migrating a 2.4 OpenLDAP docker install to 2.5.11. We have followed community recommendation and replaced the deprecated memberOf overlay with dynlist to calculate memberOf attributes, and removed the memberof elements from the data before importing it. Using dynlist, we have noticed an important drop of performance in queries involving the memberOf attributes on (relatively) large database (about 36000 users and 9000 groups dispatched in 150 branches) compared to 2.4: Queries on memberOf attributes that used to be processed in 0.200 seconds now takes about 6-7 seconds. In the example below, the search base is a branch with a relatively small number of users (about 350) and the targeted group contains 260 users: time ldapsearch -x -h 172.17.0.2 -D "cn=admin,dc=domain,dc=com" -w "secret" -b "ou=people,dc=branch,dc=domain,dc=com" "(memberof=cn=dev,ou=groups,dc=branch,dc=domain,dc=com)" uid real    0m6,358s user    0m0,024s sys    0m0,020s If the group used in memberOf has 1200 members, the query takes over 30 seconds. These results are obtained using an OpenLDAP running in a simple docker container on a host with 8CPU/16GB RAM. When using the memberOf overlay instead of dynlist to calculate memberOf, the performance are back to what is used to be in 2.4 (< 0.200 second for the above ldapsearch). We believe our config to be very standard: member attribute is indexed and dynlist is loaded and configured as below dn: olcOverlay={3}dynlist,olcDatabase={1}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcDynListConfig olcOverlay: {3}dynlist olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames Are this performance issues an expected side-effect of switching to dynlist - as the memberOf attributes are now dynamically calculated while the memberOf overlay used to writes these attributes - or something is more likely wrong with our install or could be tuned to improve these performances?

Some organizations have applications that relies on memberOf queries to be fast, and expects to keep these performances when upgrading to 2.5. From what we observed, we can only advise them to stick with the legacy memberOf overlay if they need these queries to stay fast, or rethink their approach to avoid querying memberOf on large databases.

Regards,