On 01/09/2021 17:17, Quanah Gibson-Mount wrote:

--On Wednesday, September 1, 2021 2:07 PM +0100 Mark Cairney Mark.Cairney@ed.ac.uk wrote:

...

Hi,

I've been out the LDAP loop for a bit but the recent discussion of the memberof overlay on 2.5 piqued my curiosity. Having upgraded a Dev box, removed the memberof elements from the database and replaced the memberof overlay with dynlist the queries appear to work as expected but are both a) slow and b) heavily CPU-intensive on the LDAP server.

As an aside, I would note that you appear to be indexing "pres" unnecessarily. Please read https://www.openldap.org/doc/admin25/tuning.html#Presence%20indexing

If the group object is large you may be having slow searches due to indices being collapsed to a range. You would need to run the search with trace logging to determine if that's the case as was recently discussed on the list.

Regards, Quanah

Hi,

We are also observing very long query time involving memberOf when using dynlist migrating a 2.4 OpenLDAP docker install to 2.5.11. We have followed community recommendation and replaced the deprecated memberOf overlay with dynlist to calculate memberOf attributes, and removed the memberof elements from the data before importing it.

Using dynlist, we have noticed an important drop of performance in queries involving the memberOf attributes on (relatively) large database (about 36000 users and 9000 groups dispatched in 150 branches) compared to 2.4: Queries on memberOf attributes that used to be processed in 0.200 seconds now takes about 6-7 seconds. In the example below, the search base is a branch with a relatively small number of users (about 350) and the targeted group contains 260 users:

time ldapsearch -x -h 172.17.0.2 -D "cn=admin,dc=domain,dc=com" -w "secret" -b "ou=people,dc=branch,dc=domain,dc=com" "(memberof=cn=dev,ou=groups,dc=branch,dc=domain,dc=com)" uid

real 0m6,358s user 0m0,024s sys 0m0,020s

If the group used in memberOf has 1200 members, the query takes over 30 seconds. These results are obtained using an OpenLDAP running in a simple docker container on a host with 8CPU/16GB RAM.

When using the memberOf overlay instead of dynlist to calculate memberOf, the performance are back to what is used to be in 2.4 (< 0.200 second for the above ldapsearch).

We believe our config to be very standard: member attribute is indexed and dynlist is loaded and configured as below

dn: olcOverlay={3}dynlist,olcDatabase={1}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcDynListConfig olcOverlay: {3}dynlist olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames

Are this performance issues an expected side-effect of switching to dynlist - as the memberOf attributes are now dynamically calculated while the memberOf overlay used to writes these attributes - or something is more likely wrong with our install or could be tuned to improve these performances?

Some organizations have applications that relies on memberOf queries to be fast, and expects to keep these performances when upgrading to 2.5. From what we observed, we can only advise them to stick with the legacy memberOf overlay if they need these queries to stay fast, or rethink their approach to avoid querying memberOf on large databases.

Regards,

On May 11, 2022, at 5:48 AM, Soisik Froger soisik.froger@worteks.com wrote:

If the group used in memberOf has 1200 members, the query takes over 30 seconds. These results are obtained using an OpenLDAP running in a simple docker container on a host with 8CPU/16GB RAM.

When using the memberOf overlay instead of dynlist to calculate memberOf, the performance are back to what is used to be in 2.4 (< 0.200 second for the above ldapsearch).

We believe our config to be very standard: member attribute is indexed and dynlist is loaded and configured as below

dn: olcOverlay={3}dynlist,olcDatabase={1}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcDynListConfig olcOverlay: {3}dynlist olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames

Hi Soisik,

I’d like to recreate this in a test env in order to analyze and look for solutions. Can you send me your (complete) test server config and specifics on the tests, like ldif, sample data and operations?

Thanks

— Shawn

On May 23, 2022, at 3:10 AM, Soisik Froger soisik.froger@worteks.com wrote:

I've just sent you sample slapd.conf and a data ldif that illustrate the long query time on large database. Thank you very much to take a look at this !

Hi Soisik,

I've completed a review/test of dynlist, following your excellent example as sent off list.

Here are my findings…

As a baseline, here’s an example of a 'normal' search across a well populated user tree:

# time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 'ou=people,dc=example,dc=com' "(cn=load01-TEST1-*)" | grep dn: | wc -l 17470

real 0m0.191s user 0m0.128s sys 0m0.132s

Compared with performing a search of the memberof attribute, generated by dynast. This pulls back the members of a large group (about 25K members): # time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 'ou=people,dc=example,dc=com' "(memberof=cn=load01-test1-2,ou=groups,dc=example,dc=com)" | grep dn: | wc -l 23869 real 4m6.167s user 0m2.289s sys 0m2.037s

That's 4 minutes to search across 100K users. The other search took < 200 ms.

As you (and others) have pointed out, there's a significant performance penalty for searching attributes generated by dylist.

It's possible that we can get modest improvements within a 2.7 timeframe. Unfortunately, we don't anticipate being able to get it back into range of a 'normal' search.

So, that gets us into looking at mitigation. One approach, focus on what the client does. For example, instead of searching across the user tree for membership to a particular group, it's far more efficient to just pull back that list from the group entry itself.

This is a technique that of course we’re already familiar with. I bring it up here for the sake of argument.

The question: can we sidestep the memberOf performance problems by focusing on the the client?

Or, is there a usecase that I've missed for which there's no remedy?

Thanks

— Shawn

Hello,

I have worked with Soisik on this particular topic. As you mentionned, in many cases we can just query directly the group. In our particular scenario, the software only allows one request for getting the mails of the users present in the dynamic group.

Thus I think it would be great to have a server solution (if possible in 2.6), especially if dynlist is officially the memberof replacement.

Regards,

David

Le 25 juillet 2022 19:38:52 GMT+02:00, Shawn McKinney smckinney@symas.com a écrit :

...

On May 23, 2022, at 3:10 AM, Soisik Froger soisik.froger@worteks.com wrote:

I've just sent you sample slapd.conf and a data ldif that illustrate the long query time on large database. Thank you very much to take a look at this !

Hi Soisik,

I've completed a review/test of dynlist, following your excellent example as sent off list.

Here are my findings…

As a baseline, here’s an example of a 'normal' search across a well populated user tree:

# time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 'ou=people,dc=example,dc=com' "(cn=load01-TEST1-*)" | grep dn: | wc -l 17470

real 0m0.191s user 0m0.128s sys 0m0.132s

Compared with performing a search of the memberof attribute, generated by dynast. This pulls back the members of a large group (about 25K members): # time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 'ou=people,dc=example,dc=com' "(memberof=cn=load01-test1-2,ou=groups,dc=example,dc=com)" | grep dn: | wc -l 23869 real 4m6.167s user 0m2.289s sys 0m2.037s

That's 4 minutes to search across 100K users. The other search took < 200 ms.

As you (and others) have pointed out, there's a significant performance penalty for searching attributes generated by dylist.

It's possible that we can get modest improvements within a 2.7 timeframe. Unfortunately, we don't anticipate being able to get it back into range of a 'normal' search.

So, that gets us into looking at mitigation. One approach, focus on what the client does. For example, instead of searching across the user tree for membership to a particular group, it's far more efficient to just pull back that list from the group entry itself.

This is a technique that of course we’re already familiar with. I bring it up here for the sake of argument.

The question: can we sidestep the memberOf performance problems by focusing on the the client?

Or, is there a usecase that I've missed for which there's no remedy?

Thanks

— Shawn

Hello,

Le 26/07/2022 à 00:04, Howard Chu a écrit :

David Coutadeur wrote:

...

Hello,

I have worked with Soisik on this particular topic. As you mentionned, in many cases we can just query directly the group. In our particular scenario, the software only allows one request for getting the mails of the users present in the dynamic group.

Have you already tried using the deref control to return a mail attribute instead?

Thanks for the idea.

I suppose it could work indeed. I have not tested it yet, but it could be interresting to have the performance associated to such request.

However, in my specific use case, it is not possible to configure a LDAP control in the software.

...

Thus I think it would be great to have a server solution (if possible in 2.6), especially if dynlist is officially the memberof replacement.

Regards,

David

Le 25 juillet 2022 19:38:52 GMT+02:00, Shawn McKinney smckinney@symas.com a écrit :

     On May 23, 2022, at 3:10 AM, Soisik Froger <soisik.froger@worteks.com> wrote:

     I've just sent you sample slapd.conf and a data ldif that illustrate the long query time on large database. Thank you very much to take a look at this !


 Hi Soisik,

 I've completed a review/test of dynlist, following your excellent example as sent off list.

 Here are my findings…

 As a baseline, here’s an example of a 'normal' search across a well populated user tree:

 # time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 'ou=people,dc=example,dc=com' "(cn=load01-TEST1-*)" | grep dn: | wc -l
 17470

 real    0m0.191s
 user    0m0.128s
 sys     0m0.132s

 Compared with performing a search of the memberof attribute, generated by dynast. This pulls back the members of a large group (about 25K members):
 # time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 'ou=people,dc=example,dc=com' "(memberof=cn=load01-test1-2,ou=groups,dc=example,dc=com)" | grep dn: | wc -l
 23869
 real    4m6.167s
 user    0m2.289s
 sys     0m2.037s

 That's 4 minutes to search across 100K users. The other search took < 200 ms.

 As you (and others) have pointed out, there's a significant performance penalty for searching attributes generated by dylist.

 It's possible that we can get modest improvements within a 2.7 timeframe. Unfortunately, we don't anticipate being able to get it back into range of a 'normal' search.

 So, that gets us into looking at mitigation.  One approach, focus on what the client does. For example, instead of searching across the user tree for membership to a particular group, it's far more efficient to just pull back that list from the group entry itself.

 This is a technique that of course we’re already familiar with. I bring it up here for the sake of argument.

 The question: can we sidestep the memberOf performance problems by focusing on the the client?

 Or, is there a usecase that I've missed for which there's no remedy?

 Thanks

 —
 Shawn

On Tue, Jan 03, 2023 at 11:44:30AM -0600, Shawn McKinney wrote:

Some work on the dynlist code of late to boost performance.

So, it might be worthwhile to give it a try. MR576[1] in the next release.

Is this going to hit the 2.5 LTS train or just 2.6?

Thanks...

On Mon, Jan 09, 2023 at 09:26:44AM -0600, Shawn McKinney wrote:

Yes (both)

Sweet :). We've had some performance issues with the new dynlist implementation since we upgraded to 2.5, I look forward to trying this out.

We're currently using dynlist to add the memberOf attribute to users. One thing that was nice about the new dynlist implementation in 2.5 was that it allows searching on the dynamic memberOf attribute which we couldn't do in 2.4. Looking at the commit diff, there's one part that says:

"To enable filter evaluation on the dynamic list, the configuration must be changed to explicitly map the dynamic attributes to be filtered"

This just applies to groups created dynamically, right, not static objects that get dynamic attributes added? Our current config is:

dynlist-attrset groupOfURLs memberURL member+memberOf@groupOfNames

It doesn't require any changes to keep working with a searchable memberOf attribute?

Thanks much...

On Mon, Jan 09, 2023 at 11:39:47AM -0800, Paul B. Henson wrote:

We're currently using dynlist to add the memberOf attribute to users. One thing that was nice about the new dynlist implementation in 2.5 was that it allows searching on the dynamic memberOf attribute which we couldn't do in 2.4. Looking at the commit diff, there's one part that says:

"To enable filter evaluation on the dynamic list, the configuration must be changed to explicitly map the dynamic attributes to be filtered"

This just applies to groups created dynamically, right, not static objects that get dynamic attributes added? Our current config is:

The latest manpage update should make it clearer how dynamic *lists* differ from dynamic *groups*. And yes, no need to change config for groups.

If you can give the current 2.5/2.6 branch a go, or if you decide to wait until we get a call for testing out, feedback is always welcome.

Regards,

On 7/25/2022 10:38 AM, Shawn McKinney wrote:

As you (and others) have pointed out, there's a significant performance penalty for searching attributes generated by dylist.

I'm still seeing performance issues with queries that simply return memberOf, with no reference to it in the actual search filter.

For example, this query which searches on the static uid attribute and returns memberOf:

time ldapsearch -H ldapi:/// uid=henson memberOf

Most of the time it completes in fractions of a second:

real 0m0.187s user 0m0.005s sys 0m0.003s

But sometimes it takes 5 seconds, 10 seconds, or even more. These extremely slow response times coordinate with a high read I/O percentage on the server and the high number of page faults on the slapd process.

When I first deployed 2.5, sometimes the server would get into a state where every query that requested memberOf would take in excess of 30 seconds to return until the server was restarted. I cranked up the memory on the servers and at this point I have had no more reoccurrences of that behavior, but I am still regularly seeing occasional slow performance on the queries and high read I/O percentages.

The servers have way more memory now than they should need to fully cache the entire database:

# du -sh /var/symas/openldap-data-cpp 2.6G /var/symas/openldap-data-cpp

# free -m total used free shared buff/cache available Mem: 4818 703 124 0 3991 3831 Swap: 2047 136 1911

I haven't been able to correlate the slow response times with any other external criteria such as updates or query load. Sometimes it's just slow 8-/. We never saw this problem under 2.4 which used the previous implementation of dynlist to generate memberOf. I definitely appreciate the ability to query on dynamic memberOf that was added in 2.5, but it would be nice to sort out this performance issue.

I second that, actually I was going to suggest doing exactly this test.

If possible, you should try to do it somewhere else that is not the production server, maybe a VM after restoring the database?

That would allow you to carefully execute tests without interference from other requests coming in.

If you're in Linux, strace might give you more details of what is happening in terms of system calls, but it will generate a loot of messages so you should try it as last resource.

Last, there are applications that might help you understand what is going on:

http://cnmonitor.sourceforge.net/

https://github.com/jcollie/openldap_exporter (this will require to install and configure Prometheus first)

Hope this helps.

Regards,

Alceu

On 26/07/2022 05:56, Wilkinson, Hugo (IT Dept) wrote:

If you have the ability to do so and your kernel is v5+, out-of-hours experiment with disabling system swap entirely (vm.swapiness=0 and 'swapoff -a' )  and then simulate a run of requests you'd expect to encounter the page errors with and see if it stops happening.

Depending on what else is going on this might not be a production-compatible strategy, but it will either rule that out as a potential flaw/bug in 2.5 and/or give you a workaround until it's fixed.

Hi both,

Just to let you know that with my dataset I see a consistently slow (10-20 secs) lookup of a user's memberOf group membership using dynamic dynlist overlays on 2.5 and 2.6 compared with the old "memberof" overlay.

My dynlist config is as below:

dn: olcOverlay={2}dynlist,olcDatabase={1}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcDynListConfig olcOverlay: {2}dynlist olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames structuralObjectClass: olcDynListConfig

It's fairly common to use the memberOf attribute to look up the groups a particular user is a member of e.g. for access control purposes. We use it routinely for this purpose on our Shibboleth IdPs which use OpenLDAP as a datasource.

In terms of data size/complexity we've currently got around 500K users (each with their own unique group in a separate OU) with the bulk of our group membership managed by Grouper. We've got about 100K groups in there(!) ranging in size from a single "placeholder" member up to about 200K members for our applicant/alumni groups.

The total DIT size (dumped via slapcat) is just under 2GB and our (freshly slapcat'ed and slapadd'ed mdb database size is currently sitting at 5.1GB). We're running on Centos 7 boxes with 12vCPUs and 64GB RAM each which I think should be enough CPU/memory resource for our usage?

On a related note for a production environment should we be targeting the 2.5 or 2.6 branch? I'd like to upgrade our production servers off 2.4 before the start of next term.

Kind regards,

Mark

On 26/07/2022 00:16, Paul B. Henson wrote:

This email was sent to you by someone outside the University. You should only click on links or attachments if you are certain that the email is genuine and the content is safe.

On 7/25/2022 10:38 AM, Shawn McKinney wrote:

...

As you (and others) have pointed out, there's a significant performance penalty for searching attributes generated by dylist.

I'm still seeing performance issues with queries that simply return memberOf, with no reference to it in the actual search filter.

For example, this query which searches on the static uid attribute and returns memberOf:

time ldapsearch -H ldapi:/// uid=henson memberOf

Most of the time it completes in fractions of a second:

real 0m0.187s user 0m0.005s sys 0m0.003s

But sometimes it takes 5 seconds, 10 seconds, or even more. These extremely slow response times coordinate with a high read I/O percentage on the server and the high number of page faults on the slapd process.

When I first deployed 2.5, sometimes the server would get into a state where every query that requested memberOf would take in excess of 30 seconds to return until the server was restarted. I cranked up the memory on the servers and at this point I have had no more reoccurrences of that behavior, but I am still regularly seeing occasional slow performance on the queries and high read I/O percentages.

The servers have way more memory now than they should need to fully cache the entire database:

# du -sh /var/symas/openldap-data-cpp 2.6G /var/symas/openldap-data-cpp

# free -m total used free shared buff/cache available Mem: 4818 703 124 0 3991 3831 Swap: 2047 136 1911

I haven't been able to correlate the slow response times with any other external criteria such as updates or query load. Sometimes it's just slow 8-/. We never saw this problem under 2.4 which used the previous implementation of dynlist to generate memberOf. I definitely appreciate the ability to query on dynamic memberOf that was added in 2.5, but it would be nice to sort out this performance issue.

-- /****************************

Mark Cairney ITI Enterprise Services Information Services University of Edinburgh

Tel: 0131 650 6565 Email: Mark.Cairney@ed.ac.uk

*******************************/

The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

...

...

"Paul B. Henson" henson@acm.org schrieb am 22.05.2022 um 04:51 in Nachricht

5d343067-aef3-b499-63e3-996f3d68051e@acm.org:

On 5/11/2022 3:48 AM, Soisik Froger wrote:

...

Are this performance issues an expected side-effect of switching to dynlist - as the memberOf attributes are now dynamically calculated while the memberOf overlay used to writes these attributes - or

I am also having ongoing sporadic issues with memberOf performance using the new dynlist overlay. Initially, randomly a server would get into a state where any query requesting the memberOf attribute would take in excess of 30 seconds, whereas normally it would only take a fraction of a second. The symptoms were the same, free memory, no swapping, but insanely high read IO load.

I'm wondering: If you'd make a core dump when the issue happens, how big would such a core dump be with MDB? I'm afraid it would be insanely large, containing the whole database. Am I wrong?

I cranked up the memory, which not did resolve the issue, but did help, it doesn't happen nearly as often. But still, every now and again, a server demonstrates a high read IO rate and severely degraded memberOf query performance. At this point, I just have a monitoring check that alerts on slow query performance and high read I/O and go restart them when it happens, as the additional memory made the issue go from a couple of times a week to every month or three.

I did notice now that when the issue occurs the box with the slow queries does have less memory available then when it is working normally, but still a good gigabyte of free memory not being used.

Even when the systems don't completely blow up, there are occasional slower than normal queries. Typically the test query I am doing literally takes fractions of a second:

May 21 19:47:22 ldap-01 slapd[1223]: conn=849157 op=1 SEARCH RESULT tag=101 err=0 qtime=0.000015 etime=0.198042 nentries=1 text=

Every now and again for no discernible reason it might take 5 to 10 seconds.

...

...

Howard Chu hyc@symas.com schrieb am 23.05.2022 um 18:04 in Nachricht

cc25e9ad-988c-fcd6-7bc0-0de6e3c530c7@symas.com:

Ulrich Windl wrote:

...

...

...

...

"Paul B. Henson" henson@acm.org schrieb am 22.05.2022 um 04:51 in Nachricht

5d343067-aef3-b499-63e3-996f3d68051e@acm.org:

...

On 5/11/2022 3:48 AM, Soisik Froger wrote:

...

Are this performance issues an expected side-effect of switching to dynlist - as the memberOf attributes are now dynamically calculated while the memberOf overlay used to writes these attributes - or

I am also having ongoing sporadic issues with memberOf performance using the new dynlist overlay. Initially, randomly a server would get into a state where any query requesting the memberOf attribute would take in excess of 30 seconds, whereas normally it would only take a fraction of a second. The symptoms were the same, free memory, no swapping, but insanely high read IO load.

I'm wondering: If you'd make a core dump when the issue happens, how big

would such a core dump be with MDB?

...

I'm afraid it would be insanely large, containing the whole database. Am I

wrong?

Yes, you are wrong.

mmap'd memory isn't included in core dumps by default.

OK, but that makes it hard to debug any MDB issues from the core dump unless significant parts are copied in to private memory (which I think is not the idea behind MDB).

-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/