Hi both,
Just to let you know that with my dataset I see a consistently slow (10-20 secs) lookup of a user's memberOf group membership using dynamic dynlist overlays on 2.5 and 2.6 compared with the old "memberof" overlay.
My dynlist config is as below:
dn: olcOverlay={2}dynlist,olcDatabase={1}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcDynListConfig olcOverlay: {2}dynlist olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames structuralObjectClass: olcDynListConfig
It's fairly common to use the memberOf attribute to look up the groups a particular user is a member of e.g. for access control purposes. We use it routinely for this purpose on our Shibboleth IdPs which use OpenLDAP as a datasource.
In terms of data size/complexity we've currently got around 500K users (each with their own unique group in a separate OU) with the bulk of our group membership managed by Grouper. We've got about 100K groups in there(!) ranging in size from a single "placeholder" member up to about 200K members for our applicant/alumni groups.
The total DIT size (dumped via slapcat) is just under 2GB and our (freshly slapcat'ed and slapadd'ed mdb database size is currently sitting at 5.1GB). We're running on Centos 7 boxes with 12vCPUs and 64GB RAM each which I think should be enough CPU/memory resource for our usage?
On a related note for a production environment should we be targeting the 2.5 or 2.6 branch? I'd like to upgrade our production servers off 2.4 before the start of next term.
Kind regards,
Mark
On 26/07/2022 00:16, Paul B. Henson wrote:
This email was sent to you by someone outside the University. You should only click on links or attachments if you are certain that the email is genuine and the content is safe.
On 7/25/2022 10:38 AM, Shawn McKinney wrote:
As you (and others) have pointed out, there's a significant performance penalty for searching attributes generated by dylist.
I'm still seeing performance issues with queries that simply return memberOf, with no reference to it in the actual search filter.
For example, this query which searches on the static uid attribute and returns memberOf:
time ldapsearch -H ldapi:/// uid=henson memberOf
Most of the time it completes in fractions of a second:
real 0m0.187s user 0m0.005s sys 0m0.003s
But sometimes it takes 5 seconds, 10 seconds, or even more. These extremely slow response times coordinate with a high read I/O percentage on the server and the high number of page faults on the slapd process.
When I first deployed 2.5, sometimes the server would get into a state where every query that requested memberOf would take in excess of 30 seconds to return until the server was restarted. I cranked up the memory on the servers and at this point I have had no more reoccurrences of that behavior, but I am still regularly seeing occasional slow performance on the queries and high read I/O percentages.
The servers have way more memory now than they should need to fully cache the entire database:
# du -sh /var/symas/openldap-data-cpp 2.6G /var/symas/openldap-data-cpp
# free -m total used free shared buff/cache available Mem: 4818 703 124 0 3991 3831 Swap: 2047 136 1911
I haven't been able to correlate the slow response times with any other external criteria such as updates or query load. Sometimes it's just slow 8-/. We never saw this problem under 2.4 which used the previous implementation of dynlist to generate memberOf. I definitely appreciate the ability to query on dynamic memberOf that was added in 2.5, but it would be nice to sort out this performance issue.
-- /****************************
Mark Cairney ITI Enterprise Services Information Services University of Edinburgh
Tel: 0131 650 6565 Email: Mark.Cairney@ed.ac.uk
*******************************/
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.