3:23 a.m.
Hi both,
Just to let you know that with my dataset I see a consistently slow
(10-20 secs) lookup of a user's memberOf group membership using dynamic
dynlist overlays on 2.5 and 2.6 compared with the old "memberof" overlay.
My dynlist config is as below:
dn: olcOverlay={2}dynlist,olcDatabase={1}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcDynListConfig
olcOverlay: {2}dynlist
olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames
structuralObjectClass: olcDynListConfig
It's fairly common to use the memberOf attribute to look up the groups a
particular user is a member of e.g. for access control purposes. We use
it routinely for this purpose on our Shibboleth IdPs which use OpenLDAP
as a datasource.
In terms of data size/complexity we've currently got around 500K users
(each with their own unique group in a separate OU) with the bulk of our
group membership managed by Grouper. We've got about 100K groups in
there(!) ranging in size from a single "placeholder" member up to about
200K members for our applicant/alumni groups.
The total DIT size (dumped via slapcat) is just under 2GB and our
(freshly slapcat'ed and slapadd'ed mdb database size is currently
sitting at 5.1GB). We're running on Centos 7 boxes with 12vCPUs and 64GB
RAM each which I think should be enough CPU/memory resource for our usage?
On a related note for a production environment should we be targeting
the 2.5 or 2.6 branch? I'd like to upgrade our production servers off
2.4 before the start of next term.
Kind regards,
Mark
On 26/07/2022 00:16, Paul B. Henson wrote:
This email was sent to you by someone outside the University.
You should only click on links or attachments if you are certain that
the email is genuine and the content is safe.
On 7/25/2022 10:38 AM, Shawn McKinney wrote:
> As you (and others) have pointed out, there's a significant
> performance penalty for searching attributes generated by dylist.
I'm still seeing performance issues with queries that simply return
memberOf, with no reference to it in the actual search filter.
For example, this query which searches on the static uid attribute and
returns memberOf:
time ldapsearch -H ldapi:/// uid=henson memberOf
Most of the time it completes in fractions of a second:
real 0m0.187s
user 0m0.005s
sys 0m0.003s
But sometimes it takes 5 seconds, 10 seconds, or even more. These
extremely slow response times coordinate with a high read I/O percentage
on the server and the high number of page faults on the slapd process.
When I first deployed 2.5, sometimes the server would get into a state
where every query that requested memberOf would take in excess of 30
seconds to return until the server was restarted. I cranked up the
memory on the servers and at this point I have had no more reoccurrences
of that behavior, but I am still regularly seeing occasional slow
performance on the queries and high read I/O percentages.
The servers have way more memory now than they should need to fully
cache the entire database:
# du -sh /var/symas/openldap-data-cpp
2.6G /var/symas/openldap-data-cpp
# free -m
total used free shared buff/cache
available
Mem: 4818 703 124 0 3991
3831
Swap: 2047 136 1911
I haven't been able to correlate the slow response times with any other
external criteria such as updates or query load. Sometimes it's just
slow 8-/. We never saw this problem under 2.4 which used the previous
implementation of dynlist to generate memberOf. I definitely appreciate
the ability to query on dynamic memberOf that was added in 2.5, but it
would be nice to sort out this performance issue.
--
/****************************
Mark Cairney
ITI Enterprise Services
Information Services
University of Edinburgh
Tel: 0131 650 6565
Email: Mark.Cairney(a)ed.ac.uk
*******************************/
The University of Edinburgh is a charitable body, registered in Scotland, with
registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn
Èideann, clàraichte an Alba, àireamh clàraidh SC005336.