Hi,
I've been out the LDAP loop for a bit but the recent discussion of the memberof overlay on 2.5 piqued my curiosity. Having upgraded a Dev box, removed the memberof elements from the database and replaced the memberof overlay with dynlist the queries appear to work as expected but are both a) slow and b) heavily CPU-intensive on the LDAP server.
2021-09-01T12:47:17.603513+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 fd=12 ACCEPT from IP=192.168.152.33:58738 (IP=129.215.17.9:636) 2021-09-01T12:47:17.687488+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 fd=12 TLS established tls_ssf=256 ssf=256 tls_proto=TLSv1.3 tls_cipher=TLS_AES_256_GCM_SHA384 2021-09-01T12:47:17.688032+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 op=0 SRCH base="" scope=0 deref=0 filter="(objectClass=*)" 2021-09-01T12:47:17.688470+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 op=0 SRCH attr=supportedSASLMechanisms 2021-09-01T12:47:17.688878+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 op=0 SEARCH RESULT tag=101 err=0 qtime=0.000014 etime=0.000214 nentries=1 text= 2021-09-01T12:47:17.811279+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 op=1 BIND dn="" method=163 2021-09-01T12:47:17.819249+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 op=1 RESULT tag=97 err=14 qtime=0.000030 etime=0.009084 text=SASL(0): successful result: 2021-09-01T12:47:17.908889+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 op=2 BIND dn="" method=163 2021-09-01T12:47:17.909836+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 op=2 RESULT tag=97 err=14 qtime=0.000031 etime=0.000181 text=SASL(0): successful result: 2021-09-01T12:47:17.938839+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 op=3 BIND dn="" method=163 2021-09-01T12:47:17.939621+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 op=3 BIND authcid="mcairney@EASE.ED.AC.UK" authzid="mcairney@EASE.ED.AC.UK" 2021-09-01T12:47:17.940213+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 op=3 BIND dn="uid=mcairney,ou=people,ou=central,dc=authorise-dev,dc=ed,dc=ac,dc=uk" mech=GSSAPI bind_ssf=256 ssf=256 2021-09-01T12:47:17.940616+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 op=3 RESULT tag=97 err=0 qtime=0.000024 etime=0.000409 text= 2021-09-01T12:47:18.227342+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 op=4 SRCH base="dc=authorise-dev,dc=ed,dc=ac,dc=uk" scope=2 deref=0 filter="(uid=mcairney)" 2021-09-01T12:47:18.227703+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 op=4 SRCH attr=* + 2021-09-01T12:47:31.392255+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 op=5 UNBIND 2021-09-01T12:47:31.460705+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 op=4 SEARCH RESULT tag=101 err=0 qtime=0.000031 etime=13.233679 nentries=1 text= 2021-09-01T12:47:31.461098+01:00 bonsai.authorise-dev.is.ed.ac.uk slapd[30075]: conn=1019 fd=12 closed
I'm guessing that as the values are computed that this will be heavier on the CPU but it seems a bit excessive? Has anyone else noticed any similar performance issues?
This is a relatively low-specced DEV server (2 vCPUs, 4GB RAM) so I guess this could be a factor but there's no io waiting on the server and no swapping?
The database is on a par in size with our Production service ( about 400K user objects with 1 group object per user and then about 80K actual groups of users)
The config for the primary DB (ACLs and rootPW redacted) is:
dn: olcDatabase={1}mdb,cn=config objectClass: olcDatabaseConfig objectClass: olcMdbConfig olcDatabase: {1}mdb olcDbDirectory: /opt/openldap/var/openldap-data/authorise olcSuffix: dc=authorise-dev,dc=ed,dc=ac,dc=uk
olcAddContentAcl: FALSE olcLastMod: TRUE olcMaxDerefDepth: 2 olcReadOnly: FALSE olcSecurity: ssf=1 olcSecurity: update_ssf=112 olcSecurity: simple_bind=64 olcSizeLimit: unlimited olcSyncUseSubentry: FALSE olcTimeLimit: unlimited olcMonitoring: TRUE olcDbEnvFlags: writemap olcDbEnvFlags: nometasync olcDbNoSync: FALSE olcDbIndex: objectClass eq olcDbIndex: entryUUID eq olcDbIndex: entryCSN eq olcDbIndex: cn pres,eq,sub olcDbIndex: uid pres,eq,sub olcDbIndex: uidNumber pres,eq olcDbIndex: gidNumber pres,eq olcDbIndex: eduniType eq olcDbIndex: gecos pres,eq,sub olcDbIndex: eduniCategory eq olcDbIndex: mail pres,eq,sub olcDbIndex: eduniSchoolCode eq olcDbIndex: eduniIDStatus eq olcDbIndex: eduniCollegeCode eq olcDbIndex: eduniOrgCode eq olcDbIndex: memberOf pres,eq olcDbIndex: eduniLibraryBarcode pres,eq olcDbIndex: eduniOrganisation pres,eq,sub olcDbIndex: eduniServiceCode pres,eq olcDbIndex: krbName pres,eq olcDbIndex: eduPersonAffiliation pres,eq olcDbIndex: eduPersonEntitlement pres,eq olcDbIndex: sn pres,eq,sub olcDbIndex: eduniIdmsId pres,eq olcDbIndex: member pres,eq olcDbIndex: memberUid pres,eq olcDbIndex: eduniRefNo pres,eq olcDbIndex: eduniTitle pres,eq olcDbIndex: title pres,eq,sub olcDbIndex: eduniCardNumber pres,eq olcDbIndex: eduniYearOfStudy eq olcDbIndex: description pres,eq,sub olcDbIndex: givenName pres,eq,sub olcDbIndex: aliasedObjectName eq olcDbIndex: yubiKeyId pres,eq olcDbIndex: isMemberOf pres,eq olcDbIndex: hasMember pres,eq olcDbIndex: proxyAddresses pres,eq,sub olcDbMaxReaders: 96 olcDbMaxSize: 32212254720 olcDbMode: 0600 olcDbSearchStack: 16 structuralObjectClass: olcMdbConfig
dn: olcOverlay={0}syncprov,olcDatabase={1}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcConfig objectClass: top objectClass: olcSyncProvConfig olcOverlay: {0}syncprov structuralObjectClass: olcSyncProvConfig
dn: olcOverlay={1}accesslog,olcDatabase={1}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcAccessLogConfig olcOverlay: {1}accesslog olcAccessLogDB: cn=accesslog olcAccessLogOps: writes olcAccessLogPurge: 02+00:00 00+04:00 olcAccessLogSuccess: TRUE structuralObjectClass: olcAccessLogConfig
dn: olcOverlay={2}dynlist,olcDatabase={1}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcDynListConfig olcOverlay: {2}dynlist olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames structuralObjectClass: olcDynListConfig
-- /****************************
Mark Cairney ITI Enterprise Services Information Services University of Edinburgh
Tel: 0131 650 6565 Email: Mark.Cairney@ed.ac.uk
*******************************/
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.
--On Wednesday, September 1, 2021 2:07 PM +0100 Mark Cairney Mark.Cairney@ed.ac.uk wrote:
Hi,
I've been out the LDAP loop for a bit but the recent discussion of the memberof overlay on 2.5 piqued my curiosity. Having upgraded a Dev box, removed the memberof elements from the database and replaced the memberof overlay with dynlist the queries appear to work as expected but are both a) slow and b) heavily CPU-intensive on the LDAP server.
As an aside, I would note that you appear to be indexing "pres" unnecessarily. Please read https://www.openldap.org/doc/admin25/tuning.html#Presence%20indexing
If the group object is large you may be having slow searches due to indices being collapsed to a range. You would need to run the search with trace logging to determine if that's the case as was recently discussed on the list.
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
On 01/09/2021 17:17, Quanah Gibson-Mount wrote:
--On Wednesday, September 1, 2021 2:07 PM +0100 Mark Cairney Mark.Cairney@ed.ac.uk wrote:
Hi,
I've been out the LDAP loop for a bit but the recent discussion of the memberof overlay on 2.5 piqued my curiosity. Having upgraded a Dev box, removed the memberof elements from the database and replaced the memberof overlay with dynlist the queries appear to work as expected but are both a) slow and b) heavily CPU-intensive on the LDAP server.
As an aside, I would note that you appear to be indexing "pres" unnecessarily. Please read https://www.openldap.org/doc/admin25/tuning.html#Presence%20indexing
If the group object is large you may be having slow searches due to indices being collapsed to a range. You would need to run the search with trace logging to determine if that's the case as was recently discussed on the list.
Regards, Quanah
Hi,
We are also observing very long query time involving memberOf when using dynlist migrating a 2.4 OpenLDAP docker install to 2.5.11. We have followed community recommendation and replaced the deprecated memberOf overlay with dynlist to calculate memberOf attributes, and removed the memberof elements from the data before importing it.
Using dynlist, we have noticed an important drop of performance in queries involving the memberOf attributes on (relatively) large database (about 36000 users and 9000 groups dispatched in 150 branches) compared to 2.4: Queries on memberOf attributes that used to be processed in 0.200 seconds now takes about 6-7 seconds. In the example below, the search base is a branch with a relatively small number of users (about 350) and the targeted group contains 260 users:
time ldapsearch -x -h 172.17.0.2 -D "cn=admin,dc=domain,dc=com" -w "secret" -b "ou=people,dc=branch,dc=domain,dc=com" "(memberof=cn=dev,ou=groups,dc=branch,dc=domain,dc=com)" uid
real 0m6,358s user 0m0,024s sys 0m0,020s
If the group used in memberOf has 1200 members, the query takes over 30 seconds. These results are obtained using an OpenLDAP running in a simple docker container on a host with 8CPU/16GB RAM.
When using the memberOf overlay instead of dynlist to calculate memberOf, the performance are back to what is used to be in 2.4 (< 0.200 second for the above ldapsearch).
We believe our config to be very standard: member attribute is indexed and dynlist is loaded and configured as below
dn: olcOverlay={3}dynlist,olcDatabase={1}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcDynListConfig olcOverlay: {3}dynlist olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames
Are this performance issues an expected side-effect of switching to dynlist - as the memberOf attributes are now dynamically calculated while the memberOf overlay used to writes these attributes - or something is more likely wrong with our install or could be tuned to improve these performances?
Some organizations have applications that relies on memberOf queries to be fast, and expects to keep these performances when upgrading to 2.5. From what we observed, we can only advise them to stick with the legacy memberOf overlay if they need these queries to stay fast, or rethink their approach to avoid querying memberOf on large databases.
Regards,
Soisik Froger wrote:
On 01/09/2021 17:17, Quanah Gibson-Mount wrote:
--On Wednesday, September 1, 2021 2:07 PM +0100 Mark Cairney Mark.Cairney@ed.ac.uk wrote:
Hi,
I've been out the LDAP loop for a bit but the recent discussion of the memberof overlay on 2.5 piqued my curiosity. Having upgraded a Dev box, removed the memberof elements from the database and replaced the memberof overlay with dynlist the queries appear to work as expected but are both a) slow and b) heavily CPU-intensive on the LDAP server.
As an aside, I would note that you appear to be indexing "pres" unnecessarily. Please read https://www.openldap.org/doc/admin25/tuning.html#Presence%20indexing
If the group object is large you may be having slow searches due to indices being collapsed to a range. You would need to run the search with trace logging to determine if that's the case as was recently discussed on the list.
Regards, Quanah
Hi,
We are also observing very long query time involving memberOf when using dynlist migrating a 2.4 OpenLDAP docker install to 2.5.11. We have followed community recommendation and replaced the deprecated memberOf overlay with dynlist to calculate memberOf attributes, and removed the memberof elements from the data before importing it.
Using dynlist, we have noticed an important drop of performance in queries involving the memberOf attributes on (relatively) large database (about 36000 users and 9000 groups dispatched in 150 branches) compared to 2.4: Queries on memberOf attributes that used to be processed in 0.200 seconds now takes about 6-7 seconds. In the example below, the search base is a branch with a relatively small number of users (about 350) and the targeted group contains 260 users:
time ldapsearch -x -h 172.17.0.2 -D "cn=admin,dc=domain,dc=com" -w "secret" -b "ou=people,dc=branch,dc=domain,dc=com" "(memberof=cn=dev,ou=groups,dc=branch,dc=domain,dc=com)" uid
real 0m6,358s user 0m0,024s sys 0m0,020s
If the group used in memberOf has 1200 members, the query takes over 30 seconds. These results are obtained using an OpenLDAP running in a simple docker container on a host with 8CPU/16GB RAM.
When using the memberOf overlay instead of dynlist to calculate memberOf, the performance are back to what is used to be in 2.4 (< 0.200 second for the above ldapsearch).
We believe our config to be very standard: member attribute is indexed and dynlist is loaded and configured as below
dn: olcOverlay={3}dynlist,olcDatabase={1}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcDynListConfig olcOverlay: {3}dynlist olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames
Are this performance issues an expected side-effect of switching to dynlist - as the memberOf attributes are now dynamically calculated while the memberOf overlay used to writes these attributes - or something is more likely wrong with our install or could be tuned to improve these performances?
That's just the cost of dynamically calculating these attributes, yes.
It's possible we can make some other optimizations to the overlay to reduce the amount of work in calculating memberOf if you're only using it in a filter and not returning all of its values. Looks like it could be a fairly significant refactoring to achieve that though.
Some organizations have applications that relies on memberOf queries to be fast, and expects to keep these performances when upgrading to 2.5. From what we observed, we can only advise them to stick with the legacy memberOf overlay if they need these queries to stay fast, or rethink their approach to avoid querying memberOf on large databases.
Makes sense.
Regards,
On May 11, 2022, at 5:48 AM, Soisik Froger soisik.froger@worteks.com wrote:
If the group used in memberOf has 1200 members, the query takes over 30 seconds. These results are obtained using an OpenLDAP running in a simple docker container on a host with 8CPU/16GB RAM.
When using the memberOf overlay instead of dynlist to calculate memberOf, the performance are back to what is used to be in 2.4 (< 0.200 second for the above ldapsearch).
We believe our config to be very standard: member attribute is indexed and dynlist is loaded and configured as below
dn: olcOverlay={3}dynlist,olcDatabase={1}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcDynListConfig olcOverlay: {3}dynlist olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames
Hi Soisik,
I’d like to recreate this in a test env in order to analyze and look for solutions. Can you send me your (complete) test server config and specifics on the tests, like ldif, sample data and operations?
Thanks
— Shawn
On 5/11/2022 3:48 AM, Soisik Froger wrote:
Are this performance issues an expected side-effect of switching to dynlist - as the memberOf attributes are now dynamically calculated while the memberOf overlay used to writes these attributes - or
I am also having ongoing sporadic issues with memberOf performance using the new dynlist overlay. Initially, randomly a server would get into a state where any query requesting the memberOf attribute would take in excess of 30 seconds, whereas normally it would only take a fraction of a second. The symptoms were the same, free memory, no swapping, but insanely high read IO load.
I cranked up the memory, which not did resolve the issue, but did help, it doesn't happen nearly as often. But still, every now and again, a server demonstrates a high read IO rate and severely degraded memberOf query performance. At this point, I just have a monitoring check that alerts on slow query performance and high read I/O and go restart them when it happens, as the additional memory made the issue go from a couple of times a week to every month or three.
I did notice now that when the issue occurs the box with the slow queries does have less memory available then when it is working normally, but still a good gigabyte of free memory not being used.
Even when the systems don't completely blow up, there are occasional slower than normal queries. Typically the test query I am doing literally takes fractions of a second:
May 21 19:47:22 ldap-01 slapd[1223]: conn=849157 op=1 SEARCH RESULT tag=101 err=0 qtime=0.000015 etime=0.198042 nentries=1 text=
Every now and again for no discernible reason it might take 5 to 10 seconds.
"Paul B. Henson" henson@acm.org schrieb am 22.05.2022 um 04:51 in Nachricht
5d343067-aef3-b499-63e3-996f3d68051e@acm.org:
On 5/11/2022 3:48 AM, Soisik Froger wrote:
Are this performance issues an expected side-effect of switching to dynlist - as the memberOf attributes are now dynamically calculated while the memberOf overlay used to writes these attributes - or
I am also having ongoing sporadic issues with memberOf performance using the new dynlist overlay. Initially, randomly a server would get into a state where any query requesting the memberOf attribute would take in excess of 30 seconds, whereas normally it would only take a fraction of a second. The symptoms were the same, free memory, no swapping, but insanely high read IO load.
I'm wondering: If you'd make a core dump when the issue happens, how big would such a core dump be with MDB? I'm afraid it would be insanely large, containing the whole database. Am I wrong?
I cranked up the memory, which not did resolve the issue, but did help, it doesn't happen nearly as often. But still, every now and again, a server demonstrates a high read IO rate and severely degraded memberOf query performance. At this point, I just have a monitoring check that alerts on slow query performance and high read I/O and go restart them when it happens, as the additional memory made the issue go from a couple of times a week to every month or three.
I did notice now that when the issue occurs the box with the slow queries does have less memory available then when it is working normally, but still a good gigabyte of free memory not being used.
Even when the systems don't completely blow up, there are occasional slower than normal queries. Typically the test query I am doing literally takes fractions of a second:
May 21 19:47:22 ldap-01 slapd[1223]: conn=849157 op=1 SEARCH RESULT tag=101 err=0 qtime=0.000015 etime=0.198042 nentries=1 text=
Every now and again for no discernible reason it might take 5 to 10 seconds.
Hi Shawn,
I've just sent you sample slapd.conf and a data ldif that illustrate the long query time on large database. Thank you very much to take a look at this !
Ulrich Windl wrote:
"Paul B. Henson" henson@acm.org schrieb am 22.05.2022 um 04:51 in Nachricht
5d343067-aef3-b499-63e3-996f3d68051e@acm.org:
On 5/11/2022 3:48 AM, Soisik Froger wrote:
Are this performance issues an expected side-effect of switching to dynlist - as the memberOf attributes are now dynamically calculated while the memberOf overlay used to writes these attributes - or
I am also having ongoing sporadic issues with memberOf performance using the new dynlist overlay. Initially, randomly a server would get into a state where any query requesting the memberOf attribute would take in excess of 30 seconds, whereas normally it would only take a fraction of a second. The symptoms were the same, free memory, no swapping, but insanely high read IO load.
I'm wondering: If you'd make a core dump when the issue happens, how big would such a core dump be with MDB? I'm afraid it would be insanely large, containing the whole database. Am I wrong?
Yes, you are wrong.
mmap'd memory isn't included in core dumps by default.
Howard Chu hyc@symas.com schrieb am 23.05.2022 um 18:04 in Nachricht
cc25e9ad-988c-fcd6-7bc0-0de6e3c530c7@symas.com:
Ulrich Windl wrote:
"Paul B. Henson" henson@acm.org schrieb am 22.05.2022 um 04:51 in Nachricht
5d343067-aef3-b499-63e3-996f3d68051e@acm.org:
On 5/11/2022 3:48 AM, Soisik Froger wrote:
Are this performance issues an expected side-effect of switching to dynlist - as the memberOf attributes are now dynamically calculated while the memberOf overlay used to writes these attributes - or
I am also having ongoing sporadic issues with memberOf performance using the new dynlist overlay. Initially, randomly a server would get into a state where any query requesting the memberOf attribute would take in excess of 30 seconds, whereas normally it would only take a fraction of a second. The symptoms were the same, free memory, no swapping, but insanely high read IO load.
I'm wondering: If you'd make a core dump when the issue happens, how big
would such a core dump be with MDB?
I'm afraid it would be insanely large, containing the whole database. Am I
wrong?
Yes, you are wrong.
mmap'd memory isn't included in core dumps by default.
OK, but that makes it hard to debug any MDB issues from the core dump unless significant parts are copied in to private memory (which I think is not the idea behind MDB).
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Ulrich Windl wrote:
Howard Chu hyc@symas.com schrieb am 23.05.2022 um 18:04 in Nachricht
cc25e9ad-988c-fcd6-7bc0-0de6e3c530c7@symas.com:
Ulrich Windl wrote:
"Paul B. Henson" henson@acm.org schrieb am 22.05.2022 um 04:51 in Nachricht
5d343067-aef3-b499-63e3-996f3d68051e@acm.org:
On 5/11/2022 3:48 AM, Soisik Froger wrote:
Are this performance issues an expected side-effect of switching to dynlist - as the memberOf attributes are now dynamically calculated while the memberOf overlay used to writes these attributes - or
I am also having ongoing sporadic issues with memberOf performance using the new dynlist overlay. Initially, randomly a server would get into a state where any query requesting the memberOf attribute would take in excess of 30 seconds, whereas normally it would only take a fraction of a second. The symptoms were the same, free memory, no swapping, but insanely high read IO load.
I'm wondering: If you'd make a core dump when the issue happens, how big
would such a core dump be with MDB?
I'm afraid it would be insanely large, containing the whole database. Am I
wrong?
Yes, you are wrong.
mmap'd memory isn't included in core dumps by default.
OK, but that makes it hard to debug any MDB issues from the core dump unless significant parts are copied in to private memory (which I think is not the idea behind MDB).
Thanks for your concern.
On May 23, 2022, at 3:10 AM, Soisik Froger soisik.froger@worteks.com wrote:
I've just sent you sample slapd.conf and a data ldif that illustrate the long query time on large database. Thank you very much to take a look at this !
Hi Soisik,
I've completed a review/test of dynlist, following your excellent example as sent off list.
Here are my findings…
As a baseline, here’s an example of a 'normal' search across a well populated user tree:
# time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 'ou=people,dc=example,dc=com' "(cn=load01-TEST1-*)" | grep dn: | wc -l 17470
real 0m0.191s user 0m0.128s sys 0m0.132s
Compared with performing a search of the memberof attribute, generated by dynast. This pulls back the members of a large group (about 25K members): # time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 'ou=people,dc=example,dc=com' "(memberof=cn=load01-test1-2,ou=groups,dc=example,dc=com)" | grep dn: | wc -l 23869 real 4m6.167s user 0m2.289s sys 0m2.037s
That's 4 minutes to search across 100K users. The other search took < 200 ms.
As you (and others) have pointed out, there's a significant performance penalty for searching attributes generated by dylist.
It's possible that we can get modest improvements within a 2.7 timeframe. Unfortunately, we don't anticipate being able to get it back into range of a 'normal' search.
So, that gets us into looking at mitigation. One approach, focus on what the client does. For example, instead of searching across the user tree for membership to a particular group, it's far more efficient to just pull back that list from the group entry itself.
This is a technique that of course we’re already familiar with. I bring it up here for the sake of argument.
The question: can we sidestep the memberOf performance problems by focusing on the the client?
Or, is there a usecase that I've missed for which there's no remedy?
Thanks
— Shawn
On 7/25/22 19:38, Shawn McKinney wrote:
So, that gets us into looking at mitigation. One approach, focus on what the client does. For example, instead of searching across the user tree for membership to a particular group, it's far more efficient to just pull back that list from the group entry itself.
In my case searching by 'memberOf' is not only done for determining the users' group memberships. It is used to also effeciently retrieve the users' attributes.
Example:
For Æ-DIR my aehostd uses 'memberOf' search to select only subsets of users indirectly referenced via user groups by so-called service groups (see https://www.ae-dir.com/docs.html#er-roles) to retrieve the passwd map data.
Another approach I was thinking about is using deref control on attribute 'member' when reading the group entry. But this would mean that I potentially read user data several times if the user is member of several groups referenced by same service groups (which is the case most times). In opposite to that a filter (|(memberOf=..)(memberOf=..)) only retrieves the user entry *once*.
And there were also issues with deref control and set-based ACLs:
https://bugs.openldap.org/show_bug.cgi?id=9800
I'm thinking about how to get rid of slapo-memberof for quite a while now...
BTW: You can access the Æ-DIR online demo via LDAPS and HTTPS and try out yourself: https://www.ae-dir.com/demo.html https://demo.ae-dir.com/web2ldap
Ciao, Michael.
Hello,
I have worked with Soisik on this particular topic. As you mentionned, in many cases we can just query directly the group. In our particular scenario, the software only allows one request for getting the mails of the users present in the dynamic group.
Thus I think it would be great to have a server solution (if possible in 2.6), especially if dynlist is officially the memberof replacement.
Regards,
David
Le 25 juillet 2022 19:38:52 GMT+02:00, Shawn McKinney smckinney@symas.com a écrit :
On May 23, 2022, at 3:10 AM, Soisik Froger soisik.froger@worteks.com wrote:
I've just sent you sample slapd.conf and a data ldif that illustrate the long query time on large database. Thank you very much to take a look at this !
Hi Soisik,
I've completed a review/test of dynlist, following your excellent example as sent off list.
Here are my findings…
As a baseline, here’s an example of a 'normal' search across a well populated user tree:
# time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 'ou=people,dc=example,dc=com' "(cn=load01-TEST1-*)" | grep dn: | wc -l 17470
real 0m0.191s user 0m0.128s sys 0m0.132s
Compared with performing a search of the memberof attribute, generated by dynast. This pulls back the members of a large group (about 25K members): # time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 'ou=people,dc=example,dc=com' "(memberof=cn=load01-test1-2,ou=groups,dc=example,dc=com)" | grep dn: | wc -l 23869 real 4m6.167s user 0m2.289s sys 0m2.037s
That's 4 minutes to search across 100K users. The other search took < 200 ms.
As you (and others) have pointed out, there's a significant performance penalty for searching attributes generated by dylist.
It's possible that we can get modest improvements within a 2.7 timeframe. Unfortunately, we don't anticipate being able to get it back into range of a 'normal' search.
So, that gets us into looking at mitigation. One approach, focus on what the client does. For example, instead of searching across the user tree for membership to a particular group, it's far more efficient to just pull back that list from the group entry itself.
This is a technique that of course we’re already familiar with. I bring it up here for the sake of argument.
The question: can we sidestep the memberOf performance problems by focusing on the the client?
Or, is there a usecase that I've missed for which there's no remedy?
Thanks
— Shawn
David Coutadeur wrote:
Hello,
I have worked with Soisik on this particular topic. As you mentionned, in many cases we can just query directly the group. In our particular scenario, the software only allows one request for getting the mails of the users present in the dynamic group.
Have you already tried using the deref control to return a mail attribute instead?
Thus I think it would be great to have a server solution (if possible in 2.6), especially if dynlist is officially the memberof replacement.
Regards,
David
Le 25 juillet 2022 19:38:52 GMT+02:00, Shawn McKinney smckinney@symas.com a écrit :
On May 23, 2022, at 3:10 AM, Soisik Froger <soisik.froger@worteks.com> wrote: I've just sent you sample slapd.conf and a data ldif that illustrate the long query time on large database. Thank you very much to take a look at this ! Hi Soisik, I've completed a review/test of dynlist, following your excellent example as sent off list. Here are my findings… As a baseline, here’s an example of a 'normal' search across a well populated user tree: # time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 'ou=people,dc=example,dc=com' "(cn=load01-TEST1-*)" | grep dn: | wc -l 17470 real 0m0.191s user 0m0.128s sys 0m0.132s Compared with performing a search of the memberof attribute, generated by dynast. This pulls back the members of a large group (about 25K members): # time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 'ou=people,dc=example,dc=com' "(memberof=cn=load01-test1-2,ou=groups,dc=example,dc=com)" | grep dn: | wc -l 23869 real 4m6.167s user 0m2.289s sys 0m2.037s That's 4 minutes to search across 100K users. The other search took < 200 ms. As you (and others) have pointed out, there's a significant performance penalty for searching attributes generated by dylist. It's possible that we can get modest improvements within a 2.7 timeframe. Unfortunately, we don't anticipate being able to get it back into range of a 'normal' search. So, that gets us into looking at mitigation. One approach, focus on what the client does. For example, instead of searching across the user tree for membership to a particular group, it's far more efficient to just pull back that list from the group entry itself. This is a technique that of course we’re already familiar with. I bring it up here for the sake of argument. The question: can we sidestep the memberOf performance problems by focusing on the the client? Or, is there a usecase that I've missed for which there's no remedy? Thanks — Shawn
On 7/25/2022 10:38 AM, Shawn McKinney wrote:
As you (and others) have pointed out, there's a significant performance penalty for searching attributes generated by dylist.
I'm still seeing performance issues with queries that simply return memberOf, with no reference to it in the actual search filter.
For example, this query which searches on the static uid attribute and returns memberOf:
time ldapsearch -H ldapi:/// uid=henson memberOf
Most of the time it completes in fractions of a second:
real 0m0.187s user 0m0.005s sys 0m0.003s
But sometimes it takes 5 seconds, 10 seconds, or even more. These extremely slow response times coordinate with a high read I/O percentage on the server and the high number of page faults on the slapd process.
When I first deployed 2.5, sometimes the server would get into a state where every query that requested memberOf would take in excess of 30 seconds to return until the server was restarted. I cranked up the memory on the servers and at this point I have had no more reoccurrences of that behavior, but I am still regularly seeing occasional slow performance on the queries and high read I/O percentages.
The servers have way more memory now than they should need to fully cache the entire database:
# du -sh /var/symas/openldap-data-cpp 2.6G /var/symas/openldap-data-cpp
# free -m total used free shared buff/cache available Mem: 4818 703 124 0 3991 3831 Swap: 2047 136 1911
I haven't been able to correlate the slow response times with any other external criteria such as updates or query load. Sometimes it's just slow 8-/. We never saw this problem under 2.4 which used the previous implementation of dynlist to generate memberOf. I definitely appreciate the ability to query on dynamic memberOf that was added in 2.5, but it would be nice to sort out this performance issue.
If you have the ability to do so and your kernel is v5+, out-of-hours experiment with disabling system swap entirely (vm.swapiness=0 and 'swapoff -a' ) and then simulate a run of requests you'd expect to encounter the page errors with and see if it stops happening.
Depending on what else is going on this might not be a production-compatible strategy, but it will either rule that out as a potential flaw/bug in 2.5 and/or give you a workaround until it's fixed.
________________________________ From: Paul B. Henson henson@acm.org Sent: 26 July 2022 00:16 To: openldap-technical@openldap.org openldap-technical@openldap.org Subject: Re: dynlist vs memberof performance issues
On 7/25/2022 10:38 AM, Shawn McKinney wrote:
As you (and others) have pointed out, there's a significant performance penalty for searching attributes generated by dylist.
I'm still seeing performance issues with queries that simply return memberOf, with no reference to it in the actual search filter.
For example, this query which searches on the static uid attribute and returns memberOf:
time ldapsearch -H ldapi:/// uid=henson memberOf
Most of the time it completes in fractions of a second:
real 0m0.187s user 0m0.005s sys 0m0.003s
But sometimes it takes 5 seconds, 10 seconds, or even more. These extremely slow response times coordinate with a high read I/O percentage on the server and the high number of page faults on the slapd process.
When I first deployed 2.5, sometimes the server would get into a state where every query that requested memberOf would take in excess of 30 seconds to return until the server was restarted. I cranked up the memory on the servers and at this point I have had no more reoccurrences of that behavior, but I am still regularly seeing occasional slow performance on the queries and high read I/O percentages.
The servers have way more memory now than they should need to fully cache the entire database:
# du -sh /var/symas/openldap-data-cpp 2.6G /var/symas/openldap-data-cpp
# free -m total used free shared buff/cache available Mem: 4818 703 124 0 3991 3831 Swap: 2047 136 1911
I haven't been able to correlate the slow response times with any other external criteria such as updates or query load. Sometimes it's just slow 8-/. We never saw this problem under 2.4 which used the previous implementation of dynlist to generate memberOf. I definitely appreciate the ability to query on dynamic memberOf that was added in 2.5, but it would be nice to sort out this performance issue.
********** Internet Email Confidentiality Footer **********
Privileged/Confidential information may be contained in this message. If you have received this message in error you should, without taking any copies, immediately delete it from your system and kindly notify the sender by reply email. Opinions, conclusions and other information in this message that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. All contracts for goods and services are subject to the terms and conditions of our relevant company, a copy of which is available on request or by visiting the company’s website.
Amari Metals Ltd. Registered in England and Wales number 2023155. Registered Office: Parkway House, Unit 6, Parkway Industrial Estate, Pacific Avenue, Wednesbury, WS10 7WP.
Hi both,
Just to let you know that with my dataset I see a consistently slow (10-20 secs) lookup of a user's memberOf group membership using dynamic dynlist overlays on 2.5 and 2.6 compared with the old "memberof" overlay.
My dynlist config is as below:
dn: olcOverlay={2}dynlist,olcDatabase={1}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcDynListConfig olcOverlay: {2}dynlist olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames structuralObjectClass: olcDynListConfig
It's fairly common to use the memberOf attribute to look up the groups a particular user is a member of e.g. for access control purposes. We use it routinely for this purpose on our Shibboleth IdPs which use OpenLDAP as a datasource.
In terms of data size/complexity we've currently got around 500K users (each with their own unique group in a separate OU) with the bulk of our group membership managed by Grouper. We've got about 100K groups in there(!) ranging in size from a single "placeholder" member up to about 200K members for our applicant/alumni groups.
The total DIT size (dumped via slapcat) is just under 2GB and our (freshly slapcat'ed and slapadd'ed mdb database size is currently sitting at 5.1GB). We're running on Centos 7 boxes with 12vCPUs and 64GB RAM each which I think should be enough CPU/memory resource for our usage?
On a related note for a production environment should we be targeting the 2.5 or 2.6 branch? I'd like to upgrade our production servers off 2.4 before the start of next term.
Kind regards,
Mark
On 26/07/2022 00:16, Paul B. Henson wrote:
This email was sent to you by someone outside the University. You should only click on links or attachments if you are certain that the email is genuine and the content is safe.
On 7/25/2022 10:38 AM, Shawn McKinney wrote:
As you (and others) have pointed out, there's a significant performance penalty for searching attributes generated by dylist.
I'm still seeing performance issues with queries that simply return memberOf, with no reference to it in the actual search filter.
For example, this query which searches on the static uid attribute and returns memberOf:
time ldapsearch -H ldapi:/// uid=henson memberOf
Most of the time it completes in fractions of a second:
real 0m0.187s user 0m0.005s sys 0m0.003s
But sometimes it takes 5 seconds, 10 seconds, or even more. These extremely slow response times coordinate with a high read I/O percentage on the server and the high number of page faults on the slapd process.
When I first deployed 2.5, sometimes the server would get into a state where every query that requested memberOf would take in excess of 30 seconds to return until the server was restarted. I cranked up the memory on the servers and at this point I have had no more reoccurrences of that behavior, but I am still regularly seeing occasional slow performance on the queries and high read I/O percentages.
The servers have way more memory now than they should need to fully cache the entire database:
# du -sh /var/symas/openldap-data-cpp 2.6G /var/symas/openldap-data-cpp
# free -m total used free shared buff/cache available Mem: 4818 703 124 0 3991 3831 Swap: 2047 136 1911
I haven't been able to correlate the slow response times with any other external criteria such as updates or query load. Sometimes it's just slow 8-/. We never saw this problem under 2.4 which used the previous implementation of dynlist to generate memberOf. I definitely appreciate the ability to query on dynamic memberOf that was added in 2.5, but it would be nice to sort out this performance issue.
-- /****************************
Mark Cairney ITI Enterprise Services Information Services University of Edinburgh
Tel: 0131 650 6565 Email: Mark.Cairney@ed.ac.uk
*******************************/
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.
Hello,
Le 26/07/2022 à 00:04, Howard Chu a écrit :
David Coutadeur wrote:
Hello,
I have worked with Soisik on this particular topic. As you mentionned, in many cases we can just query directly the group. In our particular scenario, the software only allows one request for getting the mails of the users present in the dynamic group.
Have you already tried using the deref control to return a mail attribute instead?
Thanks for the idea.
I suppose it could work indeed. I have not tested it yet, but it could be interresting to have the performance associated to such request.
However, in my specific use case, it is not possible to configure a LDAP control in the software.
Thus I think it would be great to have a server solution (if possible in 2.6), especially if dynlist is officially the memberof replacement.
Regards,
David
Le 25 juillet 2022 19:38:52 GMT+02:00, Shawn McKinney smckinney@symas.com a écrit :
On May 23, 2022, at 3:10 AM, Soisik Froger <soisik.froger@worteks.com> wrote: I've just sent you sample slapd.conf and a data ldif that illustrate the long query time on large database. Thank you very much to take a look at this ! Hi Soisik, I've completed a review/test of dynlist, following your excellent example as sent off list. Here are my findings… As a baseline, here’s an example of a 'normal' search across a well populated user tree: # time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 'ou=people,dc=example,dc=com' "(cn=load01-TEST1-*)" | grep dn: | wc -l 17470 real 0m0.191s user 0m0.128s sys 0m0.132s Compared with performing a search of the memberof attribute, generated by dynast. This pulls back the members of a large group (about 25K members): # time ldapsearch -x -LLL -H ldap://m01:389 -D "dc=example,dc=com" -W -s sub -b 'ou=people,dc=example,dc=com' "(memberof=cn=load01-test1-2,ou=groups,dc=example,dc=com)" | grep dn: | wc -l 23869 real 4m6.167s user 0m2.289s sys 0m2.037s That's 4 minutes to search across 100K users. The other search took < 200 ms. As you (and others) have pointed out, there's a significant performance penalty for searching attributes generated by dylist. It's possible that we can get modest improvements within a 2.7 timeframe. Unfortunately, we don't anticipate being able to get it back into range of a 'normal' search. So, that gets us into looking at mitigation. One approach, focus on what the client does. For example, instead of searching across the user tree for membership to a particular group, it's far more efficient to just pull back that list from the group entry itself. This is a technique that of course we’re already familiar with. I bring it up here for the sake of argument. The question: can we sidestep the memberOf performance problems by focusing on the the client? Or, is there a usecase that I've missed for which there's no remedy? Thanks — Shawn
I second that, actually I was going to suggest doing exactly this test.
If possible, you should try to do it somewhere else that is not the production server, maybe a VM after restoring the database?
That would allow you to carefully execute tests without interference from other requests coming in.
If you're in Linux, strace might give you more details of what is happening in terms of system calls, but it will generate a loot of messages so you should try it as last resource.
Last, there are applications that might help you understand what is going on:
http://cnmonitor.sourceforge.net/
https://github.com/jcollie/openldap_exporter (this will require to install and configure Prometheus first)
Hope this helps.
Regards,
Alceu
On 26/07/2022 05:56, Wilkinson, Hugo (IT Dept) wrote:
If you have the ability to do so and your kernel is v5+, out-of-hours experiment with disabling system swap entirely (vm.swapiness=0 and 'swapoff -a' ) and then simulate a run of requests you'd expect to encounter the page errors with and see if it stops happening.
Depending on what else is going on this might not be a production-compatible strategy, but it will either rule that out as a potential flaw/bug in 2.5 and/or give you a workaround until it's fixed.
On 7/26/2022 1:56 AM, Wilkinson, Hugo (IT Dept) wrote:
If you have the ability to do so and your kernel is v5+, out-of-hours experiment with disabling system swap entirely (vm.swapiness=0 and 'swapoff -a' ) and then simulate a run of requests you'd expect to encounter the page errors with and see if it stops happening.
We use to run our LDAP servers under Gentoo with a 5.4 kernel at the time, but currently are using Rocky Linux 8 which comes with a modified 4.18 kernel.
I did actually run a production system with no swap for a while, it still occasionally had slow responses for queries requesting the memberOf attribute.
Part of the problem is I don't expect to ever see these slow response times 8-/. As of yet I have not figured out what is going on when they are slow, I definitely don't have a reproducible query test case to make it happen <sigh>.
We were running 2.4 under rocky for a while before upgrading to 2.5 and never saw this problem, so I'm pretty sure it's the result of some change in the dynlist implementation.
On Jul 25, 2022, at 2:51 PM, David Coutadeur david.coutadeur@gmail.com wrote:
I have worked with Soisik on this particular topic. As you mentionned, in many cases we can just query directly the group. In our particular scenario, the software only allows one request for getting the mails of the users present in the dynamic group.
Thus I think it would be great to have a server solution (if possible in 2.6), especially if dynlist is officially the memberof replacement.
Hello,
Some work on the dynlist code of late to boost performance.
So, it might be worthwhile to give it a try. MR576[1] in the next release.
— Shawn
[1] https://git.openldap.org/openldap/openldap/-/merge_requests/576
On Jan 7, 2023, at 8:44 PM, Paul B. Henson henson@acm.org wrote:
On Tue, Jan 03, 2023 at 11:44:30AM -0600, Shawn McKinney wrote:
Some work on the dynlist code of late to boost performance.
So, it might be worthwhile to give it a try. MR576[1] in the next release.
Is this going to hit the 2.5 LTS train or just 2.6?
Yes (both)
On Mon, Jan 09, 2023 at 09:26:44AM -0600, Shawn McKinney wrote:
Yes (both)
Sweet :). We've had some performance issues with the new dynlist implementation since we upgraded to 2.5, I look forward to trying this out.
We're currently using dynlist to add the memberOf attribute to users. One thing that was nice about the new dynlist implementation in 2.5 was that it allows searching on the dynamic memberOf attribute which we couldn't do in 2.4. Looking at the commit diff, there's one part that says:
"To enable filter evaluation on the dynamic list, the configuration must be changed to explicitly map the dynamic attributes to be filtered"
This just applies to groups created dynamically, right, not static objects that get dynamic attributes added? Our current config is:
dynlist-attrset groupOfURLs memberURL member+memberOf@groupOfNames
It doesn't require any changes to keep working with a searchable memberOf attribute?
Thanks much...
On Jan 9, 2023, at 1:39 PM, Paul B. Henson henson@acm.org wrote:
On Mon, Jan 09, 2023 at 09:26:44AM -0600, Shawn McKinney wrote:
Yes (both)
Sweet :). We've had some performance issues with the new dynlist implementation since we upgraded to 2.5, I look forward to trying this out.
We're currently using dynlist to add the memberOf attribute to users. One thing that was nice about the new dynlist implementation in 2.5 was that it allows searching on the dynamic memberOf attribute which we couldn't do in 2.4. Looking at the commit diff, there's one part that says:
"To enable filter evaluation on the dynamic list, the configuration must be changed to explicitly map the dynamic attributes to be filtered"
This just applies to groups created dynamically, right, not static objects that get dynamic attributes added? Our current config is:
dynlist-attrset groupOfURLs memberURL member+memberOf@groupOfNames
It doesn't require any changes to keep working with a searchable memberOf attribute?
Correct, no changes to the config of dynlist.
Thanks much...
On Mon, Jan 09, 2023 at 11:39:47AM -0800, Paul B. Henson wrote:
We're currently using dynlist to add the memberOf attribute to users. One thing that was nice about the new dynlist implementation in 2.5 was that it allows searching on the dynamic memberOf attribute which we couldn't do in 2.4. Looking at the commit diff, there's one part that says:
"To enable filter evaluation on the dynamic list, the configuration must be changed to explicitly map the dynamic attributes to be filtered"
This just applies to groups created dynamically, right, not static objects that get dynamic attributes added? Our current config is:
The latest manpage update should make it clearer how dynamic *lists* differ from dynamic *groups*. And yes, no need to change config for groups.
If you can give the current 2.5/2.6 branch a go, or if you decide to wait until we get a call for testing out, feedback is always welcome.
Regards,
On Tue, Jan 10, 2023 at 01:41:58PM +0100, Ondřej Kuzník wrote:
The latest manpage update should make it clearer how dynamic *lists* differ from dynamic *groups*. And yes, no need to change config for groups.
Cool, thank you both for the clarification.
If you can give the current 2.5/2.6 branch a go, or if you decide to wait until we get a call for testing out, feedback is always welcome.
Will do, thanks again...
openldap-technical@openldap.org