dynlist vs memberof performance issues
by Mark Cairney
Hi,
I've been out the LDAP loop for a bit but the recent discussion of the
memberof overlay on 2.5 piqued my curiosity. Having upgraded a Dev box,
removed the memberof elements from the database and replaced the
memberof overlay with dynlist the queries appear to work as expected but
are both a) slow and b) heavily CPU-intensive on the LDAP server.
2021-09-01T12:47:17.603513+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 fd=12 ACCEPT from IP=192.168.152.33:58738
(IP=129.215.17.9:636)
2021-09-01T12:47:17.687488+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 fd=12 TLS established tls_ssf=256 ssf=256
tls_proto=TLSv1.3 tls_cipher=TLS_AES_256_GCM_SHA384
2021-09-01T12:47:17.688032+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=0 SRCH base="" scope=0 deref=0
filter="(objectClass=*)"
2021-09-01T12:47:17.688470+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=0 SRCH attr=supportedSASLMechanisms
2021-09-01T12:47:17.688878+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=0 SEARCH RESULT tag=101 err=0 qtime=0.000014
etime=0.000214 nentries=1 text=
2021-09-01T12:47:17.811279+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=1 BIND dn="" method=163
2021-09-01T12:47:17.819249+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=1 RESULT tag=97 err=14 qtime=0.000030
etime=0.009084 text=SASL(0): successful result:
2021-09-01T12:47:17.908889+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=2 BIND dn="" method=163
2021-09-01T12:47:17.909836+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=2 RESULT tag=97 err=14 qtime=0.000031
etime=0.000181 text=SASL(0): successful result:
2021-09-01T12:47:17.938839+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=3 BIND dn="" method=163
2021-09-01T12:47:17.939621+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=3 BIND authcid="mcairney(a)EASE.ED.AC.UK"
authzid="mcairney(a)EASE.ED.AC.UK"
2021-09-01T12:47:17.940213+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=3 BIND
dn="uid=mcairney,ou=people,ou=central,dc=authorise-dev,dc=ed,dc=ac,dc=uk"
mech=GSSAPI bind_ssf=256 ssf=256
2021-09-01T12:47:17.940616+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=3 RESULT tag=97 err=0 qtime=0.000024
etime=0.000409 text=
2021-09-01T12:47:18.227342+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=4 SRCH
base="dc=authorise-dev,dc=ed,dc=ac,dc=uk" scope=2 deref=0
filter="(uid=mcairney)"
2021-09-01T12:47:18.227703+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=4 SRCH attr=* +
2021-09-01T12:47:31.392255+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=5 UNBIND
2021-09-01T12:47:31.460705+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=4 SEARCH RESULT tag=101 err=0 qtime=0.000031
etime=13.233679 nentries=1 text=
2021-09-01T12:47:31.461098+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 fd=12 closed
I'm guessing that as the values are computed that this will be heavier
on the CPU but it seems a bit excessive? Has anyone else noticed any
similar performance issues?
This is a relatively low-specced DEV server (2 vCPUs, 4GB RAM) so I
guess this could be a factor but there's no io waiting on the server and
no swapping?
The database is on a par in size with our Production service ( about
400K user objects with 1 group object per user and then about 80K actual
groups of users)
The config for the primary DB (ACLs and rootPW redacted) is:
dn: olcDatabase={1}mdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcMdbConfig
olcDatabase: {1}mdb
olcDbDirectory: /opt/openldap/var/openldap-data/authorise
olcSuffix: dc=authorise-dev,dc=ed,dc=ac,dc=uk
olcAddContentAcl: FALSE
olcLastMod: TRUE
olcMaxDerefDepth: 2
olcReadOnly: FALSE
olcSecurity: ssf=1
olcSecurity: update_ssf=112
olcSecurity: simple_bind=64
olcSizeLimit: unlimited
olcSyncUseSubentry: FALSE
olcTimeLimit: unlimited
olcMonitoring: TRUE
olcDbEnvFlags: writemap
olcDbEnvFlags: nometasync
olcDbNoSync: FALSE
olcDbIndex: objectClass eq
olcDbIndex: entryUUID eq
olcDbIndex: entryCSN eq
olcDbIndex: cn pres,eq,sub
olcDbIndex: uid pres,eq,sub
olcDbIndex: uidNumber pres,eq
olcDbIndex: gidNumber pres,eq
olcDbIndex: eduniType eq
olcDbIndex: gecos pres,eq,sub
olcDbIndex: eduniCategory eq
olcDbIndex: mail pres,eq,sub
olcDbIndex: eduniSchoolCode eq
olcDbIndex: eduniIDStatus eq
olcDbIndex: eduniCollegeCode eq
olcDbIndex: eduniOrgCode eq
olcDbIndex: memberOf pres,eq
olcDbIndex: eduniLibraryBarcode pres,eq
olcDbIndex: eduniOrganisation pres,eq,sub
olcDbIndex: eduniServiceCode pres,eq
olcDbIndex: krbName pres,eq
olcDbIndex: eduPersonAffiliation pres,eq
olcDbIndex: eduPersonEntitlement pres,eq
olcDbIndex: sn pres,eq,sub
olcDbIndex: eduniIdmsId pres,eq
olcDbIndex: member pres,eq
olcDbIndex: memberUid pres,eq
olcDbIndex: eduniRefNo pres,eq
olcDbIndex: eduniTitle pres,eq
olcDbIndex: title pres,eq,sub
olcDbIndex: eduniCardNumber pres,eq
olcDbIndex: eduniYearOfStudy eq
olcDbIndex: description pres,eq,sub
olcDbIndex: givenName pres,eq,sub
olcDbIndex: aliasedObjectName eq
olcDbIndex: yubiKeyId pres,eq
olcDbIndex: isMemberOf pres,eq
olcDbIndex: hasMember pres,eq
olcDbIndex: proxyAddresses pres,eq,sub
olcDbMaxReaders: 96
olcDbMaxSize: 32212254720
olcDbMode: 0600
olcDbSearchStack: 16
structuralObjectClass: olcMdbConfig
dn: olcOverlay={0}syncprov,olcDatabase={1}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcConfig
objectClass: top
objectClass: olcSyncProvConfig
olcOverlay: {0}syncprov
structuralObjectClass: olcSyncProvConfig
dn: olcOverlay={1}accesslog,olcDatabase={1}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcAccessLogConfig
olcOverlay: {1}accesslog
olcAccessLogDB: cn=accesslog
olcAccessLogOps: writes
olcAccessLogPurge: 02+00:00 00+04:00
olcAccessLogSuccess: TRUE
structuralObjectClass: olcAccessLogConfig
dn: olcOverlay={2}dynlist,olcDatabase={1}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcDynListConfig
olcOverlay: {2}dynlist
olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames
structuralObjectClass: olcDynListConfig
--
/****************************
Mark Cairney
ITI Enterprise Services
Information Services
University of Edinburgh
Tel: 0131 650 6565
Email: Mark.Cairney(a)ed.ac.uk
*******************************/
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.
2 months, 1 week
LDAP VLV throws error "Other sort requests already in progress"
by rathore_pushpendra21＠yahoo.co.in
The bounty expires in 7 days. Answers to this question are eligible for a +200 reputation bounty. NG. wants to draw more attention to this question.
I am trying to implement pagination in LDAP using vlv, using reference from document https://docs.ldap.com/ldap-sdk/docs/javadoc/com/unboundid/ldap/sdk/contro...
it is working fine with single thread, but when try with multiple threads concurrently upto 5 threads it works fine, but as number of threads increased only 5 threads can run successfully exceed threads got failed with below error message:
LDAPException(resultCode=51 (busy), numEntries=0, numReferences=0, diagnostiMessage='Other sort requests already in progress', ldapSDKVersion=5.1.1..
I am using OpenLDAP, Unboundid api for connection with Java. About data size it is around 100k.
Tried with single connection and multiple connections(with multiple concurrent threads) getting same error in both cases.
Tried to synchronize block for fetching data.
On exception, make thread to wait and try again. All above things didn't worked, threads cannot fetch data from LDAP. After trying to close and reconnect connection as described in https://www.openldap.org/lists/openldap-technical/201107/msg00006.html failed thread can fetch data but after retry lot of times, in my case thread retried about 2k times then it started fetching data.
Is there any better solution, retrying 2k times and getting result is not a good option.
3 months, 2 weeks
Syncrepl has stopped 24 hours ago
by Bradley T Gill
Our accesslog mdb is 2GB and syncrepl has been 'late' (according to Nagios) on all of our servers for 24 hours now. We are in the middle of Significant Incident and am asking for suggestions of what to look for in the logs? All of the syncprov_ activities in the log seem fine.
I have done a rolling restart of our servers to see if that could spark them to sync, with no positive results.
Thanks,
[cid:image001.png@01D903D9.98037410]<http://www.aep.com/>
BRADLEY T GILL | INFRASTRUCTURE ENGINEER STAFF
BGILL(a)AEP.COM<mailto:BGILL@AEP.COM> | A:8.200.3054
1 RIVERSIDE PLAZA, COLUMBUS, OH 43215
3 months, 3 weeks
dynlist dynamic attributes (memberof) work and doesnt work in filter querys from some apps
by andreas.ladanyi＠kit.edu
Using slapd 2.5 with dynlist to generate memberof.
We use sssd ldap provider with ldap_user_search_filter parameter and memberof filter and only the user which are memberof=XY are in the sssd cache. So it works as expected, since slapd 2.5
We use ldapsearch with memberof filter and it works as expected, since slapd 2.5
Iam trying out some webapps, configure the ldap filter and iam wondering because the filter with the memberof attribute will be transmitted to slapd but there is no search result in the slapd.log. If i copy the webapp ldap filter from the slapd log and try it out with ldapsearch on the webapp server i get search results.
Could somebody clearify me ?
3 months, 3 weeks
Checking users password
by Ian Porter
Hi
I have tried to change a users password either by
ldappasswd -H ldapi:/// -x -D "ADMIN ACCOUNT" -W -S "uid=USER,ou=USER,o=ORG"
or via a ldif file with ldapmodify
ldapmodify -H ldap:// -x -D "ADMIN ACCOUNT" -W -f ./password.ldif
dn: uid=USER,ou=USER,o=ORG
changetype: modify
replace: userPassword
userPassword: {SSHA}SSHAPASSWORD HERE
where the ADMIN ACCOUNT / USER etc have been replaced with the ldap cn=manager etc, but every time I try to confirm that the password has been updated via
ldapwhoami -x -W -D "uid=USER,ou=USER,o=ORG" -H ldapi:///
Enter LDAP Password:
ldap_bind: Invalid credentials (49)
I keep on getting the ldap_bind invalid credentials, I have tested the userPassword attribute via the apache directory studio and verified the userPassword is correct.
Any advice, please
Ian
<https://www.intergence.com>[cid:banner2_39fe1610-59f4-4ba9-b81b-b4e5f6191cb2.png]<https://www.intergence.com>
Intergence Systems Ltd.
The Old Coach House, Brewery Road, Pampisford, Cambridge, CB22 3HG
Tel: +44 845 226 4167
www.intergence.com<http://www.intergence.com>
<https://www.linkedin.com/company/intergence-ltd/>[cid:linkedin_68c71b8e-8abf-4fd8-b701-d259821decc2.png]<https://www.linkedin.com/company/intergence-ltd/> <https://www.youtube.com/channel/UCwaP6ynaEpd6dFmPvWAwblA> <https://www.youtube.com/channel/UCwaP6ynaEpd6dFmPvWAwblA> [cid:twitter_60a1d18f-2ce6-4f9a-bea8-77d45272960f.png] <https://twitter.com/intergence>
Intergence Systems Limited Disclaimer: This email may contain Copyright Material and/or sensitive or protectively marked / classified material. The email is intended for the named addressee(s). Unless you are the named addressee (or authorised to receive it for the addressee), you may not copy, use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic may be subject to recording and/or monitoring in accordance with relevant legislation. Correspondence sent to Intergence Systems Limited is treated as being sent to the organisation as a whole and may be shared within the organisation and/or legitimate and authorised external organisations to enable the matter contained therein to be dealt with appropriately and/or to comply with legislative requirements. Intergence Systems Limited does not accept service of documents by e-mail.
3 months, 4 weeks
Push replication issue for the pwdHistory attribute
by Daniel Hoffend
Hello,
I'm using a master ldap instance with a push replication instance to
external slaves using the ldap backend on Debian 11 (2.4.57) and I came
across some replication issues that forces me to drop the slave dbs and
do a manually fullsync everything this error occurs.
The problem
===========
I know that replication and maintaining a password policy is a
complicated task, especially since the ppolicy overlay must be loaded
and active in every instance (master, push instance, slave). This
problem leads to interesting challanges.
First, I encountered a problem where pwdChangedTime would be duplicate
because the ppolicy overlay of the push instance and the back_ldap/slave
instance would like to set it (which leads to a duplicate attribute
error). To fix problem I backported the patch [1] to my local version of
the slapd packages. After this problem was fixed, I've encountered the
next problematic attribute: "pwdHistory". I've play around with some
syncrepl settings, but in the end, it seems to be a similar issue. It
looks likes the pwdHistory attribute is not yet present on the slave and
both instances (push and slave) try to add the pwdHistory Attributes
which leads to a problem where both entries collied (pwdHistory: value
#0 already exists). For whatever reason pwdHistory doesn't show up as
modified field on the slave in the MOD request. But anyways. Something
seems to be wrong, and it could a similar replication issue compared
with pwdChangedTime
I've lookup into the change history of the ppolicy.c file in the 2.5 and
2.6 branch but couldn't find a newer commit that would address this issue.
Does anyone has encountered a similar issue? I've not played around with
the 2.5 or 2.6 version, but looking at the code, I've either not seen a
fix or the problem might still exist, hopefully I am wrong. Any suggestions?
--
best regards
Daniel Hoffend
[1]
https://github.com/openldap/openldap/commit/7a34f46d1cabe8e80937d5167b621...
Setup
=====
Host Master
- Debian 11 slapd 2.4.57+dfsg-3
- slapd master instance with cn=config
- push replikation instance with simple config (syncrepl from localhost,
write to backend ldap)
Host Slave
- Debian 11 slapd 2.4.57+dfsg-3
- Readonly slave
On all 3 instances ppolicy is enabled otherwise the operational
attributes would be not known/available and sync of locked accounts or
per account selected password policy assignment wouldn't work.
PUSH Replication Instance
=========================
database ldap
[...]
overlay ppolicy
ppolicy_default "cn=default,ou=policies,dc=example,dc=org"
syncrepl rid=__RID__
provider=ldap://localhost:389/
binddn="cn=replication,ou=system,dc=example,dc=org"
bindmethod=simple
credentials="secret"
searchbase="dc=example,dc=org"
type=refreshAndPersist
schemachecking=off
retry="5 12 60 +"
attrs="*,memberOf,pwdPolicySubentry,pwdChangedTime,pwdAccountLockedTime,pwdHistory,creatorsName,createTimestamp,modifiersName,modifyTimestamp"
---
Nov 3 00:00:45 ldapmaster slapd[2308611]: syncrepl_message_to_entry:
rid=016 DN: uid=user1,ou=accounts,dc=example,dc=org, UUID:
db720f56-df0d-103c-8635-9543ccd6e97d
Nov 3 00:00:45 ldapmaster slapd[2308611]: syncrepl_entry: rid=016
LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_ADD) csn=(none) tid a0a89700
Nov 3 00:00:45 ldapmaster slapd[2308611]: syncrepl_entry: rid=016
be_search (0)
Nov 3 00:00:45 ldapmaster slapd[2308611]: syncrepl_entry: rid=016
uid=user1,ou=accounts,dc=example,dc=org
Nov 3 00:00:45 ldapmaster slapd[2308611]: syncrepl_null_callback :
error code 0x14
Nov 3 00:00:45 ldapmaster slapd[2308611]: syncrepl_entry: rid=016
be_modify uid=user1,ou=accounts,dc=example,dc=org (20)
Nov 3 00:00:45 ldapmaster slapd[2308611]: syncrepl_entry: rid=016
be_modify failed (20)
Nov 3 00:00:45 ldapmaster slapd[2308611]: do_syncrepl: rid=016 rc 20
retrying
SLAVE LDAP Server
=================
database mdb
[...]
overlay ppolicy
ppolicy_default "cn=default,ou=policies,dc=example,dc=org"
---
Nov 3 00:00:45 ldapslave slapd[2221506]: conn=176499 op=59513 SRCH
base="uid=user1,ou=accounts,dc=example,dc=org" scope=0 deref=0
filter="(objectClass=*)"
Nov 3 00:00:45 ldapslave slapd[2221506]: conn=176499 op=59513 SEARCH
RESULT tag=101 err=0 nentries=1 text=
Nov 3 00:00:45 ldapslave slapd[2221506]: conn=176500 op=59513 MOD
dn="uid=user1,ou=accounts,dc=example,dc=org"
Nov 3 00:00:45 ldapslave slapd[2221506]: conn=176500 op=59513 MOD
attr=structuralObjectClass creatorsName createTimestamp userPassword
pwdChangedTime memberOf entryCSN modifiersName modifyTimestamp
Nov 3 00:00:45 ldapslave slapd[2221506]: conn=176500 op=59513 RESULT
tag=103 err=20 text=modify/add: pwdHistory: value #0 already exists
4 months
Question regarding meta proxy server
by lists＠zxt10d.de
Hello list,
I set-up a meta proxy server (hopefully that term is correct), to
combine information from an AD-server and an ldap-server.
For some reasons (historical?) both are running and provide mostly
different attributes.
In principle it seems to work, as I get information from both defined
systems - but I don't get all information as there're not the same
attributes on AD and ldap.
For example, the unique userID on AD is "cn" and "sAMAccountName", on
ldap its "uid".
So, when doing a:
ldapsearch "(uid=USERID)" -H ldap://metaldapserver... I will get
information from the defined ldap-server only, and when doing a
ldapsearch "(cn=USERID)" -H ldap://metaldapserver... I will get
information from the defined AD-server only.
Is there still a way to combine both infomation?
Debian Bullseye with Symas OpenLDAP Server, slapd 2.6.3
Cheers,
Torsten
4 months
SSSD looking for password policy: "unrecognized control"
by jarett＠bioteam.net
Hi,
I am attempting to have SSSD do logins to my OpenLDAP 2.6.3 installation, however, I get "permission denied" when trying to log in because SSSD is asking for a password policy, which the server does not appear to have by default. Notably, we don't really care what "policy" the server will claim to have, because password authentication is delegated via SASL to another server which ensures strong passwords. So I just need something that will "get past" whatever checks SSSD is doing. What LDIF config can I add to my configuration to allow SSSD to let users log in properly?
The error from `journalctl -u slapd` is shown below:
Nov 01 18:16:58 ldapserver00 slapd[105481]: conn=2239 fd=11 ACCEPT from IP=10.8.8.202:41516 (IP=0.0.0.0:389)
Nov 01 18:16:58 ldapserver00 slapd[105481]: conn=2239 op=0 SRCH base="" scope=0 deref=0 filter="(objectClass=*)"
Nov 01 18:16:58 ldapserver00 slapd[105481]: conn=2239 op=0 SRCH attr=* altServer namingContexts supportedControl supportedExtension supportedFeatures supportedLDAPVersion>
Nov 01 18:16:58 ldapserver00 slapd[105481]: conn=2239 op=0 SEARCH RESULT tag=101 err=0 qtime=0.000020 etime=0.000271 nentries=1 text=
Nov 01 18:16:58 ldapserver00 slapd[105481]: conn=2239 op=1 BIND dn="cn=admin,dc=clab,dc=lab" method=128
Nov 01 18:16:58 ldapserver00 slapd[105481]: slap_global_control: unrecognized control: 1.3.6.1.4.1.42.2.27.8.5.1
Nov 01 18:16:58 ldapserver00 slapd[105481]: conn=2239 op=1 BIND dn="cn=admin,dc=clab,dc=lab" mech=SIMPLE bind_ssf=0 ssf=0
Nov 01 18:16:58 ldapserver00 slapd[105481]: conn=2239 op=1 RESULT tag=97 err=0 qtime=0.000028 etime=0.000136 text=
Nov 01 18:16:58 ldapserver00 slapd[105481]: get_filter: conn 2239 unknown attribute type=sudoHost (17)
Nov 01 18:16:58 ldapserver00 slapd[105481]: get_ssa: conn 2239 unknown attribute type=sudoHost (17)
Nov 01 18:16:58 ldapserver00 slapd[105481]: conn=2239 op=2 SRCH base="ou=users,dc=clab,dc=lab" scope=2 deref=0 filter="(&(?objectClass=sudoRole)(|(&(!(?sudoHost=*))(cn=de>
Nov 01 18:16:58 ldapserver00 slapd[105481]: conn=2239 op=2 SRCH attr=objectClass objectClass cn sudoCommand sudoHost sudoUser sudoOption sudoRunAs sudoRunAsUser sudoRunAs>
Nov 01 18:16:58 ldapserver00 slapd[105481]: conn=2239 op=2 SEARCH RESULT tag=101 err=0 qtime=0.000016 etime=0.000326 nentries=0 text=
TIA!
4 months, 3 weeks
upgrading openldap 2.4 to 2.5
by CHIROSSEL, Olivier
Hi
I have an architecture with 2 openldap masters in mirormode ( refreshAndPersist ).
and few consumers ( refreshAndPersist )
The provisionning api and consumer request masters via vip ( sorry server configuration ),
It's complex for me to upgrade all the openldap in the same operation.
Can i upgrade only the 2 masters in 2.5, and the consumers in a second time ?
The replication between master in 2.5 and consumers in 2.4 work's fine ?
Thank's in advance for your replies.
Regards
Olivier
4 months, 3 weeks
