dynlist vs memberof performance issues
by Mark Cairney
Hi,
I've been out the LDAP loop for a bit but the recent discussion of the
memberof overlay on 2.5 piqued my curiosity. Having upgraded a Dev box,
removed the memberof elements from the database and replaced the
memberof overlay with dynlist the queries appear to work as expected but
are both a) slow and b) heavily CPU-intensive on the LDAP server.
2021-09-01T12:47:17.603513+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 fd=12 ACCEPT from IP=192.168.152.33:58738
(IP=129.215.17.9:636)
2021-09-01T12:47:17.687488+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 fd=12 TLS established tls_ssf=256 ssf=256
tls_proto=TLSv1.3 tls_cipher=TLS_AES_256_GCM_SHA384
2021-09-01T12:47:17.688032+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=0 SRCH base="" scope=0 deref=0
filter="(objectClass=*)"
2021-09-01T12:47:17.688470+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=0 SRCH attr=supportedSASLMechanisms
2021-09-01T12:47:17.688878+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=0 SEARCH RESULT tag=101 err=0 qtime=0.000014
etime=0.000214 nentries=1 text=
2021-09-01T12:47:17.811279+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=1 BIND dn="" method=163
2021-09-01T12:47:17.819249+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=1 RESULT tag=97 err=14 qtime=0.000030
etime=0.009084 text=SASL(0): successful result:
2021-09-01T12:47:17.908889+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=2 BIND dn="" method=163
2021-09-01T12:47:17.909836+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=2 RESULT tag=97 err=14 qtime=0.000031
etime=0.000181 text=SASL(0): successful result:
2021-09-01T12:47:17.938839+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=3 BIND dn="" method=163
2021-09-01T12:47:17.939621+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=3 BIND authcid="mcairney(a)EASE.ED.AC.UK"
authzid="mcairney(a)EASE.ED.AC.UK"
2021-09-01T12:47:17.940213+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=3 BIND
dn="uid=mcairney,ou=people,ou=central,dc=authorise-dev,dc=ed,dc=ac,dc=uk"
mech=GSSAPI bind_ssf=256 ssf=256
2021-09-01T12:47:17.940616+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=3 RESULT tag=97 err=0 qtime=0.000024
etime=0.000409 text=
2021-09-01T12:47:18.227342+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=4 SRCH
base="dc=authorise-dev,dc=ed,dc=ac,dc=uk" scope=2 deref=0
filter="(uid=mcairney)"
2021-09-01T12:47:18.227703+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=4 SRCH attr=* +
2021-09-01T12:47:31.392255+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=5 UNBIND
2021-09-01T12:47:31.460705+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 op=4 SEARCH RESULT tag=101 err=0 qtime=0.000031
etime=13.233679 nentries=1 text=
2021-09-01T12:47:31.461098+01:00 bonsai.authorise-dev.is.ed.ac.uk
slapd[30075]: conn=1019 fd=12 closed
I'm guessing that as the values are computed that this will be heavier
on the CPU but it seems a bit excessive? Has anyone else noticed any
similar performance issues?
This is a relatively low-specced DEV server (2 vCPUs, 4GB RAM) so I
guess this could be a factor but there's no io waiting on the server and
no swapping?
The database is on a par in size with our Production service ( about
400K user objects with 1 group object per user and then about 80K actual
groups of users)
The config for the primary DB (ACLs and rootPW redacted) is:
dn: olcDatabase={1}mdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcMdbConfig
olcDatabase: {1}mdb
olcDbDirectory: /opt/openldap/var/openldap-data/authorise
olcSuffix: dc=authorise-dev,dc=ed,dc=ac,dc=uk
olcAddContentAcl: FALSE
olcLastMod: TRUE
olcMaxDerefDepth: 2
olcReadOnly: FALSE
olcSecurity: ssf=1
olcSecurity: update_ssf=112
olcSecurity: simple_bind=64
olcSizeLimit: unlimited
olcSyncUseSubentry: FALSE
olcTimeLimit: unlimited
olcMonitoring: TRUE
olcDbEnvFlags: writemap
olcDbEnvFlags: nometasync
olcDbNoSync: FALSE
olcDbIndex: objectClass eq
olcDbIndex: entryUUID eq
olcDbIndex: entryCSN eq
olcDbIndex: cn pres,eq,sub
olcDbIndex: uid pres,eq,sub
olcDbIndex: uidNumber pres,eq
olcDbIndex: gidNumber pres,eq
olcDbIndex: eduniType eq
olcDbIndex: gecos pres,eq,sub
olcDbIndex: eduniCategory eq
olcDbIndex: mail pres,eq,sub
olcDbIndex: eduniSchoolCode eq
olcDbIndex: eduniIDStatus eq
olcDbIndex: eduniCollegeCode eq
olcDbIndex: eduniOrgCode eq
olcDbIndex: memberOf pres,eq
olcDbIndex: eduniLibraryBarcode pres,eq
olcDbIndex: eduniOrganisation pres,eq,sub
olcDbIndex: eduniServiceCode pres,eq
olcDbIndex: krbName pres,eq
olcDbIndex: eduPersonAffiliation pres,eq
olcDbIndex: eduPersonEntitlement pres,eq
olcDbIndex: sn pres,eq,sub
olcDbIndex: eduniIdmsId pres,eq
olcDbIndex: member pres,eq
olcDbIndex: memberUid pres,eq
olcDbIndex: eduniRefNo pres,eq
olcDbIndex: eduniTitle pres,eq
olcDbIndex: title pres,eq,sub
olcDbIndex: eduniCardNumber pres,eq
olcDbIndex: eduniYearOfStudy eq
olcDbIndex: description pres,eq,sub
olcDbIndex: givenName pres,eq,sub
olcDbIndex: aliasedObjectName eq
olcDbIndex: yubiKeyId pres,eq
olcDbIndex: isMemberOf pres,eq
olcDbIndex: hasMember pres,eq
olcDbIndex: proxyAddresses pres,eq,sub
olcDbMaxReaders: 96
olcDbMaxSize: 32212254720
olcDbMode: 0600
olcDbSearchStack: 16
structuralObjectClass: olcMdbConfig
dn: olcOverlay={0}syncprov,olcDatabase={1}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcConfig
objectClass: top
objectClass: olcSyncProvConfig
olcOverlay: {0}syncprov
structuralObjectClass: olcSyncProvConfig
dn: olcOverlay={1}accesslog,olcDatabase={1}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcAccessLogConfig
olcOverlay: {1}accesslog
olcAccessLogDB: cn=accesslog
olcAccessLogOps: writes
olcAccessLogPurge: 02+00:00 00+04:00
olcAccessLogSuccess: TRUE
structuralObjectClass: olcAccessLogConfig
dn: olcOverlay={2}dynlist,olcDatabase={1}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcDynListConfig
olcOverlay: {2}dynlist
olcDynListAttrSet: {0}groupOfURLs memberURL member+memberOf@groupOfNames
structuralObjectClass: olcDynListConfig
--
/****************************
Mark Cairney
ITI Enterprise Services
Information Services
University of Edinburgh
Tel: 0131 650 6565
Email: Mark.Cairney(a)ed.ac.uk
*******************************/
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.
2 months, 2 weeks
upgrading openldap 2.4 to 2.5
by CHIROSSEL, Olivier
Hi
I have an architecture with 2 openldap masters in mirormode ( refreshAndPersist ).
and few consumers ( refreshAndPersist )
The provisionning api and consumer request masters via vip ( sorry server configuration ),
It's complex for me to upgrade all the openldap in the same operation.
Can i upgrade only the 2 masters in 2.5, and the consumers in a second time ?
The replication between master in 2.5 and consumers in 2.4 work's fine ?
Thank's in advance for your replies.
Regards
Olivier
4 months, 4 weeks
Two notes on slapppasswd (old version)
by Ulrich Windl
Hi!
When using an old version (from 2.4.41) of slapasswd, I noticed two things:
1) Using "-h SSHA", slappasswd was asking for passwords first, then telling me: "Password generation failed for scheme SSHA: scheme not recognized"
I think "SSHA" is unique enough to recognize; and if it's not, then complain before asking for passwords.
2) When I mistyped the option as "-h '{SSHA}>'", slappasswd did not complain and produce some output.
I think if it's picky on the first case, it should also be for the second case (assuming the output "{SSHA}oTEDKWKn0fimGo6J8de0I5qRixGWJxhJ" was correct overall)
Maybe check if these problems still exist in the current version.
Regards,
Ulrich Windl
5 months
LMDB, virtual machines, and copying mdb files
by selama＠gmail.com
I am working on developing a new document-oriented (XML+JSON) database, using LMDB as an engine, and I have two questions.
1. So far, it have been working really smoothly for me. But my one customer so far for the DB is really concerned about running LMDB in a virtual environment such as Docker, when performing reads and writes. Especially when mounting volumes. Their concern is because of the following caveat:
"Do not use LMDB databases on remote filesystems, even between processes on the same host. This breaks flock() on some OSes, possibly memory map sync, and certainly sync between programs on different hosts"
I think it shouldn't be an issue with Docker, but I want to be certain.
2. We have one server that updates the database, and another server with a read-only copy of the same database. Our plan was to simply copy the mdb files from the update machine to the read only machine, but we noticed that if we copy the file immediately after writing, the copy may end up being corrupted. My solution was to suspend all writing and wait few minutes before writing, to make sure everything back from memory, and I'm also using the "sync" command (not sure if it does anything here). It seem to be working, but I wonder if there is a more robust way of doing that? And also, is it safe to overwrite to the read-only server while it performs read transactions to the current file (or maybe rename it and copy to a new file with the same name)?
5 months
Read speeds, bottleneck, correct understanding?
by Sam Dave
Hi,
If my program reads through an LMDB database apparently slow at first run, but suddenly runs much quicker at the second run (which does exactly the same thing), can't I already say with confidence that my choice of LMDB - in this case its builtin memory mapping functionality - is already paying off in terms of read speeds?
I mean, that alone (exact same thing runs much faster the second time) already PROVES that the bottleneck was reading from disk as opposed to memory, right? I'm looking for excuses for patting myself on the back for investing in LMDB.
Regards,
Sam
5 months
cn=config TLS Configuration Problem
by Timothy Stonis
Hi,
I am trying to setup an OpenLDAP 2.6.3 server and I’d like to only use olc configuration (no slapd.conf file). So far things are going okay, but I’m having a problem with TLS configuration. I am able to enable TLS using a self-signed certificate without any problem, however, if I try to disable TLS using the following LDIF:
dn: cn=config
changetype: modify
delete: olcTLSCertificateFile
-
delete: olcTLSCertificateKeyFile
-
I get the following error:
modifying entry "cn=config"
ldap_modify: Server is unwilling to perform (53)
I enabled debugging and cannot seem to see the error. I have also tried reordering the entries, doing one at a time, disabling ldaps:// binding, etc but nothing seems to work. If I just remove the certificate and/or key files, then the server does not start. Is enabling TLS a one way street? Or, should I just use slapd.conf?
As a second question, I read in an article online that there is a way to store the TLS cert(s) and key in the LDAP database itself. However, I cannot find any info on that in the documentation. Can anyone shed some light on that?
Thank you in advance!
Tim
5 months
RE25 testing call (2.5.14)
by Quanah Gibson-Mount
This is the first testing call for OpenLDAP 2.5.14. Depending on the
results, this may be the only testing call.
Generally, get the code for RE25:
<https://git.openldap.org/openldap/openldap/-/archive/OPENLDAP_REL_ENG_2_5...>
Extract, configure, and build.
Execute the test suite (via make test) after it is built. Optionally, cd
tests && make its to run through the regression suite.
Thanks!
OpenLDAP 2.5.14 Engineering
Fixed client tools to remove 'h' and 'p' options (ITS#9917,ITS#8618)
Fixed ldapsearch memory leak with controls (ITS#9860)
Fixed libldap ldif_open_urlto check for failure (ITS#9904)
Fixed libldap ldap_url_parsehosts check for failure (ITS#9904)
Fixed slapd memory leak with olcAuthIDRewrite (ITS#6035)
Fixed slapd transactions extended operations cleanup after write
(ITS#9892)
Fixed slapd-mdb max number of index databases to 256 (ITS#9895)
Fixed slapd-monitor to free remembered cookies (ITS#9339)
Fixed slapo-deref memory leak (ITS#9924)
Fixed slapo-dynlist to ignore irrelevant objectClasses (ITS#9897)
Fixed slapo-remoteauth memory leaks (ITS#9438)
Build Environment
Fixed build process to not use gmake specific features
(ITS#9894)
Fixed slapo-otp testdir creation (ITS#9437)
Fixed slapd-tester memory leak (ITS#9908)
Fixed usage of non-standard C syntax (ITS#9898, ITS#9899,
ITS#9901)
Fixed usage of bashism (ITS#9900)
Documentation
Fixed slapo-unique(5) to clarify when quoting should be
used (ITS#9915)
Regards,
Quanah
5 months
Attribute Cert. schema (RFC 5755)
by Pascal Jakobi
Hi there
I am looking for an RFC 5755 (attribute certificates profile) schema file.
I thought it was in pmi.schema, but it appears that no, unless I am
missing sthing.
Before creating one from the RFC, I would like to be sure it does not
exist somewhere I couldn't find.
Thanks in adv.
P
5 months, 1 week
SyncRepl fails object class 'organization' requires attribute 'o'
by Sander Smeenk
Hi,
I'm trying to set up SyncRepl between two servers. When the SyncRepl
client connects and tries to start it logs:
| Entry (dc=example,dc=nl): object class 'organization' requires attribute 'o'
| syncrepl_null_callback : error code 0x41
| syncrepl_entry: rid=000 be_add dc=example,dc=nl failed (65)
There is only one such entry in the db and it has an 'o' attribute:
| dn: dc=example,dc=nl
| objectClass: top
| objectClass: dcObject
| objectClass: organization
| dc: example
| o: example.nl
| structuralObjectClass: organization
| [ .. create, modify, entry uuids etc ..]
What could i be missing here?
Regards,
-Sander.
--
| Inside every older person is a younger person wondering what happened.
| 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7 FBD6 F3A9 9442 20CC 6CD2
5 months, 1 week
replacement of memberof by dynlist.. questions..
by Frédéric Goudal
Hello,
We have to install a product which use ldap and that seems to need memberof overlay.
As I have read this overlay is deprecated is cause trouble with replication.
So I have dug to found a replacement solution, and what I have found is to add something like that :
In the olcDynamicList
olcDlAttrSet: myPerson labeledURI myMemberOf
And in each user <user> :
labeledURI: ldap:///ou=groups,dc=example,dc=com??sub?(&(objectclass=posixgroup) (memberuid=<user>))
I find this way quite heavy to deal with, adding such attribute to every user (1), but we can do it.
My other problem is that the myMemberOf may be really long to compute at each request (and for stupid historic reason some old programs do qyery on the full user set of atttributes).
So I intend to add a proxycache. But I have a questiion concerning the templates :
if I add the following template (myMemberOf=*) will it cache only the requests that are exactly (myMemberOf=XXX) or requests that contains the pattern like (&(Status=xxx)(myMemberOf=yyy)) ?
Thanks.
f.g.
Note 1 : it would be nice that we could define thinks like in a single place :
labeledURI: ldap:///ou=groups,dc=example,dc=com??sub?(&(objectclass=posixgroup) (memberuid=%uid%))
where %uid% would be the uid attribute value of the considered object. Or do I miss solething ?
—
Frédéric Goudal
Ingénieur Système, DSI Bordeaux-INP
+33 556 84 23 11
5 months, 1 week
