Index seems to return wrong amount of candidate causing really poor search performance
by chrichardso27＠gmail.com
Hi,
Considering the following assumptions;
- OpenLDAP version 2.4.51
- attributes objectClass and abc are indexed based on equality
- the EQUALITY of attribute abc is based on distinguishedNameMatch
- The database contains roughly 2 million entries
- 2 entries have defined the attribute abc with a dn value cn=foo,dc=bar and objectClass=someClass
- 2 entries have defined the attribute abc with a dn value cn=bar,dc=baz and objectClass=someClass
Now, the issue started with really slow search performance using objectClass=someClass & abc=cn=foo,dc=bar as filter criteria. Debugging a while seems to indicate that the objectClass filter returns roughly 2 million entries as candidates. Now, one would expect that the second filter would return only the 2 potential candidates from the abc index, or a subset of the whole database but this is not the case. The second filter also returns nearly the whole database entries as potential candidates and causes really slow query performance. Interestingly, this only occurs when attribute abc has value cn=foo,dc=bar, but for some reason for the entry having attribute abc with value cn=bar,dc=baz the query returns immediately. In both cases, the actual entries matching the search return immediately but for the problematic search "(&(objectClass=someClass)(abc=cn=foo,dc=bar))", the completion of the search takes a long time (around 15 seconds to be precise).
The issue started suddenly and wasn't a degradation of query performance over time.
Few things I have tried
- Rebuilt the whole database again
- Reindex the existing database again
- Testing with bdb and mdb as backends
- Increased cache sizes for bdb to hold the whole database in cache
- For bdb adjust the page size of the indexes according to suggestion by db_tuner
- Change the order of the filters
None of these made any difference. At the moment, there does not seem to be any good options to try. Any ideas or help would be greatly appreciated!
1 year, 9 months
Use-case specific changes to LMDB
by martin＠urbackup.org
This post outlines a few changes to LMDB I had to do to make it work in a specific use case. I’d like to see those changes upstream, but I understand that they may be/are not relevant for e.g. OpenLDAP.
The use case is multiple databases on disks with long running large write transactions.
1. Option to not use custom memory allocator/page pool
LMDB has a custom malloc() implementation that re-uses pages (me_dpages). I understand that this improves the performance at bit (depending on the malloc implementation). But there should at least be the option to not do that (for many reasons). I would even make not using it the default.
2. Large transactions and spilling
In a large write transaction, it will use a lot of memory per default (512MiB) which won’t get freed when the transaction commits (see 1.). If one has a lot of databases it uses a lot of memory that never gets freed.
Alternatively, one can use MDB_WRITEMAP, but (i) per default Linux isn’t tuned to delay writing pages to disk and (ii) before commit LMDB has to remove a dirty bit, so each page is written twice.
Both problems would be fixed by making when pages get spilled configurable (mt_dirty_room as MDB_IDL_UM_MAX currently) and reducing the default non-spill memory amount for at least the MDB_WRITEMAP case. If this memory amount is low mt_spill_pgs gets sorted often so maybe this needs to be converted to a different data structure (e.g. red-black tree).
3. LMDB causes crashes if database is corrupted
If the database is corrupted it can cause the application to crash. I have fixed those cases when they (randomly) occurred. Properly fixing this would probably be best done with some fuzzing.
4. Allow LMDB to reside on a device
I used dm-cache to improve LMDB read performance. It needed a bit of adjustment to get the correct size of the device via ioctl BLKGETSIZE64.
--
I’ve fixed those issues w.r.t. my application. If there is interest in any of those application specific changes, I’ll clean them up and post them.
2 years, 4 months
HDB to MDB migration results in higher CPU usage on openldap consumers
by paul.jc＠yahoo.com
Hello,
I have migrated from HDB to MDB backend and I am seeing higher CPU usage on my MDB openldap consumers. Has anyone else seen the same?
Testing in my stage environment showed MDB to use less or the same amount of CPU than HDB - but now with real traffic and a large dataset I see sustained high CPU utilization.
My production environment has the following specs:
6 consumer servers with 8vCPU x 16G RAM
openldap version 2.4.45
Syncrepl enabled (with a single openldap provider server which is also MDB and has no issues and no high cpu).
The database has ~230K users.
data.mdb is about 1.8G in size.
MDB database directives include:
olcDbCheckpoint: 102400 10
olcDbNoSync: TRUE
The rest are defaults.
Indexing includes:
olcDbIndex: businessCategory eq
olcDbIndex: cn eq,sub
olcDbIndex: description eq
olcDbIndex: displayName eq,sub
olcDbIndex: entryCSN eq
olcDbIndex: entryUUID eq
olcDbIndex: gidNumber eq
olcDbIndex: givenName eq,sub
olcDbIndex: mail eq
olcDbIndex: member eq
olcDbIndex: memberOf eq
olcDbIndex: memberUid eq
olcDbIndex: objectClass pres,eq
olcDbIndex: sn eq,sub
olcDbIndex: uid eq,sub
olcDbIndex: uidNumber eq
olcDbIndex: uniqueMember eq
These consumer servers are used for reads only.
The initial sync with the provider is ok but once the consumers are actively handling read requests, CPU jumps to 60% usage on average.
Our HDB consumers had half the resources (4vCPU and 8GB RAM) and less than half the CPU usage (average of 25% utilization).
I have tested adding other MDB directives (writemap, mapasync, nordahead) but cannot get CPU utilization to come down close to what we see with the HDB backend.
I have also load tested in my stage environment and was unable to reproduce (MDB generally utilized the same or less resources than HDB, but never double).
There has been no change in the data or traffic between migration. We have also reverted some servers back to HDB and then back to MDB to confirm the high utilization.
Has anyone else come across this with MDB and if so, were you able to alleviate CPU utilization? I can provide more details if needed. Any input welcome.
Thanks!
Paul
2 years, 6 months
Symas OpenLDAP for Linux 2.4.55-1 released
by Quanah Gibson-Mount
The latest version of Symas OpenLDAP for Linux, 2.4.55-1, is now available.
Additionally, module compilation for lastbind and noopsrch has been fixed
so they are now functional.
Regards,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
2 years, 7 months
Chain and ppolicy: what am I missing?
by Larry Rosenman
Greetings,
New poster. I've been banging my head against this wall for 3 days
straight.
I have an existing LDAP set up with 1 master and 3 replicas. We need to
add the policy overlay, and I've got that working. I'm now
trying to get the chain overlay and updateref /ppolicy_forward_updates
working,
I've followed
https://www.openldap.org/doc/admin24/overlays.html#Chaining but I'm
obviously missing <something>.
If I do a modify to one of the replicas, I get a 52 error code. I do
NOT see a successful auth (if I'm reading the debug right,
which I may not be).
This is with 2.4.42 on Ubuntu 16.04.
The replica's ARE receiving the updates, and I have the policy working.
The last remaining piece is getting the updates to be sent to the master
for ppolicy updates (or any update to a replica).
I'm obviously missing something very basic here. I'd appreciate any
pointers, help, etc.
--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 E-Mail: ler(a)lerctr.org
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
2 years, 7 months
overlay autoca in OpenLDAP 2.5
by Stefan Kania
Hello,
I play around al little bit with the OpenLDAP 2.5alpha. I'm trying the
new overlay for the certificates. I start with the configuration with
slapd.conf, because it's faster to change ;-). I started with the two
lines from the manpage:
--------
overlay autoca
caKeybits 4096
-------
The first start of the slapd failed with the error-message:
---------
Oct 20 20:39:47 ldap25 systemd[1]: slapd-current.service: Control
process exited, code=exited, status=1/FAILURE
Oct 20 20:39:47 ldap25 systemd[1]: slapd-current.service: Failed with
result 'exit-code'.
---------
I checked the config, everything was ok, I tried it a second time and
then the slapd startet without problem. This happens after neatly every
change of the parameters for this overlay. First start failed, second
start was ok without any change in the configuration.
Now, when I do an ldapsearch I see:
---------
dn: dc=example,dc=net
objectClass: domain
objectClass: dcObject
objectClass: autoCA
dc: example
cACertificate;binary::
MIIFcDCCA1igAwIBAgIJAKh3GIChqUPoMA0GCSqGSIb3DQEBCwUAMC4
...
VYd8XlDNv6d/04FDyEqKH9KAV5RMXiI9GHbQ==
---------
Then I did the following changes in my configuration:
---------
overlay autoca
caKeybits 4096
userClass inetOrgPerson
userKeybits 4096
serverClass ipHost
serverKeybits 4096
---------
Because it's a TESTSYSTEM my acl are set:
---------
access to dn.base="" by * read
access to dn.base="cn=Subschema" by * read
access to attrs=pKCS8PrivateKey by self ssf=128 write
access to *
by self write
by users read
by anonymous auth
---------
But when I create a user or a server there is no certificate. In the
manpage said:
---------
Certificates for users and servers are generated on demand using a
Search request
---------
But I never saw any certificate. As a user I search for my own Object,
but I don't see any certificate. Can I (if it works ;-) ) the
server-certificate for TLS? Where can I find some more information about
autoca.
Thanks for any help
Stefan
2 years, 7 months
Symas OpenLDAP for Linux 2.4.54-1 released
by Quanah Gibson-Mount
The latest version of Symas OpenLDAP for Linux, 2.4.54-1, is now available.
There are a few small changes for this build vs previous builds, in that
the lastbind and noopsrch overlays have been added.
Regards,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
2 years, 7 months
Modifying 'top' objectClass
by Zahoor Alizai
Can I modify the 'top' objectClass in openLDAP to incorporate some activedirectory attributes. Existing schemas in '/etc/ldap/slapd.d/cn=config/cn=schema' can be modified using ldapmodify. But i cannot find 'top' objectClass in any of these. To my knowledge, 'top' is part of system schema but can it be modified?
2 years, 7 months
ldapsearch -Y EXTERNAL not working
by Stefan Kania
Hello,
I just compiled OpenLDAP 2.5alpha on a debian 10 system. I used this howto:
https://tylersguides.com/guides/install-openldap-source-debian-stretch/
Slapd is running and I load the following ldif:
-----------------
dn: cn=config
objectClass: olcGlobal
cn: config
olcArgsFile: /opt/openldap-current/var/run/slapd.args
olcPidFile: /opt/openldap-current/var/run/slapd.pid
olcTLSCACertificateFile: /etc/ssl/certificates/demoCA/cacert.pem
olcTLSCertificateFile: /etc/ssl/certificates/ldap01-cert.pem
olcTLSCertificateKeyFile: /etc/ssl/certificates/ldap01-key.pem
olcTLSCipherSuite: TLSv1.2:HIGH:!aNULL:!eNULL
olcTLSProtocolMin: 3.3
dn: cn=schema,cn=config
objectClass: olcSchemaConfig
cn: schema
dn: cn=module,cn=config
objectClass: olcModuleList
cn: module
olcModulepath: /opt/openldap-current/libexec/openldap
olcModuleload: back_mdb.la
olcModuleload: pw-sha2.la
include: file:///opt/openldap-current/etc/openldap/schema/core.ldif
include: file:///opt/openldap-current/etc/openldap/schema/cosine.ldif
include: file:///opt/openldap-current/etc/openldap/schema/nis.ldif
include: file:///opt/openldap-current/etc/openldap/schema/inetorgperson.ldif
dn: olcDatabase=frontend,cn=config
objectClass: olcDatabaseConfig
objectClass: olcFrontendConfig
olcDatabase: frontend
olcPasswordHash: {SSHA512}
olcAccess: to * by
dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" manage
by * none
dn: olcDatabase=config,cn=config
objectClass: olcDatabaseConfig
olcDatabase: config
olcRootDN: cn=config
olcAccess: to * by
dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" manage
by * none
-----------------
When I try to do a ldapsearch with -Y EXTERNAL I get the following error:
-----------------
root@lda25:~# ldapsearch -Y EXTERNAL -H ldaps://ldap25.example.net -b
cn=config
SASL/EXTERNAL authentication started
ldap_sasl_interactive_bind: Unknown authentication method (-6)
additional info: SASL(-4): no mechanism available:
-----------------
Ldapsearch -ZZ is working:
-----------------
root@lda25:~# ldapsearch -x -ZZ -H ldap://ldap25.example.net -b
cn=config -LLL
No such object (32)
root@lda25:~# ldapsearch -x -H ldaps://ldap25.example.net -b cn=config -LLL
No such object (32)
-----------------
So ldaps and ldap+tls is working. Did I miss something during
"configure". I would like to help testing version 2.5.
Stefan
2 years, 7 months
