Index seems to return wrong amount of candidate causing really poor search performance
by chrichardso27＠gmail.com
Hi,
Considering the following assumptions;
- OpenLDAP version 2.4.51
- attributes objectClass and abc are indexed based on equality
- the EQUALITY of attribute abc is based on distinguishedNameMatch
- The database contains roughly 2 million entries
- 2 entries have defined the attribute abc with a dn value cn=foo,dc=bar and objectClass=someClass
- 2 entries have defined the attribute abc with a dn value cn=bar,dc=baz and objectClass=someClass
Now, the issue started with really slow search performance using objectClass=someClass & abc=cn=foo,dc=bar as filter criteria. Debugging a while seems to indicate that the objectClass filter returns roughly 2 million entries as candidates. Now, one would expect that the second filter would return only the 2 potential candidates from the abc index, or a subset of the whole database but this is not the case. The second filter also returns nearly the whole database entries as potential candidates and causes really slow query performance. Interestingly, this only occurs when attribute abc has value cn=foo,dc=bar, but for some reason for the entry having attribute abc with value cn=bar,dc=baz the query returns immediately. In both cases, the actual entries matching the search return immediately but for the problematic search "(&(objectClass=someClass)(abc=cn=foo,dc=bar))", the completion of the search takes a long time (around 15 seconds to be precise).
The issue started suddenly and wasn't a degradation of query performance over time.
Few things I have tried
- Rebuilt the whole database again
- Reindex the existing database again
- Testing with bdb and mdb as backends
- Increased cache sizes for bdb to hold the whole database in cache
- For bdb adjust the page size of the indexes according to suggestion by db_tuner
- Change the order of the filters
None of these made any difference. At the moment, there does not seem to be any good options to try. Any ideas or help would be greatly appreciated!
1 year, 9 months
HDB to MDB migration results in higher CPU usage on openldap consumers
by paul.jc＠yahoo.com
Hello,
I have migrated from HDB to MDB backend and I am seeing higher CPU usage on my MDB openldap consumers. Has anyone else seen the same?
Testing in my stage environment showed MDB to use less or the same amount of CPU than HDB - but now with real traffic and a large dataset I see sustained high CPU utilization.
My production environment has the following specs:
6 consumer servers with 8vCPU x 16G RAM
openldap version 2.4.45
Syncrepl enabled (with a single openldap provider server which is also MDB and has no issues and no high cpu).
The database has ~230K users.
data.mdb is about 1.8G in size.
MDB database directives include:
olcDbCheckpoint: 102400 10
olcDbNoSync: TRUE
The rest are defaults.
Indexing includes:
olcDbIndex: businessCategory eq
olcDbIndex: cn eq,sub
olcDbIndex: description eq
olcDbIndex: displayName eq,sub
olcDbIndex: entryCSN eq
olcDbIndex: entryUUID eq
olcDbIndex: gidNumber eq
olcDbIndex: givenName eq,sub
olcDbIndex: mail eq
olcDbIndex: member eq
olcDbIndex: memberOf eq
olcDbIndex: memberUid eq
olcDbIndex: objectClass pres,eq
olcDbIndex: sn eq,sub
olcDbIndex: uid eq,sub
olcDbIndex: uidNumber eq
olcDbIndex: uniqueMember eq
These consumer servers are used for reads only.
The initial sync with the provider is ok but once the consumers are actively handling read requests, CPU jumps to 60% usage on average.
Our HDB consumers had half the resources (4vCPU and 8GB RAM) and less than half the CPU usage (average of 25% utilization).
I have tested adding other MDB directives (writemap, mapasync, nordahead) but cannot get CPU utilization to come down close to what we see with the HDB backend.
I have also load tested in my stage environment and was unable to reproduce (MDB generally utilized the same or less resources than HDB, but never double).
There has been no change in the data or traffic between migration. We have also reverted some servers back to HDB and then back to MDB to confirm the high utilization.
Has anyone else come across this with MDB and if so, were you able to alleviate CPU utilization? I can provide more details if needed. Any input welcome.
Thanks!
Paul
2 years, 6 months
git repository build
by Nick Folino
Did a recent change break building from git? Make install is erroring off.
I wiped my repo, cloned from https://github.com/openldap/openldap.git and
tried a clean configure/make depend/make all went well. Make install not
so well. Looks like the install is executing the tools:
Making install in /home/nick/git/openldap/clients
Entering subdirectory tools
make[2]: Entering directory '/home/nick/git/openldap/clients/tools'
../../build/shtool mkdir -p /apps/openldap/bin
libtool: install: ../../build/shtool install -c -m 755 -s ldapsearch
/apps/openldap/bin/ldapsearch
ldap_sasl_interactive_bind: Can't contact LDAP server (-1)
libtool: install: ../../build/shtool install -c -m 755 -s ldapmodify
/apps/openldap/bin/ldapmodify
ldap_sasl_interactive_bind: Can't contact LDAP server (-1)
libtool: install: ../../build/shtool install -c -m 755 -s ldapdelete
/apps/openldap/bin/ldapdelete
ldap_sasl_interactive_bind: Can't contact LDAP server (-1)
libtool: install: ../../build/shtool install -c -m 755 -s ldapmodrdn
/apps/openldap/bin/ldapmodrdn
ldap_sasl_interactive_bind: Can't contact LDAP server (-1)
libtool: install: ../../build/shtool install -c -m 755 -s ldappasswd
/apps/openldap/bin/ldappasswd
ldap_sasl_interactive_bind: Can't contact LDAP server (-1)
libtool: install: ../../build/shtool install -c -m 755 -s ldapwhoami
/apps/openldap/bin/ldapwhoami
ldap_sasl_interactive_bind: Can't contact LDAP server (-1)
libtool: install: ../../build/shtool install -c -m 755 -s ldapvc
/apps/openldap/bin/ldapvc
ldap_sasl_interactive_bind: Can't contact LDAP server (-1)
libtool: install: ../../build/shtool install -c -m 755 -s ldapcompare
/apps/openldap/bin/ldapcompare
usage: #INST@1151854# [options] DN <attr:value|attr::b64value>
where:
DN Distinguished Name
attr assertion attribute
value assertion value
b64value base64 encoding of assertion value
Nick
2 years, 9 months
Resyncing basic question..
by Frédéric Goudal
Hello,
I’m wondering about the correct way to resync a directory in a push configuration…
The setup is the following (as described in the documentation) :
- on the master I have added a second suffix with the hidden property and the same suffix value.
- this suffix have an ldap backend on another computer which is the real slave.
I know the procedure to resynchronize two directory when the slave is in a pull configuration : get the csn of the master and restart the slave with the csn as argument.
But in the push configuration : do I need to restart the slave (the backend of the second suffix ) ? Or the master which deals with the hidden suffix ?
I hope my explanation are clear.
Thanks in advance.
f.g.
2 years, 9 months
how to clean old multiple CSN ?
by Frédéric Goudal
Hello,
I have an ldap master that from some reason has multiple CSN :
contextCSN: 20130927152219.157851Z#000000#001#000000
contextCSN: 20131127140429.597497Z#000000#002#000000
contextCSN: 20141208130549.278599Z#000000#004#000000
contextCSN: 20200831094837.531411Z#000000#00a#000000
Some are quite old. I don’t know why they are here.
Is there a way to clean the old Context CSN ?
f.g.
2 years, 9 months
Acl attribute access
by Marc Roos
If I have this acl:
to
dn="sendmailMTAKey=test(a)bbbbb.com,ou=eeee,ou=ddddd,ou=ccccc,dc=bbbbb,dc=
aaaaa,dc=local"
by ssf=64
dn.exact="uid=acctest,ou=ffff,ou=ddddd,ou=ccccc,dc=bbbbb,dc=aaaaa,dc=loc
al" read
I can access with this ldap search:
ldapsearch -LLL -W -s sub -b
"sendmailMTAKey=test(a)bbbbb.com,ou=eeee,ou=ddddd,ou=ccccc,dc=bbbbb,dc=aaa
aa,dc=local" -D
"uid=acctest,ou=ffff,ou=ddddd,ou=ccccc,dc=bbbbb,dc=aaaaa,dc=local" -H
ldaps://ldap.local sendmailMTAKey
If I change the acl to
to
dn="sendmailMTAKey=test(a)bbbbb.com,ou=eeee,ou=ddddd,ou=ccccc,dc=bbbbb,dc=
aaaaa,dc=local" attrs="sendmailMTAKey"
by ssf=64
dn.exact="uid=acctest,ou=ffff,ou=ddddd,ou=ccccc,dc=bbbbb,dc=aaaaa,dc=loc
al" read
The ldapsearch is not returning any object. How to resolve this?
2 years, 9 months
Acl with variable in filter
by Marc Roos
Is it possible to have a variable VAR in an acl filter?
Something like this:
to
dn="sendmailMTAKey=test(a)bbbbb.com,ou=eeee,ou=ddddd,ou=ccccc,dc=bbbbb,dc=
aaaaa,dc=local" filter="(sendmailMTAMapValue=VAR1)
by ssf=64
dn.exact="uid=VAR1,ou=ffff,ou=ddddd,ou=ccccc,dc=bbbbb,dc=aaaaa,dc=local"
read
2 years, 9 months
user | this expansion in acl
by Marc Roos
I am not getting this page. Maybe there should be an example or so. I
can use user anywhere in the acls and it will expand to the dn of the
binddn of the ldapsearch request?
user | this : resolves to the set {
"cn=User,cn=adfasdfa,cn=asdfadfa,ou=asdfasdfas,ou=adsasdfasdf",
"cn=Resource" }
How can I get only 'User' and not the whole dn also not 'cn=User'?
[1]
https://www.openldap.org/faq/data/cache/1133.html
2 years, 9 months
LMDB: iterating over the whole DB
by Vladimír Čunát
Hello.
(I sent the same message 24h ago before subscribing, but it hasn't arrived so far. I hope you won't see a duplicate later.)
We have a process that needs to analyze the contents of a whole LMDB. So far the approach was to open a read-only transaction and use a cursor to iterate over the whole contents in order. This long transaction apparently causes other concurrent process(es) to get MDB_MAP_FULL from mdb_put() even though that LMDB has plenty free pages at that moment.
That would be a big problem for us, but fortunately we don't need the analysis to be atomic and splitting that iterating into smaller transactions seems to avoid the problem. That were just experiments without understanding why exactly it happens (though I get that long transactions are generally problematic).
Could this behavior be considered a bug in LMDB? Is there a way of either distinguishing this MDB_MAP_FULL event from really full LMDB or avoiding it with confidence? (We only experimented with the number of keys accessed in one transaction.)
By the way, thanks a lot for LMDB!
--Vladimir
https://knot-resolver.cz
2 years, 9 months
