Hi all,
After a period of testing I upgraded our LDAP servers from 2.1.30 to 2.3.37. Many things were done, changed replication from slurpd to syncrepl, changed backend from ldbm to hdb, even changed the structure of the tree a bit, almost everything works great :)
Except some searches are slower !
I have around 160K dn's with around 3.2M attrs in a server with 4G memory with a tree looking like this (simplified):
o=sp | ou=users | namespace=domain.com | uid=username
Now, when I do this search:
# time ldapsearch -b "o=sp" "mail=user@domain.com" dn
dn: uid=user,namespace=domain.com,ou=users,o=sp
real 0m0.005s user 0m0.000s sys 0m0.000s
But when I do this search:
# time ldapsearch -b "ou=users,o=sp" "mail=user@domain.com" dn
dn: uid=user,namespace=domain.com,ou=users,o=sp
real 0m0.190s user 0m0.000s sys 0m0.000s
(even after a couple of searches, the numbers stay the same)
That's 50 times slower ! Right now all machines using LDAP are doing fine, but I'd like the searches with a different base than o=sp to be the same speed (I expected them to be even quicker because there's less to search, or am I wrong ?)
(BTW, on our old LDAP, searches within o=sp were equally quick as searches in a different basedn, around 0.009 s real)
I have indexed o, ou and namespace with pres and eq (and mail with pres, eq and sub), so I don't see a problem there. I still have to look a bit more into DB_CONFIG tuning, but still, is it normal behaviour of OpenLDAP 2.3.37 with hdb backend that searches in a subtree are slower than in the base of the tree ?
Just before sending this mail I also tried searching like this:
# time ldapsearch -b "namespace=domain.com,ou=users,o=sp" "mail=user@domain.com" dn
dn: uid=user,namespace=domain.com,ou=users,o=sp
real 0m0.015s user 0m0.000s sys 0m0.000s
So that's in between both previous searches.. strange..
thanks & regards,
Leon de Rooij leon@scarlet-internet.nl
On 7/31/07, Leon de Rooij leon@scarlet-internet.nl wrote:
Hi all,
After a period of testing I upgraded our LDAP servers from 2.1.30 to 2.3.37. Many things were done, changed replication from slurpd to syncrepl, changed backend from ldbm to hdb, even changed the structure of the tree a bit, almost everything works great :)
Except some searches are slower !
I have around 160K dn's with around 3.2M attrs in a server with 4G memory with a tree looking like this (simplified):
I have indexed o, ou and namespace with pres and eq (and mail with pres, eq and sub), so I don't see a problem there. I still have to look a bit more into DB_CONFIG tuning, but still, is it normal behaviour of OpenLDAP 2.3.37 with hdb backend that searches in a subtree are slower than in the base of the tree ?
With a relatively large database, tuning can have quite a big impact on behaviour (if the directory is busy). You really need to provide information on what tuning you have done (both in DB_CONFIG, and with cachesize and idlcachesize for this database in slapd.conf). You most likely need at least 100MB cachesize set in DB_CONFIG ... and hdb likes at least 3* idlcachesize as (entry) cachesize (as discussed in the slapd-hdb man page).
Regards, Buchan
Buchan,
Thank you very much ! I don't see how I could have missed the idlcachesize.. After setting it to 3x the cachesize (2nd and later) searches take .005s to complete :-)
I see I have a lot of reading to do still about how to set sane defaults for these options.
Thanks again,
Leon
On Tue, 2007-07-31 at 15:43 +0200, Buchan Milne wrote:
On 7/31/07, Leon de Rooij leon@scarlet-internet.nl wrote:
Hi all,
After a period of testing I upgraded our LDAP servers from 2.1.30 to 2.3.37. Many things were done, changed replication from slurpd to syncrepl, changed backend from ldbm to hdb, even changed the structure of the tree a bit, almost everything works great :)
Except some searches are slower !
I have around 160K dn's with around 3.2M attrs in a server with 4G memory with a tree looking like this (simplified):
I have indexed o, ou and namespace with pres and eq (and mail with pres, eq and sub), so I don't see a problem there. I still have to look a bit more into DB_CONFIG tuning, but still, is it normal behaviour of OpenLDAP 2.3.37 with hdb backend that searches in a subtree are slower than in the base of the tree ?
With a relatively large database, tuning can have quite a big impact on behaviour (if the directory is busy). You really need to provide information on what tuning you have done (both in DB_CONFIG, and with cachesize and idlcachesize for this database in slapd.conf). You most likely need at least 100MB cachesize set in DB_CONFIG ... and hdb likes at least 3* idlcachesize as (entry) cachesize (as discussed in the slapd-hdb man page).
Regards, Buchan
--On Tuesday, July 31, 2007 11:09 AM -0400 Aaron Richton richton@nbcs.rutgers.edu wrote:
eq and sub), so I don't see a problem there. I still have to look a bit more into DB_CONFIG tuning, but still, is it normal behaviour of
After modifying DB_CONFIG -- and perhaps you already have made intermediate edits -- make sure to stop slapd and run db_recover.
Why? If DB_CONFIG changes, slapd will automatically re-initialize the DB environment.
--Quanah
-- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
--On Tuesday, July 31, 2007 8:34 AM -0700 Quanah Gibson-Mount quanah@zimbra.com wrote:
--On Tuesday, July 31, 2007 11:09 AM -0400 Aaron Richton richton@nbcs.rutgers.edu wrote:
eq and sub), so I don't see a problem there. I still have to look a bit more into DB_CONFIG tuning, but still, is it normal behaviour of
After modifying DB_CONFIG -- and perhaps you already have made intermediate edits -- make sure to stop slapd and run db_recover.
Why? If DB_CONFIG changes, slapd will automatically re-initialize the DB environment.
[on a restart]. I.e., stopping slapd and then starting it is sufficient. There is no need to run db_recover.
--Quanah
-- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount wrote:
--On Tuesday, July 31, 2007 8:34 AM -0700 Quanah Gibson-Mount quanah@zimbra.com wrote:
--On Tuesday, July 31, 2007 11:09 AM -0400 Aaron Richton richton@nbcs.rutgers.edu wrote:
eq and sub), so I don't see a problem there. I still have to look a bit more into DB_CONFIG tuning, but still, is it normal behaviour of
After modifying DB_CONFIG -- and perhaps you already have made intermediate edits -- make sure to stop slapd and run db_recover.
Why? If DB_CONFIG changes, slapd will automatically re-initialize the DB environment.
[on a restart]. I.e., stopping slapd and then starting it is sufficient. There is no need to run db_recover.
You don't need a restart if setting via cn=config
--On Tuesday, July 31, 2007 9:43 PM +0100 Gavin Henry ghenry@suretecsystems.com wrote:
Quanah Gibson-Mount wrote:
--On Tuesday, July 31, 2007 8:34 AM -0700 Quanah Gibson-Mount quanah@zimbra.com wrote:
--On Tuesday, July 31, 2007 11:09 AM -0400 Aaron Richton richton@nbcs.rutgers.edu wrote:
eq and sub), so I don't see a problem there. I still have to look a bit more into DB_CONFIG tuning, but still, is it normal behaviour of
After modifying DB_CONFIG -- and perhaps you already have made intermediate edits -- make sure to stop slapd and run db_recover.
Why? If DB_CONFIG changes, slapd will automatically re-initialize the DB environment.
[on a restart]. I.e., stopping slapd and then starting it is sufficient. There is no need to run db_recover.
You don't need a restart if setting via cn=config
Eh, yes, you do. slapd does not (and cannot) modify the BDB environment set by DB_CONFIG while it is running. Which is what this discussion is about. You are correct that you don't have to restart slapd to make changes to the entry cache setting or idlcachesetting (both in slapd.conf) via cn=config.
--Quanah
-- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Thanks for the advice, though I didn't make changes to DB_CONFIG yet, after the initial slapadd.
regards,
Leon
On Tue, 2007-07-31 at 11:09 -0400, Aaron Richton wrote:
eq and sub), so I don't see a problem there. I still have to look a bit more into DB_CONFIG tuning, but still, is it normal behaviour of
After modifying DB_CONFIG -- and perhaps you already have made intermediate edits -- make sure to stop slapd and run db_recover.
openldap-software@openldap.org
-
Aaron Richton -
Buchan Milne -
Gavin Henry -
Leon de Rooij -
Quanah Gibson-Mount