I've been running some tests with OpenLDAP 2.4.11 lately, with an existing application that will use the LDAP as its backend for storing users and roles.
Using bdb, performance is excellent. With hdb on the other hand, the application is rendered unusable unless the searched entry is already cached, as one particular search takes minutes to execute otherwise, whereas in bdb the same search quite fast regardless. The specific search is subtree (scope=2 deref=0) filtered on "o" and "objectclass" like so: (&(objectClass=organization)(o=123456)).. Both attributes have equality indexes. The whole directory has just over 4 million entries, and about 500.000 of those are in the context being searched thanks to the scope. The strange thing is how bdb is fast from search one, while hdb with hdb it takes minutes to complete unless the entry is already cached. I've tried configuring caches according to the faq-o-matic. The docs mention that hdb requires a bigger idl cache in particular, to have good search performance, so I've tried various cache sizes, up to 1 million for idl- and 330.000 for the entry caches, but this seems to make slapd dig so deeply into virtual memory that I'm not sure if it actually does more harm than good.
As for DB_CONFIG, I've reused the same DB_CONFIG for both backends and configured according to the faq-o-matic as far as I can tell, then doubled (and then some) the size I came up with to be on the safe side with regards to overhead and such. dn2id with pagesize 4096 and approx 9300 internal pages, and id2entry with pagesize 16k and 300 internal pages and I've allocated 256mb to the dbcache which should be a lot more than the ~50mb minimum required I ended up with for internal pages and any overhead. Also tried bumping this to 512MB but as the faq-o-matic entry suggests, this doesn't seem to have any effect really.
Now, there's no requirement to use hdb as the application doesn't need subtree renames and bdb works great. I just want check that my config isn't horribly wrong, or that I'm overlooking something vital for hdb not required for bdb..
tamarin p wrote:
I've been running some tests with OpenLDAP 2.4.11 lately, with an existing application that will use the LDAP as its backend for storing users and roles.
Using bdb, performance is excellent. With hdb on the other hand, the application is rendered unusable unless the searched entry is already cached, as one particular search takes minutes to execute otherwise, whereas in bdb the same search quite fast regardless. The specific search is subtree (scope=2 deref=0) filtered on "o" and "objectclass" like so: (&(objectClass=organization)(o=123456)).. Both attributes have equality indexes. The whole directory has just over 4 million entries, and about 500.000 of those are in the context being searched thanks to the scope. The strange thing is how bdb is fast from search one, while hdb with hdb it takes minutes to complete unless the entry is already cached. I've tried configuring caches according to the faq-o-matic. The docs mention that hdb requires a bigger idl cache in particular, to have good search performance, so I've tried various cache sizes, up to 1 million for idl- and 330.000 for the entry caches, but this seems to make slapd dig so deeply into virtual memory that I'm not sure if it actually does more harm than good.
Also check your dncachesize, the DN cache tends to be the critical element in back-hdb's performance. Still, if the filter attributes are indexed it should not be so slow...
Now, there's no requirement to use hdb as the application doesn't need subtree renames and bdb works great. I just want check that my config isn't horribly wrong, or that I'm overlooking something vital for hdb not required for bdb..
Hard to say at the moment.
2008/8/26 Howard Chu hyc@symas.com
Also check your dncachesize, the DN cache tends to be the critical element in back-hdb's performance. Still, if the filter attributes are indexed it should not be so slow...
Wasn't aware of this parameter, and couldn't find it in the administrator guide. Tried setting it to about the same as cachesize but it doesn't seem to have a huge effect. The first time the search runs after server-start-up, it still takes minutes to complete. Interestingly enough, CPU usage is very low during this time, below 10%, and even below 5% most of the time. iostat reports about 3.4% iowait.
Is there a way to verify that my indexes are working properly? As far as I can tell, they are present. There's an o.bdb and an objectclass.bdb in /var/lib/ldap. Also using loglevel 256 and not seeing any warnings in the log about unindexed attributes. I tried running slapindex again but that doesn't seem to have had any effect.
tamarin p wrote:
2008/8/26 Howard Chu <hyc@symas.com mailto:hyc@symas.com>
Also check your dncachesize, the DN cache tends to be the critical element in back-hdb's performance. Still, if the filter attributes are indexed it should not be so slow...Wasn't aware of this parameter, and couldn't find it in the administrator guide. Tried setting it to about the same as cachesize but it doesn't seem to have a huge effect. The first time the search runs after server-start-up, it still takes minutes to complete. Interestingly enough, CPU usage is very low during this time, below 10%, and even below 5% most of the time. iostat reports about 3.4% iowait.
There is this that needs updating:
http://www.openldap.org/doc/admin24/tuning.html#Performance%20Factors
and "man slapd-bdb"
openldap-software@openldap.org
-
Gavin Henry -
Howard Chu -
tamarin p