Hi list,
We’ve been doing some performance testing comparing LDBM to HDB to present the business case for an upgrade but currently getting some surprising results.
A thousand searches (run consecutively) on LDBM is taking around 70 seconds to complete, whereas the same data is taking 1.2-1.3s per search with HDB. I’ve done some basic fiddling with the DB_CONFIG file and have not appeared to change the results at all. I’ve adjusted the cache size (in DB_CONFIG) to 50 meg, 500 meg, 768 meg and 1gig and I’ve played with different txn log methods (in memory, autoremove, regular logging). I am running db_recover afterwards.
The machine specs are identical (Dual P3 1266MHz, 1.25GB RAM) but I’ve also replicated it on a Dual 3GHz Xeon w/2GB RAM. The OpenLDAP versions didn’t seem to matter (currently running on 2.3.32, also tried 2.3.30 & 2.2.26). The OS has been either Debian or Ubuntu linux. Running Berkeley DB 4.2.52.
With trace & args logging enabled (-d5) the pause is on the first line of this output:
=> hdb_dn2idl("BASE_DN") <-- this line => bdb_equality_candidates (objectClass) => key_read bdb_idl_fetch_key: [b49d1940] <= bdb_index_read: failed (-30990) <= bdb_equality_candidates: id=0, first=0, last=0 => bdb_equality_candidates (uid) => key_read bdb_idl_fetch_key: [78cb9f8c] <= bdb_index_read 1 candidates <= bdb_equality_candidates: id=1, first=666704, last=666704 bdb_search_candidates: id=1 first=666704 last=666704 entry_decode: "" <= entry_decode() => send_search_entry: conn 0 dn="cn=291232744,BASE_DN" ber_flush: 810 bytes to sd 12 <= send_search_entry: conn 0 exit.
The other line that has me worried is the 5th line (bdb_index_read: failed (-30990)), I've run slapindex to restore indices but this didn't help or remove the error message.
Can anyone shed any light as to where this lag might be coming from? Can I provide any more info to help?
Thanks Mark McDonald
On Wednesday 17 January 2007 10:56, Mark Mcdonald wrote:
Hi list,
We’ve been doing some performance testing comparing LDBM to HDB to present the business case for an upgrade but currently getting some surprising results.
Is there any reason you didn't include BDB as well ?
A thousand searches (run consecutively) on LDBM is taking around 70 seconds to complete, whereas the same data is taking 1.2-1.3s per search with HDB.
On our boxes (similar I guess yo your faster box), 770 searches takes 10 seconds total (while under production load), with 3 databases totalling ~ 1.1 million entries (or more).
I’ve done some basic fiddling with the DB_CONFIG file and have not appeared to change the results at all. I’ve adjusted the cache size (in DB_CONFIG) to 50 meg, 500 meg, 768 meg and 1gig and I’ve played with different txn log methods (in memory, autoremove, regular logging). I am running db_recover afterwards.
How many entries in the database, or how large is it (du *.bdb) ?
The machine specs are identical (Dual P3 1266MHz, 1.25GB RAM) but I’ve also replicated it on a Dual 3GHz Xeon w/2GB RAM. The OpenLDAP versions didn’t seem to matter (currently running on 2.3.32, also tried 2.3.30 & 2.2.26). The OS has been either Debian or Ubuntu linux. Running Berkeley DB 4.2.52.
With trace & args logging enabled (-d5) the pause is on the first line of this output:
=> hdb_dn2idl("BASE_DN") <-- this line => bdb_equality_candidates (objectClass) => key_read bdb_idl_fetch_key: [b49d1940] <= bdb_index_read: failed (-30990) <= bdb_equality_candidates: id=0, first=0, last=0 => bdb_equality_candidates (uid) => key_read bdb_idl_fetch_key: [78cb9f8c] <= bdb_index_read 1 candidates <= bdb_equality_candidates: id=1, first=666704, last=666704 bdb_search_candidates: id=1 first=666704 last=666704 entry_decode: "" <= entry_decode() => send_search_entry: conn 0 dn="cn=291232744,BASE_DN" ber_flush: 810 bytes to sd 12 <= send_search_entry: conn 0 exit.
The other line that has me worried is the 5th line (bdb_index_read: failed (-30990)), I've run slapindex to restore indices but this didn't help or remove the error message.
AFAIK you would get an error at the bdb_equality_candidates if you didn't have indexes.
Can anyone shed any light as to where this lag might be coming from? Can I provide any more info to help?
I think you need (more?) idlcache, configured in slapd.conf inside the database. And, if you don't have any, you also need a cachesize configured (in the same place). See the man page for slapd-hdb (or slapd-bdb). IIRC the guideline for idlcache on hdb is approx three times the cachesize (which you will have to decide on).
Regards, Buchan
openldap-software@openldap.org
-
Buchan Milne -
Mark Mcdonald