We are using Symas OpenLDAP 2.4.12.1 on a Red Hat EL 5.2 64-bit with 8GB of memory.
My DB_CONFIG settings are: set_cachesize 3 0 2 set_lk_max_locks 3000 set_lk_max_objects 1500 set_lk_max_lockers 1500 set_lg_regionmax 262144 set_lg_bsize 2097152 set_lg_dir /opt/symas/var/openldap-logs/amwater set_flags DB_LOG_AUTOREMOVE set_tas_spins 1
The 10 million user load finished in about 18 hours. I believe the primary issue is the fact that we have a single disk on the system. I assume that if I were to move the database and log files off to separate disks I would get much better performance.
Thanks, Pete
On Wed, Mar 18, 2009 at 11:26 PM, Quanah Gibson-Mount quanah@zimbra.comwrote:
--On Wednesday, March 18, 2009 1:25 PM -0400 Pete Giesin < pgiesin@hubcitymedia.com> wrote:
I am trying to perform some benchmarking against OpenLDAP. So far I have
ran my tests against a 100K and 1Million entry database and I have had rather decent numbers. My final set of tests were to be ran against a 10Million entry database. Unfortunately, I am having difficult loading the database with this many entries. I have generated 10 1Million LDIF files. I am using "slapadd -c -q -v -l <file>" to import each file. The first 2 files took approximately 15 minutes each to load. The remaining 8 files are taking progressively longer and longer. So much longer that I anticipate the entire proceess to take well over 24 hours. My question is is there anything I can do to increase the performance of slapadd. I assume that since slapd is not running at this point that the normal DB_CONFIG and slapd.conf settings do not have much affect.
What are the settings in your DB_CONFIG file? It is absolutely critical to the performance of slapadd. What version of OpenLDAP are you using? What version of BDB? What operating system?
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc
Zimbra :: the leader in open source messaging and collaboration
Pete Giesin wrote:
We are using Symas OpenLDAP 2.4.12.1 on a Red Hat EL 5.2 64-bit with 8GB of memory.
My DB_CONFIG settings are: set_cachesize 3 0 2 set_lk_max_locks 3000 set_lk_max_objects 1500 set_lk_max_lockers 1500 set_lg_regionmax 262144 set_lg_bsize 2097152 set_lg_dir /opt/symas/var/openldap-logs/amwater set_flags DB_LOG_AUTOREMOVE set_tas_spins 1
The 10 million user load finished in about 18 hours. I believe the primary issue is the fact that we have a single disk on the system. I assume that if I were to move the database and log files off to separate disks I would get much better performance.
With slapadd -q there are no transactions, so there will not be any log file traffic. It sounds like your DB is very much larger than your available memory, so you're simply being limited by your disk speed. Running vmstat while you load would confirm this.
Thanks, Pete
On Wed, Mar 18, 2009 at 11:26 PM, Quanah Gibson-Mount <quanah@zimbra.com mailto:quanah@zimbra.com> wrote:
--On Wednesday, March 18, 2009 1:25 PM -0400 Pete Giesin <pgiesin@hubcitymedia.com <mailto:pgiesin@hubcitymedia.com>> wrote: I am trying to perform some benchmarking against OpenLDAP. So far I have ran my tests against a 100K and 1Million entry database and I have had rather decent numbers. My final set of tests were to be ran against a 10Million entry database. Unfortunately, I am having difficult loading the database with this many entries. I have generated 10 1Million LDIF files. I am using "slapadd -c -q -v -l <file>" to import each file. The first 2 files took approximately 15 minutes each to load. The remaining 8 files are taking progressively longer and longer. So much longer that I anticipate the entire proceess to take well over 24 hours. My question is is there anything I can do to increase the performance of slapadd. I assume that since slapd is not running at this point that the normal DB_CONFIG and slapd.conf settings do not have much affect. What are the settings in your DB_CONFIG file? It is absolutely critical to the performance of slapadd. What version of OpenLDAP are you using? What version of BDB? What operating system? --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
--On Thursday, March 19, 2009 10:11 AM -0700 Howard Chu hyc@symas.com wrote:
Pete Giesin wrote:
We are using Symas OpenLDAP 2.4.12.1 on a Red Hat EL 5.2 64-bit with 8GB of memory.
My DB_CONFIG settings are: set_cachesize 3 0 2 set_lk_max_locks 3000 set_lk_max_objects 1500 set_lk_max_lockers 1500 set_lg_regionmax 262144 set_lg_bsize 2097152 set_lg_dir /opt/symas/var/openldap-logs/amwater set_flags DB_LOG_AUTOREMOVE set_tas_spins 1
The 10 million user load finished in about 18 hours. I believe the primary issue is the fact that we have a single disk on the system. I assume that if I were to move the database and log files off to separate disks I would get much better performance.
With slapadd -q there are no transactions, so there will not be any log file traffic. It sounds like your DB is very much larger than your available memory, so you're simply being limited by your disk speed. Running vmstat while you load would confirm this.
Since you've already loaded your database, you can of course easily gather the statistics on how big your DB is, and thus what the required DB_CONFIG settings would be to load it via slapadd in the future.
Cd <data dir>
du -c -h *.bdb
Whatever that total is, is the total size you need DB_CONFIG to be to get optimal performance. In another note, there is no reason to split the BDB cache into multiple memory regions (3 0 2) on 64-bit systems, and in fact past tests I did showed that this decreases performance. I'd suggest (X 0 1) instead (where X is your updated value based on the results of the du -c -h *.bdb command.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Since you've already loaded your database, you can of course easily gather the statistics on how big your DB is, and thus what the required DB_CONFIG settings would be to load it via slapadd in the future.
Cd <data dir>
du -c -h *.bdb
Whatever that total is, is the total size you need DB_CONFIG to be to get optimal performance. In another note, there is no reason to split the BDB cache into multiple memory regions (3 0 2) on 64-bit systems, and in fact past tests I did showed that this decreases performance. I'd suggest (X 0
- instead (where X is your updated value based on the results of the
du -c -h *.bdb command.
Also have a read of these two pages for more info:
http://www.openldap.org/faq/data/cache/1075.html http://www.oracle.com/technology/documentation/berkeley-db/db/ref/am_conf/ca...
Thanks.
openldap-software@openldap.org
-
Gavin Henry -
Howard Chu -
Pete Giesin -
Quanah Gibson-Mount