I have a situation where I need to delete a major branch of my DIT and reload it with a new ldif file on live systems. My current configuration is a two node multi-master running on Red Hat Enterprise 5.4 with openldap 2.22 and BDB 4.8.26.
With both masters running, when I delete the branch it takes about 1.5 hours and the reload (ldapadd) takes about 6-7 hours. I've researched the documentation on Berkeley but didn't make any headway to reduce the time. It appeared to me that BDB was doing syncs (lots and lots of disk writes). To test this, I moved the database directory of both masters to a ram drive (mount ramfs) that could hold the whole database directory with room to spare and changed the configuration to use the new location.
I reran the tests and the results were that the delete took 6 minutes and the load took about 13 minutes. This is definitely a lot faster and the other master kept up with the updating master.
So my question is there anyway to configure Berkeley to come close to this? Is there someway to disable the syncs?
Portion of slapd.conf:==================================================== database bdb suffix "dc=htc,dc=com" rootdn "cn=XXXXXXXXXXXXXXXX,dc=htc,dc=com" rootpw XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxx directory /usr/local/var/openldap-data #directory /var/local/var.ram/openldap-data cachesize 50000 dncachesize 50000 idlcachesize 150000 checkpoint 1024 5 dbnosync dirtyread dbconfig set_cachesize 0 268435456 1 dbconfig set_lg_bsize 2097152 dbconfig set_lg_regionmax 262144 dbconfig set_flags DB_LOG_AUTOREMOVE monitoring on
DB_CONFIG:===================================== set_cachesize 0 268435456 1 set_lg_bsize 2097152 set_lg_regionmax 262144 set_flags DB_LOG_AUTOREMOVE set_flags DB_TXN_WRITE_NOSYNC set_flags DB_TXN_NOSYNC
Mike Cannady Information Services Horry Telephone Cooperative (HTC) Phone: (843)369-8212 Fax..: (843)369-7195 Pager: (843)828-5899 Email: Mike.Cannady@htcinc.net
********************************************************************** HTC Disclaimer: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you. **********************************************************************
--On April 4, 2011 3:12:40 PM -0400 "Cannady, Mike" mike.cannady@htcinc.net wrote:
I have a situation where I need to delete a major branch of my DIT and reload it with a new ldif file on live systems. My current configuration is a two node multi-master running on Red Hat Enterprise 5.4 with openldap 2.22 and BDB 4.8.26.
I strongly advise using a later version of OpenLDAP for a variety of reasons. That aside, why not use a shared memory key for the BDB backing cache instead of putting it on disk?
Finally, RAM will always be faster than disk. If your database is properly configured via the threads, entry cachesize, the idl cachesize, the checkpoint frequency, and the BDB DB_CONFIG configuration for locks, lockers, and DB cachesize, then there's probably not much more you can do.
Of course, you don't provide any data that lets us know whether or not the settings you showed are valid. I.e., you don't state the number of DNs in the database, or the size of the database, or any of your stats from the BDB database.
--Quanah
Hi,
On Mon, 4 Apr 2011, Quanah Gibson-Mount wrote:
--On April 4, 2011 3:12:40 PM -0400 "Cannady, Mike" mike.cannady@htcinc.net wrote:
I have a situation where I need to delete a major branch of my DIT and reload it with a new ldif file on live systems. My current configuration is a two node multi-master running on Red Hat Enterprise 5.4 with openldap 2.22 and BDB 4.8.26.
I strongly advise using a later version of OpenLDAP for a variety of reasons. That aside, why not use a shared memory key for the BDB backing cache instead of putting it on disk?
propably because most people and packagers just default to berkeley db file based caching and the sysv shared memory caching is largely unknown.
Perhaps this sould be featured more in the documentaion.
Finally, RAM will always be faster than disk. If your database is properly configured via the threads, entry cachesize, the idl cachesize, the checkpoint frequency, and the BDB DB_CONFIG configuration for locks, lockers, and DB cachesize, then there's probably not much more you can do.
Of course, you don't provide any data that lets us know whether or not the settings you showed are valid. I.e., you don't state the number of DNs in the database, or the size of the database, or any of your stats from the BDB database.
I have seen performance get exponentially worse on linux with the linux fsync changes that came somewhere after the 2.6.18 kernels. Performance on linux has never been fully restored with later kernels.
Greetings Christian
First I want to make a correction: The openldap version is 2.4.22, I had a typo.
-----Original Message----- From: Christian Kratzer [mailto:ck-lists@cksoft.de] Sent: Tuesday, April 05, 2011 5:53 AM To: Quanah Gibson-Mount Cc: Cannady, Mike; openldap-technical@openldap.org Subject: Re: Speeding up BDB question
Hi,
On Mon, 4 Apr 2011, Quanah Gibson-Mount wrote:
--On April 4, 2011 3:12:40 PM -0400 "Cannady, Mike"
wrote:
I have a situation where I need to delete a major branch of my DIT
and
reload it with a new ldif file on live systems. My current configuration is a two node multi-master running on Red Hat
Enterprise
5.4 with openldap 2.22 and BDB 4.8.26.
I strongly advise using a later version of OpenLDAP for a variety of
reasons.
I would like to do that but I have issues with uptime and the two-node multi-master with readonly replicates off the masters that I'll address in another post.
That aside, why not use a shared memory key for the BDB backing
cache
instead
of putting it on disk?
propably because most people and packagers just default to berkeley db file based caching and the sysv shared memory caching is largely unknown.
Perhaps this sould be featured more in the documentaion.
I agree. I saw no hint of using sysv shared memory anywhere in the openldap FAQ. I assume that "DB_SYSTEM_MEM" would have to be in the DB_CONFIG file and "dbconfig set_flags DB_SYSTEM_MEM" in slapd.conf to cover a new database. Is this correct?
Finally, RAM will always be faster than disk. If your database is
properly
configured via the threads, entry cachesize, the idl cachesize, the checkpoint frequency, and the BDB DB_CONFIG configuration for locks,
lockers,
and DB cachesize, then there's probably not much more you can do.
Of course, you don't provide any data that lets us know whether or
not
the
settings you showed are valid. I.e., you don't state the number of
DNs
in
the database, or the size of the database, or any of your stats from
the
BDB
database.
I have seen performance get exponentially worse on linux with the
linux
fsync changes that came somewhere after the 2.6.18 kernels.
Performance
on linux has never been fully restored with later kernels.
Greetings Christian
I used the ram filesystem to see if there was a lot of filesystem flushing via some kind of sync. The I/Os I saw just didn't seem right for the amount of changes happening every minute.
As to the database size and such: The ldapadd was adding 51,265 DNs back into the ldap store. The preceding delete was close to that number also. The database before any of the operations had 71,886 DNs
-- Christian Kratzer CK Software GmbH Email: ck@cksoft.de Wildberger Weg 24/2 Phone: +49 7032 893 997 - 0 D-71126 Gaeufelden Fax: +49 7032 893 997 - 9 HRB 245288, Amtsgericht
Stuttgart
Web: http://www.cksoft.de/ Geschaeftsfuehrer: Christian Kratzer
********************************************************************** HTC Disclaimer: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you. **********************************************************************
--On April 5, 2011 7:58:19 AM -0400 "Cannady, Mike" mike.cannady@htcinc.net wrote:
I agree. I saw no hint of using sysv shared memory anywhere in the openldap FAQ. I assume that "DB_SYSTEM_MEM" would have to be in the DB_CONFIG file and "dbconfig set_flags DB_SYSTEM_MEM" in slapd.conf to cover a new database. Is this correct?
I'm talking about shared memory keys, which is documented in the slapd-bdb(5) and slapd-hdb(5) manual pages.
I used the ram filesystem to see if there was a lot of filesystem flushing via some kind of sync. The I/Os I saw just didn't seem right for the amount of changes happening every minute.
As to the database size and such: The ldapadd was adding 51,265 DNs back into the ldap store. The preceding delete was close to that number also. The database before any of the operations had 71,886 DNs
Then it is clear your cachesize and idlcachesize settings are all too small. I would also note you should never, *EVER*, set the dncachesize unless you are on a terribly memory restricted machine. You should always leave it at the default of infinite.
You still fail to note the size of the BDB database (du -c -h *.bdb) so again it is impossible to say if your DB_CONFIG settings are correct.
--Quanah
-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] Sent: Tuesday, April 05, 2011 12:50 PM To: Cannady, Mike; Christian Kratzer Cc: openldap-technical@openldap.org Subject: RE: Speeding up BDB question
--On April 5, 2011 7:58:19 AM -0400 "Cannady, Mike" mike.cannady@htcinc.net wrote:
I agree. I saw no hint of using sysv shared memory anywhere in the openldap FAQ. I assume that "DB_SYSTEM_MEM" would have to be in the DB_CONFIG file and "dbconfig set_flags DB_SYSTEM_MEM" in slapd.conf
to
cover a new database. Is this correct?
I'm talking about shared memory keys, which is documented in the slapd-bdb(5) and slapd-hdb(5) manual pages.
I was looking at the BDB documentation and missed it in slapd-bdb. I reconfigured both masters to use shm_key, removed dncachesize and ran ldapadd. It took only 50 minutes to load 51,265 DNs. That is a better than the 6-8 hours before. There still seemed to be a lot of small cpu usage and a lot of I/O waiting.
I used the ram filesystem to see if there was a lot of filesystem flushing via some kind of sync. The I/Os I saw just didn't seem
right
for the amount of changes happening every minute.
As to the database size and such: The ldapadd was adding 51,265 DNs back into the ldap store. The preceding delete was close to that
number
also. The database before any of the operations had 71,886 DNs
Then it is clear your cachesize and idlcachesize settings are all too small. I would also note you should never, *EVER*, set the
dncachesize
unless you are on a terribly memory restricted machine. You should
always
leave it at the default of infinite.
You still fail to note the size of the BDB database (du -c -h *.bdb)
so
again it is impossible to say if your DB_CONFIG settings are correct.
Sorry about missing the size of the database... Here it is ( 71,914 DNs ):
[root@LDAPdev1 openldap-data]# du -c -h *.bdb 35M cn.bdb 16M dn2id.bdb 3.5M entryCSN.bdb 2.2M entryUUID.bdb 18M givenName.bdb 96M id2entry.bdb 3.2M objectClass.bdb 9.0K ou.bdb 21M sn.bdb 20M uid.bdb 2.0M uniqueIdentifier.bdb 214M total
--Quanah
-- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc
Zimbra :: the leader in open source messaging and collaboration
I'll bump up the cachesize to 100,000 and idlcachesize to 300,000 and rerun my tests.
---Thanks for the help! ---Mike Cannady
********************************************************************** HTC Disclaimer: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you. **********************************************************************
--On April 5, 2011 3:02:33 PM -0400 "Cannady, Mike" mike.cannady@htcinc.net wrote:
You still fail to note the size of the BDB database (du -c -h *.bdb)
so
again it is impossible to say if your DB_CONFIG settings are correct.
Sorry about missing the size of the database... Here it is ( 71,914 DNs ):
[root@LDAPdev1 openldap-data]# du -c -h *.bdb 35M cn.bdb 16M dn2id.bdb 3.5M entryCSN.bdb 2.2M entryUUID.bdb 18M givenName.bdb 96M id2entry.bdb 3.2M objectClass.bdb 9.0K ou.bdb 21M sn.bdb 20M uid.bdb 2.0M uniqueIdentifier.bdb 214M total
So your BDB DB_CONFIG cachesize is fine at 256MB. ;)
Also, I'm not sure if you noted this before, but the "checkpoint" directive in slapd.conf/cn=config will be affecting how often you flush. If you are doing a very large amount of write activity, you may wish to change from your default of 1024KB/5 minutes to something larger (or smaller). Personally I find small frequent flushes to be better than large periodic flushes.
--Quanah
openldap-technical@openldap.org
-
Cannady, Mike -
Christian Kratzer -
Quanah Gibson-Mount