Hi,
I have directory with a group with ~220,000 users in it. This group is frequently updated (several times an hour). This appears to be causing the BDB transaction logs to grow very large; over the course of a day they grow to about 36GB. These all appear to be active; neither 'slapd_db_archive -d' or DB_LOG_AUTOREMOVE has any effect.
Is there anything that can be done about this or is this an unavoidable side-effect of the transaction system?
Thanks, Steve
--On June 26, 2008 4:45:44 PM +1000 Steve Smith ssmith@atlassian.com wrote:
Hi,
I have directory with a group with ~220,000 users in it. This group is frequently updated (several times an hour). This appears to be causing the BDB transaction logs to grow very large; over the course of a day they grow to about 36GB. These all appear to be active; neither 'slapd_db_archive -d' or DB_LOG_AUTOREMOVE has any effect.
Is there anything that can be done about this or is this an unavoidable side-effect of the transaction system?
What version of BDB are you using? Have you configured a checkpoint in slapd.conf? What version of OpenLDAP are you using?
There were some bugs in some versions of BDB and I think some version of OpenLDAP that failed to properly checkpoint, causing the issue you are seeing. A properly patched version of BDB and a current release of OpenLDAP should certainly not have this problem.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
This is the standard RHEL 5.2 build:
openldap-2.3.27-8.el5_1.3
It looks like this RPM doesn't ship with checkpointing enabled, I'll try that and see how it fares.
Thanks, Steve
Quanah Gibson-Mount wrote:
--On June 26, 2008 4:45:44 PM +1000 Steve Smith ssmith@atlassian.com wrote:
Hi,
I have directory with a group with ~220,000 users in it. This group is frequently updated (several times an hour). This appears to be causing the BDB transaction logs to grow very large; over the course of a day they grow to about 36GB. These all appear to be active; neither 'slapd_db_archive -d' or DB_LOG_AUTOREMOVE has any effect.
Is there anything that can be done about this or is this an unavoidable side-effect of the transaction system?
What version of BDB are you using? Have you configured a checkpoint in slapd.conf? What version of OpenLDAP are you using?
There were some bugs in some versions of BDB and I think some version of OpenLDAP that failed to properly checkpoint, causing the issue you are seeing. A properly patched version of BDB and a current release of OpenLDAP should certainly not have this problem.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc
Zimbra :: the leader in open source messaging and collaboration
--On June 27, 2008 1:15:10 PM +1000 Steve Smith ssmith@atlassian.com wrote:
This is the standard RHEL 5.2 build:
openldap-2.3.27-8.el5_1.3
It looks like this RPM doesn't ship with checkpointing enabled, I'll try that and see how it fares.
Well, you don't say what version of BDB, or what patches it may or may not have, which is *extremely* important. And 2.3.27 may well be an affected OpenLDAP release. There are many reasons not to use the RHEL builds of OpenLDAP for servers, that have been hashed over this list many times. I advise reading:
http://www.openldap.org/faq/data/cache/1456.html
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
The BDB version is:
Sleepycat Software: Berkeley DB 4.4.20: (January 10, 2006)
It looks like checkpointing was the issue. Having run 'slapd_db_checkpoint -1' and 'slapd_db_archive -d' almost all the log files were removed. I've setup a cron job to checkpoint and clean-out hourly.
Thanks for the tip about the RPMs, I'll look into doing a dedicated build.
Thanks, Steve
Quanah Gibson-Mount wrote:
--On June 27, 2008 1:15:10 PM +1000 Steve Smith ssmith@atlassian.com wrote:
This is the standard RHEL 5.2 build:
openldap-2.3.27-8.el5_1.3
It looks like this RPM doesn't ship with checkpointing enabled, I'll try that and see how it fares.
Well, you don't say what version of BDB, or what patches it may or may not have, which is *extremely* important. And 2.3.27 may well be an affected OpenLDAP release. There are many reasons not to use the RHEL builds of OpenLDAP for servers, that have been hashed over this list many times. I advise reading:
http://www.openldap.org/faq/data/cache/1456.html
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc
Zimbra :: the leader in open source messaging and collaboration
On Friday 27 June 2008 06:57:13 Steve Smith wrote:
The BDB version is:
Sleepycat Software: Berkeley DB 4.4.20: (January 10, 2006)
It looks like checkpointing was the issue. Having run 'slapd_db_checkpoint -1' and 'slapd_db_archive -d' almost all the log files were removed. I've setup a cron job to checkpoint and clean-out hourly.
On OpenLDAP 2.3.x, you should not need to manually checkpoint, having a checkpoint directive in the relevant database section should be sufficient.
Regards, Buchan
openldap-software@openldap.org
-
Buchan Milne -
Quanah Gibson-Mount -
Steve Smith