--On August 15, 2008 1:51:37 PM +0200 tamarin p tamarinp@gmail.com wrote:

Hello,

I'm trying to migrate a directory from OpenLDAP 2.3.x to an OpenLDAP 2.4.11 test installation. My LDAP-experience is less than extensive, but I've run slapcat on the old base, and am now trying to import it back into 2.4.11 with slapadd. The base is large enough that the import will take several days at the current rate (3gb LDIF/approx 4 million entries), and so I can't monitor the process at all times.

My problem is with the backend transaction logs, though. These are chucked out at a rate of approximately one per minute until the disk runs out of space. I did a quick google on the issue and found that you should be able to remove logs not currently involved in any transactions by running db_archive -d (http://www.oracle.com/technology/documentation/berkeley-db/db/ref/transa pp/logfile.html). I've also tried adding a set_flags DB_LOG_AUTOREMOVE in my DB_CONFIG with no luck. Then I figured I could set up a cronjob to run db_archive hourly, but when I run db_archive -d (or more to the point, slapd_db_archive2.4 -d, as I've installed using Buchan Milne's RPMs), not a single log-file is removed. Moreover, none of the *.bdb files are increasing in size while slapadd is running, only additional log-files and __db.001 etc.Does slapadd basically do all adds in one huge transaction before committing at the end, or am I doing something wrong? Can I simply erase log-files with rm on the fly instead? The checkpointing in slapd.conf is set to 256k/5min as per default. Anything else I'm missing?

What flags are you using to slapadd? What version of BDB is your openldap build linked against? It's certainly worked for me to clean up the transaction logs while slapadding.

-Quanah

--

Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

On Friday 15 August 2008 13:51:37 tamarin p wrote:

Hello,

I'm trying to migrate a directory from OpenLDAP 2.3.x to an OpenLDAP 2.4.11 test installation. My LDAP-experience is less than extensive, but I've run slapcat on the old base, and am now trying to import it back into 2.4.11 with slapadd. The base is large enough that the import will take several days at the current rate (3gb LDIF/approx 4 million entries), and so I can't monitor the process at all times.

It shouldn't take much more than 4 hours on any hardware you should be running in production ...

My problem is with the backend transaction logs, though. These are chucked out at a rate of approximately one per minute until the disk runs out of space. I did a quick google on the issue and found that you should be able to remove logs not currently involved in any transactions by running db_archive -d ( http://www.oracle.com/technology/documentation/berkeley-db/db/ref/transapp/ logfile.html). I've also tried adding a set_flags DB_LOG_AUTOREMOVE in my DB_CONFIG with no luck. Then I figured I could set up a cronjob to run db_archive hourly, but when I run db_archive -d (or more to the point, slapd_db_archive2.4 -d, as I've installed using Buchan Milne's RPMs), not a single log-file is removed.

What is your full command line ? If the working directory is not the database directory, you need to specify the path to the database directory after the -h flag.

However, for bulk imports you should use the -q flag to slapadd, which will disable transactions for the duration of the import (and it has such a big impact on import speed that one typically doesn't worry that interrupting the import will result in a corrupt database).

Moreover, none of the *.bdb files are increasing in size while slapadd is running, only additional log-files and __db.001 etc.Does slapadd basically do all adds in one huge transaction before committing at the end, or am I doing something wrong? Can I simply erase log-files with rm on the fly instead? The checkpointing in slapd.conf is set to 256k/5min as per default. Anything else I'm missing?

-q ...

Regards, Buchan