Ralf Narozny wrote:
Hello,
I'm trying to migrate about 19 million entries from OpenLDAP 2.0 to the new 2.3. Conversion and such things are done, but inserting the data takes days.
I hope someone can point me to some helpful direction, because 22 Entries/sec is not too good, especially not for real life use...
I'm using the following to insert the data slapadd -q -v -c
My DB_CONFIG looks like this:
set_cachesize 1 524288000 1 set_lg_regionmax 262144 set_lg_bsize 2097152
Our OpenLDAP version is 2.3.27.
The speed is like following (took that from the output of my insertion script):
The gradual slowdown you're seeing indicates that the BDB cache is too small.
Since you have two CPUs you should set "tool-threads 2" in your slapd.conf.
You should not use a presence index on objectClass, just equality. Every object is required to have an objectClass attribute, so presence indexing on it is just a waste of time.
As I noted just a few hours ago on this list http://www.openldap.org/lists/openldap-software/200611/msg00051.html
you're unlikely to be able to configure sufficient cache to get good performance with a 19 million entry DB on a 32 bit server. With only ~3GB of working memory available to a process, your performance here is going to be limited to the speed of your disk drives.
Finished run at: 1162437948 Run took 620309 seconds for 18914119 entries Avg.: 30,4914 entries/second
We are using the following machine:
Linux ldaprep4 2.6.15.3 #1 SMP Mon Feb 13 09:18:43 CET 2006 i686 GNU/Linux MemTotal: 5975412 kB SwapTotal: 2150152 kB 2 * Intel(R) Pentium(R) III CPU family 1133MHz
The slapd.conf is as follows (Don't mind the /tmp as path, I changed that ;-)):
include /tmp/etc/openldap/schema/core.schema include /tmp/etc/openldap/schema/freenet.schema pidfile /tmp/var/ldap/run/slapd.pid argsfile /tmp/var/ldap/run/slapd.args modulepath /tmp/lib moduleload back_bdb.la access to * by * write loglevel 0 sizelimit 10000 timelimit 3600 cachesize 1000000 backend bdb
####################################################################### # BDB database definitions #######################################################################
# first database definition & config directives database bdb
directory /var/lib/ldap/ replogfile /tmp/log/replica.log
rootdn "cn=root,o=....." rootpw .....
suffix "o=....."
#replica uri=ldap://ldaprep1:389 binddn="cn=root,o=..." bindmethod=simple credentials=... #replica uri=ldap://ldaprep2:389 binddn="cn=root,o=..." bindmethod=simple credentials=... #replica uri=ldap://ldaprep3:389 binddn="cn=root,o=..." bindmethod=simple credentials=...
#attribute homeDirectory ces #attribute folderName ces #attribute locked ces # index cid pres,eq index cn pres,eq,sub index objectClass pres,eq index folderName pres,eq index locked pres,eq
Thans in advance, Ralf