Hello, Using openldap 2.3.38 on 2.6.24 with bdb/hdb backend.
I got about 12-13 000 entries in my directory. Slapcat takes seconds, while slapadd takes 15 minutes or more. Is there any way to speed it up, which I've missed?
to slapadd I do:
/etc/init.d/slapd stop &&
cd /var/lib/ldap && ls |grep -v DB_CONFIG|xargs rm &&
cat slapcat-output-file|slapadd
currently the best way to minimize downtime I found is to replace "directory" line in slapd.conf, then do slapadd "into" new physical location, and then just stop slapd, change dirname, change directory line back, and start again. I'm afraid it's actually not slapd issue, but just some file-reading/adding -related thing.
Is it some better workaround for this with newer release? I'd be happy to jump to 2.4.7 series, but it still seems a little bit unstable, segfaults sometimes with some searches.
Regards, PW
--On Tuesday, April 15, 2008 11:30 PM +0200 Piotr Wadas pwadas@jewish.org.pl wrote:
Hello, Using openldap 2.3.38 on 2.6.24 with bdb/hdb backend.
I got about 12-13 000 entries in my directory. Slapcat takes seconds, while slapadd takes 15 minutes or more. Is there any way to speed it up, which I've missed?
I suggest reading the man page for slapadd, and look at the options available.
I suggest reading the various documentation on properly tuning DB_CONFIG to the size of your database.
I suggest reading the man page for slapd.conf and look at the options relating to the slap* tools.
All of these things contain information that can help you decrease the amount of time it takes to slapadd the database. Assuming you don't have a massive amount of indexing and sane disks, it should take less than a minute or two to slapadd your DB on the worst case.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Well, writes are definitely slower than reads. This is especially bad with writes that sync() frequently, as the default configuration does. (Of course they're safe, so that makes a sane default.)
With that said, the first big thing you're missing is the -q flag to slapadd. Note that if the slapadd -q fails, your only recourse will be a rm. But it looks like your original procedure presumes an rm anyway.
Second, if you've got enough RAM to cache your entire database in-memory, you should adjust your DB_CONFIG and/or back-bdb directives to do so. This isn't always possible (depends on entry sizes, number of entries, and other such variables) but if you can do it, the performance is obviously fantastic. You can determine the right size using the methods in the FAQ-O-Matic or, since you're really concerned about load times, one practical method is to run "slapadd -qv" and look if the times between entries slow down -- the disk writes, being orders of magnitude slower, are often visible to the naked eye in my experience. At the end you'll experience a long flush, but that's good -- it should be gunning your I/O capabilities.
currently the best way to minimize downtime I found is to replace "directory" line in slapd.conf, then do slapadd "into" new physical location, and then just stop slapd, change dirname, change directory line back, and start again. I'm afraid it's actually not slapd issue, but just some file-reading/adding -related thing.
Well, yes, an atomic mv is going to be quicker than slapadd. IMO the best way to minimize downtime is to have lots of replicas, so you can afford to take as long as you desire with a slapadd in any directory you desire...
I'd be happy to jump to 2.4.7 series, but it still seems a little bit unstable, segfaults sometimes with some searches.
There are significant performance enhancements in 2.4. Version 2.4.9 seems to be coming along well and hopefully will fix the seg fault issues you experience (if you've can, it'd be a great contribution to grab RE24 from CVS and see if you can reproduce segfaults with those "some searches").
On Tuesday 15 April 2008 23:30:18 Piotr Wadas wrote:
Hello, Using openldap 2.3.38 on 2.6.24 with bdb/hdb backend.
I got about 12-13 000 entries in my directory. Slapcat takes seconds, while slapadd takes 15 minutes or more.
Your tuning doesn't seem sufficient, slapadding 13000 entries should take under 1 minute.
Is there any way to speed it up, which I've missed?
to slapadd I do:
/etc/init.d/slapd stop &&
cd /var/lib/ldap && ls |grep -v DB_CONFIG|xargs rm &&
cat slapcat-output-file|slapadd
Use -q flag? You don't show your DB_CONFIG file, so we can't offer many accurate suggestions there.
currently the best way to minimize downtime I found is to replace "directory" line in slapd.conf, then do slapadd "into" new physical location, and then just stop slapd, change dirname, change directory line back, and start again.
The best way to minimise downtime is to not need to do an import. Why are you exporting/importing so often that you have downtime?
I'm afraid it's actually not slapd issue, but just some file-reading/adding -related thing.
Is it some better workaround for this with newer release?
Workaround for what?
I'd be happy to jump to 2.4.7 series
Why do you think things would be so much better on 2.4 (2.4.8 is current), assuming no other changes in your environment? While there are improvements, some which would affect this, the fact that your slapadd is more than an order of magnitude slower than it should be on 2.3 seems to indicate that even a 20% performance improvement between 2.3 and 2.4 wouldn't really help you much ...
openldap-software@openldap.org
-
Aaron Richton -
Buchan Milne -
Piotr Wadas -
Quanah Gibson-Mount