Thanks, Dieter, Quanah.
I've been doing some experimenting with those mdb options.
I ran a few tests with inserting 10,000 records, wiping the DB in between, and changing just the one option at a time:
Base-line, no extra options: 4m8s With "writemap" enabled: 8m55s With "writemap+mapasync" enabled: 5m12s With "dbnosync+checkpoint 0kb/1min": 0m14s
Oddly I didn't have much luck with writemap, which by default actually slowed things down for me.
The magic bullet of the bunch is dbnosync, which completed so quickly because it didn't even hit the 1 minute mark where it would've written to disk.
That seems a bit risky to me, though. You'd have to be willing to have 1 minute worth of data be theoretically expendable with that option.
Cheers, Andrew
Andrew Eross CTO Locatrix Communications Office: +61 7 3123 1469 Mobile: +55 37 9858 9815 eross@locatrix.com
On Tue, Feb 4, 2014 at 2:04 PM, Dieter Klünter dieter@dkluenter.de wrote:
Am Tue, 4 Feb 2014 08:45:47 -0200 schrieb Andrew Eross eross@locatrix.com:
Thanks, Chris.
Yeah, I hear you on that, but sorry, to be more specific, I was running this test to get an idea of performance for regular LDAP use, and slapadd is a purely offline solution.
It would be helpful for a restore, of course, but not equivalent to when our application will be adding/modifying records (which is what I'm really trying to simulate).
Running the same test of inserting 1M rows into postgres with the same type of data record on the same machine goes about 3x faster, which just doesn't sound right, since LDAP should be way faster than Postgres, right?
Cheers, Andrew
Andrew Eross CTO Locatrix Communications Office: +61 7 3123 1469 Mobile: +55 37 9858 9815 eross@locatrix.com
On Tue, Feb 4, 2014 at 8:25 AM, Chris Card ctcard@hotmail.com wrote:
Hello all,
I've been Google'ing around and searching the archives, but I haven't quite been able to find an answer, so I wanted to ask the list.
I've been experimenting with OpenLDAP adds to see how quickly we can get data inserted into the DB.
I'm using Ubuntu 10.04, and I've tried both the packaged OpenLDAP 2.4.21 using hdb, and just recently the latest OpenLDAP 2.4.39 using lmdb, both with relatively similar results.
The short version is: to insert 1 million records, it's taking about 8 hours on a machine with 2GB RAM / 3Ghz / SSD, which seems like a long time to me.
The insert method is to use a single big ldiff file like this:
You may have a look at this paper, helt at LDAPcon 2013
http://fr.slideshare.net/ldapcon/benchmarks-on-ldap-directories
-Dieter
-- Dieter Klünter | Systemberatung http://dkluenter.de GPG Key ID: E9ED159B 53°37'09,95"N 10°08'02,42"E