On 02/04/2014 02:58 AM, Andrew Eross wrote:
Hello all,
I've been Google'ing around and searching the archives, but I haven't quite been able to find an answer, so I wanted to ask the list.
I've been experimenting with OpenLDAP adds to see how quickly we can get data inserted into the DB.
I've got a few OpenLDAP instances that I use for writing log data to, so write performance is critical, but since I'm building it from log data, absitively, posolutely, guaranteed perfect DB consistency isn't. I can always replay log data to rebuild the DB if, say, I had a power outage, the UPS failed, the RAID write-cache failed, the planets aligned, and I lost data. :-)
First off, disable logging to syslog. The first bottleneck I've run into (reads or writes) was not OpenLDAP but syslogd trying to write everything OpenLDAP told it. Set "loglevel 0" in your slapd.conf. 'Might also consider setting "threads" in slapd.conf if you've got multiple threads doing writes. (see the man pages for this)
Then, to optimize write performance, I added this to slapd.conf (in the database section): checkpoint 1024000 60 dbnosync envflags writemap
And read the man pages so you understand these options and what you're getting into. Some would consider this risky but it's perfect for my particular situation, where the data could always be rebuilt if needed (though I've never needed to).
With this setup, on a single-CPU-core VM, I got around 5400 writes per second with several threads. If memory serves, I got over 4000 writes per second with one thread. This was using the new MDB database back-end which I highly recommend. VERY fast. :-)
Brent
I'm using Ubuntu 10.04, and I've tried both the packaged OpenLDAP 2.4.21 using hdb, and just recently the latest OpenLDAP 2.4.39 using lmdb, both with relatively similar results.
The short version is: to insert 1 million records, it's taking about 8 hours on a machine with 2GB RAM / 3Ghz / SSD, which seems like a long time to me.
The insert method is to use a single big ldiff file like this:
### snip ### dn: cn=newtest0,ou=customer,dc=test,dc=com objectclass: inetOrgPerson userPassword: 90sac90sudasdjcao0sdjtest telephoneNumber: 61400000000 sn: none
dn: cn=newtest1,ou=customer,dc=test,dc=com objectclass: inetOrgPerson userPassword: 90sac90sudasdjcao0sdjtest telephoneNumber: 61400000000 sn: none ### snip ###
Then add it with ldapadd: ldapadd -x -D cn=admin,dc=test,dc=com -w test -f ./test
My configuration (for mdb in this instance) used with 2.4.39 is this: ### config ### include /usr/local/etc/openldap/schema/core.schema include /usr/local/etc/openldap/schema/cosine.schema include /usr/local/etc/openldap/schema/inetorgperson.schema
database mdb directory /usr/local/var/openldap-data suffix "dc=test,dc=com" rootdn "cn=admin,dc=test,dc=com" rootpw test maxsize 1980732000
#Indexes index cn,telephoneNumber eq index objectClass eq ### config ###
Any ideas or things I could try?
Thank you! Andrew