Howard Chu wrote:
A bit of a summary of how the backend is shaping up. I've been testing with a variety of synthetic LDIFs as well as an actual application database (Zimbra accounts).
I noted before that back-mdb's write speeds on disk are quite slow. This is because a lot of its writes will be to random disk pages, and also the data writes in a transaction commit are followed by a meta page write, which always involves a seek to page 0 or page 1 of the DB file. For slapadd -q this effect can be somewhat hidden because the writes are done with MDB_NOSYNC specified, so no explicit flushes are performed. In my current tests with synchronous writes, back-mdb is one half the speed of back-bdb/hdb. (Even in fully synchronous mode, BDB only writes its transaction logs synchronously, and those are always sequential writes so there's no seek overhead to deal with.)
With that said, slapadd -q for a 3.2M entry database on a tmpfs:
back-hdb: real 75m32.678s user 84m31.733s sys 1m0.316s back-mdb: real 63m51.048s user 50m23.125s sys 13m27.958s
On an XFS partition, the same job took back-hdb real 80m34.403s user 86m2.439s sys 1m39.662s back-mdb real 85m48.598s user 49m40.606s sys 14m48.668s
Note that back-hdb runs a trickle-sync thread to flush dirty DB pages in the background, which is why its user time is greater than the real time.
back-mdb actually completed the load in 64m16.19s according to slapadd's progress meter. But back-mdb performs an mdb_env_sync() on close, and that sync took the remaining 21 minutes. (back-hdb pretty much does the same, it does a checkpoint on close, but it skips it in Quick mode. So to be apples-to-apples, back-mdb's final sync should have been omitted as well.)
For back-hdb, BDB was configured with a 32GB environment cache. The resulting DB directory consumed 14951004KB including data files and environment files.
For back-mdb, MDB was configured with a 32GB mapsize. The resulting DB directory consumed 18299832KB. The input LDIF was 2.7GB, and there were 29 attributes indexed. Currently MDB is somewhat wasteful with space when dealing with the sorted-duplicate databases that are used for indexing, there's definitely room for improvement here.
Also this slapadd was done with tool-threads set to 1, because back-mdb only allows one writer at a time anyway. There is also obviously room for improvement here, in terms of a bulk-loading API for the MDB library.