Howard Chu wrote:
Back on the server side of things...
Using ldapadd to load my test database (380836 entries, 533MB LDIF, 1.2GB id2entry size) takes well over an hour using synchronous writes.
With DB_TXN_WRITE_NOSYNC and a 512 MB BDB cache, plus periodic checkpoints, it takes 39:21.55 minutes. With the patch in add.c:1.247, and the duplicate check in slap_mods_check replaced by a quicksort, it takes 20:23.24 minutes.
With a 1GB BDB cache it takes 14:06.48 minutes. With the above and the new ldapadd client code, it takes 12:49.74 minutes.
With synchronous writes and the original code in HEAD, 1GB BDB cache, it took 1:33:08.74 to ldapadd the database. With the patched add.c and the quicksort it took 1:14:52.11. As a point of reference, it took only 2:42.64 for slapadd -q.
Unfortunately the bulk of the ldapadd time is really determined by BerkeleyDB. With DB_TXN_NOSYNC and transaction logs written to a tmpfs, the original code in HEAD completes the ldapadd in 18:54.56. With the add.c patch that goes down to 13:35.82. With the quicksort in slap_mods_check that drops down further to 7:19.52.