Howard Chu wrote:
Howard Chu wrote:
It seems to me this can only be a viable mode of operation if you're always going to run asynch and don't care much about transaction durability or DB recoverability. Running in this mode offers absolutely zero crash resistance; the entire DB will almost always be irreparably damaged after a system crash.
Would you run like that, if it offered you the potential of maybe 10x faster write performance? (It could be useful for slapadd -q, certainly.)
OK, 10x speedup was far too optimistic. Quickly cobbling the changes together, it looks more like about a 70% speedup.
slapadd -q with 5 million entries took 24m16s as originally reported at LDAPCon last year.
With current mdb.master it takes 22m24s:
real 22m23.984s user 26m1.658s sys 8m17.415s
With the writable mmap and no msyncs it took 13m17s.
real 13m17.225s user 22m15.511s sys 1m12.533s
This code is currently available on the map2 branch of my git repo on ada.openldap.org. I'll clean it up a bit further then push it to mdb.master after some more testing.
The speedup seems to be proportional to the number of indices that are defined on the database. With Quanah's torture-test LDIF (~6 million entries, 4.9GB), there's only a small difference between the two when no indices are defined:
Original mdb.master:
real 11m27.385s user 15m5.489s sys 6m47.825s
Writemap:
real 10m27.447s user 14m35.663s sys 6m10.767s
But with 31 indices defined...
Original:
real 94m35.862s user 93m31.755s sys 20m39.693s
Writemap:
real 42m53.499s user 54m35.509s sys 7m23.364s
Over a 2:1 speedup.