Re: syncrepl consumer is slow

Emmanuel Lécharny wrote: > Le 03/02/15 05:11, Howard Chu a écrit : >> Another option here is simply to perform batching. Now that we have >> the TXN api exposed in the backend interface, we could just batch up >> e.g. 500 entries per txn. much like slapadd -q already does. >> Ultimately we ought to be able to get syncrepl refresh to occur at >> nearly the same speed as slapadd -q. > > Batching is ok, except that you never know how many entries you'll going > to have, thus you will have to actually write the data after a period of > time, even if you don't have the 500 entries. This isn't a problem - we know exactly when refresh completes, so we can finish the batch regardless of how many entries are left over. Testing this out with the experimental ITS#8040 patch - with lazy commit the 2.8M entries (2.5GB data) takes ~10 minutes for the refresh to pull them across. With batching 500 entries/txn+lazy commit it takes ~7 minutes, a decent improvement. It's still 2x slower than slapadd -q though, which loads the data in 3-1/2 minutes.