openldap-commit2devel@OpenLDAP.org wrote:
- Log -----------------------------------------------------------------
commit 14fb1f59c7e119a100952890f947377d7e95f135 Author: Howard Chu hyc@symas.com Date: Sat Jul 21 06:10:02 2012 -0700
Append tweaks, page_split fixes Append mode now does no key comparisons, input must be in sorted order. page_split was not updating cursor parents correctly.
MDB_APPEND mode is meant to be used for bulk loading of pre-sorted keys. (e.g., if someone wanted to write an mdb_load utility and the corresponding mdb_dump program, you would use this mode for the loader.) New key/data pairs are simply appended to the last page of the database, with no key comparisons at all. (That's the new behavior.) When a page is filled, instead of splitting the page into two new pages (with half the keys in each) it just allocates a new page and starts appending to it. (That's the same behavior as before.) If the parent page fills, it is split (into halves) as normal. The savings from avoiding the memcpy's associated with the split is. The savings from avoiding the key comparisons is another 45% on top of that.
In the context of the microbenchmarks posted earlier, with this new code our batched sequential write speed goes up from 1,157,407 entries/sec to 2,083,333 entries/sec, nearly 3 times faster than the closest competitor, LevelDB (at ~745,000 entries/sec). It also helps slapadd -q. It's around 11-1/2 times faster than BerkeleyDB.
We should not be hearing comments about slow backup/restore procedures any more...
Summary of changes: libraries/libmdb/mdb.c | 115 +++++++++++++++++++++++++++++++++++++++--------- libraries/libmdb/mdb.h | 24 ++++++++++- 2 files changed, 117 insertions(+), 22 deletions(-)
Howard Chu writes:
Append mode now does no key comparisons, input must be in sorted order. page_split was not updating cursor parents correctly.MDB_APPEND mode is meant to be used for bulk loading of pre-sorted keys. (e.g., if someone wanted to write an mdb_load utility and the corresponding mdb_dump program, you would use this mode for the loader.) New key/data pairs are simply appended to the last page of the database, with no key comparisons at all. (...)
As a user, I'd feel safer with a mode which kept the DB consistent: Return failure if (key to append) < (current max key). Presumably that'd mean 1 key comparison per added entry. And maybe giving the root pages a reference to the last key, I don't know.
Does this affect OpenLDAP (slapadd -q) users, or should I be saying "as a libmdb user"?
Hallvard Breien Furuseth wrote:
Howard Chu writes:
Append mode now does no key comparisons, input must be in sorted order. page_split was not updating cursor parents correctly.MDB_APPEND mode is meant to be used for bulk loading of pre-sorted keys. (e.g., if someone wanted to write an mdb_load utility and the corresponding mdb_dump program, you would use this mode for the loader.) New key/data pairs are simply appended to the last page of the database, with no key comparisons at all. (...)
As a user, I'd feel safer with a mode which kept the DB consistent: Return failure if (key to append) < (current max key). Presumably that'd mean 1 key comparison per added entry. And maybe giving the root pages a reference to the last key, I don't know.
I had a check for (key == current last key) but now I've changed it to (key <= current last key).
Does this affect OpenLDAP (slapadd -q) users, or should I be saying "as a libmdb user"?
Yes, this affects slapadd (not just -q). Append mode is used for the id2entry DB since new entries all get consecutively incremented entryIDs. Append mode is also used for the dn2id DB when adding a new entry. Append mode is also used for index records if adding a new entryID to an index slot that already exists.
-
Hallvard Breien Furuseth -
Howard Chu