At this point the libmdb library code is behaving pretty well, and the back-mdb is code-complete. It is not yet passing all of make test, but it does pass the majority of tests so far. As expected, it delivers higher throughput than back-bdb/hdb under heavy load, with a smaller memory footprint, and nothing to tune. So far so good.
Remaining bugs revealed by the test suite need to be tracked down, then some profiling/optimization work should be done. The current code is already fast, but it is far from optimal.
TODO: * the mdb library doesn't coordinate cursors with write operations. (So performing a write invalidates any open cursors in the same txn.) This needs to be fixed. * cursors only support read operations. The back-mdb indexer would benefit from being able to use put/del on a cursor, as back-bdb does. (And the back-bdb/hdb code needs to be revisited; it closes/re-opens cursors very often when it probably should just open once and keep using till an operation is done. Of course this has to be balanced against the desire to keep BDB resources locked for as little time as possible. Fortunately since MDB has no locks the same worries don't apply.) * entry_encode/decode re-work - currently back-mdb uses the same encoder as back-bdb/hdb. I plan to replace this with an encoding that uses a separate database of attributeDesc to integer mappings. This will reduce the overall DB size and will accelerate decoding. I'm backing away from the idea of a fully persistent DB with no encoding/decoding at all; maybe will try this further down the road. (The full in-memory structures are quite large, and DB size will increase dramatically. But this might be OK overall; or we might look at shrinking slapd's Entry structure etc. - quite a huge amount of grunt work involved.)
openldap-commit2devel@OpenLDAP.org wrote:
A ref change was pushed to the OpenLDAP (openldap.git) repository. It will be available in the public mirror shortly.
The branch, mdb has been updated via 5d600468011aaf773cf4cd7ed13a213d3e2e72ca (commit) via 8a6b9ea1a3274ee0525a060821db45ac5725861d (commit) from 06d590fffe107ac75e59df83516501ee1f5e46e6 (commit)
Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below.
- Log -----------------------------------------------------------------
commit 5d600468011aaf773cf4cd7ed13a213d3e2e72ca Author: Howard Chuhyc@openldap.org Date: Fri Aug 26 01:24:06 2011 -0700
fix opinfo
commit 8a6b9ea1a3274ee0525a060821db45ac5725861d Author: Howard Chuhyc@openldap.org Date: Fri Aug 26 01:18:49 2011 -0700
Fix mdb_entry_get
Summary of changes: servers/slapd/back-mdb/bind.c | 10 ++-------- servers/slapd/back-mdb/compare.c | 10 ++-------- servers/slapd/back-mdb/id2entry.c | 4 +++- servers/slapd/back-mdb/operational.c | 10 ++-------- servers/slapd/back-mdb/referral.c | 10 ++-------- servers/slapd/back-mdb/search.c | 10 ++-------- 6 files changed, 13 insertions(+), 41 deletions(-)
Howard Chu wrote:
At this point the libmdb library code is behaving pretty well, and the back-mdb is code-complete. It is not yet passing all of make test, but it does pass the majority of tests so far. As expected, it delivers higher throughput than back-bdb/hdb under heavy load, with a smaller memory footprint, and nothing to tune. So far so good.
Summary of changes: servers/slapd/back-mdb/bind.c | 10 ++-------- servers/slapd/back-mdb/compare.c | 10 ++-------- servers/slapd/back-mdb/id2entry.c | 4 +++- servers/slapd/back-mdb/operational.c | 10 ++-------- servers/slapd/back-mdb/referral.c | 10 ++-------- servers/slapd/back-mdb/search.c | 10 ++-------- 6 files changed, 13 insertions(+), 41 deletions(-)
You'll note that for most of the commits the deletions far outweigh the insertions. I made the MDB API similar to the BDB API in most areas, so the effort of porting wasn't too great. Most of the effort was simply in deleting; fully 1/3rd of back-bdb/hdb's code was deleted to get back-mdb running. This is more of what I originally intended back-hdb to be from the beginning... Now if we just go forward with my idea to move all slapd global variables into a single structure, it would be trivial to persist all of slapd's runtime state into the mmap as well.* Mebbe OpenLDAP 3.0 will get there - instant-on server...
http://www.openldap.org/lists/openldap-devel/200410/msg00000.html