MDB microbenchmark
by Howard Chu
Was reading thru Google's leveldb stuff and found their benchmark page
http://leveldb.googlecode.com/svn/trunk/doc/benchmark.html
I adapted their sqlite test driver for MDB, attached.
On my laptop I get:
violino:/home/software/leveldb> ./db_bench_mdb
MDB: version MDB 0.9.0: ("September 1, 2011")
Date: Mon Jul 2 07:17:09 2012
CPU: 4 * Intel(R) Core(TM)2 Extreme CPU Q9300 @ 2.53GHz
CPUCache: 6144 KB
Keys: 16 bytes each
Values: 100 bytes each (50 bytes after compression)
Entries: 1000000
RawSize: 110.6 MB (estimated)
FileSize: 62.9 MB (estimated)
------------------------------------------------
fillseq : 9.740 micros/op; 11.4 MB/s
fillseqsync : 8.182 micros/op; 13.5 MB/s (10000 ops)
fillseqbatch : 0.502 micros/op; 220.5 MB/s
fillrandom : 11.558 micros/op; 9.6 MB/s
fillrandint : 9.593 micros/op; 10.3 MB/s
fillrandibatch : 6.288 micros/op; 15.8 MB/s
fillrandsync : 8.399 micros/op; 13.2 MB/s (10000 ops)
fillrandbatch : 7.206 micros/op; 15.4 MB/s
overwrite : 14.253 micros/op; 7.8 MB/s
overwritebatch : 9.075 micros/op; 12.2 MB/s
readrandom : 0.261 micros/op;
readseq : 0.079 micros/op; 1392.5 MB/s
readreverse : 0.085 micros/op; 1301.9 MB/s
fillrand100K : 106.695 micros/op; 894.0 MB/s (1000 ops)
fillseq100K : 93.626 micros/op; 1018.8 MB/s (1000 ops)
readseq100K : 0.095 micros/op; 1005185.9 MB/s
readrand100K : 0.368 micros/op;
Compared to the leveldb:
violino:/home/software/leveldb> ./db_bench
LevelDB: version 1.5
Date: Mon Jul 2 07:18:35 2012
CPU: 4 * Intel(R) Core(TM)2 Extreme CPU Q9300 @ 2.53GHz
CPUCache: 6144 KB
Keys: 16 bytes each
Values: 100 bytes each (50 bytes after compression)
Entries: 1000000
RawSize: 110.6 MB (estimated)
FileSize: 62.9 MB (estimated)
WARNING: Snappy compression is not enabled
------------------------------------------------
fillseq : 1.752 micros/op; 63.1 MB/s
fillsync : 13.877 micros/op; 8.0 MB/s (1000 ops)
fillrandom : 2.836 micros/op; 39.0 MB/s
overwrite : 3.723 micros/op; 29.7 MB/s
readrandom : 5.390 micros/op; (1000000 of 1000000 found)
readrandom : 4.811 micros/op; (1000000 of 1000000 found)
readseq : 0.228 micros/op; 485.1 MB/s
readreverse : 0.520 micros/op; 212.9 MB/s
compact : 439250.000 micros/op;
readrandom : 3.269 micros/op; (1000000 of 1000000 found)
readseq : 0.197 micros/op; 560.4 MB/s
readreverse : 0.438 micros/op; 252.5 MB/s
fill100K : 504.147 micros/op; 189.2 MB/s (1000 ops)
crc32c : 4.134 micros/op; 944.9 MB/s (4K per op)
snappycomp : 6863.000 micros/op; (snappy failure)
snappyuncomp : 8145.000 micros/op; (snappy failure)
acquireload : 0.439 micros/op; (each op is 1000 loads)
Interestingly enough, MDB wins on one or two write tests. It clearly wins on
all of the read tests. MDB databases don't require compaction, so that's
another win. MDB doesn't do compression, so those tests are disabled.
I haven't duplicated all of the test scenarios described on the web page yet,
you can do that yourself with the attached code. It's pretty clear that
nothing else even begins to approach MDB's read speed.
MDB sequential write speed is dominated by the memcpy's required for
copy-on-write page updates. There's not much that can be done to eliminate
that, besides batching writes. For random writes the memcmp's on the key
comparisons become more of an issue. The fillrandi* tests use an integer key
instead of a string-based key, to show the difference due to key comparison
overhead.
For the synchronous writes, MDB is also faster, because it doesn't need to
synchronously write a transaction logfile.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
11 years, 3 months
Re: RE24 testing call#3 (OpenLDAP 2.4.32)
by Aaron Richton
mdb didn't even handle test000...just in the db_open.
Assertion failed: p != NULL, file ./../../../libraries/libmdb/mdb.c, line 3391
current thread: t@1
[1] __lwp_kill(0x0, 0x6, 0xffffffffffffffe6, 0x0, 0x0, 0x0), at 0x7fffffff7f8a900c
[2] raise(0x6, 0x0, 0xffffffff7fffe180, 0x0, 0x0, 0x0), at 0x7fffffff7f859150
[3] abort(0x4f, 0x0, 0x4f, 0x7efefeff, 0x81010100, 0xff00), at 0x7fffffff7f83eac8
[4] __assert(0x10034bd88, 0x10034bd98, 0xd3f, 0x100909480, 0x100909480, 0x64), at 0x7fffffff7f83edcc
=>[5] mdb_page_get(txn = 0x100b10dc0, pgno = 8589934594U, ret = 0x1009094d8), line 3391 in "mdb.c"
[6] mdb_page_search(mc = 0x100909490, key = 0xffffffff7fffead8, flags = 0), line 3533 in "mdb.c"
[7] mdb_cursor_set(mc = 0x100909490, key = 0xffffffff7fffead8, data = 0xffffffff7fffeac8, op = MDB_SET, exactp = 0xffffffff7fffe9c8), line 3906 in "mdb.c"
[8] mdb_cursor_get(mc = 0x100909490, key = 0xffffffff7fffead8, data = 0xffffffff7fffeac8, op = MDB_SET), line 4100 in "mdb.c"
[9] mdb_ad_read(mdb = 0x1006b3730, txn = 0x100b10dc0), line 549 in "attr.c"
[10] mdb_db_open(be = 0x100694ff0, cr = 0xffffffff7fffee4c), line 231 in "init.c"
[11] backend_startup_one(be = 0x100694ff0, cr = 0xffffffff7fffee4c), line 224 in "backend.c"
[12] backend_startup(be = 0x100694ff0), line 325 in "backend.c"
[13] slap_startup(be = (nil)), line 219 in "init.c"
[14] main(argc = 8, argv = 0xffffffff7ffff298), line 991 in "main.c"
11 years, 4 months
Re: openldap.git branch mdb.master updated. 14fb1f59c7e119a100952890f947377d7e95f135
by Howard Chu
openldap-commit2devel(a)OpenLDAP.org wrote:
> - Log -----------------------------------------------------------------
> commit 14fb1f59c7e119a100952890f947377d7e95f135
> Author: Howard Chu <hyc(a)symas.com>
> Date: Sat Jul 21 06:10:02 2012 -0700
>
> Append tweaks, page_split fixes
>
> Append mode now does no key comparisons, input must be in sorted order.
> page_split was not updating cursor parents correctly.
MDB_APPEND mode is meant to be used for bulk loading of pre-sorted keys.
(e.g., if someone wanted to write an mdb_load utility and the corresponding
mdb_dump program, you would use this mode for the loader.) New key/data pairs
are simply appended to the last page of the database, with no key comparisons
at all. (That's the new behavior.) When a page is filled, instead of splitting
the page into two new pages (with half the keys in each) it just allocates a
new page and starts appending to it. (That's the same behavior as before.) If
the parent page fills, it is split (into halves) as normal. The savings from
avoiding the memcpy's associated with the split is. The savings from avoiding
the key comparisons is another 45% on top of that.
In the context of the microbenchmarks posted earlier, with this new code our
batched sequential write speed goes up from 1,157,407 entries/sec to 2,083,333
entries/sec, nearly 3 times faster than the closest competitor, LevelDB (at
~745,000 entries/sec). It also helps slapadd -q. It's around 11-1/2 times
faster than BerkeleyDB.
We should not be hearing comments about slow backup/restore procedures any more...
> -----------------------------------------------------------------------
>
> Summary of changes:
> libraries/libmdb/mdb.c | 115 +++++++++++++++++++++++++++++++++++++++---------
> libraries/libmdb/mdb.h | 24 ++++++++++-
> 2 files changed, 117 insertions(+), 22 deletions(-)
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
11 years, 4 months
The Last Sysadmin Prayer: Discussion Returns to Wrap and Base64 World
by Angel Bosch
Hi,
I'm struggling with a new CentOS6 installation and turns out that mozldap tools aren't packaged anymore and all my scripts scream out loud because wraping and base64 encoding.
I've been reading lots of old petitions of adding wrapping and encoding, testing patches and trying to understand denials.
Without being a (truly) developer the only reason I can understand about not implementing this is complexity and lack of time/resources, but seems that some devs already sended patches for those modifications.
So I'm really lost: why could be wrong adding a feature that is asked again and again and will help lot of sysadmins make their life better?
I know about pipes and string manipulation, but why refusing a non default option that don't break anything?
This is specially dramatic for base64 manipulation. mozldap has(had?) a parameter (-e) that "minimize base-64 encoding of values", I don't really know how does it determines if a value should be decoded or not, but it Just Works!
In my script-ldap-world i just want to display names, surnames and some description with accents (à, è,...) and some catalan letter (ç). And that's the reason I use decoding 99.99% of times. No photos, blobs or any other exotic embeded object.
So please, I mean PLEASE!
Could you add a non wrapping and a decoding option to ldap tools?
I can't pay my mortage and world if falling down, but I see the light and the end of some patches.
abosch
11 years, 4 months
RE24 testing call#2 (OpenLDAP 2.4.32)
by Quanah Gibson-Mount
If you know how to build OpenLDAP manually, and would like to participate
in testing the next set of code for the 2.4.32 release, please do so.
Generally, get the code for RE24:
<http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=snapshot;h=refs...>
Configure & build.
Execute the test suite (via make test) after it is built.
Thanks!
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
11 years, 4 months
