openldap-devel July 2012

openldap-devel@openldap.org

6 participants
5 discussions

MDB microbenchmark
by Howard Chu 19 Aug '12

19 Aug '12

Was reading thru Google's leveldb stuff and found their benchmark page http://leveldb.googlecode.com/svn/trunk/doc/benchmark.html I adapted their sqlite test driver for MDB, attached. On my laptop I get: violino:/home/software/leveldb> ./db_bench_mdb MDB: version MDB 0.9.0: ("September 1, 2011") Date: Mon Jul 2 07:17:09 2012 CPU: 4 * Intel(R) Core(TM)2 Extreme CPU Q9300 @ 2.53GHz CPUCache: 6144 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) ------------------------------------------------ fillseq : 9.740 micros/op; 11.4 MB/s fillseqsync : 8.182 micros/op; 13.5 MB/s (10000 ops) fillseqbatch : 0.502 micros/op; 220.5 MB/s fillrandom : 11.558 micros/op; 9.6 MB/s fillrandint : 9.593 micros/op; 10.3 MB/s fillrandibatch : 6.288 micros/op; 15.8 MB/s fillrandsync : 8.399 micros/op; 13.2 MB/s (10000 ops) fillrandbatch : 7.206 micros/op; 15.4 MB/s overwrite : 14.253 micros/op; 7.8 MB/s overwritebatch : 9.075 micros/op; 12.2 MB/s readrandom : 0.261 micros/op; readseq : 0.079 micros/op; 1392.5 MB/s readreverse : 0.085 micros/op; 1301.9 MB/s fillrand100K : 106.695 micros/op; 894.0 MB/s (1000 ops) fillseq100K : 93.626 micros/op; 1018.8 MB/s (1000 ops) readseq100K : 0.095 micros/op; 1005185.9 MB/s readrand100K : 0.368 micros/op; Compared to the leveldb: violino:/home/software/leveldb> ./db_bench LevelDB: version 1.5 Date: Mon Jul 2 07:18:35 2012 CPU: 4 * Intel(R) Core(TM)2 Extreme CPU Q9300 @ 2.53GHz CPUCache: 6144 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) WARNING: Snappy compression is not enabled ------------------------------------------------ fillseq : 1.752 micros/op; 63.1 MB/s fillsync : 13.877 micros/op; 8.0 MB/s (1000 ops) fillrandom : 2.836 micros/op; 39.0 MB/s overwrite : 3.723 micros/op; 29.7 MB/s readrandom : 5.390 micros/op; (1000000 of 1000000 found) readrandom : 4.811 micros/op; (1000000 of 1000000 found) readseq : 0.228 micros/op; 485.1 MB/s readreverse : 0.520 micros/op; 212.9 MB/s compact : 439250.000 micros/op; readrandom : 3.269 micros/op; (1000000 of 1000000 found) readseq : 0.197 micros/op; 560.4 MB/s readreverse : 0.438 micros/op; 252.5 MB/s fill100K : 504.147 micros/op; 189.2 MB/s (1000 ops) crc32c : 4.134 micros/op; 944.9 MB/s (4K per op) snappycomp : 6863.000 micros/op; (snappy failure) snappyuncomp : 8145.000 micros/op; (snappy failure) acquireload : 0.439 micros/op; (each op is 1000 loads) Interestingly enough, MDB wins on one or two write tests. It clearly wins on all of the read tests. MDB databases don't require compaction, so that's another win. MDB doesn't do compression, so those tests are disabled. I haven't duplicated all of the test scenarios described on the web page yet, you can do that yourself with the attached code. It's pretty clear that nothing else even begins to approach MDB's read speed. MDB sequential write speed is dominated by the memcpy's required for copy-on-write page updates. There's not much that can be done to eliminate that, besides batching writes. For random writes the memcmp's on the key comparisons become more of an issue. The fillrandi* tests use an integer key instead of a string-based key, to show the difference due to key comparison overhead. For the synchronous writes, MDB is also faster, because it doesn't need to synchronously write a transaction logfile. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 4

Re: RE24 testing call#3 (OpenLDAP 2.4.32)
by Aaron Richton 04 Aug '12

04 Aug '12

mdb didn't even handle test000...just in the db_open. Assertion failed: p != NULL, file ./../../../libraries/libmdb/mdb.c, line 3391 current thread: t@1 [1] __lwp_kill(0x0, 0x6, 0xffffffffffffffe6, 0x0, 0x0, 0x0), at 0x7fffffff7f8a900c [2] raise(0x6, 0x0, 0xffffffff7fffe180, 0x0, 0x0, 0x0), at 0x7fffffff7f859150 [3] abort(0x4f, 0x0, 0x4f, 0x7efefeff, 0x81010100, 0xff00), at 0x7fffffff7f83eac8 [4] __assert(0x10034bd88, 0x10034bd98, 0xd3f, 0x100909480, 0x100909480, 0x64), at 0x7fffffff7f83edcc =>[5] mdb_page_get(txn = 0x100b10dc0, pgno = 8589934594U, ret = 0x1009094d8), line 3391 in "mdb.c" [6] mdb_page_search(mc = 0x100909490, key = 0xffffffff7fffead8, flags = 0), line 3533 in "mdb.c" [7] mdb_cursor_set(mc = 0x100909490, key = 0xffffffff7fffead8, data = 0xffffffff7fffeac8, op = MDB_SET, exactp = 0xffffffff7fffe9c8), line 3906 in "mdb.c" [8] mdb_cursor_get(mc = 0x100909490, key = 0xffffffff7fffead8, data = 0xffffffff7fffeac8, op = MDB_SET), line 4100 in "mdb.c" [9] mdb_ad_read(mdb = 0x1006b3730, txn = 0x100b10dc0), line 549 in "attr.c" [10] mdb_db_open(be = 0x100694ff0, cr = 0xffffffff7fffee4c), line 231 in "init.c" [11] backend_startup_one(be = 0x100694ff0, cr = 0xffffffff7fffee4c), line 224 in "backend.c" [12] backend_startup(be = 0x100694ff0), line 325 in "backend.c" [13] slap_startup(be = (nil)), line 219 in "init.c" [14] main(argc = 8, argv = 0xffffffff7ffff298), line 991 in "main.c"

2 5

Re: openldap.git branch mdb.master updated. 14fb1f59c7e119a100952890f947377d7e95f135
by Howard Chu 25 Jul '12

25 Jul '12

openldap-commit2devel(a)OpenLDAP.org wrote: > - Log ----------------------------------------------------------------- > commit 14fb1f59c7e119a100952890f947377d7e95f135 > Author: Howard Chu <hyc(a)symas.com> > Date: Sat Jul 21 06:10:02 2012 -0700 > > Append tweaks, page_split fixes > > Append mode now does no key comparisons, input must be in sorted order. > page_split was not updating cursor parents correctly. MDB_APPEND mode is meant to be used for bulk loading of pre-sorted keys. (e.g., if someone wanted to write an mdb_load utility and the corresponding mdb_dump program, you would use this mode for the loader.) New key/data pairs are simply appended to the last page of the database, with no key comparisons at all. (That's the new behavior.) When a page is filled, instead of splitting the page into two new pages (with half the keys in each) it just allocates a new page and starts appending to it. (That's the same behavior as before.) If the parent page fills, it is split (into halves) as normal. The savings from avoiding the memcpy's associated with the split is. The savings from avoiding the key comparisons is another 45% on top of that. In the context of the microbenchmarks posted earlier, with this new code our batched sequential write speed goes up from 1,157,407 entries/sec to 2,083,333 entries/sec, nearly 3 times faster than the closest competitor, LevelDB (at ~745,000 entries/sec). It also helps slapadd -q. It's around 11-1/2 times faster than BerkeleyDB. We should not be hearing comments about slow backup/restore procedures any more... > ----------------------------------------------------------------------- > > Summary of changes: > libraries/libmdb/mdb.c | 115 +++++++++++++++++++++++++++++++++++++++--------- > libraries/libmdb/mdb.h | 24 ++++++++++- > 2 files changed, 117 insertions(+), 22 deletions(-) -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 2

The Last Sysadmin Prayer: Discussion Returns to Wrap and Base64 World
by Angel Bosch 24 Jul '12

24 Jul '12

Hi, I'm struggling with a new CentOS6 installation and turns out that mozldap tools aren't packaged anymore and all my scripts scream out loud because wraping and base64 encoding. I've been reading lots of old petitions of adding wrapping and encoding, testing patches and trying to understand denials. Without being a (truly) developer the only reason I can understand about not implementing this is complexity and lack of time/resources, but seems that some devs already sended patches for those modifications. So I'm really lost: why could be wrong adding a feature that is asked again and again and will help lot of sysadmins make their life better? I know about pipes and string manipulation, but why refusing a non default option that don't break anything? This is specially dramatic for base64 manipulation. mozldap has(had?) a parameter (-e) that "minimize base-64 encoding of values", I don't really know how does it determines if a value should be decoded or not, but it Just Works! In my script-ldap-world i just want to display names, surnames and some description with accents (à, è,...) and some catalan letter (ç). And that's the reason I use decoding 99.99% of times. No photos, blobs or any other exotic embeded object. So please, I mean PLEASE! Could you add a non wrapping and a decoding option to ldap tools? I can't pay my mortage and world if falling down, but I see the light and the end of some patches. abosch

3 4

RE24 testing call#2 (OpenLDAP 2.4.32)
by Quanah Gibson-Mount 19 Jul '12

19 Jul '12

If you know how to build OpenLDAP manually, and would like to participate in testing the next set of code for the 2.4.32 release, please do so. Generally, get the code for RE24: <http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=snapshot;h=refs/h…> Configure & build. Execute the test suite (via make test) after it is built. Thanks! --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel July 2012