openldap-devel

openldap-devel@openldap.org

1418 discussions

Re: py-lmdb
by Howard Chu 18 May '14

18 May '14

Luke Kenneth Casson Leighton wrote: >> Not surprising. Remember, I've been writing world's-fastest <whatevers> >> since the 1980s. There is no other KV store that is full ACID and anywhere >> near as small or as fast as LMDB. > > ... or does range as well... *and* duplicate values per key. you > have a long way to go before beating tdb for number of lines of code > though :) written by rusty russell, andrew tridgell and others - did > you see it does hash buckets and then spin-locks (file-based) on each > hash chain, so you can have multiple simultaneous writers not just > readers? good trick that, i mention it just in case it's something > that could be deployed to good effect in lmdb, that would be awesome > to have parallel writes speeded up as well per core. > > ... yes i realise that lmdb is read-optimised, but hey its being > adopted elsewhere as well Fair enough, tdb is smaller, but it's also missing all those other goodies that we couldn't live without in OpenLDAP. And it doesn't perform. Something that the world needs to understand - there is no such thing as parallel writes, not in a transactional database. Every system out there gets serialized sooner or later - most just get serialized in their WAL. All the fine grained locking they do ahead of that is just mental masturbation ahead of that inevitable bottleneck. You should notice there are plenty of benchmarks where LMDB's write performance is far faster than so-called write-optimized databases too. When you jettison all the overhead of those fine-grained locks your write path can get a lot faster, and with LMDB's zero-copy writes they go faster still. We fell for the fantasy of parallel writes with BerkeleyDB, but after a dozen+ years of poking, profiling, and benchmarking, it all becomes clear - all of that locking overhead+deadlock detection/recovery is just a waste of resources. back-mdb isn't just faster than back-bdb/hdb for reads, it's also several times faster for writes, and the absence of ubiquitous locks is a good part of that. >>> i wrote some python evaluation code that stored 5,000 records with >>> 8-byte keys and 100-byte values before doing a transaction commit: it >>> managed 900,000 records per second (which is ridiculously fast even >>> what gives, there? the benchmarks show that this is supposed to be >>> faster (a *lot* faster) and that is simply not happening. is the >>> overhead from python that large it wipes out the speed advantages? >> No idea. I don't use python enough to have any insight there. But these folks have some thoughts on it https://twitter.com/hyc_symas/status/451763166985613312 > ok.. is there some c-based benchmark code somewhere i can check how > to do sequential writes, compare it with the python bindings? just to > make sure. it is very puzzling that there's a slow-down rather than a > speed-up. All of the source code for the microbenchmarks is linked from the microbench page. (Minus the actual LevelDB source tree, which you also need.) http://symas.com/mdb/microbench -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: LMDB stuff
by Emmanuel Lécharny 05 May '14

05 May '14

Le 5/5/14 1:46 PM, Hallvard Breien Furuseth a écrit : > On 05/02/2014 11:45 AM, Emmanuel Lécharny wrote: >> I don't know what's inside the Free List in LMDB, but I can only assume >> it's closed to what we have in Mavibot. Let me explain what we do (or >> will do !) in Mavibot, and what data structure we use for that. > > Thanks. > > > (...) >> I don't know how it fits with LMDB implementation, I just wanted to give >> you an insight on what we are working on. Hope it helps. > > LMDB's freeDB is simpler: A mapping > {ID of transaction which freed the pages => [page numbers]}. We used a tree because we need to be able to find fast the pages copied by a given transaction for a specific BTree. If we are to look for any older transaction belonging to the same btree, {ID, page numbers} is not enough. > A write transaction may only use page numbers from freeDB records > older than the oldest snapshot. Yes, this is the current state. In Mavibot, we distinguish the elements in the copiedPagesBTree with the pages in teh freePages list (which is a simple linked list). The rational is that it's less costly to grap a page from a simple linked list than from a b-tree, assuming that some pages will be freed without having to be put into the copiedPagesBtree : for instance, the root page can always be released if there is no read transaction using the N-1 version, or the pages used by the b-tree headers (same reason). At this point, those 'optimisation' are quite pulled out of thin air, I must admit. > It scans the reader table for > that: Each read-only txn has a slot where it puts its snapshot's > txnID. W edo have the same thing in memory. > Also the datafile has metapages for the last 2 snapshots. We have two pointers in the RecordManager header (RMH), one on the current BtreeOfBtrees and CopiedPagesBtree, and one for the previous version of those guys. We always keep a track to the previous pointers until we have done the cleanup, then we rewrite the RecordManager header with only the current version. If we have a crash before the first RMH write, we restart with the current version (which is the version before we updated whatever btrees). If the crash occurs in the middle of the two RMH writes, then at startup, we discover that we have the previous and current pointer, so we know that we have to restart the cleanup Otherwise, we are safe with the new version. In any case, we are in a state we can restart with no breakage, except that we may have lost the latest modification, and in a minimal time. I do think that LMDB and Mavibot are quite similar in the base algorithm (there is not much room for many smart and different approaches here ;), but the implementation *is* different, due to the fact that we are in Java, when LMDB is in C. Typically, serialization/deserialization is a real burden for us, so is the cache. We do'nt use MemoryMappedFile atm, but we could. > There are no concurrent write txns. > -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com

1 0

LMDB stuff
by Howard Chu 02 May '14

02 May '14

Was chatting with Emmanuel Lecharny (who is currently working on Mavibot for ApacheDS, an MVCC backend similar to LMDB) and had an interesting realization: we can avoid the current issue of long-lived reader txns preventing page reclamation. The key point is this - currently for any txn, we store in the freeDB a list of the pageIDs that were freed in that txn (due to being copied or deleted). All we need is to know that any of these pages has been copied *twice* since the txn of the outstanding reader. At that point, any such page is so new that the reader would never have seen it, and if no other readers are outstanding then the page can be safely reclaimed. Currently mavibot maintains this revision history in a dedicated Btree (much like our freeDB but with more info contained). I'm thinking, since we're already going to add a txnID to every page's page header, we can simply add a 2nd txnID, recording the txnID of the previous change to this page's ancestor. Then, any page where this prevTxnID is >= the outstanding reader's txnID can be reclaimed. Still thinking about the actual implementation of this, it may make more sense to store the prevTxnID in the freeDB than in each page header. Ideally we want to be able to grab a chunk of pageIDs unambiguously, instead of having to iterate thru each page and read its header to determine if it's safe. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

3 5

RE: slap_timestamp with microsecond granularity?
by Quanah Gibson-Mount 29 Apr '14

29 Apr '14

--On April 29, 2014 at 12:16:59 PM -0700 "Paul B. Henson" <henson(a)acm.org> wrote: > Even more conveniently, it looks like it's getting the microseconds value > from a member of the op structure, the ppolicy_bind_response function in > servers/slapd/overlays/ppolicy.c also has an op structure passed in, which > presumably should have the same microseconds value, so it looks like it > would be very trivial to emulate the above timestamp generating code in > the password policy module. As already noted by Howard, please move this discussion to the openldap-devel list. Thanks! --Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

cantor down?
by Quanah Gibson-Mount 24 Apr '14

24 Apr '14

Is cantor down? I can't seem to connect on port 22 or do a git pull of the latest source. Thanks, Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Notice to register openldap as maven group name
by Shawn McKinney 09 Apr '14

09 Apr '14

I am planning on changing the namespace used in the openldap fortress projects to org.openldap. In addition to simply changing the packages in source I need to create a new group (org.openldap) in which to publish artifacts to maven central repository. Here's an example of fortress artifacts published to old group, us.joshuatreesoftware: http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22us.joshuatreesoftware%22 Any objections to registering openldap.org as a producer of fortress software artifacts to the maven's central repo? i.e. has someone already done this? Here's more info on topic from sonatype's website: http://central.sonatype.org/pages/producers.html -- 650-963-7681 smckinney(a)symas.com

1 0

really ITS#4365?
by Michael Ströder 03 Apr '14

03 Apr '14

HI! I'm trying to figure out what was recently fixed in RE24. But this comment in CHANGES does not really match the referenced ITS: libldap CR/LF handling (ITS#4365) Maybe I'm overlooking something? Ciao, Michael.

3 2

Global modules and cn=config
by Quanah Gibson-Mount 20 Mar '14

20 Mar '14

Unfortunately, the current cn=config design makes it essentially impossible to use global modules. For example, the pw-sha2 global module for adding addtional hashing schemes cannot be used with cn=config. This is because the olcPasswordHash value is loaded up when cn=config is bootstrapped, prior to loading the global module. This means that the value fails sanity checking, and slapd aborts. See also ITS#7802. Ideas on how to address this chicken and egg issue welcome. ;) --Quanah -- Quanah Gibson-Mount Architect - Server Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 2

Indexing revisited
by Howard Chu 18 Mar '14

18 Mar '14

A few thoughts occurred to me today about our indexing code: 1) we compute a hash preset for each invocation, crunching the syntax and matching rule's OID, among other things. (It used to be worse, we used to recompute this for each individual value, even though it's a constant.) There's no need to always recompute this on each invocation, we can compute it once at first usage and reuse that result. It should speed up index generation, particularly on smaller attribute values. I'm preparing a patch to test this now. 2) using this precomputed hash, we can drop the syntax, mr, and prefix arguments from the indexer function signature. That will also speed things up. 3) I note that the 'use' argument is also never used in our indexer functions. Will drop this as well. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 3

Compact log format for faster searching?
by Hallvard Breien Furuseth 13 Mar '14

13 Mar '14

Have anyone come up with a searchable, compact slapd-log format, so the logs can be searched quicker than the current verbose format? That is, the search tool would understand the compact format without expanding it (fully) first, and only expand the final output. Our syslog from loglevel "stats" compresses to 1/20 of the original, but that's no help when we must search the entire uncompressed log. It's easy to halve the log and still keep it mostly readable: Remove syslog cruft and write it only when it changes, replace "conn= <op/fd>=" with base-32 "conn.<o/f>op", join up multiple SRCH attr= lines, etc. Or down to 1/3 of the original in our case by replacing a few common operations (filters, suffixes, etc), but that quickly makes the result unreadable without a tool to translate back and forth. So beyond that something is needed to translate back and forth, or an entirely new, human-readable, compact format. BTW, "perl -lne '/uid=xyzzy/i && print' log" is 10-15 times faster than GNU "grep -i 'uid=xyzzy' log" on my Linux box. And the initial zcat of a compressed log is faster than the perl/grep. -- Hallvard

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel