openldap-devel

openldap-devel@openldap.org

1416 discussions

RE24 Testing Call #4
by Quanah Gibson-Mount 15 Aug '14

15 Aug '14

Should be the final testing call. Please test and report back, thanks! --Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

3 5

RE24 Testing call #3 (OpenLDAP 2.4.40)
by Quanah Gibson-Mount 13 Aug '14

13 Aug '14

Hopefully the final testing call. Please test and report back, thanks! --Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

4 6

DTLS, cldaps?
by Howard Chu 07 Aug '14

07 Aug '14

Was just reviewing the latest OpenSSL security advisories, many of which apply to their DTLS implementation. I usually ignore anything that has to do with DTLS since LDAP is connection-oriented and not datagram based, but now it occurs to me to ask - is anyone using LDAP over UDP with DTLS? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 1

Re: openldap.git branch mdb.master updated. 3630066843b7ca6b2cd12911d3e2fe3314cd4549
by Hallvard Breien Furuseth 04 Aug '14

04 Aug '14

On 27/07/14 03:27, openldap-commit2devel(a)OpenLDAP.org wrote: > commit 3630066843b7ca6b2cd12911d3e2fe3314cd4549 > Author: Howard Chu <hyc(a)symas.com> > Date: Sat Jul 26 18:16:02 2014 -0700 > > Fix MIPS cache coherency on Linux > > MIPS chips require manual control of on-chip caches. The cacheflush > syscall being used here only exists on MIPS Linux, other OSs will > require revisiting. I may be guessing wrong what's going on here, but anyway: CACHEFLUSH looks like it belongs before setting me_txns->mti_txnid which tells other threads/processes to use the new metapage. If so, MDB_NOLOCK may be in trouble since it uses pick_meta() instead of mti_txnid. Should there be a separate CACHEFLUSH after writing the datapages if MDB_NOLOCK, and the current CACHEFLUSH should just flush the metapages? Does the code contradict this comment above, or is it about something else? /* Memory ordering issues are irrelevant ... */ -- Hallvard

2 7

RE24 Testing call #2 (OpenLDAP 2.4.40)
by Quanah Gibson-Mount 31 Jul '14

31 Jul '14

Current RE24 is ready for testing for the 2.4.40 release. Thanks! --Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

2.4.40 ITS possibilities
by Quanah Gibson-Mount 24 Jun '14

24 Jun '14

7877 - gcrypt optional, includes patch 6970 - autogroup + memberof crashes or deadlocks slapd 7869 - arguments to do_phk_hash are in the wrong order in contrib apr1 7868 - Sock overlay segfault 7865 - Patch to update slapo-refint manpage with cn=config format 7860 - Update ldap_get_dn(3) man page, with patch 7851 - buffer overrun issue, with patch 7838 - Add ORDERING matching rule to ppolicy (RFC issue?) 7835 - slapadd won't add LDIFv1 compliant files that start with a comment --Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: py-lmdb
by Howard Chu 22 May '14

22 May '14

Luke Kenneth Casson Leighton wrote: >> We fell for the fantasy of parallel writes with BerkeleyDB, but after a >> dozen+ years of poking, profiling, and benchmarking, it all becomes clear - >> all of that locking overhead+deadlock detection/recovery is just a waste of >> resources. > > ... which is why tdb went to the other extreme, to show it could be done. But even tdb only allows one write transaction at a time. I looked into writing a back-tdb for OpenLDAP back in 2009, before I started writing LMDB. I know pretty well how tdb works... >> https://twitter.com/hyc_symas/status/451763166985613312 > > quote: > > "The new code is faster at indexing and searching, but not so much > faster it would blow you away, even using > LMDB. Turns out the slowness of Python looping trumps the speed of a > fast datastore :(. The difference > might be bigger on a big index; I'm going to run experiments on the > Enron dataset and see." > > interesting. so why is read up at 5,000,000 per second under python > (in a python loop, obviously) but write isn't? something odd there. Good question. I'd guess there's some memory allocation overhead involved in writes. The Whoosh guys have some more perf stats here https://bitbucket.org/mchaput/whoosh/wiki/Whoosh3 (their test.Tokyo / All Keys result is highly suspect though, the timing is the same for 100,000 keys as for 1M keys. Probably a bug in their test code.) -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 8

py-lmdb
by Luke Kenneth Casson Leighton 18 May '14

18 May '14

hi all, and a special hello to howard, i had forgotten you're working on openldap. i've just discovered lmdb and its python bindings, and i am to be absolutely honest completely astounded. that's for two reasons: the first is how steeupidly quick lmdb is, and secondly because bizarrely although it walks all over the alternatives it isn't more widely adopted. mariadb added leveldb last year, mongodb i believe are looking at a leveldb port. i have been looking at a *lot* of key-value database stores recently, to find the fastest possible one after realising that standard SQL and NOSQL databases simply aren't fast enough. there is something called structchunk which instead of storing the object in a leveldb just stores an mmap file offset in the value and stores the actual object in an mmap'd file... and it's *still* nowhere near as quick as lmdb. so i am both impressed and puzzled :) i have created a debian RFP for the python bindings, in case it helps. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748373 - can i suggest that anyone wishing to see that in there send a message seconding it to be added: because as a general rule debian developers do not like my blunt honesty and they tend to ignore my advice, so if you would like to see py-lmdb packaged someone needs to speak up. questions: i wrote some python evaluation code that stored 5,000 records with 8-byte keys and 100-byte values before doing a transaction commit: it managed 900,000 records per second (which is ridiculously fast even for python. however: when i enabled append mode (on the cursor put)... and yes i used the db stats to create a key that each time was 1 bigger lexicographically than all other keys... bizarrely things *slowed down* ever so slightly - maybe about 3 to 5%. what gives, there? the benchmarks show that this is supposed to be faster (a *lot* faster) and that is simply not happening. is the overhead from python that large it wipes out the speed advantages? l.

2 2

Re: py-lmdb
by Howard Chu 18 May '14

18 May '14

Luke Kenneth Casson Leighton wrote: >> Not surprising. Remember, I've been writing world's-fastest <whatevers> >> since the 1980s. There is no other KV store that is full ACID and anywhere >> near as small or as fast as LMDB. > > ... or does range as well... *and* duplicate values per key. you > have a long way to go before beating tdb for number of lines of code > though :) written by rusty russell, andrew tridgell and others - did > you see it does hash buckets and then spin-locks (file-based) on each > hash chain, so you can have multiple simultaneous writers not just > readers? good trick that, i mention it just in case it's something > that could be deployed to good effect in lmdb, that would be awesome > to have parallel writes speeded up as well per core. > > ... yes i realise that lmdb is read-optimised, but hey its being > adopted elsewhere as well Fair enough, tdb is smaller, but it's also missing all those other goodies that we couldn't live without in OpenLDAP. And it doesn't perform. Something that the world needs to understand - there is no such thing as parallel writes, not in a transactional database. Every system out there gets serialized sooner or later - most just get serialized in their WAL. All the fine grained locking they do ahead of that is just mental masturbation ahead of that inevitable bottleneck. You should notice there are plenty of benchmarks where LMDB's write performance is far faster than so-called write-optimized databases too. When you jettison all the overhead of those fine-grained locks your write path can get a lot faster, and with LMDB's zero-copy writes they go faster still. We fell for the fantasy of parallel writes with BerkeleyDB, but after a dozen+ years of poking, profiling, and benchmarking, it all becomes clear - all of that locking overhead+deadlock detection/recovery is just a waste of resources. back-mdb isn't just faster than back-bdb/hdb for reads, it's also several times faster for writes, and the absence of ubiquitous locks is a good part of that. >>> i wrote some python evaluation code that stored 5,000 records with >>> 8-byte keys and 100-byte values before doing a transaction commit: it >>> managed 900,000 records per second (which is ridiculously fast even >>> what gives, there? the benchmarks show that this is supposed to be >>> faster (a *lot* faster) and that is simply not happening. is the >>> overhead from python that large it wipes out the speed advantages? >> No idea. I don't use python enough to have any insight there. But these folks have some thoughts on it https://twitter.com/hyc_symas/status/451763166985613312 > ok.. is there some c-based benchmark code somewhere i can check how > to do sequential writes, compare it with the python bindings? just to > make sure. it is very puzzling that there's a slow-down rather than a > speed-up. All of the source code for the microbenchmarks is linked from the microbench page. (Minus the actual LevelDB source tree, which you also need.) http://symas.com/mdb/microbench -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: LMDB stuff
by Emmanuel Lécharny 05 May '14

05 May '14

Le 5/5/14 1:46 PM, Hallvard Breien Furuseth a écrit : > On 05/02/2014 11:45 AM, Emmanuel Lécharny wrote: >> I don't know what's inside the Free List in LMDB, but I can only assume >> it's closed to what we have in Mavibot. Let me explain what we do (or >> will do !) in Mavibot, and what data structure we use for that. > > Thanks. > > > (...) >> I don't know how it fits with LMDB implementation, I just wanted to give >> you an insight on what we are working on. Hope it helps. > > LMDB's freeDB is simpler: A mapping > {ID of transaction which freed the pages => [page numbers]}. We used a tree because we need to be able to find fast the pages copied by a given transaction for a specific BTree. If we are to look for any older transaction belonging to the same btree, {ID, page numbers} is not enough. > A write transaction may only use page numbers from freeDB records > older than the oldest snapshot. Yes, this is the current state. In Mavibot, we distinguish the elements in the copiedPagesBTree with the pages in teh freePages list (which is a simple linked list). The rational is that it's less costly to grap a page from a simple linked list than from a b-tree, assuming that some pages will be freed without having to be put into the copiedPagesBtree : for instance, the root page can always be released if there is no read transaction using the N-1 version, or the pages used by the b-tree headers (same reason). At this point, those 'optimisation' are quite pulled out of thin air, I must admit. > It scans the reader table for > that: Each read-only txn has a slot where it puts its snapshot's > txnID. W edo have the same thing in memory. > Also the datafile has metapages for the last 2 snapshots. We have two pointers in the RecordManager header (RMH), one on the current BtreeOfBtrees and CopiedPagesBtree, and one for the previous version of those guys. We always keep a track to the previous pointers until we have done the cleanup, then we rewrite the RecordManager header with only the current version. If we have a crash before the first RMH write, we restart with the current version (which is the version before we updated whatever btrees). If the crash occurs in the middle of the two RMH writes, then at startup, we discover that we have the previous and current pointer, so we know that we have to restart the cleanup Otherwise, we are safe with the new version. In any case, we are in a state we can restart with no breakage, except that we may have lost the latest modification, and in a minimal time. I do think that LMDB and Mavibot are quite similar in the base algorithm (there is not much room for many smart and different approaches here ;), but the implementation *is* different, due to the fact that we are in Java, when LMDB is in C. Typically, serialization/deserialization is a real burden for us, so is the cache. We do'nt use MemoryMappedFile atm, but we could. > There are no concurrent write txns. > -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel