openldap-devel

openldap-devel@openldap.org

1418 discussions

Re: BadRSlotError: mdb_txn_begin: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot
by Howard Chu 11 Sep '14

11 Sep '14

Luke Kenneth Casson Leighton wrote: > hi all, > > the infamous obscure error which people are seeing only very > infrequently is rearing its head at least 2 to 3 times per day in a > test lab where i work. this is however a secure environment so i > cannot post core-dumps or any details of the application. > > given the restrictions, what information is needed and what approach > is needed to debug and fix this? luckily it's happening a lot so > there's the possibility of a regular iterative approach. > > the operating system(s) have been ubuntu 12.04 and also 14.04, both > have resulted in this obscure bug. bizarrely, this bug occurs in a > *single process*. it's not even multi-processing. however > metasync=False, sync=False, map_async=True, readahead=False and > writemap=True. Use the Source, Luke. MDB_BAD_RSLOT is returned only one place in mdb.c and the situation is very specific. It means you've tried to begin a new read txn on a thread that already has a read txn outstanding. The API docs are pretty clear that a thread may only have one txn at a time. You need to track down whatever is creating read txns in your code and make sure they're being properly committed or aborted. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

MDB_TOOL mode vs. large mdb_env_set_maxreaders()?
by Hallvard Breien Furuseth 28 Aug '14

28 Aug '14

If an lmdb application configures a non-default maxreaders, this setting is ignored if a tool like mdb_copy is already running (maybe from crontab). I suggest mdb_copy & co should use an MDB_TOOL env flag which requires the lockfile (if any) to be valid, and only partly reinitializes it. The upcoming MDB_ROBUST flag might also make use of this. An alternative would be to stuff this into the metapages, but that seems less flexible. -- Hallvard

2 1

RE24 Testing Call #4
by Quanah Gibson-Mount 15 Aug '14

15 Aug '14

Should be the final testing call. Please test and report back, thanks! --Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

3 5

RE24 Testing call #3 (OpenLDAP 2.4.40)
by Quanah Gibson-Mount 13 Aug '14

13 Aug '14

Hopefully the final testing call. Please test and report back, thanks! --Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

4 6

DTLS, cldaps?
by Howard Chu 07 Aug '14

07 Aug '14

Was just reviewing the latest OpenSSL security advisories, many of which apply to their DTLS implementation. I usually ignore anything that has to do with DTLS since LDAP is connection-oriented and not datagram based, but now it occurs to me to ask - is anyone using LDAP over UDP with DTLS? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 1

Re: openldap.git branch mdb.master updated. 3630066843b7ca6b2cd12911d3e2fe3314cd4549
by Hallvard Breien Furuseth 04 Aug '14

04 Aug '14

On 27/07/14 03:27, openldap-commit2devel(a)OpenLDAP.org wrote: > commit 3630066843b7ca6b2cd12911d3e2fe3314cd4549 > Author: Howard Chu <hyc(a)symas.com> > Date: Sat Jul 26 18:16:02 2014 -0700 > > Fix MIPS cache coherency on Linux > > MIPS chips require manual control of on-chip caches. The cacheflush > syscall being used here only exists on MIPS Linux, other OSs will > require revisiting. I may be guessing wrong what's going on here, but anyway: CACHEFLUSH looks like it belongs before setting me_txns->mti_txnid which tells other threads/processes to use the new metapage. If so, MDB_NOLOCK may be in trouble since it uses pick_meta() instead of mti_txnid. Should there be a separate CACHEFLUSH after writing the datapages if MDB_NOLOCK, and the current CACHEFLUSH should just flush the metapages? Does the code contradict this comment above, or is it about something else? /* Memory ordering issues are irrelevant ... */ -- Hallvard

2 7

RE24 Testing call #2 (OpenLDAP 2.4.40)
by Quanah Gibson-Mount 31 Jul '14

31 Jul '14

Current RE24 is ready for testing for the 2.4.40 release. Thanks! --Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

2.4.40 ITS possibilities
by Quanah Gibson-Mount 24 Jun '14

24 Jun '14

7877 - gcrypt optional, includes patch 6970 - autogroup + memberof crashes or deadlocks slapd 7869 - arguments to do_phk_hash are in the wrong order in contrib apr1 7868 - Sock overlay segfault 7865 - Patch to update slapo-refint manpage with cn=config format 7860 - Update ldap_get_dn(3) man page, with patch 7851 - buffer overrun issue, with patch 7838 - Add ORDERING matching rule to ppolicy (RFC issue?) 7835 - slapadd won't add LDIFv1 compliant files that start with a comment --Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: py-lmdb
by Howard Chu 22 May '14

22 May '14

Luke Kenneth Casson Leighton wrote: >> We fell for the fantasy of parallel writes with BerkeleyDB, but after a >> dozen+ years of poking, profiling, and benchmarking, it all becomes clear - >> all of that locking overhead+deadlock detection/recovery is just a waste of >> resources. > > ... which is why tdb went to the other extreme, to show it could be done. But even tdb only allows one write transaction at a time. I looked into writing a back-tdb for OpenLDAP back in 2009, before I started writing LMDB. I know pretty well how tdb works... >> https://twitter.com/hyc_symas/status/451763166985613312 > > quote: > > "The new code is faster at indexing and searching, but not so much > faster it would blow you away, even using > LMDB. Turns out the slowness of Python looping trumps the speed of a > fast datastore :(. The difference > might be bigger on a big index; I'm going to run experiments on the > Enron dataset and see." > > interesting. so why is read up at 5,000,000 per second under python > (in a python loop, obviously) but write isn't? something odd there. Good question. I'd guess there's some memory allocation overhead involved in writes. The Whoosh guys have some more perf stats here https://bitbucket.org/mchaput/whoosh/wiki/Whoosh3 (their test.Tokyo / All Keys result is highly suspect though, the timing is the same for 100,000 keys as for 1M keys. Probably a bug in their test code.) -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 8

py-lmdb
by Luke Kenneth Casson Leighton 18 May '14

18 May '14

hi all, and a special hello to howard, i had forgotten you're working on openldap. i've just discovered lmdb and its python bindings, and i am to be absolutely honest completely astounded. that's for two reasons: the first is how steeupidly quick lmdb is, and secondly because bizarrely although it walks all over the alternatives it isn't more widely adopted. mariadb added leveldb last year, mongodb i believe are looking at a leveldb port. i have been looking at a *lot* of key-value database stores recently, to find the fastest possible one after realising that standard SQL and NOSQL databases simply aren't fast enough. there is something called structchunk which instead of storing the object in a leveldb just stores an mmap file offset in the value and stores the actual object in an mmap'd file... and it's *still* nowhere near as quick as lmdb. so i am both impressed and puzzled :) i have created a debian RFP for the python bindings, in case it helps. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748373 - can i suggest that anyone wishing to see that in there send a message seconding it to be added: because as a general rule debian developers do not like my blunt honesty and they tend to ignore my advice, so if you would like to see py-lmdb packaged someone needs to speak up. questions: i wrote some python evaluation code that stored 5,000 records with 8-byte keys and 100-byte values before doing a transaction commit: it managed 900,000 records per second (which is ridiculously fast even for python. however: when i enabled append mode (on the cursor put)... and yes i used the db stats to create a key that each time was 1 bigger lexicographically than all other keys... bizarrely things *slowed down* ever so slightly - maybe about 3 to 5%. what gives, there? the benchmarks show that this is supposed to be faster (a *lot* faster) and that is simply not happening. is the overhead from python that large it wipes out the speed advantages? l.

2 2

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel