openldap-devel

openldap-devel@openldap.org

1416 discussions

Incremental backup in LMDB
by Howard Chu 01 Oct '14

01 Oct '14

A preliminary branch has been available on gitorious for a few months. It has some issues though, which still need to be resolved. https://gitorious.org/mdb/mdb/source/a7fc32c0b39c29f9afd2172338060b079d549c…: The biggest problem is that scanning every page of the DB is unsafe - if a page was an overflow page, and was freed and partially reused, then a scan may walk into intermediate chunks of data (which won't have valid page headers, although the data might coincidentally appear to have them). Two approaches to solving this are to perform the scan, but ignore pages that are currently listed in the freeDB, or to do a regular tree walk instead. At the moment I'm leaning toward just walking the trees. If the freeDB is large then comparing every scanned page to the freeDB could be expensive. Also, newer pages of the freeDB itself must be copied as well, and this would mean freeDB pages get read twice. Just thinking out loud here to document what we discussed before in IRC, so it doesn't get lost in the meantime. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

unique incrementing value with lmdb?
by CJ Ess 23 Sep '14

23 Sep '14

What is the best way to implement a unique incrementing value (for use as a key or value) with lmdb? Is there a property if the transaction mechanism I could piggy-back off of to atomically generate my number? Or should I block writers above mdb_cursor_put, do my lmdb transaction, and unblock?

2 1

Re: Problem: LDAP installation without internet
by Aaron Richton 23 Sep '14

23 Sep '14

On Fri, 22 Aug 2014, Patrick Pat wrote: > Hi, > I would like all LDAP installation procedure without internet in linux > (ubuntu (9.10 or other), Debian, ...), but by its compressed file > > namely: > openldap-2.4.8.tgz or openldap-2.4.7.tgz or openldap-2.4.9.tgz or > openldap-2.0-beta.tgz ... > And any configuration environment variables. So inside that .tgz, there's a file called "INSTALL"; you might want to take a look. None of the procedures mentioned in "INSTALL" require anything other than a vaguely sane POSIX-esque (or Windows, but let's not go there) system. This includes any particular network connectivity, Internet or otherwise. > best regards ! > >

1 0

writing about lmdb semantics, need some help
by bert hubert 23 Sep '14

23 Sep '14

Hi everybody, First, thank you for developing openldap & lmdb, both are appreciated. I ran openldap 'way back in the day' to power one of the first cable internet providers in The Netherlands, and once we shook out the bugs, it was great! Within PowerDNS, we have an LMDB backend for the highest performance, and we also want to use it for Metronome (https://github.com/ahupowerdns/metronome , a sort of mini-graphite). In order to do so, I'm documenting LMDB semantics. The LMDB documentation has most of this, but I'd like to get the semantics in one place. ( https://github.com/ahupowerdns/ahutils/blob/master/lmdb-semantics.md - with some great help from Howard already) A few questions: * - Do not have open an MDB database twice in the same process at * the same time. Not even from a plain open() call - close()ing it * breaks flock() advisory locking. I see this note, but I also see a lot about TLS locking. Is this remark still true? It is quite difficult for many applications to effectively serialize LMDB access this way, which would be required because: * - A thread can only use one transaction at a time, plus any child * transactions. Each transaction belongs to one thread. See below. * The #MDB_NOTLS flag changes this for read-only transactions. I note that the source does not actually use flock() but fcntl()-based locking, which does have the problem described above. So my question, if there is one process with many threads that need to read and update an lmdb, what is the recommended way to do that, short of single threading the access to lmdb? Bert

2 1

TXN in master
by Michael Ströder 22 Sep '14

22 Sep '14

HI! It seems TXN support has landed in git master. Is it ready for interop testing? Ciao, Michael.

2 3

writing about lmdb semantics, need some help
by bert hubert 21 Sep '14

21 Sep '14

(third attempt to get this posted) Hi everybody, First, thank you for developing openldap & lmdb, both are appreciated. I ran openldap 'way back in the day' to power one of the first cable internet providers in The Netherlands, and once we shook out the bugs, it was great! Within PowerDNS, we have an LMDB backend for the highest performance, and we also want to use it for Metronome (https://github.com/ahupowerdns/metronome , a sort of mini-graphite). In order to do so, I'm documenting LMDB semantics. The LMDB documentation has most of this, but I'd like to get the semantics in one place. ( https://github.com/ahupowerdns/ahutils/blob/master/lmdb-semantics.md - with some great help from Howard already) A few questions: * - Do not have open an MDB database twice in the same process at * the same time. Not even from a plain open() call - close()ing it * breaks flock() advisory locking. I see this note, but I also see a lot about TLS locking. Is this remark still true? It is quite difficult for many applications to effectively serialize LMDB access this way, which would be required because: * - A thread can only use one transaction at a time, plus any child * transactions. Each transaction belongs to one thread. See below. * The #MDB_NOTLS flag changes this for read-only transactions. I note that the source does not actually use flock() but fcntl()-based locking, which does have the problem described above. So my question, if there is one process with many threads that need to read and update an lmdb, what is the recommended way to do that, short of single threading the access to lmdb? Bert

1 0

Re: BadRSlotError: mdb_txn_begin: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot
by Howard Chu 11 Sep '14

11 Sep '14

Luke Kenneth Casson Leighton wrote: > garbage collection occurs periodically, at any time the interpreter > feels like it > (afaik). so, the scenario under consideration is therefore that a > cursor gets closed > at random points, well after its txn has been closed. > > however if the c api is reflected in the python layer then correspondingly the > cursor could not interfere with a txn. > > > the only other possibility under consideration is that the python > process has been > killed unexpectedly (kill -9) and has left the database in an abnormal state. Not possible; the database structure is inherently incorruptible. kill -9 won't do anything to it. > looking at the application logs that i have available (qty one only > at the moment) > i do note that the application appears to have just undergone a restart... You're talking about a database being accessed by only a single process; the lock table is reinitialized whenever a process opens it and no other processes already have it open. As such, no locktable elements from a previous crash will ever affect the running process. > > i'll have to have a closer look but would that be causing any problems (esp > with the arguments to speed up access such as sync=False etc.) Nope. sync is irrelevant for application-level crashes, all the data goes into the OS buffer cache regardless. sync=False will likely leave you a corrupted database if the OS or hardware crashes, but not if the application process crashes. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: BadRSlotError: mdb_txn_begin: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot
by Howard Chu 11 Sep '14

11 Sep '14

Luke Kenneth Casson Leighton wrote: > On Thu, Sep 11, 2014 at 11:37 AM, Howard Chu <hyc(a)symas.com> wrote: >> Luke Kenneth Casson Leighton wrote: >> >>> hi all, >>> >>> the infamous obscure error which people are seeing only very >>> infrequently is rearing its head at least 2 to 3 times per day in a >>> test lab where i work. this is however a secure environment so i >>> cannot post core-dumps or any details of the application. >>> >>> given the restrictions, what information is needed and what approach >>> is needed to debug and fix this? luckily it's happening a lot so >>> there's the possibility of a regular iterative approach. >>> >>> the operating system(s) have been ubuntu 12.04 and also 14.04, both >>> have resulted in this obscure bug. bizarrely, this bug occurs in a >>> *single process*. it's not even multi-processing. however >>> metasync=False, sync=False, map_async=True, readahead=False and >>> writemap=True. >> >> >> Use the Source, Luke. > > :) > >> MDB_BAD_RSLOT is returned only one place in mdb.c and the situation is very >> specific. It means you've tried to begin a new read txn on a thread that >> already has a read txn outstanding. > > ... but there aren't any threads... this is literally only one > process. there are no threads involved at all. the single process is > doing writes in a txn followed by reads in a separate txn. Technically, a single process is also a single thread. >> The API docs are pretty clear that a >> thread may only have one txn at a time. >> >> You need to track down whatever is creating read txns in your code and make >> sure they're being properly committed or aborted. > > this is from python, and all code is done using "with env.begin .... as txn:" > > there are no exceptions occurring within any blocks, and even if they > were the "with" statement calls the __exit__ function which closes the > transaction. I can't comment on anything python is doing, but it sounds like it's missing a step... > so, all code is as expected, hence the reason for raising it here > because this is definitely not something that should be happening. > > *thinks*... there is only one possible thing that i can think of, and > it's related to using cursors. i am not calling close or del on the > txn.cursor objects within the "with" block. could it be that python's > garbage collection is somehow collecting those txn.cursor objects at > random points, interacting in some way with the current read txn? No idea. If you're using py-lmdb it sounds like we need David Wilsom to chime in here. In the C API there's no way a cursor could interfere with a txn, no guesses what the python code is doing. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: BadRSlotError: mdb_txn_begin: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot
by Howard Chu 11 Sep '14

11 Sep '14

Luke Kenneth Casson Leighton wrote: > hi all, > > the infamous obscure error which people are seeing only very > infrequently is rearing its head at least 2 to 3 times per day in a > test lab where i work. this is however a secure environment so i > cannot post core-dumps or any details of the application. > > given the restrictions, what information is needed and what approach > is needed to debug and fix this? luckily it's happening a lot so > there's the possibility of a regular iterative approach. > > the operating system(s) have been ubuntu 12.04 and also 14.04, both > have resulted in this obscure bug. bizarrely, this bug occurs in a > *single process*. it's not even multi-processing. however > metasync=False, sync=False, map_async=True, readahead=False and > writemap=True. Use the Source, Luke. MDB_BAD_RSLOT is returned only one place in mdb.c and the situation is very specific. It means you've tried to begin a new read txn on a thread that already has a read txn outstanding. The API docs are pretty clear that a thread may only have one txn at a time. You need to track down whatever is creating read txns in your code and make sure they're being properly committed or aborted. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

MDB_TOOL mode vs. large mdb_env_set_maxreaders()?
by Hallvard Breien Furuseth 28 Aug '14

28 Aug '14

If an lmdb application configures a non-default maxreaders, this setting is ignored if a tool like mdb_copy is already running (maybe from crontab). I suggest mdb_copy & co should use an MDB_TOOL env flag which requires the lockfile (if any) to be valid, and only partly reinitializes it. The upcoming MDB_ROBUST flag might also make use of this. An alternative would be to stuff this into the metapages, but that seems less flexible. -- Hallvard

2 1

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel