openldap-devel

openldap-devel@openldap.org

1418 discussions

Notice to move the Fortress source to Apache Directory project
by Shawn McKinney 08 Oct '14

08 Oct '14

Hello, This is a notice of intent to move the following sources to apache directory project: 1. fortress core 2. realm 3. emnasse 4. commander The goal is to expand user adoption and gain commiters by working alongside projects of a similar nature, i.e. syncope, shiro, studio, etc. Over the past three years I have enjoyed the benefits of these sources being housed within the openldap project and will continue to promote fortress using openldap directory server as before. Indeed there are projects in our pipeline (rbac accelerator overlay) that will (should) remain under the openldap umbrella so my participation with openldap project will continue for the forseeable future. Shawn McKinney -- 650-963-7681 smckinney(a)symas.com

1 0

On-demand page mapping in LMDB
by Howard Chu 01 Oct '14

01 Oct '14

The vl32 branch on gitorious has some tweaks for LMDB to mmap on individual page accesses, instead of mmap'ing the entire DB file all at once. This is primarily to support usage on 32 bit machines, where the address space is too small to accomodate multiple DBs mapped in competely, all at once. https://gitorious.org/mdb/mdb/source/69d7cb8d44e04f02d8d0c923ae71fbaaa9f42f…: There's a significant performance hit since almost every page reference results in an mmap system call. But it also has a much smaller memory footprint. While the code passes simple tests, it is also broken in one significant aspect: Currently, advancing a cursor to a new page causes the previous page to be unmapped. If there are two cursors open on the same DBI in the same transaction, it's possible that the other cursor will still be pointed to the old page, and the unmap will break it. There's currently no check to prevent this, and since read txns don't track all of their open cursors, there is no way to check (unless we add refcounts to this array, which would be clumsy). The solution I'm considering is to get rid of the per-txn array of mapped pages that I added here, and just operate on the cursor stacks directly. In that case, every cursor will have its own set of mapped read-only pages, and nothing that happens on one cursor will have any effect on any other cursor. (For read-only pages. For dirty pages, the usual cursor/page fixups apply as always.) -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Incremental backup in LMDB
by Howard Chu 01 Oct '14

01 Oct '14

A preliminary branch has been available on gitorious for a few months. It has some issues though, which still need to be resolved. https://gitorious.org/mdb/mdb/source/a7fc32c0b39c29f9afd2172338060b079d549c…: The biggest problem is that scanning every page of the DB is unsafe - if a page was an overflow page, and was freed and partially reused, then a scan may walk into intermediate chunks of data (which won't have valid page headers, although the data might coincidentally appear to have them). Two approaches to solving this are to perform the scan, but ignore pages that are currently listed in the freeDB, or to do a regular tree walk instead. At the moment I'm leaning toward just walking the trees. If the freeDB is large then comparing every scanned page to the freeDB could be expensive. Also, newer pages of the freeDB itself must be copied as well, and this would mean freeDB pages get read twice. Just thinking out loud here to document what we discussed before in IRC, so it doesn't get lost in the meantime. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

unique incrementing value with lmdb?
by CJ Ess 23 Sep '14

23 Sep '14

What is the best way to implement a unique incrementing value (for use as a key or value) with lmdb? Is there a property if the transaction mechanism I could piggy-back off of to atomically generate my number? Or should I block writers above mdb_cursor_put, do my lmdb transaction, and unblock?

2 1

Re: Problem: LDAP installation without internet
by Aaron Richton 23 Sep '14

23 Sep '14

On Fri, 22 Aug 2014, Patrick Pat wrote: > Hi, > I would like all LDAP installation procedure without internet in linux > (ubuntu (9.10 or other), Debian, ...), but by its compressed file > > namely: > openldap-2.4.8.tgz or openldap-2.4.7.tgz or openldap-2.4.9.tgz or > openldap-2.0-beta.tgz ... > And any configuration environment variables. So inside that .tgz, there's a file called "INSTALL"; you might want to take a look. None of the procedures mentioned in "INSTALL" require anything other than a vaguely sane POSIX-esque (or Windows, but let's not go there) system. This includes any particular network connectivity, Internet or otherwise. > best regards ! > >

1 0

writing about lmdb semantics, need some help
by bert hubert 23 Sep '14

23 Sep '14

Hi everybody, First, thank you for developing openldap & lmdb, both are appreciated. I ran openldap 'way back in the day' to power one of the first cable internet providers in The Netherlands, and once we shook out the bugs, it was great! Within PowerDNS, we have an LMDB backend for the highest performance, and we also want to use it for Metronome (https://github.com/ahupowerdns/metronome , a sort of mini-graphite). In order to do so, I'm documenting LMDB semantics. The LMDB documentation has most of this, but I'd like to get the semantics in one place. ( https://github.com/ahupowerdns/ahutils/blob/master/lmdb-semantics.md - with some great help from Howard already) A few questions: * - Do not have open an MDB database twice in the same process at * the same time. Not even from a plain open() call - close()ing it * breaks flock() advisory locking. I see this note, but I also see a lot about TLS locking. Is this remark still true? It is quite difficult for many applications to effectively serialize LMDB access this way, which would be required because: * - A thread can only use one transaction at a time, plus any child * transactions. Each transaction belongs to one thread. See below. * The #MDB_NOTLS flag changes this for read-only transactions. I note that the source does not actually use flock() but fcntl()-based locking, which does have the problem described above. So my question, if there is one process with many threads that need to read and update an lmdb, what is the recommended way to do that, short of single threading the access to lmdb? Bert

2 1

TXN in master
by Michael Ströder 22 Sep '14

22 Sep '14

HI! It seems TXN support has landed in git master. Is it ready for interop testing? Ciao, Michael.

2 3

writing about lmdb semantics, need some help
by bert hubert 21 Sep '14

21 Sep '14

(third attempt to get this posted) Hi everybody, First, thank you for developing openldap & lmdb, both are appreciated. I ran openldap 'way back in the day' to power one of the first cable internet providers in The Netherlands, and once we shook out the bugs, it was great! Within PowerDNS, we have an LMDB backend for the highest performance, and we also want to use it for Metronome (https://github.com/ahupowerdns/metronome , a sort of mini-graphite). In order to do so, I'm documenting LMDB semantics. The LMDB documentation has most of this, but I'd like to get the semantics in one place. ( https://github.com/ahupowerdns/ahutils/blob/master/lmdb-semantics.md - with some great help from Howard already) A few questions: * - Do not have open an MDB database twice in the same process at * the same time. Not even from a plain open() call - close()ing it * breaks flock() advisory locking. I see this note, but I also see a lot about TLS locking. Is this remark still true? It is quite difficult for many applications to effectively serialize LMDB access this way, which would be required because: * - A thread can only use one transaction at a time, plus any child * transactions. Each transaction belongs to one thread. See below. * The #MDB_NOTLS flag changes this for read-only transactions. I note that the source does not actually use flock() but fcntl()-based locking, which does have the problem described above. So my question, if there is one process with many threads that need to read and update an lmdb, what is the recommended way to do that, short of single threading the access to lmdb? Bert

1 0

Re: BadRSlotError: mdb_txn_begin: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot
by Howard Chu 11 Sep '14

11 Sep '14

Luke Kenneth Casson Leighton wrote: > garbage collection occurs periodically, at any time the interpreter > feels like it > (afaik). so, the scenario under consideration is therefore that a > cursor gets closed > at random points, well after its txn has been closed. > > however if the c api is reflected in the python layer then correspondingly the > cursor could not interfere with a txn. > > > the only other possibility under consideration is that the python > process has been > killed unexpectedly (kill -9) and has left the database in an abnormal state. Not possible; the database structure is inherently incorruptible. kill -9 won't do anything to it. > looking at the application logs that i have available (qty one only > at the moment) > i do note that the application appears to have just undergone a restart... You're talking about a database being accessed by only a single process; the lock table is reinitialized whenever a process opens it and no other processes already have it open. As such, no locktable elements from a previous crash will ever affect the running process. > > i'll have to have a closer look but would that be causing any problems (esp > with the arguments to speed up access such as sync=False etc.) Nope. sync is irrelevant for application-level crashes, all the data goes into the OS buffer cache regardless. sync=False will likely leave you a corrupted database if the OS or hardware crashes, but not if the application process crashes. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: BadRSlotError: mdb_txn_begin: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot
by Howard Chu 11 Sep '14

11 Sep '14

Luke Kenneth Casson Leighton wrote: > On Thu, Sep 11, 2014 at 11:37 AM, Howard Chu <hyc(a)symas.com> wrote: >> Luke Kenneth Casson Leighton wrote: >> >>> hi all, >>> >>> the infamous obscure error which people are seeing only very >>> infrequently is rearing its head at least 2 to 3 times per day in a >>> test lab where i work. this is however a secure environment so i >>> cannot post core-dumps or any details of the application. >>> >>> given the restrictions, what information is needed and what approach >>> is needed to debug and fix this? luckily it's happening a lot so >>> there's the possibility of a regular iterative approach. >>> >>> the operating system(s) have been ubuntu 12.04 and also 14.04, both >>> have resulted in this obscure bug. bizarrely, this bug occurs in a >>> *single process*. it's not even multi-processing. however >>> metasync=False, sync=False, map_async=True, readahead=False and >>> writemap=True. >> >> >> Use the Source, Luke. > > :) > >> MDB_BAD_RSLOT is returned only one place in mdb.c and the situation is very >> specific. It means you've tried to begin a new read txn on a thread that >> already has a read txn outstanding. > > ... but there aren't any threads... this is literally only one > process. there are no threads involved at all. the single process is > doing writes in a txn followed by reads in a separate txn. Technically, a single process is also a single thread. >> The API docs are pretty clear that a >> thread may only have one txn at a time. >> >> You need to track down whatever is creating read txns in your code and make >> sure they're being properly committed or aborted. > > this is from python, and all code is done using "with env.begin .... as txn:" > > there are no exceptions occurring within any blocks, and even if they > were the "with" statement calls the __exit__ function which closes the > transaction. I can't comment on anything python is doing, but it sounds like it's missing a step... > so, all code is as expected, hence the reason for raising it here > because this is definitely not something that should be happening. > > *thinks*... there is only one possible thing that i can think of, and > it's related to using cursors. i am not calling close or del on the > txn.cursor objects within the "with" block. could it be that python's > garbage collection is somehow collecting those txn.cursor objects at > random points, interacting in some way with the current read txn? No idea. If you're using py-lmdb it sounds like we need David Wilsom to chime in here. In the C API there's no way a cursor could interfere with a txn, no guesses what the python code is doing. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel