Don't Use Copy Control (1.3.6.1.4.1.4203.666.5.15 vs. 1.3.6.1.1.22)
by Michael Ströder
HI!
Is there any issue with using OID 1.3.6.1.1.22 as controlType for Don't Use
Copy Control as defined in RFC 6171? Currently an experimental OpenLDAP .666
OID s used which makes this control rather unusable outside OpenLDAP
implementation.
Ciao, Michael.
8 years, 7 months
unique incrementing value with lmdb?
by CJ Ess
What is the best way to implement a unique incrementing value (for use as a
key or value) with lmdb?
Is there a property if the transaction mechanism I could piggy-back off of
to atomically generate my number?
Or should I block writers above mdb_cursor_put, do my lmdb transaction, and
unblock?
8 years, 8 months
Re: Problem: LDAP installation without internet
by Aaron Richton
On Fri, 22 Aug 2014, Patrick Pat wrote:
> Hi,
> I would like all LDAP installation procedure without internet in linux
> (ubuntu (9.10 or other), Debian, ...), but by its compressed file
>
> namely:
> openldap-2.4.8.tgz or openldap-2.4.7.tgz or openldap-2.4.9.tgz or
> openldap-2.0-beta.tgz ...
> And any configuration environment variables.
So inside that .tgz, there's a file called "INSTALL"; you might want to
take a look.
None of the procedures mentioned in "INSTALL" require anything other than
a vaguely sane POSIX-esque (or Windows, but let's not go there) system.
This includes any particular network connectivity, Internet or otherwise.
> best regards !
>
>
8 years, 8 months
writing about lmdb semantics, need some help
by bert hubert
Hi everybody,
First, thank you for developing openldap & lmdb, both are appreciated. I ran
openldap 'way back in the day' to power one of the first cable internet
providers in The Netherlands, and once we shook out the bugs, it was great!
Within PowerDNS, we have an LMDB backend for the highest performance, and we
also want to use it for Metronome (https://github.com/ahupowerdns/metronome
, a sort of mini-graphite).
In order to do so, I'm documenting LMDB semantics. The LMDB documentation
has most of this, but I'd like to get the semantics in one place.
( https://github.com/ahupowerdns/ahutils/blob/master/lmdb-semantics.md - with
some great help from Howard already)
A few questions:
* - Do not have open an MDB database twice in the same process at
* the same time. Not even from a plain open() call - close()ing it
* breaks flock() advisory locking.
I see this note, but I also see a lot about TLS locking. Is this remark
still true? It is quite difficult for many applications to effectively
serialize LMDB access this way, which would be required because:
* - A thread can only use one transaction at a time, plus any child
* transactions. Each transaction belongs to one thread. See below.
* The #MDB_NOTLS flag changes this for read-only transactions.
I note that the source does not actually use flock() but fcntl()-based
locking, which does have the problem described above.
So my question, if there is one process with many threads that need to read
and update an lmdb, what is the recommended way to do that, short of
single threading the access to lmdb?
Bert
8 years, 8 months
TXN in master
by Michael Ströder
HI!
It seems TXN support has landed in git master.
Is it ready for interop testing?
Ciao, Michael.
8 years, 8 months
writing about lmdb semantics, need some help
by bert hubert
(third attempt to get this posted)
Hi everybody,
First, thank you for developing openldap & lmdb, both are appreciated. I ran
openldap 'way back in the day' to power one of the first cable internet
providers in The Netherlands, and once we shook out the bugs, it was great!
Within PowerDNS, we have an LMDB backend for the highest performance, and we
also want to use it for Metronome (https://github.com/ahupowerdns/metronome
, a sort of mini-graphite).
In order to do so, I'm documenting LMDB semantics. The LMDB documentation
has most of this, but I'd like to get the semantics in one place.
( https://github.com/ahupowerdns/ahutils/blob/master/lmdb-semantics.md - with
some great help from Howard already)
A few questions:
* - Do not have open an MDB database twice in the same process at
* the same time. Not even from a plain open() call - close()ing it
* breaks flock() advisory locking.
I see this note, but I also see a lot about TLS locking. Is this remark
still true? It is quite difficult for many applications to effectively
serialize LMDB access this way, which would be required because:
* - A thread can only use one transaction at a time, plus any child
* transactions. Each transaction belongs to one thread. See below.
* The #MDB_NOTLS flag changes this for read-only transactions.
I note that the source does not actually use flock() but fcntl()-based
locking, which does have the problem described above.
So my question, if there is one process with many threads that need to read
and update an lmdb, what is the recommended way to do that, short of
single threading the access to lmdb?
Bert
8 years, 8 months
Re: BadRSlotError: mdb_txn_begin: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot
by Howard Chu
Luke Kenneth Casson Leighton wrote:
> garbage collection occurs periodically, at any time the interpreter
> feels like it
> (afaik). so, the scenario under consideration is therefore that a
> cursor gets closed
> at random points, well after its txn has been closed.
>
> however if the c api is reflected in the python layer then correspondingly the
> cursor could not interfere with a txn.
>
>
> the only other possibility under consideration is that the python
> process has been
> killed unexpectedly (kill -9) and has left the database in an abnormal state.
Not possible; the database structure is inherently incorruptible. kill -9
won't do anything to it.
> looking at the application logs that i have available (qty one only
> at the moment)
> i do note that the application appears to have just undergone a restart...
You're talking about a database being accessed by only a single process; the
lock table is reinitialized whenever a process opens it and no other processes
already have it open. As such, no locktable elements from a previous crash
will ever affect the running process.
>
> i'll have to have a closer look but would that be causing any problems (esp
> with the arguments to speed up access such as sync=False etc.)
Nope. sync is irrelevant for application-level crashes, all the data goes into
the OS buffer cache regardless. sync=False will likely leave you a corrupted
database if the OS or hardware crashes, but not if the application process
crashes.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
8 years, 8 months
Re: BadRSlotError: mdb_txn_begin: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot
by Howard Chu
Luke Kenneth Casson Leighton wrote:
> On Thu, Sep 11, 2014 at 11:37 AM, Howard Chu <hyc(a)symas.com> wrote:
>> Luke Kenneth Casson Leighton wrote:
>>
>>> hi all,
>>>
>>> the infamous obscure error which people are seeing only very
>>> infrequently is rearing its head at least 2 to 3 times per day in a
>>> test lab where i work. this is however a secure environment so i
>>> cannot post core-dumps or any details of the application.
>>>
>>> given the restrictions, what information is needed and what approach
>>> is needed to debug and fix this? luckily it's happening a lot so
>>> there's the possibility of a regular iterative approach.
>>>
>>> the operating system(s) have been ubuntu 12.04 and also 14.04, both
>>> have resulted in this obscure bug. bizarrely, this bug occurs in a
>>> *single process*. it's not even multi-processing. however
>>> metasync=False, sync=False, map_async=True, readahead=False and
>>> writemap=True.
>>
>>
>> Use the Source, Luke.
>
> :)
>
>> MDB_BAD_RSLOT is returned only one place in mdb.c and the situation is very
>> specific. It means you've tried to begin a new read txn on a thread that
>> already has a read txn outstanding.
>
> ... but there aren't any threads... this is literally only one
> process. there are no threads involved at all. the single process is
> doing writes in a txn followed by reads in a separate txn.
Technically, a single process is also a single thread.
>> The API docs are pretty clear that a
>> thread may only have one txn at a time.
>>
>> You need to track down whatever is creating read txns in your code and make
>> sure they're being properly committed or aborted.
>
> this is from python, and all code is done using "with env.begin .... as txn:"
>
> there are no exceptions occurring within any blocks, and even if they
> were the "with" statement calls the __exit__ function which closes the
> transaction.
I can't comment on anything python is doing, but it sounds like it's missing a
step...
> so, all code is as expected, hence the reason for raising it here
> because this is definitely not something that should be happening.
>
> *thinks*... there is only one possible thing that i can think of, and
> it's related to using cursors. i am not calling close or del on the
> txn.cursor objects within the "with" block. could it be that python's
> garbage collection is somehow collecting those txn.cursor objects at
> random points, interacting in some way with the current read txn?
No idea. If you're using py-lmdb it sounds like we need David Wilsom to chime
in here. In the C API there's no way a cursor could interfere with a txn, no
guesses what the python code is doing.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
8 years, 8 months
Re: BadRSlotError: mdb_txn_begin: MDB_BAD_RSLOT: Invalid reuse of reader locktable slot
by Howard Chu
Luke Kenneth Casson Leighton wrote:
> hi all,
>
> the infamous obscure error which people are seeing only very
> infrequently is rearing its head at least 2 to 3 times per day in a
> test lab where i work. this is however a secure environment so i
> cannot post core-dumps or any details of the application.
>
> given the restrictions, what information is needed and what approach
> is needed to debug and fix this? luckily it's happening a lot so
> there's the possibility of a regular iterative approach.
>
> the operating system(s) have been ubuntu 12.04 and also 14.04, both
> have resulted in this obscure bug. bizarrely, this bug occurs in a
> *single process*. it's not even multi-processing. however
> metasync=False, sync=False, map_async=True, readahead=False and
> writemap=True.
Use the Source, Luke.
MDB_BAD_RSLOT is returned only one place in mdb.c and the situation is very
specific. It means you've tried to begin a new read txn on a thread that
already has a read txn outstanding. The API docs are pretty clear that a
thread may only have one txn at a time.
You need to track down whatever is creating read txns in your code and make
sure they're being properly committed or aborted.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
8 years, 8 months
