openldap-bugs

openldap-bugs@openldap.org

1 participants
21003 discussions

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by wietse＠porcupine.org 02 Jan '14

02 Jan '14

Hallvard Breien Furuseth: > Wietse Venema writes: > > Why do you talk about map size changes when I delete a database entry?? > > That's how Postfix uses lmdb. It calls mdb_env_set_mapsize(). Looking As written in the bug report the assertion also failed with mdb_map_size set to an insanely large value (half a billion) in which case the map size never changes. But, it is good to know that the cursor is invalidated by both map size changes and transactions that change the database. I can confirm that with LMDB-0.9.10 + ITS#7756 patch, the assertion failure goes away when I close the cursor (and its read transaction) before changing the map or the database, and restore the cursor afterwards. I guess this got broken when things were changed to use external locks in order to avoid world-writable lockfiles. Wietse

1 0

Re: (ITS#7776) Bulk user add on ldap server
by quanah＠zimbra.com 02 Jan '14

02 Jan '14

--On Thursday, January 02, 2014 9:26 AM +0000 s.puspharaj(a)gmail.com wrote: > Full_Name: Puspharaj > Version: $OpenLDAP: slapd 2.4.23 (Apr 22 2013 05:03:41) > OS: Red Hat Enterprise Linux Server release 6.5 (Santiago) > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (103.4.125.30) The ITS system is for reporting bugs, not asking usage questions. I would advise you to take your question to the openldap-technical(a)openldap.org list. I would note a few things: a) The version of OpenLDAP shipped by RHEL is extremely broken. If you insist on using it, you must contact Red Hat for support. b) You can get up to date current OpenLDAP builds from Symas (http://www.symas.com) or the LTB project (<http://ltb-project.org/wiki/download#openldap>) c) I'm going to guess you are using back-bdb or back-hdb. Since you have groups with large membership, it is entirely possible you are running out of locks, lockers, or lock objects. I would advise setting an appropriate debugging level and looking for errors originating from Berkeley DB. --Quanah -- Quanah Gibson-Mount Architect - Server Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

(ITS#7776) Bulk user add on ldap server
by s.puspharaj＠gmail.com 02 Jan '14

02 Jan '14

Full_Name: Puspharaj Version: $OpenLDAP: slapd 2.4.23 (Apr 22 2013 05:03:41) OS: Red Hat Enterprise Linux Server release 6.5 (Santiago) URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (103.4.125.30) Hi There, I have around 3000 user in people section were they hold multiple attributes and other things. The problem is, I have couple of groups that holds no user in it, When i tried to add users using ldapmodify command it sometimes not responding or it says the user exist even if the users won't exist in that group. Groupadd.ldif file contains : ################################################################# # dn: cn=typo3_frontend_global_salespartner,ou=Groups,o=company # # changetype: modify # # add: member # # member: uid=alesgian,ou=internal,c=it,ou=People,o=company # # member: uid=alesspin,ou=internal,c=it,ou=People,o=company # ################################################################# Somtimes the uid contains some special characters inbetween like . # , . When the user has the above characters it won't respond or it says the user exist. But when i added manually or with single line of user instead of 1000 user in that file. It accepts the ldapmodify command with that file. The command is ldapmodify -D "cn=Directory Manager,o=company" -f Groupadd.ldif -W This openldap version has any bug related to ldapmodify when adding 2000 users to a group through a single file? Thanks, Puspharaj

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by hyc＠symas.com 01 Jan '14

01 Jan '14

Wietse Venema wrote: > Hallvard Breien Furuseth: >>> Sofar the abstraction layer already hides the LMDB-specific MAP_FULL >>> and MAP_RESIZED error conditions. If this abstraction layer needs >>> additional code in order to maintain MDB cursor sanity, then please >>> educate me. >> >> ldmb.h says -- >> >> mdb_env_set_mapsize(): >> "It may be called at later times if no transactions are active in >> this process. Note that the library does not check for this condition, >> the caller must ensure it explicitly." >> MDB_NOLOCK: >> "[caller] must ensure that no readers are using old transactions >> while a writer is active. The simplest approach is to use an >> exclusive lock so that no readers may be active at all when a writer >> begins." >> >> That's why I talked about saving the cursor position and restoring it - >> cursors are per-transaction and you need a new transaction. > > I can remember the last key from mdb_cursor_get() and set the cursor > to that key. There does not appear to be a cursor "save" operation > in the API documentation http://symas.com/mdb/doc/group__mdb.html. Hallvard is suggesting exactly that - remember the key. There is no cursor "save" operation, any saved state would be meaningless after one write transaction. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by wietse＠porcupine.org 01 Jan '14

01 Jan '14

Hallvard Breien Furuseth: > > Sofar the abstraction layer already hides the LMDB-specific MAP_FULL > > and MAP_RESIZED error conditions. If this abstraction layer needs > > additional code in order to maintain MDB cursor sanity, then please > > educate me. > > ldmb.h says -- > > mdb_env_set_mapsize(): > "It may be called at later times if no transactions are active in > this process. Note that the library does not check for this condition, > the caller must ensure it explicitly." > MDB_NOLOCK: > "[caller] must ensure that no readers are using old transactions > while a writer is active. The simplest approach is to use an > exclusive lock so that no readers may be active at all when a writer > begins." > > That's why I talked about saving the cursor position and restoring it - > cursors are per-transaction and you need a new transaction. I can remember the last key from mdb_cursor_get() and set the cursor to that key. There does not appear to be a cursor "save" operation in the API documentation http://symas.com/mdb/doc/group__mdb.html. > Maybe you should have a single write-transaction with several cursors, > instead of several transactions. That also cures the "long-lived reader" > problem. That would result in delayed delete operations, and break correctness. If a database entry is updated after it is scheduled for delete, then the entry must not be deleted. Also, I am concerned that would pollute my database-neutral API with LMDB-specific details. Wietse > Howard, maybe NOLOCK should keep the reader table and just drop the > locks? More work though. > > -- > Hallvard

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by h.b.furuseth＠usit.uio.no 31 Dec '13

31 Dec '13

Wietse Venema writes: > Why do you talk about map size changes when I delete a database entry?? That's how Postfix uses lmdb. It calls mdb_env_set_mapsize(). Looking closer though, this is in src/util/slmdb.c which is by Howard and you. Sounds like we should wait for whatever Howard has to say. > I need a software abstraction layer between Postfix and LMDB that > provides the following generic interface that is independent of > LMDB internals: > (...snip...) It does, normally. NOLOCK and map resizing interfere, and impose notable limitations. > Sofar the abstraction layer already hides the LMDB-specific MAP_FULL > and MAP_RESIZED error conditions. If this abstraction layer needs > additional code in order to maintain MDB cursor sanity, then please > educate me. ldmb.h says -- mdb_env_set_mapsize(): "It may be called at later times if no transactions are active in this process. Note that the library does not check for this condition, the caller must ensure it explicitly." MDB_NOLOCK: "[caller] must ensure that no readers are using old transactions while a writer is active. The simplest approach is to use an exclusive lock so that no readers may be active at all when a writer begins." That's why I talked about saving the cursor position and restoring it - cursors are per-transaction and you need a new transaction. Maybe you should have a single write-transaction with several cursors, instead of several transactions. That also cures the "long-lived reader" problem. Howard, maybe NOLOCK should keep the reader table and just drop the locks? More work though. -- Hallvard

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by wietse＠porcupine.org 31 Dec '13

31 Dec '13

Wietse Venema: > > You're using an old read-only transaction which cannot coexist with: > > - mdb_env_set_mapsize() which moves the map which a cursor in > > the reader is using. > > - several write-transactions + MDB_NOLOCK. The flag means the writers do > > not know about the reader, so they reuse pages from the snapshot the > > reader is using. The reader can survive while the metapages hold > > on to its snapshot, i.e. 1 or 2 write commits (I think). I am willing to be educated. Where in the LMDB API documentation would I have learned that I have to consider these details? Wietse

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by wietse＠porcupine.org 31 Dec '13

31 Dec '13

Hallvard Breien Furuseth: > [Due to a typo in your e-mail address, the ITS system did not mail out > this message anywhere. So I'm CC'ing Howard directly, just in case.] Sorry about that. Seems everyone screws up sometimes. > Wietse Venema writes: > > I wrote a test driver that reliably causes LMDB to abort during a > > simulated cache cleanup. This "exploit" produces the same result > > on Linux and FreeBSD, 32-bit and 64-bit systems. > > You're using an old read-only transaction which cannot coexist with: > - mdb_env_set_mapsize() which moves the map which a cursor in > the reader is using. > - several write-transactions + MDB_NOLOCK. The flag means the writers do > not know about the reader, so they reuse pages from the snapshot the > reader is using. The reader can survive while the metapages hold > on to its snapshot, i.e. 1 or 2 write commits (I think). > > I don't know if this is a thinko in your program or miscommunication > between you and Howard about MDB_NOLOCK and mapsize changes. With the > current liblmdb, a map change should involve: Remember the reader's > current position (key), resize the map, renew the txn and cursor, and > reposition the cursor. Why do you talk about map size changes when I delete a database entry?? I need a software abstraction layer between Postfix and LMDB that provides the following generic interface that is independent of LMDB internals: 1) Give me the first or the next element in the database. 2) Delete, look up, update, a specific database entry. 3) Operations under 1) and 2) must be possible to interleave. 4) Locking must be done outside LMDB, because world-writable lock files are not an option. (There is more, but the above is required to use LMDB as a cache with periodic cleanup. I am not makig up stuff here - the above interface has been used in Postfix for many years and it works with all supported databases that have an iterator. Sofar the abstraction layer already hides the LMDB-specific MAP_FULL and MAP_RESIZED error conditions. If this abstraction layer needs additional code in order to maintain MDB cursor sanity, then please educate me. I had expected that LMDB takes care of its cursors itself, since the side effects of other API calls are known inside LMDB only. Wietse > The test works if I (a) turn MDB_NOLOCK into MDB_NOTLS (I know that's > not what you want), and (b) detect map changes in mdb_cursor_get() and > update the cursor to match. > Old 'MDB_val's the reader fetched, are invalid after the mapsize change. > Also, remember that long-lived read-only transactions which write > transactions do know about, prevent them from reusing pages the reader > snapshot is using - resulting in further map growth. > > -- > Hallvard

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by h.b.furuseth＠usit.uio.no 31 Dec '13

31 Dec '13

Wietse Venema writes: > (...) > A few alternatives: > > - When a request fails, return a distinct error code such as > MDB_PANIC, and allow the application to look up the text for the > error. For internal errors, yes. It'll take a fair amount of work to get there. And I don't expect ldmb to become enthusiastic at catching user errors, so those may cause crashes or other misbehavior anyway. After all you can disable asserts with NDEBUG, but then you may get e.g. NULL dereferences instead. > - Invoke an application call-back function with the error code and > problem description text. If the error code is MDB_PANIC then > the application knows that it needs to make final arrangements. Sounds good to me for now, since it doesn't mean walking all over the code. mdb.c can just #define its own assert(). No error code, the context is just "we're about to abort" and I don't know of other planned callbacks which the code would help. /** User-settable callback for assert(), called before abort() * instead of printing the message. * * assert(some user errors) might be unaffected. This hack * should become obsolete as lmdb's error handling matures. * @param[in] msg The assertion message, not including newline. * @parem[in] info Currently NULL. Might change in the future. */ extern void (*mdb_assert_cb)(char *msg, void *info); (Or maybe call it -before- printing the message, you can abort or longjmp yourself if you don't want the message.) > The second option preserves 100% backwards compatibility. The first > option may cause functions to return a result that they didn't > return previously. This is about situations where mdb didn't return at all, so compatibility with old behavior will rarely be a problem with new error codes. -- Hallvard

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by h.b.furuseth＠usit.uio.no 31 Dec '13

31 Dec '13

[Due to a typo in your e-mail address, the ITS system did not mail out this message anywhere. So I'm CC'ing Howard directly, just in case.] Wietse Venema writes: > I wrote a test driver that reliably causes LMDB to abort during a > simulated cache cleanup. This "exploit" produces the same result > on Linux and FreeBSD, 32-bit and 64-bit systems. You're using an old read-only transaction which cannot coexist with: - mdb_env_set_mapsize() which moves the map which a cursor in the reader is using. - several write-transactions + MDB_NOLOCK. The flag means the writers do not know about the reader, so they reuse pages from the snapshot the reader is using. The reader can survive while the metapages hold on to its snapshot, i.e. 1 or 2 write commits (I think). I don't know if this is a thinko in your program or miscommunication between you and Howard about MDB_NOLOCK and mapsize changes. With the current liblmdb, a map change should involve: Remember the reader's current position (key), resize the map, renew the txn and cursor, and reposition the cursor. The test works if I (a) turn MDB_NOLOCK into MDB_NOTLS (I know that's not what you want), and (b) detect map changes in mdb_cursor_get() and update the cursor to match. Old 'MDB_val's the reader fetched, are invalid after the mapsize change. Also, remember that long-lived read-only transactions which write transactions do know about, prevent them from reusing pages the reader snapshot is using - resulting in further map growth. -- Hallvard

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs