openldap-bugs January 2014

openldap-bugs@openldap.org

23 participants
89 discussions

(ITS#7777) Filtering on modifyTimestamp is slower with index
by raphael.ouazana＠linagora.com 09 Jan '14

09 Jan '14

Full_Name: Raphael Ouazana Version: 2.4.38 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (88.173.78.196) Hi, I have a case where using modifyTimestamp index is far slower than with no index. First with indexing: ###Caching $ time ldapsearch -b <people> -D <manager> -W # search result search: 2 result: 0 Success # numResponses: 293886 # numEntries: 293885 real 0m12.153s user 0m7.805s sys 0m1.888s ### Request $ time ldapsearch -b <people> -D <manager> -W '(&(modifyTimestamp>=20131023163804Z)(modifyTimestamp<=20131123163804Z))' # search result search: 2 result: 0 Success # numResponses: 24277 # numEntries: 24276 real 0m23.441s user 0m0.554s sys 0m0.073s Now without modifyTimestamp index: ###Caching $ time ldapsearch -b <people> -D <manager> -W # search result search: 2 result: 0 Success # numResponses: 293886 # numEntries: 293885 real 0m11.627s user 0m7.354s sys 0m1.772s ### Request $ time ldapsearch -b <people> -D <manager> -W '(&(modifyTimestamp>=20131023163804Z)(modifyTimestamp<=20131123163804Z))' # search result search: 2 result: 0 Success # numResponses: 24277 # numEntries: 24276 real 0m0.701s user 0m0.569s sys 0m0.070s As you can see, without indexing the request takes less than 1s, while with index it takes more than 20s. I think it is not the expected result. Regards, Raphaël Ouazana.

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by wietse＠porcupine.org 08 Jan '14

08 Jan '14

Hallvard Breien Furuseth: > On 2014-01-08 18:23, Howard Chu wrote: > > Wietse Venema wrote: > >> Hallvard Breien Furuseth: > >>> Process A may see garbage data. > > > > This is wrong. The fcntl locks will definitely prevent two processes > > from interfering with each other. The reason this bug occurs is > > because fcntl locks cannot prevent two transactions within a single > > process from stomping on each other. > > Hmm. OK, after some discussion with Howard: > > This was so with Howard's original code, it wrapped each mdb_get() in > a transaction. postfix-2.11-20131228/src/util/slmdb.c breaks this by > preserving the read transaction until an error happens which is bad > enough for the code to notice. The Postfix first/next iterator has always used a long-term read transaction, ever since the first LMDB patch for Postfix. This broke with the switch to MDB_NOLOCK. > Locks around cursors are pointless as well as insufficient from > lmdb's point of view. It is transactions which must be protected. With MDB_NOLOCK the cursor's read transaction must be protected, and it must be closed before returning the (key, value) to the caller. Wietse

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by h.b.furuseth＠usit.uio.no 08 Jan '14

08 Jan '14

On 2014-01-08 18:23, Howard Chu wrote: > Wietse Venema wrote: >> Hallvard Breien Furuseth: >>> Process A may see garbage data. > > This is wrong. The fcntl locks will definitely prevent two processes > from interfering with each other. The reason this bug occurs is > because fcntl locks cannot prevent two transactions within a single > process from stomping on each other. Hmm. OK, after some discussion with Howard: This was so with Howard's original code, it wrapped each mdb_get() in a transaction. postfix-2.11-20131228/src/util/slmdb.c breaks this by preserving the read transaction until an error happens which is bad enough for the code to notice. Locks around cursors are pointless as well as insufficient from lmdb's point of view. It is transactions which must be protected. -- Hallvard

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by hyc＠symas.com 08 Jan '14

08 Jan '14

Wietse Venema wrote: > Hallvard Breien Furuseth: >> Wietse Venema writes: >>> (...) what happens when the two above activities happen >>> in different processes? That is, process A (opens the database with >>> MDB_NOLOCK. opens a cursor, and walks the database without making >>> any changes to it), while process B opens the same database with >>> MDB_NOLOCK and performs the delete transactions. >>> >>> Will the process A cursor read transaction trigger an assertion due >>> to the process B delete transactions? If not, why not? >> >> Process A may see garbage data. This is wrong. The fcntl locks will definitely prevent two processes from interfering with each other. The reason this bug occurs is because fcntl locks cannot prevent two transactions within a single process from stomping on each other. This can result in an assert(), a >> segfault or other misbehavior. Just like if you had process A == B. >> With MDB_NOLOCK, after a write transaction commits you need to >> end existing read-only transactions before starting a new writer. >> (Well, it's a bit looser than that, but that's a simple rule.) > > Thanks for confirming my suspicion. I think that the correct option > then is to use only short-lived cursor read transactions within > fcntl()-protected blocks and to save/restore the previous cursor > key where applicable. > > Wietse > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by h.b.furuseth＠usit.uio.no 08 Jan '14

08 Jan '14

I write: > Maybe MDB_NOLOCK should have kept the lockfile and reader table, and > just omitted the mutexes/semaphores that synchronize transactions. Duh, nevermind. As Howard reminded me, that'd require readers to have write access to the lockfile, which was the problem which started all this. -- Hallvard

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by wietse＠porcupine.org 08 Jan '14

08 Jan '14

Hallvard Breien Furuseth: > Wietse Venema writes: > > (...) what happens when the two above activities happen > > in different processes? That is, process A (opens the database with > > MDB_NOLOCK. opens a cursor, and walks the database without making > > any changes to it), while process B opens the same database with > > MDB_NOLOCK and performs the delete transactions. > > > > Will the process A cursor read transaction trigger an assertion due > > to the process B delete transactions? If not, why not? > > Process A may see garbage data. This can result in an assert(), a > segfault or other misbehavior. Just like if you had process A == B. > With MDB_NOLOCK, after a write transaction commits you need to > end existing read-only transactions before starting a new writer. > (Well, it's a bit looser than that, but that's a simple rule.) Thanks for confirming my suspicion. I think that the correct option then is to use only short-lived cursor read transactions within fcntl()-protected blocks and to save/restore the previous cursor key where applicable. Wietse

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by h.b.furuseth＠usit.uio.no 08 Jan '14

08 Jan '14

Wietse Venema writes: > (...) what happens when the two above activities happen > in different processes? That is, process A (opens the database with > MDB_NOLOCK. opens a cursor, and walks the database without making > any changes to it), while process B opens the same database with > MDB_NOLOCK and performs the delete transactions. > > Will the process A cursor read transaction trigger an assertion due > to the process B delete transactions? If not, why not? Process A may see garbage data. This can result in an assert(), a segfault or other misbehavior. Just like if you had process A == B. With MDB_NOLOCK, after a write transaction commits you need to end existing read-only transactions before starting a new writer. (Well, it's a bit looser than that, but that's a simple rule.) Because: A read-only transaction needs a frozen, valid snapshot of the DB. Normally it registers itself in the lockfile, so writers know to not touch the pages that make up that snapshot. MDB_NOLOCK disables the lockfile, leaving those pages unprotected from writers. Maybe MDB_NOLOCK should have kept the lockfile and reader table, and just omitted the mutexes/semaphores that synchronize transactions. OTOH mapsize changes in B are not a problem for _existing_ transactions in A, since it was actually the accompanying map _address_ change which broke existing transactions. But if A tries to begin new transactions, mdb_txn_begin() may soon fail with MDB_MAP_RESIZED until A also resizes. -- Hallvard

1 0

Re: (ITS#7774) LMDB assertion failure during Postfix cache cleanup
by wietse＠porcupine.org 08 Jan '14

08 Jan '14

This is a follow-up question, not a request to reopen the issue. It is about what happens when different processes open the same database with MDB_NOLOCK, and then perform the exact same operations that were the subject of ITS#7774. Brief summary of ITS#7774: when an LMDB database is opened with MDB_NOLOCK, a long-lived cursor's read transaction triggers an assertion when the same process performs a delete transaction on a different database element. This happened when walking an LMDB-based cache to remove "expired" entries. The application uses fcntl() shared locks around cursor_get() and mdb_get() including all related MDB operations, and it uses fcntl() exclusive locks around mdb_put() and mdb_del() including all related MDB operations. The solution was to close the long-lived cursor's read transaction before the delete operation. Now my question: what happens when the two above activities happen in different processes? That is, process A (opens the database with MDB_NOLOCK. opens a cursor, and walks the database without making any changes to it), while process B opens the same database with MDB_NOLOCK and performs the delete transactions. Will the process A cursor read transaction trigger an assertion due to the process B delete transactions? If not, why not? Wietse

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by wietse＠porcupine.org 07 Jan '14

07 Jan '14

Quanah Gibson-Mount: > --On Tuesday, January 07, 2014 10:03 PM +0000 wietse(a)porcupine.org wrote: > > > Having obtained the patch off-list, I confirm that this call-back > > feature works satisfactorily, and that this ITS can be closed. > > > > I am looking forward to the next LMDB release. > > Hi Wietse, > > I'm currently working with the dev team to get a new release out in the > next week or so. Please do not speed up the process at the cost of code quality. I'm ready to turn on support for LMDB >= 0.9.11 in Postfix 2.11.0, as long as I know that this ITS will be addressed in 0.9.11. Wietse

1 0

Re: (ITS#7775) LMDB terminates Postfix daemon process without logfile record
by quanah＠zimbra.com 07 Jan '14

07 Jan '14

--On Tuesday, January 07, 2014 10:03 PM +0000 wietse(a)porcupine.org wrote: > Having obtained the patch off-list, I confirm that this call-back > feature works satisfactorily, and that this ITS can be closed. > > I am looking forward to the next LMDB release. Hi Wietse, I'm currently working with the dev team to get a new release out in the next week or so. --Quanah -- Quanah Gibson-Mount Architect - Server Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

← Newer
1
2
3
4
5
6
7
8
9
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs January 2014