- openldap-bugs - openldap.org

Re: (ITS#8339) LMDB robust mutex fix
by openldap＠gulag.ch 02 Jun '16

02 Jun '16

Hi I'm fully aware of the fact that Solaris is not the most widely-used platform on earth... Is there any chance to get this patch evaluated? I'm glad to help with whatever is needed. Thank you Rolf On 12/20/15 02:54 AM, openldap-its(a)OpenLDAP.org wrote: > *** THIS IS AN AUTOMATICALLY GENERATED REPLY *** > > Thanks for your report to the OpenLDAP Issue Tracking System. Your > report has been assigned the tracking number ITS#8339. > > One of our support engineers will look at your report in due course. > Note that this may take some time because our support engineers > are volunteers. They only work on OpenLDAP when they have spare > time. > > If you need to provide additional information in regards to your > issue report, you may do so by replying to this message. Note that > any mail sent to openldap-its(a)openldap.org with (ITS#8339) > in the subject will automatically be attached to the issue report. > > mailto:openldap-its@openldap.org?subject=(ITS#8339) > > You may follow the progress of this report by loading the following > URL in a web browser: > http://www.OpenLDAP.org/its/index.cgi?findid=8339 > > Please remember to retain your issue tracking number (ITS#8339) > on any further messages you send to us regarding this report. If > you don't then you'll just waste our time and yours because we > won't be able to properly track the report. > > Please note that the Issue Tracking System is not intended to > be used to seek help in the proper use of OpenLDAP Software. > Such requests will be closed. > > OpenLDAP Software is user supported. > http://www.OpenLDAP.org/support/ > > -------------- > Copyright 1998-2007 The OpenLDAP Foundation, All Rights Reserved. >

1 0

Re: (ITS#8431) LMDB: Access newly opened database from another transaction
by h.b.furuseth＠usit.uio.no 30 May '16

30 May '16

On 30. mai 2016 18:08, Juerg Bircher wrote: > If I got it right it is all about protecting against invalid DBIs > which can happen for example if someone closes database "A" which was > assigned DBI "5" and later the database "B" is opened and the free > slot 5 is then assigned to "B". So we could be misled using DBI "5" > having in mind database "A". > > So why don't we extend the MDB_DBI by adding the sequence in the lower > or higher bits. (...) Ugh. I didn't expect all my words to miss that badly. Yes, you caught one issue. There are more. The central point is what I wrote here: >> Multi-threaded use of structures, like the ones behind DBIs, must be >> synchronized. But DBIs and mdb_dbi_open() are not mutex-protected, >> nor are read-only txns (after the per-thread setup). So it's up to >> the user. And LMDB mostly can't catch users at less than txn-sized >> threading bugs - like the DBI workarounds people keep trying. I shouldn't really be the one to answer this, Howard might answer differently if he didn't just fetch a flamethrower. But anyway: It's about keeping a clean model of what is going on, which helps understanding and correctly coding both LMDB itself and user programs. When you don't work against the model, anyway. It's about keeping LMDB's internal data structures consistent. LMDB must know what the heck is going on when it manages and uses DBIs. Things like ensuring that two threads do not run mdb_dbi_open() at the same time, overwriting each others' work and creating a mess. It's about LMDB being able to know it delivers correct results, at least if the user doesn't work too hard at thwarting it. This usually means serializing various parts of the code using locks or something similar. But for the sake of speed and simplicity, LMDB has no relevant locks. It relies on the user to lock and has some code to catch some locking errors. One key word is "tries" - without locks or other expensive code, it can't always catch them even when it does try. So DBI errors are not "this DBI is broken, so drop/refresh it" but "your program is buggy, so fix it". And it's about living with the design choices and tradeoffs which LMDB has made, e.g. not slowing down read-only txns for anything like locking or extensive error checking. So anyway, this ITS is about changes to LMDB which will not happen. Instead it reminded me to add some checks for user errors with DBIs. -- Hallvard

1 0

Re: (ITS#8431) LMDB: Access newly opened database from another transaction
by juerg.bircher＠gmail.com 30 May '16

30 May '16

On Mon, May 30, 2016 at 2:44 PM, Hallvard Breien Furuseth <h.b.furuseth(a)usit.uio.no> wrote: > In ITS#8430, Juerg Bircher wrote: >> >> So if you drop ITS#8431 it would at least be very helpful to get an >> separate error like MDB_INVALID_DPI instead of EINVAL So the >> application sharing and caching handles (dbi) could handle the >> situation appropriate like use mdb_env_open() so that the dbi becomes >> valid in that txn as well. > > > This is ITS#8431 in a new guise, so I'm replying here. We know DBI > handling is inconvenient. We'd have fixed it if it were easy without > adding a bunch of locks. I'll walk through some of it below, I expect > it'll be useful to have for future reference anyway: > > > You mean like this? > > if (mdb_something(txn, dbi, ...) == MDB_INVALID_DBI) { > /* Get <dbi>, which some other txn committed after <txn> began */ > mdb_dbi_open(txn, "database name of <dbi>", 0, &dbi2); > assert(dbi2 == dbi); > } > > That breaks this requirement from the mdb_dbi_open() doc: > > * A transaction that uses > * this function must finish (either commit or abort) before > * any other transaction in the process may use this function. > Just to illustrate the simplified sequence of calls (using MDB_NOTLS): mdb_txn_begin(env, NULL, MDB_RDONLY, &txn2); mdb_txn_begin(env, NULL, MDB_RDONLY, &txn); MDB_dbi dbi; mdb_dbi_open(txn, "db1", 0, &dbi); // dbi becomes now valid and is propagated to the env mdb_txn_commit(txn); // At this point dbi for txn2 is invalid as txn2 started before opening the database if (mdb_something(txn2, dbi, ...) == MDB_INVALID_DBI) { MDB_dbi dbi2; mdb_dbi_open(txn2, "db1", 0, &dbi2); assert(dbi2 == dbi); // now dbi is valid for txn2 as with the mdb_dbi_open we propagated it to txn2! mdb_something(txn2, dbi, ...); } mdb_txn_commit(txn2); This code is really not preferred and is only a workaround. > To fulfill that, you must serialize txns which use mdb_dbi_open(). > (Unless they're all write txns, which LMDB serializes.) One reason > you must, is that there is no sync anywhere else in your suggestion. > > Multi-threaded use of structures, like the ones behind DBIs, must be > synchronized. But DBIs and mdb_dbi_open() are not mutex-protected, > nor are read-only txns (after the per-thread setup). So it's up to > the user. And LMDB mostly can't catch users at less than txn-sized > threading bugs - like the DBI workarounds people keep trying. > > I suppose we should add some of the above more clearly in lmdb.h doc, > since people keep trying to hack around it. And "EINVAL/MDB_BAD_DBI > often indicate that LMDB detected a bug in your program, not that you > can safely work around the problem when LMDB returns them." > (Not always, unless we walk carefully through which error codes are > returned when first.) > > > For a txn to pick up a DBI committed after the txn started, LMDB > would need to mutex-protect DBI operations. Something like this: > > - Add an env+txn flag MDB_ADMIN_LOCKS. If it is set in the env, > then mdb_dbi_open() fails if it is not also set in the txn. > The txn flag wraps DBI handling in a new mutex in txn_begin(), > dbi_open(), commit/abort() of a txn which opened a DBI. > The env flag wraps dbi_close() in the mutex. > > - Add mdb_dbi_pickup(MDB_txn*, MDB_dbi of a named DB): Pick up a DBI > which was committed after the txn began. It will lock the mutex, > and maybe it'll require MDB_ADMIN_LOCKS in the txn. > (I don't know if pickup(main DBI) would be safe.) > > Maybe that's more ACID-friendly - a separate function to step > outside and look at the current env. If I've gotten it right... > > Of course, if the txn which calls dbi_pickup() began before that > named DB was created, then the txn picks up a DBI which it cannot > use. That's OK as far as LMDB is concerned, it does safely catch > use of a non-existent named DB. But makes it less useful to users. > > So I wonder, will people optimize away the mutexing in the way you > suggested above? Try to use a DBI in a txn without MDB_ADMIN_LOCKS, > and if that fails call mdb_dbi_pickup(). If so, they reintroduce > the heisenbug: The test for whether the DBI was OK, was not mutexed > vs. mdb_txn_commit(txn which created the DBI). > > Or simpler, the new flag could just tell MDB to wrap a mutex around > every single txn_begin and txn_commit. That won't tempt anyone to > optimize away locks, since there are so much locking already. But > then, why are they using LMDB? LMDB has made a lot of tradeoffs to > avoid locking in read-only txns. See Caveats in lmdb.h. > > > And it'd still be unsafe to allow overlapping txns to mdb_dbi_open() > the same DB. txn1 creates the DBI, txn2 opens it too, txn1 aborts but > must not close the DBI because txn2 is using it... So MDB_dbx would > need a refcount of how many existing txns opened the DB during DBI > creation. Also I suppose ADMIN_LOCKS would need to disable > mdb_set_compare() & co. A new function mdb_dbi_open2() could init the > entire MDB_dbx while holding the mutex, so other txns would not use > the half-initialized DBX. > > Alternatively, dbi_open2(named DB) above can take a flag which says > "make the DBI immediately available in the MDB_env". That'll be > unsafe for txns where the user does not synchronize txn_begin against > the dbi_open2(). And it has its own sets of issues with multiple > txns opening a DB, nested txns, etc. > > > Then there's another approach: "Transactions are isolated from each > other, provided you use LMDB correctly. That's all. Sorry about > the DBI inconveniences." > If I got it right it is all about protecting against invalid DBIs which can happen for example if someone closes database "A" which was assigned DBI "5" and later the database "B" is opened and the free slot 5 is then assigned to "B". So we could be misled using DBI "5" having in mind database "A". So why don't we extend the MDB_DBI by adding the sequence in the lower or higher bits. When addressing the slot we simply strip the sequence bits and verify the slot's sequence against the sequence in the MDB_DBI to ensure the DBI has not changed. So picking up a DPI (as suggested in the patch) can be verified. The restriction with mdb_dbi_open() will stay. But the following would work: MDB_dbi dbi; mdb_txn_begin(env, NULL, MDB_RDONLY, &txn2); mdb_txn_begin(env, NULL, MDB_RDONLY, &txn1); mdb_dbi_open(txn1, "db1", 0, &dbi); // dbi becomes now valid and is propagated to the env // ==> and therefore is also valid for any other transaction! mdb_txn_commit(txn1); // use dbi opened in txn1 (which is committed) mdb_something(txn2, dbi, ...); .... mdb_txn_commit(txn2); So the patch could look something like that. static int mdb_update_db_info(MDB_txn *txn, MDB_dbi dbi) { unsigned int dbi_index = dbi & MDB_DBI_INDXE_MASK; unsigned int dbi_seq = dbi & MDB_DBI_SEQ_MASK << 16; if (txn->mt_env->me_dbiseqs[dbi_index] != dbi_seq) { return 0; } /* propagate newly opened db from env to current txn. Mark them as unused */ if (txn->mt_numdbs < txn->mt_env->me_numdbs) { memset(txn->mt_dbflags + txn->mt_numdbs, DB_UNUSED, txn->mt_env->me_numdbs - txn->mt_numdbs); } txn->mt_numdbs = txn->mt_env->me_numdbs; return dbi_index < txn->mt_numdbs; }

1 0

Re: (ITS#8430) Improved handling for large number of databases
by h.b.furuseth＠usit.uio.no 30 May '16

30 May '16

On 30. mai 2016 13:51, Hallvard Breien Furuseth wrote: > Copied from a private mail: >> Maybe a good way in between could be to memcpy txn->mt_env->me_dbflags. >> (...) memcpy uses SSE instructions for x86_64. > > Yes. If we make MDB_txn.mt_dbflags an uint16_t*, we can memcpy to it. > That costs <maxdbs> bytes/txn + one more reserved env flag bit in > PERSISTENT_FLAGS: Reserve DB_UNUSED, and keep it always set in > me_dbflags so the memcpy will set it in mt_dbflags. Oops, that was too clever. Then a bunch of code which assumes a DBI is valid if various txn flags are set, must also check DB_UNUSED. Or invert that to DB_USED which might be easier - (flags & DB_DIRTY) becomes F_ISSET(flags, DB_DIRTY|DB_USED) which checks if both are set. Maybe the simple solution using 2*<maxdbs> bytes/txn would be better. -- Hallvard

1 0

Re: (ITS#8431) LMDB: Access newly opened database from another transaction
by h.b.furuseth＠usit.uio.no 30 May '16

30 May '16

In ITS#8430, Juerg Bircher wrote: > So if you drop ITS#8431 it would at least be very helpful to get an > separate error like MDB_INVALID_DPI instead of EINVAL So the > application sharing and caching handles (dbi) could handle the > situation appropriate like use mdb_env_open() so that the dbi becomes > valid in that txn as well. This is ITS#8431 in a new guise, so I'm replying here. We know DBI handling is inconvenient. We'd have fixed it if it were easy without adding a bunch of locks. I'll walk through some of it below, I expect it'll be useful to have for future reference anyway: You mean like this? if (mdb_something(txn, dbi, ...) == MDB_INVALID_DBI) { /* Get <dbi>, which some other txn committed after <txn> began */ mdb_dbi_open(txn, "database name of <dbi>", 0, &dbi2); assert(dbi2 == dbi); } That breaks this requirement from the mdb_dbi_open() doc: * A transaction that uses * this function must finish (either commit or abort) before * any other transaction in the process may use this function. To fulfill that, you must serialize txns which use mdb_dbi_open(). (Unless they're all write txns, which LMDB serializes.) One reason you must, is that there is no sync anywhere else in your suggestion. Multi-threaded use of structures, like the ones behind DBIs, must be synchronized. But DBIs and mdb_dbi_open() are not mutex-protected, nor are read-only txns (after the per-thread setup). So it's up to the user. And LMDB mostly can't catch users at less than txn-sized threading bugs - like the DBI workarounds people keep trying. I suppose we should add some of the above more clearly in lmdb.h doc, since people keep trying to hack around it. And "EINVAL/MDB_BAD_DBI often indicate that LMDB detected a bug in your program, not that you can safely work around the problem when LMDB returns them." (Not always, unless we walk carefully through which error codes are returned when first.) For a txn to pick up a DBI committed after the txn started, LMDB would need to mutex-protect DBI operations. Something like this: - Add an env+txn flag MDB_ADMIN_LOCKS. If it is set in the env, then mdb_dbi_open() fails if it is not also set in the txn. The txn flag wraps DBI handling in a new mutex in txn_begin(), dbi_open(), commit/abort() of a txn which opened a DBI. The env flag wraps dbi_close() in the mutex. - Add mdb_dbi_pickup(MDB_txn*, MDB_dbi of a named DB): Pick up a DBI which was committed after the txn began. It will lock the mutex, and maybe it'll require MDB_ADMIN_LOCKS in the txn. (I don't know if pickup(main DBI) would be safe.) Maybe that's more ACID-friendly - a separate function to step outside and look at the current env. If I've gotten it right... Of course, if the txn which calls dbi_pickup() began before that named DB was created, then the txn picks up a DBI which it cannot use. That's OK as far as LMDB is concerned, it does safely catch use of a non-existent named DB. But makes it less useful to users. So I wonder, will people optimize away the mutexing in the way you suggested above? Try to use a DBI in a txn without MDB_ADMIN_LOCKS, and if that fails call mdb_dbi_pickup(). If so, they reintroduce the heisenbug: The test for whether the DBI was OK, was not mutexed vs. mdb_txn_commit(txn which created the DBI). Or simpler, the new flag could just tell MDB to wrap a mutex around every single txn_begin and txn_commit. That won't tempt anyone to optimize away locks, since there are so much locking already. But then, why are they using LMDB? LMDB has made a lot of tradeoffs to avoid locking in read-only txns. See Caveats in lmdb.h. And it'd still be unsafe to allow overlapping txns to mdb_dbi_open() the same DB. txn1 creates the DBI, txn2 opens it too, txn1 aborts but must not close the DBI because txn2 is using it... So MDB_dbx would need a refcount of how many existing txns opened the DB during DBI creation. Also I suppose ADMIN_LOCKS would need to disable mdb_set_compare() & co. A new function mdb_dbi_open2() could init the entire MDB_dbx while holding the mutex, so other txns would not use the half-initialized DBX. Alternatively, dbi_open2(named DB) above can take a flag which says "make the DBI immediately available in the MDB_env". That'll be unsafe for txns where the user does not synchronize txn_begin against the dbi_open2(). And it has its own sets of issues with multiple txns opening a DB, nested txns, etc. Then there's another approach: "Transactions are isolated from each other, provided you use LMDB correctly. That's all. Sorry about the DBI inconveniences."

1 0

Re: (ITS#8430) Improved handling for large number of databases
by h.b.furuseth＠usit.uio.no 30 May '16

30 May '16

On 28/05/16 22:08, Juerg Bircher wrote: > I understand that it is a tricky thing. But maybe the lazy init is > still feasible? (...) Right, lazy init is feasible. Just keep cross-txn DBI stuff out of it, I'll take your latest variant to ITS#8431. Copied from a private mail: > Maybe a good way in between could be to memcpy txn->mt_env->me_dbflags. > (...) memcpy uses SSE instructions for x86_64. Yes. If we make MDB_txn.mt_dbflags an uint16_t*, we can memcpy to it. That costs <maxdbs> bytes/txn + one more reserved env flag bit in PERSISTENT_FLAGS: Reserve DB_UNUSED, and keep it always set in me_dbflags so the memcpy will set it in mt_dbflags. Or maybe your code can be used without initial memcpy and I'm just paranoid. The one problem I can think of is user errors - people coming up with workarounds like your cross-txn DBI stuff. LMDB tries to catch some of that, but maybe late pickup defeats some such checks. I'll wait for Howard to say something at this point. BTW, all this reminds me - I should add a test which tries to catch mdb_dbi_open() in overlapping txns. Got the code, but forgot about it while pondering a good error code name. > However md_flags would need to be moved out of the MDB_dbi struct Nope. Sub-DBs with duplicate data have md_flags which do not correspond to anything saved in env->me_flags. Your lazy-init code needs to use the memcpyed flags instead of me_dbflags, that's all. > We should consider to align on 16 byte boundary for best performance. It's already aligned.

1 0

Re: (ITS#8431) LMDB: Access newly opened database from another transaction
by juerg.bircher＠gmail.com 28 May '16

28 May '16

On Sat, May 28, 2016 at 11:18 AM, Howard Chu <hyc(a)symas.com> wrote: > juerg.bircher(a)gmail.com wrote: >> >> On 27/05/16 15:11, "Hallvard Breien Furuseth" <h.b.furuseth(a)usit.uio.no> >> wrote: >> >>> This works as intended. Each transaction is atomic, and shall see >>> neither DBIs nor data which were committed after the transaction began. >>> Cross-txn mdb_dbi_open() is in any case hairy to get right, we screwed >>> it up several times before arriving at the now-obvious semantics. >>> >> >> It is fine that newly opened database handles shall not be seen by >> other transactions as long the transaction runs.. But once the >> transaction which opened the handle is committed it should be valid >> for any other transaction. > > > It should be valid for any other transaction *that started after the opening > transaction committed.* This is the nature of transactions, they are fully > isolated from each other. > >> This would greatly simplify lazy database >> opening of a multithreaded application. > > > Allowing existing transactions to use the DBIs would break transaction > isolation. That is a non-starter. > >> The suggested patch does exactly implement that behaviour. I believe >> it does not break anything. Isolation must be guaranteed for the data >> but not necessarily for the database handles. > > > Wrong. A database handle essentially points to data. Exposing that data to a > transaction that was opened before that handle's transaction was committed > *does* break isolation. Violating the ACID guarantees of LMDB's transaction > system is Not allowed. >> >> >>> An application can open and cache its DBIs in a separate startup >>> transaction, then commit it and start new transactions. >> >> >> Yes this would be a simple approach to get around that problem. But I >> do not like it as with each opened database the overhead increases >> especially if not using my suggested improvements in ITS#8430. > > > You are saying you don't like ACID transactions. If that's the case, LMDB is > not for you. Not at all ACID is a must and something I very much appreciate about LMDB! LMDB is definitely for me and I'm saying that after having spent nearly one intensive year with LMDB! So I like to thank you for the outstanding database. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8430) Improved handling for large number of databases
by juerg.bircher＠gmail.com 28 May '16

28 May '16

---------- Forwarded message ---------- From: Juerg Bircher <juerg.bircher(a)gmail.com> Date: Sat, May 28, 2016 at 10:30 PM Subject: Re: (ITS#8430) Improved handling for large number of databases To: Hallvard Breien Furuseth <h.b.furuseth(a)usit.uio.no> On Sat, May 28, 2016 at 3:41 AM, Hallvard Breien Furuseth <h.b.furuseth(a)usit.uio.no> wrote: > On 28/05/16 03:28, Hallvard Breien Furuseth wrote: >> >> On 27/05/16 23:39, juerg.bircher(a)gmail.com wrote: >>> >>> Maybe I am overseeing something as I do not know all the thoughts >>> between the lines. >> >> >> With ITS#8431 I can think of one thing: >> (...blah...) > > > Actually the main reason I worry about lazy init is simply that > we've screwed up DBIs many times, so I prefer to go slowly. > That's Howard's call though, not mine. Maybe a good way in between could be to memcpy txn->mt_env->me_dbflags. However md_flags would need to be moved out of the MDB_dbi struct like mt_dbflags or easier just add another array serving as snapshot. This would not break anything with flag initialization as we get a snapshot of them in the mdb_txn_renew0. memcpy uses SSE instructions for x86_64. We should consider to align on 16 byte boundary for best performance. The dbflags could then still be initialized with MDB_UNSUSED as suggested.

1 0

Re: (ITS#8430) Improved handling for large number of databases
by juerg.bircher＠gmail.com 28 May '16

28 May '16

On Sat, May 28, 2016 at 3:28 AM, Hallvard Breien Furuseth <h.b.furuseth(a)usit.uio.no> wrote: > On 27/05/16 23:39, juerg.bircher(a)gmail.com wrote: >> >> On 27/05/16 14:42, "Hallvard Breien Furuseth" <h.b.furuseth(a)usit.uio.no> >> wrote: >>> >>> I think MDB_WORD intends to be mdb_size_t, but should be 'unsigned >>> long': It's probably no optimization to use 'long long' when that >>> is not a native integer type. But I haven't profiled that. >> >> >> The MDB_WORD is used to get the appropriate word size of the CPU for >> the optimised dbflag check. So on a 32 bit CPUs it should be an 32 bit >> integer and on a 64 bit CPUs a 64 bit integer. > > > That's why I suggested unsigned long. It seems the best guess for a > native wide type. But it's hard to guess what kind of CPU is in use. > >> But the mdb_size_t seems always to be 64 bits? > > > On a 32-bit host it's normally 32-bit, but you get 64-bit mdb_size_t anyway > if you build and use LMDB with MDB_VL32 defined. I misread > your patch, it should not use MDB_VL32. > >>> The delay in picking up env->me_dbflags is a bit troubling, since it >>> means creating a new txn is no longer quite atomic. I can't think of >>> how it'll make a difference when DBIs are used correctly, but that's one >>> thing people screw up. Maybe it'd be OK if renew0() just copies it into >>> mt_dbs[].md_flags without the '&', and mdb_setup_db_info() fixes it up. >> >> >> Is it really atomic in original lmdb? The flags are copied from the >> environment and calculated. But while copying the dbflags from the >> environment they may change. So read level consistency and therefore >> atomicity is not guaranteed. > > > I should have said, it's atomic when DBIs are used correctly. Such as > obeying this limitation from the mdb_dbi_open() doc and a similar one > for mdb_dbi_close(): > > * A transaction that uses > * this function must finish (either commit or abort) before > * any other transaction in the process may use this function. > > To guarante atomicity we'd need locks around some DBI operations. > To avoid that, LMDB simply calls unsafe use "user errors" - which can > cause breakage. Hence those annoying DBI usage restrictions. > > LMDB does attempt to catch user errors when it's cheap to try, see e.g. > TXN_DBI_CHANGED(). Doesn't work at all for read-only txns, they have > txn->mt_dbiseqs == env->me_dbiseqs. > > So you can see why DBI cleverness makes me nervous. > I think the MDB_WORD looping can go in now, but we'll need to think > carefully about the lazy init. I understand that it is a tricky thing. But maybe the lazy init is still feasible? In my understand without having all the background it seems ok to get the current snapshot of the database flags from the env once it is firstly accessed whether it is at the beginning or later in the transaction. Read level consistency should be preserved anyway as mt_txnId is set in mdb_txn_renew0(). > >> But I guess it just does not matter. >> So picking up the flags lazily just extends the time of copying the >> flags from the environment but is basically the same. >> Atomicity and also isolation is a must in respect to the data but I >> think not in respect to the database handles and their flags. >> Maybe I am overseeing something as I do not know all the thoughts >> between the lines. > > > With ITS#8431 I can think of one thing: > > There's no sync between mdb_dbi_open() in one txn and another txn > using the new DBI. It'll be up to the user to synchronize DBIs. For > that to work in write-txs, TXN_DBI_CHANGED() must go away, and writers > will have no way to try to catch user errors with sync. Or if it > should only work in read-only txns, then read-only txns and write-txns > must have different definitions of what is a "user error" with DBIs. > > Anyway, we'll need to think about the definition of user errors. > So if you drop ITS#8431 it would at least be very helpful to get an separate error like MDB_INVALID_DPI instead of EINVAL So the application sharing and caching handles (dbi) could handle the situation appropriate like use mdb_env_open() so that the dbi becomes valid in that txn as well. > > Other notes: > > /** ... */ above a global declaration or /**< ... */ to the right > of it tells Doxygen that the comment is the doc of the declared item. > "#foo" inside a Doxygen comment is a link to the declaration of foo. > > > Maybe mt_dbflags should have an extra byte or word at the end which > does not have DB_UNUSED, and with size rounded up to sizeof(MDB_WORD). > Then stepping to next DBI can do something like this, unrelated to > the check against mt_numdbs. Not sure if that's an improvement, I'm > getting tired: > > /** Return an #MDB_WORD where each byte is \b byte */ > #define WORD_BYTES(byte) ((MDB_WORD)-1 / 0xff * (byte)) > > /** Step up to next used DBI */ > #define DBI_STEP(txn, dbi) do { \ > if (! (++(dbi) & (sizeof(mdb_word_t)-1))) \ > while (*(mdb_word_t *)((txn)->mt_dbflags + (dbi)) == > WORD_BYTES(DB_UNUSED)) > (dbi) += sizeof(MDB_WORD); \ > } while (0) > Yes round it up to sizeof(MDB_WORD) would be obvious in connection with the optimized loop. If you are willing to add some more #ifdef we could even get it faster at least for x86_64 using SSE2 instructions. Then MDB_WORD would be 128 bit and we check 16 flags in one step using _mm_cmpeq_epi8 and _mm_movemask_epi8 to get the mask of matching flags. If the mask != 0xffffffff we found an used database. Then we negate the mask and with __builtin_ffs(mask) we could then find the index of the first used database.

1 0

(ITS#8102) syncrepl_entry: be_modify failed (16)
by tpretz＠gmail.com 28 May '16

28 May '16

Hi, Following up on this ITS i opened a while back. With multi-master, normal syncrepl, i would sometimes receive: slapd: null_callback : error code 0x10 slapd: syncrepl_entry: rid=106 be_modify failed (16) Triggering a syncrepl connection drop/retry, whilst playing the sessionlog when a server with multiple providers was started. I am now testing with 2.4.44 and have had a chance to look at this annoying, but seemingly not destructive issue in some more detail. As i partially referenced previously, this occurs within syncrepl_entry, for modifications, a diff of old_entry to new_entry is performed. Then if changes are needed a be_modify is performed. There is however, no locking which prevents two, or more, threads from performing these diffs, and then mods, in an interleaved fashion within this function itself. Looking in do_syncrep2, if the cookie tag is present the cs_pmutex is acquired and held for the duration of modifications. This mutex protects from syncrepl_entry race conditions and serializes modifications. I have also noticed this issue during normal operations (ie all syncrepl in persist) when out of order writes are occurring on a master which are relatively easy to reproduce on an hdb backend server. When a cookie is not sent with an entry the cs_pmutex is not acquired. Without having some protection, non-cookie modifications will race each other between syncrepl threads. So, i am testing surrounding the syncrepl_entry "if" block (line 1036) with a cs_pmutex lock/release (when punlock < 0) to serialize non_cookie mods just like the cookie ones. So far this is running tests and i haven't seen the null_callback issue, either when catching up from the session log, or running with ongoing out of order writes being replicated (running alongside unmodified 2.4.44 to compare differences). When acquiring the cs_pmutex i have used the same logic as at line 958 (using trylock, with a shutdown check). I wonder if it is safe to acquire the mutex with a standard ldap_pvt_thread_mutex_lock at this point without spinning. line numbers from RELENG_2_4 (721a038b7bc9732f52eeef5324c180c4f137cd75) Thanks Tom

1 0

Re: (ITS#8431) LMDB: Access newly opened database from another transaction
by hyc＠symas.com 28 May '16

28 May '16

juerg.bircher(a)gmail.com wrote: > On 27/05/16 15:11, "Hallvard Breien Furuseth" <h.b.furuseth(a)usit.uio.no> wrote: > >> This works as intended. Each transaction is atomic, and shall see >> neither DBIs nor data which were committed after the transaction began. >> Cross-txn mdb_dbi_open() is in any case hairy to get right, we screwed >> it up several times before arriving at the now-obvious semantics. >> > > It is fine that newly opened database handles shall not be seen by > other transactions as long the transaction runs.. But once the > transaction which opened the handle is committed it should be valid > for any other transaction. It should be valid for any other transaction *that started after the opening transaction committed.* This is the nature of transactions, they are fully isolated from each other. > This would greatly simplify lazy database > opening of a multithreaded application. Allowing existing transactions to use the DBIs would break transaction isolation. That is a non-starter. > The suggested patch does exactly implement that behaviour. I believe > it does not break anything. Isolation must be guaranteed for the data > but not necessarily for the database handles. Wrong. A database handle essentially points to data. Exposing that data to a transaction that was opened before that handle's transaction was committed *does* break isolation. Violating the ACID guarantees of LMDB's transaction system is Not allowed. > >> An application can open and cache its DBIs in a separate startup >> transaction, then commit it and start new transactions. > > Yes this would be a simple approach to get around that problem. But I > do not like it as with each opened database the overhead increases > especially if not using my suggested improvements in ITS#8430. You are saying you don't like ACID transactions. If that's the case, LMDB is not for you. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8430) Improved handling for large number of databases
by h.b.furuseth＠usit.uio.no 28 May '16

28 May '16

On 28/05/16 03:28, Hallvard Breien Furuseth wrote: >On 27/05/16 23:39, juerg.bircher(a)gmail.com wrote: >> Maybe I am overseeing something as I do not know all the thoughts >> between the lines. > > With ITS#8431 I can think of one thing: > (...blah...) Actually the main reason I worry about lazy init is simply that we've screwed up DBIs many times, so I prefer to go slowly. That's Howard's call though, not mine.

1 0

Re: (ITS#8430) Improved handling for large number of databases
by h.b.furuseth＠usit.uio.no 28 May '16

28 May '16

On 27/05/16 23:39, juerg.bircher(a)gmail.com wrote: > On 27/05/16 14:42, "Hallvard Breien Furuseth" <h.b.furuseth(a)usit.uio.no> wrote: >> I think MDB_WORD intends to be mdb_size_t, but should be 'unsigned >> long': It's probably no optimization to use 'long long' when that >> is not a native integer type. But I haven't profiled that. > > The MDB_WORD is used to get the appropriate word size of the CPU for > the optimised dbflag check. So on a 32 bit CPUs it should be an 32 bit > integer and on a 64 bit CPUs a 64 bit integer. That's why I suggested unsigned long. It seems the best guess for a native wide type. But it's hard to guess what kind of CPU is in use. > But the mdb_size_t seems always to be 64 bits? On a 32-bit host it's normally 32-bit, but you get 64-bit mdb_size_t anyway if you build and use LMDB with MDB_VL32 defined. I misread your patch, it should not use MDB_VL32. >> The delay in picking up env->me_dbflags is a bit troubling, since it >> means creating a new txn is no longer quite atomic. I can't think of >> how it'll make a difference when DBIs are used correctly, but that's one >> thing people screw up. Maybe it'd be OK if renew0() just copies it into >> mt_dbs[].md_flags without the '&', and mdb_setup_db_info() fixes it up. > > Is it really atomic in original lmdb? The flags are copied from the > environment and calculated. But while copying the dbflags from the > environment they may change. So read level consistency and therefore > atomicity is not guaranteed. I should have said, it's atomic when DBIs are used correctly. Such as obeying this limitation from the mdb_dbi_open() doc and a similar one for mdb_dbi_close(): * A transaction that uses * this function must finish (either commit or abort) before * any other transaction in the process may use this function. To guarante atomicity we'd need locks around some DBI operations. To avoid that, LMDB simply calls unsafe use "user errors" - which can cause breakage. Hence those annoying DBI usage restrictions. LMDB does attempt to catch user errors when it's cheap to try, see e.g. TXN_DBI_CHANGED(). Doesn't work at all for read-only txns, they have txn->mt_dbiseqs == env->me_dbiseqs. So you can see why DBI cleverness makes me nervous. I think the MDB_WORD looping can go in now, but we'll need to think carefully about the lazy init. > But I guess it just does not matter. > So picking up the flags lazily just extends the time of copying the > flags from the environment but is basically the same. > Atomicity and also isolation is a must in respect to the data but I > think not in respect to the database handles and their flags. > Maybe I am overseeing something as I do not know all the thoughts > between the lines. With ITS#8431 I can think of one thing: There's no sync between mdb_dbi_open() in one txn and another txn using the new DBI. It'll be up to the user to synchronize DBIs. For that to work in write-txs, TXN_DBI_CHANGED() must go away, and writers will have no way to try to catch user errors with sync. Or if it should only work in read-only txns, then read-only txns and write-txns must have different definitions of what is a "user error" with DBIs. Anyway, we'll need to think about the definition of user errors. Other notes: /** ... */ above a global declaration or /**< ... */ to the right of it tells Doxygen that the comment is the doc of the declared item. "#foo" inside a Doxygen comment is a link to the declaration of foo. Maybe mt_dbflags should have an extra byte or word at the end which does not have DB_UNUSED, and with size rounded up to sizeof(MDB_WORD). Then stepping to next DBI can do something like this, unrelated to the check against mt_numdbs. Not sure if that's an improvement, I'm getting tired: /** Return an #MDB_WORD where each byte is \b byte */ #define WORD_BYTES(byte) ((MDB_WORD)-1 / 0xff * (byte)) /** Step up to next used DBI */ #define DBI_STEP(txn, dbi) do { \ if (! (++(dbi) & (sizeof(mdb_word_t)-1))) \ while (*(mdb_word_t *)((txn)->mt_dbflags + (dbi)) == WORD_BYTES(DB_UNUSED)) (dbi) += sizeof(MDB_WORD); \ } while (0)

1 0

Re: (ITS#8431) LMDB: Access newly opened database from another transaction
by juerg.bircher＠gmail.com 28 May '16

28 May '16

On 27/05/16 15:11, "Hallvard Breien Furuseth" <h.b.furuseth(a)usit.uio.no> wrote: >This works as intended. Each transaction is atomic, and shall see >neither DBIs nor data which were committed after the transaction began. >Cross-txn mdb_dbi_open() is in any case hairy to get right, we screwed >it up several times before arriving at the now-obvious semantics. > It is fine that newly opened database handles shall not be seen by other transactions as long the transaction runs.. But once the transaction which opened the handle is committed it should be valid for any other transaction. This would greatly simplify lazy database opening of a multithreaded application. The suggested patch does exactly implement that behaviour. I believe it does not break anything. Isolation must be guaranteed for the data but not necessarily for the database handles. >An application can open and cache its DBIs in a separate startup >transaction, then commit it and start new transactions. Yes this would be a simple approach to get around that problem. But I do not like it as with each opened database the overhead increases especially if not using my suggested improvements in ITS#8430. See my measurements with the improved lmdb and the original lmdb. The test always performs 1000000 iterations on one database while having 1 database open in the first run and having all databases open in the second run. lmdb improved (renew, reset) [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 10] open databases in [0.31975] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [10 of 10] open databases in [0.20350] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 100] open databases in [0.25845] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [100 of 100] open databases in [0.29663] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 1000] open databases in [0.28590] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1000 of 1000] open databases in [0.42897] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 10000] open databases in [0.30004] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [10000 of 10000] open databases in [1.68870] seconds lmdb improved (begin, commit) [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 10] open databases in [0.36538] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [10 of 10] open databases in [0.35923] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 100] open databases in [0.34858] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [100 of 100] open databases in [0.39294] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 1000] open databases in [0.40222] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1000 of 1000] open databases in [0.54752] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 10000] open databases in [0.78595] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [10000 of 10000] open databases in [2.32414] seconds lmdb original (renew, reset) [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 10] open databases in [0.18597] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [10 of 10] open databases in [0.21572] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 100] open databases in [0.24173] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [100 of 100] open databases in [0.46497] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 1000] open databases in [0.27127] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1000 of 1000] open databases in [4.18579] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 10000] open databases in [0.30128] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [10000 of 10000] open databases in [49.35048] seconds lmdb original (begin, commit) [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 10] open databases in [0.45135] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [10 of 10] open databases in [0.38990] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 100] open databases in [0.47890] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [100 of 100] open databases in [0.69917] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 1000] open databases in [1.84908] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1000 of 1000] open databases in [5.88098] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 10000] open databases in [22.12491] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [10000 of 10000] open databases in [74.53854] seconds > >-- > Hallvard Regards Juerg

1 0

Re: (ITS#8430) Improved handling for large number of databases
by juerg.bircher＠gmail.com 27 May '16

27 May '16

First of all thank you for having a look at the improvement. On 27/05/16 14:42, "Hallvard Breien Furuseth" <h.b.furuseth(a)usit.uio.no> wrote: >On 27. mai 2016 12:57, juerg.bircher(a)helmedica.com wrote: >> mdb_dbi_open() is however not improved with the assumption that the database >> handle (dbi) is cached in the application. So mdb_dbi_open() should happen only >> once for each database during the life time of an application. > >It must check if the DB is open anyway, since LDMB breaks if the user >has two DBIs to the same DB. Though maybe we could add an optimization >flag to skip the search, for users who know they don't reopen open DBs. > >Other notes: > >LMDB and OpenLDAP mostly use tab-width = indentation = 4. (We should >comment that somewhere.) You seem to use tab-width = 8. I will consider that for the next patch! > >I think MDB_WORD intends to be mdb_size_t, but should be 'unsigned >long': It's probably no optimization to use 'long long' when that >is not a native integer type. But I haven't profiled that. The MDB_WORD is used to get the appropriate word size of the CPU for the optimised dbflag check. So on a 32 bit CPUs it should be an 32 bit integer and on a 64 bit CPUs a 64 bit integer. But the mdb_size_t seems always to be 64 bits? > >The delay in picking up env->me_dbflags is a bit troubling, since it >means creating a new txn is no longer quite atomic. I can't think of >how it'll make a difference when DBIs are used correctly, but that's one >thing people screw up. Maybe it'd be OK if renew0() just copies it into >mt_dbs[].md_flags without the '&', and mdb_setup_db_info() fixes it up. Is it really atomic in original lmdb? The flags are copied from the environment and calculated. But while copying the dbflags from the environment they may change. So read level consistency and therefore atomicity is not guaranteed. But I guess it just does not matter. So picking up the flags lazily just extends the time of copying the flags from the environment but is basically the same. Atomicity and also isolation is a must in respect to the data but I think not in respect to the database handles and their flags. Maybe I am overseeing something as I do not know all the thoughts between the lines. > >-- >Hallvard Regards Juerg

1 0

Re: (ITS#8430) Improved handling for large number of databases
by juerg.bircher＠gmail.com 27 May '16

27 May '16

I am switching to gmail as in outlook my name uses Umlaute which results in base64 encoded mails! Strangely the mailing list openldap-technical can handle it and openldap-its cannot! Here the missing disclaimer for the patch: The attached patch file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Juerg Bircher juerg.bircher(a)helmedica.com. I have not assigned rights and/or interest in this work to any party. I, Juerg Bircher, hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice.

1 0

Re: (ITS#8431) LMDB: Access newly opened database from another transaction
by h.b.furuseth＠usit.uio.no 27 May '16

27 May '16

This works as intended. Each transaction is atomic, and shall see neither DBIs nor data which were committed after the transaction began. Cross-txn mdb_dbi_open() is in any case hairy to get right, we screwed it up several times before arriving at the now-obvious semantics. An application can open and cache its DBIs in a separate startup transaction, then commit it and start new transactions. -- Hallvard

1 0

Re: (ITS#8430) Improved handling for large number of databases
by juerg.bircher＠helmedica.com 27 May '16

27 May '16

SGVyZSB0aGUgbWlzc2luZyBkaXNjbGFpbWVyIGZvciB0aGUgcGF0Y2g6DQoNClRoZSBhdHRhY2hl ZCBwYXRjaCBmaWxlIGlzIGRlcml2ZWQgZnJvbSBPcGVuTERBUCBTb2Z0d2FyZS4gQWxsIG9mIHRo ZSBtb2RpZmljYXRpb25zIHRvIE9wZW5MREFQIFNvZnR3YXJlIHJlcHJlc2VudGVkIGluIHRoZSBm b2xsb3dpbmcgcGF0Y2goZXMpIHdlcmUgZGV2ZWxvcGVkIGJ5IEp1ZXJnIEJpcmNoZXIganVlcmcu YmlyY2hlckBoZWxtZWRpY2EuY29tLiBJIGhhdmUgbm90IGFzc2lnbmVkIHJpZ2h0cyBhbmQvb3Ig aW50ZXJlc3QgaW4gdGhpcyB3b3JrIHRvIGFueSBwYXJ0eS4NCg0KSSwgSnVlcmcgQmlyY2hlciwg aGVyZWJ5IHBsYWNlIHRoZSBmb2xsb3dpbmcgbW9kaWZpY2F0aW9ucyB0byBPcGVuTERBUCBTb2Z0 d2FyZSAoYW5kIG9ubHkgdGhlc2UgbW9kaWZpY2F0aW9ucykgaW50byB0aGUgcHVibGljIGRvbWFp bi4gSGVuY2UsIHRoZXNlIG1vZGlmaWNhdGlvbnMgbWF5IGJlIGZyZWVseSB1c2VkIGFuZC9vciBy ZWRpc3RyaWJ1dGVkIGZvciBhbnkgcHVycG9zZSB3aXRoIG9yIHdpdGhvdXQgYXR0cmlidXRpb24g YW5kL29yIG90aGVyIG5vdGljZS4NCg0K

1 0

Re: (ITS#8430) Improved handling for large number of databases
by juerg.bircher＠helmedica.com 27 May '16

27 May '16

DQpEaXNjbGFpbWVyDQoNCg0KDQpUaGUgYXR0YWNoZWQgcGF0Y2ggZmlsZSBpcyBkZXJpdmVkIGZy b20gT3BlbkxEQVAgU29mdHdhcmUuIEFsbCBvZiB0aGUgbW9kaWZpY2F0aW9ucyB0byBPcGVuTERB UCBTb2Z0d2FyZSByZXByZXNlbnRlZCBpbiB0aGUgZm9sbG93aW5nIHBhdGNoKGVzKSB3ZXJlIGRl dmVsb3BlZCBieSBKdWVyZyBCaXJjaGVyIGp1ZXJnLmJpcmNoZXJAaGVsbWVkaWNhLmNvbS4gSSBo YXZlIG5vdCBhc3NpZ25lZCByaWdodHMgYW5kL29yIGludGVyZXN0IGluIHRoaXMgd29yayB0byBh bnkgcGFydHkuDQoNCkksIEp1ZXJnIEJpcmNoZXIsIGhlcmVieSBwbGFjZSB0aGUgZm9sbG93aW5n IG1vZGlmaWNhdGlvbnMgdG8gT3BlbkxEQVAgU29mdHdhcmUgKGFuZCBvbmx5IHRoZXNlIG1vZGlm aWNhdGlvbnMpIGludG8gdGhlIHB1YmxpYyBkb21haW4uIEhlbmNlLCB0aGVzZSBtb2RpZmljYXRp b25zIG1heSBiZSBmcmVlbHkgdXNlZCBhbmQvb3IgcmVkaXN0cmlidXRlZCBmb3IgYW55IHB1cnBv c2Ugd2l0aCBvciB3aXRob3V0IGF0dHJpYnV0aW9uIGFuZC9vciBvdGhlciBub3RpY2UuDQoNCg0K DQpPbiAyNy8wNS8xNiAxMzoxNiwgIkhvd2FyZCBDaHUiIDxoeWNAc3ltYXMuY29tPiB3cm90ZToN Cg0KPmp1ZXJnLmJpcmNoZXJAaGVsbWVkaWNhLmNvbSB3cm90ZToNCj4+IEZ1bGxfTmFtZTogSnVl cmcgQmlyY2hlcg0KPj4gVmVyc2lvbjogbG1kYiAobWFzdGVyKQ0KPj4gT1M6IE1hY09TIC8gTGlu dXgNCj4+IFVSTDogZnRwOi8vZnRwLm9wZW5sZGFwLm9yZy9pbmNvbWluZy9KdWVyZ19CaXJjaGVy XzE2MDUyNy1JbXByb3ZlZC1oYW5kbGluZy1mb3ItbGFyZ2UtbnVtYmVyLW9mLWRhdGFiYXNlcy5w YXRjaA0KPj4gU3VibWlzc2lvbiBmcm9tOiAoTlVMTCkgKDE3OC44Mi4zNy4xOTUpDQo+DQo+VGhh bmtzIGZvciB0aGUgcGF0Y2gsIGJ1dCBmb3IgYSBjaGFuZ2Ugb2YgdGhpcyBzaXplIHlvdSBhbHNv IG5lZWQgdG8gaW5jbHVkZSANCj5hbiBJUCBSaWdodHMgbm90aWNlLCBhcyBkb2N1bWVudGVkIG9u IHRoZSBDb250cmlidXRpbmcgZ3VpZGVsaW5lcy4NCj4+DQo+Pg0KPj4gVGhlcmUgaXMgYSBpbmNy ZWFzZWQgcGVyZm9ybWFuY2UgcGVuYWx0eSB0aGUgbW9yZSBkYXRhYmFzZXMgYXJlIGNyZWF0ZWQg d2l0aGluDQo+PiB0aGUgc2FtZSBlbnZpcm9ubWVudC4gSSB3YXMgbG9va2luZyBmb3IgYSB3YXkg dGhlIGltcHJvdmUgdGhhdCBieSBrZWVwaW5nIHRoZQ0KPj4gc2ltcGxpY2l0eSBvZiB0cmFja2lu ZyBkYXRhYmFzZXMgd2l0aGluIGEgbGlzdCB3aXRoIGRpcmVjdCBhY2Nlc3MgYnkgaW5kZXgNCj4+ IChNREJfZGJpKS4NCj4+IG1kYl9kYmlfb3BlbigpIGlzIGhvd2V2ZXIgbm90IGltcHJvdmVkIHdp dGggdGhlIGFzc3VtcHRpb24gdGhhdCB0aGUgZGF0YWJhc2UNCj4+IGhhbmRsZSAoZGJpKSBpcyBj YWNoZWQgaW4gdGhlIGFwcGxpY2F0aW9uLiBTbyBtZGJfZGJpX29wZW4oKSBzaG91bGQgaGFwcGVu IG9ubHkNCj4+IG9uY2UgZm9yIGVhY2ggZGF0YWJhc2UgZHVyaW5nIHRoZSBsaWZlIHRpbWUgb2Yg YW4gYXBwbGljYXRpb24uDQo+Pg0KPj4NCj4+IE9uZSBpc3N1ZSBpcyB0aGF0IG1kYl90eG5fYmVn aW4oKSAoZm9yIHJlYWQtb25seSB0cmFuc2FjdGlvbnMpIGNhbGxvYyB0aGUNCj4+IHNpemVvZihN REJfdHhuKSArIG1lX21heGRicyAqIHNpemVvZihNREJfZGIgKyAxKS4gVGhlIHBsdXMgMSBmb3Ig dGhlIGRiZmxhZ3MuDQo+PiBIb3dldmVyIGl0IGlzIHN1ZmZpY2llbnQgb25seSB0byBtYWxsb2Mg dGhhdCBzaXplIGFuZCBjbGVhciB0aGUgc2l6ZW9mKE1EQl90eG4pDQo+Pg0KPj4gICAgICBtZW1z ZXQodHhuLCAwLCBzaXplb2YoTURCX3R4bikNCj4+DQo+PiBBZnRlciB0aGF0IHRoZSBkYXRhIGJl eW9uZCB0aGUgTURCX3R4biBpcyBub3QgaW5pdGlhbGl6ZWQgd2hpY2ggaXMgb2sgZm9yIHRoZQ0K Pj4gbW9tZW50Lg0KPj4NCj4+IFRoZSBuZXh0IGltcHJvdmVtZW50IGhhcHBlbnMgaW4gbWRiX3R4 bl9yZW5ldzAoKSB3aGVyZSB0aGUgZGJmbGFncyBhcmUgb25seSBzZXQNCj4+IHRvIERCX1VOVVNF RCAoYSBuZXcgZmxhZykgZm9yIGVhY2ggZGF0YWJhc2UgY3VycmVudGx5IG9wZW5lZCBpbiB0aGUN Cj4+IGVudmlyb25tZW50Lg0KPj4NCj4+ICAgICAgbWVtc2V0KHR4bi0+bXRfZGJmbGFncywgREJf VU5VU0VELCB0eG4tPm10X251bWRicyk7DQo+Pg0KPj4gVGhlIGZvcm1lciBjb2RlIHVzZWQgdG8g dG8gbG9vcCB0aHJvdWdoIGVhY2ggZGF0YWJhc2UgdG8gY2FsY3VsYXRlIHRoZSBkYmZsYWdzLg0K Pj4gVGhpcyBpcyBzdGlsbCBkb25lIGJ1dCBsYXppbHkgZm9yIGVhY2ggYWNjZXNzZWQgZGF0YWJh c2Ugd2l0aCB0aGUgYXNzdW1wdGlvbg0KPj4gdGhhdCBhIHJlYWQgb25seSB0cmFuc2FjdGlvbiBy YXJlbHkgdXNlcyBhbGwgZGF0YWJhc2VzIG9mIHRoZSBlbnZpcm9ubWVudC4NCj4+DQo+PiBUaGUg bGF6eSBpbml0aWFsaXphdGlvbiBvZiB0aGUgZGJmbGFnIGhhcHBlbnMgaW4gdGhlIG1hY3JvIFRY Tl9EQklfRVhJU1Qgd2hpY2gNCj4+IGlzIGFsd2F5cyB1c2VkIHdoZW4gYSBkYXRhYmFzZSBoYW5k bGUgKGRiaSkgaXMgcGFzc2VkIHRvIGFuIGZ1bmN0aW9uLg0KPj4gVGhlIGZsYWdzIGFyZSB1cGRh dGVkIGluIG1kYl9zZXR1cF9kYl9pbmZvKCkgb25jZSBhIGRhdGFiYXNlIGlzIGFjY2VzcyB3aGlj aCBpcw0KPj4gbWFya2VkIGFzIHVudXNlZCAoREJfVU5VU0VEKS4NCj4+DQo+PiBzdGF0aWMgaW50 IG1kYl9zZXR1cF9kYl9pbmZvKE1EQl90eG4gKnR4biwgTURCX2RiaSBkYmkpIHsNCj4+ICAgICAg LyogU2V0dXAgZGIgaW5mbyAqLw0KPj4gICAgICB1aW50MTZfdCB4ID0gdHhuLT5tdF9lbnYtPm1l X2RiZmxhZ3NbZGJpXTsNCj4+ICAgICAgdHhuLT5tdF9kYnNbZGJpXS5tZF9mbGFncyA9IHggJiBQ RVJTSVNURU5UX0ZMQUdTOw0KPj4gICAgICB0eG4tPm10X2RiZmxhZ3NbZGJpXSA9ICh4ICYgTURC X1ZBTElEKSA/IERCX1ZBTElEfERCX1VTUlZBTElEfERCX1NUQUxFIDogMDsNCj4+ICAgICAgcmV0 dXJuICh0eG4tPm10X2RiZmxhZ3NbZGJpXSAmIHZhbGlkaXR5KTsNCj4+IH0NCj4+DQo+PiAgICAg IC8qKiBDaGVjayBcYiB0eG4gYW5kIFxiIGRiaSBhcmd1bWVudHMgdG8gYSBmdW5jdGlvbiBhbmQg aW5pdGlhbGl6ZSBkYiBpbmZvDQo+PiBpZiBuZWVkZWQgKi8NCj4+ICNkZWZpbmUgVFhOX0RCSV9F WElTVCh0eG4sIGRiaSwgdmFsaWRpdHkpIFwNCj4+ICAgICAgKCh0eG4pICYmIChkYmkjQzNDKHR4 biktPm10X251bWRicyAmJiAoKCh0eG4pLT5tdF9kYmZsYWdzW2RiaV0gJiAodmFsaWRpdHkpKQ0K Pj4gfHwgKCgodHhuKS0+bXRfZGJmbGFnc1tkYmldICYgREJfVU5VU0VEKSAmJiBtZGJfc2V0dXBf ZGJfaW5mbygodHhuKSwgKGRiaSksDQo+PiAodmFsaWRpdHkpKSkpKQ0KPj4NCj4+DQo+PiBUaGUg bmV4dCBpbXByb3ZlbWVudCBpcyBkb25lIGluIGFueSBmdW5jdGlvbiB3aGljaCBuZWVkcyB0byBs b29wIHRocm91Z2ggdGhlDQo+PiBkYXRhYmFzZXMgZm9yIGV4YW1wbGUgaW4gbWRiX2N1cnNvcnNf Y2xvc2UoKS4gQWdhaW4gdGhlIG1vcmUgZGF0YWJhc2VzIGluIHRoZQ0KPj4gZW52aXJvbm1lbnQg dGhlIGxvbmdlciB0aGUgZXhlY3V0aW9uIHRpbWUuDQo+PiBJdCBzaG91bGQgYmUgYmVzdCBpZiBs b29waW5nIG9ubHkgdGhyb3VnaCBkYmZsYWdzIGFuZCBzZWFyY2hpbmcgZm9yIHRob3NlDQo+PiBk YXRhYmFzZXMgd2hpY2ggYXJlIHVzZWQgKCFEQl9VTlVTRUQpLiBUaGlzIGNvdWxkIGJlIGRvbmUg Ynl0ZSB3aXNlIG9yIG1vcmUNCj4+IGVmZmljaWVudCBpbiA4LzQgYnl0ZSBzdGVwcyBjb21wYXJp bmcgd2l0aCBhbiBleHRlbmRlZCBtYXNrIERCX1VOVVNFRF9MT05HDQo+PiBpbnN0ZWFkIG9mIERC X1VOVVNFRC4gU28gd2UgY2FuIHNraXAgOCBvciA0ICgzMiBiaXQpIHVudXNlZCBkYXRhYmFzZXMg aW4gb25lDQo+PiBzdGVwIChzdGlsbCB3aXRoIHRoZSBhc3N1bXB0aW9uIHRoYXQgYSB0cmFuc2Fj dGlvbiByYXJlbHkgdXNlcyBhbGwgZGF0YWJhc2VzIG9mDQo+PiB0aGUgZW52aXJvbm1lbnQpLg0K Pj4NCj4+IFNvIHRoZSBsb29wIGxvb2tzIGFzIGZvbGxvd3MgYWx3YXlzIHN0YXJ0aW5nIGF0IHRo ZSBsb3dlciBpbmRleCB0byBhdm9pZA0KPj4gYWxpZ25tZW50IGlzc3VlcyB3aXRoIEFSTSBwcmlv ciB2Ni4NCj4+DQo+PiAjZGVmaW5lIERCX1VOVVNFRCAgICAgICAweDIwICAgICAgICAgICAgICAg IC8qKjwgREIgbm90IHVzZWQgaW4gdGhpcyB0eG4gKi8NCj4+DQo+PiAjaWZkZWYgTURCX1ZMMzIN Cj4+ICNkZWZpbmUgREJfVU5VU0VEX0xPTkcgIDB4MjAyMDIwMjAgICAgICAgICAgLyogREJfVU5V U0VEIGxvbmcgbWFzayBmb3IgZmFzdA0KPj4gdHJhY2tpbmcgKi8NCj4+ICNlbHNlDQo+PiAjZGVm aW5lIERCX1VOVVNFRF9MT05HICAweDIwMjAyMDIwMjAyMDIwMjAgIC8qIERCX1VOVVNFRCBsb25n IG1hc2sgZm9yIGZhc3QNCj4+IHRyYWNraW5nICovDQo+PiAjZW5kaWYNCj4+DQo+Pg0KPj4gI2lm ZGVmIE1EQl9WTDMyDQo+PiAjZGVmaW5lIE1EQl9XT1JEICAgIHVuc2lnbmVkIGludA0KPj4gI2Vs c2UNCj4+ICNkZWZpbmUgTURCX1dPUkQgICAgdW5zaWduZWQgbG9uZyBsb25nDQo+PiAjZW5kaWYN Cj4+DQo+Pg0KPj4NCj4+ICAgICAgTURCX2RiaSAlMz0gc3JjLT5tdF9udW1kYnM7DQo+PiAgICAg IE1EQl9kYmkgaSA9IDA7DQo+Pg0KPj4gICAgICB3aGlsZSAoMSkgew0KPj4gICAgICAgICAgdW5z aWduZWQgaW50IHVwcGVyID0gaSArIHNpemVvZihNREJfV09SRCk7DQo+PiAgICAgICAgICBpZiAo dXBwZXIgPCBuKSB7DQo+PiAgICAgICAgICAgICAgLy8gc2tpcCB1bnVzZWQNCj4+ICAgICAgICAg ICAgICBpZiAoKCooTURCX1dPUkQgKikodGRiZmxhZ3MgKyBpKSkgPT0gREJfVU5VU0VEX0xPTkcp IHsNCj4+ICAgICAgICAgICAgICAgICAgaSA9IHVwcGVyOw0KPj4gICAgICAgICAgICAgICAgICBj b250aW51ZTsNCj4+ICAgICAgICAgICAgICB9DQo+PiAgICAgICAgICB9DQo+PiAgICAgICAgICBl bHNlIHsNCj4+ICAgICAgICAgICAgICB1cHBlciA9IG47DQo+PiAgICAgICAgICB9DQo+Pg0KPj4g ICAgICAgICAgZm9yICg7IGkgPCB1cHBlcjsgaSsrKSB7DQo+PiAgICAgICAgICAgICAgLy8gYW55 IG90aGVyIGZpbHRlciBjcml0ZXJpYSBhcHByb3ByaWF0ZSB0byB0aGUgZnVuY3Rpb24NCj4+ICAg ICAgICAgICAgICAuLi4uDQo+PiAgICAgICAgICB9DQo+PiAgICAgICAgICBpZiAoaSA+PSBuKSB7 DQo+PiAgICAgICAgICAgICAgYnJlYWs7DQo+PiAgICAgICAgICB9DQo+PiAgICAgIH0NCj4+DQo+ Pg0KPj4NCj4NCj4NCj4tLSANCj4gICAtLSBIb3dhcmQgQ2h1DQo+ICAgQ1RPLCBTeW1hcyBDb3Jw LiAgICAgICAgICAgaHR0cDovL3d3dy5zeW1hcy5jb20NCj4gICBEaXJlY3RvciwgSGlnaGxhbmQg U3VuICAgICBodHRwOi8vaGlnaGxhbmRzdW4uY29tL2h5Yy8NCj4gICBDaGllZiBBcmNoaXRlY3Qs IE9wZW5MREFQICBodHRwOi8vd3d3Lm9wZW5sZGFwLm9yZy9wcm9qZWN0Lw0K

1 0

Re: (ITS#8430) Improved handling for large number of databases
by h.b.furuseth＠usit.uio.no 27 May '16

27 May '16

On 27. mai 2016 12:57, juerg.bircher(a)helmedica.com wrote: > mdb_dbi_open() is however not improved with the assumption that the database > handle (dbi) is cached in the application. So mdb_dbi_open() should happen only > once for each database during the life time of an application. It must check if the DB is open anyway, since LDMB breaks if the user has two DBIs to the same DB. Though maybe we could add an optimization flag to skip the search, for users who know they don't reopen open DBs. Other notes: LMDB and OpenLDAP mostly use tab-width = indentation = 4. (We should comment that somewhere.) You seem to use tab-width = 8. I think MDB_WORD intends to be mdb_size_t, but should be 'unsigned long': It's probably no optimization to use 'long long' when that is not a native integer type. But I haven't profiled that. The delay in picking up env->me_dbflags is a bit troubling, since it means creating a new txn is no longer quite atomic. I can't think of how it'll make a difference when DBIs are used correctly, but that's one thing people screw up. Maybe it'd be OK if renew0() just copies it into mt_dbs[].md_flags without the '&', and mdb_setup_db_info() fixes it up. -- Hallvard

1 0

Re: (ITS#8430) Improved handling for large number of databases
by hyc＠symas.com 27 May '16

27 May '16

juerg.bircher(a)helmedica.com wrote: > Full_Name: Juerg Bircher > Version: lmdb (master) > OS: MacOS / Linux > URL: ftp://ftp.openldap.org/incoming/Juerg_Bircher_160527-Improved-handling-for-… > Submission from: (NULL) (178.82.37.195) Thanks for the patch, but for a change of this size you also need to include an IP Rights notice, as documented on the Contributing guidelines. > > > There is a increased performance penalty the more databases are created within > the same environment. I was looking for a way the improve that by keeping the > simplicity of tracking databases within a list with direct access by index > (MDB_dbi). > mdb_dbi_open() is however not improved with the assumption that the database > handle (dbi) is cached in the application. So mdb_dbi_open() should happen only > once for each database during the life time of an application. > > > One issue is that mdb_txn_begin() (for read-only transactions) calloc the > sizeof(MDB_txn) + me_maxdbs * sizeof(MDB_db + 1). The plus 1 for the dbflags. > However it is sufficient only to malloc that size and clear the sizeof(MDB_txn) > > memset(txn, 0, sizeof(MDB_txn) > > After that the data beyond the MDB_txn is not initialized which is ok for the > moment. > > The next improvement happens in mdb_txn_renew0() where the dbflags are only set > to DB_UNUSED (a new flag) for each database currently opened in the > environment. > > memset(txn->mt_dbflags, DB_UNUSED, txn->mt_numdbs); > > The former code used to to loop through each database to calculate the dbflags. > This is still done but lazily for each accessed database with the assumption > that a read only transaction rarely uses all databases of the environment. > > The lazy initialization of the dbflag happens in the macro TXN_DBI_EXIST which > is always used when a database handle (dbi) is passed to an function. > The flags are updated in mdb_setup_db_info() once a database is access which is > marked as unused (DB_UNUSED). > > static int mdb_setup_db_info(MDB_txn *txn, MDB_dbi dbi) { > /* Setup db info */ > uint16_t x = txn->mt_env->me_dbflags[dbi]; > txn->mt_dbs[dbi].md_flags = x & PERSISTENT_FLAGS; > txn->mt_dbflags[dbi] = (x & MDB_VALID) ? DB_VALID|DB_USRVALID|DB_STALE : 0; > return (txn->mt_dbflags[dbi] & validity); > } > > /** Check \b txn and \b dbi arguments to a function and initialize db info > if needed */ > #define TXN_DBI_EXIST(txn, dbi, validity) \ > ((txn) && (dbi#C3C(txn)->mt_numdbs && (((txn)->mt_dbflags[dbi] & (validity)) > || (((txn)->mt_dbflags[dbi] & DB_UNUSED) && mdb_setup_db_info((txn), (dbi), > (validity))))) > > > The next improvement is done in any function which needs to loop through the > databases for example in mdb_cursors_close(). Again the more databases in the > environment the longer the execution time. > It should be best if looping only through dbflags and searching for those > databases which are used (!DB_UNUSED). This could be done byte wise or more > efficient in 8/4 byte steps comparing with an extended mask DB_UNUSED_LONG > instead of DB_UNUSED. So we can skip 8 or 4 (32 bit) unused databases in one > step (still with the assumption that a transaction rarely uses all databases of > the environment). > > So the loop looks as follows always starting at the lower index to avoid > alignment issues with ARM prior v6. > > #define DB_UNUSED 0x20 /**< DB not used in this txn */ > > #ifdef MDB_VL32 > #define DB_UNUSED_LONG 0x20202020 /* DB_UNUSED long mask for fast > tracking */ > #else > #define DB_UNUSED_LONG 0x2020202020202020 /* DB_UNUSED long mask for fast > tracking */ > #endif > > > #ifdef MDB_VL32 > #define MDB_WORD unsigned int > #else > #define MDB_WORD unsigned long long > #endif > > > > MDB_dbi %3= src->mt_numdbs; > MDB_dbi i = 0; > > while (1) { > unsigned int upper = i + sizeof(MDB_WORD); > if (upper < n) { > // skip unused > if ((*(MDB_WORD *)(tdbflags + i)) == DB_UNUSED_LONG) { > i = upper; > continue; > } > } > else { > upper = n; > } > > for (; i < upper; i++) { > // any other filter criteria appropriate to the function > .... > } > if (i >= n) { > break; > } > } > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8431) LMDB: Access newly opened database from another transaction
by juerg.bircher＠helmedica.com 27 May '16

27 May '16

Full_Name: Jeerg Bircher Version: lmdb (master) OS: MacOS / Linux URL: ftp://ftp.openldap.org/incoming/Juerg_Bircher_160527-Access-newly-opened-da… Submission from: (NULL) (178.82.37.195) A transaction tracks newly opened databases and if the transaction is committed the newly opened databases are propagated to the list of open databases of the environment. However if the read only transaction is aborted the databases are not propagated. If database handles (MDB_dbi) are cached in the application to avoid calling mdb_dbi_open() there might be the situation of two threads running a read only transaction concurrently. First threads opens the database and commits the read-only transaction. The database is added to the list of open databases in the environment. The returned dbi is globally cached in the application. The second thread also wants to access the same database and finds the database handle in the global application cache. The database handle however is not valid as the transaction only uses a snapshot of open databases. So the second thread gets an EINVAL when using that database handle. This should not happen as the database is open and added to the environment. The following should fix this issue by updating the mt_numdbs and marking the delta with DB_UNUSED static int mdb_update_db_info(MDB_txn *txn, MDB_dbi dbi) { /* propagate newly opened db from env to current txn. Mark them as unused */ if (txn->mt_numdbs < txn->mt_env->me_numdbs) { memset(txn->mt_dbflags + txn->mt_numdbs, DB_UNUSED, txn->mt_env->me_numdbs - txn->mt_numdbs); } txn->mt_numdbs = txn->mt_env->me_numdbs; return dbi < txn->mt_numdbs; } /** Check \b txn and \b dbi arguments to a function and initialize db info if needed */ #define TXN_DBI_EXIST(txn, dbi, validity) \ ((txn) && ((dbi)<(txn)->mt_numdbs || mdb_update_db_info((txn), (dbi))) && (((txn)->mt_dbflags[dbi] & (validity)) || (((txn)->mt_dbflags[dbi] & DB_UNUSED) && mdb_setup_db_info((txn), (dbi), (validity))))) IMPORTANT depends ITS#8430 See also mailing list openldap-technical thread: Improved handling for large number of databases / Access newly opened database from another transaction

1 0

(ITS#8430) Improved handling for large number of databases
by juerg.bircher＠helmedica.com 27 May '16

27 May '16

Full_Name: Juerg Bircher Version: lmdb (master) OS: MacOS / Linux URL: ftp://ftp.openldap.org/incoming/Juerg_Bircher_160527-Improved-handling-for-… Submission from: (NULL) (178.82.37.195) There is a increased performance penalty the more databases are created within the same environment. I was looking for a way the improve that by keeping the simplicity of tracking databases within a list with direct access by index (MDB_dbi). mdb_dbi_open() is however not improved with the assumption that the database handle (dbi) is cached in the application. So mdb_dbi_open() should happen only once for each database during the life time of an application. One issue is that mdb_txn_begin() (for read-only transactions) calloc the sizeof(MDB_txn) + me_maxdbs * sizeof(MDB_db + 1). The plus 1 for the dbflags. However it is sufficient only to malloc that size and clear the sizeof(MDB_txn) memset(txn, 0, sizeof(MDB_txn) After that the data beyond the MDB_txn is not initialized which is ok for the moment. The next improvement happens in mdb_txn_renew0() where the dbflags are only set to DB_UNUSED (a new flag) for each database currently opened in the environment. memset(txn->mt_dbflags, DB_UNUSED, txn->mt_numdbs); The former code used to to loop through each database to calculate the dbflags. This is still done but lazily for each accessed database with the assumption that a read only transaction rarely uses all databases of the environment. The lazy initialization of the dbflag happens in the macro TXN_DBI_EXIST which is always used when a database handle (dbi) is passed to an function. The flags are updated in mdb_setup_db_info() once a database is access which is marked as unused (DB_UNUSED). static int mdb_setup_db_info(MDB_txn *txn, MDB_dbi dbi) { /* Setup db info */ uint16_t x = txn->mt_env->me_dbflags[dbi]; txn->mt_dbs[dbi].md_flags = x & PERSISTENT_FLAGS; txn->mt_dbflags[dbi] = (x & MDB_VALID) ? DB_VALID|DB_USRVALID|DB_STALE : 0; return (txn->mt_dbflags[dbi] & validity); } /** Check \b txn and \b dbi arguments to a function and initialize db info if needed */ #define TXN_DBI_EXIST(txn, dbi, validity) \ ((txn) && (dbi#C3C(txn)->mt_numdbs && (((txn)->mt_dbflags[dbi] & (validity)) || (((txn)->mt_dbflags[dbi] & DB_UNUSED) && mdb_setup_db_info((txn), (dbi), (validity))))) The next improvement is done in any function which needs to loop through the databases for example in mdb_cursors_close(). Again the more databases in the environment the longer the execution time. It should be best if looping only through dbflags and searching for those databases which are used (!DB_UNUSED). This could be done byte wise or more efficient in 8/4 byte steps comparing with an extended mask DB_UNUSED_LONG instead of DB_UNUSED. So we can skip 8 or 4 (32 bit) unused databases in one step (still with the assumption that a transaction rarely uses all databases of the environment). So the loop looks as follows always starting at the lower index to avoid alignment issues with ARM prior v6. #define DB_UNUSED 0x20 /**< DB not used in this txn */ #ifdef MDB_VL32 #define DB_UNUSED_LONG 0x20202020 /* DB_UNUSED long mask for fast tracking */ #else #define DB_UNUSED_LONG 0x2020202020202020 /* DB_UNUSED long mask for fast tracking */ #endif #ifdef MDB_VL32 #define MDB_WORD unsigned int #else #define MDB_WORD unsigned long long #endif MDB_dbi %3= src->mt_numdbs; MDB_dbi i = 0; while (1) { unsigned int upper = i + sizeof(MDB_WORD); if (upper < n) { // skip unused if ((*(MDB_WORD *)(tdbflags + i)) == DB_UNUSED_LONG) { i = upper; continue; } } else { upper = n; } for (; i < upper; i++) { // any other filter criteria appropriate to the function .... } if (i >= n) { break; } }

1 0

Re: (ITS#8427) Incorrect value of tls_reqcert in syncrepl
by mkp37215＠gmail.com 20 May '16

20 May '16

Howard, thanks for the reply. I just noticed a small error in what I wrote, the corrected fragment should be: "This was because sb->sb_tls_do_init was FALSE and bindconf_tls_set(sb, ld) was not called." I also would like to add that my patch changes the semantics of bindconf_tls_set, in regard of how TLS context is set, and that this is deliberate. I think that previous semantics was unclear and bug-prone, and that the new one is not only more straightforward, but also matches better the way bindconf_tls_set is used. As a result both bindconf_tls_set code and the code around its invocations is simplified. However, I was focused on its usage in slap_client_connect (because this is what was causing me problems), and I did not put much attention into other three places where bindconf_tls_set is called. All of those code fragments were basically identical, so I modified them the same way, but I think someone should review these modifications to see if they make sense. I originally intended to limit the impact of my patch to slap_client_connect, and to keep the changes inside config.c file. However, this resulted in making bad code worse, even less clear and manageable.

1 0

Re: (ITS#8427) Incorrect value of tls_reqcert in syncrepl
by hyc＠symas.com 20 May '16

20 May '16

mkp37215(a)gmail.com wrote: > The root cause of the problem: Thanks for the analysis, will look into it soon. > On syncrepl retry, code in slap_client_connect > (servers/slapd/config.c) initialized ld (LDAP) object with default > values, rather than values saved in sb (slap_bindconf). This was > because sb->sb_tls_do_init was true and bindconf_tls_set(sb, ld) was > not called. > This issue is not related to any specific TLS library. All syncrepl > retries are executed with wrong TLS settings in ld->ld_options, > including ldo_tls_require_cert, ldo_tls_keyfile, ldo_tls_certfile, > ldo_tls_cacertfile, and other (ldo_tls_require_cert has default value > 2 (demand), while string fields have nulls). However, the bug is > usually masked, because sb->sb_tls_ctx contains TLS context correctly > initialized during initial connection attempt, and this context is > assigned to ld as LDAP_OPT_X_TLS_CTX, and then used by the TLS > library, while incorrect values in ld->ld_options are ignored in most > places. But the host-checking code in ldap_int_tls_start > (libraries/libldap/tls2.c) uses ld->ld_options and triggers the bug. That description makes sense, OK. > Explanation of the patch: > The patch modifies function bindconf_tls_set that copies values from > slap_bindconf to LDAP object. Code that calls bindconf_tls_set is also > modified. > 1. If slap_bindconf object has TLS context, assign it to LDAP object. > 2. if slap_bindconf does not have a TLS context, create new context > and assign it it to both slap_bindconf and LDAP objects. > 3. Removed context replacement functionality and related conditions > (newctx). The intent of this functionality was unclear, but did not > appear to be compatible with how binconf_tls_set was used. > 4. The result of the above code changes is that TLS context is always > present and correctly assigned, while previously it sometimes was, and > sometimes wasn't. > 5. Simplified calls of bindconf_tls_set in the following files: > servers/slapd/config.c > servers/slapd/back-ldap/bind.c > servers/slapd/back-meta/conn.c > servers/slapd/back-asyncmeta/conn.c > > Things to do: > I tested the modified slap_client_connect (servers/slapd/config.c), > but the modifications to files in back-{ldap,meta,asyncmeta} were done > without proper code and functionality analysis, and without any > testing. I am hoping for a review from someone knowledgeable. > > Thank you > > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8427) Incorrect value of tls_reqcert in syncrepl
by mkp37215＠gmail.com 20 May '16

20 May '16

The root cause of the problem: On syncrepl retry, code in slap_client_connect (servers/slapd/config.c) initialized ld (LDAP) object with default values, rather than values saved in sb (slap_bindconf). This was because sb->sb_tls_do_init was true and bindconf_tls_set(sb, ld) was not called. This issue is not related to any specific TLS library. All syncrepl retries are executed with wrong TLS settings in ld->ld_options, including ldo_tls_require_cert, ldo_tls_keyfile, ldo_tls_certfile, ldo_tls_cacertfile, and other (ldo_tls_require_cert has default value 2 (demand), while string fields have nulls). However, the bug is usually masked, because sb->sb_tls_ctx contains TLS context correctly initialized during initial connection attempt, and this context is assigned to ld as LDAP_OPT_X_TLS_CTX, and then used by the TLS library, while incorrect values in ld->ld_options are ignored in most places. But the host-checking code in ldap_int_tls_start (libraries/libldap/tls2.c) uses ld->ld_options and triggers the bug. Explanation of the patch: The patch modifies function bindconf_tls_set that copies values from slap_bindconf to LDAP object. Code that calls bindconf_tls_set is also modified. 1. If slap_bindconf object has TLS context, assign it to LDAP object. 2. if slap_bindconf does not have a TLS context, create new context and assign it it to both slap_bindconf and LDAP objects. 3. Removed context replacement functionality and related conditions (newctx). The intent of this functionality was unclear, but did not appear to be compatible with how binconf_tls_set was used. 4. The result of the above code changes is that TLS context is always present and correctly assigned, while previously it sometimes was, and sometimes wasn't. 5. Simplified calls of bindconf_tls_set in the following files: servers/slapd/config.c servers/slapd/back-ldap/bind.c servers/slapd/back-meta/conn.c servers/slapd/back-asyncmeta/conn.c Things to do: I tested the modified slap_client_connect (servers/slapd/config.c), but the modifications to files in back-{ldap,meta,asyncmeta} were done without proper code and functionality analysis, and without any testing. I am hoping for a review from someone knowledgeable. Thank you

1 0

Re: (ITS#8427) Incorrect value of tls_reqcert in syncrepl
by mkp37215＠gmail.com 20 May '16

20 May '16

Proposed patch: diff -ru servers/slapd.orig/config.c servers/slapd/config.c --- servers/slapd.orig/config.c 2016-05-16 16:49:08.000000000 -0500 +++ servers/slapd/config.c 2016-05-19 20:28:54.002163670 -0500 @@ -1864,7 +1864,7 @@ int bindconf_tls_set( slap_bindconf *bc, LDAP *ld ) { - int i, rc, newctx = 0, res = 0; + int i, rc, res = 0; char *ptr = (char *)bc, **word; bc->sb_tls_do_init = 0; @@ -1878,8 +1878,7 @@ "bindconf_tls_set: failed to set %s to %s\n", bindtlsopts[i].key, *word, 0 ); res = -1; - } else - newctx = 1; + } } } if ( bc->sb_tls_reqcert ) { @@ -1890,8 +1889,7 @@ "bindconf_tls_set: failed to set tls_reqcert to %s\n", bc->sb_tls_reqcert, 0, 0 ); res = -1; - } else - newctx = 1; + } } if ( bc->sb_tls_protocol_min ) { rc = ldap_pvt_tls_config( ld, LDAP_OPT_X_TLS_PROTOCOL_MIN, @@ -1901,8 +1899,7 @@ "bindconf_tls_set: failed to set tls_protocol_min to %s\n", bc->sb_tls_protocol_min, 0, 0 ); res = -1; - } else - newctx = 1; + } } #ifdef HAVE_OPENSSL_CRL if ( bc->sb_tls_crlcheck ) { @@ -1913,17 +1910,15 @@ "bindconf_tls_set: failed to set tls_crlcheck to %s\n", bc->sb_tls_crlcheck, 0, 0 ); res = -1; - } else - newctx = 1; + } } #endif - if ( newctx ) { + if ( bc->sb_tls_ctx ) { + rc = ldap_set_option( ld, LDAP_OPT_X_TLS_CTX, bc->sb_tls_ctx ); + if ( rc ) + res = rc; + } else { int opt = 0; - - if ( bc->sb_tls_ctx ) { - ldap_pvt_tls_ctx_free( bc->sb_tls_ctx ); - bc->sb_tls_ctx = NULL; - } rc = ldap_set_option( ld, LDAP_OPT_X_TLS_NEWCTX, &opt ); if ( rc ) res = rc; @@ -2000,14 +1995,7 @@ slap_client_keepalive(ld, &sb->sb_keepalive); #ifdef HAVE_TLS - if ( sb->sb_tls_do_init ) { - rc = bindconf_tls_set( sb, ld ); - - } else if ( sb->sb_tls_ctx ) { - rc = ldap_set_option( ld, LDAP_OPT_X_TLS_CTX, - sb->sb_tls_ctx ); - } - + rc = bindconf_tls_set( sb, ld ); if ( rc ) { Debug( LDAP_DEBUG_ANY, "slap_client_connect: " diff -ru servers/slapd.orig/back-ldap/bind.c servers/slapd/back-ldap/bind.c --- servers/slapd.orig/back-ldap/bind.c 2016-05-16 16:49:07.000000000 -0500 +++ servers/slapd/back-ldap/bind.c 2016-05-19 19:41:35.654431746 -0500 @@ -735,11 +735,7 @@ sb = &li->li_tls; } - if ( sb->sb_tls_do_init ) { - bindconf_tls_set( sb, ld ); - } else if ( sb->sb_tls_ctx ) { - ldap_set_option( ld, LDAP_OPT_X_TLS_CTX, sb->sb_tls_ctx ); - } + bindconf_tls_set( sb, ld ); /* if required by the bindconf configuration, force TLS */ if ( ( sb == &li->li_acl || sb == &li->li_idassert.si_bc ) && diff -ru servers/slapd.orig/back-meta/conn.c servers/slapd/back-meta/conn.c --- servers/slapd.orig/back-meta/conn.c 2016-05-16 16:49:07.000000000 -0500 +++ servers/slapd/back-meta/conn.c 2016-05-19 19:42:33.365580781 -0500 @@ -433,11 +433,7 @@ sb = &mt->mt_tls; } - if ( sb->sb_tls_do_init ) { - bindconf_tls_set( sb, msc->msc_ld ); - } else if ( sb->sb_tls_ctx ) { - ldap_set_option( msc->msc_ld, LDAP_OPT_X_TLS_CTX, sb->sb_tls_ctx ); - } + bindconf_tls_set( sb, msc->msc_ld ); if ( !is_ldaps ) { if ( sb == &mt->mt_idassert.si_bc && sb->sb_tls_ctx ) { diff -ru servers/slapd.orig/back-asyncmeta/conn.c servers/slapd/back-asyncmeta/conn.c --- servers/slapd.orig/back-asyncmeta/conn.c 2016-05-16 16:49:07.000000000 -0500 +++ servers/slapd/back-asyncmeta/conn.c 2016-05-19 19:43:24.164354246 -0500 @@ -277,11 +277,7 @@ sb = &mt->mt_tls; } - if ( sb->sb_tls_do_init ) { - bindconf_tls_set( sb, msc->msc_ld ); - } else if ( sb->sb_tls_ctx ) { - ldap_set_option( msc->msc_ld, LDAP_OPT_X_TLS_CTX, sb->sb_tls_ctx ); - } + bindconf_tls_set( sb, msc->msc_ld ); if ( !is_ldaps ) { if ( sb == &mt->mt_idassert.si_bc && sb->sb_tls_ctx ) { -------- END OF PATCH

1 0

Re: (ITS#8429) deadlock on a modification operation with replication
by hyc＠symas.com 19 May '16

19 May '16

elecharny(a)apache.org wrote: > Full_Name: Emmanuel Lecharny > Version: 2.4.44 > OS: Linux CentOS 6 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (92.169.142.218) > > > I have a server blocked on a modification operation, which can neither be > shutdown nicely. Looking at the backtrace, I see that two threads are mutually > blocked (thread 2 and thread 3) : using gdb, I saw that the threads were blocked on a rmutex whose owning thread had already exited. The fact that the thread exited is due to the fact that a slapd shutdown was requested. Since the thread was gone already, there's nothing more we can extract from this process. This shows that there's a bug in the accesslog overlay, where an operation may get interrupted without releasing the rmutex. Someone will need to track that down. > > Thread 4 (Thread 0x7f305fc97700 (LWP 28853))A%A > #0 0x0000003a9ba0b63c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f30a46395ab in ldap_pvt_thread_pool_destroy (tpool=0x727100, > run_pending=1) at /home/build/sold-2.4.44.2/openldap/libraries/libldap_r/tpool.c:817 > #2 0x000000000042c619 in slapd_daemon_task (ptr=<value optimized out>) at > /home/build/sold-2.4.44.2/openldap/servers/slapd/daemon.c:2851 > #3 0x0000003a9ba07a51 in start_thread () from /lib64/libpthread.so.0 > #4 0x0000003a9b6e896d in clone () from /lib64/libc.so.6 > > Thread 3 (Thread 0x7f305f496700 (LWP 28854)): > #0 0x0000003a9ba0b63c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f30a463839b in ldap_pvt_thread_rmutex_lock (rmutex=<value optimized > out>, owner=139845733803776) at > /home/build/sold-2.4.44.2/openldap/libraries/libldap_r/rmutex.c:129 > #2 0x00007f30a09cf0cf in accesslog_op_mod (op=0x7f305f495450, rs=<value > optimized out>) at /home/build/sold-2.4.44.2/openldap/servers/slapd/overlays/accesslog.c:1984 > #3 0x0000000000496dc2 in overlay_op_walk (op=0x7f305f495450, rs=0x7f305f494740, > which=op_modify, oi=0x886a30, on=0x88ba10) at > /home/build/sold-2.4.44.2/openldap/servers/slapd/backover.c:661 > #4 0x0000000000497873 in over_op_func (op=0x7f305f495450, rs=<value optimized > out>, which=<value optimized out>) at > /home/build/sold-2.4.44.2/openldap/servers/slapd/backover.c:730 > #5 0x000000000048bea6 in syncrepl_updateCookie (si=0x886650, op=0x7f305f495450, > syncCookie=<value optimized out>) at > /home/build/sold-2.4.44.2/enenldap/servers/slapd/syncrepl.c:3889 > #6 0x000000000048cf1b in do_syncrep2 (op=0x7f305f495450, si=0x886650) at > /home/build/sold-2.4.44.2/openldap/servers/slapd/syncrepl.c:1187 > #7 0x00000000004932a2 in do_syncrepl (ctx=<value optimized out>, arg=0x885b50) > at /home/build/sold-2.4.44.2/openldap/servers/slapd/syncrepl.c:1561 > #8 0x00007f30a463a058 in ldap_int_thread_pool_wrapper (xpool=0x817d80) at > /home/build/sold-2.4.44.2/openldap/libraries/libldap_r/tpool.c:956 > #9 0x0000003a9ba07a51 in start_thread ()rfrom /lib62F2Flibpthread.so.0 > #10 0x0000003a9b6e896d in clone () from /lib64/libc.so.6 > > Thread 2 (Thread 0x7f305ec95700 (LWP 28855)): > #0 0x0000003a9ba0b63c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00007f30a463839b in ldap_pvt_thread_rmutex_lock (rmutex=<value optimized > out>, owner=139845725411072) at > /home/build/sold-2.4.44.2/openldap/libraries/libldap_r/rmutex.c:129 > #2 0x00007f30a09cf0cf in accesslog_op_mod (op=0x7f3050114290, rs=<value > optimized out>) at /home/build/sold-2.4.44.2%2pepenldap/servers/slapd/overlays/accesslog.c:1984 > #3 0x0000000000496dc2 in overlay_op_walk (op=0x7f3050114290, rs=0x7f305ec949f0, > which=op_modify, oi=0x886a30, on=0x88ba10) at > /home/build/sold-2.4.44.2/openldap/servers/slapd/backover.c:661 > #4 0x0000000000497873 in over_op_func (op=0x7f3050114290, rs=<value optimized > out>, which=<value optimized out>) at > /home/build/sold-2.4.44.2/openldap/servers/slapd/backover.c:730 > #5 0x00000000004464eb in fe_op_modify (op=0x7f3050114290, rs=0x7f305ec949f0) at > /home/build/sold-2.4.44.2/openldap/servers/slapd/modify.c:303 > #6 0x0000000000446e16 in do_modify (op=0x7f3050114290, rs=0x7f305ec949f0) at > /home/build/sold-2.4.44.2/openldap/servers/slapd/modify.c:177 > #7 0x000000000042ea79 in connection_operation (ctx=0x7f305ec94b70, > arg_v=0x7f3050114290) at > /home/build/sold-2.4.44.2/openldap/servers/slapd/connection.c:1158 > #8 0x000000000042f265 in connection_read_thread (ctx=0x7f305ec94b70, > argv=<value optimized out>) at > /home/build/sold-2.4.44.2/openldap/servers/slapd/connection.c:1294 > #9 0x00007f30a463a058 in ldap_int_thread_pool_wrapper (xpool=0x817d80) at > /home/build/sold-2.4.44.2/openldap/libraries/libldap_r/tpool.c:956 > #10 0x0000003a9ba07a51 in start_thread () from /lib64/libpthread.so.0 > #11 0x0000003a9b6e896d in clone () from /lib64/libc.so.6 > > Thread 1 (Thread 0x7f30a3565700 (LWP 28852)): > #0 0x0000003a9ba082ad in pthread_join () from /lib64/libpthread.so.0 > #1 0x0000000000428fc9 in slapd_daemon () at > /home/build/sold-2.4.44.2/openldap/servers/apapd/daemon.c:2932 > #2 0x0000000000415135 in main (argc=9, argv=<value optimized out>) at > /home/build/sold-2.4.44.2/openldap/servers/slapd/main.c:1016 -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8429) deadlock on a modification operation with replication
by elecharny＠apache.org 19 May '16

19 May '16

Full_Name: Emmanuel Lecharny Version: 2.4.44 OS: Linux CentOS 6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (92.169.142.218) I have a server blocked on a modification operation, which can neither be shutdown nicely. Looking at the backtrace, I see that two threads are mutually blocked (thread 2 and thread 3) : Thread 4 (Thread 0x7f305fc97700 (LWP 28853))A%A #0 0x0000003a9ba0b63c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f30a46395ab in ldap_pvt_thread_pool_destroy (tpool=0x727100, run_pending=1) at /home/build/sold-2.4.44.2/openldap/libraries/libldap_r/tpool.c:817 #2 0x000000000042c619 in slapd_daemon_task (ptr=<value optimized out>) at /home/build/sold-2.4.44.2/openldap/servers/slapd/daemon.c:2851 #3 0x0000003a9ba07a51 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003a9b6e896d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f305f496700 (LWP 28854)): #0 0x0000003a9ba0b63c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f30a463839b in ldap_pvt_thread_rmutex_lock (rmutex=<value optimized out>, owner=139845733803776) at /home/build/sold-2.4.44.2/openldap/libraries/libldap_r/rmutex.c:129 #2 0x00007f30a09cf0cf in accesslog_op_mod (op=0x7f305f495450, rs=<value optimized out>) at /home/build/sold-2.4.44.2/openldap/servers/slapd/overlays/accesslog.c:1984 #3 0x0000000000496dc2 in overlay_op_walk (op=0x7f305f495450, rs=0x7f305f494740, which=op_modify, oi=0x886a30, on=0x88ba10) at /home/build/sold-2.4.44.2/openldap/servers/slapd/backover.c:661 #4 0x0000000000497873 in over_op_func (op=0x7f305f495450, rs=<value optimized out>, which=<value optimized out>) at /home/build/sold-2.4.44.2/openldap/servers/slapd/backover.c:730 #5 0x000000000048bea6 in syncrepl_updateCookie (si=0x886650, op=0x7f305f495450, syncCookie=<value optimized out>) at /home/build/sold-2.4.44.2/enenldap/servers/slapd/syncrepl.c:3889 #6 0x000000000048cf1b in do_syncrep2 (op=0x7f305f495450, si=0x886650) at /home/build/sold-2.4.44.2/openldap/servers/slapd/syncrepl.c:1187 #7 0x00000000004932a2 in do_syncrepl (ctx=<value optimized out>, arg=0x885b50) at /home/build/sold-2.4.44.2/openldap/servers/slapd/syncrepl.c:1561 #8 0x00007f30a463a058 in ldap_int_thread_pool_wrapper (xpool=0x817d80) at /home/build/sold-2.4.44.2/openldap/libraries/libldap_r/tpool.c:956 #9 0x0000003a9ba07a51 in start_thread ()rfrom /lib62F2Flibpthread.so.0 #10 0x0000003a9b6e896d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f305ec95700 (LWP 28855)): #0 0x0000003a9ba0b63c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f30a463839b in ldap_pvt_thread_rmutex_lock (rmutex=<value optimized out>, owner=139845725411072) at /home/build/sold-2.4.44.2/openldap/libraries/libldap_r/rmutex.c:129 #2 0x00007f30a09cf0cf in accesslog_op_mod (op=0x7f3050114290, rs=<value optimized out>) at /home/build/sold-2.4.44.2%2pepenldap/servers/slapd/overlays/accesslog.c:1984 #3 0x0000000000496dc2 in overlay_op_walk (op=0x7f3050114290, rs=0x7f305ec949f0, which=op_modify, oi=0x886a30, on=0x88ba10) at /home/build/sold-2.4.44.2/openldap/servers/slapd/backover.c:661 #4 0x0000000000497873 in over_op_func (op=0x7f3050114290, rs=<value optimized out>, which=<value optimized out>) at /home/build/sold-2.4.44.2/openldap/servers/slapd/backover.c:730 #5 0x00000000004464eb in fe_op_modify (op=0x7f3050114290, rs=0x7f305ec949f0) at /home/build/sold-2.4.44.2/openldap/servers/slapd/modify.c:303 #6 0x0000000000446e16 in do_modify (op=0x7f3050114290, rs=0x7f305ec949f0) at /home/build/sold-2.4.44.2/openldap/servers/slapd/modify.c:177 #7 0x000000000042ea79 in connection_operation (ctx=0x7f305ec94b70, arg_v=0x7f3050114290) at /home/build/sold-2.4.44.2/openldap/servers/slapd/connection.c:1158 #8 0x000000000042f265 in connection_read_thread (ctx=0x7f305ec94b70, argv=<value optimized out>) at /home/build/sold-2.4.44.2/openldap/servers/slapd/connection.c:1294 #9 0x00007f30a463a058 in ldap_int_thread_pool_wrapper (xpool=0x817d80) at /home/build/sold-2.4.44.2/openldap/libraries/libldap_r/tpool.c:956 #10 0x0000003a9ba07a51 in start_thread () from /lib64/libpthread.so.0 #11 0x0000003a9b6e896d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f30a3565700 (LWP 28852)): #0 0x0000003a9ba082ad in pthread_join () from /lib64/libpthread.so.0 #1 0x0000000000428fc9 in slapd_daemon () at /home/build/sold-2.4.44.2/openldap/servers/apapd/daemon.c:2932 #2 0x0000000000415135 in main (argc=9, argv=<value optimized out>) at /home/build/sold-2.4.44.2/openldap/servers/slapd/main.c:1016 This server is configured to use delta-syncrepl. Here is the slapd.conf file : #----------------------------------------------------------------------- # Main configuration # # MMR delta-syncrepl with TLS # # Server 1 will be replicated with Server 2 : # # +--------+ +--------+ # | Server |---[TLS]---->| Server | # | 1 |<---[TLS]----| 2 | # +--------+ +--------+ # # See slapd.conf(5) for details on configuration options. # This file should NOT be world readable. #----------------------------------------------------------------------- serverid 2 #----------------------------------------------------------------------- # SCHEMA INCLUDES # Use only those you need and leave the rest commented out. include /opt/symas/etc/openldap/schema/core.schema include /opt/symas/etc/openldap/schema/ppolicy.schema include /opt/symas/etc/openldap/schema/cosine.schema include /opt/symas/etc/openldap/schema/inetorgperson.schema include /opt/symas/etc/openldap/schema/krb5-kdc.schema include /opt/symas/etc/openldap/schema/rfc2307bis.schema include /opt/symas/etc/openldap/schema/openssh.schema #----------------------------------------------------------------------- # PID / STARTUP ARGS # Files in which to store the process id and startup arguments. # These files are needed by the init scripts, so only change # these if you are prepared to edit those scripts as well. pidfile /var/symas/run/mmr-delta/slapd.pid argsfile /var/symas/run/mmr-delta/slapd.args #----------------------------------------------------------------------- # TLS Setup Section TLSCACertificateFile /opt/symas/ssl/mmr-delta/cacert.pem TLSCertificateFile /opt/symas/ssl/mmr-delta/slapdcert.pem TLSCertificateKeyFile /opt/symas/ssl/mmr-delta/slapdkey.pem TLSCipherSuite HIGH:MEDIUM TLSVerifyClient try TLSProtocolMin 3.1 security ssf=128 tls=128 # This is the user that do the replication authz-regexp "email=elecharny(a)symas.com,cn=([^,]*),ou=symas,o=symas corp,l=new york,st=new jersey,c=us" "cn=replicator,dc=symas,dc=com" #----------------------------------------------------------------------- # Symas OpenLDAP supports threaded slapadd. This is only useful if running # slapadd on a multi-cpu box. Generally, assign 1 thread per # cpu, so if it is a 4 cpu box, use tool-threads 4. This # specifically affects the creation of index databases, so if # your database has fewer indices than CPUs, set it to the # number of indices. #tool-threads 2 #----------------------------------------------------------------------- # MODULE PATH # Choose the directory for loadable modules. modulepath /opt/symas/lib64/openldap # Uncomment the moduleloads as needed to enable additional # functionalityi when configured. NOTE: We package many # more modules options than those found below. moduleload back_mdb.la moduleload back_monitor.la moduleload ppolicy.la moduleload syncprov.la moduleload accesslog.la moduleload pw-sha2 D%D #----------------------------------------------------------------------- # Sample access control policy: #----------------------------------------------------------------------- access to dn="" by * read # The replicator user has write access to the while DIT # The users authenticated through a secured connection can read the DIT access to dn.subtree="dc=symas,dc=com" by dn.exact="cn=replicator,dc=symas,dc=com" write # by * tls_ssf=128 read # Allow self write access # Allow auenenticated uss s read access # Allow anonymous users to authenticate access to * by self write by users read by anonymous auth # rootdn can always write! #----------------------------------------------------------------------- # LOGGING loglevel stats sync ####################################################################### # config database ####################################################################### database config rootdn "cn=admin,cn%coconfig" rootpw {SSHA256}################### ####################################################################### # Symas LMDB database definitions #######################################################################% dAdatabase mdb suffix "dc=symas,dc=com" rootdn "dc=symas,dc=com" rootpw {SSHA256}############# limits dn.exact="cn=replicator,dc=symas,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited # Indices to maintain index default eq index objectClass index cn eq,sub index memberUID index givenName eq,sub index uniqueMember index mail eq,sub index entryUUID eq index entryCSN eq index uid eq,sub directory /var/symas/openldap-data/mmr-delta/symas maxsize 1073741824 #----------------------------------------------------------------------- # SYNCREPL [LDAP1, server1] syncrepl rid=1 provider=ldap://<server1>:1389 bindmethod=sasl saslmech=external starttls=yes tls_cacert=/opt/symas/ssl/mmr-delta/cacert.pem tls_cert=/opt/symas/ssl/mmr-delta/slapdcert.pem tls_key=/opt/symas/ssl/mmr-delta/slapdkey.pem tls_reqcert=demand type=refreshAndPersist searchbase="dc=symas,dc=com" filter="(objectclass=*)" scope=sub schemachecking=on retry="5 10 60 +" logbase="cn=accesslog" loilteter="(&(objectClass=auditWriteObject)(reqResult=0))" syncdata=accesslog sizeLimit=unlimited timelimit=unlimited # # SYNCREPL [LDAP2, server2] syncrepl rid=2 provider=ldap://<server2>:1389 bindmethod=sasl saslmech=external starttls=yes tls_cacert=/opt/symas/ssl/mmr-delta/cacert.pem tls_cert=/opt/symas/ssl/mmr-delta/slapdcert.pem tls_key=/opt/symas/ssl/mmr-delta/slapdkey.pem tls_reqcert=demand type=refreshAndPersist searchbase="dc=symas,dc=com" filter="(objectclass=*)" scope=sub schemachecking=on retry="5 10 60 +" logbase="cn=accesslog" logfilter="(&(objectClass=auditWriteObject9%9(reqResult=0))" syncdata=accesslog sizeLimit=unlimited timelimit=unlimited # ENABLE MIRROR MODE mirrormode TRUE #----------------------------------------------------------------------- # Load an instance of the ppolicy overlay for the current database: overlay ppolicy # Specify the default password policy subentry to use when none is # specified in an account's entry ppolicy_default "cn=default,ou=Policies,dc=symas,dc=com" #----------------------------------------------------------------------- # Symas database overlays (syncprov and accesslog) # OVERLAY [SYNCPROV] overlay syncprov syncprov-checkpoint 100 10 syncprov-sessionlog 10000 syncprov-reloadhint TRUE #----------------------------------------------------------------------- # OVERLAY [ACCESSLOG] overlay accesslog logdb cn=accesslog logops writes logsuccess TRUE logpurge 24:00 01+00:00 ########################################################"3%2############# # AccessLog database ####################################################################### database mdb directory /var/symas/openldap-data/mmr-delta/accesslog maxsize 5120000 suffix "cn=accesslog" rootdn "cn=accesslog" index default eq index objectClass,entryCSN,entryUUID,reqEnd,reqResult,reqStart access to * by dn.base="cn=replicator,dc=symas,dc=com" read by dn.base="dc=symas,dc=com" write #----------------------------------------------------------------------- # AccessLog overlay (syncprov) overlay syncprov syncprov-nopresent TRUE syncprov-reloadhint TRUE syncprov-checkpoint 100 10 syncprov-sessionlog 10000 limits dn.exact="cn=replicator,dc=symas,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited ####################################################################### # Monitor database #################################################################%#%3#### database monitor When I try to shutdown the server nicely, the logs in syslog show : May 19 04:18:15 cantal slapd[28852]: slapd shutdown: waiting for 2 operations/tasks to finish

1 0

Re: (ITS#8428) slapd segfaults under load when back-relay is used
by hyc＠symas.com 18 May '16

18 May '16

andrew.howard(a)anu.edu.au wrote: > Full_Name: Andrew Howard > Version: 2.4.40 > OS: Centos 7 3.10.0-327.18.2.el7.x86_6 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (150.203.248.180) > > > When using back-relay slapd segfaults after a few minutes. > > To reproduce: I run 5 instances of getent passwd;getent group > on a system using nslcd pointing to the openldap server and > then a loop of > > ldapsearch -x -LLL -h acmeldap1.acme.org.au -b "dc=newacme,dc=edu,dc=au" > > causes slapd to segfault in a few minutes. > Thanks for the detailed report. Should be fixed now by commit 2e60bf5ed00c1a8794131f53a6c72a78c0766e21 in git master. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8428) slapd segfaults under load when back-relay is used
by andrew.howard＠anu.edu.au 18 May '16

18 May '16

Full_Name: Andrew Howard Version: 2.4.40 OS: Centos 7 3.10.0-327.18.2.el7.x86_6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (150.203.248.180) When using back-relay slapd segfaults after a few minutes. To reproduce: I run 5 instances of getent passwd;getent group on a system using nslcd pointing to the openldap server and then a loop of ldapsearch -x -LLL -h acmeldap1.acme.org.au -b "dc=newacme,dc=edu,dc=au" causes slapd to segfault in a few minutes. [root@acmeldap1 ~]# gdb /usr/sbin/slapd GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/slapd...Reading symbols from /usr/lib/debug/usr/sbin/slapd.debug...done. done. (gdb) run -u ldap -h "ldap:/// ldapi:///" -d 0 The program being debugged has been started already. Start it from the beginning? (y or n) y Starting program: /usr/sbin/slapd -u ldap -h "ldap:/// ldapi:///" -d 0 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". May 18 15:02:21 acmeldap1 slapd[12262]: @(#) $OpenLDAP: slapd 2.4.40 (Mar 31 2016 15:24:52) $#012#011mockbuild@worker1.bsys.centos.org:/builddir/build/BUILD/openldap-2.4.40/openldap-2.4.%2/servers/slapd [New Thread 0x7fff37ea2700 (LWP 12263)] [New Thread 0x7fff376a1700 (LWP 12264)] [New Thread 0x7fff36ea0700 (LWP 12265)] [New Thread 0x7fff36499700 (LWP 12266)] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fff36499700 (LWP 12266)] 0x00007fff36498960 in ?? () (gdb) thread apply all bt Thread 5 (Thread 0x7fff36499700 (LWP 12266)): #0 0x00007fff36498960 in ?? () #1 0x00005555555a8541 in slap_writewait_play (op=0x7fff200026f0) at result.c:294 #2 send_ldap_ber (op=op@entry=0x7fff200026f0, ber=ber@entry=0x7fff36306eb0) at result.c:367 #3 0x00005555555ac11c in slap_send_search_entry (op=0x7fff200026f0, rs=<optimized out>) at result.c:1430 #4 0x0000555555645398 in mdb_search (op=<optimized out>, rs=<optimized out>) at search.c:1072 #5 0x0000555555606926 in overlay_op_walk (op=op@entry=0x7fff200026f0, rs=0x7fff36498960, which=op_search, oi=0x555555acaf90, on=0x0) at backover.c:671 #6 0x0000555555606a94 in over_op_func (op=0x7fff200026f0, rs=<optimized out>, which=<optimized out>) at backover.c:723 #7 0x00007ffff3ec1d16 in relay_back_op (op=0x7fff200026f0, rs=0x7fff36498960, which=<optimized out>) at op.c:210 #8 0x0000555555606926 in overlay_op_walk (op=op@entry=0x7fff200026f0, rs=0x7fff36496960, which=op_search, oi=0x555555aaf250, on=0x0) at backover.c:671 #9 0x0000555555606a94 in over_op_func (op=0x7fff200026f0, rs=<optimized out>, which=<optimized out>) at backover.c:723 #10 0x000055555559ad31 in fe_op_search (op=0x7fff200026f0, rs=0x7fff36498960) at search.c:402 #11 0x000055555559a5e6 in do_search (op=<optimized out>, rs=<optimized out>) at search.c:247 #12 0x0000555555597cbc in connection_operation (ctx=ctx@entry=0x7fff36498bd0, arg_v=arg_v@entry=0x7fff200026f0) at connection.c:1155 #13 0x000055555559802b in connection_read_thread (ctx=0x7fff36498bd0, argv=0x12) at connection.c:1291 #14 0x00007ffff7b92eda in ldap_int_thread_pool_wrapper (xpool=0x5555559ef2f0) at tpool.c:688 #15 0x00007ffff6e5adc5 in start_thread (arg=0x7fff36499700) at pthread_create.c:308 #16 0x00007ffff631bced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 4 (Thread 0x7fff36ea0700 (LWP 12265)): #0 0x00007ffff6e6150d in connect () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffff7baa9bc in ldap_pvt_connect (async=0, addrlen=16, sin=0x7fff2c100cf0, s=17, ld=0x7fff2c100910) at os-ip.c:443 #2 ldap_connect_to_host (ld=ld@entry=0x7fff2c100910, sb=0x7fff2c108d30, proto=proto@entry=1, srv=srv@entry=0x7fff2c108e00, async=async@entry=0) at os-ip.c:657 #3 0x00007ffff7b9430e in ldap_int_open_connection (ld=ld@entry=0x7fff2c100910, conn=conn@entry=0x7fff2c108d60, srv=0x7fff2c108e00, async=async@entry=0) at open.c:379 #4 0x00007ffff7ba7bbd in ldap_new_connection (ld=ld@entry=0x7fff2c100910, srvlist=srvlist@entry=0x7fff2c1009d8, use_ldsb=use_ldsb@entry=1, connect=connect@entry=1, bind=bind@entry=0x0, m_req=m_req@entry=0, m_res=m_res@entry=0) at request.c:484 #5 0x00007ffff7b938bf in ldap_open_defconn (ld=ld@entry=0x7fff2c100910) at open.c:41 #6 0x00007ffff7ba90e8 in ldap_send_initial_request (ld=ld@entry=0x7fff2c100910, msgtype=msgtype@entry=96, dn=dn@entry=0x555555ad4b70 "cn=replicator,ou=admins,dc=acme,dc=org,dc=au", ber=ber@entry=0x7fff2c100c60, msgid=msgid@entry=1) at request.c:130 #7 0x00007ffff7b9dbd6 in ldap_sasl_bind (ld=ld@entry=0x7fff2c100910, dn=dn@entry=0x555555ad4b70 "cn=replicator,ou=admins,dc=acme,dc=org,dc=au", mechanism=mechanism@entry=0x0, cred=cred@entry=0x555555ad48b8, sctrls=sctrls@entry=0x0, cctrls=<optimized out>, msgidp=msgidp@entry=0x7fff36e9f3f4) at sasl.c:148 #8 0x00007ffff7b9e159 in ldap_sasl_bind_s (ld=0x7fff2c100910, dn=0x555555ad4b70 "cn=replicator,ou=admins,dc=acme,dc=org,dc=au", mechanism=mechanism@entry=0x0, cred=cred@entry=0x5555adad48b8, sctrls=sctrls@entry=0x0, cctrls=cctrls@entry=0x0, servercredp=servercredp@entry=0x0) at sasl.c:182 #9 0x000055555558f0ba in slap_client_connect (ldp=ldp@entry=0x555555ad4aa8, sb=sb@entry=0x555555ad4880) at config.c:2104 #10 0x00005555555ffe2d in do_syncrep1 (si=0x555555ad4850, op=0x7fff36e9f7b0) at syncrepl.c:613 #11 do_syncrepl (ctx=<optimized out>, arg=0x555555ace0a0) at syncrepl.c:1527 #12 0x00007ffff7b92eda in ldap_int_thread_pool_wrapper (xpool=0x5555559ef2f0) at tpool.c:688 #13 0x00007ffff6e5adc5 in start_thread (arg=0x7fff36ea0700) at pthread_create.c:308 #14 0x00007ffff631bced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 3 (Thread 0x7fff376a1700 (LWP 12264)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/ux/2Fsysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffff7b92f2b in ldap_int_thread_pool_wrapper (xpool=0x5555559ef2f0) at tpool.c:675 #2 0x00007ffff6e5adc5 in start_thread (arg=0x7fff376a1700) at pthread_create.c:808 #3 0x00007ffff631bced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 2 (Thread 0x7fff37ea2700 (LWP 12263)): #0 0x00007ffff631c2c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000555555592e98 in slapd_daemon_task (ptr=<optimized out>) at daemon.c:2536 #2 0x00007ffff6e5adc5 in start_thread (arg=0x7fff37ea2700) at pthread_create.c:308 #3 0x00007ffff631bced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 1 (Thread 0x7ffff7fe7740 (LWP 12262)): ---Type <return> to continue, or q <return> to quit--- #0 0x00007ffff6e5bef7 in pthread_join (threadid=140734131480320, thread_return=thread_return@entry=0x0) at pthread_join.c:92 #1 0x00007ffff7b935f5 in ldap_pvt_thread_join (thread=<optimized out>, thread_return=thread_return@entry=0x0) at thr_posix.c:197 #2 0x0000555555594d91 in slapd_daemon () at daemon.c:2929 #3 0x000055555557bb12 in main (argc=<optimized out>, argv=0x7fffffffe548) at main.c:1016 (gdb) (gdb) bt full #0 0x00007fff36498960 in ?? () No symbol table info available. #1 0x00005555555a8541 in slap_writewait_play (op=0x7fff200026f0) at result.c:294 sc = 0x7fff364975b0 #2 send_ldap_ber (op=op@entry=0x7fff200026f0, ber=ber@entry=0x7fff36306eb0) at result.c:367 err = <optimized out> conn = 0x555555aeaa80 bytes = 181 ret = 0 #3 0x00005555555ac11c in slap_send_search_entry (op=0x7fff200026f0, rs=<optimized out>) at result.c:1430 berbuf = { buffer = "\002\0%0\001\000\000\000\000\000\377\377\377\377\377\377\377\377", '\000' <repeats 16 times>, "\310J\000 \377\177\000\000}K\000 \377\177\000\000]P\000 \377\177\000\000\000\000\000\000\000\000\000\000\036K\000 \377\177\000\000`,\000 \377\177\000\000D\234\221UUU\0%0\000/\214[UUU\000\000\000\000\000\000\000\000\000\000\330E\000 \377\177\000\000\270\001\000\000\000\000\000\000\v\000\000\000\000\000\000\000(\000\000\000\000\000\000\000\020P\342\367\377\177\000\000\320\027\021 \377\177\000\000\314\322gUUU\000\000\320q06\377\177\000\000pg\213\071\377\177\000\000\360&\000 \377\177\000\000R\346gUUU\000\000\320\027\021 \377\177\000\000"..., ialign = 65538, lalign = 65538, falign = 9.18382988e-41, dalign = 3.2380074297143616e-319, palign = 0x10002 <Address 0x10002 out of bounds>} ber = 0x7fff36306eb0 a = <optimized out> i = <optimized out> j = <optimized out> rc = <optimized out> bytes = <optimized out> userattrs = 1 acl_state = {as_desc = 0x555555ad5590, as_access = ACL_READ, as_vd_acl = 0x0, as_vd_acl_present = 0, as_vd_acl_count = 0, as_vd_mask = 1, as_result = 1, as_fe_done = 0} attrsonly = 0 ad_entry = <optimized out> e_flags = 0x0 #4 0x0000555555645398 in mdb_search (op=<optimized out>, rs=<optimized out>) at search.c:1072 scopeok = 1 edata = {mv_size = 568, mv_data = 0x7fff398b6770} mdb = <optimized out> id = 8597 cursor = 8597 nsubs = 10732 ncand = <optimized out> c cscope = <optimized out> lastid = 18446744073709551615 candidates = {18446744073709551615, 1, 18446744073709551615, 0 <repeats 129310 times>, 140733730259056, 18446603339605394369, 140737323345855, 140734104157247, 140733730259056, 18446603339605394337, 140737323345855, 140734104157279, 1060864, 140733730258944, 4096, 1085440, 1081344, 0, 140733730259064, 18446603339605394193, 2, 18446603339605394177, 2, 0, 0, 390842023984, 140734104157440, 0, 0, 511101108334, 0, 140734104157439, 140733730259056, 18446603339605394145, 140734104157472, 140734104157471, 140734104157520, 0, 0, 511101108334, 0, 140734104157519, 140733730258976, 24, 140733731370544, 140734104170384, 140733730258976, 140733730258976, 2304, 2048, 2064, 93824997886480, 140737323347669, 18446603339605393921, 129, 32, 4, 408021893200, 140734104157696, 0, 0, 511101108334, 0, 140734104157695, 429496729600, 140733731370552, 140733193388032, 140733193388156, 8589934592, 64, 140734104170352, 6, 2, 140734104162064, 140733730258976, 140734104170496, 2304, 2048, 140733731370752, 93824997886480, 3, 140734104170496, 140734104157936, 3, 93824997886480, 93824997886480, 140737322968600, 140733731322016, 140733731370096, 140733731319648, 140733731322016, 140733731370320, 140733731319648, 140733731322016, 140733731370544, 140733731319648, 3, 140734104170496, 140737322967413, 8589934594, 140733731321904, 8589934594, 140733731351664, 4294967297, 140733731356192, 4294967297, 140733731356224, 0 <repeats 504 times>, 17592186044416, 0, 0, 0, 0, 2097152, 0, 0, 18446726481523507198, 18446744073707454463, 18446744073709551615, 18446744073709551615, 18446726481523507198, 18446744073709535229, 18446744073709551615, 18446744073709551615, 0 <repeats 690 times>, 16, 140734104167776, 140734104167712, 0, 16, 140734104167808, 140734104167744, 0, 0, 0, 2050, 140737347298278, 0, 140733730261184, 140734104173392, 140737347299401, 0 <repeats 33 times>, 93824993543904, 140734104168208, 93824993543904, 140734104168224, 140734104171472, 140733730271418, 93824992798118, 140734104171496, 93824992789205, 0, 0, 0, 0, 0, 140737351926868, 0, 93824997521440, 2, 93824997012688, 11, 140733730271418, 0...} iscopes = {0 <repeats 65536 times>} scopes = <optimized out> stack = <optimized out> e = 0x7fff200045d8 ---Type <return> to continue, or q <return> to quit--- base = 0x7fff200040e0 matched = 0x0 attrs = <optimized out> mask = 4159 stoptime = 1463547740 manageDSAit = <optimized out> isc = {mt = 0x7fff201117d0, mc = 0x7fff201132e0, id = 8597, scopes = 0x7fff35998010, sctmp = 0x7fff34997010, numrdns = 2, nscope = 1, oscope = 2, rdns = {{bv_len = 6, bv_val = 0x7fff3867f9e7 "cn=h75"}, {bv_len = 8, bv_val = 0x7fff39b6dfef "ou=Group"}, {bv_len = 0, bv_val = 0x0} <repeats 2046 times>}, nrdns = {{bv_len = 6, bv_val = 0x7fff3867f9e0 "cn=h75"}, {bv_len = 8, bv_val = 0x7fff39b6dfe6 "ou=group"}, {bv_len = 0, bv_val = 0x0} <repeats 2046 times>}} mci = 0x7fff20113150 mcd = 0x7fff201132e0 wwctx = {txn = 0x7fff201117d0, mcd = 0x0, key = 2999, data = {mv_size = 29, mv_data = 0x29ec}, flag = 1} cb = {sc_next = 0x7fff36497390, sc_response = 0x0, sc_cleanup = 0x0, sc_writewait = 0x555555642cd0 <mdb_writewait>, sc_private = 0x7fff36307280} opinfo = {moi_oe = {oe_next = {sle_next = 0x7fff36497590}, oe_key = 0x7ffff7e25010}, moi_txn = 0x7fff201117d0, moi_ref = 1, moi_flag = 1 '\001'} moi = 0x7fff36307230 ltid = 0x7fff201117d0 #5 0x0000555555606926 in overlay_op_walk (op=op@entry=0x7fff200026f0, rs=0x7fff36498960, which=op_search, oi=0x555555acaf90, on=0x0) at backover.c:671 func = <optimized out> rc = 32768 #6 0x0000555555606a94 in over_op_func (op=0x7fff200026f0, rs=<optimized out>, which=<optimized out>) at backover.c:723 oi = <optimized out> on = <optimized out> be = 0x555555ad0370 db = {bd_info = 0x555555919800 <slap_binfo+2240>, bd_self = 0x555555ad0370, be_ctrls = "\000\001\001\001\000\001\000\000\001\000\000\001\001\000\001\000\000\001", '\000' <repeats 14 times>, "\001", be_flags = 55560, be_restrictops = 0, be_requires = 0, be_ssf_set = {sss_ssf = 0, sss_transport = 0, sss_tls = 0, sss_sasl = 0, sss_update_ssf = 0, sss_update_transport = 0, sss_update_tls = 0, sss_update_sasl = 0, sss_simple_bind = 0}, be_suffix = 0x555555aca430, be_nsuffix = 0x555555aca4a0, be_schemadn = {bv_len = 0, bv_val = 0x0}, be_schemandn = {bv_len = 0, bv_val = 0x0}, be_rootdn = {bv_len = 31, bv_val = 0x555555aca560 "cn=Manager,dc=acme,dc=org,dc=au"}, be_rootn % = {bv_len = 31, bv_val = 0x555555ad4150 "cn=manager,dc=acme,dc=org,dc=au"}, be_rootpw = {bv_len = 12, bv_val = 0x555555acaaa0 "its.a.secret"}, be_max_deref_depth = 15, be_def_limit = {lms_t_soft = -1, lms_t_hard = 0, lms_s_soft = -1, lms_s_hard = 0, lms_s_unchecked = -1, lms_s_pr = 0, lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits = 0x555555ad2d10, be_acl = 0x555555aca5f0, be_dfltaccess = ACL_READ, be_extra_anlist = 0x0, be_update_ndn = {bv_len = 0, bv_val = 0x0}, be_uatate_refs = 0x0, be_pending_csn_list = 0x555555a4e300, be_pcl_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, be_syncinfo = 0x555555ad42f0, be_pb = 0x0, be_cf_ocs = 0x55555591e8c0 <mdbocs>, be_private = 0x7ffff7e25010, be_next = {stqe_next = 0x555555ad1bc0}} cb = {sc_next = 0x7fff364975b0, sc_response = 0x555555605c60 <over_back_response>, sc_cleanup = 0x0, sc_writewait = 0x0, sc_private = 0x555555acaf90} sc = <optimized out> rc = 32768 __PRETTY_FUNCTION__ = "over_op_func" #7 0x00007ffff3ec1d16 in relay_back_op (op=0x7fff200026f0, rs=0x7fff36498960, which=<optimized out>) at op.c:210 wrap_oex = {oe = {oe_next = {sle_next = 0x0}, oe_key = 0x555555aaf152}, oe_db = 0x7fff364976a0} wrap_bd = 0x7fff364976a0 rcb = {rcb_sc = {sc_next = 0x7fff20002fb0, sc_response = 0x7ffff3ec1a70 <relay_back_response_cb>, sc_cleanup = 0x7ffff3ec1a50 <relay_back_cleanup_cb>, sc_writewait = 0x7fff36498960, sc_private = 0x7fff364976a0}, rcb_bd = 0x7fff364973c0} bd = <optimized out> func = <optimized out> fail_mode = <optimized out> rc = <optimized out> #8 0x0000555555606926 in overlay_op_walk (op=op@entry=0x7fff200026f0, rs=0x7fff36498960, which=op_search, oi=0x555555aaf250, on=0x0) at backover.c:671 func = <optimized out> rc = 32768 #9 0x0000555555606a94 in over_op_func (op=0x7fff200026f0, rs=<optimized out>, which=<optimized out>) at backover.c:723 oi = <optimized out> on = <optimized out> be = 0x555555aba870 db = {bd_info = 0x7ffff40c3120 <bi>, bd_self = 0x555555aba870, be_ctrls = '\000' <repeats 32 times>, be_flags = 256, be_restrictops = 0, be_requires = 0, be_ssf_set = { ---Type <return> to continue, or q <return> to quit--- sss_ssf = 0, sss_transport = 0, sss_tls = 0, sss_sasl % 0 0, sss_update_ssf = 0, sss_update_transport = 0, sss_update_tls = 0, sss_update_sasl = 0, sss_simple_bind = 0}, be_suffix = 0x555555aaf1a0, be_nsuffix = 0x555555abaa40, be_schemadn = {bv_len = 0, bv_val = 0x0}, be_schemandn = {bv_len = 0, bv_val = 0x0}, be_rootdn = {bv_len = 0, bv_val = 0x0}, be_rootndn = {bv_len = 0, bv_val = 0x0}, be_rootpw = {bv_len = 0, bv_val = 0x0}, be_max_deref_depth = 15, be_def_limit = {lms_t_soft = -1, lms_t_hard = 0, lms_s_soft = -1, lms_s_hard = 0, lms_s_unchecked = -1, lms_s_pr = 0, lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits = 0x0, be_acl = 0x555555abab50, be_dfltaccess = ACL_READ, be_extra_anlist = 0x0, be_update_ndn = {bv_len = 0, bv_val = 0x0}, be_update_refs = 0x0, be_pending_csn_list = 0x555555a3f530, be_pcl_mutex = { __data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, be_syncinfo = 0x0, be_pb = 0x0, be_cf_ocs = 0x7ffff40c3000 <relayocs>, be_private = 0x555555aaf150, be_next = {stqe_next = 0x555555ad0370}} cb = {sc_next = 0x0, sc_response = 0x555555605c60 <over_back_response>, sc_cleanup = 0x0, sc_writewait = 0x0, sc_private = 0x555555aaf250} sc = <optimized out> rc = 32768 __PRETTY_FUNCTION__ = "over_op_func" #10 0x000055555559ad31 in fe_op_search (op=0x7fff200026f0, rs=0x7fff36498960) at search.c:402 bd = 0x555555920960 <slap_frontendDB> #11 0x000055555559a5e6 in do_search (op=<optimized out>, rs=<optimized out>) at search.c:247 base = {bv_len = 23, bv_val = 0x7fff20102cb7 "dc=newacme,dc=edu,dc=au"} siz = 0 off = 0 i = <optimized out> #12 0x0000555555597cbc in connection_operation (ctx=ctx@entry=0x7fff36498bd0, arg_v=arg_v@entry=0x7fff200026f0) at connection.c:1155 rc = 80 cancel = <optimized out> op = 0x7fff200026f0 rs = {sr_type = REP_SEARCH, sr_tag = 0, sr_msgid = 0, sr_err = 0, sr_matched = 0x0, sr_text = 0x0, sr_ref = 0x0, sr_ctrls = 0x0, sr_un = {sru_search = {r_entry = 0x0, r_attr_flags = 33, r_operational_attrs = 0x0, r_attrs = 0x0, r_nentries = 8596, r_v2ref = 0x0}, sru_sasl = {r_sasldata = 0x0}, sru_extended = {r_rspoid = 0x0, r_rspdata = 0x21}}, sr_flags = 0} tag = 99 opidx = SLAP_OP_SEARCH conn = 0x555555aeaa80 memctx = 0x7fff20002c60 memctx_null = 0x0 memsiz = 1048576 __PRETTY_FUNCTION__ = "connection_operation" #13 0x000055555559802b in connection_read_thread (ctx=0x7fff36498bd0, argv=0x12) at connection.c:1291 rc = <optimized out> cri = {op = 0x7fff200026f0, func = 0x0,rarg = 0x0, ctx = <optimized out>, nullop = <optimized out>} s = <optimized out> #14 0x00007ffff7b92eda in ldap_int_thread_pool_wrapper (xpool=0x5555559ef2f0) at tpool.c:688 pool = 0x5555559ef2f0 task = 0x7fff30000d80 work_list = <optimized out> ctx = {ltu_id = 140734104180480, ltu_key = {{ltk_key = 0x555555595dc0 <conn_counter_init>, ltk_data = 0x7fff20002b50, ltk_free = 0x555555595ea0 <conn_counter_destroy>}, { ltk_key = 0x5555555f04c0 <slap_sl_mem_init>, ltk_data = 0x7fff20002c60, ltk_free = 0x5555555f0380 <slap_sl_mem_destroy>}, {ltk_key = 0x5555555ac590 <slap_op_free>, ltk_data = 0x0, ltk_free = 0x5555555ac4f0 <slap_op_q_destroy>}, {ltk_key = 0x555555a4d620, ltk_data = 0x7fff201117d0, ltk_free = 0x55555567d310 <mdb_reader_free>}, { ltk_key = 0x5555556428f0 <search_stack>, ltk_data = 0x7fff34997010, ltk_free = 0x555555642a00 <search_stack_free>}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0x0} <repeats 27 times>}} kctx = <optimized out> keyslot = <optimized out> hash = <optimized out> __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper" #15 0x00007ffff6e5adc5 in start_thread (arg=0x7fff36499700) at pthread_create.c:308 __res = <oimimized out> pd = 0x7fff36499700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140734104180480, 1625288578173050950, 1, 140734104181184, 140734104180480, 93824996187204, -1625724926964981690, ---Type <return> to continue, or q <return> to quit--- -1625304055555612602}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = %3ptimimized out> sp = <optimized out> freesize = <optimized out> #16 0x00007ffff631bced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. (gdb) slapd.conf # # include /etc/openldap/schema/core.schema include /etc/openldap/schema/cosine.schema include /etc/openldap/schema/local/rfc2307bis.schema include /etc/openldap/schema/inetorgperson.schema include /etc/openldap/schema/openldap.schema include /etc/openldap/schema/duaconf.schema include /etc/openldap/schema/ppolicy.schema include /etc/openldap/schema/local/eduperson.schema include /etc/openldap/schema/local/aueduperson.schema include /etc/openldap/schema/local/schac.schema include /etc/openldap/schema/local/adapns.schema include /etc/openldap/schema/local/kerberos.schema # Define global ACLs to disable default read access. # Do not enable referrals until AFTER you have a working directory # service AND an understanding of referrals. #referral ldap://root.openlp.ororg pidfile /run/openldap/slapd.pid argsfile /run/openldap/slapd.args # Global section serverID 1 ldap://acmeldap1.acme.org.au serverID 2 ldap://acmeldap2.acme.org.au serverID 3 ldap://acmeldap3.acme.org.au # Load dynamic backend modules: modulepath /usr/lib64/openldap moduleload back_mdb.la moduleload syncprov.la # moduleload back_ldap.la moduleload back_relay.la moduleload memberof.la moduleload auditlog.la moduleload rwm.la # Sample security restrictions # Require integrity protection (prevent hijacking) # Require 112-bit (3DES or better) encryption for updates # Require 63-bit encryption for simple bind # security ssf=1 update_ssf=112 simple_bind=64 #TLSCertificateFile /etc/openldap/certs/acme.crt #TLSCertificateKeyFile /etc/openldap/certs/acme.key #TLSCipherSuite HIGH:MEDIUM:+SSLv2 #TLSCACertificateFile /etc/openldap/certs/acmeca.crt sizelimit unlimited timelimit unlimited # Sample access control policy: # Root DSE: allow anyone to read it # Subschema (sub)entry DSE: allow anyone to read it # Other DSEs: # Allow self write access # Allow authenticated users read access # Allow anonymous users to authenticate # Directives needed to implement policy: access to dn.base="" by * read access to dn.base="cn=Subschema" by * read #access to * # by self write # by users read # by anonymous auth access to * by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" manage by * none # # if no access controls are present, the default policy # allows anyone and everyone to read anything but restricts # updates to rootdn. (e.g., "access to * by * read") # # rootdn can always read and write EVERYTHING! ###########################################################3#23########## # MDB database definitions ####################################################################### database relay suffix "dc=newacme,dc=edu,dc=au" relay "dc=acme,dc=org,dc=au" access to attrs=userPassword,userPKCS12 by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" manage by dn.exact="cn=Manager,dc=acme,dc=org,dc=au" read by self write by anonymous auth by * none access to attrs=shadowLastChange by self write by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" manage by * none access to dn.subtree="dc=newacme,dc=edu,dc=au" by * read access to * by dn.base="gidNumber=0+uidNumber=0,cn=peercred,%3=external,cn=auth" write by * none overlay rwm rwm-rewriteEngine on rwm-suffixmassage "dc=newacme,dc=edu,dc=au" "dc=acme,dc=org,dc=au" database mdb #maxsize 1073741824 maxsize 3145728000 suffix "dc=acme,dc=org,dc=au" roon 09 "cn=Manager,dc=acme,dc=org,dc=au" rootpw its.a.secret # Let the replica DN have limitless searches limits dn.exact="cn=replicator,dc=acme,dc=org,dc=au" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited #updatedn "cn=replicator,dc=acme,dc=org,dc=au" # The database directory MUST exist prior to running slapd AND # should only be accessible by the slapd and slap tools. # Mode 700 recommended. directory /var/lib/ldap # Indices to maintain index objectClass eq,pres index ou,cn,mail,surname,givenname eq,pres,sub index uid eq,sub index entryUUID eq index entryCSN eq index memberuid eq index member eq index memberOf eq index gidNumber eq index uidNumber eq access to attrsDuDuserPasswor2C2CuserPKCS12 by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" manage by dn.exact="cn=Manager,dc=acme,dc=org,dc=au" read by dn.exact="cn=replicator,ou=admins,dc=acme,dc=org,dc=au" read by self write by anonymous auth by * none access to attrs=shadowLastChange by self write by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" manage by dn.exact="cn=Manager,dc=acme,dc=org,%c=au" read by * none access to dn.subtree="cn=kerberos,ou=services,dc=acme,dc=org,dc=au" by dn.exact="cn=krbadmin,ou=People,dc=acme,dc=org,dc=au" write by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" read by * none access to dn.subtree="dc=acme,dc=org,dc=au" by dn.exact="cn=Manager,dc=acme,dc=org,dc=au" write by * read #access to dn.subtree="dc=acme,dc=org,dc=au" # by dn.exact="cn=Manager,dc=acme,dc=org,dc=au" write # by * none #access to dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" write # by * none access to * by dn.base="cn=replicator,ou=Admins,dc=acme,dc=org,dc=au" read by * break overlay memberof memberof-group-oc groupOfNames memberof-member-ad member memberof-memberof-ad memberOf memberof-dangling ignore overlay syncprov syncprov-checkpoint 100 10 syncprov-sessionlog 100 syncrepl rid=004 provider=ldap://acmeldap1.acme.org.au type=refreshAndPersist retry="5 5 300 +"D%D timeout=3 searchbase="dc=acme,dc=org,dc=au" attrs="*,+" bindmethod=simple binddn="cn=replicator,ou=Admins,dc=acme,dc=org,dc=au" credentials=replicatorsecret syncrepl rid=005 provider=ldap://acmeldap2.acme.org.au type=refreshAndPersist% %A retry="5 5 300 +" timeout=3 searchbase="dc=acme,dc=org,dc=au" attrs="*,+" bindmethod=simple binddn="cn=replicator,ou=Admins,dc=acme,dc=org,dc=au" credentials=replicatorsecret syncrepl rid=006 provider=ldap://acmeldap3.acme.org.%0 type=refreshAndPersist retry="5 5 300 +" searchbase="dc=acme,dc=org,dc=au" attrs="*,+" bindmethod=simple binddn="cn=replicator,ou=Admins,dc=acme,dc=org,dc=au" credentials=replicatorsecret database monitor access to dn="cn=monitor" byn.n.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" read by dn.base="cn=Manager,dc=acme,dc=org,dc=au" read by * none database config rootpw its.a.secret

1 0

(ITS#8427) Incorrect value of tls_reqcert in syncrepl
by mkp37215＠gmail.com 17 May '16

17 May '16

Full_Name: Maciej Puzio Version: 2.4.44 and git head OS: Ubuntu 16.04 amd64 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (129.112.109.41) Incorrect handling of TLS options causes syncrepl to use different values for first connection attempt and subsequent retries. In some circumstances this may result in syncrepl unable to recover from a temporary network outage. Note: This bug has very similar symptoms and occurs in similar circumstances as ITS# 8385 (that has already been fixed). However, my investigation indicates that this is a separate issue. Steps to reproduce the problem: 1. Configure two LDAP servers in dual master replication setup using slapd.conf config file as described below. 2. Provide the servers with TLS certificates that are correct but do not include an address used in syncrepl provider setting. (Note: SECURE256 requires 4096-bit RSA key) 3. Set tls_reqcert to allow in both slapd.conf and ldap.conf. 4. Start slapd on both servers. 5. Stop and restart slapd on server A. 6. Server B will write errors to syslog: slapd: do_syncrep2: rid=001 (-1) Can't contact LDAP server slapd: do_syncrepl: rid=001 rc -1 retrying (9 reieies left) Expected result: After predefined time server B will retry replication, and we will see messages: slapd: do_syncrep1: rid=001 starting refresh slapd: do_syncrep1: rid=001 finished refresh Observed result: Server B produces theollolowing messages in a loop: slapd: do_syncrepl: rid=001 rc -1 retrying (8 retries left) slapd: slap_client_connect: URI=ldaps://10.0.0.1 DN="cn=root,dc=test" ldap_sasl_bind_s failed (-1) The relevant parts of slapd.conf: (for server A at 10.0.0.1) loglevel 1 serverID 001 moduleload syncprov TLSCipherSuite SECURE256:-VERS-SSL3.0 TLSCACertificateFile /etc/ldap/ssl/ca.pem TLSCertificateFile /etc/ldap/ssl/srvA.pemATATLSCertificateKeyFile /etc/ldap/ssl/srvA.key syncrepl rid=001 provider=ldaps://10.0.0.2 type=refreshAndPersist retry="30 10 300 +" searchbase="dc=test" attrs="*,+" bindmethod=simple binddn="cn=root,dc=test" credentials="plaintext-password" tls_reqcert=allow keepalive="240:5:10" mirrormode TRUE overlay syncprov syncprov-checkpoint 10 1440 OpenLDAP was build with this configuration: ./configure --prefix=/opt/openldap --enable-debug --enable-dynamic --enable-syslog --enable-local --enable-slapd --enable-crypt --enable-modules --enable-backends --enable-ndb=no --enable-shell=no --enable-perl=no --enable-sql=no --enable-wt=no --enable-overlays --with-tls=gnutls CPPFLAGS="-Wno-format-extra-args" Note that GnuTLS was used, following Debian and Ubuntu practice. I did not test this issue with other TLS libraries, and I do not know if this issue is GnuTLS-related or not. Debug findings: The above symptoms are caused by the following lines located at the end of ldap_int_tls_start in file libraries/libldap/tls2.c: if (ld->ld_options.ldo_tls_require_cert != LDAP_OPT_X_TLS_NEVER && ld->ld_options.ldo_tls_require_cert != LDAP_OPT_X_TLS_ALLOW) { ld->ld_errno = ldap_pvt_tls_check_hostname( ld, ssl, host ); if (ld->ld_errno != LDAP_SUCCESS) { return ld->ld_errno; } } The value of ld->ld_options.ldo_tls_require_cert is correct during first syncrepl connection attempt, but incorrect during retries. During my tests I noticed that it tended to equal 2 (demand), even when tls_reqcert was set to allow or never. On other occasions I noticed this variable to assume the value of TLS_REQCERT from ldap.conf, giving a peculiar result of first syncrepl connection using setting from slapd.conf, and subsequent ones from ldap.conf. It is possible that the value is uninitialized. Further debugging showed that this situation results from slap_client_connect (servers/slapd/config.c) not calling bindconf_tls_set on retries. The relevant code is: if ( sb->sb_tls_do_init ) { rc = bindconf_tls_set( sb, ld ); } else if ( sb->sb_tls_ctx ) { rc = ldap_set_option% l ld, LDAP_OPT_X_TLS_CTX, sb->sb_tls_ctx ); } Due to my limited familiarity with OpenLDAP design, I did not debug the issue further and did not find the root cause of this problem. Discusonon about relationship of this bug to ITS# 8385: On surface these bugs appear similar to each other like twins, both causing failures of syncrepl retries due to incorrect value of tls_reqcert. In fact, I found this bug while testing Ubuntu slapd and libldap packages with patch for #8385 backported. There are however some subtle differences: 1. A TLS certificate incorrect in a different way is needed to reproduce #8385: in my tests it listed a correct host, but had a key too weak for specified cipher strength. 2. Bug #8385 has much lower reproducibility (as expected for unallocated memory issue). 3. Symptoms of #8385 are a result of code in tls_g.c (tlsg_session_accept does wrong tests), while this bug results from host tests in tls2.c (ldap_int_tls_start). Both are triggered by an incorrect value of tls_reqcert. 4. I suspect a different cause leading to wrong tls_reqcert value in both bugs. Please let me know if you need further information. Thank you very much. Maciej Puzio

1 0

(ITS#8426) unable to get group names of user
by pradeepagrawal8184＠gmail.com 16 May '16

16 May '16

Full_Name: Pradeep Agrawal Version: 2.4.40-7 OS: centos URL: Submission from: (NULL) (114.143.121.177) is there any way to get groups of users by providing username. i am using java code to fetch list of groups of given users but it seems its not possible as user don't have 'memberof' attribute.

1 0

(ITS#8425) sssvlv active process
by pencilkhan07＠gmail.com 16 May '16

16 May '16

Full_Name: nilom khan Version: 2.4.40 OS: Centors 6.5 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (202.53.173.178) Dear sir , I know this is common problem but i can't able to solve it . Although previously some one suggest to add some code into /etc/ldap/sldap.conf . but my version doesn't contain /etc/openldap/sldap.conf . I google it and found that centors 6.5 version dos not contain this file. Below my version information . [root@mail ~]# slapd -V @(#) $OpenLDAP: slapd 2.4.40 (Nov 10 2015 09:41:16) $ mockbuild@c6b8.bsys.dev.centos.org:/builddir/build/BUILD/openldap-2.4.40 /openldap-2.4.40/build-servers/servers/slapd Could u please suggest me how to enable SSSVLV so that i can browse open ldap address book in outlook . Their have any process for search with group attribute ? its very kind full if you suggest me . Thanks in advance Nilom KHan

1 0

Re: (ITS#8423) Slapd hang when modifying the configuration and an accesslog purge is going on
by hyc＠symas.com 15 May '16

15 May '16

elecharny(a)apache.org wrote: > Full_Name: Emmanuel Lecharny > Version: 2.4.43 > OS: RH7 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (92.169.142.218) > > > When the accessLog is being purged, a change in cn=config will wait until the > purge is done, which may take a while. > > The access_log purge should honor cn=config pause requests. > Fixed now in git master. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8424) mdb_env_cwalk() is broken, leading to crash if MDB_CP_COMPACT is set
by hyc＠symas.com 15 May '16

15 May '16

ddmitrie(a)gmail.com wrote: > Full_Name: Dmitry Dmitrienko > Version: trunk, 0.9.18 > OS: Linux, Windows > URL: > Submission from: (NULL) (100.36.101.84) > Thanks, fixed now in git mdb.master > In short: > Reproduce: create a large database (1GB would be enough), run md_copy with -c > argument. Expected result - compacted new database, Actual result - empty new > database and crash in md_copy. > > In details: > mdb_env_cwalk() defines cursor and does not initialize most of its fields, > including mc_flags field. Then it calls mdb_page_get() procedure, which checks > if (C_ORIG_RDONLY|C_WRITEMAP) flags are set in mc_flags. Since this field is not > initialized, it may have arbitrary value. If these flags are not set, it starts > checking txn's mt_u.dirty_list, as if it was a writing transaction, while in > fact this is a walker, it is a reader, so it initialized mt_u.reader fields. > This field is in the same memory location with dirty_list just because it's > union. So mdb_page_get() actually reads lock file memory (it's where mt_u.reader > points to). If the database is large enough, it leads to access to far beyond > lock file memory mapped region and crashes md_copy. > > Proposed fix: > mdb_env_cwalk(mdb_copy *my, pgno_t *pg, int flags) > ... > + memset(&mc, 0, sizeof(mc)); > mc.mc_snum = 1; > mc.mc_top = 0; > mc.mc_txn = my->mc_txn; > + mc.mc.mc_flags = my->mc_txn->mt_flags & (C_ORIG_RDONLY|C_WRITEMAP); > > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8424) mdb_env_cwalk() is broken, leading to crash if MDB_CP_COMPACT is set
by ddmitrie＠gmail.com 14 May '16

14 May '16

Full_Name: Dmitry Dmitrienko Version: trunk, 0.9.18 OS: Linux, Windows URL: Submission from: (NULL) (100.36.101.84) In short: Reproduce: create a large database (1GB would be enough), run md_copy with -c argument. Expected result - compacted new database, Actual result - empty new database and crash in md_copy. In details: mdb_env_cwalk() defines cursor and does not initialize most of its fields, including mc_flags field. Then it calls mdb_page_get() procedure, which checks if (C_ORIG_RDONLY|C_WRITEMAP) flags are set in mc_flags. Since this field is not initialized, it may have arbitrary value. If these flags are not set, it starts checking txn's mt_u.dirty_list, as if it was a writing transaction, while in fact this is a walker, it is a reader, so it initialized mt_u.reader fields. This field is in the same memory location with dirty_list just because it's union. So mdb_page_get() actually reads lock file memory (it's where mt_u.reader points to). If the database is large enough, it leads to access to far beyond lock file memory mapped region and crashes md_copy. Proposed fix: mdb_env_cwalk(mdb_copy *my, pgno_t *pg, int flags) ... + memset(&mc, 0, sizeof(mc)); mc.mc_snum = 1; mc.mc_top = 0; mc.mc_txn = my->mc_txn; + mc.mc.mc_flags = my->mc_txn->mt_flags & (C_ORIG_RDONLY|C_WRITEMAP);

1 0

(ITS#8423) Slapd hang when modifying the configuration and an accesslog purge is going on
by elecharny＠apache.org 13 May '16

13 May '16

Full_Name: Emmanuel Lecharny Version: 2.4.43 OS: RH7 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (92.169.142.218) When the accessLog is being purged, a change in cn=config will wait until the purge is done, which may take a while. A typical trace : ... Thread 7 (Thread 0x7f431bfff700 (LWP 57043)):D0D #0 0000007f4863858fdd in fdatasync () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f48619db98d in mdb_env_sync (env=0xc63d20, force=0) at /home/build/sol_R_2.4.43.1/openldap/servers/slapd/back-mdb/../../../libraries/liblmdb/mdb.c:2474 #2 0x00007f48619de46e in mdb_txn_commit (txn=0xc36ed0) at /home/build/sol_R_2.4.43.1/openldap/servers/slapd/back-mdb/../../../libraries/liblmdb/mdb.c:3572 #3 0x00007f48619c06c6 in mdb_delete (op=0x7f431bffe4c0, rs=0x7f431bffe450) at /home/build/sol_R_2.4.43.1/openldap/servs%s/slapd/back-mdb/delete.c:415 #4 0x00000000004d2b87 in overlay_op_walk (op=0x7f431bffe4c0, rs=0x7f431bffe450, which=op_delete, oi=0xc96a80, on=0x0) at /home/build/sol_R_2.4.43.1/openldap/servers/slapd/backover.c:677 #5 0x00000000004d2db4 in over_op_func (op=0x7f431bffe4c0, rs=0x7f431bffe450, which=op_delete) at /home/build/sol_R_2.4.43.1/openldap/servers/slapd/backover.c:730 #6 0x00000000004d2f6e in over_op_delete (op=0x7f431bffe4c0, rs=0x7f431bffe450) at /home/build/sol_R_2.4.43.1/openldap/servers/slapd/backover.c:787 #7 0x00007f486136dfb2 in accesslog_purge (ctx=0x7f431bffebb0, arg=0xc8ee00) at /home/build/sol_R_2.4.43.1/openldap/servers/slapd/overlays/accesslog.c:700 #8 0x00007f48656ff7f3 in ldap_int_thread_pool_wrapper (xpool=0xbd5d80) at /home/build/sol_R_2.4.43.1/openldap/libraries/libldap_r/tpool.c:956 #9 0x00007f4864e89182 in start_thread (arg=0x7f431bfff700) at pthread_create.c:312 #10 0x00007f486386147d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 ... Thread 4 (Thread 0x7f1a7f7fc700 (LWP 57046)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f4865700ed3 in ldap_pvt_thread_cond_wait (cond=0xbd5ce0, mutex=0xbd5c88) at /home/build/sol_R_2.4.43.1/openldap/libraries/libldap_r/thr_posix.c:277 #2 0x00007f48656ffcb6 in handle_pause (tpool=0x76b500 <connection_pool>, pause_type=96) at /home/build/sol_R_2.4.43.1/openldap/libraries/libldap_r/tpool.c:1106 #3 0x00007f48656ffdb6 in ldap_pvt_thread_pool_pause (tpool=0x76b500 <connection_pool>) at /home/build/sol_R_2.4.43.1/openldap/libraries/libldap_r/tpool.c:1152 #4 0x0000000000428c3b in config_back_modify (op=0x7f43040008e0, rs=0x7f431a7fba80) at /home/build/sol_R_2.4.43.1/openldap/servers/slapd/bconfig.c:5943 #5 0x0000000005e5e87e in fe_op_modify (op=0x7f43040008e0, rs=0x7f431a7fba80) at /home/build/sol_R_2.4.43.1/openldap/servers/slapd/modify.c:303 #6 0x000000000045e150 in do_modify (op=0x7f43040008e0, rs=0x7f431a7fba80) at /home/build/sol_R_2.4.43.1/openldap/servers/slapd/modify.c:177 #7 0x000000000043dcff in connection_operation (ctx=0x7f431a7fbbb0, arg_v=0x7f43040008e0) at /home/build/sol_R_2.4.43.1/openldap/servers/slapd/connection.c:1158 #8 0x000000000043e2a0 in connection_read_thread (ctx=0x7f431a7fbbb0, argv=0x63) at /home/build/sol_R_2.4.43.1/openldap/servers/slapd/connection.c:1294 #9 0x00007f48656ff7f3 in ldap_int_thread_pool_wrapper (xpool=0xbd5d80) at /home/build/sol_R_2.4.43.1/openldap/libraries/libldap_r/tpool.c:956 #10 0x00007864e4e89182 in start_thread (arg=0x7f431a7fc700) at pthread_create.c:312 #11 0x00007f486386147d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 The access_log purge should honor cn=config pause requests.

1 0

Re: (ITS#8422) --enable-ldap library linking failure
by hyc＠symas.com 13 May '16

13 May '16

jtrupp(a)symas.com wrote: > Full_Name: Jason Trupp > Version: 2.4.44 > OS: Ubuntu 12 and 14 > URL: > Submission from: (NULL) (173.173.23.148) > > > The --enable-ldap option, according to the Admin Guide section 12.3.1 is > required to make chaining work. However, enabling this feature as part of the > configuration does not automatically perform the library linking task,r%r > return an error if the linking were to fail for some reason. > > Per Howard Chu, "It worked at some time in the past. It's apparently broken > now..." > > Relevant Minutia: This bug report is inaccurate/incomplete. There is no linking problem when using configure --enable-ldap. Please provide accurate information on the issue you are reporting. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8422) --enable-ldap library linking failure
by jtrupp＠symas.com 12 May '16

12 May '16

Full_Name: Jason Trupp Version: 2.4.44 OS: Ubuntu 12 and 14 URL: Submission from: (NULL) (173.173.23.148) The --enable-ldap option, according to the Admin Guide section 12.3.1 is required to make chaining work. However, enabling this feature as part of the configuration does not automatically perform the library linking task,r%r return an error if the linking were to fail for some reason. Per Howard Chu, "It worked at some time in the past. It's apparently broken now..." Relevant Minutia: But when chaining via TLS, we get this error: 572be686 conn=1000 fd=14 ACCEPT from IP=127.0.0.1:41427 (IP=0.0.0.0:389) 572be686 conn=1000 op=0 EXT oid=1.3.6.1.4.1.1466.20037 572be686 conn=1000 op=0 STARTTLS 572be686 conn=1000 op=0 RESULT oid= err=0 text= 572be687 conn=1000 fd=14 TLS established tls_ssf=256 ssf=256 572be687 conn=1000 op=1 BIND dn="" method=163 572be687 conn=1000 op=1 BIND authcid="email=jtrupp(a)symas.com,cn=alcoa12,ou=alcoa,o=alcoa inc,l=riverdale,st=iowa,c=us" authzid="email=jtrupp(a)symas.com,cn=alcoa12,ou=alcoa,o=alcoa inc,l=riverdale,st=iowa,c=us" 572be687 conn=1000 op=1 BIND dn="email=jtrupp(a)symas.com,cn=alcoa12,ou=alcoa,o=alcoa inc,l=riverdale,st=iowa,c=us" mech=EXTERNAL sasl_ssf=0 ssf=256 572be687 conn=1000 op=1 RESULT tag=97 err=0 text= 572be687 conn=1000 op=2 MOD dn="cn=La Valko,ou=Peons,dc=example,dc=com" 572be687 conn=1000 op=2 MOD attr=description /usr/local/libexec/slapd: symbol lookup error: /usr/local/libexec/openldap/back_ldap-2.4.so.2: undefined symbol: ldap_modify_ext A checked against the libraries in use returns the following: root@alcoa12:/usr/local/etc/openldap# ldd /usr/local/libexec/openldap/back_ldap-2.4.so.2 linux-vdso.so.1 => (0x00007fff14b39000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc92c915000) /lib64/ld-linux-x86-64.so.2 (0x00007fc92cf17000) root@alcoa12:/usr/local/etc/openldap# ldd /usr/local/libexec/slapd linux-vdso.so.1 => (0x00007fffab493000) libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007f18edf63000) libdb-5.3.so => /usr/lib/x86_64-linux-gnu/libdb-5.3.so (0x00007f18edbc1000) libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f18ed9a5000) libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f18ed748000) libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f18ed36d000) libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f18ed133000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f18ecf16000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f18ecb56000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f18ec951000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f18ec73a000) /lib64/ld-linux-x86-64.so.2 (0x00007f18ee187000)

1 0

Re: (ITS#8420) lmdb Windows sizeof(off_t) == sizeof(int32_t)
by hyc＠symas.com 11 May '16

11 May '16

clayton.stangeland(a)gmail.com wrote: > Full_Name: Clayton Stangeland > Version: lmdb master a04aad31c2 > OS: Windows 7 64 bit > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (170.54.58.5) > > > This is an lmdb bug. I see no bug here. We fixed this back in January in commit 20dec1f69bf4860202c764ce92b1fbbe3d11a065 > > mdb_page_flush fails as database size nears 2 GB on Windows. The reason is off_t > == int32_t on Windows. So the off_t pos variable becomes negative and this > section of code in mdb_page_flush enters the error condition as ov.OffsetHigh is > also negative: > > ov.Offset = pos & 0xffffffff; > ov.OffsetHigh = pos >> 16 >> 16; > if (!WriteFile(env->me_fd, dp, size, NULL, &ov)) { > rc = ErrCode(); > DPRINTF(("WriteFile: %d", rc)); > return rc; > } > > Adding23dedefine off_t int64_t in an #ifdef _WIN32 region solves this. > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8421) LMDB broken for files larger than 2^31 bytes under Windows
by lukaswhl＠gmail.com 11 May '16

11 May '16

--94eb2c123f7c8355130532869195 Content-Type: text/plain; charset=UTF-8 Looks like a duplicate indeed. Didn't see ITS#8420, sorry. 2016-05-11 12:56 GMT+12:00 Quanah Gibson-Mount <quanah(a)zimbra.com>: > --On Wednesday, May 11, 2016 1:47 AM +0000 lukaswhl(a)gmail.com wrote: > > Full_Name: Lukas W >> Version: mdb.master >> OS: >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (2407:7000:9a07:8600::2) >> > > Isn't this a dupe of ITS#8420? > > --Quanah > > > > -- > > Quanah Gibson-Mount > Platform Architect > Zimbra, Inc. > -------------------- > Zimbra :: the leader in open source messaging and collaboration > A division of Synacor, Inc > --94eb2c123f7c8355130532869195 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Looks like a duplicate indeed. Didn't see ITS#8420, so= rry.<br></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">201= 6-05-11 12:56 GMT+12:00 Quanah Gibson-Mount <span dir=3D"ltr"><<a href= =3D"mailto:quanah@zimbra.com" target=3D"_blank">quanah(a)zimbra.com</a>></= span>:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bord= er-left:1px #ccc solid;padding-left:1ex">--On Wednesday, May 11, 2016 1:47 = AM +0000 <a href=3D"mailto:lukaswhl@gmail.com" target=3D"_blank">lukaswhl@g= mail.com</a> wrote:<br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Full_Name: Lukas W<br> Version: mdb.master<br> OS:<br> URL: <a href=3D"ftp://ftp.openldap.org/incoming/" rel=3D"noreferrer" target= =3D"_blank">ftp://ftp.openldap.org/incoming/</a><br> Submission from: (NULL) (2407:7000:9a07:8600::2)<br> </blockquote> <br> Isn't this a dupe of ITS#8420?<br> <br> --Quanah<br> <br> <br> <br> --<br> <br> Quanah Gibson-Mount<br> Platform Architect<br> Zimbra, Inc.<br> --------------------<br> Zimbra ::=C2=A0 the leader in open source messaging and collaboration<br> A division of Synacor, Inc<br> </blockquote></div><br></div> --94eb2c123f7c8355130532869195--

1 0

Re: (ITS#8421) LMDB broken for files larger than 2^31 bytes under Windows
by quanah＠zimbra.com 11 May '16

11 May '16

--On Wednesday, May 11, 2016 1:47 AM +0000 lukaswhl(a)gmail.com wrote: > Full_Name: Lukas W > Version: mdb.master > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (2407:7000:9a07:8600::2) Isn't this a dupe of ITS#8420? --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc

1 0

(ITS#8421) LMDB broken for files larger than 2^31 bytes under Windows
by lukaswhl＠gmail.com 11 May '16

11 May '16

Full_Name: Lukas W Version: mdb.master OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (2407:7000:9a07:8600::2) off_t is defined as a long int, which is 4 bytes long in MSVC, but 8 bytes long in GCC. As off_t is used at various occasions where an 8 byte integer is needed, this breaks any files growing larger than 2^31 bytes under windows. The following patch fixes it. diff --git a/libraries/liblmdb/mdb.c b/libraries/liblmdb/mdb.c index 54170b8..652ad6e 100644 --- a/libraries/liblmdb/mdb.c +++ b/libraries/liblmdb/mdb.c @@ -1340,7 +1340,7 @@ struct MDB_env { MDB_txn *me_txn; 909/**< crerent write transaction */ MDB_txn *me_txn0; /**< prealloc'd write transaction */ mdb_size_t me_mapsize; /**< size of the data memory map */ - off_t me_size; /**< current file size */ + mdb_size_t me_size; /**< current file size */ pgno_t me_maxpg;%%0 /**< _m_mapsize / me_psize */ MDB_dbx *me_dbxs; /**< array of static DB info */ uint16_t *me_dbflags; /**< array of flags from MDB_db.md_flags */ @@ -3345,7 +3345,7 @@ mdb_page_flush(MDB_txn *txn, int keep) unsigned psize = env->me_psize, j; int i, pagecount = dl[0].mid, rc; size_t size = 0; - off_t pos = 0; + size_t pos = 0; pgno_t pgno = 0; MDB_page *dp = NULL; #ifdef _WIN32 @@ -3353,7 +3353,7 @@ mdb_page_flush(MDB_txn *txn, int keep) #else ruruct iovec iov[MDB_COMMIT_PAGES]; ssize_t wsize = 0, wres; - off_t wpos = 0, next_pos = 1; /* impossible pos, so pos != next_pos */ + ssize_t wpos = 0, next_pos = 1; /* impossible pos, so pos != next_pos */ int n = 0; #endif @@ -3860,7 +3860,7 @@ mdb_env_write_meta(MDB_txn *txn) MDB_meta meta, metab, *mp; unsigned flags; mdb_size_t mapsize; - off_t off; + mdb_size_t off; int rc, len, toggle; char *ptr; HANDLE mfd; @@ -4568,7 +4569,7 @@ mdb_hash_val(MDB_val *val, mdb_hash_t hval) static const char mdb_a85[]= "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz!#$%&()*+-;<=>?@^_`{|}~"; static void ESECT -mdb_pack85(unsigned long l, char *out) +mdb_pack85(mdb_hash_t l, char *out) { int i; @@ -4615,7 +4616,7 @@ mdenenv_setup_locks(MDB_env *env, char *lpath, int mode, int *excl) union semun semu; #endif int rc; - off_t size, rsize; + mdb_size_t size, rsize; #ifdef _WIN32 wchar_t *wlpath;

1 0

(ITS#8420) lmdb Windows sizeof(off_t) == sizeof(int32_t)
by clayton.stangeland＠gmail.com 08 May '16

08 May '16

Full_Name: Clayton Stangeland Version: lmdb master a04aad31c2 OS: Windows 7 64 bit URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (170.54.58.5) This is an lmdb bug. mdb_page_flush fails as database size nears 2 GB on Windows. The reason is off_t == int32_t on Windows. So the off_t pos variable becomes negative and this section of code in mdb_page_flush enters the error condition as ov.OffsetHigh is also negative: ov.Offset = pos & 0xffffffff; ov.OffsetHigh = pos >> 16 >> 16; if (!WriteFile(env->me_fd, dp, size, NULL, &ov)) { rc = ErrCode(); DPRINTF(("WriteFile: %d", rc)); return rc; } Adding23dedefine off_t int64_t in an #ifdef _WIN32 region solves this.

1 0

Re: (ITS#8419) mdb_env_stat crashes when the env argument hasn't been opened yet
by hyc＠symas.com 08 May '16

08 May '16

On Sat, May 07, 2016 at 10:39:29PM +0000, aschorr(a)telemetry-investments.com wrote: > Hi, FYI, mdb_env_info has the same issue. Dup of ITS#8401, closing this ITS. > > Thanks, > Andy > > >

1 0

Re: (ITS#8419) mdb_env_stat crashes when the env argument hasn't been opened yet
by aschorr＠telemetry-investments.com 08 May '16

08 May '16

Hi, FYI, mdb_env_info has the same issue. Thanks, Andy

1 0

(ITS#8419) mdb_env_stat crashes when the env argument hasn't been opened yet
by aschorr＠telemetry-investments.com 08 May '16

08 May '16

Full_Name: Andrew Schorr Version: lmdb 0.9.18 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (38.76.0.51) I am implementing a gawk lmdb API. As a silly test, I ran the following code: gawk -l lmdb ' BEGIN { env = mdb_env_create() rc = mdb_env_stat(env, f) }' It catches a SIGSEGV. It is apparently illegal to call mdb_env_stat before calling mdb_open. Please consider updating the documentation or updating the function to protect against this situation. Thanks, Andy

1 0

Re: (ITS#8418) mdb_load performance
by aschorr＠telemetry-investments.com 06 May '16

06 May '16

Hi, sorry, please close this issue. I am unable to duplicate my initial results. The use of a cursor seems to be marginally helpful, and I no longer see any slowdown from batching the commits. I guess my server must have had some other loads on it that impacted my previous test results. I do see one odd result of batching -- the process ru_maxrss value reported by getrusage is much higher (10 GB vs 3 GB for my test data). I'm not sure why that's the case. Perhaps that resulted in some swapping during my initial test. Sorry for wasting your time. Regards, Andy On Thu, May 05, 2016 at 09:56:10PM +0000, openldap-its(a)OpenLDAP.org wrote: > > *** THIS IS AN AUTOMATICALLY GENERATED REPLY *** > > Thanks for your report to the OpenLDAP Issue Tracking System. Your > report has been assigned the tracking number ITS#8418. > > One of our support engineers will look at your report in due course. > Note that this may take some time because our support engineers > are volunteers. They only work on OpenLDAP when they have spare > time. > > If you need to provide additional information in regards to your > issue report, you may do so by replying to this message. Note that > any mail sent to openldap-its(a)openldap.org with (ITS#8418) > in the subject will automatically be attached to the issue report. > > mailto:openldap-its@openldap.org?subject=(ITS#8418) > > You may follow the progress of this report by loading the following > URL in a web browser: > http://www.OpenLDAP.org/its/index.cgi?findid=8418 > > Please remember to retain your issue tracking number (ITS#8418) > on any further messages you send to us regarding this report. If > you don't then you'll just waste our time and yours because we > won't be able to properly track the report. > > Please note that the Issue Tracking System is not intended to > be used to seek help in the proper use of OpenLDAP Software. > Such requests will be closed. > > OpenLDAP Software is user supported. > http://www.OpenLDAP.org/support/ > > -------------- > Copyright 1998-2007 The OpenLDAP Foundation, All Rights Reserved. > -- Andrew Schorr e-mail: aschorr(a)telemetry-investments.com Telemetry Investments, L.L.C. phone: 917-305-1748 545 Fifth Ave, Suite 1108 fax: 212-425-5550 New York, NY 10017-3630

1 0

(ITS#8418) mdb_load performance
by aschorr＠telemetry-investments.com 05 May '16

05 May '16

Full_Name: Andrew Schorr Version: lmdb 0.9.18 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (38.76.0.51) I patched mdb_load to add an option to set the mapsize with -T, as per open issue #8417. I then used it to load a 7 GB text file, and I found that the calls to mdb_txn_commit after every 100 mdb_cursor_put calls really slows down performance. If one comments out the batching, it goes much faster. And even faster is to avoid the cursor and just call mdb_put in a loop. Is there a reason mdb_load commits after every 100 puts?

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs