May 2016 - openldap-bugs

Re: (ITS#8431) LMDB: Access newly opened database from another transaction
by h.b.furuseth＠usit.uio.no 30 May '16

30 May '16

On 30. mai 2016 18:08, Juerg Bircher wrote: > If I got it right it is all about protecting against invalid DBIs > which can happen for example if someone closes database "A" which was > assigned DBI "5" and later the database "B" is opened and the free > slot 5 is then assigned to "B". So we could be misled using DBI "5" > having in mind database "A". > > So why don't we extend the MDB_DBI by adding the sequence in the lower > or higher bits. (...) Ugh. I didn't expect all my words to miss that badly. Yes, you caught one issue. There are more. The central point is what I wrote here: >> Multi-threaded use of structures, like the ones behind DBIs, must be >> synchronized. But DBIs and mdb_dbi_open() are not mutex-protected, >> nor are read-only txns (after the per-thread setup). So it's up to >> the user. And LMDB mostly can't catch users at less than txn-sized >> threading bugs - like the DBI workarounds people keep trying. I shouldn't really be the one to answer this, Howard might answer differently if he didn't just fetch a flamethrower. But anyway: It's about keeping a clean model of what is going on, which helps understanding and correctly coding both LMDB itself and user programs. When you don't work against the model, anyway. It's about keeping LMDB's internal data structures consistent. LMDB must know what the heck is going on when it manages and uses DBIs. Things like ensuring that two threads do not run mdb_dbi_open() at the same time, overwriting each others' work and creating a mess. It's about LMDB being able to know it delivers correct results, at least if the user doesn't work too hard at thwarting it. This usually means serializing various parts of the code using locks or something similar. But for the sake of speed and simplicity, LMDB has no relevant locks. It relies on the user to lock and has some code to catch some locking errors. One key word is "tries" - without locks or other expensive code, it can't always catch them even when it does try. So DBI errors are not "this DBI is broken, so drop/refresh it" but "your program is buggy, so fix it". And it's about living with the design choices and tradeoffs which LMDB has made, e.g. not slowing down read-only txns for anything like locking or extensive error checking. So anyway, this ITS is about changes to LMDB which will not happen. Instead it reminded me to add some checks for user errors with DBIs. -- Hallvard

1 0

Re: (ITS#8431) LMDB: Access newly opened database from another transaction
by juerg.bircher＠gmail.com 30 May '16

30 May '16

On Mon, May 30, 2016 at 2:44 PM, Hallvard Breien Furuseth <h.b.furuseth(a)usit.uio.no> wrote: > In ITS#8430, Juerg Bircher wrote: >> >> So if you drop ITS#8431 it would at least be very helpful to get an >> separate error like MDB_INVALID_DPI instead of EINVAL So the >> application sharing and caching handles (dbi) could handle the >> situation appropriate like use mdb_env_open() so that the dbi becomes >> valid in that txn as well. > > > This is ITS#8431 in a new guise, so I'm replying here. We know DBI > handling is inconvenient. We'd have fixed it if it were easy without > adding a bunch of locks. I'll walk through some of it below, I expect > it'll be useful to have for future reference anyway: > > > You mean like this? > > if (mdb_something(txn, dbi, ...) == MDB_INVALID_DBI) { > /* Get <dbi>, which some other txn committed after <txn> began */ > mdb_dbi_open(txn, "database name of <dbi>", 0, &dbi2); > assert(dbi2 == dbi); > } > > That breaks this requirement from the mdb_dbi_open() doc: > > * A transaction that uses > * this function must finish (either commit or abort) before > * any other transaction in the process may use this function. > Just to illustrate the simplified sequence of calls (using MDB_NOTLS): mdb_txn_begin(env, NULL, MDB_RDONLY, &txn2); mdb_txn_begin(env, NULL, MDB_RDONLY, &txn); MDB_dbi dbi; mdb_dbi_open(txn, "db1", 0, &dbi); // dbi becomes now valid and is propagated to the env mdb_txn_commit(txn); // At this point dbi for txn2 is invalid as txn2 started before opening the database if (mdb_something(txn2, dbi, ...) == MDB_INVALID_DBI) { MDB_dbi dbi2; mdb_dbi_open(txn2, "db1", 0, &dbi2); assert(dbi2 == dbi); // now dbi is valid for txn2 as with the mdb_dbi_open we propagated it to txn2! mdb_something(txn2, dbi, ...); } mdb_txn_commit(txn2); This code is really not preferred and is only a workaround. > To fulfill that, you must serialize txns which use mdb_dbi_open(). > (Unless they're all write txns, which LMDB serializes.) One reason > you must, is that there is no sync anywhere else in your suggestion. > > Multi-threaded use of structures, like the ones behind DBIs, must be > synchronized. But DBIs and mdb_dbi_open() are not mutex-protected, > nor are read-only txns (after the per-thread setup). So it's up to > the user. And LMDB mostly can't catch users at less than txn-sized > threading bugs - like the DBI workarounds people keep trying. > > I suppose we should add some of the above more clearly in lmdb.h doc, > since people keep trying to hack around it. And "EINVAL/MDB_BAD_DBI > often indicate that LMDB detected a bug in your program, not that you > can safely work around the problem when LMDB returns them." > (Not always, unless we walk carefully through which error codes are > returned when first.) > > > For a txn to pick up a DBI committed after the txn started, LMDB > would need to mutex-protect DBI operations. Something like this: > > - Add an env+txn flag MDB_ADMIN_LOCKS. If it is set in the env, > then mdb_dbi_open() fails if it is not also set in the txn. > The txn flag wraps DBI handling in a new mutex in txn_begin(), > dbi_open(), commit/abort() of a txn which opened a DBI. > The env flag wraps dbi_close() in the mutex. > > - Add mdb_dbi_pickup(MDB_txn*, MDB_dbi of a named DB): Pick up a DBI > which was committed after the txn began. It will lock the mutex, > and maybe it'll require MDB_ADMIN_LOCKS in the txn. > (I don't know if pickup(main DBI) would be safe.) > > Maybe that's more ACID-friendly - a separate function to step > outside and look at the current env. If I've gotten it right... > > Of course, if the txn which calls dbi_pickup() began before that > named DB was created, then the txn picks up a DBI which it cannot > use. That's OK as far as LMDB is concerned, it does safely catch > use of a non-existent named DB. But makes it less useful to users. > > So I wonder, will people optimize away the mutexing in the way you > suggested above? Try to use a DBI in a txn without MDB_ADMIN_LOCKS, > and if that fails call mdb_dbi_pickup(). If so, they reintroduce > the heisenbug: The test for whether the DBI was OK, was not mutexed > vs. mdb_txn_commit(txn which created the DBI). > > Or simpler, the new flag could just tell MDB to wrap a mutex around > every single txn_begin and txn_commit. That won't tempt anyone to > optimize away locks, since there are so much locking already. But > then, why are they using LMDB? LMDB has made a lot of tradeoffs to > avoid locking in read-only txns. See Caveats in lmdb.h. > > > And it'd still be unsafe to allow overlapping txns to mdb_dbi_open() > the same DB. txn1 creates the DBI, txn2 opens it too, txn1 aborts but > must not close the DBI because txn2 is using it... So MDB_dbx would > need a refcount of how many existing txns opened the DB during DBI > creation. Also I suppose ADMIN_LOCKS would need to disable > mdb_set_compare() & co. A new function mdb_dbi_open2() could init the > entire MDB_dbx while holding the mutex, so other txns would not use > the half-initialized DBX. > > Alternatively, dbi_open2(named DB) above can take a flag which says > "make the DBI immediately available in the MDB_env". That'll be > unsafe for txns where the user does not synchronize txn_begin against > the dbi_open2(). And it has its own sets of issues with multiple > txns opening a DB, nested txns, etc. > > > Then there's another approach: "Transactions are isolated from each > other, provided you use LMDB correctly. That's all. Sorry about > the DBI inconveniences." > If I got it right it is all about protecting against invalid DBIs which can happen for example if someone closes database "A" which was assigned DBI "5" and later the database "B" is opened and the free slot 5 is then assigned to "B". So we could be misled using DBI "5" having in mind database "A". So why don't we extend the MDB_DBI by adding the sequence in the lower or higher bits. When addressing the slot we simply strip the sequence bits and verify the slot's sequence against the sequence in the MDB_DBI to ensure the DBI has not changed. So picking up a DPI (as suggested in the patch) can be verified. The restriction with mdb_dbi_open() will stay. But the following would work: MDB_dbi dbi; mdb_txn_begin(env, NULL, MDB_RDONLY, &txn2); mdb_txn_begin(env, NULL, MDB_RDONLY, &txn1); mdb_dbi_open(txn1, "db1", 0, &dbi); // dbi becomes now valid and is propagated to the env // ==> and therefore is also valid for any other transaction! mdb_txn_commit(txn1); // use dbi opened in txn1 (which is committed) mdb_something(txn2, dbi, ...); .... mdb_txn_commit(txn2); So the patch could look something like that. static int mdb_update_db_info(MDB_txn *txn, MDB_dbi dbi) { unsigned int dbi_index = dbi & MDB_DBI_INDXE_MASK; unsigned int dbi_seq = dbi & MDB_DBI_SEQ_MASK << 16; if (txn->mt_env->me_dbiseqs[dbi_index] != dbi_seq) { return 0; } /* propagate newly opened db from env to current txn. Mark them as unused */ if (txn->mt_numdbs < txn->mt_env->me_numdbs) { memset(txn->mt_dbflags + txn->mt_numdbs, DB_UNUSED, txn->mt_env->me_numdbs - txn->mt_numdbs); } txn->mt_numdbs = txn->mt_env->me_numdbs; return dbi_index < txn->mt_numdbs; }

1 0

Re: (ITS#8430) Improved handling for large number of databases
by h.b.furuseth＠usit.uio.no 30 May '16

30 May '16

On 30. mai 2016 13:51, Hallvard Breien Furuseth wrote: > Copied from a private mail: >> Maybe a good way in between could be to memcpy txn->mt_env->me_dbflags. >> (...) memcpy uses SSE instructions for x86_64. > > Yes. If we make MDB_txn.mt_dbflags an uint16_t*, we can memcpy to it. > That costs <maxdbs> bytes/txn + one more reserved env flag bit in > PERSISTENT_FLAGS: Reserve DB_UNUSED, and keep it always set in > me_dbflags so the memcpy will set it in mt_dbflags. Oops, that was too clever. Then a bunch of code which assumes a DBI is valid if various txn flags are set, must also check DB_UNUSED. Or invert that to DB_USED which might be easier - (flags & DB_DIRTY) becomes F_ISSET(flags, DB_DIRTY|DB_USED) which checks if both are set. Maybe the simple solution using 2*<maxdbs> bytes/txn would be better. -- Hallvard

1 0

Re: (ITS#8431) LMDB: Access newly opened database from another transaction
by h.b.furuseth＠usit.uio.no 30 May '16

30 May '16

In ITS#8430, Juerg Bircher wrote: > So if you drop ITS#8431 it would at least be very helpful to get an > separate error like MDB_INVALID_DPI instead of EINVAL So the > application sharing and caching handles (dbi) could handle the > situation appropriate like use mdb_env_open() so that the dbi becomes > valid in that txn as well. This is ITS#8431 in a new guise, so I'm replying here. We know DBI handling is inconvenient. We'd have fixed it if it were easy without adding a bunch of locks. I'll walk through some of it below, I expect it'll be useful to have for future reference anyway: You mean like this? if (mdb_something(txn, dbi, ...) == MDB_INVALID_DBI) { /* Get <dbi>, which some other txn committed after <txn> began */ mdb_dbi_open(txn, "database name of <dbi>", 0, &dbi2); assert(dbi2 == dbi); } That breaks this requirement from the mdb_dbi_open() doc: * A transaction that uses * this function must finish (either commit or abort) before * any other transaction in the process may use this function. To fulfill that, you must serialize txns which use mdb_dbi_open(). (Unless they're all write txns, which LMDB serializes.) One reason you must, is that there is no sync anywhere else in your suggestion. Multi-threaded use of structures, like the ones behind DBIs, must be synchronized. But DBIs and mdb_dbi_open() are not mutex-protected, nor are read-only txns (after the per-thread setup). So it's up to the user. And LMDB mostly can't catch users at less than txn-sized threading bugs - like the DBI workarounds people keep trying. I suppose we should add some of the above more clearly in lmdb.h doc, since people keep trying to hack around it. And "EINVAL/MDB_BAD_DBI often indicate that LMDB detected a bug in your program, not that you can safely work around the problem when LMDB returns them." (Not always, unless we walk carefully through which error codes are returned when first.) For a txn to pick up a DBI committed after the txn started, LMDB would need to mutex-protect DBI operations. Something like this: - Add an env+txn flag MDB_ADMIN_LOCKS. If it is set in the env, then mdb_dbi_open() fails if it is not also set in the txn. The txn flag wraps DBI handling in a new mutex in txn_begin(), dbi_open(), commit/abort() of a txn which opened a DBI. The env flag wraps dbi_close() in the mutex. - Add mdb_dbi_pickup(MDB_txn*, MDB_dbi of a named DB): Pick up a DBI which was committed after the txn began. It will lock the mutex, and maybe it'll require MDB_ADMIN_LOCKS in the txn. (I don't know if pickup(main DBI) would be safe.) Maybe that's more ACID-friendly - a separate function to step outside and look at the current env. If I've gotten it right... Of course, if the txn which calls dbi_pickup() began before that named DB was created, then the txn picks up a DBI which it cannot use. That's OK as far as LMDB is concerned, it does safely catch use of a non-existent named DB. But makes it less useful to users. So I wonder, will people optimize away the mutexing in the way you suggested above? Try to use a DBI in a txn without MDB_ADMIN_LOCKS, and if that fails call mdb_dbi_pickup(). If so, they reintroduce the heisenbug: The test for whether the DBI was OK, was not mutexed vs. mdb_txn_commit(txn which created the DBI). Or simpler, the new flag could just tell MDB to wrap a mutex around every single txn_begin and txn_commit. That won't tempt anyone to optimize away locks, since there are so much locking already. But then, why are they using LMDB? LMDB has made a lot of tradeoffs to avoid locking in read-only txns. See Caveats in lmdb.h. And it'd still be unsafe to allow overlapping txns to mdb_dbi_open() the same DB. txn1 creates the DBI, txn2 opens it too, txn1 aborts but must not close the DBI because txn2 is using it... So MDB_dbx would need a refcount of how many existing txns opened the DB during DBI creation. Also I suppose ADMIN_LOCKS would need to disable mdb_set_compare() & co. A new function mdb_dbi_open2() could init the entire MDB_dbx while holding the mutex, so other txns would not use the half-initialized DBX. Alternatively, dbi_open2(named DB) above can take a flag which says "make the DBI immediately available in the MDB_env". That'll be unsafe for txns where the user does not synchronize txn_begin against the dbi_open2(). And it has its own sets of issues with multiple txns opening a DB, nested txns, etc. Then there's another approach: "Transactions are isolated from each other, provided you use LMDB correctly. That's all. Sorry about the DBI inconveniences."

1 0

Re: (ITS#8430) Improved handling for large number of databases
by h.b.furuseth＠usit.uio.no 30 May '16

30 May '16

On 28/05/16 22:08, Juerg Bircher wrote: > I understand that it is a tricky thing. But maybe the lazy init is > still feasible? (...) Right, lazy init is feasible. Just keep cross-txn DBI stuff out of it, I'll take your latest variant to ITS#8431. Copied from a private mail: > Maybe a good way in between could be to memcpy txn->mt_env->me_dbflags. > (...) memcpy uses SSE instructions for x86_64. Yes. If we make MDB_txn.mt_dbflags an uint16_t*, we can memcpy to it. That costs <maxdbs> bytes/txn + one more reserved env flag bit in PERSISTENT_FLAGS: Reserve DB_UNUSED, and keep it always set in me_dbflags so the memcpy will set it in mt_dbflags. Or maybe your code can be used without initial memcpy and I'm just paranoid. The one problem I can think of is user errors - people coming up with workarounds like your cross-txn DBI stuff. LMDB tries to catch some of that, but maybe late pickup defeats some such checks. I'll wait for Howard to say something at this point. BTW, all this reminds me - I should add a test which tries to catch mdb_dbi_open() in overlapping txns. Got the code, but forgot about it while pondering a good error code name. > However md_flags would need to be moved out of the MDB_dbi struct Nope. Sub-DBs with duplicate data have md_flags which do not correspond to anything saved in env->me_flags. Your lazy-init code needs to use the memcpyed flags instead of me_dbflags, that's all. > We should consider to align on 16 byte boundary for best performance. It's already aligned.

1 0

Re: (ITS#8431) LMDB: Access newly opened database from another transaction
by juerg.bircher＠gmail.com 28 May '16

28 May '16

On Sat, May 28, 2016 at 11:18 AM, Howard Chu <hyc(a)symas.com> wrote: > juerg.bircher(a)gmail.com wrote: >> >> On 27/05/16 15:11, "Hallvard Breien Furuseth" <h.b.furuseth(a)usit.uio.no> >> wrote: >> >>> This works as intended. Each transaction is atomic, and shall see >>> neither DBIs nor data which were committed after the transaction began. >>> Cross-txn mdb_dbi_open() is in any case hairy to get right, we screwed >>> it up several times before arriving at the now-obvious semantics. >>> >> >> It is fine that newly opened database handles shall not be seen by >> other transactions as long the transaction runs.. But once the >> transaction which opened the handle is committed it should be valid >> for any other transaction. > > > It should be valid for any other transaction *that started after the opening > transaction committed.* This is the nature of transactions, they are fully > isolated from each other. > >> This would greatly simplify lazy database >> opening of a multithreaded application. > > > Allowing existing transactions to use the DBIs would break transaction > isolation. That is a non-starter. > >> The suggested patch does exactly implement that behaviour. I believe >> it does not break anything. Isolation must be guaranteed for the data >> but not necessarily for the database handles. > > > Wrong. A database handle essentially points to data. Exposing that data to a > transaction that was opened before that handle's transaction was committed > *does* break isolation. Violating the ACID guarantees of LMDB's transaction > system is Not allowed. >> >> >>> An application can open and cache its DBIs in a separate startup >>> transaction, then commit it and start new transactions. >> >> >> Yes this would be a simple approach to get around that problem. But I >> do not like it as with each opened database the overhead increases >> especially if not using my suggested improvements in ITS#8430. > > > You are saying you don't like ACID transactions. If that's the case, LMDB is > not for you. Not at all ACID is a must and something I very much appreciate about LMDB! LMDB is definitely for me and I'm saying that after having spent nearly one intensive year with LMDB! So I like to thank you for the outstanding database. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8430) Improved handling for large number of databases
by juerg.bircher＠gmail.com 28 May '16

28 May '16

---------- Forwarded message ---------- From: Juerg Bircher <juerg.bircher(a)gmail.com> Date: Sat, May 28, 2016 at 10:30 PM Subject: Re: (ITS#8430) Improved handling for large number of databases To: Hallvard Breien Furuseth <h.b.furuseth(a)usit.uio.no> On Sat, May 28, 2016 at 3:41 AM, Hallvard Breien Furuseth <h.b.furuseth(a)usit.uio.no> wrote: > On 28/05/16 03:28, Hallvard Breien Furuseth wrote: >> >> On 27/05/16 23:39, juerg.bircher(a)gmail.com wrote: >>> >>> Maybe I am overseeing something as I do not know all the thoughts >>> between the lines. >> >> >> With ITS#8431 I can think of one thing: >> (...blah...) > > > Actually the main reason I worry about lazy init is simply that > we've screwed up DBIs many times, so I prefer to go slowly. > That's Howard's call though, not mine. Maybe a good way in between could be to memcpy txn->mt_env->me_dbflags. However md_flags would need to be moved out of the MDB_dbi struct like mt_dbflags or easier just add another array serving as snapshot. This would not break anything with flag initialization as we get a snapshot of them in the mdb_txn_renew0. memcpy uses SSE instructions for x86_64. We should consider to align on 16 byte boundary for best performance. The dbflags could then still be initialized with MDB_UNSUSED as suggested.

1 0

Re: (ITS#8430) Improved handling for large number of databases
by juerg.bircher＠gmail.com 28 May '16

28 May '16

On Sat, May 28, 2016 at 3:28 AM, Hallvard Breien Furuseth <h.b.furuseth(a)usit.uio.no> wrote: > On 27/05/16 23:39, juerg.bircher(a)gmail.com wrote: >> >> On 27/05/16 14:42, "Hallvard Breien Furuseth" <h.b.furuseth(a)usit.uio.no> >> wrote: >>> >>> I think MDB_WORD intends to be mdb_size_t, but should be 'unsigned >>> long': It's probably no optimization to use 'long long' when that >>> is not a native integer type. But I haven't profiled that. >> >> >> The MDB_WORD is used to get the appropriate word size of the CPU for >> the optimised dbflag check. So on a 32 bit CPUs it should be an 32 bit >> integer and on a 64 bit CPUs a 64 bit integer. > > > That's why I suggested unsigned long. It seems the best guess for a > native wide type. But it's hard to guess what kind of CPU is in use. > >> But the mdb_size_t seems always to be 64 bits? > > > On a 32-bit host it's normally 32-bit, but you get 64-bit mdb_size_t anyway > if you build and use LMDB with MDB_VL32 defined. I misread > your patch, it should not use MDB_VL32. > >>> The delay in picking up env->me_dbflags is a bit troubling, since it >>> means creating a new txn is no longer quite atomic. I can't think of >>> how it'll make a difference when DBIs are used correctly, but that's one >>> thing people screw up. Maybe it'd be OK if renew0() just copies it into >>> mt_dbs[].md_flags without the '&', and mdb_setup_db_info() fixes it up. >> >> >> Is it really atomic in original lmdb? The flags are copied from the >> environment and calculated. But while copying the dbflags from the >> environment they may change. So read level consistency and therefore >> atomicity is not guaranteed. > > > I should have said, it's atomic when DBIs are used correctly. Such as > obeying this limitation from the mdb_dbi_open() doc and a similar one > for mdb_dbi_close(): > > * A transaction that uses > * this function must finish (either commit or abort) before > * any other transaction in the process may use this function. > > To guarante atomicity we'd need locks around some DBI operations. > To avoid that, LMDB simply calls unsafe use "user errors" - which can > cause breakage. Hence those annoying DBI usage restrictions. > > LMDB does attempt to catch user errors when it's cheap to try, see e.g. > TXN_DBI_CHANGED(). Doesn't work at all for read-only txns, they have > txn->mt_dbiseqs == env->me_dbiseqs. > > So you can see why DBI cleverness makes me nervous. > I think the MDB_WORD looping can go in now, but we'll need to think > carefully about the lazy init. I understand that it is a tricky thing. But maybe the lazy init is still feasible? In my understand without having all the background it seems ok to get the current snapshot of the database flags from the env once it is firstly accessed whether it is at the beginning or later in the transaction. Read level consistency should be preserved anyway as mt_txnId is set in mdb_txn_renew0(). > >> But I guess it just does not matter. >> So picking up the flags lazily just extends the time of copying the >> flags from the environment but is basically the same. >> Atomicity and also isolation is a must in respect to the data but I >> think not in respect to the database handles and their flags. >> Maybe I am overseeing something as I do not know all the thoughts >> between the lines. > > > With ITS#8431 I can think of one thing: > > There's no sync between mdb_dbi_open() in one txn and another txn > using the new DBI. It'll be up to the user to synchronize DBIs. For > that to work in write-txs, TXN_DBI_CHANGED() must go away, and writers > will have no way to try to catch user errors with sync. Or if it > should only work in read-only txns, then read-only txns and write-txns > must have different definitions of what is a "user error" with DBIs. > > Anyway, we'll need to think about the definition of user errors. > So if you drop ITS#8431 it would at least be very helpful to get an separate error like MDB_INVALID_DPI instead of EINVAL So the application sharing and caching handles (dbi) could handle the situation appropriate like use mdb_env_open() so that the dbi becomes valid in that txn as well. > > Other notes: > > /** ... */ above a global declaration or /**< ... */ to the right > of it tells Doxygen that the comment is the doc of the declared item. > "#foo" inside a Doxygen comment is a link to the declaration of foo. > > > Maybe mt_dbflags should have an extra byte or word at the end which > does not have DB_UNUSED, and with size rounded up to sizeof(MDB_WORD). > Then stepping to next DBI can do something like this, unrelated to > the check against mt_numdbs. Not sure if that's an improvement, I'm > getting tired: > > /** Return an #MDB_WORD where each byte is \b byte */ > #define WORD_BYTES(byte) ((MDB_WORD)-1 / 0xff * (byte)) > > /** Step up to next used DBI */ > #define DBI_STEP(txn, dbi) do { \ > if (! (++(dbi) & (sizeof(mdb_word_t)-1))) \ > while (*(mdb_word_t *)((txn)->mt_dbflags + (dbi)) == > WORD_BYTES(DB_UNUSED)) > (dbi) += sizeof(MDB_WORD); \ > } while (0) > Yes round it up to sizeof(MDB_WORD) would be obvious in connection with the optimized loop. If you are willing to add some more #ifdef we could even get it faster at least for x86_64 using SSE2 instructions. Then MDB_WORD would be 128 bit and we check 16 flags in one step using _mm_cmpeq_epi8 and _mm_movemask_epi8 to get the mask of matching flags. If the mask != 0xffffffff we found an used database. Then we negate the mask and with __builtin_ffs(mask) we could then find the index of the first used database.

1 0

(ITS#8102) syncrepl_entry: be_modify failed (16)
by tpretz＠gmail.com 28 May '16

28 May '16

Hi, Following up on this ITS i opened a while back. With multi-master, normal syncrepl, i would sometimes receive: slapd: null_callback : error code 0x10 slapd: syncrepl_entry: rid=106 be_modify failed (16) Triggering a syncrepl connection drop/retry, whilst playing the sessionlog when a server with multiple providers was started. I am now testing with 2.4.44 and have had a chance to look at this annoying, but seemingly not destructive issue in some more detail. As i partially referenced previously, this occurs within syncrepl_entry, for modifications, a diff of old_entry to new_entry is performed. Then if changes are needed a be_modify is performed. There is however, no locking which prevents two, or more, threads from performing these diffs, and then mods, in an interleaved fashion within this function itself. Looking in do_syncrep2, if the cookie tag is present the cs_pmutex is acquired and held for the duration of modifications. This mutex protects from syncrepl_entry race conditions and serializes modifications. I have also noticed this issue during normal operations (ie all syncrepl in persist) when out of order writes are occurring on a master which are relatively easy to reproduce on an hdb backend server. When a cookie is not sent with an entry the cs_pmutex is not acquired. Without having some protection, non-cookie modifications will race each other between syncrepl threads. So, i am testing surrounding the syncrepl_entry "if" block (line 1036) with a cs_pmutex lock/release (when punlock < 0) to serialize non_cookie mods just like the cookie ones. So far this is running tests and i haven't seen the null_callback issue, either when catching up from the session log, or running with ongoing out of order writes being replicated (running alongside unmodified 2.4.44 to compare differences). When acquiring the cs_pmutex i have used the same logic as at line 958 (using trylock, with a shutdown check). I wonder if it is safe to acquire the mutex with a standard ldap_pvt_thread_mutex_lock at this point without spinning. line numbers from RELENG_2_4 (721a038b7bc9732f52eeef5324c180c4f137cd75) Thanks Tom

1 0

Re: (ITS#8431) LMDB: Access newly opened database from another transaction
by hyc＠symas.com 28 May '16

28 May '16

juerg.bircher(a)gmail.com wrote: > On 27/05/16 15:11, "Hallvard Breien Furuseth" <h.b.furuseth(a)usit.uio.no> wrote: > >> This works as intended. Each transaction is atomic, and shall see >> neither DBIs nor data which were committed after the transaction began. >> Cross-txn mdb_dbi_open() is in any case hairy to get right, we screwed >> it up several times before arriving at the now-obvious semantics. >> > > It is fine that newly opened database handles shall not be seen by > other transactions as long the transaction runs.. But once the > transaction which opened the handle is committed it should be valid > for any other transaction. It should be valid for any other transaction *that started after the opening transaction committed.* This is the nature of transactions, they are fully isolated from each other. > This would greatly simplify lazy database > opening of a multithreaded application. Allowing existing transactions to use the DBIs would break transaction isolation. That is a non-starter. > The suggested patch does exactly implement that behaviour. I believe > it does not break anything. Isolation must be guaranteed for the data > but not necessarily for the database handles. Wrong. A database handle essentially points to data. Exposing that data to a transaction that was opened before that handle's transaction was committed *does* break isolation. Violating the ACID guarantees of LMDB's transaction system is Not allowed. > >> An application can open and cache its DBIs in a separate startup >> transaction, then commit it and start new transactions. > > Yes this would be a simple approach to get around that problem. But I > do not like it as with each opened database the overhead increases > especially if not using my suggested improvements in ITS#8430. You are saying you don't like ACID transactions. If that's the case, LMDB is not for you. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8430) Improved handling for large number of databases
by h.b.furuseth＠usit.uio.no 27 May '16

27 May '16

On 28/05/16 03:28, Hallvard Breien Furuseth wrote: >On 27/05/16 23:39, juerg.bircher(a)gmail.com wrote: >> Maybe I am overseeing something as I do not know all the thoughts >> between the lines. > > With ITS#8431 I can think of one thing: > (...blah...) Actually the main reason I worry about lazy init is simply that we've screwed up DBIs many times, so I prefer to go slowly. That's Howard's call though, not mine.

1 0

Re: (ITS#8430) Improved handling for large number of databases
by h.b.furuseth＠usit.uio.no 27 May '16

27 May '16

On 27/05/16 23:39, juerg.bircher(a)gmail.com wrote: > On 27/05/16 14:42, "Hallvard Breien Furuseth" <h.b.furuseth(a)usit.uio.no> wrote: >> I think MDB_WORD intends to be mdb_size_t, but should be 'unsigned >> long': It's probably no optimization to use 'long long' when that >> is not a native integer type. But I haven't profiled that. > > The MDB_WORD is used to get the appropriate word size of the CPU for > the optimised dbflag check. So on a 32 bit CPUs it should be an 32 bit > integer and on a 64 bit CPUs a 64 bit integer. That's why I suggested unsigned long. It seems the best guess for a native wide type. But it's hard to guess what kind of CPU is in use. > But the mdb_size_t seems always to be 64 bits? On a 32-bit host it's normally 32-bit, but you get 64-bit mdb_size_t anyway if you build and use LMDB with MDB_VL32 defined. I misread your patch, it should not use MDB_VL32. >> The delay in picking up env->me_dbflags is a bit troubling, since it >> means creating a new txn is no longer quite atomic. I can't think of >> how it'll make a difference when DBIs are used correctly, but that's one >> thing people screw up. Maybe it'd be OK if renew0() just copies it into >> mt_dbs[].md_flags without the '&', and mdb_setup_db_info() fixes it up. > > Is it really atomic in original lmdb? The flags are copied from the > environment and calculated. But while copying the dbflags from the > environment they may change. So read level consistency and therefore > atomicity is not guaranteed. I should have said, it's atomic when DBIs are used correctly. Such as obeying this limitation from the mdb_dbi_open() doc and a similar one for mdb_dbi_close(): * A transaction that uses * this function must finish (either commit or abort) before * any other transaction in the process may use this function. To guarante atomicity we'd need locks around some DBI operations. To avoid that, LMDB simply calls unsafe use "user errors" - which can cause breakage. Hence those annoying DBI usage restrictions. LMDB does attempt to catch user errors when it's cheap to try, see e.g. TXN_DBI_CHANGED(). Doesn't work at all for read-only txns, they have txn->mt_dbiseqs == env->me_dbiseqs. So you can see why DBI cleverness makes me nervous. I think the MDB_WORD looping can go in now, but we'll need to think carefully about the lazy init. > But I guess it just does not matter. > So picking up the flags lazily just extends the time of copying the > flags from the environment but is basically the same. > Atomicity and also isolation is a must in respect to the data but I > think not in respect to the database handles and their flags. > Maybe I am overseeing something as I do not know all the thoughts > between the lines. With ITS#8431 I can think of one thing: There's no sync between mdb_dbi_open() in one txn and another txn using the new DBI. It'll be up to the user to synchronize DBIs. For that to work in write-txs, TXN_DBI_CHANGED() must go away, and writers will have no way to try to catch user errors with sync. Or if it should only work in read-only txns, then read-only txns and write-txns must have different definitions of what is a "user error" with DBIs. Anyway, we'll need to think about the definition of user errors. Other notes: /** ... */ above a global declaration or /**< ... */ to the right of it tells Doxygen that the comment is the doc of the declared item. "#foo" inside a Doxygen comment is a link to the declaration of foo. Maybe mt_dbflags should have an extra byte or word at the end which does not have DB_UNUSED, and with size rounded up to sizeof(MDB_WORD). Then stepping to next DBI can do something like this, unrelated to the check against mt_numdbs. Not sure if that's an improvement, I'm getting tired: /** Return an #MDB_WORD where each byte is \b byte */ #define WORD_BYTES(byte) ((MDB_WORD)-1 / 0xff * (byte)) /** Step up to next used DBI */ #define DBI_STEP(txn, dbi) do { \ if (! (++(dbi) & (sizeof(mdb_word_t)-1))) \ while (*(mdb_word_t *)((txn)->mt_dbflags + (dbi)) == WORD_BYTES(DB_UNUSED)) (dbi) += sizeof(MDB_WORD); \ } while (0)

1 0

Re: (ITS#8431) LMDB: Access newly opened database from another transaction
by juerg.bircher＠gmail.com 27 May '16

27 May '16

On 27/05/16 15:11, "Hallvard Breien Furuseth" <h.b.furuseth(a)usit.uio.no> wrote: >This works as intended. Each transaction is atomic, and shall see >neither DBIs nor data which were committed after the transaction began. >Cross-txn mdb_dbi_open() is in any case hairy to get right, we screwed >it up several times before arriving at the now-obvious semantics. > It is fine that newly opened database handles shall not be seen by other transactions as long the transaction runs.. But once the transaction which opened the handle is committed it should be valid for any other transaction. This would greatly simplify lazy database opening of a multithreaded application. The suggested patch does exactly implement that behaviour. I believe it does not break anything. Isolation must be guaranteed for the data but not necessarily for the database handles. >An application can open and cache its DBIs in a separate startup >transaction, then commit it and start new transactions. Yes this would be a simple approach to get around that problem. But I do not like it as with each opened database the overhead increases especially if not using my suggested improvements in ITS#8430. See my measurements with the improved lmdb and the original lmdb. The test always performs 1000000 iterations on one database while having 1 database open in the first run and having all databases open in the second run. lmdb improved (renew, reset) [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 10] open databases in [0.31975] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [10 of 10] open databases in [0.20350] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 100] open databases in [0.25845] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [100 of 100] open databases in [0.29663] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 1000] open databases in [0.28590] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1000 of 1000] open databases in [0.42897] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 10000] open databases in [0.30004] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [10000 of 10000] open databases in [1.68870] seconds lmdb improved (begin, commit) [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 10] open databases in [0.36538] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [10 of 10] open databases in [0.35923] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 100] open databases in [0.34858] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [100 of 100] open databases in [0.39294] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 1000] open databases in [0.40222] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1000 of 1000] open databases in [0.54752] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 10000] open databases in [0.78595] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [10000 of 10000] open databases in [2.32414] seconds lmdb original (renew, reset) [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 10] open databases in [0.18597] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [10 of 10] open databases in [0.21572] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 100] open databases in [0.24173] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [100 of 100] open databases in [0.46497] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 1000] open databases in [0.27127] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1000 of 1000] open databases in [4.18579] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [1 of 10000] open databases in [0.30128] seconds [1000000] iterations (renew, cursor_open, cursor_close, reset) on 1 database with [10000 of 10000] open databases in [49.35048] seconds lmdb original (begin, commit) [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 10] open databases in [0.45135] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [10 of 10] open databases in [0.38990] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 100] open databases in [0.47890] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [100 of 100] open databases in [0.69917] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 1000] open databases in [1.84908] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1000 of 1000] open databases in [5.88098] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [1 of 10000] open databases in [22.12491] seconds [1000000] iterations (begin, cursor_open, cursor_close, abort) on 1 database with [10000 of 10000] open databases in [74.53854] seconds > >-- > Hallvard Regards Juerg

1 0

Re: (ITS#8430) Improved handling for large number of databases
by juerg.bircher＠gmail.com 27 May '16

27 May '16

First of all thank you for having a look at the improvement. On 27/05/16 14:42, "Hallvard Breien Furuseth" <h.b.furuseth(a)usit.uio.no> wrote: >On 27. mai 2016 12:57, juerg.bircher(a)helmedica.com wrote: >> mdb_dbi_open() is however not improved with the assumption that the database >> handle (dbi) is cached in the application. So mdb_dbi_open() should happen only >> once for each database during the life time of an application. > >It must check if the DB is open anyway, since LDMB breaks if the user >has two DBIs to the same DB. Though maybe we could add an optimization >flag to skip the search, for users who know they don't reopen open DBs. > >Other notes: > >LMDB and OpenLDAP mostly use tab-width = indentation = 4. (We should >comment that somewhere.) You seem to use tab-width = 8. I will consider that for the next patch! > >I think MDB_WORD intends to be mdb_size_t, but should be 'unsigned >long': It's probably no optimization to use 'long long' when that >is not a native integer type. But I haven't profiled that. The MDB_WORD is used to get the appropriate word size of the CPU for the optimised dbflag check. So on a 32 bit CPUs it should be an 32 bit integer and on a 64 bit CPUs a 64 bit integer. But the mdb_size_t seems always to be 64 bits? > >The delay in picking up env->me_dbflags is a bit troubling, since it >means creating a new txn is no longer quite atomic. I can't think of >how it'll make a difference when DBIs are used correctly, but that's one >thing people screw up. Maybe it'd be OK if renew0() just copies it into >mt_dbs[].md_flags without the '&', and mdb_setup_db_info() fixes it up. Is it really atomic in original lmdb? The flags are copied from the environment and calculated. But while copying the dbflags from the environment they may change. So read level consistency and therefore atomicity is not guaranteed. But I guess it just does not matter. So picking up the flags lazily just extends the time of copying the flags from the environment but is basically the same. Atomicity and also isolation is a must in respect to the data but I think not in respect to the database handles and their flags. Maybe I am overseeing something as I do not know all the thoughts between the lines. > >-- >Hallvard Regards Juerg

1 0

Re: (ITS#8430) Improved handling for large number of databases
by juerg.bircher＠gmail.com 27 May '16

27 May '16

I am switching to gmail as in outlook my name uses Umlaute which results in base64 encoded mails! Strangely the mailing list openldap-technical can handle it and openldap-its cannot! Here the missing disclaimer for the patch: The attached patch file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Juerg Bircher juerg.bircher(a)helmedica.com. I have not assigned rights and/or interest in this work to any party. I, Juerg Bircher, hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice.

1 0

Re: (ITS#8431) LMDB: Access newly opened database from another transaction
by h.b.furuseth＠usit.uio.no 27 May '16

27 May '16

This works as intended. Each transaction is atomic, and shall see neither DBIs nor data which were committed after the transaction began. Cross-txn mdb_dbi_open() is in any case hairy to get right, we screwed it up several times before arriving at the now-obvious semantics. An application can open and cache its DBIs in a separate startup transaction, then commit it and start new transactions. -- Hallvard

1 0

Re: (ITS#8430) Improved handling for large number of databases
by juerg.bircher＠helmedica.com 27 May '16

27 May '16

SGVyZSB0aGUgbWlzc2luZyBkaXNjbGFpbWVyIGZvciB0aGUgcGF0Y2g6DQoNClRoZSBhdHRhY2hl ZCBwYXRjaCBmaWxlIGlzIGRlcml2ZWQgZnJvbSBPcGVuTERBUCBTb2Z0d2FyZS4gQWxsIG9mIHRo ZSBtb2RpZmljYXRpb25zIHRvIE9wZW5MREFQIFNvZnR3YXJlIHJlcHJlc2VudGVkIGluIHRoZSBm b2xsb3dpbmcgcGF0Y2goZXMpIHdlcmUgZGV2ZWxvcGVkIGJ5IEp1ZXJnIEJpcmNoZXIganVlcmcu YmlyY2hlckBoZWxtZWRpY2EuY29tLiBJIGhhdmUgbm90IGFzc2lnbmVkIHJpZ2h0cyBhbmQvb3Ig aW50ZXJlc3QgaW4gdGhpcyB3b3JrIHRvIGFueSBwYXJ0eS4NCg0KSSwgSnVlcmcgQmlyY2hlciwg aGVyZWJ5IHBsYWNlIHRoZSBmb2xsb3dpbmcgbW9kaWZpY2F0aW9ucyB0byBPcGVuTERBUCBTb2Z0 d2FyZSAoYW5kIG9ubHkgdGhlc2UgbW9kaWZpY2F0aW9ucykgaW50byB0aGUgcHVibGljIGRvbWFp bi4gSGVuY2UsIHRoZXNlIG1vZGlmaWNhdGlvbnMgbWF5IGJlIGZyZWVseSB1c2VkIGFuZC9vciBy ZWRpc3RyaWJ1dGVkIGZvciBhbnkgcHVycG9zZSB3aXRoIG9yIHdpdGhvdXQgYXR0cmlidXRpb24g YW5kL29yIG90aGVyIG5vdGljZS4NCg0K

1 0

Re: (ITS#8430) Improved handling for large number of databases
by juerg.bircher＠helmedica.com 27 May '16

27 May '16

DQpEaXNjbGFpbWVyDQoNCg0KDQpUaGUgYXR0YWNoZWQgcGF0Y2ggZmlsZSBpcyBkZXJpdmVkIGZy b20gT3BlbkxEQVAgU29mdHdhcmUuIEFsbCBvZiB0aGUgbW9kaWZpY2F0aW9ucyB0byBPcGVuTERB UCBTb2Z0d2FyZSByZXByZXNlbnRlZCBpbiB0aGUgZm9sbG93aW5nIHBhdGNoKGVzKSB3ZXJlIGRl dmVsb3BlZCBieSBKdWVyZyBCaXJjaGVyIGp1ZXJnLmJpcmNoZXJAaGVsbWVkaWNhLmNvbS4gSSBo YXZlIG5vdCBhc3NpZ25lZCByaWdodHMgYW5kL29yIGludGVyZXN0IGluIHRoaXMgd29yayB0byBh bnkgcGFydHkuDQoNCkksIEp1ZXJnIEJpcmNoZXIsIGhlcmVieSBwbGFjZSB0aGUgZm9sbG93aW5n IG1vZGlmaWNhdGlvbnMgdG8gT3BlbkxEQVAgU29mdHdhcmUgKGFuZCBvbmx5IHRoZXNlIG1vZGlm aWNhdGlvbnMpIGludG8gdGhlIHB1YmxpYyBkb21haW4uIEhlbmNlLCB0aGVzZSBtb2RpZmljYXRp b25zIG1heSBiZSBmcmVlbHkgdXNlZCBhbmQvb3IgcmVkaXN0cmlidXRlZCBmb3IgYW55IHB1cnBv c2Ugd2l0aCBvciB3aXRob3V0IGF0dHJpYnV0aW9uIGFuZC9vciBvdGhlciBub3RpY2UuDQoNCg0K DQpPbiAyNy8wNS8xNiAxMzoxNiwgIkhvd2FyZCBDaHUiIDxoeWNAc3ltYXMuY29tPiB3cm90ZToN Cg0KPmp1ZXJnLmJpcmNoZXJAaGVsbWVkaWNhLmNvbSB3cm90ZToNCj4+IEZ1bGxfTmFtZTogSnVl cmcgQmlyY2hlcg0KPj4gVmVyc2lvbjogbG1kYiAobWFzdGVyKQ0KPj4gT1M6IE1hY09TIC8gTGlu dXgNCj4+IFVSTDogZnRwOi8vZnRwLm9wZW5sZGFwLm9yZy9pbmNvbWluZy9KdWVyZ19CaXJjaGVy XzE2MDUyNy1JbXByb3ZlZC1oYW5kbGluZy1mb3ItbGFyZ2UtbnVtYmVyLW9mLWRhdGFiYXNlcy5w YXRjaA0KPj4gU3VibWlzc2lvbiBmcm9tOiAoTlVMTCkgKDE3OC44Mi4zNy4xOTUpDQo+DQo+VGhh bmtzIGZvciB0aGUgcGF0Y2gsIGJ1dCBmb3IgYSBjaGFuZ2Ugb2YgdGhpcyBzaXplIHlvdSBhbHNv IG5lZWQgdG8gaW5jbHVkZSANCj5hbiBJUCBSaWdodHMgbm90aWNlLCBhcyBkb2N1bWVudGVkIG9u IHRoZSBDb250cmlidXRpbmcgZ3VpZGVsaW5lcy4NCj4+DQo+Pg0KPj4gVGhlcmUgaXMgYSBpbmNy ZWFzZWQgcGVyZm9ybWFuY2UgcGVuYWx0eSB0aGUgbW9yZSBkYXRhYmFzZXMgYXJlIGNyZWF0ZWQg d2l0aGluDQo+PiB0aGUgc2FtZSBlbnZpcm9ubWVudC4gSSB3YXMgbG9va2luZyBmb3IgYSB3YXkg dGhlIGltcHJvdmUgdGhhdCBieSBrZWVwaW5nIHRoZQ0KPj4gc2ltcGxpY2l0eSBvZiB0cmFja2lu ZyBkYXRhYmFzZXMgd2l0aGluIGEgbGlzdCB3aXRoIGRpcmVjdCBhY2Nlc3MgYnkgaW5kZXgNCj4+ IChNREJfZGJpKS4NCj4+IG1kYl9kYmlfb3BlbigpIGlzIGhvd2V2ZXIgbm90IGltcHJvdmVkIHdp dGggdGhlIGFzc3VtcHRpb24gdGhhdCB0aGUgZGF0YWJhc2UNCj4+IGhhbmRsZSAoZGJpKSBpcyBj YWNoZWQgaW4gdGhlIGFwcGxpY2F0aW9uLiBTbyBtZGJfZGJpX29wZW4oKSBzaG91bGQgaGFwcGVu IG9ubHkNCj4+IG9uY2UgZm9yIGVhY2ggZGF0YWJhc2UgZHVyaW5nIHRoZSBsaWZlIHRpbWUgb2Yg YW4gYXBwbGljYXRpb24uDQo+Pg0KPj4NCj4+IE9uZSBpc3N1ZSBpcyB0aGF0IG1kYl90eG5fYmVn aW4oKSAoZm9yIHJlYWQtb25seSB0cmFuc2FjdGlvbnMpIGNhbGxvYyB0aGUNCj4+IHNpemVvZihN REJfdHhuKSArIG1lX21heGRicyAqIHNpemVvZihNREJfZGIgKyAxKS4gVGhlIHBsdXMgMSBmb3Ig dGhlIGRiZmxhZ3MuDQo+PiBIb3dldmVyIGl0IGlzIHN1ZmZpY2llbnQgb25seSB0byBtYWxsb2Mg dGhhdCBzaXplIGFuZCBjbGVhciB0aGUgc2l6ZW9mKE1EQl90eG4pDQo+Pg0KPj4gICAgICBtZW1z ZXQodHhuLCAwLCBzaXplb2YoTURCX3R4bikNCj4+DQo+PiBBZnRlciB0aGF0IHRoZSBkYXRhIGJl eW9uZCB0aGUgTURCX3R4biBpcyBub3QgaW5pdGlhbGl6ZWQgd2hpY2ggaXMgb2sgZm9yIHRoZQ0K Pj4gbW9tZW50Lg0KPj4NCj4+IFRoZSBuZXh0IGltcHJvdmVtZW50IGhhcHBlbnMgaW4gbWRiX3R4 bl9yZW5ldzAoKSB3aGVyZSB0aGUgZGJmbGFncyBhcmUgb25seSBzZXQNCj4+IHRvIERCX1VOVVNF RCAoYSBuZXcgZmxhZykgZm9yIGVhY2ggZGF0YWJhc2UgY3VycmVudGx5IG9wZW5lZCBpbiB0aGUN Cj4+IGVudmlyb25tZW50Lg0KPj4NCj4+ICAgICAgbWVtc2V0KHR4bi0+bXRfZGJmbGFncywgREJf VU5VU0VELCB0eG4tPm10X251bWRicyk7DQo+Pg0KPj4gVGhlIGZvcm1lciBjb2RlIHVzZWQgdG8g dG8gbG9vcCB0aHJvdWdoIGVhY2ggZGF0YWJhc2UgdG8gY2FsY3VsYXRlIHRoZSBkYmZsYWdzLg0K Pj4gVGhpcyBpcyBzdGlsbCBkb25lIGJ1dCBsYXppbHkgZm9yIGVhY2ggYWNjZXNzZWQgZGF0YWJh c2Ugd2l0aCB0aGUgYXNzdW1wdGlvbg0KPj4gdGhhdCBhIHJlYWQgb25seSB0cmFuc2FjdGlvbiBy YXJlbHkgdXNlcyBhbGwgZGF0YWJhc2VzIG9mIHRoZSBlbnZpcm9ubWVudC4NCj4+DQo+PiBUaGUg bGF6eSBpbml0aWFsaXphdGlvbiBvZiB0aGUgZGJmbGFnIGhhcHBlbnMgaW4gdGhlIG1hY3JvIFRY Tl9EQklfRVhJU1Qgd2hpY2gNCj4+IGlzIGFsd2F5cyB1c2VkIHdoZW4gYSBkYXRhYmFzZSBoYW5k bGUgKGRiaSkgaXMgcGFzc2VkIHRvIGFuIGZ1bmN0aW9uLg0KPj4gVGhlIGZsYWdzIGFyZSB1cGRh dGVkIGluIG1kYl9zZXR1cF9kYl9pbmZvKCkgb25jZSBhIGRhdGFiYXNlIGlzIGFjY2VzcyB3aGlj aCBpcw0KPj4gbWFya2VkIGFzIHVudXNlZCAoREJfVU5VU0VEKS4NCj4+DQo+PiBzdGF0aWMgaW50 IG1kYl9zZXR1cF9kYl9pbmZvKE1EQl90eG4gKnR4biwgTURCX2RiaSBkYmkpIHsNCj4+ICAgICAg LyogU2V0dXAgZGIgaW5mbyAqLw0KPj4gICAgICB1aW50MTZfdCB4ID0gdHhuLT5tdF9lbnYtPm1l X2RiZmxhZ3NbZGJpXTsNCj4+ICAgICAgdHhuLT5tdF9kYnNbZGJpXS5tZF9mbGFncyA9IHggJiBQ RVJTSVNURU5UX0ZMQUdTOw0KPj4gICAgICB0eG4tPm10X2RiZmxhZ3NbZGJpXSA9ICh4ICYgTURC X1ZBTElEKSA/IERCX1ZBTElEfERCX1VTUlZBTElEfERCX1NUQUxFIDogMDsNCj4+ICAgICAgcmV0 dXJuICh0eG4tPm10X2RiZmxhZ3NbZGJpXSAmIHZhbGlkaXR5KTsNCj4+IH0NCj4+DQo+PiAgICAg IC8qKiBDaGVjayBcYiB0eG4gYW5kIFxiIGRiaSBhcmd1bWVudHMgdG8gYSBmdW5jdGlvbiBhbmQg aW5pdGlhbGl6ZSBkYiBpbmZvDQo+PiBpZiBuZWVkZWQgKi8NCj4+ICNkZWZpbmUgVFhOX0RCSV9F WElTVCh0eG4sIGRiaSwgdmFsaWRpdHkpIFwNCj4+ICAgICAgKCh0eG4pICYmIChkYmkjQzNDKHR4 biktPm10X251bWRicyAmJiAoKCh0eG4pLT5tdF9kYmZsYWdzW2RiaV0gJiAodmFsaWRpdHkpKQ0K Pj4gfHwgKCgodHhuKS0+bXRfZGJmbGFnc1tkYmldICYgREJfVU5VU0VEKSAmJiBtZGJfc2V0dXBf ZGJfaW5mbygodHhuKSwgKGRiaSksDQo+PiAodmFsaWRpdHkpKSkpKQ0KPj4NCj4+DQo+PiBUaGUg bmV4dCBpbXByb3ZlbWVudCBpcyBkb25lIGluIGFueSBmdW5jdGlvbiB3aGljaCBuZWVkcyB0byBs b29wIHRocm91Z2ggdGhlDQo+PiBkYXRhYmFzZXMgZm9yIGV4YW1wbGUgaW4gbWRiX2N1cnNvcnNf Y2xvc2UoKS4gQWdhaW4gdGhlIG1vcmUgZGF0YWJhc2VzIGluIHRoZQ0KPj4gZW52aXJvbm1lbnQg dGhlIGxvbmdlciB0aGUgZXhlY3V0aW9uIHRpbWUuDQo+PiBJdCBzaG91bGQgYmUgYmVzdCBpZiBs b29waW5nIG9ubHkgdGhyb3VnaCBkYmZsYWdzIGFuZCBzZWFyY2hpbmcgZm9yIHRob3NlDQo+PiBk YXRhYmFzZXMgd2hpY2ggYXJlIHVzZWQgKCFEQl9VTlVTRUQpLiBUaGlzIGNvdWxkIGJlIGRvbmUg Ynl0ZSB3aXNlIG9yIG1vcmUNCj4+IGVmZmljaWVudCBpbiA4LzQgYnl0ZSBzdGVwcyBjb21wYXJp bmcgd2l0aCBhbiBleHRlbmRlZCBtYXNrIERCX1VOVVNFRF9MT05HDQo+PiBpbnN0ZWFkIG9mIERC X1VOVVNFRC4gU28gd2UgY2FuIHNraXAgOCBvciA0ICgzMiBiaXQpIHVudXNlZCBkYXRhYmFzZXMg aW4gb25lDQo+PiBzdGVwIChzdGlsbCB3aXRoIHRoZSBhc3N1bXB0aW9uIHRoYXQgYSB0cmFuc2Fj dGlvbiByYXJlbHkgdXNlcyBhbGwgZGF0YWJhc2VzIG9mDQo+PiB0aGUgZW52aXJvbm1lbnQpLg0K Pj4NCj4+IFNvIHRoZSBsb29wIGxvb2tzIGFzIGZvbGxvd3MgYWx3YXlzIHN0YXJ0aW5nIGF0IHRo ZSBsb3dlciBpbmRleCB0byBhdm9pZA0KPj4gYWxpZ25tZW50IGlzc3VlcyB3aXRoIEFSTSBwcmlv ciB2Ni4NCj4+DQo+PiAjZGVmaW5lIERCX1VOVVNFRCAgICAgICAweDIwICAgICAgICAgICAgICAg IC8qKjwgREIgbm90IHVzZWQgaW4gdGhpcyB0eG4gKi8NCj4+DQo+PiAjaWZkZWYgTURCX1ZMMzIN Cj4+ICNkZWZpbmUgREJfVU5VU0VEX0xPTkcgIDB4MjAyMDIwMjAgICAgICAgICAgLyogREJfVU5V U0VEIGxvbmcgbWFzayBmb3IgZmFzdA0KPj4gdHJhY2tpbmcgKi8NCj4+ICNlbHNlDQo+PiAjZGVm aW5lIERCX1VOVVNFRF9MT05HICAweDIwMjAyMDIwMjAyMDIwMjAgIC8qIERCX1VOVVNFRCBsb25n IG1hc2sgZm9yIGZhc3QNCj4+IHRyYWNraW5nICovDQo+PiAjZW5kaWYNCj4+DQo+Pg0KPj4gI2lm ZGVmIE1EQl9WTDMyDQo+PiAjZGVmaW5lIE1EQl9XT1JEICAgIHVuc2lnbmVkIGludA0KPj4gI2Vs c2UNCj4+ICNkZWZpbmUgTURCX1dPUkQgICAgdW5zaWduZWQgbG9uZyBsb25nDQo+PiAjZW5kaWYN Cj4+DQo+Pg0KPj4NCj4+ICAgICAgTURCX2RiaSAlMz0gc3JjLT5tdF9udW1kYnM7DQo+PiAgICAg IE1EQl9kYmkgaSA9IDA7DQo+Pg0KPj4gICAgICB3aGlsZSAoMSkgew0KPj4gICAgICAgICAgdW5z aWduZWQgaW50IHVwcGVyID0gaSArIHNpemVvZihNREJfV09SRCk7DQo+PiAgICAgICAgICBpZiAo dXBwZXIgPCBuKSB7DQo+PiAgICAgICAgICAgICAgLy8gc2tpcCB1bnVzZWQNCj4+ICAgICAgICAg ICAgICBpZiAoKCooTURCX1dPUkQgKikodGRiZmxhZ3MgKyBpKSkgPT0gREJfVU5VU0VEX0xPTkcp IHsNCj4+ICAgICAgICAgICAgICAgICAgaSA9IHVwcGVyOw0KPj4gICAgICAgICAgICAgICAgICBj b250aW51ZTsNCj4+ICAgICAgICAgICAgICB9DQo+PiAgICAgICAgICB9DQo+PiAgICAgICAgICBl bHNlIHsNCj4+ICAgICAgICAgICAgICB1cHBlciA9IG47DQo+PiAgICAgICAgICB9DQo+Pg0KPj4g ICAgICAgICAgZm9yICg7IGkgPCB1cHBlcjsgaSsrKSB7DQo+PiAgICAgICAgICAgICAgLy8gYW55 IG90aGVyIGZpbHRlciBjcml0ZXJpYSBhcHByb3ByaWF0ZSB0byB0aGUgZnVuY3Rpb24NCj4+ICAg ICAgICAgICAgICAuLi4uDQo+PiAgICAgICAgICB9DQo+PiAgICAgICAgICBpZiAoaSA+PSBuKSB7 DQo+PiAgICAgICAgICAgICAgYnJlYWs7DQo+PiAgICAgICAgICB9DQo+PiAgICAgIH0NCj4+DQo+ Pg0KPj4NCj4NCj4NCj4tLSANCj4gICAtLSBIb3dhcmQgQ2h1DQo+ICAgQ1RPLCBTeW1hcyBDb3Jw LiAgICAgICAgICAgaHR0cDovL3d3dy5zeW1hcy5jb20NCj4gICBEaXJlY3RvciwgSGlnaGxhbmQg U3VuICAgICBodHRwOi8vaGlnaGxhbmRzdW4uY29tL2h5Yy8NCj4gICBDaGllZiBBcmNoaXRlY3Qs IE9wZW5MREFQICBodHRwOi8vd3d3Lm9wZW5sZGFwLm9yZy9wcm9qZWN0Lw0K

1 0

Re: (ITS#8430) Improved handling for large number of databases
by h.b.furuseth＠usit.uio.no 27 May '16

27 May '16

On 27. mai 2016 12:57, juerg.bircher(a)helmedica.com wrote: > mdb_dbi_open() is however not improved with the assumption that the database > handle (dbi) is cached in the application. So mdb_dbi_open() should happen only > once for each database during the life time of an application. It must check if the DB is open anyway, since LDMB breaks if the user has two DBIs to the same DB. Though maybe we could add an optimization flag to skip the search, for users who know they don't reopen open DBs. Other notes: LMDB and OpenLDAP mostly use tab-width = indentation = 4. (We should comment that somewhere.) You seem to use tab-width = 8. I think MDB_WORD intends to be mdb_size_t, but should be 'unsigned long': It's probably no optimization to use 'long long' when that is not a native integer type. But I haven't profiled that. The delay in picking up env->me_dbflags is a bit troubling, since it means creating a new txn is no longer quite atomic. I can't think of how it'll make a difference when DBIs are used correctly, but that's one thing people screw up. Maybe it'd be OK if renew0() just copies it into mt_dbs[].md_flags without the '&', and mdb_setup_db_info() fixes it up. -- Hallvard

1 0

Re: (ITS#8430) Improved handling for large number of databases
by hyc＠symas.com 27 May '16

27 May '16

juerg.bircher(a)helmedica.com wrote: > Full_Name: Juerg Bircher > Version: lmdb (master) > OS: MacOS / Linux > URL: ftp://ftp.openldap.org/incoming/Juerg_Bircher_160527-Improved-handling-for-… > Submission from: (NULL) (178.82.37.195) Thanks for the patch, but for a change of this size you also need to include an IP Rights notice, as documented on the Contributing guidelines. > > > There is a increased performance penalty the more databases are created within > the same environment. I was looking for a way the improve that by keeping the > simplicity of tracking databases within a list with direct access by index > (MDB_dbi). > mdb_dbi_open() is however not improved with the assumption that the database > handle (dbi) is cached in the application. So mdb_dbi_open() should happen only > once for each database during the life time of an application. > > > One issue is that mdb_txn_begin() (for read-only transactions) calloc the > sizeof(MDB_txn) + me_maxdbs * sizeof(MDB_db + 1). The plus 1 for the dbflags. > However it is sufficient only to malloc that size and clear the sizeof(MDB_txn) > > memset(txn, 0, sizeof(MDB_txn) > > After that the data beyond the MDB_txn is not initialized which is ok for the > moment. > > The next improvement happens in mdb_txn_renew0() where the dbflags are only set > to DB_UNUSED (a new flag) for each database currently opened in the > environment. > > memset(txn->mt_dbflags, DB_UNUSED, txn->mt_numdbs); > > The former code used to to loop through each database to calculate the dbflags. > This is still done but lazily for each accessed database with the assumption > that a read only transaction rarely uses all databases of the environment. > > The lazy initialization of the dbflag happens in the macro TXN_DBI_EXIST which > is always used when a database handle (dbi) is passed to an function. > The flags are updated in mdb_setup_db_info() once a database is access which is > marked as unused (DB_UNUSED). > > static int mdb_setup_db_info(MDB_txn *txn, MDB_dbi dbi) { > /* Setup db info */ > uint16_t x = txn->mt_env->me_dbflags[dbi]; > txn->mt_dbs[dbi].md_flags = x & PERSISTENT_FLAGS; > txn->mt_dbflags[dbi] = (x & MDB_VALID) ? DB_VALID|DB_USRVALID|DB_STALE : 0; > return (txn->mt_dbflags[dbi] & validity); > } > > /** Check \b txn and \b dbi arguments to a function and initialize db info > if needed */ > #define TXN_DBI_EXIST(txn, dbi, validity) \ > ((txn) && (dbi#C3C(txn)->mt_numdbs && (((txn)->mt_dbflags[dbi] & (validity)) > || (((txn)->mt_dbflags[dbi] & DB_UNUSED) && mdb_setup_db_info((txn), (dbi), > (validity))))) > > > The next improvement is done in any function which needs to loop through the > databases for example in mdb_cursors_close(). Again the more databases in the > environment the longer the execution time. > It should be best if looping only through dbflags and searching for those > databases which are used (!DB_UNUSED). This could be done byte wise or more > efficient in 8/4 byte steps comparing with an extended mask DB_UNUSED_LONG > instead of DB_UNUSED. So we can skip 8 or 4 (32 bit) unused databases in one > step (still with the assumption that a transaction rarely uses all databases of > the environment). > > So the loop looks as follows always starting at the lower index to avoid > alignment issues with ARM prior v6. > > #define DB_UNUSED 0x20 /**< DB not used in this txn */ > > #ifdef MDB_VL32 > #define DB_UNUSED_LONG 0x20202020 /* DB_UNUSED long mask for fast > tracking */ > #else > #define DB_UNUSED_LONG 0x2020202020202020 /* DB_UNUSED long mask for fast > tracking */ > #endif > > > #ifdef MDB_VL32 > #define MDB_WORD unsigned int > #else > #define MDB_WORD unsigned long long > #endif > > > > MDB_dbi %3= src->mt_numdbs; > MDB_dbi i = 0; > > while (1) { > unsigned int upper = i + sizeof(MDB_WORD); > if (upper < n) { > // skip unused > if ((*(MDB_WORD *)(tdbflags + i)) == DB_UNUSED_LONG) { > i = upper; > continue; > } > } > else { > upper = n; > } > > for (; i < upper; i++) { > // any other filter criteria appropriate to the function > .... > } > if (i >= n) { > break; > } > } > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8431) LMDB: Access newly opened database from another transaction
by juerg.bircher＠helmedica.com 27 May '16

27 May '16

Full_Name: Jeerg Bircher Version: lmdb (master) OS: MacOS / Linux URL: ftp://ftp.openldap.org/incoming/Juerg_Bircher_160527-Access-newly-opened-da… Submission from: (NULL) (178.82.37.195) A transaction tracks newly opened databases and if the transaction is committed the newly opened databases are propagated to the list of open databases of the environment. However if the read only transaction is aborted the databases are not propagated. If database handles (MDB_dbi) are cached in the application to avoid calling mdb_dbi_open() there might be the situation of two threads running a read only transaction concurrently. First threads opens the database and commits the read-only transaction. The database is added to the list of open databases in the environment. The returned dbi is globally cached in the application. The second thread also wants to access the same database and finds the database handle in the global application cache. The database handle however is not valid as the transaction only uses a snapshot of open databases. So the second thread gets an EINVAL when using that database handle. This should not happen as the database is open and added to the environment. The following should fix this issue by updating the mt_numdbs and marking the delta with DB_UNUSED static int mdb_update_db_info(MDB_txn *txn, MDB_dbi dbi) { /* propagate newly opened db from env to current txn. Mark them as unused */ if (txn->mt_numdbs < txn->mt_env->me_numdbs) { memset(txn->mt_dbflags + txn->mt_numdbs, DB_UNUSED, txn->mt_env->me_numdbs - txn->mt_numdbs); } txn->mt_numdbs = txn->mt_env->me_numdbs; return dbi < txn->mt_numdbs; } /** Check \b txn and \b dbi arguments to a function and initialize db info if needed */ #define TXN_DBI_EXIST(txn, dbi, validity) \ ((txn) && ((dbi)<(txn)->mt_numdbs || mdb_update_db_info((txn), (dbi))) && (((txn)->mt_dbflags[dbi] & (validity)) || (((txn)->mt_dbflags[dbi] & DB_UNUSED) && mdb_setup_db_info((txn), (dbi), (validity))))) IMPORTANT depends ITS#8430 See also mailing list openldap-technical thread: Improved handling for large number of databases / Access newly opened database from another transaction

1 0

(ITS#8430) Improved handling for large number of databases
by juerg.bircher＠helmedica.com 27 May '16

27 May '16

Full_Name: Juerg Bircher Version: lmdb (master) OS: MacOS / Linux URL: ftp://ftp.openldap.org/incoming/Juerg_Bircher_160527-Improved-handling-for-… Submission from: (NULL) (178.82.37.195) There is a increased performance penalty the more databases are created within the same environment. I was looking for a way the improve that by keeping the simplicity of tracking databases within a list with direct access by index (MDB_dbi). mdb_dbi_open() is however not improved with the assumption that the database handle (dbi) is cached in the application. So mdb_dbi_open() should happen only once for each database during the life time of an application. One issue is that mdb_txn_begin() (for read-only transactions) calloc the sizeof(MDB_txn) + me_maxdbs * sizeof(MDB_db + 1). The plus 1 for the dbflags. However it is sufficient only to malloc that size and clear the sizeof(MDB_txn) memset(txn, 0, sizeof(MDB_txn) After that the data beyond the MDB_txn is not initialized which is ok for the moment. The next improvement happens in mdb_txn_renew0() where the dbflags are only set to DB_UNUSED (a new flag) for each database currently opened in the environment. memset(txn->mt_dbflags, DB_UNUSED, txn->mt_numdbs); The former code used to to loop through each database to calculate the dbflags. This is still done but lazily for each accessed database with the assumption that a read only transaction rarely uses all databases of the environment. The lazy initialization of the dbflag happens in the macro TXN_DBI_EXIST which is always used when a database handle (dbi) is passed to an function. The flags are updated in mdb_setup_db_info() once a database is access which is marked as unused (DB_UNUSED). static int mdb_setup_db_info(MDB_txn *txn, MDB_dbi dbi) { /* Setup db info */ uint16_t x = txn->mt_env->me_dbflags[dbi]; txn->mt_dbs[dbi].md_flags = x & PERSISTENT_FLAGS; txn->mt_dbflags[dbi] = (x & MDB_VALID) ? DB_VALID|DB_USRVALID|DB_STALE : 0; return (txn->mt_dbflags[dbi] & validity); } /** Check \b txn and \b dbi arguments to a function and initialize db info if needed */ #define TXN_DBI_EXIST(txn, dbi, validity) \ ((txn) && (dbi#C3C(txn)->mt_numdbs && (((txn)->mt_dbflags[dbi] & (validity)) || (((txn)->mt_dbflags[dbi] & DB_UNUSED) && mdb_setup_db_info((txn), (dbi), (validity))))) The next improvement is done in any function which needs to loop through the databases for example in mdb_cursors_close(). Again the more databases in the environment the longer the execution time. It should be best if looping only through dbflags and searching for those databases which are used (!DB_UNUSED). This could be done byte wise or more efficient in 8/4 byte steps comparing with an extended mask DB_UNUSED_LONG instead of DB_UNUSED. So we can skip 8 or 4 (32 bit) unused databases in one step (still with the assumption that a transaction rarely uses all databases of the environment). So the loop looks as follows always starting at the lower index to avoid alignment issues with ARM prior v6. #define DB_UNUSED 0x20 /**< DB not used in this txn */ #ifdef MDB_VL32 #define DB_UNUSED_LONG 0x20202020 /* DB_UNUSED long mask for fast tracking */ #else #define DB_UNUSED_LONG 0x2020202020202020 /* DB_UNUSED long mask for fast tracking */ #endif #ifdef MDB_VL32 #define MDB_WORD unsigned int #else #define MDB_WORD unsigned long long #endif MDB_dbi %3= src->mt_numdbs; MDB_dbi i = 0; while (1) { unsigned int upper = i + sizeof(MDB_WORD); if (upper < n) { // skip unused if ((*(MDB_WORD *)(tdbflags + i)) == DB_UNUSED_LONG) { i = upper; continue; } } else { upper = n; } for (; i < upper; i++) { // any other filter criteria appropriate to the function .... } if (i >= n) { break; } }

1 0

Re: (ITS#8427) Incorrect value of tls_reqcert in syncrepl
by mkp37215＠gmail.com 20 May '16

20 May '16

Howard, thanks for the reply. I just noticed a small error in what I wrote, the corrected fragment should be: "This was because sb->sb_tls_do_init was FALSE and bindconf_tls_set(sb, ld) was not called." I also would like to add that my patch changes the semantics of bindconf_tls_set, in regard of how TLS context is set, and that this is deliberate. I think that previous semantics was unclear and bug-prone, and that the new one is not only more straightforward, but also matches better the way bindconf_tls_set is used. As a result both bindconf_tls_set code and the code around its invocations is simplified. However, I was focused on its usage in slap_client_connect (because this is what was causing me problems), and I did not put much attention into other three places where bindconf_tls_set is called. All of those code fragments were basically identical, so I modified them the same way, but I think someone should review these modifications to see if they make sense. I originally intended to limit the impact of my patch to slap_client_connect, and to keep the changes inside config.c file. However, this resulted in making bad code worse, even less clear and manageable.

1 0

Re: (ITS#8427) Incorrect value of tls_reqcert in syncrepl
by hyc＠symas.com 19 May '16

19 May '16

mkp37215(a)gmail.com wrote: > The root cause of the problem: Thanks for the analysis, will look into it soon. > On syncrepl retry, code in slap_client_connect > (servers/slapd/config.c) initialized ld (LDAP) object with default > values, rather than values saved in sb (slap_bindconf). This was > because sb->sb_tls_do_init was true and bindconf_tls_set(sb, ld) was > not called. > This issue is not related to any specific TLS library. All syncrepl > retries are executed with wrong TLS settings in ld->ld_options, > including ldo_tls_require_cert, ldo_tls_keyfile, ldo_tls_certfile, > ldo_tls_cacertfile, and other (ldo_tls_require_cert has default value > 2 (demand), while string fields have nulls). However, the bug is > usually masked, because sb->sb_tls_ctx contains TLS context correctly > initialized during initial connection attempt, and this context is > assigned to ld as LDAP_OPT_X_TLS_CTX, and then used by the TLS > library, while incorrect values in ld->ld_options are ignored in most > places. But the host-checking code in ldap_int_tls_start > (libraries/libldap/tls2.c) uses ld->ld_options and triggers the bug. That description makes sense, OK. > Explanation of the patch: > The patch modifies function bindconf_tls_set that copies values from > slap_bindconf to LDAP object. Code that calls bindconf_tls_set is also > modified. > 1. If slap_bindconf object has TLS context, assign it to LDAP object. > 2. if slap_bindconf does not have a TLS context, create new context > and assign it it to both slap_bindconf and LDAP objects. > 3. Removed context replacement functionality and related conditions > (newctx). The intent of this functionality was unclear, but did not > appear to be compatible with how binconf_tls_set was used. > 4. The result of the above code changes is that TLS context is always > present and correctly assigned, while previously it sometimes was, and > sometimes wasn't. > 5. Simplified calls of bindconf_tls_set in the following files: > servers/slapd/config.c > servers/slapd/back-ldap/bind.c > servers/slapd/back-meta/conn.c > servers/slapd/back-asyncmeta/conn.c > > Things to do: > I tested the modified slap_client_connect (servers/slapd/config.c), > but the modifications to files in back-{ldap,meta,asyncmeta} were done > without proper code and functionality analysis, and without any > testing. I am hoping for a review from someone knowledgeable. > > Thank you > > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8427) Incorrect value of tls_reqcert in syncrepl
by mkp37215＠gmail.com 19 May '16

19 May '16

The root cause of the problem: On syncrepl retry, code in slap_client_connect (servers/slapd/config.c) initialized ld (LDAP) object with default values, rather than values saved in sb (slap_bindconf). This was because sb->sb_tls_do_init was true and bindconf_tls_set(sb, ld) was not called. This issue is not related to any specific TLS library. All syncrepl retries are executed with wrong TLS settings in ld->ld_options, including ldo_tls_require_cert, ldo_tls_keyfile, ldo_tls_certfile, ldo_tls_cacertfile, and other (ldo_tls_require_cert has default value 2 (demand), while string fields have nulls). However, the bug is usually masked, because sb->sb_tls_ctx contains TLS context correctly initialized during initial connection attempt, and this context is assigned to ld as LDAP_OPT_X_TLS_CTX, and then used by the TLS library, while incorrect values in ld->ld_options are ignored in most places. But the host-checking code in ldap_int_tls_start (libraries/libldap/tls2.c) uses ld->ld_options and triggers the bug. Explanation of the patch: The patch modifies function bindconf_tls_set that copies values from slap_bindconf to LDAP object. Code that calls bindconf_tls_set is also modified. 1. If slap_bindconf object has TLS context, assign it to LDAP object. 2. if slap_bindconf does not have a TLS context, create new context and assign it it to both slap_bindconf and LDAP objects. 3. Removed context replacement functionality and related conditions (newctx). The intent of this functionality was unclear, but did not appear to be compatible with how binconf_tls_set was used. 4. The result of the above code changes is that TLS context is always present and correctly assigned, while previously it sometimes was, and sometimes wasn't. 5. Simplified calls of bindconf_tls_set in the following files: servers/slapd/config.c servers/slapd/back-ldap/bind.c servers/slapd/back-meta/conn.c servers/slapd/back-asyncmeta/conn.c Things to do: I tested the modified slap_client_connect (servers/slapd/config.c), but the modifications to files in back-{ldap,meta,asyncmeta} were done without proper code and functionality analysis, and without any testing. I am hoping for a review from someone knowledgeable. Thank you

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs May 2016