openldap-bugs

openldap-bugs@openldap.org

1 participants
20959 discussions

Re: (ITS#7975) MDB search scope one and filter returns search base
by hyc＠symas.com 29 Oct '14

29 Oct '14

requate(a)univention.de wrote: > Full_Name: Arvid Requate > Version: 2.4.40 > OS: Debian / UCS > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (82.198.197.8) > > > With an mdb backend the command > > ldapsearch -xLLL -b dc=ar40i1,dc=qa -s one objectClass=domain dn > > returns the base DN. With a bdb backend it doesn't. > > The crucial point seems to be the filter in this case, which matches the base > object only, and none of it's children. If I change the filter to > objectclass=top (or leave it away) then only the children are returned, just > like the behaviour of the bdb backend. Fixed now in git master. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#7975) MDB search scope one and filter returns search base
by requate＠univention.de 29 Oct '14

29 Oct '14

Full_Name: Arvid Requate Version: 2.4.40 OS: Debian / UCS URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (82.198.197.8) With an mdb backend the command ldapsearch -xLLL -b dc=ar40i1,dc=qa -s one objectClass=domain dn returns the base DN. With a bdb backend it doesn't. The crucial point seems to be the filter in this case, which matches the base object only, and none of it's children. If I change the filter to objectclass=top (or leave it away) then only the children are returned, just like the behaviour of the bdb backend.

1 0

Re: (ITS#7974) LDBM's "laggard reader" flaw still present, in continue of ITS#7904
by leo＠yuriev.ru 23 Oct '14

23 Oct '14

I assume ITS#7830 is the same issue.

1 0

Re: (ITS#7830) MDB Size grown a lot during massive updates and concurrent read access
by leo＠yuriev.ru 23 Oct '14

23 Oct '14

Seems to will be fixed by ITS#7974.

1 0

Re: (ITS#7974) LDBM's "laggard reader" flaw still present, in continue of ITS#7904
by leo＠yuriev.ru 23 Oct '14

23 Oct '14

This is a multi-part message in MIME format. --------------060806060106060808080102 Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit The attached files is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Peter-Service LLC, Moscow, Russia. Peter-Service LLC has not assigned rights and/or interest in this work to any party. I, Leonid Yuriev am authorized by Peter-Service LLC, my employer, to release this work under the following terms. Peter-Service LLC hereby places the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice. --------------060806060106060808080102 Content-Type: text/x-patch; name="0001-lmdb-ITS-7974-a-reading-lag-for-dreamcatcher.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0001-lmdb-ITS-7974-a-reading-lag-for-dreamcatcher.patch" >From a30ece9b236b7217481c086fd27b133bd3404317 Mon Sep 17 00:00:00 2001 From: Leo Yuriev <leo(a)yuriev.ru> Date: Tue, 21 Oct 2014 15:34:22 +0400 Subject: [PATCH 1/2] lmdb: ITS#7974 a reading lag for dreamcatcher. --- libraries/liblmdb/lmdb.h | 11 +++++++++++ libraries/liblmdb/mdb.c | 20 ++++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/libraries/liblmdb/lmdb.h b/libraries/liblmdb/lmdb.h index a3ca62e..82eff14 100644 --- a/libraries/liblmdb/lmdb.h +++ b/libraries/liblmdb/lmdb.h @@ -1571,6 +1571,17 @@ void mdb_env_set_oomkiller(MDB_env *env, MDB_oomkiller_func *oomkiller); * @return A #MDB_oomkiller_func function or NULL if disabled. */ MDB_oomkiller_func* mdb_env_get_oomkiller(MDB_env *env); + + /** @brief Returns a reading lag. + * + * Returns an information for estimate how much given read-only + * transaction is lagging relative the to actual head. + * + * @param[in] txn A transaction handle returned by #mdb_txn_begin() + * @param[out] percent Percentage of page allocation in the database. + * @return Number of transactions committed after the given was started for read, or -1 on failure. + */ +int mdb_txn_straggler(MDB_txn *txnm, int *percent); /** @} */ #ifdef __cplusplus diff --git a/libraries/liblmdb/mdb.c b/libraries/liblmdb/mdb.c index e60d83d..a417c9b 100644 --- a/libraries/liblmdb/mdb.c +++ b/libraries/liblmdb/mdb.c @@ -2823,6 +2823,26 @@ mdb_dbis_update(MDB_txn *txn, int keep) env->me_numdbs = n; } +int +mdb_txn_straggler(MDB_txn *txn, int *percent) +{ + MDB_env *env; + MDB_meta *meta; + txnid_t lag; + + if (! txn || ! txn->mt_u.reader) + return -1; + + env = txn->mt_env; + meta = env->me_metas[ mdb_env_pick_meta(env) ]; + if (percent) { + long cent = env->me_maxpg / 100; + *percent = (meta->mm_last_pg + cent / 2 + 1) / (cent ? cent : 1); + } + lag = meta->mm_txnid - txn->mt_u.reader->mr_txnid; + return (0 > (int) lag) ? ~0u >> 1: lag; +} + /** Common code for #mdb_txn_reset() and #mdb_txn_abort(). * May be called twice for readonly txns: First reset it, then abort. * @param[in] txn the transaction handle to reset -- 2.1.0 --------------060806060106060808080102 Content-Type: text/x-patch; name="0002-slapd-ITS-7974-dreamcatcher-feature.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-slapd-ITS-7974-dreamcatcher-feature.patch" >From 133cec8eadc93fe3083d110e58259ba2a067908c Mon Sep 17 00:00:00 2001 From: Leo Yuriev <leo(a)yuriev.ru> Date: Tue, 21 Oct 2014 15:38:28 +0400 Subject: [PATCH 2/2] slapd: ITS#7974 dreamcatcher feature. --- servers/slapd/back-mdb/config.c | 46 +++++++++++++++++++++++++++++- servers/slapd/back-mdb/search.c | 62 ++++++++++++++++++++++------------------- 2 files changed, 79 insertions(+), 29 deletions(-) diff --git a/servers/slapd/back-mdb/config.c b/servers/slapd/back-mdb/config.c index b54da49..65034b1 100644 --- a/servers/slapd/back-mdb/config.c +++ b/servers/slapd/back-mdb/config.c @@ -39,7 +39,8 @@ enum { MDB_MAXREADERS, MDB_MAXSIZE, MDB_MODE, - MDB_SSTACK + MDB_SSTACK, + MDB_DREAMCATCHER }; static ConfigTable mdbcfg[] = { @@ -74,6 +75,10 @@ static ConfigTable mdbcfg[] = { mdb_cf_gen, "( OLcfgDbAt:12.2 NAME 'olcDbMaxSize' " "DESC 'Maximum size of DB in bytes' " "SYNTAX OMsInteger SINGLE-VALUE )", NULL, NULL }, + { "dreamcatcher", "lag> <percentage", 3, 3, 0, ARG_MAGIC|MDB_DREAMCATCHER, + mdb_cf_gen, "( OLcfgDbAt:12.4 NAME 'olcDbDreamcatcher' " + "DESC 'Dreamcatcher to avoids withhold of reclaiming' " + "SYNTAX OMsDirectoryString SINGLE-VALUE )",NULL, NULL }, { "mode", "mode", 2, 2, 0, ARG_MAGIC|MDB_MODE, mdb_cf_gen, "( OLcfgDbAt:0.3 NAME 'olcDbMode' " "DESC 'Unix permissions of database files' " @@ -319,6 +324,23 @@ mdb_cf_gen( ConfigArgs *c ) } break; + case MDB_DREAMCATCHER: + if ( mdb->mi_renew_lag ) { + char buf[64]; + struct berval bv; + bv.bv_len = snprintf( buf, sizeof(buf), "%ld %ld", + (long) mdb->mi_renew_lag, (long) mdb->mi_renew_percent ); + if ( bv.bv_len > 0 && bv.bv_len < sizeof(buf) ) { + bv.bv_val = buf; + value_add_one( &c->rvalue_vals, &bv ); + } else { + rc = 1; + } + } else { + rc = 1; + } + break; + case MDB_DIRECTORY: if ( mdb->mi_dbenv_home ) { c->value_string = ch_strdup( mdb->mi_dbenv_home ); @@ -389,6 +411,10 @@ mdb_cf_gen( ConfigArgs *c ) } mdb->mi_txn_cp = 0; break; + case MDB_DREAMCATCHER: + mdb->mi_renew_lag = 0; + mdb->mi_renew_percent = 0; + break; case MDB_DIRECTORY: mdb->mi_flags |= MDB_RE_OPEN; ch_free( mdb->mi_dbenv_home ); @@ -574,6 +600,24 @@ mdb_cf_gen( ConfigArgs *c ) } } break; + case MDB_DREAMCATCHER: { + long l; + if ( lutil_atolx( &l, c->argv[1], 0 ) != 0 || l < 1 ) { + fprintf( stderr, "%s: " + "invalid lag \"%s\" in \"dreamcatcher\".\n", + c->log, c->argv[1] ); + return 1; + } + mdb->mi_renew_lag = l; + if ( lutil_atolx( &l, c->argv[2], 0 ) != 0 || l < 0 || l > 100 ) { + fprintf( stderr, "%s: " + "invalid percentage \"%s\" in \"dreamcatcher\".\n", + c->log, c->argv[2] ); + return 1; + } + mdb->mi_renew_percent = l; + } break; + case MDB_DIRECTORY: { FILE *f; char *ptr, *testpath; diff --git a/servers/slapd/back-mdb/search.c b/servers/slapd/back-mdb/search.c index 98b5f25..381cb80 100644 --- a/servers/slapd/back-mdb/search.c +++ b/servers/slapd/back-mdb/search.c @@ -340,22 +340,29 @@ typedef struct ww_ctx { * case return an LDAP_BUSY error - let the client know this search * couldn't succeed, but might succeed on a retry. */ + +static void +mdb_befree( Operation *op, ww_ctx *ww ) +{ + assert(! ww->flag); + if ( ww->mcd ) { + MDB_val key, data; + mdb_cursor_get( ww->mcd, &key, &data, MDB_GET_CURRENT ); + memcpy( &ww->key, key.mv_data, sizeof(ID) ); + ww->data.mv_size = data.mv_size; + ww->data.mv_data = op->o_tmpalloc( data.mv_size, op->o_tmpmemctx ); + memcpy(ww->data.mv_data, data.mv_data, data.mv_size); + } + mdb_txn_reset( ww->txn ); + ww->flag = 1; +} + static void mdb_writewait( Operation *op, slap_callback *sc ) { ww_ctx *ww = sc->sc_private; - if ( !ww->flag ) { - if ( ww->mcd ) { - MDB_val key, data; - mdb_cursor_get( ww->mcd, &key, &data, MDB_GET_CURRENT ); - memcpy( &ww->key, key.mv_data, sizeof(ID) ); - ww->data.mv_size = data.mv_size; - ww->data.mv_data = op->o_tmpalloc( data.mv_size, op->o_tmpmemctx ); - memcpy(ww->data.mv_data, data.mv_data, data.mv_size); - } - mdb_txn_reset( ww->txn ); - ww->flag = 1; - } + if ( !ww->flag ) + mdb_befree( op, ww ); } static int @@ -796,7 +803,6 @@ loop_begin: goto done; } - if ( nsubs < ncand ) { unsigned i; /* Is this entry in the candidate list? */ @@ -1034,14 +1040,6 @@ notfound: ber_bvarray_free( erefs ); rs->sr_ref = NULL; - if ( wwctx.flag ) { - rs->sr_err = mdb_waitfixup( op, &wwctx, mci, mcd ); - if ( rs->sr_err ) { - send_ldap_result( op, rs ); - goto done; - } - } - goto loop_continue; } @@ -1096,13 +1094,6 @@ notfound: } goto done; } - if ( wwctx.flag ) { - rs->sr_err = mdb_waitfixup( op, &wwctx, mci, mcd ); - if ( rs->sr_err ) { - send_ldap_result( op, rs ); - goto done; - } - } } } else { @@ -1113,6 +1104,21 @@ notfound: } loop_continue: + if ( ! wwctx.flag && mdb->mi_renew_lag ) { + int percentage, lag = mdb_txn_straggler( ltid, &percentage ); + if ( lag >= mdb->mi_renew_lag && percentage >= mdb->mi_renew_percent ) { + Debug( LDAP_DEBUG_TRACE, "dreamcather: lag %d, percentage %u%%\n", lag, percentage, 0 ); + mdb_befree( op, &wwctx ); + } + } + if ( wwctx.flag ) { + rs->sr_err = mdb_waitfixup( op, &wwctx, mci, mcd ); + if ( rs->sr_err ) { + send_ldap_result( op, rs ); + goto done; + } + } + if( e != NULL ) { if ( e != base ) mdb_entry_return( op, e ); -- 2.1.0 --------------060806060106060808080102--

1 0

Re: (ITS#7974) LDBM's "laggard reader" flaw still present, in continue of ITS#7904
by leo＠yuriev.ru 23 Oct '14

23 Oct '14

This is a multi-part message in MIME format. --------------000000040604020102000407 Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit The attached files is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Peter-Service LLC, Moscow, Russia. Peter-Service LLC has not assigned rights and/or interest in this work to any party. I, Leonid Yuriev am authorized by Peter-Service LLC, my employer, to release this work under the following terms. Peter-Service LLC hereby places the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice. --------------000000040604020102000407 Content-Type: text/x-patch; name="0001-lmdb-ITS-7974-oomkiller-feature.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0001-lmdb-ITS-7974-oomkiller-feature.patch" >From 85fce95eaa0e71ee43625ccc202c173f7d4acb4a Mon Sep 17 00:00:00 2001 From: Leo Yuriev <leo(a)yuriev.ru> Date: Tue, 21 Oct 2014 19:25:32 +0400 Subject: [PATCH 1/2] lmdb: ITS#7974 oomkiller feature. --- libraries/liblmdb/lmdb.h | 34 +++++++++++++++++ libraries/liblmdb/mdb.c | 95 ++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 126 insertions(+), 3 deletions(-) diff --git a/libraries/liblmdb/lmdb.h b/libraries/liblmdb/lmdb.h index bdbb0b9..a3ca62e 100644 --- a/libraries/liblmdb/lmdb.h +++ b/libraries/liblmdb/lmdb.h @@ -1537,6 +1537,40 @@ int mdb_reader_list(MDB_env *env, MDB_msg_func *func, void *ctx); * @return 0 on success, non-zero on failure. */ int mdb_reader_check(MDB_env *env, int *dead); + + /** @brief A callback function for killing a laggard readers, + * called in case of MDB_MAP_FULL error. + * + * @param[in] env An environment handle returned by #mdb_env_create(). + * @param[in] pid pid of the reader process. + * @param[in] thread_id thread_id of the reader thread. + * @param[in] txn Transaction number on which stalled. + * @return -1 on failure (reader is not killed), + * 0 on a race condition (no such reader), + * 1 on success (reader was killed), + * >1 on success (reader was SURE killed). + */ +typedef int (MDB_oomkiller_func)(MDB_env *env, int pid, void* thread_id, size_t txn); + + /** @brief Set the oomkiller callback. + * + * Callback will be called only on out-of-pages case for killing + * a laggard readers to allowing reclaiming of freeDB. + * + * @param[in] env An environment handle returned by #mdb_env_create(). + * @param[in] oomkiller A #MDB_oomkiller_func function or NULL to disable. + */ +void mdb_env_set_oomkiller(MDB_env *env, MDB_oomkiller_func *oomkiller); + + /** @brief Get the current oomkiller callback. + * + * Callback will be called only on out-of-pages case for killing + * a laggard readers to allowing reclaiming of freeDB. + * + * @param[in] env An environment handle returned by #mdb_env_create(). + * @return A #MDB_oomkiller_func function or NULL if disabled. + */ +MDB_oomkiller_func* mdb_env_get_oomkiller(MDB_env *env); /** @} */ #ifdef __cplusplus diff --git a/libraries/liblmdb/mdb.c b/libraries/liblmdb/mdb.c index 6cc3433..e60d83d 100644 --- a/libraries/liblmdb/mdb.c +++ b/libraries/liblmdb/mdb.c @@ -1145,6 +1145,7 @@ struct MDB_env { #endif void *me_userctx; /**< User-settable context */ MDB_assert_func *me_assert_func; /**< Callback for assertion failures */ + MDB_oomkiller_func *me_oomkiller; /**< Callback for killing laggard readers */ }; /** Nested transaction */ @@ -1900,6 +1901,77 @@ mdb_find_oldest(MDB_txn *txn) return oldest; } +static txnid_t +mdb_laggard_reader(MDB_env *env, int *laggard) +{ + txnid_t tail = 0; + if (laggard) + *laggard = -1; + if (env->me_txns->mti_txnid > 1) { + int i; + MDB_reader *r = env->me_txns->mti_readers; + + tail = env->me_txns->mti_txnid - 1; + for (i = env->me_txns->mti_numreaders; --i >= 0; ) { + if (r[i].mr_pid) { + txnid_t mr = r[i].mr_txnid; + if (tail > mr) { + tail = mr; + if (laggard) + *laggard = i; + } + } + } + } + + return tail; +} + +static int +mdb_oomkill_laggard(MDB_env *env) +{ + int dead, idx; + txnid_t tail = mdb_laggard_reader(env, &idx); + if (idx < 0) + return 0; + + for(;;) { + MDB_reader *r; + MDB_THR_T tid; + pid_t pid; + int rc; + + if (mdb_reader_check(env, &dead)) + break; + + if (dead && tail < mdb_laggard_reader(env, NULL)) + return 1; + + if (!env->me_oomkiller) + break; + + r = &env->me_txns->mti_readers[ idx ]; + pid = r->mr_pid; + tid = r->mr_tid; + if (r->mr_txnid != tail || pid <= 0) + continue; + + rc = env->me_oomkiller(env, pid, (void*) tid, tail); + if (rc < 0) + break; + + if (rc) { + r->mr_txnid = (txnid_t)-1; + if (rc > 1) { + r->mr_tid = 0; + r->mr_pid = 0; + } + } + } + + return tail < mdb_laggard_reader(env, NULL); +} + /** Add a page to the txn's dirty list */ static void mdb_page_dirty(MDB_txn *txn, MDB_page *mp) @@ -1978,6 +2050,7 @@ mdb_page_alloc(MDB_cursor *mc, int num, MDB_page **mp) goto fail; } +oomkill_retry:; for (op = MDB_FIRST;; op = MDB_NEXT) { MDB_val key, data; MDB_node *leaf; @@ -2073,9 +2146,11 @@ mdb_page_alloc(MDB_cursor *mc, int num, MDB_page **mp) i = 0; pgno = txn->mt_next_pgno; if (pgno + num >= env->me_maxpg) { - DPUTS("DB size maxed out"); - rc = MDB_MAP_FULL; - goto fail; + DPUTS("DB size maxed out"); + if (mdb_oomkill_laggard(env)) + goto oomkill_retry; + rc = MDB_MAP_FULL; + goto fail; } search_done: @@ -9403,4 +9478,18 @@ mdb_reader_check(MDB_env *env, int *dead) *dead = count; return MDB_SUCCESS; } + +void +mdb_env_set_oomkiller(MDB_env *env, MDB_oomkiller_func *oomkiller) +{ + if (env) + env->me_oomkiller = oomkiller; +} + +MDB_oomkiller_func* +mdb_env_get_oomkiller(MDB_env *env) +{ + return env ? env->me_oomkiller : NULL; +} + /** @} */ -- 2.1.0 --------------000000040604020102000407 Content-Type: text/x-patch; name="0002-slapd-ITS-7974-oomkiller-feature.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-slapd-ITS-7974-oomkiller-feature.patch" >From caf076698e093077fa44e490797847c3187d485b Mon Sep 17 00:00:00 2001 From: Leo Yuriev <leo(a)yuriev.ru> Date: Tue, 21 Oct 2014 19:49:05 +0400 Subject: [PATCH 2/2] slapd: ITS#7974 oomkiller feature. --- servers/slapd/back-mdb/back-mdb.h | 3 +++ servers/slapd/back-mdb/config.c | 40 ++++++++++++++++++++++++++++++++++---- servers/slapd/back-mdb/init.c | 2 ++ servers/slapd/back-mdb/proto-mdb.h | 2 ++ 4 files changed, 43 insertions(+), 4 deletions(-) diff --git a/servers/slapd/back-mdb/back-mdb.h b/servers/slapd/back-mdb/back-mdb.h index 9d5d4b1..be82da8 100644 --- a/servers/slapd/back-mdb/back-mdb.h +++ b/servers/slapd/back-mdb/back-mdb.h @@ -81,6 +81,9 @@ struct mdb_info { uint32_t mi_txn_cp_kbyte; struct re_s *mi_txn_cp_task; struct re_s *mi_index_task; + uint32_t mi_renew_lag; + uint32_t mi_renew_percent; + int mi_oomkill; mdb_monitor_t mi_monitor; diff --git a/servers/slapd/back-mdb/config.c b/servers/slapd/back-mdb/config.c index 5b402c5..b54da49 100644 --- a/servers/slapd/back-mdb/config.c +++ b/servers/slapd/back-mdb/config.c @@ -106,6 +106,8 @@ static slap_verbmasks mdb_envflags[] = { { BER_BVC("writemap"), MDB_WRITEMAP }, { BER_BVC("mapasync"), MDB_MAPASYNC }, { BER_BVC("nordahead"), MDB_NORDAHEAD }, +#define MDB_OOMKILL (MDB_NOMEMINIT << 4) + { BER_BVC("oomkill"), MDB_OOMKILL }, { BER_BVNULL, 0 } }; @@ -123,6 +125,23 @@ mdb_checkpoint( void *ctx, void *arg ) return NULL; } +/* perform killing a laggard readers */ +int +mdb_oomkiller(MDB_env *env, int pid, void* thread_id, size_t txn) +{ + if ( pid != getpid() ) { + if ( kill( pid, SIGKILL ) == 0 ) { + Debug( LDAP_DEBUG_ANY, "oomkiller: SIGKILL to pid %i\n", pid, 0, 0 ); + sched_yield(); + return 2; + } + if ( errno == ESRCH ) + return 0; + Debug( LDAP_DEBUG_ANY, "oomkiller: SIGKILL to pid %i: %s\n", pid, strerror(errno), 0 ); + } + return -1; +} + /* reindex entries on the fly */ static void * mdb_online_index( void *ctx, void *arg ) @@ -313,12 +332,15 @@ mdb_cf_gen( ConfigArgs *c ) c->value_int = 1; break; - case MDB_ENVFLAGS: - if ( mdb->mi_dbenv_flags ) { - mask_to_verbs( mdb_envflags, mdb->mi_dbenv_flags, &c->rvalue_vals ); - } + case MDB_ENVFLAGS: { + long flags = mdb->mi_dbenv_flags; + if ( mdb->mi_oomkill ) + flags |= MDB_OOMKILL; + if ( flags ) + mask_to_verbs( mdb_envflags, flags, &c->rvalue_vals ); if ( !c->rvalue_vals ) rc = 1; break; + } case MDB_INDEX: mdb_attr_index_unparse( mdb, &c->rvalue_vals ); @@ -380,6 +402,8 @@ mdb_cf_gen( ConfigArgs *c ) break; case MDB_ENVFLAGS: + mdb->mi_oomkill = 0; + mdb_env_set_oomkiller( mdb->mi_dbenv, NULL ); if ( c->valx == -1 ) { int i; for ( i=0; mdb_envflags[i].mask; i++) { @@ -596,6 +620,14 @@ mdb_cf_gen( ConfigArgs *c ) for ( i=1; i<c->argc; i++ ) { j = verb_to_mask( c->argv[i], mdb_envflags ); if ( mdb_envflags[j].mask ) { + + if ( MDB_OOMKILL == mdb_envflags[j].mask ) { + mdb->mi_oomkill = 1; + if ( mdb->mi_flags & MDB_IS_OPEN ) + mdb_env_set_oomkiller( mdb->mi_dbenv, mdb_oomkiller ); + break; + } + if ( mdb->mi_flags & MDB_IS_OPEN ) rc = mdb_env_set_flags( mdb->mi_dbenv, mdb_envflags[j].mask, 1 ); else diff --git a/servers/slapd/back-mdb/init.c b/servers/slapd/back-mdb/init.c index 1c5ab83..c7c09d4 100644 --- a/servers/slapd/back-mdb/init.c +++ b/servers/slapd/back-mdb/init.c @@ -150,6 +150,8 @@ mdb_db_open( BackendDB *be, ConfigReply *cr ) goto fail; } + mdb_env_set_oomkiller( mdb->mi_dbenv, mdb->mi_oomkill ? mdb_oomkiller : NULL); + #ifdef HAVE_EBCDIC strcpy( path, mdb->mi_dbenv_home ); __atoe( path ); diff --git a/servers/slapd/back-mdb/proto-mdb.h b/servers/slapd/back-mdb/proto-mdb.h index b6b8d7c..3ec986e 100644 --- a/servers/slapd/back-mdb/proto-mdb.h +++ b/servers/slapd/back-mdb/proto-mdb.h @@ -20,6 +20,8 @@ LDAP_BEGIN_DECL #define MDB_UCTYPE "MDB" +MDB_oomkiller_func mdb_oomkiller; + /* * attr.c */ -- 2.1.0 --------------000000040604020102000407--

1 0

LDBM's "laggard reader" flaw still present, in continue of ITS#7904
by leo＠yuriev.ru 23 Oct '14

23 Oct '14

Full_Name: Leonid Yuriev Version: 2.4.40 OS: RHEL7 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (31.130.36.33) Currently there is flaw that does not allow using OpenLDAP + LMDB in projects with high rate of updates (add/modify/delete). The root of these problems is that LMDB cannot reclaim freed pages by a presence of a "laggard reader", or in other words if they are still referenced by an active read. It should be noted, that withholding of reclaiming while the high update rate, burns free pages very quickly. Fix of the ITS#7904 significantly improves the situation, but does not solve all the problems completely. Firstly, seemingly innocuous use of something like a "mdb_stat -efff | less" can lead to the MDB_MAP_FULL and paralyze update. Second, ITS#7904 affects the syncrepl only partially. Approximately half of the "long read" operations occur without sending data to the network. Therefore, in many cases get MDB_MAP_FULL easily enough. This leads to a chain of problems and in some cases makes the replication impossible. To solve these problems, I made two simple improvements. 1) OOMKiller feature just a fuse likely Linux kernel oomkiller. In generally, in case of MDB_MAP_FULL will send the SIGKILL to a laggard reader, but not to self. On success will retry to reclaim and continue. Engaged by envflags oomkill. 2) Dreamcatcher feature really, it has caught and forced vanish our nightmares with syncrepl & MDB MAP_FULL ;) Based on ITS#7904 fix. In generally, renew read-txt when the lag from last txn is greater than a configured threshold and the percentage of pages allocated is greater than the configured value. Engaged by dreamcatcher lag percentage. Two patchsets will be attached soon.

1 0

(ITS#7973) LDAP not seeing Berkeley DB
by jfha73＠gmail.com 20 Oct '14

20 Oct '14

Full_Name: Jorge F. Hernandez Version: 2.4 OS: Mac OS X 10.10 URL: Submission from: (NULL) (74.8.80.62) Hey guys, I'm trying to build OpenLDAP from source on a Mac OS X 10.10 using Berkeley DB 4.6, but when it checks for it it says it had version 1.0, I tried building Berkeley DB using --prefix/usr and leaving it as /usr/cacal and LDAP doesn't see it. I was able to do the same on Mac OS X 10.9 so I guess it may be something related to the OS recognition (Yosemite - 10.10) just came out. Thanks.

1 0

Re: (ITS#7972) zgrim
by hyc＠symas.com 20 Oct '14

20 Oct '14

zzgrim(a)gmail.com wrote: > Full_Name: zgrim > Version: 2.4.40 > OS: Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (79.115.186.237) > > > Hello, thank you for lmdb. > I'm using lmdb via a Go wrapper and under some not-yet-clear circumstances, > under high concurrency stress testing, > I sometimes get "double free or corruption" SIGABRT related to > mdb_dbis_update(). > The setup is using the NOTLS and RDONLY flags on a 64bit 3.17.0 Arch Linux > system. > If I protect the free() inside mdb_dbis_update() (in mdb.c) with a mutex (code > inline below), the symptom disappears. > I'm sorry I can't provide a simple test case, but in my code, without the > mutex, launching some tens of test clients simultaneously trigger the > corruption fairly quick, i.e. a few minutes or even seconds. Sounds like you have not read the mdb_dbi_open() doc closely. The code in mdb_dbis_update() only triggers if you've opened new DBs in a transaction. The doc for mdb_dbi_open() specifically says you cannot call it from multiple concurrent transactions. Closing this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7958) LMDB: LIFO-reclaiming, write-performance improvement & bugfixes
by leo＠yuriev.ru 19 Oct '14

19 Oct '14

>>> We are using the combination: >>> envflags writemap nosync lifo >>> checkpoint 0 1 >>> >>> If the checkpoint is set in seconds, it gives us the assurance >>> consistent state database on disk. >>> However, without this patch meta-pages can be written by the kernel >>> before the data. >>> >>> In fact, for a full guarantee in case of death slapd process, >>> meta-page should be written explicitly. > > > No, the DB can never go inconsistent due to a process crashing - the pages > in OS cache are always correct. It can only go inconsistent if the OS > crashes and a proper sync has not occurred. Yes, Howard, you are right. But apparently I need to be more precise. Talking about "death" of slapd, I meant all the reasons, including power off. For example, a power-off case: - The main power is turned off and the system switches to the UPS. - Given the notice, OS starts an emergency stop processes. - For some reason (does not have enough time to stop) slapd receives SIGKILL. - OS tries to write mmap-region of the DB-file and begins with the lower address. - let the meta-pages has written completely, but for the rest of the data is not enough battery power. - Now DB is completely destroyed on the disk. To avoid this, the meta-pages should not be included in the rw-mapped region, and should be written explicity after a data pages. >>>>> commit 8ddd63161aeb2689822d1a8d27385d62e4e341ae >>>>> Author: Leo Yuriev <leo(a)yuriev.ru> >>>>> Date: 2014-09-19 22:47:19 +0400 >>>>> >>>>> BUGFIX - lmdb: properly sync meta-pages in mdb_sync_env(). >>>>> >>>>> Meta-pages may be updated during data-syncing in mdb_sync_env(), >>>>> in this case database would be inconsistent. >>>>> >>>>> Check-and-retry if lead txn-id changed during flushing data in >>>>> mdb_sync_env(). > > Fundamentally, you are trying to make an inherently unsafe configuration > "safer", but it's impossible. Assume you have mlock'd the meta pages into > memory, so the OS never flushes them itself any more, and you're running > with NOSYNC. That means, within 3 transactions, the data pages on disk will > be out of sync with the meta pages on disk. If the OS crashes at that point, > the entire DB will be lost. Not a problem. I had explained above - we should write meta-pages explicitly after the data sync. But also we should not perform reclaiming ahead of the last checkpoint. > The only way to make this mode of operation somewhat safe is to defer > reclaiming pages for even longer. E.g., instead of halting at current_txnid > - 3, halt at current_txnid - 22, in which case the data pointed to by the > on-disk meta pages cannot get obsolete until 20 transactions have occurred. > > Note that in combination with your LIFO patch, it's pretty much guaranteed > that the on-disk meta pages will be useless after only 2 un-sync'd > transactions. Yes, Howard, you are right. But I think there is confusion in the discussion because of mixing of LIFO-feature and changes for checkpoints consistency in a NOSYNC and WRITEMAP+NOSYNC modes. For a "NOSYNC + checkpoints" topic I will submit a separate ITS (like a 'volaile' related 7969,7970,7971). My opinion - it is a flaw, and no reason to don't fix it. Continuing the conversation about checkpoints in a LIFO context. I saw the problem, that you specified, and thinking over its solution, but have not yet found "golden ratio". And since we are having a serious problem with syncrepl, then I put off this task with an excuse "LIFO-patch not does worse than it was." In general, we should do not reclaim anything ahead of the txn, that is synced to the disk (let this be named a R-rule). To do so we need a second field like mti_txnid, but which will be update only at the end of mdb_env_write_meta(). Finally we should start search in mdb_find_oldest() from value of this new field instead of the current txn number. This seems to will be work fine. However, I stopped on a reasoning - about the purpose of the checkpoints, about design LMDB as a product, about the expectations of the user and the necessary configuration parameters: - checkpoints are needed ONLY in nosync modes; - if the user does NOT activate the checkpoints, he do not care about consistency; - but if it is turned on, we MUST provide consistency on the checkpoints; - otherwise a checkpoints feature is thoughtless and should be REMOVED. Therefore implementation of checkpoints & reclaiming should be updated to conform to the "R-rule", that noted above. >From this point of view a LIFO-feature also should be refined, but nevertheless can be very useful. - SYNC mode = takes a benefit from storage with write-back cache (assume powered by battery). - ASYNC/NOSYNC without checkpoint = significant reduction of committed/dirty pages and thereby much less write-iops. - ASYNC/NOSYNC with checkpoint = seems to same as a SYNC case. Total of all the above - I think first we need to fix a reclaiming or delete the checkpoints, and then I will complete LIFO. > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ Thank for conversation. Leonid.

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs