openldap-devel

openldap-devel@openldap.org

1416 discussions

MDB v2: Replace meta pages with "meta position" word
by Hallvard Breien Furuseth 16 Nov '12

16 Nov '12

I think MDB v2 should move the variable parts of MDB_meta into the data pages. The datafile header would retain a word with the position of the last *synced* MDB_meta, or of the last meta when MDB_NOSYNC. The lockfile header would hold the position of the *last* MDB_meta. All transactions start from the lockfile->metapos commit. Write txns do not reuse free pages younger than the datafile->metapos commit. mdb_env_sync() called by the user does roughly: size_t lastpos = lockfile->metapos; sync; # define pos2id(env, pos) ((MDB_meta*)((env)->me_map+(pos)))->mt_txnid if (pos2id(env, lastpos) > pos2id(env, datafile->metapos)) write lastpos to &datafile->metapos; Called from mdb_txn_commit(), this may need lastpos as an argument. Results, if I'm keeping this straight: Setting the latest commit becomes atomic: Just change metapos. (Field MDB_txninfo.mti_txnid goes away.) No sync issues with copying 'MDB_db's from the meta, since the meta will not be overwritten during the txn. Users can sync infrequently yet preserve consistency, a generalization of MDB_NOMETASYNC. An application crash will then lose unsynced commits, since resetting the lockfile must reset lockfile->metapos. MDB cannot know if a system crash left those commits unsynced. mdb_env_sync() needs a mutex - either its own or the write lock. (A soft mode could trylock and do nothing if that fails.) If mdb_env_sync() gets its own mutex, then mdb_txn_commit() can announce the commit at lockfile->metapos and unlock the write lock _before_ doing mdb_env_sync. With multiple writer threads, that's like an ACID-safe MDB_MAPASYNC. However, that has quirks. I don't know how serious they are: - mdb_txn_commit() can fail after other txns see the commit, or succeed but set a failure flag for other txns to react to. Delayed mdb_env_sync can fail today too, but it will also happen if mdb_env_sync cannot set datafile->metapos. - mdb_txn_commit() may not return immediately after the commit becomes visible to other txns. Unless it is set up to queue the {sync; set datafile->metapos} actions for a maintenance thread. More detailed draft code, still ignoring various flags: typedef struct MDB_meta { /* Meta info about a commit */ MDB_db mm_dbs[2]; txnid_t mm_txnid; pgno_t mm_last_pg; } MDB_meta; typedef struct MDB_header { /* Datafile header */ ...; /* Position of last synced meta - or last known if MDB_NOSYNC */ size_t mh_metapos; } MDB_header; typedef struct MDB_txbody { /* Lockfile header */ ...; /* Position of last meta, possibly not synced. Both read and write * txns start at this commit. Replaces the old member mtb_txnid. */ size_t mtb_metapos; } MDB_txbody; mdb_txn_commit(MDB_txn *txn) { ...; /* Commit a write txn: */ pwritev(env->me_fd, <data pages including MDB_meta>); /* Make the commit visible to other txns */ lockfile->mtb_metapos = <offset of MDB_meta in me_map>; unlock(write_mutex); /* Preserve the commit */ mdb_env_sync(env, 0); } # define pos2id(env, pos) (((MDB_meta*)((env)->me_map+(pos)))->mt_txnid) mdb_txn_sync(MDB_txn *txn, int force) { MDB_env *txn->mt_env; MDB_txninfo *txns = env->me_txns; enum { metapos_pos = offsetof(MDB_header, mh_metapos) }; lock(meta_mutex); /* Positions of meta pages known to datafile and lockfile */ size_t cur = *(size_t *)(env->me_map + metapos_pos); size_t lastpos = txns->mtb_metapos; int got_new = pos2id(lastpos) > pos2id(cur); if (force || (got_new && !(env->me_flags & MDB_NOSYNC))) fdatasync(env->me_fd); /* Make datafile catch up with pre-fdatasync lockfile */ if (got_new) pwrite(env->me_mfd, &lastpos, sizeof(lastpos), metapos_pos); unlock(meta_mutex); } -- Hallvard

2 11

slapd detach and listen order
by Greg Hudson 09 Nov '12

09 Nov '12

Up through 2.4.26, slapd detached after binding server sockets (slapd_daemon_init) but before initializing back ends (slap_startup) or calling listen on the server sockets (slapd_daemon). Since initializing DB back ends could take some time, there were some practical issues like http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=589915 . The intended fix for this, 03ac02c7215fb592a75370b86069ee2372abec49 and f77e475967cb71ce925941b4660c4d7a04b3b8bf and 9f66f51a7189e6bf73ae6f2ff5da11487a94664e, cause the parent process to wait until the child has finished running slap_startup(). But unless I'm misreading the code, it seems like there is still a window after the parent exits before the child calls listen on the server sockets, during which a client could try to connect and fail. Is there a reason why the call to listen needs to be deferred until slapd_daemon_task, instead of doing it in slap_open_listener when the server sockets are first bound? (This issue came up for me because I'm writing automated tests for the MIT krb5 LDAP KDB module. I haven't been able to produce a failure to connect after starting the 2.4.28 slapd, I think because the window is very brief, but any window is likely to eventually cause an issue on someone's machine, in my experience.)

1 0

Re: (ITS#7428) libldap: use non-blocking IO during TLS handshake
by Ralf Haferkamp 08 Nov '12

08 Nov '12

Hi, On Thu, Nov 01, 2012 at 05:36:54PM +0000, I wrote: > I've just uploaded: > > ftp://ftp.openldap.org/incoming/rhafer-Use-non-blocking-IO-during-SSL-Hands… > > which tries to address the issue. If LDAP_OPT_NETWORK_TIMEOUT is set > ldap_int_tls_start will switch to non-blocking IO and call > ldap_int_tls_connect as often as needed unless it times out inbetween. > Currently I have only tested this with openssl but AFAICS this should also work > with the NSS and gnutls backends > > Please review and comment. Did somebody have a chance already to look at this? I'd really like to know if it would be ok to push this patch into master, and if not, what needs to be done to make it acceptable for master. Meanwhile I already tried testing it with the NSS and gnutls backends, with mixed success so far. But as of now I think the issues I ran into can be blame to bugs in either gnutls or NSS. Applying this patch to a supported (by libldap) gnutls release e.g. fixes the problems in gnutls: http://git.savannah.gnu.org/gitweb/?p=gnutls.git;a=blobdiff;f=lib/gnutls_bu… NSS otoh seems to have problems handling short writes during the TLS Handshake, I haven't been able to track this down as I don't know NSS well enough. But as I am also able to produces weird failures on the server side using an unmodified libldap it seems the problem is not related to my patch (to test this I used a stock FC17 with and reduced the tcp buffers to really small values (/proc/sys/net/ipv4/tcp_rmem and tcp_wmem) on both the client and the server. After doing some searches the client locked up the server log seem to indicate that it did a short write and never retries to write the rest of the buffer)). Would be good if someone with NSS knowledge could look into this (hello Rich ;)) Ralf

1 0

Re: (ITS#7432) slapd crashes when removing members from a large group
by Howard Chu 07 Nov '12

07 Nov '12

hyc(a)symas.com wrote: > marvin.mundry(a)uni-hamburg.de wrote: >> Full_Name: Marvin Mundry >> Version: 2.4.33 >> OS: Ubuntu 12.10 >> URL: https://idmswiki.rrz.uni-hamburg.de:8005/debug.tar.bz >> Submission from: (NULL) (134.100.2.183) >> > > Thanks for the report. The crash has been fixed in git, but your test runs > into another (known) issue in MDB. > > You're working with a very large entry, which libmdb stores in overflow pages. > In the current version of libmdb, freespace management for overflow pages is > not fully implemented, so every time you update the entry libmdb will always > use new pages (instead of reusing old pages). Thus, after a few hundred > operations, your 1GB map will be exhausted. > > It looks like you won't be able to use back-mdb until this feature is fully > implemented in libmdb. So the issue is how to find a contiguous run of pages large enough to satisfy the overflow page, in the current freelist. This takes us into the realm of malloc algorithms, first-fit/best-fit/..., etc. I think first we scan whatever freelist we have in memory, to see if a suitable run of pages is already present. If not, and there are additional freelists still available: 1) we could just merge all of them, and then search again or 2) merge one at a time, and search again Leaning toward #2, I suspect we don't need to coalesce all freelists all the time. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 2

Support readonly MDB slaptools on readonly filesystems
by Hallvard Breien Furuseth 27 Oct '12

27 Oct '12

Back-mdb needs this test, copied from libmdb, to make use of the libmdb support for read-only filesystems. I don't know if Windows sets errno=EROFS for fopen(fname, "w") when fname is on a read-only FS, if so the #ifndef _WIN32 is unnecessary. --- servers/slapd/back-mdb/config.c +++ servers/slapd/back-mdb/config.c @@ -561,3 +561,8 @@ mdb_cf_gen( ConfigArgs *c ) ch_free( testpath ); - if ( !f ) { +#ifndef _WIN32 + if ( !(f || ((slapMode & SLAP_TOOL_READONLY) && errno == EROFS)) ) +#else + if ( !(f || ((slapMode & SLAP_TOOL_READONLY) && GetLastError() == ERROR_WRITE_PROTECT)) ) +#endif + { snprintf( c->cr_msg, sizeof( c->cr_msg ), "%s: invalid path: %s", -- Hallvard

1 0

Re: openldap.git branch mdb.master updated. ef2092eac9f35b77eae250801c96dfa515c6d4b2
by Hallvard Breien Furuseth 25 Oct '12

25 Oct '12

> commit ef2092eac9f35b77eae250801c96dfa515c6d4b2 > Author: Howard Chu <hyc(a)symas.com> > Date: Thu Oct 25 05:28:55 2012 -0700 > > MDB_DEBUG tweak for old compilers This changes which page DPRINTF says got popped, since mc_top is now read after the "mc->mc_top--;" statement. I think it should have been: --- mdb.c +++ mdb.c @@ -298,4 +298,5 @@ #else # define DPRINTF(fmt, ...) ((void) 0) +# define MDB_DEBUG_SKIP #endif /** Print a debug string. @@ -3684,5 +3685,5 @@ { if (mc->mc_snum) { -#if MDB_DEBUG +#ifndef MDB_DEBUG_SKIP MDB_page *top = mc->mc_pg[mc->mc_top]; #endif Or use the DPRINTF((args)) hack after all, and a helper macro to unpack the arguments. -- Hallvard

2 1

RE24 testing call #2 (OpenLDAP 2.4.33)
by Quanah Gibson-Mount 13 Oct '12

13 Oct '12

If you know how to build OpenLDAP manually, and would like to participate in testing the next set of code for the 2.4.33 release, please do so. Generally, get the code for RE24: <http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=snapshot;h=refs/h…> Configure & build. Execute the test suite (via make test) after it is built. Thanks! --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

2 1

2.4.33 is ready for release (Contributors)
by Quanah Gibson-Mount 08 Oct '12

08 Oct '12

2.4.33 is ready for release. Significant contributors are: Howard Chu (Symas Corp) Hallvard Furuseth (University of Oslo) Quanah Gibson-Mount (VMware, Inc) --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Manual should tell the data range of the configuration parameter
by Tianyin Xu 04 Oct '12

04 Oct '12

Hi, guys, I was quite confused with "index_intlen". ---------- what the manual says ---------- * * *index_intlen* <*integer*> Specify the key length for ordered integer indices. The most significant bytes of the binary integer will be used for index keys. The default value is 4, which provides exact indexing for 31 bit values. A floating point representation is used to index too large values. ------------------------------------------------------ Ok, what the manual told me is that 4 is the default value, and large values can be handled. But I finally figured out that 4 is not only the default value but also the minimum value. >From the source code, I knew the index_intlen is with the data range [4, 255]. Any out-of-range value will be converted to the boundaries without notifying users: #define SLAP_INDEX_INTLEN_DEFAULT 4 /* servers/slapd/bconfig.c */ static int config_generic(ConfigArgs *c) { ... case CFG_IX_INTLEN: if ( c->value_int < SLAP_INDEX_INTLEN_DEFAULT ) c->value_int = SLAP_INDEX_INTLEN_DEFAULT; else if ( c->value_int > 255 ) c->value_int = 255; index_intlen = c->value_int; ... } To my opinion, as a configuration knob for user tuning, such behavior (especially the range) should be explicitly exposed to users. I would suggest to make it explicit by 1. print out logs to notify users 2. write in the manual. * * Thanks a lot! Tianyin -- Tianyin XU, http://cseweb.ucsd.edu/~tixu/

2 1

RE24 testing all #1 (OpenLDAP 2.4.33)
by Quanah Gibson-Mount 03 Oct '12

03 Oct '12

3 4

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel