openldap-devel

openldap-devel@openldap.org

4 participants
1420 discussions

Re: RE24 Testing call #1 (OpenLDAP 2.4.34)
by Marco Pizzoli 11 Dec '12

11 Dec '12

All hdb and mdb tests OK on RHEL6.3 X86_64with bdb 5.3.21. I also successfully slapadded in a mdb database a known problematic (with 2.4.33) database. Marco On Tue, Dec 11, 2012 at 12:49 AM, Quanah Gibson-Mount <quanah(a)zimbra.com>wrote: > Current RE24 is ready for testing for the 2.4.34 release. > > Thanks! > > --Quanah > > -- > > Quanah Gibson-Mount > Sr. Member of Technical Staff > Zimbra, Inc > A Division of VMware, Inc. > -------------------- > Zimbra :: the leader in open source messaging and collaboration > >

1 0

indexing optimization
by Howard Chu 04 Dec '12

04 Dec '12

In schema_init.c hashPreset() we compute the initial value of the index hash. This includes a prefix string which is usually the attribute's canonical name, a prefix character which is 0 or a particular substring index type, and then the syntax OID and matching rule OID. This is computed once per attribute. We could instead compute this one-time only, and save the result in the Attribute structure, or the AttributeDescription structure. (It probably makes most sense in the latter, since the prefix also may include any tags on the attributeDescription.) This change would require changing the signature of the indexer functions, to pass in the AttrDesc pointer or something to convey the precomputed hash. Or we could do away with some of this. Hashing against the syntax and mr OID seems gratuitous, since the index values are already going to be segregated by prefix and pre. One way or another, the hash over these two OIDs should be factored out, because even if we want to keep them, they're constants and we shouldn't need to keep recomputing them. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

mdb_dcmp() = EINVAL?
by Hallvard Breien Furuseth 03 Dec '12

03 Dec '12

mdb_dcmp() returns EINVAL if no dupsort function is set. I'm not sure it does the caller any favors in doing so: It claims (A>B && B>A) if it gets called with both (A,B) and (B,A). That can confuse the caller, e.g. a sort function. I expect the sensible results without a dcmp function are 0 or a crash. Plus to document this clearly. If that EINVAL is useful, i.e. if there are times when the user wishes to ask if a dupsort function is set, then mdb can grow an mdb_dbi_info(env, dbi, &result) function to describe the database: Name, flags, sort funcs. -- Hallvard

2 1

MDB library naming conflict
by Quanah Gibson-Mount 02 Dec '12

02 Dec '12

Unfortunately, libmdb already exists in modern linux systems as part of the mdb-tools package. This would be a problematic conflict for getting MDB packaged in general. As a proposal, how about liblmdb as the new name? I haven't been able to find any instances of that existing in the wild. --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

5 13

MDB v2: Replace meta pages with "meta position" word
by Hallvard Breien Furuseth 16 Nov '12

16 Nov '12

I think MDB v2 should move the variable parts of MDB_meta into the data pages. The datafile header would retain a word with the position of the last *synced* MDB_meta, or of the last meta when MDB_NOSYNC. The lockfile header would hold the position of the *last* MDB_meta. All transactions start from the lockfile->metapos commit. Write txns do not reuse free pages younger than the datafile->metapos commit. mdb_env_sync() called by the user does roughly: size_t lastpos = lockfile->metapos; sync; # define pos2id(env, pos) ((MDB_meta*)((env)->me_map+(pos)))->mt_txnid if (pos2id(env, lastpos) > pos2id(env, datafile->metapos)) write lastpos to &datafile->metapos; Called from mdb_txn_commit(), this may need lastpos as an argument. Results, if I'm keeping this straight: Setting the latest commit becomes atomic: Just change metapos. (Field MDB_txninfo.mti_txnid goes away.) No sync issues with copying 'MDB_db's from the meta, since the meta will not be overwritten during the txn. Users can sync infrequently yet preserve consistency, a generalization of MDB_NOMETASYNC. An application crash will then lose unsynced commits, since resetting the lockfile must reset lockfile->metapos. MDB cannot know if a system crash left those commits unsynced. mdb_env_sync() needs a mutex - either its own or the write lock. (A soft mode could trylock and do nothing if that fails.) If mdb_env_sync() gets its own mutex, then mdb_txn_commit() can announce the commit at lockfile->metapos and unlock the write lock _before_ doing mdb_env_sync. With multiple writer threads, that's like an ACID-safe MDB_MAPASYNC. However, that has quirks. I don't know how serious they are: - mdb_txn_commit() can fail after other txns see the commit, or succeed but set a failure flag for other txns to react to. Delayed mdb_env_sync can fail today too, but it will also happen if mdb_env_sync cannot set datafile->metapos. - mdb_txn_commit() may not return immediately after the commit becomes visible to other txns. Unless it is set up to queue the {sync; set datafile->metapos} actions for a maintenance thread. More detailed draft code, still ignoring various flags: typedef struct MDB_meta { /* Meta info about a commit */ MDB_db mm_dbs[2]; txnid_t mm_txnid; pgno_t mm_last_pg; } MDB_meta; typedef struct MDB_header { /* Datafile header */ ...; /* Position of last synced meta - or last known if MDB_NOSYNC */ size_t mh_metapos; } MDB_header; typedef struct MDB_txbody { /* Lockfile header */ ...; /* Position of last meta, possibly not synced. Both read and write * txns start at this commit. Replaces the old member mtb_txnid. */ size_t mtb_metapos; } MDB_txbody; mdb_txn_commit(MDB_txn *txn) { ...; /* Commit a write txn: */ pwritev(env->me_fd, <data pages including MDB_meta>); /* Make the commit visible to other txns */ lockfile->mtb_metapos = <offset of MDB_meta in me_map>; unlock(write_mutex); /* Preserve the commit */ mdb_env_sync(env, 0); } # define pos2id(env, pos) (((MDB_meta*)((env)->me_map+(pos)))->mt_txnid) mdb_txn_sync(MDB_txn *txn, int force) { MDB_env *txn->mt_env; MDB_txninfo *txns = env->me_txns; enum { metapos_pos = offsetof(MDB_header, mh_metapos) }; lock(meta_mutex); /* Positions of meta pages known to datafile and lockfile */ size_t cur = *(size_t *)(env->me_map + metapos_pos); size_t lastpos = txns->mtb_metapos; int got_new = pos2id(lastpos) > pos2id(cur); if (force || (got_new && !(env->me_flags & MDB_NOSYNC))) fdatasync(env->me_fd); /* Make datafile catch up with pre-fdatasync lockfile */ if (got_new) pwrite(env->me_mfd, &lastpos, sizeof(lastpos), metapos_pos); unlock(meta_mutex); } -- Hallvard

2 11

slapd detach and listen order
by Greg Hudson 09 Nov '12

09 Nov '12

Up through 2.4.26, slapd detached after binding server sockets (slapd_daemon_init) but before initializing back ends (slap_startup) or calling listen on the server sockets (slapd_daemon). Since initializing DB back ends could take some time, there were some practical issues like http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=589915 . The intended fix for this, 03ac02c7215fb592a75370b86069ee2372abec49 and f77e475967cb71ce925941b4660c4d7a04b3b8bf and 9f66f51a7189e6bf73ae6f2ff5da11487a94664e, cause the parent process to wait until the child has finished running slap_startup(). But unless I'm misreading the code, it seems like there is still a window after the parent exits before the child calls listen on the server sockets, during which a client could try to connect and fail. Is there a reason why the call to listen needs to be deferred until slapd_daemon_task, instead of doing it in slap_open_listener when the server sockets are first bound? (This issue came up for me because I'm writing automated tests for the MIT krb5 LDAP KDB module. I haven't been able to produce a failure to connect after starting the 2.4.28 slapd, I think because the window is very brief, but any window is likely to eventually cause an issue on someone's machine, in my experience.)

1 0

Re: (ITS#7428) libldap: use non-blocking IO during TLS handshake
by Ralf Haferkamp 08 Nov '12

08 Nov '12

Hi, On Thu, Nov 01, 2012 at 05:36:54PM +0000, I wrote: > I've just uploaded: > > ftp://ftp.openldap.org/incoming/rhafer-Use-non-blocking-IO-during-SSL-Hands… > > which tries to address the issue. If LDAP_OPT_NETWORK_TIMEOUT is set > ldap_int_tls_start will switch to non-blocking IO and call > ldap_int_tls_connect as often as needed unless it times out inbetween. > Currently I have only tested this with openssl but AFAICS this should also work > with the NSS and gnutls backends > > Please review and comment. Did somebody have a chance already to look at this? I'd really like to know if it would be ok to push this patch into master, and if not, what needs to be done to make it acceptable for master. Meanwhile I already tried testing it with the NSS and gnutls backends, with mixed success so far. But as of now I think the issues I ran into can be blame to bugs in either gnutls or NSS. Applying this patch to a supported (by libldap) gnutls release e.g. fixes the problems in gnutls: http://git.savannah.gnu.org/gitweb/?p=gnutls.git;a=blobdiff;f=lib/gnutls_bu… NSS otoh seems to have problems handling short writes during the TLS Handshake, I haven't been able to track this down as I don't know NSS well enough. But as I am also able to produces weird failures on the server side using an unmodified libldap it seems the problem is not related to my patch (to test this I used a stock FC17 with and reduced the tcp buffers to really small values (/proc/sys/net/ipv4/tcp_rmem and tcp_wmem) on both the client and the server. After doing some searches the client locked up the server log seem to indicate that it did a short write and never retries to write the rest of the buffer)). Would be good if someone with NSS knowledge could look into this (hello Rich ;)) Ralf

1 0

Re: (ITS#7432) slapd crashes when removing members from a large group
by Howard Chu 07 Nov '12

07 Nov '12

hyc(a)symas.com wrote: > marvin.mundry(a)uni-hamburg.de wrote: >> Full_Name: Marvin Mundry >> Version: 2.4.33 >> OS: Ubuntu 12.10 >> URL: https://idmswiki.rrz.uni-hamburg.de:8005/debug.tar.bz >> Submission from: (NULL) (134.100.2.183) >> > > Thanks for the report. The crash has been fixed in git, but your test runs > into another (known) issue in MDB. > > You're working with a very large entry, which libmdb stores in overflow pages. > In the current version of libmdb, freespace management for overflow pages is > not fully implemented, so every time you update the entry libmdb will always > use new pages (instead of reusing old pages). Thus, after a few hundred > operations, your 1GB map will be exhausted. > > It looks like you won't be able to use back-mdb until this feature is fully > implemented in libmdb. So the issue is how to find a contiguous run of pages large enough to satisfy the overflow page, in the current freelist. This takes us into the realm of malloc algorithms, first-fit/best-fit/..., etc. I think first we scan whatever freelist we have in memory, to see if a suitable run of pages is already present. If not, and there are additional freelists still available: 1) we could just merge all of them, and then search again or 2) merge one at a time, and search again Leaning toward #2, I suspect we don't need to coalesce all freelists all the time. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 2

Support readonly MDB slaptools on readonly filesystems
by Hallvard Breien Furuseth 27 Oct '12

27 Oct '12

Back-mdb needs this test, copied from libmdb, to make use of the libmdb support for read-only filesystems. I don't know if Windows sets errno=EROFS for fopen(fname, "w") when fname is on a read-only FS, if so the #ifndef _WIN32 is unnecessary. --- servers/slapd/back-mdb/config.c +++ servers/slapd/back-mdb/config.c @@ -561,3 +561,8 @@ mdb_cf_gen( ConfigArgs *c ) ch_free( testpath ); - if ( !f ) { +#ifndef _WIN32 + if ( !(f || ((slapMode & SLAP_TOOL_READONLY) && errno == EROFS)) ) +#else + if ( !(f || ((slapMode & SLAP_TOOL_READONLY) && GetLastError() == ERROR_WRITE_PROTECT)) ) +#endif + { snprintf( c->cr_msg, sizeof( c->cr_msg ), "%s: invalid path: %s", -- Hallvard

1 0

Re: openldap.git branch mdb.master updated. ef2092eac9f35b77eae250801c96dfa515c6d4b2
by Hallvard Breien Furuseth 25 Oct '12

25 Oct '12

> commit ef2092eac9f35b77eae250801c96dfa515c6d4b2 > Author: Howard Chu <hyc(a)symas.com> > Date: Thu Oct 25 05:28:55 2012 -0700 > > MDB_DEBUG tweak for old compilers This changes which page DPRINTF says got popped, since mc_top is now read after the "mc->mc_top--;" statement. I think it should have been: --- mdb.c +++ mdb.c @@ -298,4 +298,5 @@ #else # define DPRINTF(fmt, ...) ((void) 0) +# define MDB_DEBUG_SKIP #endif /** Print a debug string. @@ -3684,5 +3685,5 @@ { if (mc->mc_snum) { -#if MDB_DEBUG +#ifndef MDB_DEBUG_SKIP MDB_page *top = mc->mc_pg[mc->mc_top]; #endif Or use the DPRINTF((args)) hack after all, and a helper macro to unpack the arguments. -- Hallvard

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel