January 2007 - openldap-devel

syncrepl
by Howard Chu 06 Feb '07

06 Feb '07

ITS#4618, 4623, 4626 and 4703 all basically have to do with trying to use multiple replication contexts with a single provider. This is a behavior that the 2.3 syncprov implementation just wasn't designed for; it was meant to handle only a single context. Looking at the ideas in 2.2's syncrepl, it might have gone in the direction of solving these problems if it weren't weighed down by so many insurmountable design and implementation flaws. 2.2 probably tried to do too much too soon, and got waylaid by the devil in the details. At this point, this solution for multiple contexts presents itself: 1) We assign distinct searchbases to each context. 2) Every distinct source of changes must have its own unique rid. E.g., if a database is a provider for a context, it must have an rid. Every consumer within its namingContext must have their own rid's just as before. (The new requirement here is assigning rid's to providers that are masters of their data.) 3) Currently the provider hands a consumer a cookie consisting of the rid that the consumer supplied, plus a single contextCSN from the provider. This single contextCSN is inadequate for accurately capturing all of the changes that may come from multiple sources in a namingContext. Instead, the provider will send out a cookie consisting of multiple rid,CSN pairs - one for every rid of the provider's that resides in the consumer's search space. This is the only reliable way to make sure that all changes are tracked and propagated. This says that in general, rids should not need to be configured on consumers - they should be dictated solely by the providers. It may be a good idea to allow them to be configured on consumers as an override, but for now that seems unimportant. So: 1) the provider must have its own unique rid configured 2) the consumer's rid is optional 3) the provider must be told about all of the consumers living under it 4) the provider must aggregate all of the consumer cookies under it with its own context info when generating a cookie for its own consumers Currently slapd treats an entire database as read-only when it has a consumer configured on it. This raises the question of how to allow multiple consumers in a single context - should we allow multiple consumers per DB, as 2.2 tried (and failed) to do, or should we continue with the current approach of one consumer per DB, and use glue to collect multiple consumers under one roof? -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

3 9

slapo-dynlist desgin question(s)
by Quanah Gibson-Mount 06 Feb '07

06 Feb '07

Stanford is looking at implementing groups into our LDAP servers, and in particular, looking at using slapo-dynlist. However, it does not behave as I expected it to. Basically, it uses the credentials of whomever bound to determine the membership list. This means I would have to give access to a privileged attribute to those who wished to use groups, which is exactly what I'm trying to avoid. What I wanted to do, was specifically control the access to the group objects themselves. If an entity has access to the group object, they would then be able to see all current members of the group. I believe this would mean adding functionality to slapo-dynlist to where it uses the rootdn to perform the internal search instead of the credentials. Would it be possible to have this sort of addition? --Quanah -- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html

3 34

GSS-SPNEGO Protocol Details
by Michael B Allen 30 Jan '07

30 Jan '07

Hello, I've implemented SASL binds for GSSAPI and GSS-SPNEGO using a Sockbuf_IO_Desc handler instead of libsasl. Everything works great but I've noticed some behavior from the server I'm using that is not consistent with the available documentation (RFC 2222 and draft-ietf-sasl-gssapi-03 by Melnikov). Would anyone happen to know where I might ask about GSS-SPNEGO protocol details? Is there an IETF mailing list somewhere? There are three issues: 1) GSS-SPNEGO search replies are sealed even though the request was not and a capture of another client talking to the same server shows replies as integ-only. A examination of the captures of my code and the other client shows the packets are identical (minus ber encoding differences and encrypted krb5 bits). 2) GSS-SPNEGO does not appear to use the additional bind exchange to negotiate the security-layer bit mask like GSSAPI does. 3) GSSAPI can use what is apparently the DN of an account called the "authorization identity". The actual values for this field do not appear to be documented anywhere. I don't suppose I should care since the code works fine but I do. Any pointers are appreciated. Mike

3 6

Re: commit: ldap/servers/slapd schema_init.c
by Kurt Zeilenga 29 Jan '07

29 Jan '07

While not exactly incorrect, I think it best to On Jan 29, 2007, at 3:39 AM, hyc(a)OpenLDAP.org wrote: > Update of /repo/OpenLDAP/pkg/ldap/servers/slapd > > Modified Files: > schema_init.c 1.393 -> 1.394 > > Log Message: > ITS#4823 allow zero-length IA5String > > > CVS Web URLs: > http://www.openldap.org/devel/cvsweb.cgi/servers/slapd/ > http://www.openldap.org/devel/cvsweb.cgi/servers/slapd/ > schema_init.c > > Changes are generally available on cvs.openldap.org (and CVSweb) > within 30 minutes of being committed. >

1 1

sb_sasl_write short-count bug?
by Michael B Allen 28 Jan '07

28 Jan '07

Hi, Consider the sb_sasl_write function: 341 static ber_slen_t 342 sb_sasl_write( Sockbuf_IO_Desc *sbiod, void *buf, ber_len_t len) 343 { <snip> 352 /* Are there anything left in the buffer? */ 353 if ( p->buf_out.buf_ptr != p->buf_out.buf_end ) { 354 ret = ber_pvt_sb_do_write( sbiod, &p->buf_out ); <snip> 391 ret = ber_pvt_sb_do_write( sbiod, &p->buf_out ); 392 393 /* return number of bytes encoded, not written, to ensure 394 * no byte is encoded twice (even if only sent once). 395 */ 396 return len; 397 } This optimistically returns the len supplied. If ber_pvt_sb_do_write returns a short-count then data will be left in the p->buf_out. The remaining data will not be written until the next sb_sasl_write call which may not happen. I have not observed a problem but just from examining the logic I thought I might say something. Is this bug? Would putting ber_pvt_sb_do_write in a loop do any good? Mike

2 1

Re: commit: ldap/tests/scripts test049-sync-config
by Howard Chu 28 Jan '07

28 Jan '07

ando(a)OpenLDAP.org wrote: > Update of /repo/OpenLDAP/pkg/ldap/tests/scripts > > Modified Files: > test049-sync-config 1.3 -> 1.4 > > Log Message: > make sure replication finished before comparing data (under valgrind, replication may take ages) I debated adding such a check before. I think the first check you added should be removed. In my initial testing there were timing-dependent errors that cropped up when refreshing was occurring while the ldapadd was running. I think it's important that we continue to test for this case, however crudely it's done. Too bad we don't have a reliable means to notify the test script when the consumer has actually started its work. We could query back-monitor but normally (not under valgrind) the refresh could complete before we got the back-monitor search result. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

2 1

Re: cn=include
by Eric Irrgang 26 Jan '07

26 Jan '07

Since 1) the behavior is different 2) the slapd.conf to cn=config conversion sucks in the relevant information anyway and 3) use of includes is inappropriate under cn=config is it maybe time for the slap* tools stop creating cn=Includes? On Tue, 23 Jan 2007, Howard Chu wrote: > Eric Irrgang wrote: >> Are olcInclude attributes in cn=config honored as per the Admin Guide >> section 5.2.2 or is that documentation misleading? > > Good question. The short answer is - the use of include files is not > recommended for cn=config. They really only work correctly when slapd is using > slapd.conf. > > Keep in mind - in slapd.conf, you can insert include statements anywhere at > all in the config file, you can order them completely arbitrarily, > interleaving them with any other config statements. Under cn=config, all of > the cn=Includes are grouped under one place, they can't have anything else > inserted between them, so if they needed to have other intervening directives > processed first, they would fail. > > Also, the point of using cn=config is to make every part of the configuration > accessible/modifiable using LDAP. slapd.conf-formatted files (e.g. include > files) are not accessible or modifiable using LDAP. > -- > -- Howard Chu > Chief Architect, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc > OpenLDAP Core Team http://www.openldap.org/project/ > -- Eric Irrgang - UT Austin ITS Unix Systems - (512)475-9342

3 3

Search timeout?
by Pierangelo Masarati 25 Jan '07

25 Jan '07

I'm facing an issue: when a proxy is searching a remote server, and the server does not respond in the sense that it happily accepts a search request but never returns any type of response (a "silly" way of reproducing this issue consists in sending a SIGSTOP to the remote server: it accepts requests but does nothing), right now the proxy honors the timelimit. However, if the search is performed as the rootdn of the proxy, it will last forever. So I'd like to introduce the concept of "search timeout". This differs from the already existing "timelimit" in that it is not a client-requested (or server-imposed) limit on the overall duration of a search operation, but rather a server-imposed limitation on how long the proxy can wait consecutively for any sort of response from a remote server. This should, optionally, apply to the rootdn as well, since it's more about the sanity of the connection than on the properties of the operation. This type of limitation has already been introduced for other operations, including compare and all write operations, for a similar purpose. Comments? p.

2 1

Re: commit: ldap/tests/scripts test049-sync-config defines.sh
by Howard Chu 23 Jan '07

23 Jan '07

hyc(a)OpenLDAP.org wrote: > Update of /repo/OpenLDAP/pkg/ldap/tests/scripts > > Modified Files: > defines.sh 1.147 -> 1.148 > Added Files: > test049-sync-config NONE -> 1.1 > > Log Message: > Test slave bootstrapping via syncrepl For anyone curious, this test shows how to use back-config and syncrepl to dynamically configure and populate a server. Two stub entries (cn=config and olcdatabase=config,cn=config) are slapadded to initialize the master and slave configurations. Then ldapadd/ldapmodify are used to load the syncprov module, add a syncrepl consumer to the config database, and add the syncprov overlay to the config database on the master. Then ldapmodify is used to start a syncrepl consumer on the slave's config database. The rest of the script adds schema, a database backend and content for the backend on the master, all of which get replicated to the slave. The dilemma of how not to wipe out the consumer configuration once the complete master configuration is replicated onto it is solved with this trick: the master has both a provider and a consumer configured on it, and the consumer points at the master. The syncrepl config handler checks to see if its providerURI matches any of the current server's listenerURIs. If there's a match, the config is a no-op. (It gets parsed but no consumer task is triggered.) -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

2 3

Modrdn replication
by Pierangelo Masarati 21 Jan '07

21 Jan '07

ITS#4809 reports that when replicating modrdn via slurpd, operational attributes don't get replicated. This appears to be intrinsically caused by the definition of the modrdn operation, which, unlike the modify and add operations, doesn't contemplate the possibility to add/modify attributes other than the naming ones. So add/modify can be easily exploited for replication by adding/modifying the write-related operational attributes during replication, while modrdn can't. Assuming there's any intention to fix slurpd replication until slurpd is dismissed, we need to find a means to attach modification of write-related operational attributes to a modrdn operation, to complement the modrdn operation itself. The proposed solution consists in explicitly modifying the necessary (operational) attributes by means of an additional modify operation that is attached to modrdn. This may be occasionally useful regardless of slurpd replication, which makes it more appealing for OpenLDAP developers, so that the required effort wouldn't be just wasted by slurpd dismissal. The additional modify could be wrapped into a control's value, and that control might be the "relax" control itself, so that the original operation, augmented by the optional modify, would need to succeed as a whole or abort, extending the capabilities of the "relax" control by allowing extra modifications to be added, in order to preserve the integrity of the object after the operation. Comments? p.

3 4

Re: commit: ldap/servers/slapd acl.c slap.h
by Pierangelo Masarati 16 Jan '07

16 Jan '07

ando(a)OpenLDAP.org wrote: > allow to specify the required access privileges in internal operations (ITS#4806) > Note that one does not need to use the same access privileges for testing the filter and testing data return: in fact, when issuing an internal search, when required one could set o_acl_priv to the privilege desired for filter testing; then, in sc_response(), one could change it to the privilege desired for data return testing, and return it back in sc_cleanup(), and so back and forth. p.

1 0

Re: slapo-dynlist desgin question(s)
by Quanah Gibson-Mount 16 Jan '07

16 Jan '07

--On January 16, 2007 6:34:39 PM +0100 Pierangelo Masarati <ando(a)sys-net.it> wrote: > Quanah Gibson-Mount wrote: > >> This patch also does not work, continuing to use the credentials of the >> bound user. > > > What operation are you performing when it gets to evaluate that filter? > Can you describe it a little bit further? ldapsearch -LLL -Q -h ldap-dev1 -b "cn=groups,cn=applications,dc=stanford,dc=edu" cn=registry-consult -- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html

1 0

subclass matching in be_fetch()
by Pierangelo Masarati 11 Jan '07

11 Jan '07

I think in many cases it would be useful to have be_entry_get_rw() match entries based on their class not only exactly, as now, but also if the entry's class is subordinate to the one passed to be_entry_get_rw(). I have a patch that adds a flag parameter for the objectClass value. In most cases, to preserve the current behavior, the flag is set to 0; however I believe in many cases where be_entry_get_rw() or be_fetch() are used, it should definitely be set to SLAP_OCF_CHECK_SUP. If there's no objection, and I didn't overlook anything serious, I'd commit it. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.n.c. Via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ------------------------------------------ Office: +39.02.23998309 Mobile: +39.333.4963172 Email: pierangelo.masarati(a)sys-net.it ------------------------------------------

1 0

rootdn -> admindn?
by Hallvard B Furuseth 10 Jan '07

10 Jan '07

I've had it with explaining that the suffix is the root of the database which is not the rootdn which is not the Unix user "root". Any hope of renaming rootdn and rootpw to admindn and adminpw, with rootdn/pw as backwards-compatible and eventually undocumented aliases? -- Hallvard

8 10

Notice of disconnection
by Lars Hesel Christensen XX (AH/LMD) 10 Jan '07

10 Jan '07

Hi all I've been looking at the Notice of disconnection handling in OpenLDAP and there is a thing that is confusing me a bit. in result.c, the function try_read1msg reads data from socket and creates a new message from that. If we receive an unsolicited message, e.g, notice of disconnection, then the msgId will be zero, and we will go through the following code(taken from /* $OpenLDAP: /libraries/libldap/result.c,v 1.144 2006/12/17 21:04:25 ando Exp $ */): /* message id */ if ( ber_get_int( ber, &id ) == LBER_ERROR ) { ber_free( ber, 1 ); ld->ld_errno = LDAP_DECODING_ERROR; return( -1 ); } /* if it's been abandoned, toss it */ if ( ldap_abandoned( ld, id, &idx ) ) { <REMOVED since the msgId is not abandoned> retry_ber: ber_free( ber, 1 ); if ( ber_sockbuf_ctrl( lc->lconn_sb, LBER_SB_OPT_DATA_READY, NULL ) ) { goto retry; } --> return( LDAP_MSG_X_KEEP_LOOKING ); /* continue looking */ } lr = ldap_find_request_by_msgid( ld, id ); if ( lr == NULL ) { const char *msg = "unknown"; /* the message type */ tag = ber_peek_tag( ber, &len ); switch ( tag ) { case LBER_ERROR: break; default: msg = ldap_int_msgtype2str( tag ); break; } Debug( LDAP_DEBUG_ANY, "no request for response on ld %p msgid %ld message type %s (tossing)\n", (void *)ld, (long)id, msg ); goto retry_ber; } Assuming there is no more data to be read from the socket we will return in the line marked with the arrow '-->' since ldap_find_request_by_msgid() returns NULL. What confuses me is that later on in try_read1msg we get to (after constructing the message: /* is this the one we're looking for? */ if ( msgid == LDAP_RES_ANY || id == msgid ) { if ( all == LDAP_MSG_ONE || ( newmsg->lm_msgtype != LDAP_RES_SEARCH_RESULT && newmsg->lm_msgtype != LDAP_RES_SEARCH_ENTRY && newmsg->lm_msgtype != LDAP_RES_SEARCH_REFERENCE ) ) { *result = newmsg; ld->ld_errno = LDAP_SUCCESS; return( tag ); } else if ( newmsg->lm_msgtype == LDAP_RES_SEARCH_RESULT) { foundit = 1; /* return the chain later */ } } Do we have a test case testing notice of disconnection? I tried to grep the 'tests' directory for 'notice', 'disconnection' and 'unsolicited' but nothing came up. Perhaps I'm misunderstanding something here, so if I am I would very much appreciate a hint. Happy newyear everyone, Lars Hesel Christensen -- Lars Hesel Christensen Ericsson Denmark A/S, Telebit lars.hesel.xx.christensen(a)ericsson.com

3 9

Re: pthread rdwr locks
by Howard Chu 09 Jan '07

09 Jan '07

Howard Chu wrote: > I just noticed that we're explicitly disabling support for native rdwr > locks in ldap_int_thread.h, and it's been that way since 2000. Any > reason not to use the native locks instead of our emulation routines? At least on Linux the native pthread locks are faster than our emulation. However, the glibc header files don't expose the phtread_rwlock_t definition by default, and getting the definition exposed involves a bit of pain. I haven't investigated on any other platforms. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

1 0

Re: BerkeleyDB performance on Linux
by Howard Chu 08 Jan '07

08 Jan '07

Rick Jones wrote: > Howard Chu wrote: >> The attached patch makes O_DIRECT work on Linux in BerkeleyDB 4.5.20. >> (You will need to manually define LINUX_NEEDS_PAGE_ALIGNMENT if you're >> using a kernel older than 2.6.) >> >> The main reason to use this patch is to conserve memory - ordinarily, >> all the I/O that BDB does to its files gets cached in the Linux >> filesystem buffer cache. This caching is redundant since BDB always >> does its own caching, and it effectively makes the BDB environment >> consume twice as much memory as it needs. Using O_DIRECT on I/Os >> disables the filesystem buffer cache for those I/Os, thus freeing up a >> sizable chunk of memory. > > Long ago and far away I was having "fun" with SPECweb99 on Linux and > found that eventually the access logs would grow and cast-out otherwise > usefully cached URLs. I then found the posix_fadvise call, which can be > used to tell the system that a certain range is no longer required: Yes, but posix_fadvise is not supported on Linux kernels before 2.6, returns ENOSYS. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

1 0

BerkeleyDB performance on Linux
by Howard Chu 07 Jan '07

07 Jan '07

The attached patch makes O_DIRECT work on Linux in BerkeleyDB 4.5.20. (You will need to manually define LINUX_NEEDS_PAGE_ALIGNMENT if you're using a kernel older than 2.6.) The main reason to use this patch is to conserve memory - ordinarily, all the I/O that BDB does to its files gets cached in the Linux filesystem buffer cache. This caching is redundant since BDB always does its own caching, and it effectively makes the BDB environment consume twice as much memory as it needs. Using O_DIRECT on I/Os disables the filesystem buffer cache for those I/Os, thus freeing up a sizable chunk of memory. The caching problem is particularly aggravated on Linux because the memory manager doesn't give program pages higher priority than cache pages. So when your system is tight on memory, the kernel will start swapping program data pages before it starts reclaiming buffer cache pages, and application performance plummets. (Possibly that indicates a kernel bug, or at least a misfeature.) Note that you must configure BerkeleyDB with --enable-o_direct to enable the support, and you must add "set_flags DB_DIRECT_DB" to your DB_CONFIG to enable it in a particular environment. With this patch, a slapd that occupies 6.8GB on a system with 8GB of RAM can run continuously without swapping, delivering a sustained 11,500 authentications per second. Without the patch, swapping starts when the process hits the 4.5GB mark (because over 3GB of buffer cache is in use), and performance drops to only *hundreds* of authentications per second. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/ --- dbinc/mp.h.orig 2006-09-07 14:31:58.000000000 -0700 +++ dbinc/mp.h 2007-01-06 19:14:56.000000000 -0800 @@ -378,6 +378,23 @@ #define BH_FREE_REUSE 0x02 #define BH_FREE_UNLOCKED 0x04 +#ifdef DIAG_MVCC +#define BH_ALIGNED +#define VM_PAGESIZE 4096 +#endif + +/* Linux O_DIRECT needs aligned buffers. 2.6 kernel allows 512 byte + * alignment, otherwise need page sized (4096). + */ +#if defined(linux) && !defined(BH_ALIGNED) +#define BH_ALIGNED +#ifdef LINUX_NEEDS_PAGE_ALIGNMENT +#define VM_PAGESIZE 4096 +#else /* Linux 2.6+ */ +#define VM_PAGESIZE 512 +#endif +#endif + /* * BH -- * Buffer header. @@ -404,7 +421,7 @@ roff_t td_off; /* MVCC: creating TXN_DETAIL offset. */ SH_CHAIN_ENTRY vc; /* MVCC: version chain. */ -#ifdef DIAG_MVCC +#ifdef BH_ALIGNED u_int16_t align_off; /* Alignment offset for diagnostics.*/ #endif @@ -465,15 +482,14 @@ (dbc->txn != NULL && F_ISSET(dbc->txn, TXN_SNAPSHOT) && \ dbc->txn->td != NULL && __memp_skip_curadj(dbc, pgno)) -#if defined(DIAG_MVCC) && defined(HAVE_MPROTECT) -#define VM_PAGESIZE 4096 -#define MVCC_BHSIZE(mfp, sz) do { \ +#ifdef BH_ALIGNED +#define BHSIZE(mfp, sz) do { \ sz += VM_PAGESIZE + sizeof(BH); \ if (mfp->stat.st_pagesize < VM_PAGESIZE) \ sz += VM_PAGESIZE - mfp->stat.st_pagesize; \ } while (0) -#define MVCC_BHALIGN(mfp, p) do { \ +#define BHALIGN(mfp, p) do { \ if (mfp != NULL) { \ BH *__bhp; \ void *__orig = (p); \ @@ -493,13 +509,19 @@ } \ } while (0) -#define MVCC_BHUNALIGN(mfp, p) do { \ +#define BHUNALIGN(mfp, p) do { \ if ((mfp) != NULL) { \ BH *bhp = (BH *)(p); \ (p) = ((u_int8_t *)bhp - bhp->align_off); \ } \ } while (0) +#else +#define BHSIZE(mfp, sz) do {} while (0) +#define BHALIGN(mfp, p) do {} while (0) +#define BHUNALIGN(mfp, p) do {} while (0) +#endif +#if defined(DIAG_MVCC) && defined(HAVE_MPROTECT) #ifdef linux #define MVCC_MPROTECT(buf, sz, mode) do { \ int __ret = mprotect((buf), (sz), (mode)); \ @@ -513,11 +535,7 @@ } \ } while (0) #endif /* linux */ - -#else /* defined(DIAG_MVCC) && defined(HAVE_MPROTECT) */ -#define MVCC_BHSIZE(mfp, sz) do {} while (0) -#define MVCC_BHALIGN(mfp, p) do {} while (0) -#define MVCC_BHUNALIGN(mfp, p) do {} while (0) +#else #define MVCC_MPROTECT(buf, size, mode) do {} while (0) #endif --- mp/mp_alloc.c.orig 2006-09-07 14:32:03.000000000 -0700 +++ mp/mp_alloc.c 2007-01-06 19:14:56.000000000 -0800 @@ -66,7 +66,7 @@ if (mfp != NULL) { len = SSZA(BH, buf) + mfp->stat.st_pagesize; /* Add space for alignment padding for MVCC diagnostics. */ - MVCC_BHSIZE(mfp, len); + BHSIZE(mfp, len); } MPOOL_REGION_LOCK(dbenv, infop); @@ -91,10 +91,10 @@ c_mp->stat.st_pages++; MPOOL_REGION_UNLOCK(dbenv, infop); /* - * For MVCC diagnostics, align the pointer so that the buffer + * If necessary, align the pointer so that the buffer * starts on a page boundary. */ - MVCC_BHALIGN(mfp, p); + BHALIGN(mfp, p); found: if (offsetp != NULL) *offsetp = R_OFFSET(infop, p); @@ -447,7 +447,7 @@ MPOOLFILE *mfp; void *buf; { - MVCC_BHUNALIGN(mfp, buf); + BHUNALIGN(mfp, buf); COMPQUIET(mfp, NULL); __db_shalloc_free(infop, buf); } --- mp/mp_fget.c.orig 2006-09-13 09:22:42.000000000 -0700 +++ mp/mp_fget.c 2007-01-06 19:14:56.000000000 -0800 @@ -708,7 +708,7 @@ * the hash bucket's priority. */ /*lint --e{668} (flexelint: bhp cannot be NULL). */ -#ifdef DIAG_MVCC +#ifdef BH_ALIGNED memset(bhp, 0, SSZ(BH, align_off)); #else memset(bhp, 0, sizeof(BH));

2 6

LDAP Sync API
by Pierangelo Masarati 06 Jan '07

06 Jan '07

It's now in HEAD; typedef's, data structure and function declarations are in include/ldap.h; the bulk of the code is in libraries/libldap/ldap_sync.c. A man page is coming, although there might be need to freeze the API first. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.n.c. Via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ------------------------------------------ Office: +39.02.23998309 Mobile: +39.333.4963172 Email: pierangelo.masarati(a)sys-net.it ------------------------------------------

1 0

Re: better malloc strategies
by Howard Chu 05 Jan '07

05 Jan '07

Howard Chu wrote: > I've tested again with libhoard 3.5.1, and it's actually superior to > libumem for both speed and fragmentation. Here are some results against > glibc 2.3.3, libhoard, and libumem: I've retested with current HEAD, adding Google's tcmalloc to the mix. glibc hoard umem tcmalloc size time size time size time size time initial 731 741 741 744 single 1368 2:03.02 1809 1:26.67 1778 1:43.49 1453 0:58.77 single 1369 2:23.61 1827 0:25.68 1825 0:18.40 1512 0:21.74 single 1368 1:48.41 1828 0:16.52 1826 0:21.07 1529 0:22.97 single 1368 1:48.59 1829 0:16.59 1827 0:16.95 1529 0:17.07 single 1368 1:48.72 1829 0:16.53 1827 0:16.61 1529 0:17.01 single 1368 1:48.39 1829 0:20.70 1827 0:16.56 1529 0:16.99 single 1368 1:48.63 1830 0:16.56 1828 0:17.48 1529 0:17.29 single 1384 1:48.14 1829 0:16.64 1828 0:22.17 1529 0:16.94 four 1967 1:20.21 1918 0:35.96 1891 0:29.95 1606 0:42.48 four 2002 1:10.58 1919 0:30.07 1911 0:29.00 1622 0:42.38 four 2009 1:33.45 1920 0:42.06 1911 0:40.01 1628 0:40.68 four 1998 1:32.94 1920 0:35.62 1911 0:39.11 1634 0:30.41 four 1995 1:35.47 1920 0:34.20 1911 0:28.40 1634 0:40.80 four 1986 1:34.38 1920 0:28.92 1911 0:31.16 1634 0:40.42 four 1989 1:33.23 1920 0:31.48 1911 0:33.73 1634 0:33.97 four 1999 1:33.04 1920 0:33.47 1911 0:38.33 1634 0:40.91 slapd CPU 26:31.56 8:34.78 8:33.39 9:19.87 > The initial size is the size of the slapd process right after startup, > with the process totally idle. The id2entry DB is 1.3GB with about > 360,000 entries, BDB cache at 512M, entry cache at 70,000 entries, > cachefree 7000. The subsequent statistics are the size of the slapd > process after running a single search filtering on an unindexed > attribute, basically spanning the entire DB. The entries range in size > from a few K to a couple megabytes. The BDB and slapd cache configurations are the same, but the machine now has 4GB of RAM so the entire DB fits in the filesystem buffer cache. As such there is no disk I/O during these tests. > > After running the single ldapsearch 4 times, I then ran the same search > again with 4 jobs in parallel. There should of course be some process > growth for the resources for 3 additional threads (about 60MB is about > right since this is an x86_64 system). This time I ran the tests 8 times each. Basically I was looking for the slapd process size to stabilize at a constant number... > > The machine only had 2GB of RAM, and you can see that with glibc malloc > the kswapd got really busy in the 4-way run. The times might improve > slightly after I add more RAM to the box. But clearly glibc malloc is > fragmenting the heap like crazy. The current version of libhoard looks > like the winner here. There was no swap activity (or any other disk activity) this time, so the glibc numbers don't explode like they did before. But even so, the other allocators are 5-6 times faster. The Google folks claim tcmalloc is the fastest allocator they have ever seen. These tests show it is fast, but it is not the fastest. It definitely is the most space-efficient multi-threaded allocator though. It's hard to judge just by the execution times of the ldapsearch commands, so I also recorded the amount of CPU time the slapd process consumed by the end of each test. That gives a clearer picture of the performance differences for each allocator. These numbers aren't directly comparable to the ones I posted on August 30, because I was using a hacked up RE23 there and used HEAD here. Going thru these tests is a little disturbing; I would expect the single-threaded results to be 100% repeatable but they're not quite. In one run I saw glibc use up 2.1GB during the 4-way test, and never shrink back down. I later re-ran the same test (because I hadn't recorded the CPU usage the first time around) and got these numbers instead. The other thing that's disturbing is just how bad glibc's malloc really is, even in the single-threaded case which is supposed to be its ideal situation. The other thing to note is that for this run with libhoard, I doubled its SUPERBLOCK_SIZE from 64K to 128K. That's probably why it's less space-efficient here than umem, while it was more efficient in the August results. I guess I'll have to recompile that with the original size to see what difference that makes. For now I'd say my preference would be tcmalloc... -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

2 10

ldap_start_tls(), ldap_install_tls() and async
by Frank Cusack 02 Jan '07

02 Jan '07

ldap_start_tls() is async, but ldap_install_tls() appears to send stuff over the network. (I get errors on the server side about unknown protocol if my client app does not call ldap_install_tls(), so I'm just guessing that ldap_install_tls() sends network traffic.) So the problem is that ldap_install_tls() is not async and does not accept a timeout parameter. I guess that ldap_install_tls() does some crypto stuff as well, eg obtaining random data which might block. So doesn't ldap_install_tls() need to be async in order to make the start tls operation as a whole async? Or am I missing some information. thanks -frank

1 0

MSYS, Windows
by Howard Chu 02 Jan '07

02 Jan '07

Mostly off-topic, but may be of some interest... I recently noticed while running the test suite on Windows with MSYS that the server processes were not getting killed cleanly, they were just being terminated abruptly. This tended to do weird things for subsequent tests, showing complaints about not being able to delete the testrun directory for a prior test because it wasn't empty. I found that one way to work around this problem was to invoke Cygwin's "sync" command after each test. (Added into tests/scripts/all.) In some cases it would cause a test to fail that would otherwise succeed when run on its own. But the actual problem is that the MSYS DLL doesn't propagate signals to non-MSYS apps. And it turns out, I had actually noticed this back in 2003 and subsequently forgotten. http://news.gmane.org/find-root.php?group=gmane.comp.gnu.mingw.msys&article… Anyway, I've written a patch for the MSYS DLL which you can find here http://sourceforge.net/tracker/index.php?func=detail&aid=1624635&group_id=2… and I have a binary you can download here http://highlandsun.com/hyc/msys-1.0.dll.gz cksum: 2315022199 326195 msys-1.0.dll.gz It's based off a recent pull from their CVS, so roughly version 1.0.11. It also includes both of these patches http://sourceforge.net/tracker/index.php?func=detail&aid=1174106&group_id=2… If you're doing a lot of work with OpenLDAP in a Windows environment this will definitely make life more tolerable. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel January 2007