December 2006 - openldap-devel

by Howard Chu

ITS#4618, 4623, 4626 and 4703 all basically have to do with trying to use multiple replication contexts with a single provider. This is a behavior that the 2.3 syncprov implementation just wasn't designed for; it was meant to handle only a single context. Looking at the ideas in 2.2's syncrepl, it might have gone in the direction of solving these problems if it weren't weighed down by so many insurmountable design and implementation flaws. 2.2 probably tried to do too much too soon, and got waylaid by the devil in the details. At this point, this solution for multiple contexts presents itself: 1) We assign distinct searchbases to each context. 2) Every distinct source of changes must have its own unique rid. E.g., if a database is a provider for a context, it must have an rid. Every consumer within its namingContext must have their own rid's just as before. (The new requirement here is assigning rid's to providers that are masters of their data.) 3) Currently the provider hands a consumer a cookie consisting of the rid that the consumer supplied, plus a single contextCSN from the provider. This single contextCSN is inadequate for accurately capturing all of the changes that may come from multiple sources in a namingContext. Instead, the provider will send out a cookie consisting of multiple rid,CSN pairs - one for every rid of the provider's that resides in the consumer's search space. This is the only reliable way to make sure that all changes are tracked and propagated. This says that in general, rids should not need to be configured on consumers - they should be dictated solely by the providers. It may be a good idea to allow them to be configured on consumers as an override, but for now that seems unimportant. So: 1) the provider must have its own unique rid configured 2) the consumer's rid is optional 3) the provider must be told about all of the consumers living under it 4) the provider must aggregate all of the consumer cookies under it with its own context info when generating a cookie for its own consumers Currently slapd treats an entire database as read-only when it has a consumer configured on it. This raises the question of how to allow multiple consumers in a single context - should we allow multiple consumers per DB, as 2.2 tried (and failed) to do, or should we continue with the current approach of one consumer per DB, and use glue to collect multiple consumers under one roof? -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

16 years, 3 months

3
9
0 / 0

Notice of disconnection

by Lars Hesel Christensen XX (AH/LMD)

Hi all I've been looking at the Notice of disconnection handling in OpenLDAP and there is a thing that is confusing me a bit. in result.c, the function try_read1msg reads data from socket and creates a new message from that. If we receive an unsolicited message, e.g, notice of disconnection, then the msgId will be zero, and we will go through the following code(taken from /* $OpenLDAP: /libraries/libldap/result.c,v 1.144 2006/12/17 21:04:25 ando Exp $ */): /* message id */ if ( ber_get_int( ber, &id ) == LBER_ERROR ) { ber_free( ber, 1 ); ld->ld_errno = LDAP_DECODING_ERROR; return( -1 ); } /* if it's been abandoned, toss it */ if ( ldap_abandoned( ld, id, &idx ) ) { <REMOVED since the msgId is not abandoned> retry_ber: ber_free( ber, 1 ); if ( ber_sockbuf_ctrl( lc->lconn_sb, LBER_SB_OPT_DATA_READY, NULL ) ) { goto retry; } --> return( LDAP_MSG_X_KEEP_LOOKING ); /* continue looking */ } lr = ldap_find_request_by_msgid( ld, id ); if ( lr == NULL ) { const char *msg = "unknown"; /* the message type */ tag = ber_peek_tag( ber, &len ); switch ( tag ) { case LBER_ERROR: break; default: msg = ldap_int_msgtype2str( tag ); break; } Debug( LDAP_DEBUG_ANY, "no request for response on ld %p msgid %ld message type %s (tossing)\n", (void *)ld, (long)id, msg ); goto retry_ber; } Assuming there is no more data to be read from the socket we will return in the line marked with the arrow '-->' since ldap_find_request_by_msgid() returns NULL. What confuses me is that later on in try_read1msg we get to (after constructing the message: /* is this the one we're looking for? */ if ( msgid == LDAP_RES_ANY || id == msgid ) { if ( all == LDAP_MSG_ONE || ( newmsg->lm_msgtype != LDAP_RES_SEARCH_RESULT && newmsg->lm_msgtype != LDAP_RES_SEARCH_ENTRY && newmsg->lm_msgtype != LDAP_RES_SEARCH_REFERENCE ) ) { *result = newmsg; ld->ld_errno = LDAP_SUCCESS; return( tag ); } else if ( newmsg->lm_msgtype == LDAP_RES_SEARCH_RESULT) { foundit = 1; /* return the chain later */ } } Do we have a test case testing notice of disconnection? I tried to grep the 'tests' directory for 'notice', 'disconnection' and 'unsolicited' but nothing came up. Perhaps I'm misunderstanding something here, so if I am I would very much appreciate a hint. Happy newyear everyone, Lars Hesel Christensen -- Lars Hesel Christensen Ericsson Denmark A/S, Telebit lars.hesel.xx.christensen(a)ericsson.com

16 years, 4 months

3
9
0 / 0

Re: better malloc strategies

by Howard Chu

Howard Chu wrote: > I've tested again with libhoard 3.5.1, and it's actually superior to > libumem for both speed and fragmentation. Here are some results against > glibc 2.3.3, libhoard, and libumem: I've retested with current HEAD, adding Google's tcmalloc to the mix. glibc hoard umem tcmalloc size time size time size time size time initial 731 741 741 744 single 1368 2:03.02 1809 1:26.67 1778 1:43.49 1453 0:58.77 single 1369 2:23.61 1827 0:25.68 1825 0:18.40 1512 0:21.74 single 1368 1:48.41 1828 0:16.52 1826 0:21.07 1529 0:22.97 single 1368 1:48.59 1829 0:16.59 1827 0:16.95 1529 0:17.07 single 1368 1:48.72 1829 0:16.53 1827 0:16.61 1529 0:17.01 single 1368 1:48.39 1829 0:20.70 1827 0:16.56 1529 0:16.99 single 1368 1:48.63 1830 0:16.56 1828 0:17.48 1529 0:17.29 single 1384 1:48.14 1829 0:16.64 1828 0:22.17 1529 0:16.94 four 1967 1:20.21 1918 0:35.96 1891 0:29.95 1606 0:42.48 four 2002 1:10.58 1919 0:30.07 1911 0:29.00 1622 0:42.38 four 2009 1:33.45 1920 0:42.06 1911 0:40.01 1628 0:40.68 four 1998 1:32.94 1920 0:35.62 1911 0:39.11 1634 0:30.41 four 1995 1:35.47 1920 0:34.20 1911 0:28.40 1634 0:40.80 four 1986 1:34.38 1920 0:28.92 1911 0:31.16 1634 0:40.42 four 1989 1:33.23 1920 0:31.48 1911 0:33.73 1634 0:33.97 four 1999 1:33.04 1920 0:33.47 1911 0:38.33 1634 0:40.91 slapd CPU 26:31.56 8:34.78 8:33.39 9:19.87 > The initial size is the size of the slapd process right after startup, > with the process totally idle. The id2entry DB is 1.3GB with about > 360,000 entries, BDB cache at 512M, entry cache at 70,000 entries, > cachefree 7000. The subsequent statistics are the size of the slapd > process after running a single search filtering on an unindexed > attribute, basically spanning the entire DB. The entries range in size > from a few K to a couple megabytes. The BDB and slapd cache configurations are the same, but the machine now has 4GB of RAM so the entire DB fits in the filesystem buffer cache. As such there is no disk I/O during these tests. > > After running the single ldapsearch 4 times, I then ran the same search > again with 4 jobs in parallel. There should of course be some process > growth for the resources for 3 additional threads (about 60MB is about > right since this is an x86_64 system). This time I ran the tests 8 times each. Basically I was looking for the slapd process size to stabilize at a constant number... > > The machine only had 2GB of RAM, and you can see that with glibc malloc > the kswapd got really busy in the 4-way run. The times might improve > slightly after I add more RAM to the box. But clearly glibc malloc is > fragmenting the heap like crazy. The current version of libhoard looks > like the winner here. There was no swap activity (or any other disk activity) this time, so the glibc numbers don't explode like they did before. But even so, the other allocators are 5-6 times faster. The Google folks claim tcmalloc is the fastest allocator they have ever seen. These tests show it is fast, but it is not the fastest. It definitely is the most space-efficient multi-threaded allocator though. It's hard to judge just by the execution times of the ldapsearch commands, so I also recorded the amount of CPU time the slapd process consumed by the end of each test. That gives a clearer picture of the performance differences for each allocator. These numbers aren't directly comparable to the ones I posted on August 30, because I was using a hacked up RE23 there and used HEAD here. Going thru these tests is a little disturbing; I would expect the single-threaded results to be 100% repeatable but they're not quite. In one run I saw glibc use up 2.1GB during the 4-way test, and never shrink back down. I later re-ran the same test (because I hadn't recorded the CPU usage the first time around) and got these numbers instead. The other thing that's disturbing is just how bad glibc's malloc really is, even in the single-threaded case which is supposed to be its ideal situation. The other thing to note is that for this run with libhoard, I doubled its SUPERBLOCK_SIZE from 64K to 128K. That's probably why it's less space-efficient here than umem, while it was more efficient in the August results. I guess I'll have to recompile that with the original size to see what difference that makes. For now I'd say my preference would be tcmalloc... -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

16 years, 4 months

2
10
0 / 0

Re: commit: ldap/servers/slapd/overlays accesslog.c

by Kurt D. Zeilenga

It might be more appropriate to handle this issue on the consumer than the provider. An arbitrary LDAP sync client might want this and other DSA specific attributes included in the content. That is, the provider should not assume the consumer is doing server-to-server replication. Kurt At 01:10 PM 12/23/2006, hyc(a)OpenLDAP.org wrote: >Update of /repo/OpenLDAP/pkg/ldap/servers/slapd/overlays > >Modified Files: > accesslog.c 1.68 -> 1.69 > >Log Message: >ITS#4788 don't return auditContext attr on syncrepl searches > > >CVS Web URLs: > http://www.openldap.org/devel/cvsweb.cgi/servers/slapd/overlays/ > http://www.openldap.org/devel/cvsweb.cgi/servers/slapd/overlays/accesslog.c > >Changes are generally available on cvs.openldap.org (and CVSweb) >within 30 minutes of being committed.

16 years, 5 months

4
7
0 / 0

Privileged connections in back-ldap/back-meta

by Pierangelo Masarati

I'm facing the problem of having too many threads operating on the same privileged connections in back-ldap/back-meta, which seems to cause poor scalability. This mainly occurs because we want all operations to be performed as a privileged identity with proxyAuthz. For this purpose I plan to move privileged connections out of the connection AVL tree into ldapinfo_t, so that we save a search in the tree, and keep multiple instances of privileged connections in a LDAP_TAILQ structure of fixed maximum length to use them round-robin. Each type of privileged connection uses a dedicated list, so in our case we basically keep two lists: one for the privileged user and one for binds, which are now serialized into one privileged connection since all subsequent operations use proxyAuthz. A prototype is being tested, which doesn't work too bad (scalability did not dramatically improve yet, so there might be other bottlenecks, but at least some repeatable transient hangs disappeared). Comments? p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.n.c. Via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ------------------------------------------ Office: +39.02.23998309 Mobile: +39.333.4963172 Email: pierangelo.masarati(a)sys-net.it ------------------------------------------

16 years, 5 months

2
1
0 / 0

Re: Lifetime and Documenting of LDAP_DEPRECATED, outside ldap.h

by Kurt D. Zeilenga

At 06:35 AM 12/20/2006, Gavin Henry wrote: >> Is there a date set that LDAP_DEPRECATED will not be supported anymore? No. >> And should we (docs team) put something on Faq-O-Matic (devel section), or >> in the new devel guide (once an outline/template has been put in place) >> that presents the "Best Practice" for updating their code? Personally, I think it would be better to focus the limited documentation time/energy we have toward documenting non-deprecated code (and discussing how to write clients using non-deprecated code). But as I've noted before, (code and documentation) developers should work on what they need/want... and, if its generally useful, it will find its way into OpenLDAP Software (and/or other materials provided by the Project). That is, don't let my (or other statements) stop anyone from working on what they need/want. -- Kurt

16 years, 5 months

2
1
0 / 0

Lifetime and Documenting of LDAP_DEPRECATED, outside ldap.h

by Gavin Henry

Dear All, We've been working on some LDAP client code for other Open Source projects and most of them, as well as Linux Distros define LDAP_DEPRECATED to keep the code working etc., as they haven't been updated for the new libs. Is there a date set that LDAP_DEPRECATED will not be supported anymore? And should we (docs team) put something on Faq-O-Matic (devel section), or in the new devel guide (once an outline/template has been put in place) that presents the "Best Practice" for updating their code? Or should we leave it up to them to read over ldap.h, as Kurt put: /* deprecated, please use foobar function */ in it. Thoughts? Thanks, Gavin. -- Kind Regards, Gavin Henry. Managing Director. T +44 (0) 1224 279484 M +44 (0) 7930 323266 F +44 (0) 1224 824887 E ghenry(a)suretecsystems.com Open Source. Open Solutions(tm). http://www.suretecsystems.com/

16 years, 5 months

1
1
0 / 0

Re: commit: ldap/servers/slapd/back-ldap init.c bind.c

by Pierangelo Masarati

ando(a)OpenLDAP.org wrote: > Log Message: > (almost) blind fixes to ITS#4782 > ldap_back_is_proxy_authz() should be fixed now; monitor should be fixed as well, I'll check later with all backends built as modules. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.n.c. Via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ------------------------------------------ Office: +39.02.23998309 Mobile: +39.333.4963172 Email: pierangelo.masarati(a)sys-net.it ------------------------------------------

16 years, 5 months

1
0
0 / 0

Cancel operation in RE23?

by Michael Ströder

HI! Is the cancel operation already supported in RE23? It seems so (playing around with it with python-ldap). But supportedFeatures in root DSE does not contain 1.3.6.1.1.8. RFC 3909 states: Servers SHOULD indicate their support for this extended operation by providing 1.3.6.1.1.8 as a value of the 'supportedExtension' attribute type in their root DSE. A server MAY choose to advertise this extension only when the client is authorized to use it. Ciao, Michael.

16 years, 5 months

2
2
0 / 0

Re: commit: ldap/servers/slapd/overlays rwm.c

by Pierangelo Masarati

hyc(a)OpenLDAP.org wrote: > Log Message: > Fix callback handling > There seems to be many problems passing test047. I'm fixing a couple of them, but more remain. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.n.c. Via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ------------------------------------------ Office: +39.02.23998309 Mobile: +39.333.4963172 Email: pierangelo.masarati(a)sys-net.it ------------------------------------------

16 years, 5 months

1
0
0 / 0

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel December 2006