- openldap-bugs - openldap.org

Re: (ITS#6280) Race condition in TLS/SSL init code
by hyc＠symas.com 03 Sep '09

03 Sep '09

donald.lewis(a)teradata.com wrote: > Full_Name: Jeff Lewis > Version: 2.4.17 > OS: SuSE Linux, ES9& 10 > URL: > Submission from: (NULL) (153.65.16.10) > > > There appears to be a race condition in SSL/TLS setup in the LDAP client library > bind fucntions. We have workarounds for the problem in 2.3 and 2.4 which we > plan to use until we can get a proper fix so there is no time pressure on you > from my end. Since we have a viable workaround, we're happy leaving the choice > to address the issue or not to you. libldap is not documented to be thread-safe, nor is it intended to be. > 2. If, before the first bind attempt, I invoke ldap_set_option passing a NULL > LDAP pointer, LDAP_OPT_X_TLS_NEWCTX and a value of 0, I do not experience the > problem. We're quite happy with this as a workaround. This supports my belief > that the race is somewhere in the TLS initialization code. > > In 2.3.39 (our current OpenLDAP client), the problem is more severe. When > multiple threads collide in the TLS setup logic, the problem causes TLS to > become completely unusable and no connection can be established to the server. > We've repaired this problem in libraries/libldap/tls.c by calling > ldap_pvt_tls_init through pthread_once and we're quite happy to live with this > until we can get a 2.4 OpenLDAP imported. Your app is responsible for making sure that all libldap init functions are processed before multiple threads are created. Using pthread_once or whatever approach you take is up to you. Closing this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6280) Race condition in TLS/SSL init code
by Donald.Lewis＠Teradata.com 03 Sep '09

03 Sep '09

This is a multi-part message in MIME format. ------_=_NextPart_001_01CA2C19.8E008473 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Forgot to say the the bind functions return -1 when the problem occurs. ------_=_NextPart_001_01CA2C19.8E008473 Content-Type: text/html; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN"> <HTML> <HEAD> <META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; = charset=3Dus-ascii"> <META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version = 6.5.7654.12"> <TITLE>Re: (ITS#6280) Race condition in TLS/SSL init code</TITLE> </HEAD> <BODY>  <P><FONT SIZE=3D2 FACE=3D"Arial">Forgot to say the the bind functions = return -1 when the problem occurs.</FONT> </P> </BODY> </HTML> ------_=_NextPart_001_01CA2C19.8E008473--

1 0

(ITS#6280) Race condition in TLS/SSL init code
by donald.lewis＠teradata.com 03 Sep '09

03 Sep '09

Full_Name: Jeff Lewis Version: 2.4.17 OS: SuSE Linux, ES9 & 10 URL: Submission from: (NULL) (153.65.16.10) There appears to be a race condition in SSL/TLS setup in the LDAP client library bind fucntions. We have workarounds for the problem in 2.3 and 2.4 which we plan to use until we can get a proper fix so there is no time pressure on you from my end. Since we have a viable workaround, we're happy leaving the choice to address the issue or not to you. In OpenLDAP release 2.4.17, there is a race condition of some sort when SSL or TLS is used. When two or more threads collide in the SSL/TLS setup code, one or both threads fail to connect. I believe this to be some sort of a one-time initialization problem because it is the first few bind attempts that seem to fail and once one bind succeeds, the rest seem to succeed. We observe this with simple binds using ldap_sasl_bind_s, ldap_simple_bind_s, or ldap_bind_s. We observe it on connections protected with SSL. We also observe it on connections protected with TLS. We do not observe any problems involving normal, unprotected LDAP connections. It would appear that SSL or TLS has to be in play. I've got two workarounds: 1. If I serialize the binds with a mutex, the problem disappears. This is not something we'd want to do in our product because of the large number of binds we have to do in a short period of time. 2. If, before the first bind attempt, I invoke ldap_set_option passing a NULL LDAP pointer, LDAP_OPT_X_TLS_NEWCTX and a value of 0, I do not experience the problem. We're quite happy with this as a workaround. This supports my belief that the race is somewhere in the TLS initialization code. In 2.3.39 (our current OpenLDAP client), the problem is more severe. When multiple threads collide in the TLS setup logic, the problem causes TLS to become completely unusable and no connection can be established to the server. We've repaired this problem in libraries/libldap/tls.c by calling ldap_pvt_tls_init through pthread_once and we're quite happy to live with this until we can get a 2.4 OpenLDAP imported.

1 0

Re: (ITS#6277) Missing documentation for cn=config
by gavin.henry＠gmail.com 02 Sep '09

02 Sep '09

Of course, but Ryan also mentioned the password policy overlay here and loading overlays in general via cn=config. On 01/09/2009, hyc(a)symas.com <hyc(a)symas.com> wrote: > ryans(a)aweber.com wrote: >> Gavin, >> >>> Well there are two places that talk about how to convert from slapd.conf >>> to cn=config formats. In the guide and man pages, so that is the best >>> way to do a full conversion and see the end result. >>> >>> Where would you like to see these added? >>> >>> Thanks for the feedback as always! >> >> My vote would be section 5.2.2.3 of the Admin Guide. Currently, it has 2 >> examples of loading modules consisting of >> libtool libraries, but I would think that it would be good to give an >> example which, for pedagogical purposes, explained >> how to instantiate an overlay in cn=config. For example, ppolicy and >> autogroup: > > autogroup is a contrib module. We only document usage of core features in > the > Admin Guide. > >> dn: olcOverlay=ppolicy,olcDatabase={1}hdb,cn=config >> objectClass: olcOverlayConfig >> objectClass: olcPPolicyConfig >> olcOverlay: ppolicy > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ > > > -- Sent from my mobile device http://www.suretecsystems.com/services/openldap/ http://www.suretectelecom.com

1 0

Re: (ITS#6277) Missing documentation for cn=config
by hyc＠symas.com 01 Sep '09

01 Sep '09

ryans(a)aweber.com wrote: > Gavin, > >> Well there are two places that talk about how to convert from slapd.conf >> to cn=config formats. In the guide and man pages, so that is the best >> way to do a full conversion and see the end result. >> >> Where would you like to see these added? >> >> Thanks for the feedback as always! > > My vote would be section 5.2.2.3 of the Admin Guide. Currently, it has 2 examples of loading modules consisting of > libtool libraries, but I would think that it would be good to give an example which, for pedagogical purposes, explained > how to instantiate an overlay in cn=config. For example, ppolicy and autogroup: autogroup is a contrib module. We only document usage of core features in the Admin Guide. > dn: olcOverlay=ppolicy,olcDatabase={1}hdb,cn=config > objectClass: olcOverlayConfig > objectClass: olcPPolicyConfig > olcOverlay: ppolicy -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6277) Missing documentation for cn=config
by ghenry＠OpenLDAP.org 01 Sep '09

01 Sep '09

----- ryans(a)aweber.com wrote: > Gavin, > > > Well there are two places that talk about how to convert from > slapd.conf > > to cn=config formats. In the guide and man pages, so that is the > best > > way to do a full conversion and see the end result. > > > > Where would you like to see these added? > > > > Thanks for the feedback as always! > > My vote would be section 5.2.2.3 of the Admin Guide. Hi Ryan, Thanks for the time you spent replying in length. Oh, maybe that time could have been spent writing a patch to docs! ;-) I'll take on board your suggestions and make some changes when I can. Patches welcome! Cheers. -- Kind Regards, Gavin Henry. OpenLDAP Engineering Team. E ghenry(a)OpenLDAP.org Community developed LDAP software. http://www.openldap.org/project/

1 0

Re: (ITS#6277) Missing documentation for cn=config
by ryans＠aweber.com 01 Sep '09

01 Sep '09

Gavin, > Well there are two places that talk about how to convert from slapd.conf > to cn=config formats. In the guide and man pages, so that is the best > way to do a full conversion and see the end result. > > Where would you like to see these added? > > Thanks for the feedback as always! My vote would be section 5.2.2.3 of the Admin Guide. Currently, it has 2 examples of loading modules consisting of libtool libraries, but I would think that it would be good to give an example which, for pedagogical purposes, explained how to instantiate an overlay in cn=config. For example, ppolicy and autogroup: dn: olcOverlay=ppolicy,olcDatabase={1}hdb,cn=config objectClass: olcOverlayConfig objectClass: olcPPolicyConfig olcOverlay: ppolicy dn: olcOverlay=autogroup,olcDatabase={1}hdb,cn=config objectClass: olcOverlayConfig objectClass: olcAutomaticGroups olcOverlay: autogroup olcAGattrSet: groupOfNames labeledURI member Perhaps it would also be good to emphasize the need for overlay-specific objectclasses, such as the aforementioned 'olcOverlayConfig' and 'olcPPolicyConfig'. It probably also wouldn't hurt to mention that there are certain overlay-specific attributes, i.e. olcAGattrSet, that are necessary to make full use of the module, and that grepping through the contrib module's source can help one identify said attributes' names in lieu of adequate documentation. Something like: grep -C3 NAME contrib/slapd-modules/autogroup/autogroup.c, maybe? Thanks as always, Ryan

1 0

(ITS#6279) autogroup overlay causes slapcat segfault
by ryans＠aweber.com 31 Aug '09

31 Aug '09

Full_Name: Ryan Steele Version: 2.4.17 OS: Ubuntu URL: http://pastie.org/600973 Submission from: (NULL) (71.230.37.172) When the autogroup overlay directives are included in slapd.conf, it seems to work okay. When adding it to the cn=config DIT, however, it causes segfaults on slapcat. This is version 1.6 of the autogroup overlay (which fixed segfaults when using slapd.conf). Steps to reproduce (starting from a pristine installation) and strace of slapcat are in the pastebin linked below.

1 0

Re: (ITS#6278) Patch - Enhancement - provide support for PEM files in MozNSS crypto
by hyc＠symas.com 29 Aug '09

29 Aug '09

rmeggins(a)redhat.com wrote: > Full_Name: Rich Megginson > Version: 2.4.18 (current CVS HEAD) > OS: Fedora > URL: ftp://ftp.openldap.org/incoming/openldap-2.4.18-moznss-20090828.patch > Submission from: (NULL) (76.113.59.19) > > > This patch adds support for reading PEM encoded cert and key files to the MozNSS > crypto implementation. It depends on having the nsspem module library available > somewhere for the runtime linker to find it (e.g. on a linux system, it uses > dlopen() to load libnsspem.so). This module is available on Fedora and is > provided by the package nss-3.12.3.99. Work is underway to incorporate the PEM > module into MozNSS upstream at mozilla.org. The current source code repository > is > git://fedorapeople.org/~rcritten/pemnss.git > > The tlsm_init code attempts to determine if you are using PEM files or are using > a MozNSS key/cert database. If you specify the TLS cacertdir directive, and > that directory has valid key/cert databases in it, it will use them. Otherwise, > it will load the PEM module and attempt to load the certs and keys specified by > cacertdir, cacertfile, certfile, and keyfile. Thanks. I tweaked the #includes this time around because /usr/include/nspr and /usr/include/nss already exist on my Ubuntu machine. Unfortunately, several of the nspr header files still use unqualified paths to reference each other, so you still need -I/usr/include/nspr to get this to compile successfully. Needless to say, I think nspr needs to be cleaned up a bit more before it's ready for primetime. With that small change it compiles OK so I've committed it to HEAD. I'm not too thrilled to see PL_strdup / PL_strfree and friends here; everything else in the source tree uses our wrappers that funnel into ber_memalloc and we rely on this for accounting, leak testing and other validation. (Not to mention avoiding silly crashes when running on Windows where each DLL has its own heap manager...) It looks like all of the uses are self-contained and won't escape this module so it's probably OK. I'd suggest overriding the NSPR malloc hooks but that would probably mess up any other applications. We can probably roll this code into RE24 as well but I'm still going to omit the configure switch for the moment. > This patch file is derived from OpenLDAP Software. All of the > modifications to OpenLDAP Software represented in the following > patch(es) were developed by Red Hat, Inc.. Red Hat, Inc. has not > assigned rights and/or interest in this work to any party. I, Richard > Megginson am authorized by Red Hat, Inc., my employer, to release this > work under the following terms. > > Red Hat, Inc. hereby place the following modifications to OpenLDAP > Software (and only these modifications) into the public domain. Hence, > these modifications may be freely used and/or redistributed for any > purpose with or without attribution and/or other notice. > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#6278) Patch - Enhancement - provide support for PEM files in MozNSS crypto
by rmeggins＠redhat.com 28 Aug '09

28 Aug '09

Full_Name: Rich Megginson Version: 2.4.18 (current CVS HEAD) OS: Fedora URL: ftp://ftp.openldap.org/incoming/openldap-2.4.18-moznss-20090828.patch Submission from: (NULL) (76.113.59.19) This patch adds support for reading PEM encoded cert and key files to the MozNSS crypto implementation. It depends on having the nsspem module library available somewhere for the runtime linker to find it (e.g. on a linux system, it uses dlopen() to load libnsspem.so). This module is available on Fedora and is provided by the package nss-3.12.3.99. Work is underway to incorporate the PEM module into MozNSS upstream at mozilla.org. The current source code repository is git://fedorapeople.org/~rcritten/pemnss.git The tlsm_init code attempts to determine if you are using PEM files or are using a MozNSS key/cert database. If you specify the TLS cacertdir directive, and that directory has valid key/cert databases in it, it will use them. Otherwise, it will load the PEM module and attempt to load the certs and keys specified by cacertdir, cacertfile, certfile, and keyfile. This patch file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Red Hat, Inc.. Red Hat, Inc. has not assigned rights and/or interest in this work to any party. I, Richard Megginson am authorized by Red Hat, Inc., my employer, to release this work under the following terms. Red Hat, Inc. hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice.

1 0

Re: (ITS#6277) Missing documentation for cn=config
by hyc＠symas.com 27 Aug '09

27 Aug '09

masarati(a)aero.polimi.it wrote: >>> Full_Name: Ryan Steele >>> Version: 2.4.15 >>> OS: Ubuntu 8.04 LTD >>> URL: ftp://ftp.openldap.org/incoming/ >>> Submission from: (NULL) (207.106.239.81) >>> >>> >>> According to chapter 5 of the admin guide, "some of the backends and of >>> the >>> distributed overlays do not support runtime configuration yet. In those >> cases, >>> the old style slapd.conf(5) file must be used." However, there is no >>> documentation on which backends and overlays do and don't have said >>> support. >>> Without grokking the code, it is a trial-and-error operation at best. >> >> Point taken. I'm pretty sure they all do now. Will check and update that >> section. > > back-meta and back-sql don't yet. I have half (er, somewhere between 0 > and 100%, boundaries not included) in a working dir somewhere, and rough > ideas about what to do with back-meta. All of the core overlays now support cn=config. Yes, contrib is trial-and-error. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6277) Missing documentation for cn=config
by masarati＠aero.polimi.it 27 Aug '09

27 Aug '09

>> Full_Name: Ryan Steele >> Version: 2.4.15 >> OS: Ubuntu 8.04 LTD >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (207.106.239.81) >> >> >> According to chapter 5 of the admin guide, "some of the backends and of >> the >> distributed overlays do not support runtime configuration yet. In those > cases, >> the old style slapd.conf(5) file must be used." However, there is no >> documentation on which backends and overlays do and don't have said >> support. >> Without grokking the code, it is a trial-and-error operation at best. > > Point taken. I'm pretty sure they all do now. Will check and update that > section. back-meta and back-sql don't yet. I have half (er, somewhere between 0 and 100%, boundaries not included) in a working dir somewhere, and rough ideas about what to do with back-meta. p.

1 0

Re: (ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by rlvcosta＠yahoo.com 27 Aug '09

27 Aug '09

Quanah, Please see my comments in your previous e-mail. Thanks, Rodrigo. Quanah Gibson-Mount wrote: > --On Thursday, August 27, 2009 6:39 AM -0700 Rodrigo Costa > <rlvcosta(a)yahoo.com> wrote: > >> Quanah, >> >> Please see answer in your previous e-mail below. >> >> I'm also sending the information I could collect attached since it is a >> small file(5KB). >> >> The behavior that appears strange and that could indicate a problem is >> the fact that even when consumer is stopped the provider still doing >> something for a long time. This doesn't appear to be correct. >> >> Other strange behavior is that when system enters in this state one >> provider CPU stays running around 100% CPU usage. I made a jmeter script >> to test individual bind/search(no ldapsearch *) and then even with some >> load(like 200 simultaneous query) I do not see CPU in 100%. Something >> doesn't appear to be ok since I do not see why CPU should enter in 100% >> permanently. > > I explained to you previously why this would be. Other comments inline. > >>> Why are you stopping the provider to do a slapcat? >> [Rodrigo]Faster dump of data. And in any case if other situation like a >> problema occurs the secondary system could stay disconnect for other >> reasons. > [Rodrigo] I have 2 reasons : 1)Since backup takes sometime and DB has multiple branches for the same record the only way to have a consistent backup is executing a cold backup; 2)slapcat in a stop slapd could perform faster and also fulfill item 1 above(cold backup) > Do you have any evidence that an offline slapcat is faster than one > while slapd is running? I don't understand what you mean in the rest > of that sentence. > [Rodrigo] I didn't try with load traffic but it seems reasonable if a > cold backup is faster and cleaner than a hot backup. >>>> Even a small number of entrances are different when consumer in >>>> Provider 2 >>>> connects to Provider 1 then syncrepl enters in the full DB search as >>>> expected. >>> >>> >>> What is your sessionlog setting on each provider for the syncprov >>> overlay? >> [Rodrigo] >> syncprov-checkpoint 10000 120 >> syncprov-sessionlog 100000 > > Hm, I would probably checkpoint the cookie a lot more frequently than > you have it set to. The sessionlog setting seems fine to me. [Rodrigo] Ok > >> Same configuration in both systems. >>> >>>> For definition purposes I have some memory limitations where I need to >>>> limit dncachesize for around 80% of DB entrances. >>> >>> We already went through other things you could do to reduce your >>> memory footprint in other ways. You've completely ignored that >>> advice. As long as your dncachesize is in this state, I don't expect >>> things to behave normally. >> [Rodrigo]I implemented what was possible. The end is this cache config >> possible by the memory constraints : >> # Cache values >> # cachesize 10000 >> cachesize 20000 >> dncachesize 3000000 >> # dncachesize 400000 >> # idlcachesize 10000 >> idlcachesize 30000 >> # cachefree 10 >> cachefree 100 > > You don't say anything in here about your DB_CONFIG settings, which is > where you could stand to gain the most amount of memory back. I do > see you're definitely running a very restricted > cachesize/idlcachesize. ;) > [Rodrigo]DB_CONFIG is using only 100MB of memory and DB_LOG_AUTOREMOVE. > > >>> What value did you set for "cachefree"? >> [Rodrigo] cachefree 100 > [Rodrigo] I made the change proposed and tested. The behavior was > really better since after dncachesize was filled the issue did not > repeated as before. BUT it just took more time until the behavior repeats. After some more time then just after dncachesize reaches around 3Mi the behavior returned. What happens is : 1-> Provider 1 CPU start to consume around 100%; 2-> Consumer 2 CPU goes to 0% consumption(before it was around 10% when replication in place); 3-> Replication never ends(I cannot see in the Provider 2 data) and even I stop Consumer 2(or slapd) the CPU in Provider 1 remains days in 100%. Looks like code enter in a dead loop which I could not identify the condition or the requirement to avoid it. I generated some GDB traces and as soon as possible(there is space) I will put in the ftp. > > This value is likely substantially way too low for your system > configuration. This is how many entries get freed from any of the > caches. With your dncachesize being 3,000,000, removing 100 entries > from it will do hardly anything, and may be part of the issue. If it > wasn't for the major imbalance between your entry, idl, and > dncachesizes, I would suggest a fairly high value like 100,000. But > given your entry cache is 20,000, you'll probably have to limit the > cachefree to 5000-10000. But it is going to need to be higher than 100. > > --Quanah > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and collaboration > >

1 0

Re: (ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by quanah＠zimbra.com 27 Aug '09

27 Aug '09

--On Thursday, August 27, 2009 6:39 AM -0700 Rodrigo Costa <rlvcosta(a)yahoo.com> wrote: > Quanah, > > Please see answer in your previous e-mail below. > > I'm also sending the information I could collect attached since it is a > small file(5KB). > > The behavior that appears strange and that could indicate a problem is > the fact that even when consumer is stopped the provider still doing > something for a long time. This doesn't appear to be correct. > > Other strange behavior is that when system enters in this state one > provider CPU stays running around 100% CPU usage. I made a jmeter script > to test individual bind/search(no ldapsearch *) and then even with some > load(like 200 simultaneous query) I do not see CPU in 100%. Something > doesn't appear to be ok since I do not see why CPU should enter in 100% > permanently. I explained to you previously why this would be. Other comments inline. >> Why are you stopping the provider to do a slapcat? > [Rodrigo]Faster dump of data. And in any case if other situation like a > problema occurs the secondary system could stay disconnect for other > reasons. Do you have any evidence that an offline slapcat is faster than one while slapd is running? I don't understand what you mean in the rest of that sentence. >>> Even a small number of entrances are different when consumer in >>> Provider 2 >>> connects to Provider 1 then syncrepl enters in the full DB search as >>> expected. >> >> >> What is your sessionlog setting on each provider for the syncprov >> overlay? > [Rodrigo] > syncprov-checkpoint 10000 120 > syncprov-sessionlog 100000 Hm, I would probably checkpoint the cookie a lot more frequently than you have it set to. The sessionlog setting seems fine to me. > Same configuration in both systems. >> >>> For definition purposes I have some memory limitations where I need to >>> limit dncachesize for around 80% of DB entrances. >> >> We already went through other things you could do to reduce your >> memory footprint in other ways. You've completely ignored that >> advice. As long as your dncachesize is in this state, I don't expect >> things to behave normally. > [Rodrigo]I implemented what was possible. The end is this cache config > possible by the memory constraints : ># Cache values ># cachesize 10000 > cachesize 20000 > dncachesize 3000000 ># dncachesize 400000 ># idlcachesize 10000 > idlcachesize 30000 ># cachefree 10 > cachefree 100 You don't say anything in here about your DB_CONFIG settings, which is where you could stand to gain the most amount of memory back. I do see you're definitely running a very restricted cachesize/idlcachesize. ;) >> What value did you set for "cachefree"? > [Rodrigo] cachefree 100 This value is likely substantially way too low for your system configuration. This is how many entries get freed from any of the caches. With your dncachesize being 3,000,000, removing 100 entries from it will do hardly anything, and may be part of the issue. If it wasn't for the major imbalance between your entry, idl, and dncachesizes, I would suggest a fairly high value like 100,000. But given your entry cache is 20,000, you'll probably have to limit the cachefree to 5000-10000. But it is going to need to be higher than 100. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#6277) Missing documentation for cn=config
by Gavin Henry 27 Aug '09

27 Aug '09

> Full_Name: Ryan Steele > Version: 2.4.15 > OS: Ubuntu 8.04 LTD > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (207.106.239.81) > > > According to chapter 5 of the admin guide, "some of the backends and of the > distributed overlays do not support runtime configuration yet. In those cases, > the old style slapd.conf(5) file must be used." However, there is no > documentation on which backends and overlays do and don't have said support. > Without grokking the code, it is a trial-and-error operation at best. Point taken. I'm pretty sure they all do now. Will check and update that section. > Also, there is no documentation on how to add overlay-specific directives (man > pages or otherwise). Take, for example, autogroup-attrset; the olcAGattrSet > directive is only described in autogroup.c, and nowhere else. This is not the > only instance of missing module documentation, but it should give a general idea > of where to look. Some of the contribs one don't come with a man page unfortunately. Core ones do. I'll look into the relevant sections and add one or two examples. > Of course, grepping the code is easy enough, but you shouldn't have to do that > to learn how to achieve simple configurations. If there's going to be a > fundamental paradigm shift from slapd.conf to cn=config, there has to be > documentation (man pages, admin guide sections, et. al.) on the appropriate > methods for achieving what once was done through slapd.conf. IMHO, that should > include concrete examples, such as the following, pulled from the Courier > documentation: > > The following LDIF could be used to add [the auditlog] overlay to cn=config > (adjust to suit): > > dn: olcOverlay=auditlog,olcDatabase={1}hdb,cn=config > changetype: add > objectClass: olcOverlayConfig > objectClass: olcAuditLogConfig > olcOverlay: auditlog > olcAuditlogFile: /tmp/auditlog.ldif > Well there are two places that talk about how to convert from slapd.conf to cn=config formats. In the guide and man pages, so that is the best way to do a full conversion and see the end result. Where would you like to see these added? Thanks for the feedback as always! Gavin.

1 0

(ITS#6277) Missing documentation for cn=config
by ryans＠aweber.com 27 Aug '09

27 Aug '09

Full_Name: Ryan Steele Version: 2.4.15 OS: Ubuntu 8.04 LTD URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (207.106.239.81) According to chapter 5 of the admin guide, "some of the backends and of the distributed overlays do not support runtime configuration yet. In those cases, the old style slapd.conf(5) file must be used." However, there is no documentation on which backends and overlays do and don't have said support. Without grokking the code, it is a trial-and-error operation at best. Also, there is no documentation on how to add overlay-specific directives (man pages or otherwise). Take, for example, autogroup-attrset; the olcAGattrSet directive is only described in autogroup.c, and nowhere else. This is not the only instance of missing module documentation, but it should give a general idea of where to look. Of course, grepping the code is easy enough, but you shouldn't have to do that to learn how to achieve simple configurations. If there's going to be a fundamental paradigm shift from slapd.conf to cn=config, there has to be documentation (man pages, admin guide sections, et. al.) on the appropriate methods for achieving what once was done through slapd.conf. IMHO, that should include concrete examples, such as the following, pulled from the Courier documentation: The following LDIF could be used to add [the auditlog] overlay to cn=config (adjust to suit): dn: olcOverlay=auditlog,olcDatabase={1}hdb,cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcAuditLogConfig olcOverlay: auditlog olcAuditlogFile: /tmp/auditlog.ldif

1 0

Re: (ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by quanah＠zimbra.com 27 Aug '09

27 Aug '09

--On Thursday, August 27, 2009 6:39 AM -0700 Rodrigo Costa <rlvcosta(a)yahoo.com> wrote: > Quanah, > > Please see answer in your previous e-mail below. > > I'm also sending the information I could collect attached since it is a > small file(5KB). The ITS system does not accept attachments. You'll need to upload it to the FTP site, which had space cleared up. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#6276) paused pool can deadlock if writers are waiting
by h.b.furuseth＠usit.uio.no 27 Aug '09

27 Aug '09

Re-reading this, I'm not sure what either of us are talking about... About reading a byte and putting it back if select() says the socket is readable: I suppose this is only necessary if conn->c_writewaiter != 0, and maybe only if the pool is paused. And remember that the next select() will often also find the socket is readable, so we must read the next byte (from the socket, not the sockbuf) and put that on the sockbut stack, and so on. About multiple types of pauses: The concept must be rephrased for that to make sense. The pause mechanism is in effect a pool-wide read/write lock which favors writers over readers. A pool thread readlocks it while running a task. pool_pause() does unlock - writelock. pool_resume() and pool_pausecheck() do unlock - readlock. Now it's possible to talk about moving a few things out of this particular lock, and if necessary protect it with another lock. -- Hallvard

1 0

Re: (ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by rlvcosta＠yahoo.com 27 Aug '09

27 Aug '09

--0-698620411-1251380367=:64533 Content-Type: text/plain; charset=us-ascii Quanah, Please see answer in your previous e-mail below. I'm also sending the information I could collect attached since it is a small file(5KB). The behavior that appears strange and that could indicate a problem is the fact that even when consumer is stopped the provider still doing something for a long time. This doesn't appear to be correct. Other strange behavior is that when system enters in this state one provider CPU stays running around 100% CPU usage. I made a jmeter script to test individual bind/search(no ldapsearch *) and then even with some load(like 200 simultaneous query) I do not see CPU in 100%. Something doesn't appear to be ok since I do not see why CPU should enter in 100% permanently. Regards, Rodrigo. Quanah Gibson-Mount wrote: > --On Tuesday, August 25, 2009 6:16 PM +0000 rlvcosta(a)yahoo.com wrote: > >> The issue I'm facing is related, in a general user view, is when I stop >> the secondary Provider2(master 2) for backup purposes using slapcat. The >> Provider1(master 1) continues to provide ldap service where some >> entrances can be created during the time backup is running(no consumer >> from Provider 2). > > Why are you stopping the provider to do a slapcat? [Rodrigo]Faster dump of data. And in any case if other situation like a problema occurs the secondary system could stay disconnect for other reasons. > >> Even a small number of entrances are different when consumer in >> Provider 2 >> connects to Provider 1 then syncrepl enters in the full DB search as >> expected. > > > What is your sessionlog setting on each provider for the syncprov > overlay? [Rodrigo] syncprov-checkpoint 10000 120 syncprov-sessionlog 100000 Same configuration in both systems. > >> For definition purposes I have some memory limitations where I need to >> limit dncachesize for around 80% of DB entrances. > > We already went through other things you could do to reduce your > memory footprint in other ways. You've completely ignored that > advice. As long as your dncachesize is in this state, I don't expect > things to behave normally. [Rodrigo]I implemented what was possible. The end is this cache config possible by the memory constraints : #Cache values #cachesize 10000 cachesize 20000 dncachesize 3000000 #dncachesize 400000 #idlcachesize 10000 idlcachesize 30000 #cachefree 10 cachefree 100 > >> I could also note that when in this situation the monitor cache, in a >> very slow pace, changes the cache in a single entrance. Being more >> specific : >> >> dn: cn=Database 1,cn=Databases,cn=Monitor >> structuralObjectClass: monitoredObject >> creatorsName: >> modifiersName: >> createTimestamp: 20090821145848Z >> modifyTimestamp: 20090821145848Z >> monitoredInfo: bdb >> monitorIsShadow: TRUE >> namingContexts: ou=CONTENT,o=domain,c=fr >> readOnly: FALSE >> monitorOverlay: syncprov >> olmBDBEntryCache: 19920 >> olmBDBDNCache: 3896287 >> olmBDBIDLCache: 2 >> olmDbDirectory: /var/openldap-data/bdb1/ >> entryDN: cn=Database 1,cn=Databases,cn=Monitor >> subschemaSubentry: cn=Subschema >> hasSubordinates: TRUE >> >> Stays running in the values 3896287 and 3896288. Looks like the memory >> re-use is being too short causing locks that takes long time causing >> a non >> synchronization. > > > What value did you set for "cachefree"? [Rodrigo] cachefree 100 > >> PS-> I could not put the file in the openldap ftp. It says device full. >> Please let me know how can I send this file. > > I've let the maintainer of the system know, hopefully there'll be > space available soon. [Rodrigo]I'm sending file attached since it is very small 5KB. > > --Quanah > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and collaboration > > --0-698620411-1251380367=:64533 Content-Type: application/octet-stream; name="replication_too_slow.zip" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="replication_too_slow.zip" UEsDBBQAAgAIABd5GTtTPer8EQEAAMUBAAAJABUAREJfQ09ORklHVVQJAAOu KJRKsCiUSlV4BADuAwECXZBbT4NAEIXf+RWT0EcLuysI8qYVmyYKia1Gn8jC TpFkuQRWUv31LiCofZxvLufMMWG1EwHc3SabOLrfba1UpHQdxfu3aHPRA7UY MEIcm3j2JQHiBy4LHAZCSQhPDawME/aoQL0jCK54yjuEooISy7r9hIxnutEV X2gZpmF2qJIRDQQIuMxhvk8I0PMehQGez1Pi+K53NS38KC+qR8nzbpEZq+Gt w2uUTO8Y//hDvE1ung/xU/gYv4TLOVnn0HP5geOpYUPmSYt5UVclP80O5sYf RGaWjmYZufaoyyYzmoqiBbvnrV03WEnBG1sHvdZq3SKtsGzqluvYjoVEyFrk SsuCXsVM6ThnR6psfu/pwvgGUEsDBBQAAgAIAFFzFTsgtwjuGwkAAAlhAAAw ABUAaXRzX3JlcGxpY2F0aW9uX29wZW5sZGFwX2dkYl90cmFjZXNfMjAwOTA4 MjEudHh0VVQJAAPK2I5KaA2PSlV4BADuAwEC7Vxtb9s4Ev4eIP+B2EWB5Jp4 +SaKCppu79rsXYBNNmjTOxyKhUBJVOKNbHklOZtgr/fbj5RkS7LlWH7LZe8o N6hNDofkzGieISnNFwThq5/39w5uAu8QZLeJFAEQo1H0CEQUAS/b39vfuy6K LXBQfjtGFnEYRBaG4ODHf1wBTLDjHB6e7O99CwGADxBSQinEoD8Ernsnk6GM 3Pv0MfU114NDRYcKOksyGjBNJ0dxFLm/iX6mCECYxAPwXdT39J/fS+MeU41w 3ohDGweC6kZpJEaBGwg5iIduINMsiR8L/qTgz7iklpeTZiLJ3HKKsz2MinLd D1SN6WRw+qMb+1E8lAvHNRURrYuIEttxnEpEBMK1RMQcjK1cROUoXT8eBrmk 3r3764/nf3nv4h7p4WWTKoSnZMcCyDW7KBAjd3Q/EUrFFhzor6fwgQe2z6F/ BAbjTD6UBXZI1eiAukSmTcYdxWn/oeefYNuuBG9jy5F02k1/OO1mFMdKz4my MpmAgwf9c8oZHuZMdZliyGxcKWM9TVprapI0NKkUyUldk6ihSUIhL4zYdePU 7ddGNU6TSR+Bd0x7tupoolqPQa7uJF83/P77Que4qqCwXlGKtbhqFft7x8fH 148jCd4kMhsnw7cgi4HSX9YfjuURiBPwa6Pq13E/U02mM8X1mTLMHGozPpmp TYmx2T+KzaKaJjHklo0RpZWD5utpkoeWZdc1+UusfnTWmw9Jm94KJsWPU2xb FmGWbbOjEoPcwmCVjGG73pBT6U3hARGWmMWDYhq0JLFsPww0yUD0y6qOksix UYu5+nxBEBvYNLBpXNAmsIl9B/NSk1Hs37mjceYOY/21G4BCLjGSlM8BqAgC iALMFwEoqlfkghJEhNSS9Rbr4mohPBH4gkG/geGsrPB91hybvRDcDUgbkDYg vVwS7Z8vDjcwbWDaOKFNYJo50MFOXUO5vNwseWxCdfsYUaUYx29TzAy3iTIE cyxCebsyCMSVzqVAfph7PxUiuL7wb6UbJWMVTiQ3ShiqMNeAj4ksNJCTKC42 pZVKpcCMoCaXsK9sRi/n45EeEfE86kmu/F8/tx9J7FCon+qXA23KLHIE1o8b 9DRlv+hIBSOcKcsMI3Fzio+AFs20wmrOAikUqQxJOphhbzKNVIrEv50bf5JO fwo/Z1YQaplANjUsDkPL93OfHd/LJBKPbjxS91B0V4tncipphwup7ClVWMBM KxUvY0JoYT/XZfo49EdJfJ8rwU+H5SS00EWg5jCIA3n6w/nlB/fq49mns8vr ylImTfPp6LvWmTAXnt1krsbQkNGU/VREDErayhorFWn7hlNR0cWiQqiLqBDu IipESpxwwkJUoazPIyehExKH5SRB3Ky3ynpmF3eOMtCh9LO+BprChWgqNqEK yCyVH/Wl8nppFo9KYnviGQXzN/SMuUEjvoFnRM56nvElB9prL0gMNpoA/eUE 6CYaN9G48Thrb5pRioNSkyrAy0RaBtArbpvZz7DRtDZgGWs0+Pc/vEF1vPpl oNNAp3FW/33o1BcNYMuJ0/xxTy5izwownKkoRUUYr1c82f1mh1FtB2VseuzV dhjVdoLGp8d2UqLasd2NzLThqHtDREvF6EzOwOwgbHAv91HmB5pvnegW3A6a FbgcDw+LgdaOETuPp9hMMUdvJrL540Y2m+w4v6QjOxPemPDGeKwNHqcJfX9d XEYlLgceRPPhTRsuk2mFcNrCm41weT2XBmpP3wTSEtwAugF0s1WxlWdpDEAb gDYeaDOA5oywJkDH3i8r7DwgQleA5vzyRPuzrs1F/462F8y62sCwgeGtwLCB XgO9xutsAL0U1qE3TKRc8cR8ZpXbBXxbXjR5zm1/vQoOlcrmcbll2OW2vy8h bgB2te2/yTZ7+QQmxIKFJhww4YAJB7a3Ku8QJIBwHEW7ihQuY5A+Drw4Apnw Im0qYQzEvehH+mdvzVBiOdfuscZyXpsEI8u5rxitLGW4m3Cmsx635Fa7anjX fleNQwGaiNLeDnwwKC9dBk7BgTLKsZ+1c03Bnw7BlNdm53r6ykR6p/uc7UyX u1nRGazIf4uTOzfqp1mzTT60kS6ftMllbNda+tmDavN7lI312zKnQHlShJhN tCfVhXfyUdcrgrvyu2ICg8DBgX5jpmCiKwORiaJWEio8fpSX6mBt2sZmFLzR d7qbRu5ADiY3+tuvR+D3Oq+qI5sR4R/NdcApgs3uax3ZDPkeeDPzXsRrm+T9 NPhLTBHniyfCLQvOTEQKQnEA3uh3dLRKZJLXPTEHLplDWuaALL5wDqoNgqWw 4tFEUK8xnJsDXDD6mXF/1fY2kiJLgQoFs/5Apm+/fq3s4K4whDmDG6cyUVUt NtdX5KR2m6gBpVGsLRAzqyq+FemtKrMIUnemU5W7+uWf6+t/uj98vnx/ff7T pesqsm+eume/2TAuWu61rF15++dZ8iwdz4aP9c6vpTpD/dyqpTOwz63GOoP2 Ksu1jrbRsi5b3pKtLYJFK7vlLbex9Fvey6K1YfcbY6uLx+cOg9a9nUz4tCR8 egFhEMFYiQLzdcMgITiCWwqDipc+juY74GRRCPFUmDITPsyGDW38auEDgbsO HxCbDx8wpozR/7vwYcMMcVvd0OnuXdfZ8VnFJ+5yS6jFJz61PdQ1GGnbP1re 1pw3mfMms639zDlTEJWi0GQza8p4mEfQOaQNs47JUxQ7zuF8CpaC2Qpq9sLF GVgmzKYJWCyGQrgwAQuvp05BIbLWSMDCMdnkUU+6ncwtjhKvPYlYNsjAYm0z AwvrlIHF7pSBhXfKwOLsMgNL8XbPjjKwoE4ZWHCnDCykUwYWujwDi7UkAwvr lIHFXiUDC992BhZnAz+L4QvIwGLeQDcHyOYAeT6+/5LEcfbOS7IoGCnH+O+f vwWjFBzL8F83aoVedL+/p4nydTImDlW9OBxqWAKInFhQyTH9DupaCE/yf6De tvoAcHX+AXz+dPaxWHRfqf8vzwH4+/nHawAUcgDw6W8fwSfw6v3VZ/Dq4uxC U12fX5y9BuD9TxcXf778AKprfy83glwTRYlytfmoIMEDwKk9UANEA8VQFakB 2j2kvrATiHtKh/nwQOPSg/wPUEsDBBQAAgAIAPV4GTsJpOScnQcAAJoWAAAK ABUAc2xhcGQuY29uZlVUCQADbiiUSnEolEpVeAQA7gMBAtVYXW8buxF91v4K Vn6xU1my5CS9NbBFHVu516jjBJFT3DwZ3CUlMeaSG5IrWy3633uGuyvJH7px UMNAhSDZJWfI+ThzZjY7yQ6bSMm85qXo59ZMd9/ssal1TMjAlfbMGkbLalY5 HhTebEn/+D40L+fKs6nS0J/bSgt28fGSZZLdWIcXJ7ngmZaQTJTJdSVkpzOQ IR/YUhoteDnw+VwWfJBbJ/v18w8lvTJPlIVgsG5WSueteZqKUf6eIGt/j8p/ LxCjn9IopPd8pszs6hGn/1i1LPh+UemgfvpSLoTDve+svW71kL1jre0NOz89 /rQYsVwraQKl2si8TjCr84s/lNYwl8DElMOAfsKjaqaMuFqM6KxTy4wNTBpK ODI/lc5xoKcyQWl2/P5y/JktbcXmfCEZJ4BcIwZMKIfbrFviCC/dQuWSHV+c Mm6gKZC4wI0gQTtdHwo8tc8d8vJoMHDWhn7rdR9JT5JSCUImEr3gbuAqsxGV CHYIJNzNfJTaJkQCSaLtTMuF1J3o67nlgoml4YXKWcbza2kEK6yotPRHSf1Q 8jDvDCrvBlpl8lauc9IIaBzSIeWrTGR9zR8sF9YoRObuVp1jvzR56eyC9nbY fcUYgi07WfH4Tsm9vxGP7/m51DpuJZQi6c5O2RAvVP6SGXkbgA0HDtEoOM9q aFReUs4uzyeRTKTJ3RK8gUxuIAxStEKXiqoolixIDwhKF9RU5TyASOYqn0fg 5IDETBrpaDlbsnzODZURCxbqEfXltRoE7Qd0gO8xZNOQQLfg1y3BlbLo9oAu Ada6pU2QQ6G8j9ZYQyhs5Zi38IuHCHyKKfnkGJyZOVuV0SBiOKYCKuWrrVxb Q95Oww13kmLJl0CIvmY4x0s93fdqZqTY9BGWzu0N0OUio55PTo5P1rvviVwf ejfI+X6GCgG55i40ej/WWjn3QOMfcvljJeoVvCiJ7mVeORWWiIAPTtXpTHY6 n+X3CiXNlAlyFgWA01Cnm+2WDm4iQHP1DcBC+Pc2VIbD0X6mAts9PB1PKMyZ DEG6vRV2cAJBqSoFRW1D8+1hVLwn51W0lDgqkktjsPfTdNgcchVfhqNG9opk 07evN/zkeQ7eJMwGZzUrrVb58ojuBuGw08n4qME7N0trJMDYYgIykyqryZbt +irbg+du+SOdj0CbIyFPt3RqiibksBuY3xq03uIVFMCxlEMREerr0x4IGmuW ha18I4RbN3UheBq5WC1Qw0ZKgeMgEwNTUNJWvrdBwa4w/Yx7mXa7VJKv4s2P C+QmXYVjm/ArGIGdtbf1+9qp+n3tCTkQiUhN0X3uJQvbwAYg52F9b7N9Na7Q 3RQa3yaCaIHq8E5WsBfmxBRZFVZoR1xbHEZBgAGeMrYr+7N+j3U3fFr72t2L ttbCkT64vuHLNl+4vE7x+J/jz18vfzu7+PVPQOLz/HAv2syQwWROCaFQKPSX umyf6Y728A6uSnw1BcWuxhLWtVV68vHicnxx2bOpsJhhTC9Pp66bNBFZiwIs XBTY366zw0605C5Q94nNyzoBJpW+lLlCWpeRAyjp9em9ZkKlQCDTC6sAcGSs nX3rBrj7y15MxNZpmCj6S93agARLzWtdRMQ8ICF0Aj7D4dGv8qbTAfc4GZrU 08K/I1H9R317//Uin1zk4d3XWdL003WG2tGIffgyuWTyVqE3lk6RW3bV3KKl cWQikquHcGvgftZyhaKBDCCkWNTSrYc4x9YufbCYIP9ycAAo5rZAuVNwkrUF zWhJM1I7xuyToQMCFVl+hiktr4uBkoRYmSYBmB5blzCrCnkb61mJXmlMVWTS 9cCNxS0WaKrFoIUU8wyDKteKe4WK/N6jIm6UbfYNNp1opAs7zWJk1pPJRS8+ fPmCAQV7yc4JB9+wBdcV9Yuc3rz6VzsrDw/wS1arnVF8F+au3OFB/CU7Gxud 182aEvqOdH3k5mrnsJaMC1OaktrLk9VSB2pJQkOdk6VmCE56cDBMGPXOBXDq 0mbKHR70h69H/eGb/uvR0eEvf4VIWJYydXSMnx8b8QlUCZxgA4hzy7T7hg3f kA/sz10sepRMPq8ZeXttQS7yNP6KXTq10ylyjBkmOJ92X/XiWdQsCxnmVqR1 +2zWhEmfUsCMwVtBlYOJPm1LJCmUc9YVhEeMYxZsrDFCtQMvjaDN4360rrQA Wx12NhwdrHe9jEMd8FTvIgXsM300bPI2lUTBPaaMZKdeRiTZtmDTASUaB/dU /neRvUOZs7P4KRzLBFP/IH5GrOqlvmi/FqwVcBZmPx9ifveHfXkbpw4QT0E3 gmLAcBjE0hyNAcKassDuhp57zeivQubz9NfJ5PjTWSNFzJSfiZRuGGyYsHnN 38e/H3/4dD7un3z88Ly9ZvQyvabtGdtaztnF6fj38eSnWs4WnWdoOS/ScNZe rRrPyzSWQd1P/qduMgJV3i/stwcs/Rs7xteAiS7z6pa+26b4kIMqEM1CFW1+ J9211BJD9rvHe1LTL25kVjeadcuhp1VLutdzQChU2GhRURbf/fLFulKz+vTe ROtPb0+rU/64Q61te9CnRi/Qp7YU5PP2qa2X/P/0qWHdp56Nxpv/gnpZJm8u Tf4LUEsBAhcDFAACAAgAF3kZO1M96vwRAQAAxQEAAAkADQAAAAAAAQAAAKSB AAAAAERCX0NPTkZJR1VUBQADriiUSlV4AABQSwECFwMUAAIACABRcxU7ILcI 7hsJAAAJYQAAMAANAAAAAAABAAAAwIFNAQAAaXRzX3JlcGxpY2F0aW9uX29w ZW5sZGFwX2dkYl90cmFjZXNfMjAwOTA4MjEudHh0VVQFAAPK2I5KVXgAAFBL AQIXAxQAAgAIAPV4GTsJpOScnQcAAJoWAAAKAA0AAAAAAAEAAACkgcsKAABz bGFwZC5jb25mVVQFAANuKJRKVXgAAFBLBQYAAAAAAwADAPQAAAClEgAAAAA= --0-698620411-1251380367=:64533--

1 0

Re: (ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by quanah＠zimbra.com 26 Aug '09

26 Aug '09

--On Tuesday, August 25, 2009 6:16 PM +0000 rlvcosta(a)yahoo.com wrote: > The issue I'm facing is related, in a general user view, is when I stop > the secondary Provider2(master 2) for backup purposes using slapcat. The > Provider1(master 1) continues to provide ldap service where some > entrances can be created during the time backup is running(no consumer > from Provider 2). Why are you stopping the provider to do a slapcat? > Even a small number of entrances are different when consumer in Provider 2 > connects to Provider 1 then syncrepl enters in the full DB search as > expected. What is your sessionlog setting on each provider for the syncprov overlay? > For definition purposes I have some memory limitations where I need to > limit dncachesize for around 80% of DB entrances. We already went through other things you could do to reduce your memory footprint in other ways. You've completely ignored that advice. As long as your dncachesize is in this state, I don't expect things to behave normally. > I could also note that when in this situation the monitor cache, in a > very slow pace, changes the cache in a single entrance. Being more > specific : > > dn: cn=Database 1,cn=Databases,cn=Monitor > structuralObjectClass: monitoredObject > creatorsName: > modifiersName: > createTimestamp: 20090821145848Z > modifyTimestamp: 20090821145848Z > monitoredInfo: bdb > monitorIsShadow: TRUE > namingContexts: ou=CONTENT,o=domain,c=fr > readOnly: FALSE > monitorOverlay: syncprov > olmBDBEntryCache: 19920 > olmBDBDNCache: 3896287 > olmBDBIDLCache: 2 > olmDbDirectory: /var/openldap-data/bdb1/ > entryDN: cn=Database 1,cn=Databases,cn=Monitor > subschemaSubentry: cn=Subschema > hasSubordinates: TRUE > > Stays running in the values 3896287 and 3896288. Looks like the memory > re-use is being too short causing locks that takes long time causing a non > synchronization. What value did you set for "cachefree"? > PS-> I could not put the file in the openldap ftp. It says device full. > Please let me know how can I send this file. I've let the maintainer of the system know, hopefully there'll be space available soon. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#6276) paused pool can deadlock if writers are waiting
by h.b.furuseth＠usit.uio.no 26 Aug '09

26 Aug '09

h.b.furuseth(a)usit.uio.no writes: > Then we could add the option (c) a *small* set of (groups of?) config > variables which each will need their own mutex or pause or something. > Such as loglevel, since the code is doing Debug() all over the place. I forgot: Simplest way here is to have two versions of the variable: the one accessed by cn=config, and a copy which is the one slapd obeys. The cn=config variable is pause-protected as usual, and slapd copies it to the active variable at its leisure. This will be acceptable for config changes that cannot fail once the value has been verified and need not take effect immediately. Then the copy task needs locks both for the configurable variable ("lock" = prevent_pause() ... "unlock" = allow_pause()?) and for the active variable, but not at the same time. -- Hallvard

1 0

Re: (ITS#6276) paused pool can deadlock if writers are waiting
by hyc＠symas.com 26 Aug '09

26 Aug '09

Hallvard B Furuseth wrote: > But I quite agree > that this is hopeless if that "small" set cannot be kept small. For one > thing it might introduce yet another "lock order", threatening to be > inconsistent with the lock order of other mutexes. >> Another possibility is to just try to read 1 byte in the listener >> thread, to detect the hangup there when we have no other means to >> discover it. We would have to make sure to be able to unget this byte >> back to the bottom of the sockbuf stack if there's valid data. This >> will affect the throughput of the listener thread, but it may not be >> too terrible a hit. > > Does that help for a socket which has gotten blocked on select() due to > full socket buffers? A socket that's blocked on a write will eventually clear itself up - either the client will catch up or it will go away. So there's really no reason to worry about them at all. > Also if doing - a read() OS call anyway, why not read() a large enough > chunk that it would commonly be a full PDU? As long as it's read to a > sockbuf buffer rather than into slapd's data. Because we don't want to keep the listener thread busy for any longer than necessary. In terms of system call overhead it might be reasonable to read as much as one memory page (e.g. 4KB) but even that involves a lot of cycles for memcpy, and the idea is to shunt all the time-consuming work elsewhere so that the listener can return to listening ASAP. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6276) paused pool can deadlock if writers are waiting
by h.b.furuseth＠usit.uio.no 26 Aug '09

26 Aug '09

Howard Chu writes: >Hallvard B Furuseth wrote: >> Beyond that, my immediate reaction was that pauses are implemented at >> the wrong level and/or need to be split up in different types of pauses. >> There is no good reason cn=config's need to have slapd config variables >> for itself to affect network operations - nor, I hope, affect pool-level >> operations like pool_purgekey(). > > I suppose implementing things this way can appear to be too blunt. But the > amount of locks required to do it at a finer level is, IMO, unmanageable. I know. That's why I'm not suggesting to do away with pauses, only to make them a little less crude than "1 type of pause for everything". See below. > configurations are changed from under them.) > >> E.g. cn=config could use a slapd_config_pause() call to ask for lone >> access to the config variables instead of thread_pool_pause() call. >> Slapd operations that must respect such pauses should thus call a >> slapd_config_pausecheck() macro/function, not thread_pool_pausecheck() >> or depend on the pool to respect slapd-level pauses. >> connection_read_thread() can then go ahead and use the connection >> independently of those pauses. If it reaches connection_operation(), >> it'll need to check for slapd pause. > > Yes, I've thought about this approach too. That makes the pause mechanisms > even messier. It does? I was hoping that would be the cleaner solution. Lifting them from pool level to slapd level would mean that every task now submitted to the pool will be responsible for (a) calling pausecheck itself or (b) avoiding all variables/features that today need pauses. Then we could add the option (c) a *small* set of (groups of?) config variables which each will need their own mutex or pause or something. Such as loglevel, since the code is doing Debug() all over the place. And presumably some network variables, for this ITS. But I quite agree that this is hopeless if that "small" set cannot be kept small. For one thing it might introduce yet another "lock order", threatening to be inconsistent with the lock order of other mutexes. > Another possibility is to just try to read 1 byte in the listener > thread, to detect the hangup there when we have no other means to > discover it. We would have to make sure to be able to unget this byte > back to the bottom of the sockbuf stack if there's valid data. This > will affect the throughput of the listener thread, but it may not be > too terrible a hit. Does that help for a socket which has gotten blocked on select() due to full socket buffers? Also if doing - a read() OS call anyway, why not read() a large enough chunk that it would commonly be a full PDU? As long as it's read to a sockbuf buffer rather than into slapd's data. -- Hallvard

1 0

Re: (ITS#6257) libldap: getopt flag to return the SASL username
by hyc＠symas.com 26 Aug '09

26 Aug '09

masarati(a)aero.polimi.it wrote: > My concern was not from an operational point of view: the simple concept > of having a library dynamically loading something that could no longer be > present is calling for trouble, unless handled appropriately, and probably > there is no way to do it safely as one could always remove a .so while > it's in use (although I guess on any decent system the object will be > cached or loaded somewhere until it's in use). > > My concern is about the char* array returned by that call: if for any > reason the library decides to refresh it, but the caller of > ldap_get_option() is still holding a pointer to that array, this calls for > pointing to freed memory and things like that, as far as I understand. > For this reason, returning a copy sounds wiser. Whether the contents of > that copy is valid or not, namely the related mechanism is available or > not, that's an entirely different issue. The Cyrus mechlist is only initialized once during the life of the library. (Subsequent init calls just increment an init counter and then return.) It will not change in normal use. If an application loads / unloads / reloads the library, the list may change, but any app that goes thru this trouble will already know they have to call the init functions all over again, and re-fetch this list. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6257) libldap: getopt flag to return the SASL username
by masarati＠aero.polimi.it 26 Aug '09

26 Aug '09

>> On a somewhat related issue, I note that LDAP_OPT_X_SASL_MECHLIST >> returns >> a pointer to an array of chars that apparently cannot be mucked with. >> >> Assuming my understanding is correct, I wonder if this behavior is >> desirable or not, given the fact that if another mech is added, e.g. by >> adding a dynamic module, I expect this list to change. > > These are SASL mechs with the plugin modules. Right? > >>From an operational standpoint: If a SASL plugin module for a mech was >> added I > think it's acceptable that a software which queries this option is > restarted > before this SASL mech is known to the software. Probably one has to add > additional configuration for this SASL mech. > > Now the question is what happens if a SASL plugin module is removed and > the > software trys to use the removed SASL mech. Clearly removing plugin > modules in > a running system is asking for trouble anyway... > > Having said this I would not care too much about this list going to > change... My concern was not from an operational point of view: the simple concept of having a library dynamically loading something that could no longer be present is calling for trouble, unless handled appropriately, and probably there is no way to do it safely as one could always remove a .so while it's in use (although I guess on any decent system the object will be cached or loaded somewhere until it's in use). My concern is about the char* array returned by that call: if for any reason the library decides to refresh it, but the caller of ldap_get_option() is still holding a pointer to that array, this calls for pointing to freed memory and things like that, as far as I understand. For this reason, returning a copy sounds wiser. Whether the contents of that copy is valid or not, namely the related mechanism is available or not, that's an entirely different issue. p.

1 0

Re: (ITS#6276) paused pool can deadlock if writers are waiting
by hyc＠symas.com 26 Aug '09

26 Aug '09

Hallvard B Furuseth wrote: > This is a quick abstract answer, I haven't dived into the code to see > what my suggestions would mean in practice. Anyway: > It sounds to me like this problem cannot be fully prevented when network > operations share the same pool queue as everything else, and we have > operations that can wait for other operations. All pool threads can be > occupied with LDAP-level operations, so that network-level operations > which the LDAP-level operations depend on can get blocked. If slapd has > a design which even tries to guarantee forward progress, I'm not aware > of it. > > So LDAP-level operations ought to leave at least one thread free for for > network-level operations. Not necessary a designated thread: A thread > moving from network-level to LDAP-level operation like > connection_read_thread() does could first check that at least one other > thread remains available for network-level operation. Yes... > Beyond that, my immediate reaction was that pauses are implemented at > the wrong level and/or need to be split up in different types of pauses. > There is no good reason cn=config's need to have slapd config variables > for itself to affect network operations - nor, I hope, affect pool-level > operations like pool_purgekey(). I suppose implementing things this way can appear to be too blunt. But the amount of locks required to do it at a finer level is, IMO, unmanageable. The fact that you can change global slapd config parameters (such as the number of threads, size of sockbuf buffers, etc.) makes it inherently safe for *anything* else to be active while config changes are being made. And sifting thru each variable to decide how sensitive they are, and when they are unsafe, will inevitably lead to requiring locks on every single piece of configuration data. (Want to look up an attribute type, or objectclass? Or a database suffix? etc. etc. etc... There are countless things we do arbitrarily that simply won't work if we allow arbitrary threads to continue to run while their configurations are changed from under them.) > E.g. cn=config could use a slapd_config_pause() call to ask for lone > access to the config variables instead of thread_pool_pause() call. > Slapd operations that must respect such pauses should thus call a > slapd_config_pausecheck() macro/function, not thread_pool_pausecheck() > or depend on the pool to respect slapd-level pauses. > connection_read_thread() can then go ahead and use the connection > independently of those pauses. If it reaches connection_operation(), > it'll need to check for slapd pause. Yes, I've thought about this approach too. That makes the pause mechanisms even messier. Another possibility is to just try to read 1 byte in the listener thread, to detect the hangup there when we have no other means to discover it. We would have to make sure to be able to unget this byte back to the bottom of the sockbuf stack if there's valid data. This will affect the throughput of the listener thread, but it may not be too terrible a hit. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6276) paused pool can deadlock if writers are waiting
by h.b.furuseth＠usit.uio.no 26 Aug '09

26 Aug '09

This is a quick abstract answer, I haven't dived into the code to see what my suggestions would mean in practice. Anyway: It sounds to me like this problem cannot be fully prevented when network operations share the same pool queue as everything else, and we have operations that can wait for other operations. All pool threads can be occupied with LDAP-level operations, so that network-level operations which the LDAP-level operations depend on can get blocked. If slapd has a design which even tries to guarantee forward progress, I'm not aware of it. So LDAP-level operations ought to leave at least one thread free for for network-level operations. Not necessary a designated thread: A thread moving from network-level to LDAP-level operation like connection_read_thread() does could first check that at least one other thread remains available for network-level operation. Beyond that, my immediate reaction was that pauses are implemented at the wrong level and/or need to be split up in different types of pauses. There is no good reason cn=config's need to have slapd config variables for itself to affect network operations - nor, I hope, affect pool-level operations like pool_purgekey(). E.g. cn=config could use a slapd_config_pause() call to ask for lone access to the config variables instead of thread_pool_pause() call. Slapd operations that must respect such pauses should thus call a slapd_config_pausecheck() macro/function, not thread_pool_pausecheck() or depend on the pool to respect slapd-level pauses. connection_read_thread() can then go ahead and use the connection independently of those pauses. If it reaches connection_operation(), it'll need to check for slapd pause. Alternatively, with different types of pauses/operations: The pool could have different sets of pauses (a pause-type bitmask?) and if one type of pauses happens, operations that ignore that type of pause will not be affected. This messes up the pool code further though, and I already hate what pauses do to the otherwise clean pool code:-( -- Hallvard

1 0

(ITS#6276) paused pool can deadlock if writers are waiting
by hyc＠OpenLDAP.org 26 Aug '09

26 Aug '09

Full_Name: Howard Chu Version: RE24/HEAD OS: Solaris 10 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (76.91.220.157) Submitted by: hyc test050 hung on me after some number of iterations. Unfortunately I didn't save the stack traces, but basically there was one thread waiting in send_ldap_ber() on the write2 cv, and another thread in config_back_add() waiting for a pool pause to succeed. netstat showed that no connections had queued data, so there should have been no reason for the writer to still be waiting. I believe what happened here is that while the writer was waiting (it was a syncprov qtask replaying events for a psearch) the psearch connection got closed. Solaris is using select, and select() doesn't specially distinguish socket close events - they're reported as read events. The deadlock is because we queue read events into the thread pool, and we don't discover they're actually closed sockets until the read thread gets to run and tries to read from the socket (and gets zero bytes back). But since the pool is entering a pause, the reader thread cannot run, so it can't detect the hangup and dispose of the waiting writer. The ideal fix for this is to process hangup events inline in the listener thread instead of pushing them into the thread pool. But that requires being able to cheaply determine that a hangup actually occurred, and select() doesn't give us this information. We could get this info using poll() instead. Since nowadays any POSIX platform that implements select() also implements poll() we can probably just switch to poll() and drop select(). One exception is Windows; Winsock only supports poll() on Windows Vista and newer. (Note, we had a patch that added a connection_hangup() handler for Linux epoll() at one point, but I dropped it later because it seemed to have strange interactions with Samba. Should look into resurrecting it again.) I don't think we can really fix this issue without knowing for certain when hangup events occur. If we're forced to keep using select, that implies that the main listener thread must attempt a read on the socket before deciding how to dispatch the connection. Any thoughts?

1 0

Re: (ITS#6274) nssov: makefile suffix rules too greedy
by hyc＠symas.com 25 Aug '09

25 Aug '09

jonathan(a)phillipoux.net wrote: > Full_Name: Jonathan Clarke > Version: RE24 > OS: Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (82.67.204.30) > > > When running "make" in the nssov module from contrib/slapd-modules, I get this: > 8<------------------- > [...]/contrib/slapd-modules/nssov$ make > ../../../libtool --mode=compile gcc -g -O2 -I../../../include > -I../../../include -I../../../servers/slapd -Inss-ldapd -c alias.c nssov.h > libtool: compile: cannot determine name of library object from `nssov.h' > 8<------------------- > > This trivial patch to Makefile corrects this: > 8<------------------- > - $(LIBTOOL) --mode=compile $(CC) $(OPT) $(DEFS) $(INCS) -c $? > + $(LIBTOOL) --mode=compile $(CC) $(OPT) $(DEFS) $(INCS) -c $< > 8<------------------- > > Thanks, fixed in HEAD. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by rlvcosta＠yahoo.com 25 Aug '09

25 Aug '09

Full_Name: Rodrigo Luiz Vargas Costa Version: 2.4.17 OS: CentOS release 5.2 (Final) URL: ftp://ftp.openldap.org/incoming/<TBD> Submission from: (NULL) (135.245.8.5) Openldap developers, I have being exchange some information at openldap lists where looks like some improvements are being done in replication for release 2.4.18. The architecture I'm running has 2 machines in MirrorMode in the same subnet(at the same switch). These systems are part of a HA system sharing a VIP and where both machines have slapd running simultaneously(bind to any local interface) and only VIP is exchanged for HA purposes. The issue I'm facing is related, in a general user view, is when I stop the secondary Provider2(master 2) for backup purposes using slapcat. The Provider1(master 1) continues to provide ldap service where some entrances can be created during the time backup is running(no consumer from Provider 2). Even a small number of entrances are different when consumer in Provider 2 connects to Provider 1 then syncrepl enters in the full DB search as expected. For definition purposes I have some memory limitations where I need to limit dncachesize for around 80% of DB entrances. >From a user perspective I see that after cache is filled system enters in some state where synchronization doesn't happen anymore. For full reference(config, gdb, etc), please see file attached in FTP. Then I see 2 issues : 1)Consumer from Provider2, even passed days and only a small number of differences for test purpose happen(no traffic), the syncrepl never ends and there isn't replication(Provider 1 stay continuously consuming 100% CPU); 2)Even I stop the Provider2(then its consumer) I do not see any change in Provider 1 activities. The CPU continues in 100% even passed days what suggest some hang in the thread or logic. I compiled openldap with GDB symbols and then execute some traces in the threads during the state 2 report above. Looks like it stay looping forever locked in some thread lock. I could also note that when in this situation the monitor cache, in a very slow pace, changes the cache in a single entrance. Being more specific : dn: cn=Database 1,cn=Databases,cn=Monitor structuralObjectClass: monitoredObject creatorsName: modifiersName: createTimestamp: 20090821145848Z modifyTimestamp: 20090821145848Z monitoredInfo: bdb monitorIsShadow: TRUE namingContexts: ou=CONTENT,o=domain,c=fr readOnly: FALSE monitorOverlay: syncprov olmBDBEntryCache: 19920 olmBDBDNCache: 3896287 olmBDBIDLCache: 2 olmDbDirectory: /var/openldap-data/bdb1/ entryDN: cn=Database 1,cn=Databases,cn=Monitor subschemaSubentry: cn=Subschema hasSubordinates: TRUE Stays running in the values 3896287 and 3896288. Looks like the memory re-use is being too short causing locks that takes long time causing a non synchronization. I made several GDB traces for different conditions. Please see ftp attachment file for details. Thanks, Rodrigo. PS-> I could not put the file in the openldap ftp. It says device full. Please let me know how can I send this file.

1 0

Re: (ITS#6257) libldap: getopt flag to return the SASL username
by michael＠stroeder.com 25 Aug '09

25 Aug '09

masarati(a)aero.polimi.it wrote: >> masarati(a)aero.polimi.it wrote: > >> I'd appreciate it very much if it would be exactly behave in the same way >> like >> all other string-valued options. > > On a somewhat related issue, I note that LDAP_OPT_X_SASL_MECHLIST returns > a pointer to an array of chars that apparently cannot be mucked with. > > Assuming my understanding is correct, I wonder if this behavior is > desirable or not, given the fact that if another mech is added, e.g. by > adding a dynamic module, I expect this list to change. These are SASL mechs with the plugin modules. Right? >From an operational standpoint: If a SASL plugin module for a mech was added I think it's acceptable that a software which queries this option is restarted before this SASL mech is known to the software. Probably one has to add additional configuration for this SASL mech. Now the question is what happens if a SASL plugin module is removed and the software trys to use the removed SASL mech. Clearly removing plugin modules in a running system is asking for trouble anyway... Having said this I would not care too much about this list going to change... Ciao, Michael.

1 0

(ITS#6274) nssov: makefile suffix rules too greedy
by jonathan＠phillipoux.net 25 Aug '09

25 Aug '09

Full_Name: Jonathan Clarke Version: RE24 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (82.67.204.30) When running "make" in the nssov module from contrib/slapd-modules, I get this: 8<------------------- [...]/contrib/slapd-modules/nssov$ make ../../../libtool --mode=compile gcc -g -O2 -I../../../include -I../../../include -I../../../servers/slapd -Inss-ldapd -c alias.c nssov.h libtool: compile: cannot determine name of library object from `nssov.h' 8<------------------- This trivial patch to Makefile corrects this: 8<------------------- - $(LIBTOOL) --mode=compile $(CC) $(OPT) $(DEFS) $(INCS) -c $? + $(LIBTOOL) --mode=compile $(CC) $(OPT) $(DEFS) $(INCS) -c $< 8<-------------------

1 0

Re: (ITS#6200) slapd crashes under load w/ syncrepl
by richton＠rci.rutgers.edu 24 Aug '09

24 Aug '09

What does this look with top and/or in dmesg over the run time? Is it a simple out-of-memory? Definitely a bit of a gross method, but what's ls -lh core show for size? (If so, is it warranted given your load or is there a leak, etc etc...and of course make sure you're up to date on the surrounding packages, OpenLDAP isn't the only thing that can leak.)

1 0

Re: (ITS#6270) Conflict between ppolicy (pwdReset flag) and unique overlays
by michael＠stroeder.com 24 Aug '09

24 Aug '09

michael(a)stroeder.com wrote: > clem.oudot(a)gmail.com wrote: >> I have a rootdn. An extract of my slapd.conf is : >> >> --- >> database bdb >> suffix dc=3Dexample,dc=3Dcom >> rootdn cn=3Dmanager,dc=3Dexample,dc=3Dcom >> rootpw secret >> directory /var/lib/ldap >> >> overlay ppolicy >> ppolicy_use_lockout >> ppolicy_hash_cleartext >> >> overlay unique >> unique_uri ldap:///ou=3Dusers,dc=3Dexample,dc=3Dcom?uid?sub?(objectClass=3D= >> inetOrgPerson) > > Could you please repost this without the broken quoted printables? Especially > the filter part of the value for 'unique_uri'. I suspect the ITS software messed this up because the direct Cc:-ed messages to me did not contain the messed up quoted printables. So here's what Clément orginally sent as excerpt of his config: --- database bdb suffix dc=example,dc=com rootdn cn=manager,dc=example,dc=com rootpw secret directory /var/lib/ldap overlay ppolicy ppolicy_use_lockout ppolicy_hash_cleartext overlay unique unique_uri ldap:///ou=users,dc=example,dc=com?uid?sub?(objectClass=inetOrgPerson) --- Ciao, Michael.

1 0

Re: (ITS#6270) Conflict between ppolicy (pwdReset flag) and unique overlays
by michael＠stroeder.com 24 Aug '09

24 Aug '09

clem.oudot(a)gmail.com wrote: > I have a rootdn. An extract of my slapd.conf is : > > --- > database bdb > suffix dc=3Dexample,dc=3Dcom > rootdn cn=3Dmanager,dc=3Dexample,dc=3Dcom > rootpw secret > directory /var/lib/ldap > > overlay ppolicy > ppolicy_use_lockout > ppolicy_hash_cleartext > > overlay unique > unique_uri ldap:///ou=3Dusers,dc=3Dexample,dc=3Dcom?uid?sub?(objectClass=3D= > inetOrgPerson) Could you please repost this without the broken quoted printables? Especially the filter part of the value for 'unique_uri'. Ciao, Michael.

1 0

(ITS#6273) NSS overlay (nssov) fails to load
by battery＠writeme.com 24 Aug '09

24 Aug '09

Full_Name: Matt Kassawara Version: 2.4.17 OS: Ubuntu 9.10 (Karmic) URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (8.7.94.151) Loading the nssov module using 'ldapadd' reports an olcModuleLoad handler error... # ldapadd -H ldapi:/// -Y external SASL/EXTERNAL authentication started SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth SASL SSF: 0 dn: cn=module{2},cn=config objectclass: olcmodulelist cn: module{2} olcmoduleload: {0}nssov olcmodulepath: /usr/lib/ldap adding new entry "cn=module{2},cn=config" ldap_add: Other (e.g., implementation specific) error (80) additional info: <olcModuleLoad> handler exited with 1 Output from 'slapd' in debug level 7 (incorrectly) reports file not found... oc_check_required entry (cn=module{2},cn=config), objectClass "olcModuleList" oc_check_allowed type "objectClass" oc_check_allowed type "cn" oc_check_allowed type "olcModuleLoad" oc_check_allowed type "olcModulePath" oc_check_allowed type "structuralObjectClass" lt_dlopenext failed: (nssov) file not found Output from 'LD_DEBUG' reveals undefined symbol 'ber_bvmatch' in nssov.so.0... 18016: /usr/lib/ldap/nssov.so.0: error: symbol lookup error: undefined symbol: ber_bvmatch (fatal) 18016: 18016: file=/usr/lib/ldap/nssov.so.0 [0]; destroying link map

1 0

(ITS#6272) test045 freed memory access
by richton＠nbcs.rutgers.edu 24 Aug '09

24 Aug '09

Full_Name: Aaron Richton Version: RE24 OS: Solaris 9 URL: https://www.nbcs.rutgers.edu/~richton/richton-bt-200908221228.txt Submission from: (NULL) (128.6.31.135) t@5 (l@5) terminated by signal SEGV (no mapping at the fault address) Current function is connection_abandon 729 op.orn_msgid = o->o_msgid; (dbx) where current thread: t@5 =>[1] connection_abandon(c = 0x106df088), line 729 in "connection.c" [2] connection_closing(c = 0x106df088, why = 0x2859e0 "connection lost"), line 777 in "connection.c" [3] connection_read(s = 11, cri = 0xfd3ffd64), line 1427 in "connection.c" [4] connection_read_thread(ctx = 0xfd3ffe0c, argv = 0xb), line 1245 in "connection.c" [5] ldap_int_thread_pool_wrapper(xpool = 0x10491f08), line 685 in "tpool.c" (dbx) print o->o_hdr o->o_hdr = 0xdeadbeef Full backtrace in ITS link. testrun directory: https://www.nbcs.rutgers.edu/~richton/richton-testrun-200908221228.tar.bz2

1 0

Re: (ITS#6271) syncprov use of freed mutex
by masarati＠aero.polimi.it 24 Aug '09

24 Aug '09

richton(a)nbcs.rutgers.edu wrote: > Full_Name: Aaron Richton > Version: RE24 > OS: Solaris 9 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (128.6.31.135) > > > Sol9+libumem+RE24 (current). I have five core files all with an aborting thread > showing: > > current thread: t@8 > [1] __lwp_kill(0x0, 0x6, 0x0, 0xfffffff8, 0x0, 0xfb3feec1), at 0xff31feb0 > [2] Abort(0xfb3fef18, 0xfb3fef18, 0x3a, 0x39, 0x81010100, 0xff00), at > 0xff056f9c > [3] panic(0xff065e34, 0x0, 0x0, 0x1, 0x0, 0xfb3ff139), at 0xff057094 > [4] _ceil_prio_inherit(0xef, 0x0, 0x10807258, 0xff078000, 0xff0613bc, 0x0), at > 0xff05ff10 > [5] mutex_lock_internal(0xad, 0x0, 0xff040e00, 0x23, 0x7, 0x10fed159), at > 0xff0614bc > =>[6] ldap_pvt_thread_mutex_lock(mutex = 0x10807258), line 296 in "thr_posix.c" > [7] syncprov_op_mod(op = 0xfb3ff8e8, rs = 0xfb3ff5a0), line 1965 in > "syncprov.c" > [8] overlay_op_walk(op = 0xfb3ff8e8, rs = 0xfb3ff5a0, which = op_modify, oi = > 0x1083b088, on = 0x1083ad88), line 659 in "backover.c" > [9] over_op_func(op = 0xfb3ff8e8, rs = 0xfb3ff5a0, which = op_modify), line > 721 in "backover.c" > [10] over_op_modify(op = 0xfb3ff8e8, rs = 0xfb3ff5a0), line 755 in > "backover.c" > [11] syncrepl_updateCookie(si = 0x10417a88, op = 0xfb3ff8e8, pdn = 0x10845b68, > syncCookie = 0xfb3ff710), line 3045 in "syncrepl.c" > [12] do_syncrep2(op = 0xfb3ff8e8, si = 0x10417a88), line 1177 in "syncrepl.c" > [13] do_syncrepl(ctx = 0xfb3ffe0c, arg = 0x10807d08), line 1358 in > "syncrepl.c" > [14] connection_read_thread(ctx = 0xfb3ffe0c, argv = 0x1c), line 1254 in > "connection.c" > [15] ldap_int_thread_pool_wrapper(xpool = 0x10491f08), line 685 in "tpool.c" > > In Frame 7, mt->mt_mods = 0xdeadbeef. Looks like libumem caught a post-free() > access. Looks like someone else got to the end of the modifications list and destroyed everything... p.

1 0

(ITS#6271) syncprov use of freed mutex
by richton＠nbcs.rutgers.edu 24 Aug '09

24 Aug '09

Full_Name: Aaron Richton Version: RE24 OS: Solaris 9 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (128.6.31.135) Sol9+libumem+RE24 (current). I have five core files all with an aborting thread showing: current thread: t@8 [1] __lwp_kill(0x0, 0x6, 0x0, 0xfffffff8, 0x0, 0xfb3feec1), at 0xff31feb0 [2] Abort(0xfb3fef18, 0xfb3fef18, 0x3a, 0x39, 0x81010100, 0xff00), at 0xff056f9c [3] panic(0xff065e34, 0x0, 0x0, 0x1, 0x0, 0xfb3ff139), at 0xff057094 [4] _ceil_prio_inherit(0xef, 0x0, 0x10807258, 0xff078000, 0xff0613bc, 0x0), at 0xff05ff10 [5] mutex_lock_internal(0xad, 0x0, 0xff040e00, 0x23, 0x7, 0x10fed159), at 0xff0614bc =>[6] ldap_pvt_thread_mutex_lock(mutex = 0x10807258), line 296 in "thr_posix.c" [7] syncprov_op_mod(op = 0xfb3ff8e8, rs = 0xfb3ff5a0), line 1965 in "syncprov.c" [8] overlay_op_walk(op = 0xfb3ff8e8, rs = 0xfb3ff5a0, which = op_modify, oi = 0x1083b088, on = 0x1083ad88), line 659 in "backover.c" [9] over_op_func(op = 0xfb3ff8e8, rs = 0xfb3ff5a0, which = op_modify), line 721 in "backover.c" [10] over_op_modify(op = 0xfb3ff8e8, rs = 0xfb3ff5a0), line 755 in "backover.c" [11] syncrepl_updateCookie(si = 0x10417a88, op = 0xfb3ff8e8, pdn = 0x10845b68, syncCookie = 0xfb3ff710), line 3045 in "syncrepl.c" [12] do_syncrep2(op = 0xfb3ff8e8, si = 0x10417a88), line 1177 in "syncrepl.c" [13] do_syncrepl(ctx = 0xfb3ffe0c, arg = 0x10807d08), line 1358 in "syncrepl.c" [14] connection_read_thread(ctx = 0xfb3ffe0c, argv = 0x1c), line 1254 in "connection.c" [15] ldap_int_thread_pool_wrapper(xpool = 0x10491f08), line 685 in "tpool.c" In Frame 7, mt->mt_mods = 0xdeadbeef. Looks like libumem caught a post-free() access.

1 0

Re: (ITS#6270) Conflict between ppolicy (pwdReset flag) and unique overlays
by clem.oudot＠gmail.com 24 Aug '09

24 Aug '09

2009/8/24 <michael(a)stroeder.com>: > clem.oudot(a)gmail.com wrote: >> Full_Name: Clement OUDOT >> Version: 2.4.16 >> OS: RHEL 5.2 >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (83.145.72.122) >> >> I use both ppolicy and unique overlays. >> >> I try to modify the password of an account whose pwdReset attribute is s= et to >> TRUE. I get this LDAP error: >> >> ldap_modify: Insufficient access (50) >> =A0 =A0 =A0 =A0 additional info: unique_search failed > > Without seeing your configuration one cannot determine wheter you're hitt= ing > ITS#6108: "unique overlay and rootdn". > > http://www.openldap.org/its/index.cgi/Documentation?id=3D6108;selectid=3D= 6108 > > Ciao, Michael. Hello, I have a rootdn. An extract of my slapd.conf is : --- database bdb suffix dc=3Dexample,dc=3Dcom rootdn cn=3Dmanager,dc=3Dexample,dc=3Dcom rootpw secret directory /var/lib/ldap overlay ppolicy ppolicy_use_lockout ppolicy_hash_cleartext overlay unique unique_uri ldap:///ou=3Dusers,dc=3Dexample,dc=3Dcom?uid?sub?(objectClass=3D= inetOrgPerson) --- Cl=E9ment.

1 0

Re: (ITS#6270) Conflict between ppolicy (pwdReset flag) and unique overlays
by michael＠stroeder.com 24 Aug '09

24 Aug '09

clem.oudot(a)gmail.com wrote: > Full_Name: Clement OUDOT > Version: 2.4.16 > OS: RHEL 5.2 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (83.145.72.122) > > I use both ppolicy and unique overlays. > > I try to modify the password of an account whose pwdReset attribute is set to > TRUE. I get this LDAP error: > > ldap_modify: Insufficient access (50) > additional info: unique_search failed Without seeing your configuration one cannot determine wheter you're hitting ITS#6108: "unique overlay and rootdn". http://www.openldap.org/its/index.cgi/Documentation?id=6108;selectid=6108 Ciao, Michael.

1 0

(ITS#6270) Conflict between ppolicy (pwdReset flag) and unique overlays
by clem.oudot＠gmail.com 24 Aug '09

24 Aug '09

Full_Name: Clement OUDOT Version: 2.4.16 OS: RHEL 5.2 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (83.145.72.122) Hello, I use both ppolicy and unique overlays. I try to modify the password of an account whose pwdReset attribute is set to TRUE. I get this LDAP error: ldap_modify: Insufficient access (50) additional info: unique_search failed In OpenLDAP logs, we can see: connection restricted to password changing only send_ldap_result: conn=20 op=2 p=3 send_ldap_result: err=50 matched="" text="Operations are restricted to bind/unbind/abandon/StartTLS/modify password" send_ldap_result: conn=20 op=2 p=3 send_ldap_result: err=50 matched="" text="unique_search failed" send_ldap_response: msgid=3 tag=103 err=50 So it seems the unique overlay cannot do a search because the connection is restricted by the ppolicy overlay.

1 0

Re: (ITS#6266) uninitialized var in overlays/dynlist.c:dl_cfgen()
by masarati＠aero.polimi.it 24 Aug '09

24 Aug '09

> hyc(a)symas.com writes: >> In the actual code, dlm is initialized to NULL. Therefore, dlml will get >> initialized to NULL > > Only initialized when dml is NULL, but the test > 'if ( dlml != NULL )' below is unconditional. Then dlml goes > out of scope, it doesn't keep the value between iterations. Should be fixed now, thanks. Please test. p.

1 0

Re: (ITS#6267) test030-relay fails with back-meta: line 79: kill: (23804) - No such process
by masarati＠aero.polimi.it 24 Aug '09

24 Aug '09

> masarati(a)aero.polimi.it wrote: >> Can't reproduce. > > Also happens with new checkout. See testrun/ available here: > > http://www.stroeder.com/temp/openldap-its6267-testrun-test030.tar.gz OK, that's about building as modules (back-meta needs back-ldap, should have tested straight ahead). Fixing... p.

1 0

Re: (ITS#6267) test030-relay fails with back-meta: line 79: kill: (23804) - No such process
by michael＠stroeder.com 22 Aug '09

22 Aug '09

masarati(a)aero.polimi.it wrote: > Can't reproduce. Also happens with new checkout. See testrun/ available here: http://www.stroeder.com/temp/openldap-its6267-testrun-test030.tar.gz > Did you start from a fresh checkout? Is your working directory in sync with what's available through anon CVS? Ciao, Michael.

1 0

(ITS#6269) Fails to include inter-library dependencies on QNX
by mkraai＠beckman.com 21 Aug '09

21 Aug '09

Full_Name: Matt Kraai Version: 2.4.17 OS: QNX 6.4.1 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (134.217.96.252) When built on QNX 6.4.1, libldap.so doesn't include inter-library dependency information. When an application tries to link against it, it fails because the symbols provided by the libraries that libldap.so should depend on aren't defined. Upgrading to a more recent version of libtool (e.g., 2.2.6) fixes this problem, since it knows how to handle inter-library dependencies on QNX. This can be done by installing libtool 2.2.6 and running the following commands in openldap's top-level source directory: aclocal libtoolize autoconf

1 0

Re: (ITS#6268) multi-master sync replication ldap_add error code 68 bug
by hyc＠symas.com 20 Aug '09

20 Aug '09

bcolston(a)xtec.com wrote: > Full_Name: Barry Colston > Version: 2.4.17 > OS: Fedora 10 > URL: > Submission from: (NULL) (209.255.208.219) > > > While testing sync replication, I encountered a situation in which a previously > deleted DN cannot be added again because the ldapadd command receives a 68 error > code. If I perform an ldapsearch command for the DN, the DN is not found. If I > try to add the DN, the add fails with a 68 error code. If I perform a slapcat > command for that DN, slapcat displays the record. This sounds like it's related to ITS#6097. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6257) libldap: getopt flag to return the SASL username
by masarati＠aero.polimi.it 20 Aug '09

20 Aug '09

> masarati(a)aero.polimi.it wrote: > I'd appreciate it very much if it would be exactly behave in the same way > like > all other string-valued options. On a somewhat related issue, I note that LDAP_OPT_X_SASL_MECHLIST returns a pointer to an array of chars that apparently cannot be mucked with. Assuming my understanding is correct, I wonder if this behavior is desirable or not, given the fact that if another mech is added, e.g. by adding a dynamic module, I expect this list to change. p.

1 0

(ITS#6268) multi-master sync replication ldap_add error code 68 bug
by bcolston＠xtec.com 20 Aug '09

20 Aug '09

Full_Name: Barry Colston Version: 2.4.17 OS: Fedora 10 URL: Submission from: (NULL) (209.255.208.219) While testing sync replication, I encountered a situation in which a previously deleted DN cannot be added again because the ldapadd command receives a 68 error code. If I perform an ldapsearch command for the DN, the DN is not found. If I try to add the DN, the add fails with a 68 error code. If I perform a slapcat command for that DN, slapcat displays the record. I have 3 multi-master servers in my configuration; all 3 are executing on the same physical server, listening on different ports and with separate copies of BDB databases. Each server is replicating to the other 2 servers (e.g., server 1 replicates to server 2 and server 3, server 2 replicates to server 1 and server 3, and server 3 replicates to server 1 and server 2) using the refreshAndPersist mode. I execute 3 shell scripts simultaneously, each of which adds a set of parent/child records using the ldapadd command and ldif files, then deletes the records using the ldapdelete command (each shell scripts adds and deletes about 267 records and each shell script operates on a separate set of DNs). The 3 shell scripts issue the ldapadd and ldapdelete commands against server 1 and repeat the add/delete cycle 10 times before exiting. After all 3 shell scripts finish, I compare the server 1 records against the server 2 records and compare the server 1 records against the server 3 records listing any differences. The method I normally execute the shell scripts results in 800 records being added then deleted 10 times, for a total of 8000 adds/deletes occurring. After executing the above 3 shell scripts multiple times (without bringing down slapd between executes), some records will fail to be added by the ldapadd command because the ldapadd command returns an error code of 68. After this occurs, I execute an ldapsearch command for the DN of the record that received the 68 error; the ldapsearch command fails to find the record (which is correct because the record was deleted). If I perform a slapcat command for the DN against server 1's BDB files, slapcat finds the record, but it is listed with an objectClass and structuralObjectClass of "glue" (which are different than when the record was added.) Slapcat performed against the 2 other master servers (server 2 and server 3) do not display a record. When this error occurs, there are usually multiple records that fail to add with an error code of 68. I have removed all BDB index files and rerun the slapindex command, but the DN is still not found with the ldapsearch command and fails to be added because of a 68 error code. This condition is not repeatable on demand, but if I run my scripts doing ldapadd/ldapdelete multiple times, it will eventually occur. This error appears to be related to the value specified in my slapd.conf file for "syncprov-sessionlog". I have changed the value of "syncprov-sessionlog", with the following results: syncprov-sessionlog 5000 - usually execution 3 or 4 results in the ldapadd 68 error (e.g., the first 2 executions of 8000 adds/deletes work OK) syncprov-sessionlog 50000 - usually execution 6 or 7 results in the ldapadd 68 error (e.g., the first 5 executions of 8000 adds/deletes work OK) syncprov-sessionlog 200 - usually execution 2 or 3 results in the ldapadd 68 error (e.g., the first execution of 8000 adds/deletes works OK) syncprov-sessionlog not specified - usually execution 2 or 3 results in 1 of the slapd servers crashing with a segmentation fault (usually server 1, but sometimes the other servers) (example of crash output is *** glibc detected *** /tmp/reptest/openldap/openldap-2.4.17/libexec/slapd: malloc(): memory corruption (fast): 0x9e61f860 ***) I am using BDB 4.6.21 and have tested with the 4 BDB patches applied and not applied (the 68 error occurs using BDB without the patches and BDB with the patches).

1 0

Re: (ITS#6152) proxycache enhancements
by hyc＠symas.com 20 Aug '09

20 Aug '09

hyc(a)symas.com wrote: > hyc(a)symas.com wrote: >> The TTR feature is now available for testing in HEAD. (As well as the offline >> mode switch.) > Bind caching is now working in HEAD as well. manpage is up to date, test020 > needs some new tests still. Working on that... And passwordModify will now update the cached Bind info. So that's it for the code changes, I'm just looking at the test020 script and config files to decide how to add tests for refresh, expiration, bind, and pwdMod. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs