- openldap-bugs - openldap.org

Re: (ITS#6277) Missing documentation for cn=config
by ghenry＠OpenLDAP.org 01 Sep '09

01 Sep '09

----- ryans(a)aweber.com wrote: > Gavin, > > > Well there are two places that talk about how to convert from > slapd.conf > > to cn=config formats. In the guide and man pages, so that is the > best > > way to do a full conversion and see the end result. > > > > Where would you like to see these added? > > > > Thanks for the feedback as always! > > My vote would be section 5.2.2.3 of the Admin Guide. Hi Ryan, Thanks for the time you spent replying in length. Oh, maybe that time could have been spent writing a patch to docs! ;-) I'll take on board your suggestions and make some changes when I can. Patches welcome! Cheers. -- Kind Regards, Gavin Henry. OpenLDAP Engineering Team. E ghenry(a)OpenLDAP.org Community developed LDAP software. http://www.openldap.org/project/

1 0

Re: (ITS#6277) Missing documentation for cn=config
by ryans＠aweber.com 01 Sep '09

01 Sep '09

Gavin, > Well there are two places that talk about how to convert from slapd.conf > to cn=config formats. In the guide and man pages, so that is the best > way to do a full conversion and see the end result. > > Where would you like to see these added? > > Thanks for the feedback as always! My vote would be section 5.2.2.3 of the Admin Guide. Currently, it has 2 examples of loading modules consisting of libtool libraries, but I would think that it would be good to give an example which, for pedagogical purposes, explained how to instantiate an overlay in cn=config. For example, ppolicy and autogroup: dn: olcOverlay=ppolicy,olcDatabase={1}hdb,cn=config objectClass: olcOverlayConfig objectClass: olcPPolicyConfig olcOverlay: ppolicy dn: olcOverlay=autogroup,olcDatabase={1}hdb,cn=config objectClass: olcOverlayConfig objectClass: olcAutomaticGroups olcOverlay: autogroup olcAGattrSet: groupOfNames labeledURI member Perhaps it would also be good to emphasize the need for overlay-specific objectclasses, such as the aforementioned 'olcOverlayConfig' and 'olcPPolicyConfig'. It probably also wouldn't hurt to mention that there are certain overlay-specific attributes, i.e. olcAGattrSet, that are necessary to make full use of the module, and that grepping through the contrib module's source can help one identify said attributes' names in lieu of adequate documentation. Something like: grep -C3 NAME contrib/slapd-modules/autogroup/autogroup.c, maybe? Thanks as always, Ryan

1 0

(ITS#6279) autogroup overlay causes slapcat segfault
by ryans＠aweber.com 31 Aug '09

31 Aug '09

Full_Name: Ryan Steele Version: 2.4.17 OS: Ubuntu URL: http://pastie.org/600973 Submission from: (NULL) (71.230.37.172) When the autogroup overlay directives are included in slapd.conf, it seems to work okay. When adding it to the cn=config DIT, however, it causes segfaults on slapcat. This is version 1.6 of the autogroup overlay (which fixed segfaults when using slapd.conf). Steps to reproduce (starting from a pristine installation) and strace of slapcat are in the pastebin linked below.

1 0

Re: (ITS#6278) Patch - Enhancement - provide support for PEM files in MozNSS crypto
by hyc＠symas.com 28 Aug '09

28 Aug '09

rmeggins(a)redhat.com wrote: > Full_Name: Rich Megginson > Version: 2.4.18 (current CVS HEAD) > OS: Fedora > URL: ftp://ftp.openldap.org/incoming/openldap-2.4.18-moznss-20090828.patch > Submission from: (NULL) (76.113.59.19) > > > This patch adds support for reading PEM encoded cert and key files to the MozNSS > crypto implementation. It depends on having the nsspem module library available > somewhere for the runtime linker to find it (e.g. on a linux system, it uses > dlopen() to load libnsspem.so). This module is available on Fedora and is > provided by the package nss-3.12.3.99. Work is underway to incorporate the PEM > module into MozNSS upstream at mozilla.org. The current source code repository > is > git://fedorapeople.org/~rcritten/pemnss.git > > The tlsm_init code attempts to determine if you are using PEM files or are using > a MozNSS key/cert database. If you specify the TLS cacertdir directive, and > that directory has valid key/cert databases in it, it will use them. Otherwise, > it will load the PEM module and attempt to load the certs and keys specified by > cacertdir, cacertfile, certfile, and keyfile. Thanks. I tweaked the #includes this time around because /usr/include/nspr and /usr/include/nss already exist on my Ubuntu machine. Unfortunately, several of the nspr header files still use unqualified paths to reference each other, so you still need -I/usr/include/nspr to get this to compile successfully. Needless to say, I think nspr needs to be cleaned up a bit more before it's ready for primetime. With that small change it compiles OK so I've committed it to HEAD. I'm not too thrilled to see PL_strdup / PL_strfree and friends here; everything else in the source tree uses our wrappers that funnel into ber_memalloc and we rely on this for accounting, leak testing and other validation. (Not to mention avoiding silly crashes when running on Windows where each DLL has its own heap manager...) It looks like all of the uses are self-contained and won't escape this module so it's probably OK. I'd suggest overriding the NSPR malloc hooks but that would probably mess up any other applications. We can probably roll this code into RE24 as well but I'm still going to omit the configure switch for the moment. > This patch file is derived from OpenLDAP Software. All of the > modifications to OpenLDAP Software represented in the following > patch(es) were developed by Red Hat, Inc.. Red Hat, Inc. has not > assigned rights and/or interest in this work to any party. I, Richard > Megginson am authorized by Red Hat, Inc., my employer, to release this > work under the following terms. > > Red Hat, Inc. hereby place the following modifications to OpenLDAP > Software (and only these modifications) into the public domain. Hence, > these modifications may be freely used and/or redistributed for any > purpose with or without attribution and/or other notice. > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#6278) Patch - Enhancement - provide support for PEM files in MozNSS crypto
by rmeggins＠redhat.com 28 Aug '09

28 Aug '09

Full_Name: Rich Megginson Version: 2.4.18 (current CVS HEAD) OS: Fedora URL: ftp://ftp.openldap.org/incoming/openldap-2.4.18-moznss-20090828.patch Submission from: (NULL) (76.113.59.19) This patch adds support for reading PEM encoded cert and key files to the MozNSS crypto implementation. It depends on having the nsspem module library available somewhere for the runtime linker to find it (e.g. on a linux system, it uses dlopen() to load libnsspem.so). This module is available on Fedora and is provided by the package nss-3.12.3.99. Work is underway to incorporate the PEM module into MozNSS upstream at mozilla.org. The current source code repository is git://fedorapeople.org/~rcritten/pemnss.git The tlsm_init code attempts to determine if you are using PEM files or are using a MozNSS key/cert database. If you specify the TLS cacertdir directive, and that directory has valid key/cert databases in it, it will use them. Otherwise, it will load the PEM module and attempt to load the certs and keys specified by cacertdir, cacertfile, certfile, and keyfile. This patch file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Red Hat, Inc.. Red Hat, Inc. has not assigned rights and/or interest in this work to any party. I, Richard Megginson am authorized by Red Hat, Inc., my employer, to release this work under the following terms. Red Hat, Inc. hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice.

1 0

Re: (ITS#6277) Missing documentation for cn=config
by hyc＠symas.com 27 Aug '09

27 Aug '09

masarati(a)aero.polimi.it wrote: >>> Full_Name: Ryan Steele >>> Version: 2.4.15 >>> OS: Ubuntu 8.04 LTD >>> URL: ftp://ftp.openldap.org/incoming/ >>> Submission from: (NULL) (207.106.239.81) >>> >>> >>> According to chapter 5 of the admin guide, "some of the backends and of >>> the >>> distributed overlays do not support runtime configuration yet. In those >> cases, >>> the old style slapd.conf(5) file must be used." However, there is no >>> documentation on which backends and overlays do and don't have said >>> support. >>> Without grokking the code, it is a trial-and-error operation at best. >> >> Point taken. I'm pretty sure they all do now. Will check and update that >> section. > > back-meta and back-sql don't yet. I have half (er, somewhere between 0 > and 100%, boundaries not included) in a working dir somewhere, and rough > ideas about what to do with back-meta. All of the core overlays now support cn=config. Yes, contrib is trial-and-error. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6277) Missing documentation for cn=config
by masarati＠aero.polimi.it 27 Aug '09

27 Aug '09

>> Full_Name: Ryan Steele >> Version: 2.4.15 >> OS: Ubuntu 8.04 LTD >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (207.106.239.81) >> >> >> According to chapter 5 of the admin guide, "some of the backends and of >> the >> distributed overlays do not support runtime configuration yet. In those > cases, >> the old style slapd.conf(5) file must be used." However, there is no >> documentation on which backends and overlays do and don't have said >> support. >> Without grokking the code, it is a trial-and-error operation at best. > > Point taken. I'm pretty sure they all do now. Will check and update that > section. back-meta and back-sql don't yet. I have half (er, somewhere between 0 and 100%, boundaries not included) in a working dir somewhere, and rough ideas about what to do with back-meta. p.

1 0

Re: (ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by rlvcosta＠yahoo.com 27 Aug '09

27 Aug '09

Quanah, Please see my comments in your previous e-mail. Thanks, Rodrigo. Quanah Gibson-Mount wrote: > --On Thursday, August 27, 2009 6:39 AM -0700 Rodrigo Costa > <rlvcosta(a)yahoo.com> wrote: > >> Quanah, >> >> Please see answer in your previous e-mail below. >> >> I'm also sending the information I could collect attached since it is a >> small file(5KB). >> >> The behavior that appears strange and that could indicate a problem is >> the fact that even when consumer is stopped the provider still doing >> something for a long time. This doesn't appear to be correct. >> >> Other strange behavior is that when system enters in this state one >> provider CPU stays running around 100% CPU usage. I made a jmeter script >> to test individual bind/search(no ldapsearch *) and then even with some >> load(like 200 simultaneous query) I do not see CPU in 100%. Something >> doesn't appear to be ok since I do not see why CPU should enter in 100% >> permanently. > > I explained to you previously why this would be. Other comments inline. > >>> Why are you stopping the provider to do a slapcat? >> [Rodrigo]Faster dump of data. And in any case if other situation like a >> problema occurs the secondary system could stay disconnect for other >> reasons. > [Rodrigo] I have 2 reasons : 1)Since backup takes sometime and DB has multiple branches for the same record the only way to have a consistent backup is executing a cold backup; 2)slapcat in a stop slapd could perform faster and also fulfill item 1 above(cold backup) > Do you have any evidence that an offline slapcat is faster than one > while slapd is running? I don't understand what you mean in the rest > of that sentence. > [Rodrigo] I didn't try with load traffic but it seems reasonable if a > cold backup is faster and cleaner than a hot backup. >>>> Even a small number of entrances are different when consumer in >>>> Provider 2 >>>> connects to Provider 1 then syncrepl enters in the full DB search as >>>> expected. >>> >>> >>> What is your sessionlog setting on each provider for the syncprov >>> overlay? >> [Rodrigo] >> syncprov-checkpoint 10000 120 >> syncprov-sessionlog 100000 > > Hm, I would probably checkpoint the cookie a lot more frequently than > you have it set to. The sessionlog setting seems fine to me. [Rodrigo] Ok > >> Same configuration in both systems. >>> >>>> For definition purposes I have some memory limitations where I need to >>>> limit dncachesize for around 80% of DB entrances. >>> >>> We already went through other things you could do to reduce your >>> memory footprint in other ways. You've completely ignored that >>> advice. As long as your dncachesize is in this state, I don't expect >>> things to behave normally. >> [Rodrigo]I implemented what was possible. The end is this cache config >> possible by the memory constraints : >> # Cache values >> # cachesize 10000 >> cachesize 20000 >> dncachesize 3000000 >> # dncachesize 400000 >> # idlcachesize 10000 >> idlcachesize 30000 >> # cachefree 10 >> cachefree 100 > > You don't say anything in here about your DB_CONFIG settings, which is > where you could stand to gain the most amount of memory back. I do > see you're definitely running a very restricted > cachesize/idlcachesize. ;) > [Rodrigo]DB_CONFIG is using only 100MB of memory and DB_LOG_AUTOREMOVE. > > >>> What value did you set for "cachefree"? >> [Rodrigo] cachefree 100 > [Rodrigo] I made the change proposed and tested. The behavior was > really better since after dncachesize was filled the issue did not > repeated as before. BUT it just took more time until the behavior repeats. After some more time then just after dncachesize reaches around 3Mi the behavior returned. What happens is : 1-> Provider 1 CPU start to consume around 100%; 2-> Consumer 2 CPU goes to 0% consumption(before it was around 10% when replication in place); 3-> Replication never ends(I cannot see in the Provider 2 data) and even I stop Consumer 2(or slapd) the CPU in Provider 1 remains days in 100%. Looks like code enter in a dead loop which I could not identify the condition or the requirement to avoid it. I generated some GDB traces and as soon as possible(there is space) I will put in the ftp. > > This value is likely substantially way too low for your system > configuration. This is how many entries get freed from any of the > caches. With your dncachesize being 3,000,000, removing 100 entries > from it will do hardly anything, and may be part of the issue. If it > wasn't for the major imbalance between your entry, idl, and > dncachesizes, I would suggest a fairly high value like 100,000. But > given your entry cache is 20,000, you'll probably have to limit the > cachefree to 5000-10000. But it is going to need to be higher than 100. > > --Quanah > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and collaboration > >

1 0

Re: (ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by quanah＠zimbra.com 27 Aug '09

27 Aug '09

--On Thursday, August 27, 2009 6:39 AM -0700 Rodrigo Costa <rlvcosta(a)yahoo.com> wrote: > Quanah, > > Please see answer in your previous e-mail below. > > I'm also sending the information I could collect attached since it is a > small file(5KB). > > The behavior that appears strange and that could indicate a problem is > the fact that even when consumer is stopped the provider still doing > something for a long time. This doesn't appear to be correct. > > Other strange behavior is that when system enters in this state one > provider CPU stays running around 100% CPU usage. I made a jmeter script > to test individual bind/search(no ldapsearch *) and then even with some > load(like 200 simultaneous query) I do not see CPU in 100%. Something > doesn't appear to be ok since I do not see why CPU should enter in 100% > permanently. I explained to you previously why this would be. Other comments inline. >> Why are you stopping the provider to do a slapcat? > [Rodrigo]Faster dump of data. And in any case if other situation like a > problema occurs the secondary system could stay disconnect for other > reasons. Do you have any evidence that an offline slapcat is faster than one while slapd is running? I don't understand what you mean in the rest of that sentence. >>> Even a small number of entrances are different when consumer in >>> Provider 2 >>> connects to Provider 1 then syncrepl enters in the full DB search as >>> expected. >> >> >> What is your sessionlog setting on each provider for the syncprov >> overlay? > [Rodrigo] > syncprov-checkpoint 10000 120 > syncprov-sessionlog 100000 Hm, I would probably checkpoint the cookie a lot more frequently than you have it set to. The sessionlog setting seems fine to me. > Same configuration in both systems. >> >>> For definition purposes I have some memory limitations where I need to >>> limit dncachesize for around 80% of DB entrances. >> >> We already went through other things you could do to reduce your >> memory footprint in other ways. You've completely ignored that >> advice. As long as your dncachesize is in this state, I don't expect >> things to behave normally. > [Rodrigo]I implemented what was possible. The end is this cache config > possible by the memory constraints : ># Cache values ># cachesize 10000 > cachesize 20000 > dncachesize 3000000 ># dncachesize 400000 ># idlcachesize 10000 > idlcachesize 30000 ># cachefree 10 > cachefree 100 You don't say anything in here about your DB_CONFIG settings, which is where you could stand to gain the most amount of memory back. I do see you're definitely running a very restricted cachesize/idlcachesize. ;) >> What value did you set for "cachefree"? > [Rodrigo] cachefree 100 This value is likely substantially way too low for your system configuration. This is how many entries get freed from any of the caches. With your dncachesize being 3,000,000, removing 100 entries from it will do hardly anything, and may be part of the issue. If it wasn't for the major imbalance between your entry, idl, and dncachesizes, I would suggest a fairly high value like 100,000. But given your entry cache is 20,000, you'll probably have to limit the cachefree to 5000-10000. But it is going to need to be higher than 100. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#6277) Missing documentation for cn=config
by Gavin Henry 27 Aug '09

27 Aug '09

> Full_Name: Ryan Steele > Version: 2.4.15 > OS: Ubuntu 8.04 LTD > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (207.106.239.81) > > > According to chapter 5 of the admin guide, "some of the backends and of the > distributed overlays do not support runtime configuration yet. In those cases, > the old style slapd.conf(5) file must be used." However, there is no > documentation on which backends and overlays do and don't have said support. > Without grokking the code, it is a trial-and-error operation at best. Point taken. I'm pretty sure they all do now. Will check and update that section. > Also, there is no documentation on how to add overlay-specific directives (man > pages or otherwise). Take, for example, autogroup-attrset; the olcAGattrSet > directive is only described in autogroup.c, and nowhere else. This is not the > only instance of missing module documentation, but it should give a general idea > of where to look. Some of the contribs one don't come with a man page unfortunately. Core ones do. I'll look into the relevant sections and add one or two examples. > Of course, grepping the code is easy enough, but you shouldn't have to do that > to learn how to achieve simple configurations. If there's going to be a > fundamental paradigm shift from slapd.conf to cn=config, there has to be > documentation (man pages, admin guide sections, et. al.) on the appropriate > methods for achieving what once was done through slapd.conf. IMHO, that should > include concrete examples, such as the following, pulled from the Courier > documentation: > > The following LDIF could be used to add [the auditlog] overlay to cn=config > (adjust to suit): > > dn: olcOverlay=auditlog,olcDatabase={1}hdb,cn=config > changetype: add > objectClass: olcOverlayConfig > objectClass: olcAuditLogConfig > olcOverlay: auditlog > olcAuditlogFile: /tmp/auditlog.ldif > Well there are two places that talk about how to convert from slapd.conf to cn=config formats. In the guide and man pages, so that is the best way to do a full conversion and see the end result. Where would you like to see these added? Thanks for the feedback as always! Gavin.

1 0

(ITS#6277) Missing documentation for cn=config
by ryans＠aweber.com 27 Aug '09

27 Aug '09

Full_Name: Ryan Steele Version: 2.4.15 OS: Ubuntu 8.04 LTD URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (207.106.239.81) According to chapter 5 of the admin guide, "some of the backends and of the distributed overlays do not support runtime configuration yet. In those cases, the old style slapd.conf(5) file must be used." However, there is no documentation on which backends and overlays do and don't have said support. Without grokking the code, it is a trial-and-error operation at best. Also, there is no documentation on how to add overlay-specific directives (man pages or otherwise). Take, for example, autogroup-attrset; the olcAGattrSet directive is only described in autogroup.c, and nowhere else. This is not the only instance of missing module documentation, but it should give a general idea of where to look. Of course, grepping the code is easy enough, but you shouldn't have to do that to learn how to achieve simple configurations. If there's going to be a fundamental paradigm shift from slapd.conf to cn=config, there has to be documentation (man pages, admin guide sections, et. al.) on the appropriate methods for achieving what once was done through slapd.conf. IMHO, that should include concrete examples, such as the following, pulled from the Courier documentation: The following LDIF could be used to add [the auditlog] overlay to cn=config (adjust to suit): dn: olcOverlay=auditlog,olcDatabase={1}hdb,cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcAuditLogConfig olcOverlay: auditlog olcAuditlogFile: /tmp/auditlog.ldif

1 0

Re: (ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by quanah＠zimbra.com 27 Aug '09

27 Aug '09

--On Thursday, August 27, 2009 6:39 AM -0700 Rodrigo Costa <rlvcosta(a)yahoo.com> wrote: > Quanah, > > Please see answer in your previous e-mail below. > > I'm also sending the information I could collect attached since it is a > small file(5KB). The ITS system does not accept attachments. You'll need to upload it to the FTP site, which had space cleared up. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#6276) paused pool can deadlock if writers are waiting
by h.b.furuseth＠usit.uio.no 27 Aug '09

27 Aug '09

Re-reading this, I'm not sure what either of us are talking about... About reading a byte and putting it back if select() says the socket is readable: I suppose this is only necessary if conn->c_writewaiter != 0, and maybe only if the pool is paused. And remember that the next select() will often also find the socket is readable, so we must read the next byte (from the socket, not the sockbuf) and put that on the sockbut stack, and so on. About multiple types of pauses: The concept must be rephrased for that to make sense. The pause mechanism is in effect a pool-wide read/write lock which favors writers over readers. A pool thread readlocks it while running a task. pool_pause() does unlock - writelock. pool_resume() and pool_pausecheck() do unlock - readlock. Now it's possible to talk about moving a few things out of this particular lock, and if necessary protect it with another lock. -- Hallvard

1 0

Re: (ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by rlvcosta＠yahoo.com 27 Aug '09

27 Aug '09

--0-698620411-1251380367=:64533 Content-Type: text/plain; charset=us-ascii Quanah, Please see answer in your previous e-mail below. I'm also sending the information I could collect attached since it is a small file(5KB). The behavior that appears strange and that could indicate a problem is the fact that even when consumer is stopped the provider still doing something for a long time. This doesn't appear to be correct. Other strange behavior is that when system enters in this state one provider CPU stays running around 100% CPU usage. I made a jmeter script to test individual bind/search(no ldapsearch *) and then even with some load(like 200 simultaneous query) I do not see CPU in 100%. Something doesn't appear to be ok since I do not see why CPU should enter in 100% permanently. Regards, Rodrigo. Quanah Gibson-Mount wrote: > --On Tuesday, August 25, 2009 6:16 PM +0000 rlvcosta(a)yahoo.com wrote: > >> The issue I'm facing is related, in a general user view, is when I stop >> the secondary Provider2(master 2) for backup purposes using slapcat. The >> Provider1(master 1) continues to provide ldap service where some >> entrances can be created during the time backup is running(no consumer >> from Provider 2). > > Why are you stopping the provider to do a slapcat? [Rodrigo]Faster dump of data. And in any case if other situation like a problema occurs the secondary system could stay disconnect for other reasons. > >> Even a small number of entrances are different when consumer in >> Provider 2 >> connects to Provider 1 then syncrepl enters in the full DB search as >> expected. > > > What is your sessionlog setting on each provider for the syncprov > overlay? [Rodrigo] syncprov-checkpoint 10000 120 syncprov-sessionlog 100000 Same configuration in both systems. > >> For definition purposes I have some memory limitations where I need to >> limit dncachesize for around 80% of DB entrances. > > We already went through other things you could do to reduce your > memory footprint in other ways. You've completely ignored that > advice. As long as your dncachesize is in this state, I don't expect > things to behave normally. [Rodrigo]I implemented what was possible. The end is this cache config possible by the memory constraints : #Cache values #cachesize 10000 cachesize 20000 dncachesize 3000000 #dncachesize 400000 #idlcachesize 10000 idlcachesize 30000 #cachefree 10 cachefree 100 > >> I could also note that when in this situation the monitor cache, in a >> very slow pace, changes the cache in a single entrance. Being more >> specific : >> >> dn: cn=Database 1,cn=Databases,cn=Monitor >> structuralObjectClass: monitoredObject >> creatorsName: >> modifiersName: >> createTimestamp: 20090821145848Z >> modifyTimestamp: 20090821145848Z >> monitoredInfo: bdb >> monitorIsShadow: TRUE >> namingContexts: ou=CONTENT,o=domain,c=fr >> readOnly: FALSE >> monitorOverlay: syncprov >> olmBDBEntryCache: 19920 >> olmBDBDNCache: 3896287 >> olmBDBIDLCache: 2 >> olmDbDirectory: /var/openldap-data/bdb1/ >> entryDN: cn=Database 1,cn=Databases,cn=Monitor >> subschemaSubentry: cn=Subschema >> hasSubordinates: TRUE >> >> Stays running in the values 3896287 and 3896288. Looks like the memory >> re-use is being too short causing locks that takes long time causing >> a non >> synchronization. > > > What value did you set for "cachefree"? [Rodrigo] cachefree 100 > >> PS-> I could not put the file in the openldap ftp. It says device full. >> Please let me know how can I send this file. > > I've let the maintainer of the system know, hopefully there'll be > space available soon. [Rodrigo]I'm sending file attached since it is very small 5KB. > > --Quanah > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and collaboration > > --0-698620411-1251380367=:64533 Content-Type: application/octet-stream; name="replication_too_slow.zip" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="replication_too_slow.zip" UEsDBBQAAgAIABd5GTtTPer8EQEAAMUBAAAJABUAREJfQ09ORklHVVQJAAOu KJRKsCiUSlV4BADuAwECXZBbT4NAEIXf+RWT0EcLuysI8qYVmyYKia1Gn8jC TpFkuQRWUv31LiCofZxvLufMMWG1EwHc3SabOLrfba1UpHQdxfu3aHPRA7UY MEIcm3j2JQHiBy4LHAZCSQhPDawME/aoQL0jCK54yjuEooISy7r9hIxnutEV X2gZpmF2qJIRDQQIuMxhvk8I0PMehQGez1Pi+K53NS38KC+qR8nzbpEZq+Gt w2uUTO8Y//hDvE1ung/xU/gYv4TLOVnn0HP5geOpYUPmSYt5UVclP80O5sYf RGaWjmYZufaoyyYzmoqiBbvnrV03WEnBG1sHvdZq3SKtsGzqluvYjoVEyFrk SsuCXsVM6ThnR6psfu/pwvgGUEsDBBQAAgAIAFFzFTsgtwjuGwkAAAlhAAAw ABUAaXRzX3JlcGxpY2F0aW9uX29wZW5sZGFwX2dkYl90cmFjZXNfMjAwOTA4 MjEudHh0VVQJAAPK2I5KaA2PSlV4BADuAwEC7Vxtb9s4Ev4eIP+B2EWB5Jp4 +SaKCppu79rsXYBNNmjTOxyKhUBJVOKNbHklOZtgr/fbj5RkS7LlWH7LZe8o N6hNDofkzGieISnNFwThq5/39w5uAu8QZLeJFAEQo1H0CEQUAS/b39vfuy6K LXBQfjtGFnEYRBaG4ODHf1wBTLDjHB6e7O99CwGADxBSQinEoD8Ernsnk6GM 3Pv0MfU114NDRYcKOksyGjBNJ0dxFLm/iX6mCECYxAPwXdT39J/fS+MeU41w 3ohDGweC6kZpJEaBGwg5iIduINMsiR8L/qTgz7iklpeTZiLJ3HKKsz2MinLd D1SN6WRw+qMb+1E8lAvHNRURrYuIEttxnEpEBMK1RMQcjK1cROUoXT8eBrmk 3r3764/nf3nv4h7p4WWTKoSnZMcCyDW7KBAjd3Q/EUrFFhzor6fwgQe2z6F/ BAbjTD6UBXZI1eiAukSmTcYdxWn/oeefYNuuBG9jy5F02k1/OO1mFMdKz4my MpmAgwf9c8oZHuZMdZliyGxcKWM9TVprapI0NKkUyUldk6ihSUIhL4zYdePU 7ddGNU6TSR+Bd0x7tupoolqPQa7uJF83/P77Que4qqCwXlGKtbhqFft7x8fH 148jCd4kMhsnw7cgi4HSX9YfjuURiBPwa6Pq13E/U02mM8X1mTLMHGozPpmp TYmx2T+KzaKaJjHklo0RpZWD5utpkoeWZdc1+UusfnTWmw9Jm94KJsWPU2xb FmGWbbOjEoPcwmCVjGG73pBT6U3hARGWmMWDYhq0JLFsPww0yUD0y6qOksix UYu5+nxBEBvYNLBpXNAmsIl9B/NSk1Hs37mjceYOY/21G4BCLjGSlM8BqAgC iALMFwEoqlfkghJEhNSS9Rbr4mohPBH4gkG/geGsrPB91hybvRDcDUgbkDYg vVwS7Z8vDjcwbWDaOKFNYJo50MFOXUO5vNwseWxCdfsYUaUYx29TzAy3iTIE cyxCebsyCMSVzqVAfph7PxUiuL7wb6UbJWMVTiQ3ShiqMNeAj4ksNJCTKC42 pZVKpcCMoCaXsK9sRi/n45EeEfE86kmu/F8/tx9J7FCon+qXA23KLHIE1o8b 9DRlv+hIBSOcKcsMI3Fzio+AFs20wmrOAikUqQxJOphhbzKNVIrEv50bf5JO fwo/Z1YQaplANjUsDkPL93OfHd/LJBKPbjxS91B0V4tncipphwup7ClVWMBM KxUvY0JoYT/XZfo49EdJfJ8rwU+H5SS00EWg5jCIA3n6w/nlB/fq49mns8vr ylImTfPp6LvWmTAXnt1krsbQkNGU/VREDErayhorFWn7hlNR0cWiQqiLqBDu IipESpxwwkJUoazPIyehExKH5SRB3Ky3ynpmF3eOMtCh9LO+BprChWgqNqEK yCyVH/Wl8nppFo9KYnviGQXzN/SMuUEjvoFnRM56nvElB9prL0gMNpoA/eUE 6CYaN9G48Thrb5pRioNSkyrAy0RaBtArbpvZz7DRtDZgGWs0+Pc/vEF1vPpl oNNAp3FW/33o1BcNYMuJ0/xxTy5izwownKkoRUUYr1c82f1mh1FtB2VseuzV dhjVdoLGp8d2UqLasd2NzLThqHtDREvF6EzOwOwgbHAv91HmB5pvnegW3A6a FbgcDw+LgdaOETuPp9hMMUdvJrL540Y2m+w4v6QjOxPemPDGeKwNHqcJfX9d XEYlLgceRPPhTRsuk2mFcNrCm41weT2XBmpP3wTSEtwAugF0s1WxlWdpDEAb gDYeaDOA5oywJkDH3i8r7DwgQleA5vzyRPuzrs1F/462F8y62sCwgeGtwLCB XgO9xutsAL0U1qE3TKRc8cR8ZpXbBXxbXjR5zm1/vQoOlcrmcbll2OW2vy8h bgB2te2/yTZ7+QQmxIKFJhww4YAJB7a3Ku8QJIBwHEW7ihQuY5A+Drw4Apnw Im0qYQzEvehH+mdvzVBiOdfuscZyXpsEI8u5rxitLGW4m3Cmsx635Fa7anjX fleNQwGaiNLeDnwwKC9dBk7BgTLKsZ+1c03Bnw7BlNdm53r6ykR6p/uc7UyX u1nRGazIf4uTOzfqp1mzTT60kS6ftMllbNda+tmDavN7lI312zKnQHlShJhN tCfVhXfyUdcrgrvyu2ICg8DBgX5jpmCiKwORiaJWEio8fpSX6mBt2sZmFLzR d7qbRu5ADiY3+tuvR+D3Oq+qI5sR4R/NdcApgs3uax3ZDPkeeDPzXsRrm+T9 NPhLTBHniyfCLQvOTEQKQnEA3uh3dLRKZJLXPTEHLplDWuaALL5wDqoNgqWw 4tFEUK8xnJsDXDD6mXF/1fY2kiJLgQoFs/5Apm+/fq3s4K4whDmDG6cyUVUt NtdX5KR2m6gBpVGsLRAzqyq+FemtKrMIUnemU5W7+uWf6+t/uj98vnx/ff7T pesqsm+eume/2TAuWu61rF15++dZ8iwdz4aP9c6vpTpD/dyqpTOwz63GOoP2 Ksu1jrbRsi5b3pKtLYJFK7vlLbex9Fvey6K1YfcbY6uLx+cOg9a9nUz4tCR8 egFhEMFYiQLzdcMgITiCWwqDipc+juY74GRRCPFUmDITPsyGDW38auEDgbsO HxCbDx8wpozR/7vwYcMMcVvd0OnuXdfZ8VnFJ+5yS6jFJz61PdQ1GGnbP1re 1pw3mfMms639zDlTEJWi0GQza8p4mEfQOaQNs47JUxQ7zuF8CpaC2Qpq9sLF GVgmzKYJWCyGQrgwAQuvp05BIbLWSMDCMdnkUU+6ncwtjhKvPYlYNsjAYm0z AwvrlIHF7pSBhXfKwOLsMgNL8XbPjjKwoE4ZWHCnDCykUwYWujwDi7UkAwvr lIHFXiUDC992BhZnAz+L4QvIwGLeQDcHyOYAeT6+/5LEcfbOS7IoGCnH+O+f vwWjFBzL8F83aoVedL+/p4nydTImDlW9OBxqWAKInFhQyTH9DupaCE/yf6De tvoAcHX+AXz+dPaxWHRfqf8vzwH4+/nHawAUcgDw6W8fwSfw6v3VZ/Dq4uxC U12fX5y9BuD9TxcXf778AKprfy83glwTRYlytfmoIMEDwKk9UANEA8VQFakB 2j2kvrATiHtKh/nwQOPSg/wPUEsDBBQAAgAIAPV4GTsJpOScnQcAAJoWAAAK ABUAc2xhcGQuY29uZlVUCQADbiiUSnEolEpVeAQA7gMBAtVYXW8buxF91v4K Vn6xU1my5CS9NbBFHVu516jjBJFT3DwZ3CUlMeaSG5IrWy3633uGuyvJH7px UMNAhSDZJWfI+ThzZjY7yQ6bSMm85qXo59ZMd9/ssal1TMjAlfbMGkbLalY5 HhTebEn/+D40L+fKs6nS0J/bSgt28fGSZZLdWIcXJ7ngmZaQTJTJdSVkpzOQ IR/YUhoteDnw+VwWfJBbJ/v18w8lvTJPlIVgsG5WSueteZqKUf6eIGt/j8p/ LxCjn9IopPd8pszs6hGn/1i1LPh+UemgfvpSLoTDve+svW71kL1jre0NOz89 /rQYsVwraQKl2si8TjCr84s/lNYwl8DElMOAfsKjaqaMuFqM6KxTy4wNTBpK ODI/lc5xoKcyQWl2/P5y/JktbcXmfCEZJ4BcIwZMKIfbrFviCC/dQuWSHV+c Mm6gKZC4wI0gQTtdHwo8tc8d8vJoMHDWhn7rdR9JT5JSCUImEr3gbuAqsxGV CHYIJNzNfJTaJkQCSaLtTMuF1J3o67nlgoml4YXKWcbza2kEK6yotPRHSf1Q 8jDvDCrvBlpl8lauc9IIaBzSIeWrTGR9zR8sF9YoRObuVp1jvzR56eyC9nbY fcUYgi07WfH4Tsm9vxGP7/m51DpuJZQi6c5O2RAvVP6SGXkbgA0HDtEoOM9q aFReUs4uzyeRTKTJ3RK8gUxuIAxStEKXiqoolixIDwhKF9RU5TyASOYqn0fg 5IDETBrpaDlbsnzODZURCxbqEfXltRoE7Qd0gO8xZNOQQLfg1y3BlbLo9oAu Ada6pU2QQ6G8j9ZYQyhs5Zi38IuHCHyKKfnkGJyZOVuV0SBiOKYCKuWrrVxb Q95Oww13kmLJl0CIvmY4x0s93fdqZqTY9BGWzu0N0OUio55PTo5P1rvviVwf ejfI+X6GCgG55i40ej/WWjn3QOMfcvljJeoVvCiJ7mVeORWWiIAPTtXpTHY6 n+X3CiXNlAlyFgWA01Cnm+2WDm4iQHP1DcBC+Pc2VIbD0X6mAts9PB1PKMyZ DEG6vRV2cAJBqSoFRW1D8+1hVLwn51W0lDgqkktjsPfTdNgcchVfhqNG9opk 07evN/zkeQ7eJMwGZzUrrVb58ojuBuGw08n4qME7N0trJMDYYgIykyqryZbt +irbg+du+SOdj0CbIyFPt3RqiibksBuY3xq03uIVFMCxlEMREerr0x4IGmuW ha18I4RbN3UheBq5WC1Qw0ZKgeMgEwNTUNJWvrdBwa4w/Yx7mXa7VJKv4s2P C+QmXYVjm/ArGIGdtbf1+9qp+n3tCTkQiUhN0X3uJQvbwAYg52F9b7N9Na7Q 3RQa3yaCaIHq8E5WsBfmxBRZFVZoR1xbHEZBgAGeMrYr+7N+j3U3fFr72t2L ttbCkT64vuHLNl+4vE7x+J/jz18vfzu7+PVPQOLz/HAv2syQwWROCaFQKPSX umyf6Y728A6uSnw1BcWuxhLWtVV68vHicnxx2bOpsJhhTC9Pp66bNBFZiwIs XBTY366zw0605C5Q94nNyzoBJpW+lLlCWpeRAyjp9em9ZkKlQCDTC6sAcGSs nX3rBrj7y15MxNZpmCj6S93agARLzWtdRMQ8ICF0Aj7D4dGv8qbTAfc4GZrU 08K/I1H9R317//Uin1zk4d3XWdL003WG2tGIffgyuWTyVqE3lk6RW3bV3KKl cWQikquHcGvgftZyhaKBDCCkWNTSrYc4x9YufbCYIP9ycAAo5rZAuVNwkrUF zWhJM1I7xuyToQMCFVl+hiktr4uBkoRYmSYBmB5blzCrCnkb61mJXmlMVWTS 9cCNxS0WaKrFoIUU8wyDKteKe4WK/N6jIm6UbfYNNp1opAs7zWJk1pPJRS8+ fPmCAQV7yc4JB9+wBdcV9Yuc3rz6VzsrDw/wS1arnVF8F+au3OFB/CU7Gxud 182aEvqOdH3k5mrnsJaMC1OaktrLk9VSB2pJQkOdk6VmCE56cDBMGPXOBXDq 0mbKHR70h69H/eGb/uvR0eEvf4VIWJYydXSMnx8b8QlUCZxgA4hzy7T7hg3f kA/sz10sepRMPq8ZeXttQS7yNP6KXTq10ylyjBkmOJ92X/XiWdQsCxnmVqR1 +2zWhEmfUsCMwVtBlYOJPm1LJCmUc9YVhEeMYxZsrDFCtQMvjaDN4360rrQA Wx12NhwdrHe9jEMd8FTvIgXsM300bPI2lUTBPaaMZKdeRiTZtmDTASUaB/dU /neRvUOZs7P4KRzLBFP/IH5GrOqlvmi/FqwVcBZmPx9ifveHfXkbpw4QT0E3 gmLAcBjE0hyNAcKassDuhp57zeivQubz9NfJ5PjTWSNFzJSfiZRuGGyYsHnN 38e/H3/4dD7un3z88Ly9ZvQyvabtGdtaztnF6fj38eSnWs4WnWdoOS/ScNZe rRrPyzSWQd1P/qduMgJV3i/stwcs/Rs7xteAiS7z6pa+26b4kIMqEM1CFW1+ J9211BJD9rvHe1LTL25kVjeadcuhp1VLutdzQChU2GhRURbf/fLFulKz+vTe ROtPb0+rU/64Q61te9CnRi/Qp7YU5PP2qa2X/P/0qWHdp56Nxpv/gnpZJm8u Tf4LUEsBAhcDFAACAAgAF3kZO1M96vwRAQAAxQEAAAkADQAAAAAAAQAAAKSB AAAAAERCX0NPTkZJR1VUBQADriiUSlV4AABQSwECFwMUAAIACABRcxU7ILcI 7hsJAAAJYQAAMAANAAAAAAABAAAAwIFNAQAAaXRzX3JlcGxpY2F0aW9uX29w ZW5sZGFwX2dkYl90cmFjZXNfMjAwOTA4MjEudHh0VVQFAAPK2I5KVXgAAFBL AQIXAxQAAgAIAPV4GTsJpOScnQcAAJoWAAAKAA0AAAAAAAEAAACkgcsKAABz bGFwZC5jb25mVVQFAANuKJRKVXgAAFBLBQYAAAAAAwADAPQAAAClEgAAAAA= --0-698620411-1251380367=:64533--

1 0

Re: (ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by quanah＠zimbra.com 26 Aug '09

26 Aug '09

--On Tuesday, August 25, 2009 6:16 PM +0000 rlvcosta(a)yahoo.com wrote: > The issue I'm facing is related, in a general user view, is when I stop > the secondary Provider2(master 2) for backup purposes using slapcat. The > Provider1(master 1) continues to provide ldap service where some > entrances can be created during the time backup is running(no consumer > from Provider 2). Why are you stopping the provider to do a slapcat? > Even a small number of entrances are different when consumer in Provider 2 > connects to Provider 1 then syncrepl enters in the full DB search as > expected. What is your sessionlog setting on each provider for the syncprov overlay? > For definition purposes I have some memory limitations where I need to > limit dncachesize for around 80% of DB entrances. We already went through other things you could do to reduce your memory footprint in other ways. You've completely ignored that advice. As long as your dncachesize is in this state, I don't expect things to behave normally. > I could also note that when in this situation the monitor cache, in a > very slow pace, changes the cache in a single entrance. Being more > specific : > > dn: cn=Database 1,cn=Databases,cn=Monitor > structuralObjectClass: monitoredObject > creatorsName: > modifiersName: > createTimestamp: 20090821145848Z > modifyTimestamp: 20090821145848Z > monitoredInfo: bdb > monitorIsShadow: TRUE > namingContexts: ou=CONTENT,o=domain,c=fr > readOnly: FALSE > monitorOverlay: syncprov > olmBDBEntryCache: 19920 > olmBDBDNCache: 3896287 > olmBDBIDLCache: 2 > olmDbDirectory: /var/openldap-data/bdb1/ > entryDN: cn=Database 1,cn=Databases,cn=Monitor > subschemaSubentry: cn=Subschema > hasSubordinates: TRUE > > Stays running in the values 3896287 and 3896288. Looks like the memory > re-use is being too short causing locks that takes long time causing a non > synchronization. What value did you set for "cachefree"? > PS-> I could not put the file in the openldap ftp. It says device full. > Please let me know how can I send this file. I've let the maintainer of the system know, hopefully there'll be space available soon. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#6276) paused pool can deadlock if writers are waiting
by h.b.furuseth＠usit.uio.no 26 Aug '09

26 Aug '09

h.b.furuseth(a)usit.uio.no writes: > Then we could add the option (c) a *small* set of (groups of?) config > variables which each will need their own mutex or pause or something. > Such as loglevel, since the code is doing Debug() all over the place. I forgot: Simplest way here is to have two versions of the variable: the one accessed by cn=config, and a copy which is the one slapd obeys. The cn=config variable is pause-protected as usual, and slapd copies it to the active variable at its leisure. This will be acceptable for config changes that cannot fail once the value has been verified and need not take effect immediately. Then the copy task needs locks both for the configurable variable ("lock" = prevent_pause() ... "unlock" = allow_pause()?) and for the active variable, but not at the same time. -- Hallvard

1 0

Re: (ITS#6276) paused pool can deadlock if writers are waiting
by hyc＠symas.com 26 Aug '09

26 Aug '09

Hallvard B Furuseth wrote: > But I quite agree > that this is hopeless if that "small" set cannot be kept small. For one > thing it might introduce yet another "lock order", threatening to be > inconsistent with the lock order of other mutexes. >> Another possibility is to just try to read 1 byte in the listener >> thread, to detect the hangup there when we have no other means to >> discover it. We would have to make sure to be able to unget this byte >> back to the bottom of the sockbuf stack if there's valid data. This >> will affect the throughput of the listener thread, but it may not be >> too terrible a hit. > > Does that help for a socket which has gotten blocked on select() due to > full socket buffers? A socket that's blocked on a write will eventually clear itself up - either the client will catch up or it will go away. So there's really no reason to worry about them at all. > Also if doing - a read() OS call anyway, why not read() a large enough > chunk that it would commonly be a full PDU? As long as it's read to a > sockbuf buffer rather than into slapd's data. Because we don't want to keep the listener thread busy for any longer than necessary. In terms of system call overhead it might be reasonable to read as much as one memory page (e.g. 4KB) but even that involves a lot of cycles for memcpy, and the idea is to shunt all the time-consuming work elsewhere so that the listener can return to listening ASAP. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6276) paused pool can deadlock if writers are waiting
by h.b.furuseth＠usit.uio.no 26 Aug '09

26 Aug '09

Howard Chu writes: >Hallvard B Furuseth wrote: >> Beyond that, my immediate reaction was that pauses are implemented at >> the wrong level and/or need to be split up in different types of pauses. >> There is no good reason cn=config's need to have slapd config variables >> for itself to affect network operations - nor, I hope, affect pool-level >> operations like pool_purgekey(). > > I suppose implementing things this way can appear to be too blunt. But the > amount of locks required to do it at a finer level is, IMO, unmanageable. I know. That's why I'm not suggesting to do away with pauses, only to make them a little less crude than "1 type of pause for everything". See below. > configurations are changed from under them.) > >> E.g. cn=config could use a slapd_config_pause() call to ask for lone >> access to the config variables instead of thread_pool_pause() call. >> Slapd operations that must respect such pauses should thus call a >> slapd_config_pausecheck() macro/function, not thread_pool_pausecheck() >> or depend on the pool to respect slapd-level pauses. >> connection_read_thread() can then go ahead and use the connection >> independently of those pauses. If it reaches connection_operation(), >> it'll need to check for slapd pause. > > Yes, I've thought about this approach too. That makes the pause mechanisms > even messier. It does? I was hoping that would be the cleaner solution. Lifting them from pool level to slapd level would mean that every task now submitted to the pool will be responsible for (a) calling pausecheck itself or (b) avoiding all variables/features that today need pauses. Then we could add the option (c) a *small* set of (groups of?) config variables which each will need their own mutex or pause or something. Such as loglevel, since the code is doing Debug() all over the place. And presumably some network variables, for this ITS. But I quite agree that this is hopeless if that "small" set cannot be kept small. For one thing it might introduce yet another "lock order", threatening to be inconsistent with the lock order of other mutexes. > Another possibility is to just try to read 1 byte in the listener > thread, to detect the hangup there when we have no other means to > discover it. We would have to make sure to be able to unget this byte > back to the bottom of the sockbuf stack if there's valid data. This > will affect the throughput of the listener thread, but it may not be > too terrible a hit. Does that help for a socket which has gotten blocked on select() due to full socket buffers? Also if doing - a read() OS call anyway, why not read() a large enough chunk that it would commonly be a full PDU? As long as it's read to a sockbuf buffer rather than into slapd's data. -- Hallvard

1 0

Re: (ITS#6257) libldap: getopt flag to return the SASL username
by hyc＠symas.com 26 Aug '09

26 Aug '09

masarati(a)aero.polimi.it wrote: > My concern was not from an operational point of view: the simple concept > of having a library dynamically loading something that could no longer be > present is calling for trouble, unless handled appropriately, and probably > there is no way to do it safely as one could always remove a .so while > it's in use (although I guess on any decent system the object will be > cached or loaded somewhere until it's in use). > > My concern is about the char* array returned by that call: if for any > reason the library decides to refresh it, but the caller of > ldap_get_option() is still holding a pointer to that array, this calls for > pointing to freed memory and things like that, as far as I understand. > For this reason, returning a copy sounds wiser. Whether the contents of > that copy is valid or not, namely the related mechanism is available or > not, that's an entirely different issue. The Cyrus mechlist is only initialized once during the life of the library. (Subsequent init calls just increment an init counter and then return.) It will not change in normal use. If an application loads / unloads / reloads the library, the list may change, but any app that goes thru this trouble will already know they have to call the init functions all over again, and re-fetch this list. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6257) libldap: getopt flag to return the SASL username
by masarati＠aero.polimi.it 26 Aug '09

26 Aug '09

>> On a somewhat related issue, I note that LDAP_OPT_X_SASL_MECHLIST >> returns >> a pointer to an array of chars that apparently cannot be mucked with. >> >> Assuming my understanding is correct, I wonder if this behavior is >> desirable or not, given the fact that if another mech is added, e.g. by >> adding a dynamic module, I expect this list to change. > > These are SASL mechs with the plugin modules. Right? > >>From an operational standpoint: If a SASL plugin module for a mech was >> added I > think it's acceptable that a software which queries this option is > restarted > before this SASL mech is known to the software. Probably one has to add > additional configuration for this SASL mech. > > Now the question is what happens if a SASL plugin module is removed and > the > software trys to use the removed SASL mech. Clearly removing plugin > modules in > a running system is asking for trouble anyway... > > Having said this I would not care too much about this list going to > change... My concern was not from an operational point of view: the simple concept of having a library dynamically loading something that could no longer be present is calling for trouble, unless handled appropriately, and probably there is no way to do it safely as one could always remove a .so while it's in use (although I guess on any decent system the object will be cached or loaded somewhere until it's in use). My concern is about the char* array returned by that call: if for any reason the library decides to refresh it, but the caller of ldap_get_option() is still holding a pointer to that array, this calls for pointing to freed memory and things like that, as far as I understand. For this reason, returning a copy sounds wiser. Whether the contents of that copy is valid or not, namely the related mechanism is available or not, that's an entirely different issue. p.

1 0

Re: (ITS#6276) paused pool can deadlock if writers are waiting
by hyc＠symas.com 26 Aug '09

26 Aug '09

Hallvard B Furuseth wrote: > This is a quick abstract answer, I haven't dived into the code to see > what my suggestions would mean in practice. Anyway: > It sounds to me like this problem cannot be fully prevented when network > operations share the same pool queue as everything else, and we have > operations that can wait for other operations. All pool threads can be > occupied with LDAP-level operations, so that network-level operations > which the LDAP-level operations depend on can get blocked. If slapd has > a design which even tries to guarantee forward progress, I'm not aware > of it. > > So LDAP-level operations ought to leave at least one thread free for for > network-level operations. Not necessary a designated thread: A thread > moving from network-level to LDAP-level operation like > connection_read_thread() does could first check that at least one other > thread remains available for network-level operation. Yes... > Beyond that, my immediate reaction was that pauses are implemented at > the wrong level and/or need to be split up in different types of pauses. > There is no good reason cn=config's need to have slapd config variables > for itself to affect network operations - nor, I hope, affect pool-level > operations like pool_purgekey(). I suppose implementing things this way can appear to be too blunt. But the amount of locks required to do it at a finer level is, IMO, unmanageable. The fact that you can change global slapd config parameters (such as the number of threads, size of sockbuf buffers, etc.) makes it inherently safe for *anything* else to be active while config changes are being made. And sifting thru each variable to decide how sensitive they are, and when they are unsafe, will inevitably lead to requiring locks on every single piece of configuration data. (Want to look up an attribute type, or objectclass? Or a database suffix? etc. etc. etc... There are countless things we do arbitrarily that simply won't work if we allow arbitrary threads to continue to run while their configurations are changed from under them.) > E.g. cn=config could use a slapd_config_pause() call to ask for lone > access to the config variables instead of thread_pool_pause() call. > Slapd operations that must respect such pauses should thus call a > slapd_config_pausecheck() macro/function, not thread_pool_pausecheck() > or depend on the pool to respect slapd-level pauses. > connection_read_thread() can then go ahead and use the connection > independently of those pauses. If it reaches connection_operation(), > it'll need to check for slapd pause. Yes, I've thought about this approach too. That makes the pause mechanisms even messier. Another possibility is to just try to read 1 byte in the listener thread, to detect the hangup there when we have no other means to discover it. We would have to make sure to be able to unget this byte back to the bottom of the sockbuf stack if there's valid data. This will affect the throughput of the listener thread, but it may not be too terrible a hit. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6276) paused pool can deadlock if writers are waiting
by h.b.furuseth＠usit.uio.no 26 Aug '09

26 Aug '09

This is a quick abstract answer, I haven't dived into the code to see what my suggestions would mean in practice. Anyway: It sounds to me like this problem cannot be fully prevented when network operations share the same pool queue as everything else, and we have operations that can wait for other operations. All pool threads can be occupied with LDAP-level operations, so that network-level operations which the LDAP-level operations depend on can get blocked. If slapd has a design which even tries to guarantee forward progress, I'm not aware of it. So LDAP-level operations ought to leave at least one thread free for for network-level operations. Not necessary a designated thread: A thread moving from network-level to LDAP-level operation like connection_read_thread() does could first check that at least one other thread remains available for network-level operation. Beyond that, my immediate reaction was that pauses are implemented at the wrong level and/or need to be split up in different types of pauses. There is no good reason cn=config's need to have slapd config variables for itself to affect network operations - nor, I hope, affect pool-level operations like pool_purgekey(). E.g. cn=config could use a slapd_config_pause() call to ask for lone access to the config variables instead of thread_pool_pause() call. Slapd operations that must respect such pauses should thus call a slapd_config_pausecheck() macro/function, not thread_pool_pausecheck() or depend on the pool to respect slapd-level pauses. connection_read_thread() can then go ahead and use the connection independently of those pauses. If it reaches connection_operation(), it'll need to check for slapd pause. Alternatively, with different types of pauses/operations: The pool could have different sets of pauses (a pause-type bitmask?) and if one type of pauses happens, operations that ignore that type of pause will not be affected. This messes up the pool code further though, and I already hate what pauses do to the otherwise clean pool code:-( -- Hallvard

1 0

(ITS#6276) paused pool can deadlock if writers are waiting
by hyc＠OpenLDAP.org 25 Aug '09

25 Aug '09

Full_Name: Howard Chu Version: RE24/HEAD OS: Solaris 10 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (76.91.220.157) Submitted by: hyc test050 hung on me after some number of iterations. Unfortunately I didn't save the stack traces, but basically there was one thread waiting in send_ldap_ber() on the write2 cv, and another thread in config_back_add() waiting for a pool pause to succeed. netstat showed that no connections had queued data, so there should have been no reason for the writer to still be waiting. I believe what happened here is that while the writer was waiting (it was a syncprov qtask replaying events for a psearch) the psearch connection got closed. Solaris is using select, and select() doesn't specially distinguish socket close events - they're reported as read events. The deadlock is because we queue read events into the thread pool, and we don't discover they're actually closed sockets until the read thread gets to run and tries to read from the socket (and gets zero bytes back). But since the pool is entering a pause, the reader thread cannot run, so it can't detect the hangup and dispose of the waiting writer. The ideal fix for this is to process hangup events inline in the listener thread instead of pushing them into the thread pool. But that requires being able to cheaply determine that a hangup actually occurred, and select() doesn't give us this information. We could get this info using poll() instead. Since nowadays any POSIX platform that implements select() also implements poll() we can probably just switch to poll() and drop select(). One exception is Windows; Winsock only supports poll() on Windows Vista and newer. (Note, we had a patch that added a connection_hangup() handler for Linux epoll() at one point, but I dropped it later because it seemed to have strange interactions with Samba. Should look into resurrecting it again.) I don't think we can really fix this issue without knowing for certain when hangup events occur. If we're forced to keep using select, that implies that the main listener thread must attempt a read on the socket before deciding how to dispatch the connection. Any thoughts?

1 0

Re: (ITS#6274) nssov: makefile suffix rules too greedy
by hyc＠symas.com 25 Aug '09

25 Aug '09

jonathan(a)phillipoux.net wrote: > Full_Name: Jonathan Clarke > Version: RE24 > OS: Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (82.67.204.30) > > > When running "make" in the nssov module from contrib/slapd-modules, I get this: > 8<------------------- > [...]/contrib/slapd-modules/nssov$ make > ../../../libtool --mode=compile gcc -g -O2 -I../../../include > -I../../../include -I../../../servers/slapd -Inss-ldapd -c alias.c nssov.h > libtool: compile: cannot determine name of library object from `nssov.h' > 8<------------------- > > This trivial patch to Makefile corrects this: > 8<------------------- > - $(LIBTOOL) --mode=compile $(CC) $(OPT) $(DEFS) $(INCS) -c $? > + $(LIBTOOL) --mode=compile $(CC) $(OPT) $(DEFS) $(INCS) -c $< > 8<------------------- > > Thanks, fixed in HEAD. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by rlvcosta＠yahoo.com 25 Aug '09

25 Aug '09

Full_Name: Rodrigo Luiz Vargas Costa Version: 2.4.17 OS: CentOS release 5.2 (Final) URL: ftp://ftp.openldap.org/incoming/<TBD> Submission from: (NULL) (135.245.8.5) Openldap developers, I have being exchange some information at openldap lists where looks like some improvements are being done in replication for release 2.4.18. The architecture I'm running has 2 machines in MirrorMode in the same subnet(at the same switch). These systems are part of a HA system sharing a VIP and where both machines have slapd running simultaneously(bind to any local interface) and only VIP is exchanged for HA purposes. The issue I'm facing is related, in a general user view, is when I stop the secondary Provider2(master 2) for backup purposes using slapcat. The Provider1(master 1) continues to provide ldap service where some entrances can be created during the time backup is running(no consumer from Provider 2). Even a small number of entrances are different when consumer in Provider 2 connects to Provider 1 then syncrepl enters in the full DB search as expected. For definition purposes I have some memory limitations where I need to limit dncachesize for around 80% of DB entrances. >From a user perspective I see that after cache is filled system enters in some state where synchronization doesn't happen anymore. For full reference(config, gdb, etc), please see file attached in FTP. Then I see 2 issues : 1)Consumer from Provider2, even passed days and only a small number of differences for test purpose happen(no traffic), the syncrepl never ends and there isn't replication(Provider 1 stay continuously consuming 100% CPU); 2)Even I stop the Provider2(then its consumer) I do not see any change in Provider 1 activities. The CPU continues in 100% even passed days what suggest some hang in the thread or logic. I compiled openldap with GDB symbols and then execute some traces in the threads during the state 2 report above. Looks like it stay looping forever locked in some thread lock. I could also note that when in this situation the monitor cache, in a very slow pace, changes the cache in a single entrance. Being more specific : dn: cn=Database 1,cn=Databases,cn=Monitor structuralObjectClass: monitoredObject creatorsName: modifiersName: createTimestamp: 20090821145848Z modifyTimestamp: 20090821145848Z monitoredInfo: bdb monitorIsShadow: TRUE namingContexts: ou=CONTENT,o=domain,c=fr readOnly: FALSE monitorOverlay: syncprov olmBDBEntryCache: 19920 olmBDBDNCache: 3896287 olmBDBIDLCache: 2 olmDbDirectory: /var/openldap-data/bdb1/ entryDN: cn=Database 1,cn=Databases,cn=Monitor subschemaSubentry: cn=Subschema hasSubordinates: TRUE Stays running in the values 3896287 and 3896288. Looks like the memory re-use is being too short causing locks that takes long time causing a non synchronization. I made several GDB traces for different conditions. Please see ftp attachment file for details. Thanks, Rodrigo. PS-> I could not put the file in the openldap ftp. It says device full. Please let me know how can I send this file.

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs