openldap-bugs August 2009

openldap-bugs@openldap.org

35 participants
225 discussions

(ITS#6279) autogroup overlay causes slapcat segfault
by ryans＠aweber.com 01 Sep '09

01 Sep '09

Full_Name: Ryan Steele Version: 2.4.17 OS: Ubuntu URL: http://pastie.org/600973 Submission from: (NULL) (71.230.37.172) When the autogroup overlay directives are included in slapd.conf, it seems to work okay. When adding it to the cn=config DIT, however, it causes segfaults on slapcat. This is version 1.6 of the autogroup overlay (which fixed segfaults when using slapd.conf). Steps to reproduce (starting from a pristine installation) and strace of slapcat are in the pastebin linked below.

1 0

Re: (ITS#6278) Patch - Enhancement - provide support for PEM files in MozNSS crypto
by hyc＠symas.com 29 Aug '09

29 Aug '09

rmeggins(a)redhat.com wrote: > Full_Name: Rich Megginson > Version: 2.4.18 (current CVS HEAD) > OS: Fedora > URL: ftp://ftp.openldap.org/incoming/openldap-2.4.18-moznss-20090828.patch > Submission from: (NULL) (76.113.59.19) > > > This patch adds support for reading PEM encoded cert and key files to the MozNSS > crypto implementation. It depends on having the nsspem module library available > somewhere for the runtime linker to find it (e.g. on a linux system, it uses > dlopen() to load libnsspem.so). This module is available on Fedora and is > provided by the package nss-3.12.3.99. Work is underway to incorporate the PEM > module into MozNSS upstream at mozilla.org. The current source code repository > is > git://fedorapeople.org/~rcritten/pemnss.git > > The tlsm_init code attempts to determine if you are using PEM files or are using > a MozNSS key/cert database. If you specify the TLS cacertdir directive, and > that directory has valid key/cert databases in it, it will use them. Otherwise, > it will load the PEM module and attempt to load the certs and keys specified by > cacertdir, cacertfile, certfile, and keyfile. Thanks. I tweaked the #includes this time around because /usr/include/nspr and /usr/include/nss already exist on my Ubuntu machine. Unfortunately, several of the nspr header files still use unqualified paths to reference each other, so you still need -I/usr/include/nspr to get this to compile successfully. Needless to say, I think nspr needs to be cleaned up a bit more before it's ready for primetime. With that small change it compiles OK so I've committed it to HEAD. I'm not too thrilled to see PL_strdup / PL_strfree and friends here; everything else in the source tree uses our wrappers that funnel into ber_memalloc and we rely on this for accounting, leak testing and other validation. (Not to mention avoiding silly crashes when running on Windows where each DLL has its own heap manager...) It looks like all of the uses are self-contained and won't escape this module so it's probably OK. I'd suggest overriding the NSPR malloc hooks but that would probably mess up any other applications. We can probably roll this code into RE24 as well but I'm still going to omit the configure switch for the moment. > This patch file is derived from OpenLDAP Software. All of the > modifications to OpenLDAP Software represented in the following > patch(es) were developed by Red Hat, Inc.. Red Hat, Inc. has not > assigned rights and/or interest in this work to any party. I, Richard > Megginson am authorized by Red Hat, Inc., my employer, to release this > work under the following terms. > > Red Hat, Inc. hereby place the following modifications to OpenLDAP > Software (and only these modifications) into the public domain. Hence, > these modifications may be freely used and/or redistributed for any > purpose with or without attribution and/or other notice. > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#6278) Patch - Enhancement - provide support for PEM files in MozNSS crypto
by rmeggins＠redhat.com 29 Aug '09

29 Aug '09

Full_Name: Rich Megginson Version: 2.4.18 (current CVS HEAD) OS: Fedora URL: ftp://ftp.openldap.org/incoming/openldap-2.4.18-moznss-20090828.patch Submission from: (NULL) (76.113.59.19) This patch adds support for reading PEM encoded cert and key files to the MozNSS crypto implementation. It depends on having the nsspem module library available somewhere for the runtime linker to find it (e.g. on a linux system, it uses dlopen() to load libnsspem.so). This module is available on Fedora and is provided by the package nss-3.12.3.99. Work is underway to incorporate the PEM module into MozNSS upstream at mozilla.org. The current source code repository is git://fedorapeople.org/~rcritten/pemnss.git The tlsm_init code attempts to determine if you are using PEM files or are using a MozNSS key/cert database. If you specify the TLS cacertdir directive, and that directory has valid key/cert databases in it, it will use them. Otherwise, it will load the PEM module and attempt to load the certs and keys specified by cacertdir, cacertfile, certfile, and keyfile. This patch file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Red Hat, Inc.. Red Hat, Inc. has not assigned rights and/or interest in this work to any party. I, Richard Megginson am authorized by Red Hat, Inc., my employer, to release this work under the following terms. Red Hat, Inc. hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice.

1 0

Re: (ITS#6277) Missing documentation for cn=config
by hyc＠symas.com 28 Aug '09

28 Aug '09

masarati(a)aero.polimi.it wrote: >>> Full_Name: Ryan Steele >>> Version: 2.4.15 >>> OS: Ubuntu 8.04 LTD >>> URL: ftp://ftp.openldap.org/incoming/ >>> Submission from: (NULL) (207.106.239.81) >>> >>> >>> According to chapter 5 of the admin guide, "some of the backends and of >>> the >>> distributed overlays do not support runtime configuration yet. In those >> cases, >>> the old style slapd.conf(5) file must be used." However, there is no >>> documentation on which backends and overlays do and don't have said >>> support. >>> Without grokking the code, it is a trial-and-error operation at best. >> >> Point taken. I'm pretty sure they all do now. Will check and update that >> section. > > back-meta and back-sql don't yet. I have half (er, somewhere between 0 > and 100%, boundaries not included) in a working dir somewhere, and rough > ideas about what to do with back-meta. All of the core overlays now support cn=config. Yes, contrib is trial-and-error. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6277) Missing documentation for cn=config
by masarati＠aero.polimi.it 28 Aug '09

28 Aug '09

>> Full_Name: Ryan Steele >> Version: 2.4.15 >> OS: Ubuntu 8.04 LTD >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (207.106.239.81) >> >> >> According to chapter 5 of the admin guide, "some of the backends and of >> the >> distributed overlays do not support runtime configuration yet. In those > cases, >> the old style slapd.conf(5) file must be used." However, there is no >> documentation on which backends and overlays do and don't have said >> support. >> Without grokking the code, it is a trial-and-error operation at best. > > Point taken. I'm pretty sure they all do now. Will check and update that > section. back-meta and back-sql don't yet. I have half (er, somewhere between 0 and 100%, boundaries not included) in a working dir somewhere, and rough ideas about what to do with back-meta. p.

1 0

Re: (ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by rlvcosta＠yahoo.com 28 Aug '09

28 Aug '09

Quanah, Please see my comments in your previous e-mail. Thanks, Rodrigo. Quanah Gibson-Mount wrote: > --On Thursday, August 27, 2009 6:39 AM -0700 Rodrigo Costa > <rlvcosta(a)yahoo.com> wrote: > >> Quanah, >> >> Please see answer in your previous e-mail below. >> >> I'm also sending the information I could collect attached since it is a >> small file(5KB). >> >> The behavior that appears strange and that could indicate a problem is >> the fact that even when consumer is stopped the provider still doing >> something for a long time. This doesn't appear to be correct. >> >> Other strange behavior is that when system enters in this state one >> provider CPU stays running around 100% CPU usage. I made a jmeter script >> to test individual bind/search(no ldapsearch *) and then even with some >> load(like 200 simultaneous query) I do not see CPU in 100%. Something >> doesn't appear to be ok since I do not see why CPU should enter in 100% >> permanently. > > I explained to you previously why this would be. Other comments inline. > >>> Why are you stopping the provider to do a slapcat? >> [Rodrigo]Faster dump of data. And in any case if other situation like a >> problema occurs the secondary system could stay disconnect for other >> reasons. > [Rodrigo] I have 2 reasons : 1)Since backup takes sometime and DB has multiple branches for the same record the only way to have a consistent backup is executing a cold backup; 2)slapcat in a stop slapd could perform faster and also fulfill item 1 above(cold backup) > Do you have any evidence that an offline slapcat is faster than one > while slapd is running? I don't understand what you mean in the rest > of that sentence. > [Rodrigo] I didn't try with load traffic but it seems reasonable if a > cold backup is faster and cleaner than a hot backup. >>>> Even a small number of entrances are different when consumer in >>>> Provider 2 >>>> connects to Provider 1 then syncrepl enters in the full DB search as >>>> expected. >>> >>> >>> What is your sessionlog setting on each provider for the syncprov >>> overlay? >> [Rodrigo] >> syncprov-checkpoint 10000 120 >> syncprov-sessionlog 100000 > > Hm, I would probably checkpoint the cookie a lot more frequently than > you have it set to. The sessionlog setting seems fine to me. [Rodrigo] Ok > >> Same configuration in both systems. >>> >>>> For definition purposes I have some memory limitations where I need to >>>> limit dncachesize for around 80% of DB entrances. >>> >>> We already went through other things you could do to reduce your >>> memory footprint in other ways. You've completely ignored that >>> advice. As long as your dncachesize is in this state, I don't expect >>> things to behave normally. >> [Rodrigo]I implemented what was possible. The end is this cache config >> possible by the memory constraints : >> # Cache values >> # cachesize 10000 >> cachesize 20000 >> dncachesize 3000000 >> # dncachesize 400000 >> # idlcachesize 10000 >> idlcachesize 30000 >> # cachefree 10 >> cachefree 100 > > You don't say anything in here about your DB_CONFIG settings, which is > where you could stand to gain the most amount of memory back. I do > see you're definitely running a very restricted > cachesize/idlcachesize. ;) > [Rodrigo]DB_CONFIG is using only 100MB of memory and DB_LOG_AUTOREMOVE. > > >>> What value did you set for "cachefree"? >> [Rodrigo] cachefree 100 > [Rodrigo] I made the change proposed and tested. The behavior was > really better since after dncachesize was filled the issue did not > repeated as before. BUT it just took more time until the behavior repeats. After some more time then just after dncachesize reaches around 3Mi the behavior returned. What happens is : 1-> Provider 1 CPU start to consume around 100%; 2-> Consumer 2 CPU goes to 0% consumption(before it was around 10% when replication in place); 3-> Replication never ends(I cannot see in the Provider 2 data) and even I stop Consumer 2(or slapd) the CPU in Provider 1 remains days in 100%. Looks like code enter in a dead loop which I could not identify the condition or the requirement to avoid it. I generated some GDB traces and as soon as possible(there is space) I will put in the ftp. > > This value is likely substantially way too low for your system > configuration. This is how many entries get freed from any of the > caches. With your dncachesize being 3,000,000, removing 100 entries > from it will do hardly anything, and may be part of the issue. If it > wasn't for the major imbalance between your entry, idl, and > dncachesizes, I would suggest a fairly high value like 100,000. But > given your entry cache is 20,000, you'll probably have to limit the > cachefree to 5000-10000. But it is going to need to be higher than 100. > > --Quanah > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and collaboration > >

1 0

Re: (ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by quanah＠zimbra.com 28 Aug '09

28 Aug '09

--On Thursday, August 27, 2009 6:39 AM -0700 Rodrigo Costa <rlvcosta(a)yahoo.com> wrote: > Quanah, > > Please see answer in your previous e-mail below. > > I'm also sending the information I could collect attached since it is a > small file(5KB). > > The behavior that appears strange and that could indicate a problem is > the fact that even when consumer is stopped the provider still doing > something for a long time. This doesn't appear to be correct. > > Other strange behavior is that when system enters in this state one > provider CPU stays running around 100% CPU usage. I made a jmeter script > to test individual bind/search(no ldapsearch *) and then even with some > load(like 200 simultaneous query) I do not see CPU in 100%. Something > doesn't appear to be ok since I do not see why CPU should enter in 100% > permanently. I explained to you previously why this would be. Other comments inline. >> Why are you stopping the provider to do a slapcat? > [Rodrigo]Faster dump of data. And in any case if other situation like a > problema occurs the secondary system could stay disconnect for other > reasons. Do you have any evidence that an offline slapcat is faster than one while slapd is running? I don't understand what you mean in the rest of that sentence. >>> Even a small number of entrances are different when consumer in >>> Provider 2 >>> connects to Provider 1 then syncrepl enters in the full DB search as >>> expected. >> >> >> What is your sessionlog setting on each provider for the syncprov >> overlay? > [Rodrigo] > syncprov-checkpoint 10000 120 > syncprov-sessionlog 100000 Hm, I would probably checkpoint the cookie a lot more frequently than you have it set to. The sessionlog setting seems fine to me. > Same configuration in both systems. >> >>> For definition purposes I have some memory limitations where I need to >>> limit dncachesize for around 80% of DB entrances. >> >> We already went through other things you could do to reduce your >> memory footprint in other ways. You've completely ignored that >> advice. As long as your dncachesize is in this state, I don't expect >> things to behave normally. > [Rodrigo]I implemented what was possible. The end is this cache config > possible by the memory constraints : ># Cache values ># cachesize 10000 > cachesize 20000 > dncachesize 3000000 ># dncachesize 400000 ># idlcachesize 10000 > idlcachesize 30000 ># cachefree 10 > cachefree 100 You don't say anything in here about your DB_CONFIG settings, which is where you could stand to gain the most amount of memory back. I do see you're definitely running a very restricted cachesize/idlcachesize. ;) >> What value did you set for "cachefree"? > [Rodrigo] cachefree 100 This value is likely substantially way too low for your system configuration. This is how many entries get freed from any of the caches. With your dncachesize being 3,000,000, removing 100 entries from it will do hardly anything, and may be part of the issue. If it wasn't for the major imbalance between your entry, idl, and dncachesizes, I would suggest a fairly high value like 100,000. But given your entry cache is 20,000, you'll probably have to limit the cachefree to 5000-10000. But it is going to need to be higher than 100. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#6277) Missing documentation for cn=config
by Gavin Henry 28 Aug '09

28 Aug '09

> Full_Name: Ryan Steele > Version: 2.4.15 > OS: Ubuntu 8.04 LTD > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (207.106.239.81) > > > According to chapter 5 of the admin guide, "some of the backends and of the > distributed overlays do not support runtime configuration yet. In those cases, > the old style slapd.conf(5) file must be used." However, there is no > documentation on which backends and overlays do and don't have said support. > Without grokking the code, it is a trial-and-error operation at best. Point taken. I'm pretty sure they all do now. Will check and update that section. > Also, there is no documentation on how to add overlay-specific directives (man > pages or otherwise). Take, for example, autogroup-attrset; the olcAGattrSet > directive is only described in autogroup.c, and nowhere else. This is not the > only instance of missing module documentation, but it should give a general idea > of where to look. Some of the contribs one don't come with a man page unfortunately. Core ones do. I'll look into the relevant sections and add one or two examples. > Of course, grepping the code is easy enough, but you shouldn't have to do that > to learn how to achieve simple configurations. If there's going to be a > fundamental paradigm shift from slapd.conf to cn=config, there has to be > documentation (man pages, admin guide sections, et. al.) on the appropriate > methods for achieving what once was done through slapd.conf. IMHO, that should > include concrete examples, such as the following, pulled from the Courier > documentation: > > The following LDIF could be used to add [the auditlog] overlay to cn=config > (adjust to suit): > > dn: olcOverlay=auditlog,olcDatabase={1}hdb,cn=config > changetype: add > objectClass: olcOverlayConfig > objectClass: olcAuditLogConfig > olcOverlay: auditlog > olcAuditlogFile: /tmp/auditlog.ldif > Well there are two places that talk about how to convert from slapd.conf to cn=config formats. In the guide and man pages, so that is the best way to do a full conversion and see the end result. Where would you like to see these added? Thanks for the feedback as always! Gavin.

1 0

(ITS#6277) Missing documentation for cn=config
by ryans＠aweber.com 28 Aug '09

28 Aug '09

Full_Name: Ryan Steele Version: 2.4.15 OS: Ubuntu 8.04 LTD URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (207.106.239.81) According to chapter 5 of the admin guide, "some of the backends and of the distributed overlays do not support runtime configuration yet. In those cases, the old style slapd.conf(5) file must be used." However, there is no documentation on which backends and overlays do and don't have said support. Without grokking the code, it is a trial-and-error operation at best. Also, there is no documentation on how to add overlay-specific directives (man pages or otherwise). Take, for example, autogroup-attrset; the olcAGattrSet directive is only described in autogroup.c, and nowhere else. This is not the only instance of missing module documentation, but it should give a general idea of where to look. Of course, grepping the code is easy enough, but you shouldn't have to do that to learn how to achieve simple configurations. If there's going to be a fundamental paradigm shift from slapd.conf to cn=config, there has to be documentation (man pages, admin guide sections, et. al.) on the appropriate methods for achieving what once was done through slapd.conf. IMHO, that should include concrete examples, such as the following, pulled from the Courier documentation: The following LDIF could be used to add [the auditlog] overlay to cn=config (adjust to suit): dn: olcOverlay=auditlog,olcDatabase={1}hdb,cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcAuditLogConfig olcOverlay: auditlog olcAuditlogFile: /tmp/auditlog.ldif

1 0

Re: (ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by quanah＠zimbra.com 28 Aug '09

28 Aug '09

--On Thursday, August 27, 2009 6:39 AM -0700 Rodrigo Costa <rlvcosta(a)yahoo.com> wrote: > Quanah, > > Please see answer in your previous e-mail below. > > I'm also sending the information I could collect attached since it is a > small file(5KB). The ITS system does not accept attachments. You'll need to upload it to the FTP site, which had space cleared up. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs August 2009