(ITS#6279) autogroup overlay causes slapcat segfault
by ryans＠aweber.com
Full_Name: Ryan Steele
Version: 2.4.17
OS: Ubuntu
URL: http://pastie.org/600973
Submission from: (NULL) (71.230.37.172)
When the autogroup overlay directives are included in slapd.conf, it seems to
work okay. When adding it to the cn=config DIT, however, it causes segfaults on
slapcat. This is version 1.6 of the autogroup overlay (which fixed segfaults
when using slapd.conf). Steps to reproduce (starting from a pristine
installation) and strace of slapcat are in the pastebin linked below.
14 years, 1 month
Re: (ITS#6278) Patch - Enhancement - provide support for PEM files in MozNSS crypto
by hyc＠symas.com
rmeggins(a)redhat.com wrote:
> Full_Name: Rich Megginson
> Version: 2.4.18 (current CVS HEAD)
> OS: Fedora
> URL: ftp://ftp.openldap.org/incoming/openldap-2.4.18-moznss-20090828.patch
> Submission from: (NULL) (76.113.59.19)
>
>
> This patch adds support for reading PEM encoded cert and key files to the MozNSS
> crypto implementation. It depends on having the nsspem module library available
> somewhere for the runtime linker to find it (e.g. on a linux system, it uses
> dlopen() to load libnsspem.so). This module is available on Fedora and is
> provided by the package nss-3.12.3.99. Work is underway to incorporate the PEM
> module into MozNSS upstream at mozilla.org. The current source code repository
> is
> git://fedorapeople.org/~rcritten/pemnss.git
>
> The tlsm_init code attempts to determine if you are using PEM files or are using
> a MozNSS key/cert database. If you specify the TLS cacertdir directive, and
> that directory has valid key/cert databases in it, it will use them. Otherwise,
> it will load the PEM module and attempt to load the certs and keys specified by
> cacertdir, cacertfile, certfile, and keyfile.
Thanks. I tweaked the #includes this time around because /usr/include/nspr and
/usr/include/nss already exist on my Ubuntu machine. Unfortunately, several of
the nspr header files still use unqualified paths to reference each other, so
you still need -I/usr/include/nspr to get this to compile successfully.
Needless to say, I think nspr needs to be cleaned up a bit more before it's
ready for primetime.
With that small change it compiles OK so I've committed it to HEAD.
I'm not too thrilled to see PL_strdup / PL_strfree and friends here;
everything else in the source tree uses our wrappers that funnel into
ber_memalloc and we rely on this for accounting, leak testing and other
validation. (Not to mention avoiding silly crashes when running on Windows
where each DLL has its own heap manager...) It looks like all of the uses are
self-contained and won't escape this module so it's probably OK. I'd suggest
overriding the NSPR malloc hooks but that would probably mess up any other
applications.
We can probably roll this code into RE24 as well but I'm still going to omit
the configure switch for the moment.
> This patch file is derived from OpenLDAP Software. All of the
> modifications to OpenLDAP Software represented in the following
> patch(es) were developed by Red Hat, Inc.. Red Hat, Inc. has not
> assigned rights and/or interest in this work to any party. I, Richard
> Megginson am authorized by Red Hat, Inc., my employer, to release this
> work under the following terms.
>
> Red Hat, Inc. hereby place the following modifications to OpenLDAP
> Software (and only these modifications) into the public domain. Hence,
> these modifications may be freely used and/or redistributed for any
> purpose with or without attribution and/or other notice.
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
14 years, 1 month
(ITS#6278) Patch - Enhancement - provide support for PEM files in MozNSS crypto
by rmeggins＠redhat.com
Full_Name: Rich Megginson
Version: 2.4.18 (current CVS HEAD)
OS: Fedora
URL: ftp://ftp.openldap.org/incoming/openldap-2.4.18-moznss-20090828.patch
Submission from: (NULL) (76.113.59.19)
This patch adds support for reading PEM encoded cert and key files to the MozNSS
crypto implementation. It depends on having the nsspem module library available
somewhere for the runtime linker to find it (e.g. on a linux system, it uses
dlopen() to load libnsspem.so). This module is available on Fedora and is
provided by the package nss-3.12.3.99. Work is underway to incorporate the PEM
module into MozNSS upstream at mozilla.org. The current source code repository
is
git://fedorapeople.org/~rcritten/pemnss.git
The tlsm_init code attempts to determine if you are using PEM files or are using
a MozNSS key/cert database. If you specify the TLS cacertdir directive, and
that directory has valid key/cert databases in it, it will use them. Otherwise,
it will load the PEM module and attempt to load the certs and keys specified by
cacertdir, cacertfile, certfile, and keyfile.
This patch file is derived from OpenLDAP Software. All of the
modifications to OpenLDAP Software represented in the following
patch(es) were developed by Red Hat, Inc.. Red Hat, Inc. has not
assigned rights and/or interest in this work to any party. I, Richard
Megginson am authorized by Red Hat, Inc., my employer, to release this
work under the following terms.
Red Hat, Inc. hereby place the following modifications to OpenLDAP
Software (and only these modifications) into the public domain. Hence,
these modifications may be freely used and/or redistributed for any
purpose with or without attribution and/or other notice.
14 years, 1 month
Re: (ITS#6277) Missing documentation for cn=config
by hyc＠symas.com
masarati(a)aero.polimi.it wrote:
>>> Full_Name: Ryan Steele
>>> Version: 2.4.15
>>> OS: Ubuntu 8.04 LTD
>>> URL: ftp://ftp.openldap.org/incoming/
>>> Submission from: (NULL) (207.106.239.81)
>>>
>>>
>>> According to chapter 5 of the admin guide, "some of the backends and of
>>> the
>>> distributed overlays do not support runtime configuration yet. In those
>> cases,
>>> the old style slapd.conf(5) file must be used." However, there is no
>>> documentation on which backends and overlays do and don't have said
>>> support.
>>> Without grokking the code, it is a trial-and-error operation at best.
>>
>> Point taken. I'm pretty sure they all do now. Will check and update that
>> section.
>
> back-meta and back-sql don't yet. I have half (er, somewhere between 0
> and 100%, boundaries not included) in a working dir somewhere, and rough
> ideas about what to do with back-meta.
All of the core overlays now support cn=config. Yes, contrib is trial-and-error.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
14 years, 1 month
Re: (ITS#6277) Missing documentation for cn=config
by masarati＠aero.polimi.it
>> Full_Name: Ryan Steele
>> Version: 2.4.15
>> OS: Ubuntu 8.04 LTD
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (207.106.239.81)
>>
>>
>> According to chapter 5 of the admin guide, "some of the backends and of
>> the
>> distributed overlays do not support runtime configuration yet. In those
> cases,
>> the old style slapd.conf(5) file must be used." However, there is no
>> documentation on which backends and overlays do and don't have said
>> support.
>> Without grokking the code, it is a trial-and-error operation at best.
>
> Point taken. I'm pretty sure they all do now. Will check and update that
> section.
back-meta and back-sql don't yet. I have half (er, somewhere between 0
and 100%, boundaries not included) in a working dir somewhere, and rough
ideas about what to do with back-meta.
p.
14 years, 1 month
Re: (ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by rlvcosta＠yahoo.com
Quanah,
Please see my comments in your previous e-mail.
Thanks,
Rodrigo.
Quanah Gibson-Mount wrote:
> --On Thursday, August 27, 2009 6:39 AM -0700 Rodrigo Costa
> <rlvcosta(a)yahoo.com> wrote:
>
>> Quanah,
>>
>> Please see answer in your previous e-mail below.
>>
>> I'm also sending the information I could collect attached since it is a
>> small file(5KB).
>>
>> The behavior that appears strange and that could indicate a problem is
>> the fact that even when consumer is stopped the provider still doing
>> something for a long time. This doesn't appear to be correct.
>>
>> Other strange behavior is that when system enters in this state one
>> provider CPU stays running around 100% CPU usage. I made a jmeter script
>> to test individual bind/search(no ldapsearch *) and then even with some
>> load(like 200 simultaneous query) I do not see CPU in 100%. Something
>> doesn't appear to be ok since I do not see why CPU should enter in 100%
>> permanently.
>
> I explained to you previously why this would be. Other comments inline.
>
>>> Why are you stopping the provider to do a slapcat?
>> [Rodrigo]Faster dump of data. And in any case if other situation like a
>> problema occurs the secondary system could stay disconnect for other
>> reasons.
> [Rodrigo] I have 2 reasons :
1)Since backup takes sometime and DB has multiple branches for the same
record the only way to have a consistent backup is executing a cold backup;
2)slapcat in a stop slapd could perform faster and also fulfill item 1
above(cold backup)
> Do you have any evidence that an offline slapcat is faster than one
> while slapd is running? I don't understand what you mean in the rest
> of that sentence.
> [Rodrigo] I didn't try with load traffic but it seems reasonable if a
> cold backup is faster and cleaner than a hot backup.
>>>> Even a small number of entrances are different when consumer in
>>>> Provider 2
>>>> connects to Provider 1 then syncrepl enters in the full DB search as
>>>> expected.
>>>
>>>
>>> What is your sessionlog setting on each provider for the syncprov
>>> overlay?
>> [Rodrigo]
>> syncprov-checkpoint 10000 120
>> syncprov-sessionlog 100000
>
> Hm, I would probably checkpoint the cookie a lot more frequently than
> you have it set to. The sessionlog setting seems fine to me.
[Rodrigo] Ok
>
>> Same configuration in both systems.
>>>
>>>> For definition purposes I have some memory limitations where I need to
>>>> limit dncachesize for around 80% of DB entrances.
>>>
>>> We already went through other things you could do to reduce your
>>> memory footprint in other ways. You've completely ignored that
>>> advice. As long as your dncachesize is in this state, I don't expect
>>> things to behave normally.
>> [Rodrigo]I implemented what was possible. The end is this cache config
>> possible by the memory constraints :
>> # Cache values
>> # cachesize 10000
>> cachesize 20000
>> dncachesize 3000000
>> # dncachesize 400000
>> # idlcachesize 10000
>> idlcachesize 30000
>> # cachefree 10
>> cachefree 100
>
> You don't say anything in here about your DB_CONFIG settings, which is
> where you could stand to gain the most amount of memory back. I do
> see you're definitely running a very restricted
> cachesize/idlcachesize. ;)
> [Rodrigo]DB_CONFIG is using only 100MB of memory and DB_LOG_AUTOREMOVE.
>
>
>>> What value did you set for "cachefree"?
>> [Rodrigo] cachefree 100
> [Rodrigo] I made the change proposed and tested. The behavior was
> really better since after dncachesize was filled the issue did not
> repeated as before.
BUT it just took more time until the behavior repeats. After some more
time then just after dncachesize reaches around 3Mi the behavior
returned. What happens is :
1-> Provider 1 CPU start to consume around 100%;
2-> Consumer 2 CPU goes to 0% consumption(before it was around 10% when
replication in place);
3-> Replication never ends(I cannot see in the Provider 2 data) and even
I stop Consumer 2(or slapd) the CPU in Provider 1 remains days in 100%.
Looks like code enter in a dead loop which I could not identify the
condition or the requirement to avoid it. I generated some GDB traces
and as soon as possible(there is space) I will put in the ftp.
>
> This value is likely substantially way too low for your system
> configuration. This is how many entries get freed from any of the
> caches. With your dncachesize being 3,000,000, removing 100 entries
> from it will do hardly anything, and may be part of the issue. If it
> wasn't for the major imbalance between your entry, idl, and
> dncachesizes, I would suggest a fairly high value like 100,000. But
> given your entry cache is 20,000, you'll probably have to limit the
> cachefree to 5000-10000. But it is going to need to be higher than 100.
>
> --Quanah
>
> --
>
> Quanah Gibson-Mount
> Principal Software Engineer
> Zimbra, Inc
> --------------------
> Zimbra :: the leader in open source messaging and collaboration
>
>
14 years, 1 month
Re: (ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by quanah＠zimbra.com
--On Thursday, August 27, 2009 6:39 AM -0700 Rodrigo Costa
<rlvcosta(a)yahoo.com> wrote:
> Quanah,
>
> Please see answer in your previous e-mail below.
>
> I'm also sending the information I could collect attached since it is a
> small file(5KB).
>
> The behavior that appears strange and that could indicate a problem is
> the fact that even when consumer is stopped the provider still doing
> something for a long time. This doesn't appear to be correct.
>
> Other strange behavior is that when system enters in this state one
> provider CPU stays running around 100% CPU usage. I made a jmeter script
> to test individual bind/search(no ldapsearch *) and then even with some
> load(like 200 simultaneous query) I do not see CPU in 100%. Something
> doesn't appear to be ok since I do not see why CPU should enter in 100%
> permanently.
I explained to you previously why this would be. Other comments inline.
>> Why are you stopping the provider to do a slapcat?
> [Rodrigo]Faster dump of data. And in any case if other situation like a
> problema occurs the secondary system could stay disconnect for other
> reasons.
Do you have any evidence that an offline slapcat is faster than one while
slapd is running? I don't understand what you mean in the rest of that
sentence.
>>> Even a small number of entrances are different when consumer in
>>> Provider 2
>>> connects to Provider 1 then syncrepl enters in the full DB search as
>>> expected.
>>
>>
>> What is your sessionlog setting on each provider for the syncprov
>> overlay?
> [Rodrigo]
> syncprov-checkpoint 10000 120
> syncprov-sessionlog 100000
Hm, I would probably checkpoint the cookie a lot more frequently than you
have it set to. The sessionlog setting seems fine to me.
> Same configuration in both systems.
>>
>>> For definition purposes I have some memory limitations where I need to
>>> limit dncachesize for around 80% of DB entrances.
>>
>> We already went through other things you could do to reduce your
>> memory footprint in other ways. You've completely ignored that
>> advice. As long as your dncachesize is in this state, I don't expect
>> things to behave normally.
> [Rodrigo]I implemented what was possible. The end is this cache config
> possible by the memory constraints :
># Cache values
># cachesize 10000
> cachesize 20000
> dncachesize 3000000
># dncachesize 400000
># idlcachesize 10000
> idlcachesize 30000
># cachefree 10
> cachefree 100
You don't say anything in here about your DB_CONFIG settings, which is
where you could stand to gain the most amount of memory back. I do see
you're definitely running a very restricted cachesize/idlcachesize. ;)
>> What value did you set for "cachefree"?
> [Rodrigo] cachefree 100
This value is likely substantially way too low for your system
configuration. This is how many entries get freed from any of the caches.
With your dncachesize being 3,000,000, removing 100 entries from it will do
hardly anything, and may be part of the issue. If it wasn't for the major
imbalance between your entry, idl, and dncachesizes, I would suggest a
fairly high value like 100,000. But given your entry cache is 20,000,
you'll probably have to limit the cachefree to 5000-10000. But it is going
to need to be higher than 100.
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
14 years, 1 month
Re: (ITS#6277) Missing documentation for cn=config
by Gavin Henry
> Full_Name: Ryan Steele
> Version: 2.4.15
> OS: Ubuntu 8.04 LTD
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (207.106.239.81)
>
>
> According to chapter 5 of the admin guide, "some of the backends and of the
> distributed overlays do not support runtime configuration yet. In those
cases,
> the old style slapd.conf(5) file must be used." However, there is no
> documentation on which backends and overlays do and don't have said support.
> Without grokking the code, it is a trial-and-error operation at best.
Point taken. I'm pretty sure they all do now. Will check and update that
section.
> Also, there is no documentation on how to add overlay-specific directives
(man
> pages or otherwise). Take, for example, autogroup-attrset; the olcAGattrSet
> directive is only described in autogroup.c, and nowhere else. This is not
the
> only instance of missing module documentation, but it should give a general
idea
> of where to look.
Some of the contribs one don't come with a man page unfortunately. Core ones
do.
I'll look into the relevant sections and add one or two examples.
> Of course, grepping the code is easy enough, but you shouldn't have to do
that
> to learn how to achieve simple configurations. If there's going to be a
> fundamental paradigm shift from slapd.conf to cn=config, there has to be
> documentation (man pages, admin guide sections, et. al.) on the appropriate
> methods for achieving what once was done through slapd.conf. IMHO, that
should
> include concrete examples, such as the following, pulled from the Courier
> documentation:
>
> The following LDIF could be used to add [the auditlog] overlay to cn=config
> (adjust to suit):
>
> dn: olcOverlay=auditlog,olcDatabase={1}hdb,cn=config
> changetype: add
> objectClass: olcOverlayConfig
> objectClass: olcAuditLogConfig
> olcOverlay: auditlog
> olcAuditlogFile: /tmp/auditlog.ldif
>
Well there are two places that talk about how to convert from slapd.conf to
cn=config formats. In the guide and man pages, so that is the best way to do a
full conversion and see the end result.
Where would you like to see these added?
Thanks for the feedback as always!
Gavin.
14 years, 1 month
(ITS#6277) Missing documentation for cn=config
by ryans＠aweber.com
Full_Name: Ryan Steele
Version: 2.4.15
OS: Ubuntu 8.04 LTD
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (207.106.239.81)
According to chapter 5 of the admin guide, "some of the backends and of the
distributed overlays do not support runtime configuration yet. In those cases,
the old style slapd.conf(5) file must be used." However, there is no
documentation on which backends and overlays do and don't have said support.
Without grokking the code, it is a trial-and-error operation at best.
Also, there is no documentation on how to add overlay-specific directives (man
pages or otherwise). Take, for example, autogroup-attrset; the olcAGattrSet
directive is only described in autogroup.c, and nowhere else. This is not the
only instance of missing module documentation, but it should give a general idea
of where to look.
Of course, grepping the code is easy enough, but you shouldn't have to do that
to learn how to achieve simple configurations. If there's going to be a
fundamental paradigm shift from slapd.conf to cn=config, there has to be
documentation (man pages, admin guide sections, et. al.) on the appropriate
methods for achieving what once was done through slapd.conf. IMHO, that should
include concrete examples, such as the following, pulled from the Courier
documentation:
The following LDIF could be used to add [the auditlog] overlay to cn=config
(adjust to suit):
dn: olcOverlay=auditlog,olcDatabase={1}hdb,cn=config
changetype: add
objectClass: olcOverlayConfig
objectClass: olcAuditLogConfig
olcOverlay: auditlog
olcAuditlogFile: /tmp/auditlog.ldif
14 years, 1 month
Re: (ITS#6275) syncrepl taking long(not sync) when consumer not connect for a moment
by quanah＠zimbra.com
--On Thursday, August 27, 2009 6:39 AM -0700 Rodrigo Costa
<rlvcosta(a)yahoo.com> wrote:
> Quanah,
>
> Please see answer in your previous e-mail below.
>
> I'm also sending the information I could collect attached since it is a
> small file(5KB).
The ITS system does not accept attachments. You'll need to upload it to
the FTP site, which had space cleared up.
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
14 years, 1 month
