openldap-bugs

openldap-bugs@openldap.org

1 participants
20961 discussions

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by hyc＠symas.com 22 Jan '09

22 Jan '09

rlvcosta(a)yahoo.com wrote: > See that I have 2 DBs and in each DB area in slapd.conf, read without error by slapd as seen above, I have the cachesize, idlcachesize and dncachesize specified with small values just to monitor the memory usage by slapd. > > Unfortunately even with this configuration I still having slapd consuming memory without release it. So if a new entrance is queried by some LDAP client, slapd will consume more memory. Since my DB dn's are bigger than 3GB then soon or later the slapd will crash. > > See my DB information : > > 7.2G dn2id.bdb > 12G id2entry.bdb > 110M maillogin.bdb > 3.3M objectClass.bdb > 108M pnnumber.bdb > 1.3M submxid.bdb > 323M uid.bdb > > See my dn2id is much bigger than yours and maybe you just did not saw this issue happenig since your dn can be loaded in your memory and you in a 64 bits environment can use more than 3GB per process. > > But in any case looks like there is a problem in slapd that do not respect the cache limitations imposed. > > In this way we cannot use slapd for large databases. Please see my configuration above and let me know if a made a mistake(I hope so) since dncachesize is not documented at man pages. You're mistaken, dncachesize is in the 2.4 manpages. You seem to not be reading the documentation that came with the software release you're using. > Just as example, with the DB I loaded for tests with 1 million entrances, with the cache sizes definitions as in the slapd.conf described above and making a ldapsearch on all entrances from the CONTENT DB, in the end I have : > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 3189 ldap 15 0 842m 724m 67m S 99 6.1 7:00.48 slapd > > See the slapd process already consumed 842m and I just read 1 million entrances, even dncachesize is defined to be 1000. > > I do not see anyway to control the memory usage by slapd. > > Is this "disrespect" to slapd.conf directives a possible problem? Looks like slapd is behaving normally so far, though it's hard to tell without also seeing your DB_CONFIG settings. Test again with your full database and report the results. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: Re (ITS#5843) N-way Multi-master doesn't replicate deletes under certain circumstances.
by ghenry＠OpenLDAP.org 22 Jan '09

22 Jan '09

> So I guess this was not a bug, but a mistake in the documentation : > http://www.openldap.org/doc/admin24/replication.html#N-Way%20Multi-Master > > Can anyone confirm ? Yes, you are correct, apologies. Ando fixed this in test050: ----------- 1.14 Sun Nov 16 22:06:30 2008 UTC; 2 months ago by ando Changed since 1.13: +28 -10 lines Diffs to 1.13 (colored diff) add indexes when supported; syncrepl on configuration should always be refreshAndPersist ----------- and I missed it for updating the docs. The update to the guide will be in the next release, 2.4.14 I'll close this ITS now. Thanks! -- Kind Regards, Gavin Henry. OpenLDAP Engineering Team. E ghenry(a)OpenLDAP.org Community developed LDAP software. http://www.openldap.org/project/

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by rlvcosta＠yahoo.com 22 Jan '09

22 Jan '09

Quanah, I follow your recommendation and compiled the Berkeley Db 4.7.25NC and openldap 2.4.13. After that load my LDIF file to have 1million entraces in my DB just to test the dncachesize control. During my initialization I have : module back_bdb.la: null module registered line 29 (moduleload back_monitor.la) loaded module back_monitor.la module back_monitor.la: null module registered line 74 (database bdb) line 75 (suffix "ou=CONTENT,o=alcatel,c=fr") line 76 (rootdn "cn=admin,ou=CONTENT,o=alcatel,c=fr") line 80 (rootpw ***) line 86 (directory /var/openldap-data/bdb1) line 89 (index userid,pnnumber,submxid,maillogin,abookaliasid eq,pres) index uid 0x0006 index pnnumber 0x0006 index submxid 0x0006 index maillogin 0x0006 index abookaliasid 0x0006 line 90 (index objectClass eq) index objectClass 0x0004 line 93 (cachesize 1000) line 94 (idlcachesize 1000) line 95 (dncachesize 1000) line 105 (database bdb) line 106 (suffix "ou=INDEXES,o=alcatel,c=fr") line 107 (rootdn "cn=admin,ou=INDEXES,o=alcatel,c=fr") line 111 (rootpw ***) line 114 (directory /var/openldap-data/bdb2) line 117 (index weblogin,maillogin,mail,pnnumber eq,pres) index weblogin 0x0006 index maillogin 0x0006 index mail 0x0006 index pnnumber 0x0006 line 118 (index profileid,loginname eq,pres) index profileid 0x0006 index loginname 0x0006 line 119 (index objectClass eq) index objectClass 0x0004 line 122 (cachesize 1000) line 123 (idlcachesize 1000) line 124 (dncachesize 1000) line 129 (database monitor) slapd starting See that I have 2 DBs and in each DB area in slapd.conf, read without error by slapd as seen above, I have the cachesize, idlcachesize and dncachesize specified with small values just to monitor the memory usage by slapd. Unfortunately even with this configuration I still having slapd consuming memory without release it. So if a new entrance is queried by some LDAP client, slapd will consume more memory. Since my DB dn's are bigger than 3GB then soon or later the slapd will crash. See my DB information : 7.2G dn2id.bdb 12G id2entry.bdb 110M maillogin.bdb 3.3M objectClass.bdb 108M pnnumber.bdb 1.3M submxid.bdb 323M uid.bdb See my dn2id is much bigger than yours and maybe you just did not saw this issue happenig since your dn can be loaded in your memory and you in a 64 bits environment can use more than 3GB per process. But in any case looks like there is a problem in slapd that do not respect the cache limitations imposed. In this way we cannot use slapd for large databases. Please see my configuration above and let me know if a made a mistake(I hope so) since dncachesize is not documented at man pages. Just as example, with the DB I loaded for tests with 1 million entrances, with the cache sizes definitions as in the slapd.conf described above and making a ldapsearch on all entrances from the CONTENT DB, in the end I have : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3189 ldap 15 0 842m 724m 67m S 99 6.1 7:00.48 slapd See the slapd process already consumed 842m and I just read 1 million entrances, even dncachesize is defined to be 1000. I do not see anyway to control the memory usage by slapd. Is this "disrespect" to slapd.conf directives a possible problem? I hope there is a configuration problem with my slapd.conf where this could be work around. Regards, Rodrigo. --- On Wed, 1/21/09, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > From: Quanah Gibson-Mount <quanah(a)zimbra.com> > Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 > To: rlvcosta(a)yahoo.com, openldap-its(a)openldap.org > Date: Wednesday, January 21, 2009, 8:30 PM > --On Wednesday, January 21, 2009 12:49 PM -0800 Rodrigo > Costa <rlvcosta(a)yahoo.com> wrote: > > > Quanah, > > > > That's is exactly the point. My system is a 32 > bits and with 64 bits if > > there are enough physical memory available even slapd > cache grows without > > never release it will be difficult(or very long) to > cache enough > > entrances before system consumes all memory. > > > > With the 32 bits 3GB maximum, depend on OS too, this > issue appears more > > often. My impression is if you ldapsearch all DB in a > 32 bits HW/OS slapd > > will consume all possible memory unless to keep > boundaries for its > > cache(erase and overwrite). > > As I noted, in 2.3, you cannot constrain the DN cachesize. > You can do this with OL 2.4. You need to try a current > RE2.4 release, and set the dncachesize. There was no dn > cache at all in OL 2.1. > > --Quanah > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and > collaboration

1 0

Re (ITS#5843) N-way Multi-master doesn't replicate deletes under certain circumstances.
by adrien.futschik＠atosorigin.com 22 Jan '09

22 Jan '09

I Think I found the problem. I have to confirm this, but the "olcSpSessionlog" parameter doesn't seem to be necessary. I made a mistake in my configuration. I used "refreshOnly" mode for the backend and with "refreshOnly", consumer isn't aware when a provider deletes an entry. So I understood at least. Here is a extract from the documentation about syncrepl : "Also as a consequence of the search filter used in the syncrepl specification, it is possible for a modification to remove an entry from the replication scope even though the entry has not been deleted on the provider. Logically the entry must be deleted on the consumer but in refreshOnly mode the provider cannot detect and propagate this change without the use of the session log." Therefore I changed "refreshOnly" for "refreshAndPersist", I removed the "olcSpSessionlog" parameter, and everything seems to be working fine. At least, deletes are correctly replicated to the second master. Here is the final configuration I am using : m1 -config : dn: cn=config objectClass: olcGlobal cn: config olcServerID: 1 dn: olcDatabase={0}config,cn=config objectClass: olcDatabaseConfig olcDatabase: {0}config olcRootPW:< file://$CONFIGPWF m2 - config : dn: cn=config objectClass: olcGlobal cn: config olcServerID: 2 dn: olcDatabase={0}config,cn=config objectClass: olcDatabaseConfig olcDatabase: {0}config olcRootPW:< file://$CONFIGPWF m1 - syncprov : dn: cn=config changetype: modify replace: olcServerID olcServerID: 1 $URI1 olcServerID: 2 $URI2 dn: olcOverlay=syncprov,olcDatabase={0}config,cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: syncprov dn: olcDatabase={0}config,cn=config changetype: modify add: olcSyncRepl olcSyncRepl: rid=001 provider=$URI1 binddn="cn=config" bindmethod=simple credentials=$CONFIGPW searchbase="cn=config" type=refreshAndPersist retry="5 5 300 5" timeout=3 olcSyncRepl: rid=002 provider=$URI2 binddn="cn=config" bindmethod=simple credentials=$CONFIGPW searchbase="cn=config" type=refreshAndPersist retry="5 5 300 5" timeout=3 - add: olcMirrorMode olcMirrorMode: TRUE m2 - syncrepl : dn: olcDatabase={0}config,cn=config changetype: modify add: olcSyncRepl olcSyncRepl: rid=001 provider=$URI1 binddn="cn=config" bindmethod=simple credentials=$CONFIGPW searchbase="cn=config" type=refreshAndPersist retry="5 5 300 5" timeout=3 olcSyncRepl: rid=002 provider=$URI2 binddn="cn=config" bindmethod=simple credentials=$CONFIGPW searchbase="cn=config" type=refreshAndPersist retry="5 5 300 5" timeout=3 - add: olcMirrorMode olcMirrorMode: TRUE m1 - schema : include: file://$ABS_SCHEMADIR/core.ldif include: file://$ABS_SCHEMADIR/cosine.ldif include: file://$ABS_SCHEMADIR/inetorgperson.ldif include: file://$ABS_SCHEMADIR/openldap.ldif include: file://$ABS_SCHEMADIR/nis.ldif m1 - backend : dn: olcDatabase={1}$BACKEND,cn=config objectClass: olcDatabaseConfig objectClass: olc${BACKEND}Config olcDatabase: {1}$BACKEND olcSuffix: $BASEDN olcDbDirectory: ./openldap-data olcRootDN: $MANAGERDN olcRootPW: $PASSWD olcSyncRepl: rid=004 provider=$URI1 binddn="$MANAGERDN" bindmethod=simple credentials=$PASSWD searchbase="$BASEDN" type=refreshAndPersist interval=$INTERVAL retry="5 5 300 5" timeout=3 olcSyncRepl: rid=005 provider=$URI2 binddn="$MANAGERDN" bindmethod=simple credentials=$PASSWD searchbase="$BASEDN" type=refreshAndPersist interval=$INTERVAL retry="5 5 300 5" timeout=3 olcMirrorMode: TRUE dn: olcOverlay=syncprov,olcDatabase={1}${BACKEND},cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: syncprov So I guess this was not a bug, but a mistake in the documentation : http://www.openldap.org/doc/admin24/replication.html#N-Way%20Multi-Master Can anyone confirm ? Adrien Futschik

1 0

Re: (ITS#5883) [PATCH] syncrepl test enhancements
by hyc＠symas.com 22 Jan '09

22 Jan '09

richton(a)nbcs.rutgers.edu wrote: > Full_Name: Aaron Richton > Version: HEAD > OS: irrelevant > URL: https://www.nbcs.rutgers.edu/~richton/syncrepl-modrdn-tests.patch > Submission from: (NULL) (67.85.181.229) > > > Per http://www.openldap.org/lists/openldap-devel/200812/msg00004.html these > patches enhance tests to detect ITS#5809-like behavior. > > The actual test changes are at the address linked to this report. They should > apply clean to HEAD and can be used immediately with, for example, back-ldif. I > note that I wasn't able to reproduce a failure under test018; if anybody can > suggest a different test there, that'd be great, or perhaps the stopping and > starting somehow masks the condition? > > While verifying the tests, I noticed that some backends (such as back-bdb) were > failing because the consumer and provider returned the attributes in a different > order. Since attribute ordering isn't guaranteed, I viewed this as a test > failure rather than a slapd failure, and used the back-ndb gawk acfilter.sh to > work around this. In the end I just made it apply to all backends: Since we want to minimize the dependency on gawk, I took a look at the ordering failure. It occurs because the syncrepl consumer internally processes renames before it processes modifies. As such, just changing the test script to execute in this order seems to avoid the problem. > > https://www.nbcs.rutgers.edu/~richton/acfilter.patch > > but we'll probably want to discuss this on -devel: it adds a gawk dependency to > the tests. > > With the acfilter.sh sorting enabled, I found three tests that don't filter the > original data prior to ldapsearch comparison. I added these filters in: > > https://www.nbcs.rutgers.edu/~richton/openldap-testflt.patch Sounds like it should be harmless to include this patch... > > We all know that nothing's free -- a couple extra commands in the name of bug > coverage is probably worth it -- but the gawk dependency may be troublesome. > Anyway, do what you will...I consider these pretty trivial, so: > > These patch files are derived from OpenLDAP Software. All of the modifications > to OpenLDAP Software represented in these patches were developed by Aaron > Richton<richton(a)nbcs.rutgers.edu>. I have not assigned rights and/or interest > in this work to any party. > > I, Aaron Richton, hereby place these modifications to OpenLDAP Software (and > only these modifications) into the public domain. Hence, these modifications may > be freely used and/or redistributed for any purpose with or without attribution > and/or other notice. > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#5901) modrdn syncing busted
by quanah＠zimbra.com 22 Jan '09

22 Jan '09

Full_Name: Quanah Gibson-Mount Version: 2.4.13 OS: NA URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (75.111.29.239) As noted in a post to the bugs list: Hello, I have configured a central OpenLDAP, using Debian Etch, version: slapd 2.3.30-5+etch1, which is also a sync provider. Another Debian Etch box uses the sync: syncrepl rid=123 provider=ldap://example.com type=refreshAndPersist retry="5,2,30,2,60,+" interval=00:01:00:00 searchbase="basedn" filter="(objectClass=*)" scope=sub attrs="*,+" sizelimit=0 timelimit=0 schemachecking=on bindmethod=simple binddn="someDN" credentials="somePwd" starttls=critical This works fine, but moddn fails: for testing purpose I ldapadd : dn: c=testtt,ou=basedn c: testtt objectClass: country then: ldapmodrdn -r 'c=testtt,ou=basedn' 'c=testww' It works on the sync provider (schemacheck on) and the result is: dn: c=testww,ou=basedn objectClass: country c: testww But on the sync consumer: : syncrepl_entry: c=testww,ou=basedn : Entry (c=testww,ou=basedn attribute 'c' cannot have multiple values : entry failed schema check: attribute 'c' cannot have multiple values : null_callback : error code 0x13 : syncrepl_entry: be_modrdn (19) The testtt entry is still present on the secondary box. If I delete "testww" on the sync provider, the "testt" entry in the consumere is deleted. As noted by Howard in follow up: Actually... this would have worked correctly in 2.4.11 and 2.4.12. In 2.4.13 I reverted back to the 2.3 behavior due to other concerns, so this will probably fail again. And the test script does a rename on an entry whose distinguished attribute is multivalued, so it doesn't catch this case. I think we'll have to revisit this patch for 2.4.14 and add an additional rename test to test017 and test018.

1 0

Re: (ITS#5701) connection.c asserting during test008
by quanah＠zimbra.com 21 Jan '09

21 Jan '09

--On Saturday, September 20, 2008 8:03 PM +0000 richton(a)nbcs.rutgers.edu wrote: > On Sat, 20 Sep 2008, Pierangelo Masarati wrote: > Connection code has been reworked in current RE24 and may have resolved this issue. Can you please test? Thanks, Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5880) back-bdb segfault
by quanah＠zimbra.com 21 Jan '09

21 Jan '09

--On Saturday, January 17, 2009 3:13 AM +0000 quanah(a)zimbra.com wrote: > I'm just warning that current RE24 is not expected to be at all stable. In fact, I note that both your reports are against RE24 CVS checkouts, and not a release. Please reproduce in a release (I expect it will still occur there), rather than unstable CVS. Thanks, Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by quanah＠zimbra.com 21 Jan '09

21 Jan '09

--On Wednesday, January 21, 2009 12:49 PM -0800 Rodrigo Costa <rlvcosta(a)yahoo.com> wrote: > Quanah, > > That's is exactly the point. My system is a 32 bits and with 64 bits if > there are enough physical memory available even slapd cache grows without > never release it will be difficult(or very long) to cache enough > entrances before system consumes all memory. > > With the 32 bits 3GB maximum, depend on OS too, this issue appears more > often. My impression is if you ldapsearch all DB in a 32 bits HW/OS slapd > will consume all possible memory unless to keep boundaries for its > cache(erase and overwrite). As I noted, in 2.3, you cannot constrain the DN cachesize. You can do this with OL 2.4. You need to try a current RE2.4 release, and set the dncachesize. There was no dn cache at all in OL 2.1. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by rlvcosta＠yahoo.com 21 Jan '09

21 Jan '09

Quanah, That's is exactly the point. My system is a 32 bits and with 64 bits if there are enough physical memory available even slapd cache grows without never release it will be difficult(or very long) to cache enough entrances before system consumes all memory. With the 32 bits 3GB maximum, depend on OS too, this issue appears more often. My impression is if you ldapsearch all DB in a 32 bits HW/OS slapd will consume all possible memory unless to keep boundaries for its cache(erase and overwrite). In 64 bits with a lot of memory this will be more difficult to happen but eventually will happen if the logic is the same. I was wondering if in a sittuation where this cache memory issue would appear, like ldapsearch, if there is a feature in openldap where it would guarantee the cache boundaries and the reutilization of the space after the limits are reached. Looks like even with the slapd.conf configurations it doesn't keeps its limits. Regards, Rodrigo. --- On Wed, 1/21/09, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > From: Quanah Gibson-Mount <quanah(a)zimbra.com> > Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 > To: rlvcosta(a)yahoo.com, openldap-its(a)openldap.org > Date: Wednesday, January 21, 2009, 6:41 PM > --On Wednesday, January 21, 2009 12:23 PM -0800 Rodrigo > Costa <rlvcosta(a)yahoo.com> wrote: > > > Quanah, > > > > My initial attempt was using openldap2.4.11 since this > is the stable > > version. At this version Berkeley DB 4.7 isn't > supported and then I used > > the BDB 4.6 with all patches. > > I hardly consider 2.4.11 stable, regardless of what the tag > is. I would advise trying something more current (such as > 2.4.13). And with BDB 4.7 + patches. > > > At this configuration, having the DB_CONFIG setup for > 50MB, and even > > having the idlcache and cache sizes defined I continue > to see the memory > > being allocated and never released by slapd. > > What about dncachesize as well? > > > 2.3.27-8.el5_2.4 where the CentOS5.4 uses a built-in > backend library for > > BDB4.4. > > If you are looking for stability, I would hardly go with > the crappy release CentOS ships. The final 2.3 release was > 2.3.43. Use it instead if you want to use 2.3. > > Also, are you on a 32 or 64-bit system? I certainly have > no problems going over 3GB on my 64-bit systems. > > For example: > [zimbra@ldap openldap-data]$ du -c -h *.bdb > 1.1G cn.bdb > 564M displayName.bdb > 702M dn2id.bdb > 158M entryCSN.bdb > 122M entryUUID.bdb > 335M givenName.bdb > 6.5G id2entry.bdb > 1.1G mail.bdb > 5.5M objectClass.bdb > 855M sn.bdb > 121M uid.bdb > 8.0K zimbraDomainName.bdb > 122M zimbraId.bdb > 20K zimbraMailAlias.bdb > 1.1G zimbraMailDeliveryAddress.bdb > 2.1M zimbraMailForwardingAddress.bdb > 3.1M zimbraMailTransport.bdb > 8.0K zimbraVirtualHostname.bdb > 8.0K zimbraVirtualIPAddress.bdb > 13G total > > > DB is 13GB in size. > > [zimbra@ldap openldap-data]$ cat DB_CONFIG > set_cachesize 8 0 2 > set_lg_regionmax 262144 > set_lg_bsize 2097152 > set_lk_max_locks 1500 > set_flags DB_LOG_AUTOREMOVE > set_lg_dir /var/log/zimbra-ldap-journals > > Process size: > > 8447 zimbra 17 0 9.9g 8.9g 8.1g S 34 57.5 39:20.57 > slapd > > > Using OpenLDAP 2.3.42. > > > --Quanah > > > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and > collaboration

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs