openldap-bugs

openldap-bugs@openldap.org

1 participants
20961 discussions

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by quanah＠zimbra.com 21 Jan '09

21 Jan '09

--On Wednesday, January 21, 2009 12:23 PM -0800 Rodrigo Costa <rlvcosta(a)yahoo.com> wrote: > Quanah, > > My initial attempt was using openldap2.4.11 since this is the stable > version. At this version Berkeley DB 4.7 isn't supported and then I used > the BDB 4.6 with all patches. I hardly consider 2.4.11 stable, regardless of what the tag is. I would advise trying something more current (such as 2.4.13). And with BDB 4.7 + patches. > At this configuration, having the DB_CONFIG setup for 50MB, and even > having the idlcache and cache sizes defined I continue to see the memory > being allocated and never released by slapd. What about dncachesize as well? > 2.3.27-8.el5_2.4 where the CentOS5.4 uses a built-in backend library for > BDB4.4. If you are looking for stability, I would hardly go with the crappy release CentOS ships. The final 2.3 release was 2.3.43. Use it instead if you want to use 2.3. Also, are you on a 32 or 64-bit system? I certainly have no problems going over 3GB on my 64-bit systems. For example: [zimbra@ldap openldap-data]$ du -c -h *.bdb 1.1G cn.bdb 564M displayName.bdb 702M dn2id.bdb 158M entryCSN.bdb 122M entryUUID.bdb 335M givenName.bdb 6.5G id2entry.bdb 1.1G mail.bdb 5.5M objectClass.bdb 855M sn.bdb 121M uid.bdb 8.0K zimbraDomainName.bdb 122M zimbraId.bdb 20K zimbraMailAlias.bdb 1.1G zimbraMailDeliveryAddress.bdb 2.1M zimbraMailForwardingAddress.bdb 3.1M zimbraMailTransport.bdb 8.0K zimbraVirtualHostname.bdb 8.0K zimbraVirtualIPAddress.bdb 13G total DB is 13GB in size. [zimbra@ldap openldap-data]$ cat DB_CONFIG set_cachesize 8 0 2 set_lg_regionmax 262144 set_lg_bsize 2097152 set_lk_max_locks 1500 set_flags DB_LOG_AUTOREMOVE set_lg_dir /var/log/zimbra-ldap-journals Process size: 8447 zimbra 17 0 9.9g 8.9g 8.1g S 34 57.5 39:20.57 slapd Using OpenLDAP 2.3.42. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by rlvcosta＠yahoo.com 21 Jan '09

21 Jan '09

Quanah, My initial attempt was using openldap2.4.11 since this is the stable version. At this version Berkeley DB 4.7 isn't supported and then I used the BDB 4.6 with all patches. At this configuration, having the DB_CONFIG setup for 50MB, and even having the idlcache and cache sizes defined I continue to see the memory being allocated and never released by slapd. Now I tried to evolve my OS to CentOS5.2 and then see if it could be an issue with my compilation using a community compiled openldap version that could have syncrepl available. Based in my issues I'm not using syncrepl at this time since I'm trying to check a stable sittuation for my DB. What I'm observing is that under openldap version 2.3 and 2.4 if I have a DB, at least with BDB, that could be searched and generate indexes bigger than 3GB(max OS memory space per process), the search will make slapd load all indexes in memory and never release them. This make that openldap would not be used for DBs which size would become bigger than 3GB. What would be the major part of cases. Once a DB entrance is searched, it is loaded into memory and never more released. So, for example, if I made several binds to the same entrance in DB, slapd never allocates more memory and uses the already cached entrance. But if I search for different entrances in DB, slapd always cache in memory and continues to do it until it reaches the 3GB barrier and then process cannot allocate more memory. slapd never release cache memory, respect the default idlcachesize and cachesize that is 1000, or even release cache memory. So if there is not ldapsearch for a backup, that could speed this behavior, with the time if the queries are done randomly at certain time the cache will become of size 3GB and no more memory can be allocated. My actual version now is: 2.3.27-8.el5_2.4 where the CentOS5.4 uses a built-in backend library for BDB4.4. My DB_CONFIG is(without comments) : set_cachesize 0 52428800 1 set_lg_regionmax 1048576 set_lg_max 10485760 set_lg_bsize 2097152 set_tmp_dir /var/tmp And the slapd.conf I included(the indexes always existed) : # The database directory MUST exist prior to running slapd AND # should only be accessible by the slapd and slap tools. # Mode 700 recommended. directory /var/openldap-data/bdb1 # Indices to maintain for this database index userid,pnnumber,submxid,maillogin,abookaliasid eq,pres index objectClass eq #Cache values cachesize 1000 idlcachesize 1000 This issue, by other ITSs, appear to exist before. Not sure if this is really a configuration problem. Running slapd with -d 64 showed the cachesize and idlcachesize did not return any complain. Please let me know if you have some idea about this problem. I will try to use the not yet stable 2.4.13 but I would like to have your feedback if this could really be caused by other problem or even if in openldap 2.4.13 there are some detection/solution for this kind of issue. Regards, Rodrigo. --- On Wed, 1/21/09, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > From: Quanah Gibson-Mount <quanah(a)zimbra.com> > Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 > To: rlvcosta(a)yahoo.com, openldap-its(a)openldap.org > Date: Wednesday, January 21, 2009, 5:57 PM > --On Tuesday, January 20, 2009 12:07 AM +0000 > rlvcosta(a)yahoo.com wrote: > > > Pierangelo, > > > > Executing some test I'm not sure anymore if this > is really a memory > > leakage or if some cache issue. > > > > I decided to install CentOS5.2 where I could have > OpenLdap 2.3.27 > > compiled by the distribution and then execute more > tests to check if some > > library in RHEL4 would be causing some problem. > > > > I then use a similar slapd.conf as in my original > e-mail just without > > explicitly define an size for cache and idlcache > there. My DB_CONFIG file > > has a cache defined as 50MB what is respected. > > > > Looking at some other ITS I found the ITS#3938 what I > believe is very > > similar to my problem. > > > > I still have a DB with 3882998 entrances and where the > DB files at disk > > is around 15GB. My machine has 12GB of memory where > for sure all the DB > > cannot be cached into memory. > > > > In any case what is happening is that every time a > transaction is done > > with OpenLdap, even a distribution formal compilation, > the cache > > continues to grow until consumes all memory. If the DB > is bigger than > > machine memory then soon or later a memory out of > range will happen. > > > > The cache is never release by OpenLdap and then if a > sequential search is > > done and since DB is bigger than memory we can speed > up the problem. > > I would suggest you use OpenLDAP 2.4, and set the > slapd.conf cachesize, > idlcachesize, and dncachesize. I would also review your > settings for > DB_CONFIG, and use a newer BDB version with OpenLDAP 2.4 > (i.e., BDB 4.7 > with all 3 patches). See if that helps resolve the issue > you face. In > OpenLDAP 2.3, there is no way to control the dncachesize. > > --Quanah > > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and > collaboration

1 0

Re: (ITS#5860) slapd memeory leak under openldap 2.4
by quanah＠zimbra.com 21 Jan '09

21 Jan '09

--On Tuesday, January 20, 2009 12:07 AM +0000 rlvcosta(a)yahoo.com wrote: > Pierangelo, > > Executing some test I'm not sure anymore if this is really a memory > leakage or if some cache issue. > > I decided to install CentOS5.2 where I could have OpenLdap 2.3.27 > compiled by the distribution and then execute more tests to check if some > library in RHEL4 would be causing some problem. > > I then use a similar slapd.conf as in my original e-mail just without > explicitly define an size for cache and idlcache there. My DB_CONFIG file > has a cache defined as 50MB what is respected. > > Looking at some other ITS I found the ITS#3938 what I believe is very > similar to my problem. > > I still have a DB with 3882998 entrances and where the DB files at disk > is around 15GB. My machine has 12GB of memory where for sure all the DB > cannot be cached into memory. > > In any case what is happening is that every time a transaction is done > with OpenLdap, even a distribution formal compilation, the cache > continues to grow until consumes all memory. If the DB is bigger than > machine memory then soon or later a memory out of range will happen. > > The cache is never release by OpenLdap and then if a sequential search is > done and since DB is bigger than memory we can speed up the problem. I would suggest you use OpenLDAP 2.4, and set the slapd.conf cachesize, idlcachesize, and dncachesize. I would also review your settings for DB_CONFIG, and use a newer BDB version with OpenLDAP 2.4 (i.e., BDB 4.7 with all 3 patches). See if that helps resolve the issue you face. In OpenLDAP 2.3, there is no way to control the dncachesize. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5900) Problem to populate LDAP 2.4.12 with an Ldiff file on SUNBLADE
by quanah＠zimbra.com 21 Jan '09

21 Jan '09

--On Wednesday, January 21, 2009 6:41 PM +0000 fethi.tekitek(a)alcatel-lucent.com wrote: > Full_Name: Fathy Tekitek > Version: 2.4.12 > OS: Solaris 10 (rev 8) > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (62.23.212.109) > > > Hi, > I try to populate the openldap 2.4.12 running with BerkeleyDB 4.7.25 > using an ldiff file > The first two entry are populate but It remains blocked > This problem is reproducible on SunBlade 100 servers but not in T5000, > SunBlade2500 and Ultra45 where it works OK. > I can enter the content of this file one bye one but all in the ldiff file > "/export/home/build26/openldap/bin/ldapadd -d 1 -H "ldaps://sunval233 " > -x -D "cn=root,dc=alu,dc=com" -f > /export/home/build26/CSA/data/installation/ngsecPopulate.ldif -w newuser1" Usage questions should generally be directed to openldap-software(a)openldap.org. I will note that (a) There are 3 patches now for BDB 4.7, make sure you are using them all, and (b) if your database is stored in NFS, that is not supported. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5894) test039 fails: ldap_back_conn_delete: Assertion
by quanah＠zimbra.com 21 Jan '09

21 Jan '09

--On Wednesday, January 21, 2009 2:35 PM +0100 Michael Ströder <michael(a)stroeder.com> wrote: > I'm using libdb-4_5-4.5.20-94.17 from openSUSE 11.1 which is AFAIK fully > patched. > > The interesting fact is that HEAD currently hangs all the time in > test008. RE24 with your recent changes now fails as described in > test039. Before these changes RE24 just worked fine. > > I think this is related to modifications in connection.c because trouble > started after HEAD contained these changes. All tests pass for me with current RE24 and HEAD. What Linux kernel are you using? The RE24 changes to the connection code are the same as what are in HEAD. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

(ITS#5900) Problem to populate LDAP 2.4.12 with an Ldiff file on SUNBLADE
by fethi.tekitek＠alcatel-lucent.com 21 Jan '09

21 Jan '09

Full_Name: Fathy Tekitek Version: 2.4.12 OS: Solaris 10 (rev 8) URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (62.23.212.109) Hi, I try to populate the openldap 2.4.12 running with BerkeleyDB 4.7.25 using an ldiff file The first two entry are populate but It remains blocked This problem is reproducible on SunBlade 100 servers but not in T5000, SunBlade2500 and Ultra45 where it works OK. I can enter the content of this file one bye one but all in the ldiff file "/export/home/build26/openldap/bin/ldapadd -d 1 -H "ldaps://sunval233 " -x -D "cn=root,dc=alu,dc=com" -f /export/home/build26/CSA/data/installation/ngsecPopulate.ldif -w newuser1" here the ldap traces ldap_url_parse_ext(ldaps://sunval233) ldap_create ldap_url_parse_ext(ldaps://sunval233:636/??base) ldap_sasl_bind ldap_send_initial_request ldap_new_connection 1 1 0 ldap_int_open_connection ldap_connect_to_host: TCP sunval233:636 ldap_new_socket: 4 ldap_prepare_socket: 4 ldap_connect_to_host: Trying 10.0.0.233:636 ldap_pvt_connect: fd: 4 tm: -1 async: 0 TLS trace: SSL_connect:before/connect initialization TLS trace: SSL_connect:SSLv2/v3 write client hello A TLS trace: SSL_connect:SSLv3 read server hello A TLS certificate verification: depth: 1, err: 0, subject: /C=ES/ST=Madrid/L=Madrid/O=Alcatel-Lucent/OU=NGSEC, issuer: /C=ES/ST=Madrid/L=Madrid/O=Alcatel-Lucent/OU=NGSEC TLS certificate verification: depth: 0, err: 0, subject: /C=ES/ST=Madrid/L=Madrid/O=Alcatel-Lucent/OU=NGSEC/CN=sunval233, issuer: /C=ES/ST=Madrid/L=Madrid/O=Alcatel-Lucent/OU=NGSEC TLS trace: SSL_connect:SSLv3 read server certificate A TLS trace: SSL_connect:SSLv3 read server certificate request A TLS trace: SSL_connect:SSLv3 read server done A TLS trace: SSL_connect:SSLv3 write client certificate A TLS trace: SSL_connect:SSLv3 write client key exchange A TLS trace: SSL_connect:SSLv3 write change cipher spec A TLS trace: SSL_connect:SSLv3 write finished A TLS trace: SSL_connect:SSLv3 flush data TLS trace: SSL_connect:SSLv3 read finished A ldap_open_defconn: successful ldap_send_server_request ber_scanf fmt ({it) ber: ber_scanf fmt ({i) ber: ber_flush2: 43 bytes to sd 4 ldap_result ld 90c48 msgid 1 wait4msg ld 90c48 msgid 1 (infinite timeout) wait4msg continue ld 90c48 msgid 1 all 1 ** ld 90c48 Connections: * host: sunval233 port: 636 (default) refcnt: 2 status: Connected last used: Thu May 12 08:04:02 2009 ** ld 90c48 Outstanding Requests: * msgid 1, origid 1, status InProgress outstanding referrals 0, parent count 0 ld 90c48 request count 1 (abandoned 0) ** ld 90c48 Response Queue: Empty ld 90c48 response count 0 ldap_chkResponseList ld 90c48 msgid 1 all 1 ldap_chkResponseList returns ld 90c48 NULL ldap_int_select read1msg: ld 90c48 msgid 1 all 1 ber_get_next ber_get_next: tag 0x30 len 12 contents: read1msg: ld 90c48 msgid 1 message type bind ber_scanf fmt ({eAA) ber: read1msg: ld 90c48 0 new referrals read1msg: mark request completed, ld 90c48 msgid 1 request done: ld 90c48 msgid 1 res_errno: 0, res_error: <>, res_matched: <> ldap_free_request (origid 1, msgid 1) ldap_free_connection 0 1 ldap_free_connection: refcnt 1 ldap_parse_result ber_scanf fmt ({iAA) ber: ber_scanf fmt (}) ber: ldap_msgfree adding new entry "dc=alu,dc=com" ldap_add_ext ldap_send_initial_request ldap_send_server_request ber_scanf fmt ({it) ber: ber_scanf fmt ({) ber: ber_flush2: 101 bytes to sd 4 ldap_result ld 90c48 msgid 2 wait4msg ld 90c48 msgid 2 (timeout 100000 usec) wait4msg continue ld 90c48 msgid 2 all 1 ** ld 90c48 Connections: * host: sunval233 port: 636 (default) refcnt: 2 status: Connected last used: Thu May 12 08:04:02 2009 ** ld 90c48 Outstanding Requests: * msgid 2, origid 2, status InProgress outstanding referrals 0, parent count 0 ld 90c48 request count 1 (abandoned 0) ** ld 90c48 Response Queue: Empty ld 90c48 response count 0 ldap_chkResponseList ld 90c48 msgid 2 all 1 ldap_chkResponseList returns ld 90c48 NULL ldap_int_select ldap_result ld 90c48 msgid 2 wait4msg ld 90c48 msgid 2 (timeout 100000 usec) wait4msg continue ld 90c48 msgid 2 all 1 ** ld 90c48 Connections: * host: sunval233 port: 636 (default) refcnt: 2 status: Connected last used: Thu May 12 08:04:02 2009 ** ld 90c48 Outstanding Requests: * msgid 2, origid 2, status InProgress outstanding referrals 0, parent count 0 ld 90c48 request count 1 (abandoned 0) ** ld 90c48 Response Queue: Empty ld 90c48 response count 0 ldap_chkResponseList ld 90c48 msgid 2 all 1 ldap_chkResponseList returns ld 90c48 NULL ldap_int_select ldap_result ld 90c48 msgid 2 wait4msg ld 90c48 msgid 2 (timeout 100000 usec) wait4msg continue ld 90c48 msgid 2 all 1 ** ld 90c48 Connections: * host: sunval233 port: 636 (default) refcnt: 2 status: Connected last used: Thu May 12 08:04:02 2009 ** ld 90c48 Outstanding Requests: * msgid 2, origid 2, status InProgress outstanding referrals 0, parent count 0 ld 90c48 request count 1 (abandoned 0) ** ld 90c48 Response Queue: Empty ld 90c48 response count 0 ldap_chkResponseList ld 90c48 msgid 2 all 1 ldap_chkResponseList returns ld 90c48 NULL ldap_int_select ^C #

1 0

Re: (ITS#5899) make fails on AIX5.3
by quanah＠zimbra.com 21 Jan '09

21 Jan '09

--On Wednesday, January 21, 2009 5:12 PM +0000 masterson.andrew(a)gmail.com wrote: > Full_Name: Andrew > Version: 2.4.11 > OS: AIX5.3 64 bit > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (208.98.237.232) > > > configure runs > > CPPFLAGS=-I/usr/local/BerkeleyDB.4.7/include > LDFLAGS=-L/usr/local/BerkeleyDB.4.7/lib ./configure --enable-ipv6 OpenLDAP 2.4.11 does not support BDB 4.7. Use OpenLDAP 2.4.12 or later for BDB 4.7 support. And make sure you have all 3 patches for BDB 4.7. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

(ITS#5899) make fails on AIX5.3
by masterson.andrew＠gmail.com 21 Jan '09

21 Jan '09

Full_Name: Andrew Version: 2.4.11 OS: AIX5.3 64 bit URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (208.98.237.232) configure runs CPPFLAGS=-I/usr/local/BerkeleyDB.4.7/include LDFLAGS=-L/usr/local/BerkeleyDB.4.7/lib ./configure --enable-ipv6 (and gives warnings about openssl and Cyrus SASL not being installed - otherwise fine) make fails with the following error: cd back-bdb; make all rm -f version.c ../../../build/mkversion -v "2.4.11" back_bdb > version.c /bin/sh ../../..//libtool --tag=disable-shared --mode=compile cc_r -g -I../../../include -I../../../include -I.. -I./.. -I/usr/local/BerkeleyDB.4.7/include -c init.c cc_r -g -I../../../include -I../../../include -I.. -I./.. -I/usr/local/BerkeleyDB.4.7/include -c init.c -o init.o "init.c", line 509.49: 1506-022 (S) "lk_handle" is not a member of "struct __db_env". "init.c", line 752.32: 1506-280 (W) Function argument assignment between types "int(*)(unsigned long,unsigned long)" and "int(*)(void)" is not allowed. make: 1254-004 The error code from the last command is 1. Stop. make: 1254-004 The error code from the last command is 2. Stop. make: 1254-004 The error code from the last command is 2. Stop. make: 1254-004 The error code from the last command is 2. Stop.

1 0

Re: (ITS#5894) test039 fails: ldap_back_conn_delete: Assertion
by michael＠stroeder.com 21 Jan '09

21 Jan '09

quanah(a)zimbra.com wrote: > --On Saturday, January 17, 2009 3:25 PM +0000 michael(a)stroeder.com wrote: > >> Howard Chu wrote: >>> michael(a)stroeder.com wrote: >>>> Full_Name: Michael Str?der >>>> Version: HEAD >>>> OS: Linux >>>> URL: ftp://ftp.openldap.org/incoming/ >>>> Submission from: (NULL) (84.163.100.169) >>>> >>>> >>>> lt-slapd: bind.c:157: ldap_back_conn_delete: Assertion >>>> `!(*(&((lc))->lc_lcflags) >>>> & (((0x00000020U))))' failed. >>> I'm not seeing that here. Ran into a couple other problems though, fixed >>> now in HEAD. Please re-test... >> Now test008 hangs... > > Multiple runs of test008 work for me, using BDB 4.7.25 with all 3 patches, > and HEAD CVS as of 1/20/2009 @ 12:00 PST. > > Did you fully rebuild OpenLDAP after the latest commits? What version of > BDB are you using? Is it fully patched? What backend was it using when it > hung? Did you get a backtrace of where it hung? I'm using libdb-4_5-4.5.20-94.17 from openSUSE 11.1 which is AFAIK fully patched. The interesting fact is that HEAD currently hangs all the time in test008. RE24 with your recent changes now fails as described in test039. Before these changes RE24 just worked fine. I think this is related to modifications in connection.c because trouble started after HEAD contained these changes. Ciao, Michael.

1 0

Re: (ITS#5895) Example in constraint manual and documentation
by ando＠sys-net.it 21 Jan '09

21 Jan '09

ghenry(a)suretecsystems.com wrote: > ----- ando(a)sys-net.it wrote: > >> Gavin Henry wrote: >>> Sorry, I presume this was a bug in the code and the docs document >> the >>> right behaviour? >>> >>> I'll revert them when you've confirmed. >> As far as I remember, the bug was in the example in >> slapo-constraint(5) >> man page ("[:digit:]" indicates the collection of values ":", "d", >> "i", >> ..., while "[[:digit:]]" indicates "0"->"9"). The man page is correct >> > > OK, it was actually this bit: > > constraint_attribute mail regex ^[[:alnum:]]+@mydomain.com$ > > and in HEAD it still only had one set of []. Both the admin guide and man page > have been updated for this in my last commit. Definitely my fault: apparently, I fixed it in my working copy long ago (maybe while offline) and forgot to commit. I just found it out because of a conflict when updating with your fix. Thanks, p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ----------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Fax: +39 0382 476497 Email: ando(a)sys-net.it -----------------------------------

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs