openldap-bugs October 2013

openldap-bugs@openldap.org

25 participants
95 discussions

(ITS#7736) back-mdb corrupts during write operations
by quanah＠OpenLDAP.org 30 Oct '13

30 Oct '13

Full_Name: Quanah Gibson-Mount Version: 2.4.37 OS: Linux 2.6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (75.111.58.125) Found after upgrading to OpenLDAP 2.4.37 + the last two commits to liblmdb: 527168fe mdb_search_candidates: id=1 first=77 last=77 slapd: ./../../../libraries/liblmdb/mdb.c:4596: mdb_page_get: Assertion `p != ((void *)0)' failed. Aborted slapd last logged: Oct 30 15:50:14 zqa-061 slapd[25442]: slap_queue_csn: queing 0x7f761e2091a0 20131030155014.945701Z#000000#000#000000 Oct 30 15:50:14 zqa-061 slapd[25442]: slap_graduate_commit_csn: removing 0x4cb8b70 20131030155014.945701Z#000000#000#000000

1 0

Re: (ITS#7735) Memory leaks on Multi-Master replications
by quanah＠zimbra.com 30 Oct '13

30 Oct '13

--On Wednesday, October 30, 2013 11:48 AM +0400 Dmitrii Fonariuk <dmitrii.fonariuk(a)gmail.com> wrote: > > use libtcmalloc_minimal.so.4.1.2 > add export LD_PRELOAD=/opt/local/tcmalloc/lib/libtcmalloc_minimal.so to > my run script > run slapd and view pmap pid to make sure that tcmalloc are using > run load script - 3K mps on two servers (1.5K mps per server) > load for 30 minutes memory consumption increased from 366 Mb to 660 Mb > and rising. Ok. You will need to provide your full configuration for each server, minus passwords, as well as sample data for this to be pursued. You may also want to run slapd under oprofile (without tcmalloc) to show the existence of any leaks. --Quanah -- Quanah Gibson-Mount Architect - Server Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#7735) Memory leaks on Multi-Master replications
by dmitrii.fonariuk＠gmail.com 30 Oct '13

30 Oct '13

--047d7bf0f220ae630704e9f08f20 Content-Type: text/plain; charset=ISO-8859-1 use libtcmalloc_minimal.so.4.1.2 add export LD_PRELOAD=/opt/local/tcmalloc/lib/libtcmalloc_minimal.so to my run script run slapd and view pmap pid to make sure that tcmalloc are using run load script - 3K mps on two servers (1.5K mps per server) load for 30 minutes memory consumption increased from 366 Mb to 660 Mb and rising. libtcmalloc did not help me 2013/10/30 Quanah Gibson-Mount <quanah(a)zimbra.com> > > > On Oct 29, 2013, at 10:32 PM, Dmitrii Fonariuk <dmitrii.fonariuk(a)gmail.com> > wrote: > > hi. > visually. two days of use 3K load modify per second memory consumption > increased from 200 Mb to 16 Gb. After that, the system killed the process. > when modify send only to one server use 200 Mb memory and its consumption > does not increase. > no I do not use an alternative memory allocator. > > best regards, > Dmitrii > > > I strongly advise you use tcmalloc and report back. > > > --047d7bf0f220ae630704e9f08f20 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">use libtcmalloc_minimal.so.4.1.2<div><font face=3D"arial, = helvetica, sans-serif" color=3D"#000000" style=3D"background-color:rgb(255,= 255,255)">add=A0export LD_PRELOAD=3D/opt/local/tcmalloc/lib/libtcmalloc_min= imal.so to my run script</font></div> <div><font color=3D"#000000" face=3D"arial, helvetica, sans-serif">run=A0sl= apd and view pmap pid to make=A0sure that=A0tcmalloc are using</font></div>= <div><font color=3D"#000000" face=3D"arial, helvetica, sans-serif">run load= script - 3K mps on two servers (1.5K mps per server)</font></div> <div><font color=3D"#000000" face=3D"arial, helvetica, sans-serif">load for= 30 minutes memory consumption increased from 366 Mb to 660 Mb and rising.<= /font></div><div><br></div><div>libtcmalloc did not help me<br></div><div><= br> </div><div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">20= 13/10/30 Quanah Gibson-Mount <span dir=3D"ltr"><<a href=3D"mailto:quanah= @zimbra.com" target=3D"_blank">quanah(a)zimbra.com</a>></span><br><blockqu= ote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-wid= th:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-l= eft:1ex"> <div dir=3D"auto"><div class=3D"im"><div><br></div><div><br>On Oct 29, 2013= , at 10:32 PM, Dmitrii Fonariuk <<a href=3D"mailto:dmitrii.fonariuk@gmai= l.com" target=3D"_blank">dmitrii.fonariuk(a)gmail.com</a>> wrote:<br><br><= /div> <blockquote type=3D"cite"><div><div dir=3D"ltr">hi.<div>visually.=A0two day= s of use 3K load modify per second memory consumption increased from 200 Mb= to 16 Gb. After that, the system killed the process.</div><div>when modify= send only to one server use 200 Mb memory and its consumption does not inc= rease.</div> <div>no I do not use an=A0<span style=3D"font-family:arial,sans-serif;font-= size:13px">alternative memory allocator.</span></div><div><font face=3D"ari= al, sans-serif"><br></font></div><div><font face=3D"arial, sans-serif">best= regards,</font></div> <div><font face=3D"arial, sans-serif">Dmitrii</font></div></div></div></blo= ckquote><div><br></div></div><div>I strongly advise you use tcmalloc and re= port back.=A0</div><span class=3D""><font color=3D"#888888"><div><br></div>= <div> <br></div></font></span></div></blockquote></div><div><br></div> </div></div></div> --047d7bf0f220ae630704e9f08f20--

1 0

Re: (ITS#7735) Memory leaks on Multi-Master replications
by quanah＠zimbra.com 29 Oct '13

29 Oct '13

--Apple-Mail-A17D5106-BB00-42CE-B576-6B323E42E55E Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable > On Oct 29, 2013, at 10:32 PM, Dmitrii Fonariuk <dmitrii.fonariuk(a)gmail.com= > wrote: >=20 > hi. > visually. two days of use 3K load modify per second memory consumption inc= reased from 200 Mb to 16 Gb. After that, the system killed the process. > when modify send only to one server use 200 Mb memory and its consumption d= oes not increase. > no I do not use an alternative memory allocator. >=20 > best regards, > Dmitrii I strongly advise you use tcmalloc and report back.=20 --Quanah >=20 > 2013/10/29 Quanah Gibson-Mount <quanah(a)zimbra.com> >> --On Tuesday, October 29, 2013 10:45 AM +0000 dmitrii.fonariuk(a)gmail.com w= rote: >>=20 >>> Full_Name: Dmitrii Fonariuk >>> Version: 2.4.37 >>> OS: RHEL6.x86_64 >>> URL: ftp://ftp.openldap.org/incoming/ >>> Submission from: (NULL) (91.210.4.1) >>>> memory leaks >>>=20 >>> * in square brackets ranges. >>>=20 >>> Ranges for DN do not overlap for different servers. State is numericStri= ng >>> attribute. >>=20 >> How are you determining there is a memory leak? Are you using an alterna= tive memory allocator like tcmalloc instead of glibc? >>=20 >> --Quanah >>=20 >>=20 >> -- >>=20 >> Quanah Gibson-Mount >> Architect - Server >> Zimbra, Inc. >> -------------------- >> Zimbra :: the leader in open source messaging and collaboration >=20 --Apple-Mail-A17D5106-BB00-42CE-B576-6B323E42E55E Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div><br></div><div><br>On Oct 29, 2013, at 10:32 PM, Dmitrii Fonariuk <<a href="mailto:dmitrii.fonariuk@gmail.com">dmitrii.fonariuk(a)gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">hi.<div>visually. two days of use 3K load modify per second memory consumption increased from 200 Mb to 16 Gb. After that, the system killed the process.</div><div>when modify send only to one server use 200 Mb memory and its consumption does not increase.</div> <div>no I do not use an <span style="font-family:arial,sans-serif;font-size:13px">alternative memory allocator.</span></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">best regards,</font></div> <div><font face="arial, sans-serif">Dmitrii</font></div></div></div></blockquote><div><br></div><div>I strongly advise you use tcmalloc and report back. </div><div><br></div><div>--Quanah</div><div><br></div><div><br></div><br><blockquote type="cite"><div><div dir="ltr"><div><div class="gmail_extra"><br><div class="gmail_quote">2013/10/29 Quanah Gibson-Mount <span dir="ltr"><<a href="mailto:quanah@zimbra.com" target="_blank">quanah(a)zimbra.com</a>></span><br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">--On Tuesday, October 29, 2013 10:45 AM +0000 <a href="mailto:dmitrii.fonariuk@gmail.com" target="_blank">dmitrii.fonariuk(a)gmail.com</a> wrote:<br> <br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> Full_Name: Dmitrii Fonariuk<br> Version: 2.4.37<br> OS: RHEL6.x86_64<br> URL: <a href="ftp://ftp.openldap.org/incoming/" target="_blank">ftp://ftp.openldap.org/<u></u>incoming/</a><br> Submission from: (NULL) (91.210.4.1)<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> memory leaks<br> </blockquote> <br> * in square brackets ranges.<br> <br> Ranges for DN do not overlap for different servers. State is numericString<br> attribute.<br> </blockquote> <br> How are you determining there is a memory leak?  Are you using an alternative memory allocator like tcmalloc instead of glibc?<br> <br> --Quanah<br> <br> <br> --<br> <br> Quanah Gibson-Mount<br> Architect - Server<br> Zimbra, Inc.<br> --------------------<br> Zimbra ::  the leader in open source messaging and collaboration<br> </blockquote></div><br></div></div></div> </div></blockquote></body></html> --Apple-Mail-A17D5106-BB00-42CE-B576-6B323E42E55E--

1 0

Re: (ITS#7735) Memory leaks on Multi-Master replications
by dmitrii.fonariuk＠gmail.com 29 Oct '13

29 Oct '13

--e89a8ff1c04499e9af04e9eea81c Content-Type: text/plain; charset=ISO-8859-1 hi. visually. two days of use 3K load modify per second memory consumption increased from 200 Mb to 16 Gb. After that, the system killed the process. when modify send only to one server use 200 Mb memory and its consumption does not increase. no I do not use an alternative memory allocator. best regards, Dmitrii 2013/10/29 Quanah Gibson-Mount <quanah(a)zimbra.com> > --On Tuesday, October 29, 2013 10:45 AM +0000 dmitrii.fonariuk(a)gmail.comwrote: > > Full_Name: Dmitrii Fonariuk >> Version: 2.4.37 >> OS: RHEL6.x86_64 >> URL: ftp://ftp.openldap.org/**incoming/<ftp://ftp.openldap.org/incoming/> >> Submission from: (NULL) (91.210.4.1) >> >>> memory leaks >>> >> >> * in square brackets ranges. >> >> Ranges for DN do not overlap for different servers. State is numericString >> attribute. >> > > How are you determining there is a memory leak? Are you using an > alternative memory allocator like tcmalloc instead of glibc? > > --Quanah > > > -- > > Quanah Gibson-Mount > Architect - Server > Zimbra, Inc. > -------------------- > Zimbra :: the leader in open source messaging and collaboration > --e89a8ff1c04499e9af04e9eea81c Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">hi.<div>visually.=A0two days of use 3K load modify per sec= ond memory consumption increased from 200 Mb to 16 Gb. After that, the syst= em killed the process.</div><div>when modify send only to one server use 20= 0 Mb memory and its consumption does not increase.</div> <div>no I do not use an=A0<span style=3D"font-family:arial,sans-serif;font-= size:13px">alternative memory allocator.</span></div><div><font face=3D"ari= al, sans-serif"><br></font></div><div><font face=3D"arial, sans-serif">best= regards,</font></div> <div><font face=3D"arial, sans-serif">Dmitrii<br></font><div class=3D"gmail= _extra"><br><br><div class=3D"gmail_quote">2013/10/29 Quanah Gibson-Mount <= span dir=3D"ltr"><<a href=3D"mailto:quanah@zimbra.com" target=3D"_blank"= >quanah(a)zimbra.com</a>></span><br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex">--On Tuesday, October 29, 2013 10:45 AM +0000 <a href=3D"m= ailto:dmitrii.fonariuk@gmail.com" target=3D"_blank">dmitrii.fonariuk(a)gmail.= com</a> wrote:<br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> Full_Name: Dmitrii Fonariuk<br> Version: 2.4.37<br> OS: RHEL6.x86_64<br> URL: <a href=3D"ftp://ftp.openldap.org/incoming/" target=3D"_blank">ftp://f= tp.openldap.org/<u></u>incoming/</a><br> Submission from: (NULL) (91.210.4.1)<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> memory leaks<br> </blockquote> <br> * in square brackets ranges.<br> <br> Ranges for DN do not overlap for different servers. State is numericString<= br> attribute.<br> </blockquote> <br> How are you determining there is a memory leak? =A0Are you using an alterna= tive memory allocator like tcmalloc instead of glibc?<br> <br> --Quanah<br> <br> <br> --<br> <br> Quanah Gibson-Mount<br> Architect - Server<br> Zimbra, Inc.<br> --------------------<br> Zimbra :: =A0the leader in open source messaging and collaboration<br> </blockquote></div><br></div></div></div> --e89a8ff1c04499e9af04e9eea81c--

1 0

Re: (ITS#7735) Memory leaks on Multi-Master replications
by quanah＠zimbra.com 29 Oct '13

29 Oct '13

--On Tuesday, October 29, 2013 10:45 AM +0000 dmitrii.fonariuk(a)gmail.com wrote: > Full_Name: Dmitrii Fonariuk > Version: 2.4.37 > OS: RHEL6.x86_64 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (91.210.4.1) >> memory leaks > > * in square brackets ranges. > > Ranges for DN do not overlap for different servers. State is numericString > attribute. How are you determining there is a memory leak? Are you using an alternative memory allocator like tcmalloc instead of glibc? --Quanah -- Quanah Gibson-Mount Architect - Server Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

(ITS#7735) Memory leaks on Multi-Master replications
by dmitrii.fonariuk＠gmail.com 29 Oct '13

29 Oct '13

Full_Name: Dmitrii Fonariuk Version: 2.4.37 OS: RHEL6.x86_64 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (91.210.4.1) Hi. I have 2 servers with OpenLDAP 2.4.37 in Multi-Master replication. If I send modify request to only one server no memory leaks. If I send different modify request on both servers I see a memory leaks on both servers. Here are the settings of one of the servers: dn: olcDatabase={1}mdb,cn=config objectClass: olcDatabaseConfig objectClass: olcConfig objectClass: top objectClass: olcMdbConfig olcDatabase: {1}mdb olcDbDirectory: /opt/local/OpenLDAP/data/mdb3 olcDbEnvFlags: writemap olcDbEnvFlags: mapasync olcDbIndex: entryUUID eq olcDbIndex: entryCSN eq olcDbIndex: objectClass eq olcDbMaxSize: 5368709120 olcLimits: {0}dn.exact="uid=replica1,uid=admin,dc=rrr" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited olcMirrorMode: TRUE olcRootDN: dc=rrr olcRootPW: secret olcSuffix: dc=rrr olcSyncrepl: {0}rid=010 provider=ldap://srv-rrr03:3903 binddn="uid=replica1,uid=admin,dc=rrr" bindmethod=simple credentials=replica1 searchbase="dc=rrr" schemachecking=off type=refreshAndPersist retry="5 5 60 +" dn: olcOverlay={0}syncprov,olcDatabase={1}mdb,cn=config objectClass: olcSyncProvConfig objectClass: olcOverlayConfig objectClass: olcConfig objectClass: top olcOverlay: {0}syncprov olcSpCheckpoint: 1000 5 Requests on one server: dn: id=1000000000[10000-30000],ou=subscriber,o=Megatron,dc=rrr changetype: modify replace: state state: [1-9] no memory leaks Requests on both servers: 1: dn: id=1000000000[10000-20000],ou=subscriber,o=Megatron,dc=rrr changetype: modify replace: state state: [1-9] 2: dn: id=1000000000[20001-30000],ou=subscriber,o=Megatron,dc=rrr changetype: modify replace: state state: [1-9] memory leaks * in square brackets ranges. Ranges for DN do not overlap for different servers. State is numericString attribute.

1 0

Re: (ITS#7705) mdb back-end segfaults sporadically with paged searches
by openldap＠semyon.org 28 Oct '13

28 Oct '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 The problem seems to be resolved in OPENLDAP_REL_ENG_2_4_37. Please close this ticket. Best Semyon -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.20 (Darwin) Comment: GPGTools - https://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQGcBAEBCgAGBQJSbugyAAoJEDnsppBba4kKtNcL/0YHZaQfYOL7X5LBkYlsHX86 JtLPPT8mCax9ehrTQUmz5ZzuzV2YkQ2KnzO45Gb8K721bNTCuNBwlXvT33yh+I0T QCEOAdMOxAasC27BW3L46Bbw+lW9/GPfxW8p6eGaw2qDCpIwRluxApysi+paRZR1 VCXu/z8/zLIEVMDKq/lktGnCBFR7uNBr575G4ePZVgytDO8aEk/F2VE+LQRBbfVL 25zlQesiO8EEPk5blbBhRFxTUZEBs/0EAnBB7hoBVKDlbog4tjOav6aNq1o4TegS +0oNgZS+K0Krop/3v27dr+LzaFy07jySE4aDgXKstgqzM4uvZzCSxarsFPZyUqbd sJNXBEKM5HUMn6orl6Ip7Z7h0MSkPYAH7OZLWXYTOmgl5wgJ8ArQSMMTD9+vC/yk McIDElkqCoRce/IRfRYjS+Ct+qaEB7AYKhtbzVAwPjJBIj4S5N24xUShjDCvkjay K6oj3GQFq78jp2ehITHn9zytiw12gmhv+V42hEQY+g== =VzO+ -----END PGP SIGNATURE-----

1 0

ITS#7731
by hyc＠symas.com 28 Oct '13

28 Oct '13

The ITS is for actual bug reports, not for software usage questions. This ITS will be closed. Since you're using an RHEL build you should contact RedHat for support. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#7734) MDB_NORDAHEAD missing from valid flags bitfields
by dw＠botanicus.net 27 Oct '13

27 Oct '13

Full_Name: David Wilson Version: LMDB 0.9.9 OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (178.238.153.20) Unless I'm missing something, this new flag cannot be set during mdb_env_open(), and since it is not in CHANGEABLE either, cannot be set with mdb_env_set_flags() commit dd0df21ce0a935955966aa7d3077d666741d027f Author: David Wilson <dw(a)botanicus.net> Date: Mon Oct 28 01:28:00 2013 +0000 Add MDB_NORDAHEAD to CHANGELESS. diff --git a/lib/mdb.c b/lib/mdb.c index 33be876..3ec7302 100644 --- a/lib/mdb.c +++ b/lib/mdb.c @@ -3929,7 +3929,7 @@ fail: * environment and re-opening it with the new flags. */ #define CHANGEABLE (MDB_NOSYNC|MDB_NOMETASYNC|MDB_MAPASYNC) -#define CHANGELESS (MDB_FIXEDMAP|MDB_NOSUBDIR|MDB_RDONLY|MDB_WRITEMAP|MDB_NOTLS|MDB_NOLOCK) +#define CHANGELESS (MDB_FIXEDMAP|MDB_NOSUBDIR|MDB_RDONLY|MDB_WRITEMAP|MDB_NOTLS|MDB_NOLOCK|MDB_NORDAHEAD) int mdb_env_open(MDB_env *env, const char *path, unsigned int flags, mdb_mode_t mode)

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs October 2013