--0-1358063323-1230648878=:64021 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable
Pierangelo,
I could run the valgrind but at this time I do not have the DB with the aro= und 4 million subscribers. In this condition, with a small number of users,= I already could see some "definitive" memory leaks and I believe they are = associated with OpenLdap.
Just for a history I already compiled the openldap 2.4.11 souce code withou= t modules and TLS since many lists make reference to some possible issues i= n RHEL with TLS. Even so I could not have a load without the possible memor= y leak.
So I do not believe the problems are in the RHEL libtool or glibc.
In any case the result for the valgrind is (short number of entrances at th= is time):
=3D=3D32608=3D=3D Memcheck, a memory error detector. =3D=3D32608=3D=3D Copyright (C) 2002-2005, and GNU GPL'd, by Julian Seward = et al. =3D=3D32608=3D=3D Using LibVEX rev 1575, a library for dynamic binary trans= lation. =3D=3D32608=3D=3D Copyright (C) 2004-2005, and GNU GPL'd, by OpenWorks LLP. =3D=3D32608=3D=3D Using valgrind-3.1.1, a dynamic binary instrumentation fr= amework. =3D=3D32608=3D=3D Copyright (C) 2000-2005, and GNU GPL'd, by Julian Seward = et al. =3D=3D32608=3D=3D For more details, rerun with: -v =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 86 f= rom 2) =3D=3D32608=3D=3D malloc/free: in use at exit: 751 bytes in 32 blocks. =3D=3D32608=3D=3D malloc/free: 29,775 allocs, 29,743 frees, 22,150,722 byte= s allocated. =3D=3D32608=3D=3D For counts of detected errors, rerun with: -v =3D=3D32608=3D=3D searching for pointers to 32 not-freed blocks. =3D=3D32608=3D=3D checked 8,191,292 bytes. =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D 16 bytes in 1 blocks are still reachable in loss record 1= of 9 =3D=3D32608=3D=3D=A0=A0=A0 at 0x40056BF: calloc (vg_replace_malloc.c:279) =3D=3D32608=3D=3D=A0=A0=A0 by 0x519364: _dlerror_run (in /lib/libdl-2.3.4.s= o) =3D=3D32608=3D=3D=A0=A0=A0 by 0x518D10: dlopen@@GLIBC_2.1 (in /lib/libdl-2.= 3.4.so) =3D=3D32608=3D=3D=A0=A0=A0 by 0x52D80D: _sasl_get_plugin (in /usr/lib/libsa= sl2.so.2.0.19) =3D=3D32608=3D=3D=A0=A0=A0 by 0x52DC61: _sasl_load_plugins (in /usr/lib/lib= sasl2.so.2.0.19) =3D=3D32608=3D=3D=A0=A0=A0 by 0x52B063: sasl_server_init (in /usr/lib/libsa= sl2.so.2.0.19) =3D=3D32608=3D=3D=A0=A0=A0 by 0x80C82C0: slap_sasl_init (in /usr/libexec/sl= apd) =3D=3D32608=3D=3D=A0=A0=A0 by 0x3FFDE2: (below main) (in /lib/tls/libc-2.3.= 4.so) =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D 22 bytes in 1 blocks are still reachable in loss record 2= of 9 =3D=3D32608=3D=3D=A0=A0=A0 at 0x4004405: malloc (vg_replace_malloc.c:149) =3D=3D32608=3D=3D=A0=A0=A0 by 0xB77567: (within /usr/lib/libltdl.so.3.1.0) =3D=3D32608=3D=3D=A0=A0=A0 by 0xB780C8: (within /usr/lib/libltdl.so.3.1.0) =3D=3D32608=3D=3D=A0=A0=A0 by 0xB7A79D: lt_dlsetsearchpath (in /usr/lib/lib= ltdl.so.3.1.0) =3D=3D32608=3D=3D=A0=A0=A0 by 0x8068147: (within /usr/libexec/slapd) =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D 24 bytes in 1 blocks are still reachable in loss record 3= of 9 =3D=3D32608=3D=3D=A0=A0=A0 at 0x400579F: realloc (vg_replace_malloc.c:306) =3D=3D32608=3D=3D=A0=A0=A0 by 0x8151E6F: ber_bvarray_add (in /usr/libexec/s= lapd) =3D=3D32608=3D=3D=A0=A0=A0 by 0x44DD627: ??? =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D 28 bytes in 2 blocks are definitely lost in loss record 4= of 9 =3D=3D32608=3D=3D=A0=A0=A0 at 0x4004405: malloc (vg_replace_malloc.c:149) =3D=3D32608=3D=3D=A0=A0=A0 by 0xB77567: (within /usr/lib/libltdl.so.3.1.0) =3D=3D32608=3D=3D=A0=A0=A0 by 0xB780C8: (within /usr/lib/libltdl.so.3.1.0) =3D=3D32608=3D=3D=A0=A0=A0 by 0xB78CDC: (within /usr/lib/libltdl.so.3.1.0) =3D=3D32608=3D=3D=A0=A0=A0 by 0xB7972B: lt_dlopen (in /usr/lib/libltdl.so.3= .1.0) =3D=3D32608=3D=3D=A0=A0=A0 by 0xB7980C: lt_dlopenext (in /usr/lib/libltdl.s= o.3.1.0) =3D=3D32608=3D=3D=A0=A0=A0 by 0x80CA1E3: module_load (in /usr/libexec/slapd= ) =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D 28 bytes in 1 blocks are still reachable in loss record 5= of 9 =3D=3D32608=3D=3D=A0=A0=A0 at 0x4004405: malloc (vg_replace_malloc.c:149) =3D=3D32608=3D=3D=A0=A0=A0 by 0x3D81D7: _dl_map_object_deps (in /lib/ld-2.3= .4.so) =3D=3D32608=3D=3D=A0=A0=A0 by 0x4E7158: dl_open_worker (in /lib/tls/libc-2.= 3.4.so) =3D=3D32608=3D=3D=A0=A0=A0 by 0x3D90FD: _dl_catch_error (in /lib/ld-2.3.4.s= o) =3D=3D32608=3D=3D=A0=A0=A0 by 0x4E7CB7: _dl_open (in /lib/tls/libc-2.3.4.so= ) =3D=3D32608=3D=3D=A0=A0=A0 by 0x4E904C: do_dlopen (in /lib/tls/libc-2.3.4.s= o) =3D=3D32608=3D=3D=A0=A0=A0 by 0x3D90FD: _dl_catch_error (in /lib/ld-2.3.4.s= o) =3D=3D32608=3D=3D=A0=A0=A0 by 0x4E912D: __libc_dlopen_mode (in /lib/tls/lib= c-2.3.4.so) =3D=3D32608=3D=3D=A0=A0=A0 by 0x561087: pthread_cancel_init (in /lib/tls/li= bpthread-2.3.4.so) =3D=3D32608=3D=3D=A0=A0=A0 by 0x5611FC: _Unwind_ForcedUnwind (in /lib/tls/l= ibpthread-2.3.4.so) =3D=3D32608=3D=3D=A0=A0=A0 by 0x55F020: __pthread_unwind (in /lib/tls/libpt= hread-2.3.4.so) =3D=3D32608=3D=3D=A0=A0=A0 by 0x55AF8F: pthread_exit (in /lib/tls/libpthrea= d-2.3.4.so) =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D 32 bytes in 1 blocks are still reachable in loss record 6= of 9 =3D=3D32608=3D=3D=A0=A0=A0 at 0x4004405: malloc (vg_replace_malloc.c:149) =3D=3D32608=3D=3D=A0=A0=A0 by 0x4005728: realloc (vg_replace_malloc.c:306) =3D=3D32608=3D=3D=A0=A0=A0 by 0x810E504: rewrite_mapper_register (in /usr/l= ibexec/slapd) =3D=3D32608=3D=3D=A0=A0=A0 by 0x3FFDE2: (below main) (in /lib/tls/libc-2.3.= 4.so) =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D 77 bytes in 5 blocks are still reachable in loss record 7= of 9 =3D=3D32608=3D=3D=A0=A0=A0 at 0x4004405: malloc (vg_replace_malloc.c:149) =3D=3D32608=3D=3D=A0=A0=A0 by 0x8150E2E: ber_memalloc_x (in /usr/libexec/sl= apd) =3D=3D32608=3D=3D=A0=A0=A0 by 0x3D6363FF: ??? =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D 252 bytes in 18 blocks are definitely lost in loss record= 8 of 9 =3D=3D32608=3D=3D=A0=A0=A0 at 0x4004405: malloc (vg_replace_malloc.c:149) =3D=3D32608=3D=3D=A0=A0=A0 by 0x8150E2E: ber_memalloc_x (in /usr/libexec/sl= apd) =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D 272 bytes in 2 blocks are possibly lost in loss record 9 = of 9 =3D=3D32608=3D=3D=A0=A0=A0 at 0x40056BF: calloc (vg_replace_malloc.c:279) =3D=3D32608=3D=3D=A0=A0=A0 by 0x3DB71A: _dl_allocate_tls (in /lib/ld-2.3.4.= so) =3D=3D32608=3D=3D=A0=A0=A0 by 0x55A91E: pthread_create@@GLIBC_2.1 (in /lib/= tls/libpthread-2.3.4.so) =3D=3D32608=3D=3D=A0=A0=A0 by 0x811D4A9: ldap_pvt_thread_create (in /usr/li= bexec/slapd) =3D=3D32608=3D=3D=A0=A0=A0 by 0x3FFDE2: (below main) (in /lib/tls/libc-2.3.= 4.so) =3D=3D32608=3D=3D=20 =3D=3D32608=3D=3D LEAK SUMMARY: =3D=3D32608=3D=3D=A0=A0=A0 definitely lost: 280 bytes in 20 blocks. =3D=3D32608=3D=3D=A0=A0=A0=A0=A0 possibly lost: 272 bytes in 2 blocks. =3D=3D32608=3D=3D=A0=A0=A0 still reachable: 199 bytes in 10 blocks. =3D=3D32608=3D=3D=A0=A0=A0=A0=A0=A0=A0=A0 suppressed: 0 bytes in 0 blocks.
My suspicious is this memory leak is happening at ber_memalloc_x (in /usr/l= ibexec/slapd). Or in the part above :
=3D=3D32608=3D=3D 252 bytes in 18 blocks are definitely lost in loss record= 8 of 9
=3D=3D32608=3D=3D=A0=A0=A0 at 0x4004405: malloc (vg_replace_malloc.c:149)
=3D=3D32608=3D=3D=A0=A0=A0 by 0x8150E2E: ber_memalloc_x (in /usr/libexec/sl= apd)
Since right now I do not have a large number of entrances in the DB the sea= rch is easily cached in memory and the response is very quick.
I will load again the DB but since this takes around 6-8hours I sent this i= nitial result from valgrind since it can already give some directions. Tomo= rrow I should send more information related with a more heavy load DB.
I would like your help to see if this can identify something related with t= his memory leakage.
Please let me know if I can help in this meantime with something else.
Best regards,
Rodrigo.
--- On Sat, 12/20/08, Pierangelo Masarati ando@sys-net.it wrote: From: Pierangelo Masarati ando@sys-net.it Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 To: rlvcosta@yahoo.com Cc: openldap-its@openldap.org Date: Saturday, December 20, 2008, 7:05 PM
rlvcosta@yahoo.com wrote:
Full_Name: Rodrigo Costa Version: 2.4.11 OS: RHEL4 U4 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (201.82.161.220) =20 =20 Opeldap dev team, =20 I'm facing an issue and after many tentatives I believe this is really
a bug
that maybe could be identified and solved by openldap community. I tried
to
include dmalloc into openldap 2.4.11 code but even I could compile with i=
t any
tentative to execute the code generated "Segmentation Fault" so
I could not use
dmalloc to identify where the memory leak is happening.
Can you try with valgrind?
p.
Ing. Pierangelo Masarati OpenLDAP Core Team
SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ----------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Fax: +39 0382 476497 Email: ando@sys-net.it -----------------------------------
=0A=0A=0A --0-1358063323-1230648878=:64021 Content-Type: text/html; charset=us-ascii
<table cellspacing="0" cellpadding="0" border="0" ><tr><td valign="top" style="font: inherit;">Pierangelo,<br><br>I could run the valgrind but at this time I do not have the DB with the around 4 million subscribers. In this condition, with a small number of users, I already could see some "definitive" memory leaks and I believe they are associated with OpenLdap.<br><br>Just for a history I already compiled the openldap 2.4.11 souce code without modules and TLS since many lists make reference to some possible issues in RHEL with TLS. Even so I could not have a load without the possible memory leak.<br><br>So I do not believe the problems are in the RHEL libtool or glibc.<br><br>In any case the result for the valgrind is (short number of entrances at this time):<br><br>==32608== Memcheck, a memory error detector.<br>==32608== Copyright (C) 2002-2005, and GNU GPL'd, by Julian Seward et al.<br>==32608== Using LibVEX rev 1575, a library for dynamic binary translation.<br>==32608== Copyright (C) 2004-2005, and GNU GPL'd, by OpenWorks LLP.<br>==32608== Using valgrind-3.1.1, a dynamic binary instrumentation framework.<br>==32608== Copyright (C) 2000-2005, and GNU GPL'd, by Julian Seward et al.<br>==32608== For more details, rerun with: -v<br>==32608== <br>==32608== <br>==32608== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 86 from 2)<br>==32608== malloc/free: in use at exit: 751 bytes in 32 blocks.<br>==32608== malloc/free: 29,775 allocs, 29,743 frees, 22,150,722 bytes allocated.<br>==32608== For counts of detected errors, rerun with: -v<br>==32608== searching for pointers to 32 not-freed blocks.<br>==32608== checked 8,191,292 bytes.<br>==32608== <br>==32608== 16 bytes in 1 blocks are still reachable in loss record 1 of 9<br>==32608== at 0x40056BF: calloc (vg_replace_malloc.c:279)<br>==32608== by 0x519364: _dlerror_run (in /lib/libdl-2.3.4.so)<br>==32608== by 0x518D10: dlopen@@GLIBC_2.1 (in /lib/libdl-2.3.4.so)<br>==32608== by 0x52D80D: _sasl_get_plugin (in /usr/lib/libsasl2.so.2.0.19)<br>==32608== by 0x52DC61: _sasl_load_plugins (in /usr/lib/libsasl2.so.2.0.19)<br>==32608== by 0x52B063: sasl_server_init (in /usr/lib/libsasl2.so.2.0.19)<br>==32608== by 0x80C82C0: slap_sasl_init (in /usr/libexec/slapd)<br>==32608== by 0x3FFDE2: (below main) (in /lib/tls/libc-2.3.4.so)<br>==32608== <br>==32608== <br>==32608== 22 bytes in 1 blocks are still reachable in loss record 2 of 9<br>==32608== at 0x4004405: malloc (vg_replace_malloc.c:149)<br>==32608== by 0xB77567: (within /usr/lib/libltdl.so.3.1.0)<br>==32608== by 0xB780C8: (within /usr/lib/libltdl.so.3.1.0)<br>==32608== by 0xB7A79D: lt_dlsetsearchpath (in /usr/lib/libltdl.so.3.1.0)<br>==32608== by 0x8068147: (within /usr/libexec/slapd)<br>==32608== <br>==32608== <br>==32608== 24 bytes in 1 blocks are still reachable in loss record 3 of 9<br>==32608== at 0x400579F: realloc (vg_replace_malloc.c:306)<br>==32608== by 0x8151E6F: ber_bvarray_add (in /usr/libexec/slapd)<br>==32608== by 0x44DD627: ???<br>==32608== <br>==32608== <br>==32608== 28 bytes in 2 blocks are definitely lost in loss record 4 of 9<br>==32608== at 0x4004405: malloc (vg_replace_malloc.c:149)<br>==32608== by 0xB77567: (within /usr/lib/libltdl.so.3.1.0)<br>==32608== by 0xB780C8: (within /usr/lib/libltdl.so.3.1.0)<br>==32608== by 0xB78CDC: (within /usr/lib/libltdl.so.3.1.0)<br>==32608== by 0xB7972B: lt_dlopen (in /usr/lib/libltdl.so.3.1.0)<br>==32608== by 0xB7980C: lt_dlopenext (in /usr/lib/libltdl.so.3.1.0)<br>==32608== by 0x80CA1E3: module_load (in /usr/libexec/slapd)<br>==32608== <br>==32608== <br>==32608== 28 bytes in 1 blocks are still reachable in loss record 5 of 9<br>==32608== at 0x4004405: malloc (vg_replace_malloc.c:149)<br>==32608== by 0x3D81D7: _dl_map_object_deps (in /lib/ld-2.3.4.so)<br>==32608== by 0x4E7158: dl_open_worker (in /lib/tls/libc-2.3.4.so)<br>==32608== by 0x3D90FD: _dl_catch_error (in /lib/ld-2.3.4.so)<br>==32608== by 0x4E7CB7: _dl_open (in /lib/tls/libc-2.3.4.so)<br>==32608== by 0x4E904C: do_dlopen (in /lib/tls/libc-2.3.4.so)<br>==32608== by 0x3D90FD: _dl_catch_error (in /lib/ld-2.3.4.so)<br>==32608== by 0x4E912D: __libc_dlopen_mode (in /lib/tls/libc-2.3.4.so)<br>==32608== by 0x561087: pthread_cancel_init (in /lib/tls/libpthread-2.3.4.so)<br>==32608== by 0x5611FC: _Unwind_ForcedUnwind (in /lib/tls/libpthread-2.3.4.so)<br>==32608== by 0x55F020: __pthread_unwind (in /lib/tls/libpthread-2.3.4.so)<br>==32608== by 0x55AF8F: pthread_exit (in /lib/tls/libpthread-2.3.4.so)<br>==32608== <br>==32608== <br>==32608== 32 bytes in 1 blocks are still reachable in loss record 6 of 9<br>==32608== at 0x4004405: malloc (vg_replace_malloc.c:149)<br>==32608== by 0x4005728: realloc (vg_replace_malloc.c:306)<br>==32608== by 0x810E504: rewrite_mapper_register (in /usr/libexec/slapd)<br>==32608== by 0x3FFDE2: (below main) (in /lib/tls/libc-2.3.4.so)<br>==32608== <br>==32608== <br>==32608== 77 bytes in 5 blocks are still reachable in loss record 7 of 9<br>==32608== at 0x4004405: malloc (vg_replace_malloc.c:149)<br>==32608== by 0x8150E2E: ber_memalloc_x (in /usr/libexec/slapd)<br>==32608== by 0x3D6363FF: ???<br>==32608== <br>==32608== <br>==32608== 252 bytes in 18 blocks are definitely lost in loss record 8 of 9<br>==32608== at 0x4004405: malloc (vg_replace_malloc.c:149)<br>==32608== by 0x8150E2E: ber_memalloc_x (in /usr/libexec/slapd)<br>==32608== <br>==32608== <br>==32608== 272 bytes in 2 blocks are possibly lost in loss record 9 of 9<br>==32608== at 0x40056BF: calloc (vg_replace_malloc.c:279)<br>==32608== by 0x3DB71A: _dl_allocate_tls (in /lib/ld-2.3.4.so)<br>==32608== by 0x55A91E: pthread_create@@GLIBC_2.1 (in /lib/tls/libpthread-2.3.4.so)<br>==32608== by 0x811D4A9: ldap_pvt_thread_create (in /usr/libexec/slapd)<br>==32608== by 0x3FFDE2: (below main) (in /lib/tls/libc-2.3.4.so)<br>==32608== <br>==32608== LEAK SUMMARY:<br>==32608== definitely lost: 280 bytes in 20 blocks.<br>==32608== possibly lost: 272 bytes in 2 blocks.<br>==32608== still reachable: 199 bytes in 10 blocks.<br>==32608== suppressed: 0 bytes in 0 blocks.<br><br>My suspicious is this memory leak is happening at ber_memalloc_x (in /usr/libexec/slapd). Or in the part above :<br><br>==32608== 252 bytes in 18 blocks are definitely lost in loss record 8 of 9<br> ==32608== at 0x4004405: malloc (vg_replace_malloc.c:149)<br> ==32608== by 0x8150E2E: ber_memalloc_x (in /usr/libexec/slapd)<br><br>Since right now I do not have a large number of entrances in the DB the search is easily cached in memory and the response is very quick.<br><br>I will load again the DB but since this takes around 6-8hours I sent this initial result from valgrind since it can already give some directions. Tomorrow I should send more information related with a more heavy load DB.<br><br>I would like your help to see if this can identify something related with this memory leakage.<br><br>Please let me know if I can help in this meantime with something else.<br><br>Best regards,<br><br>Rodrigo.<span style="font-family: monospace;"><br><br></span> --- On <b>Sat, 12/20/08, Pierangelo Masarati <i><ando@sys-net.it></i></b> wrote:<br><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px;">From: Pierangelo Masarati <ando@sys-net.it><br>Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4<br>To: rlvcosta@yahoo.com<br>Cc: openldap-its@openldap.org<br>Date: Saturday, December 20, 2008, 7:05 PM<br><br><pre>rlvcosta@yahoo.com wrote:<br>> Full_Name: Rodrigo Costa<br>> Version: 2.4.11<br>> OS: RHEL4 U4<br>> URL: ftp://ftp.openldap.org/incoming/<br>> Submission from: (NULL) (201.82.161.220)<br>> <br>> <br>> Opeldap dev team,<br>> <br>> I'm facing an issue and after many tentatives I believe this is really<br>a bug<br>> that maybe could be identified and solved by openldap community. I tried<br>to<br>> include dmalloc into openldap 2.4.11 code but even I could compile with it<br>any<br>> tentative to execute the code generated "Segmentation Fault" so<br>I could not use<br>> dmalloc to identify where the memory leak is happening.<br><br>Can you try with valgrind?<br><br>p.<br><br><br>Ing. Pierangelo Masarati<br>OpenLDAP Core Team<br><br>SysNet s.r.l.<br>via Dossi, 8 - 27100 Pavia - ITALIA<br>http://www.sys-net.it<br>-----------------------------------<br>Office: +39 02 23998309<br>Mobile: +39 333 4963172<br>Fax: +39 0382 476497<br>Email: ando@sys-net.it<br>-----------------------------------<br><br></pre></blockquote></td></tr></table><br>
--0-1358063323-1230648878=:64021--
-
rlvcosta＠yahoo.com