11:07 a.m.
https://bugs.openldap.org/show_bug.cgi?id=9365
Issue ID: 9365
Summary: Mem leaks with Æ-DIR providers
Product: OpenLDAP
Version: 2.4.53
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: michael(a)stroeder.com
Target Milestone: ---
Created attachment 772
--> https://bugs.openldap.org/attachment.cgi?id=772&action=edit
valgrind output on openSUSE Tumbleweed x86_64
An Æ-DIR installation with self-compiled OpenLDAP 2.4.53 on Debian (now
buster) has memory leak issues on the Æ-DIR providers. The read-only
consumers do not have this issue. The provider config is more complex
with more overlays and more ACLs.
In this production deployment slapd is automatically restarted (by monit) when
memory consumption reaches 80%. Thus monitoring clearly shows a frequent saw
tooth pattern.
I've also tested on openSUSE Tumbleweed x86_64 with a RE24 build [1] by running
slapd under control of valgrind for a couple of minutes continously sending
simple bind operations (additional to the monitoring and other back-ground jobs
running).
Find valgrind output of my first attempt attached.
Does that make sense at all?
--
You are receiving this mail because:
You are on the CC list for the issue.
11:09 a.m.
New subject: [Issue 9365] Mem leaks with Æ-DIR providers
https://bugs.openldap.org/show_bug.cgi?id=9365
--- Comment #1 from Michael Ströder <michael(a)stroeder.com> ---
[1] OpenLDAP packages used:
https://build.opensuse.org/package/show/home:stroeder:branches:home:stroe...
--
You are receiving this mail because:
You are on the CC list for the issue.
11:11 a.m.
New subject: [Issue 9365] Mem leaks with Æ-DIR providers
https://bugs.openldap.org/show_bug.cgi?id=9365
--- Comment #2 from Michael Ströder <michael(a)stroeder.com> ---
valgrind command used:
valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes
--fullpath-after= --keep-debuginfo=yes --verbose --log-file=valgrind-out.txt --
/usr/lib64/slapd -d stats -n ae-slapd -l LOCAL4 -s 7 -f
/opt/ae-dir/etc/openldap/slapd.conf -h
'ldapi://%2Fopt%2Fae-dir%2Frun%2Fslapd%2Fldapi ldap://*:389 ldaps://*:636' -o
slp=off -u ae-dir-slapd
--
You are receiving this mail because:
You are on the CC list for the issue.
11:16 a.m.
New subject: [Issue 9365] Mem leaks with Æ-DIR providers
https://bugs.openldap.org/show_bug.cgi?id=9365
--- Comment #3 from Michael Ströder <michael(a)stroeder.com> ---
Created attachment 773
--> https://bugs.openldap.org/attachment.cgi?id=773&action=edit
provider configuration
--
You are receiving this mail because:
You are on the CC list for the issue.
11:48 a.m.
New subject: [Issue 9365] Mem leaks with Æ-DIR providers
https://bugs.openldap.org/show_bug.cgi?id=9365
--- Comment #4 from Howard Chu <hyc(a)openldap.org> ---
Comment on attachment 772
--> https://bugs.openldap.org/attachment.cgi?id=772
valgrind output on openSUSE Tumbleweed x86_64
There are two items of interest here, one is an uninit'd variable, which was
fixed in master 3 years ago in commit 5bd89a1f1f1 but apparently never merged
into RE24.
The other appears to be a leak in regexec(), which would be a libc bug. But to
be sure - how many operations did you perform, before stopping slapd? The line
==7378== 47,104 bytes in 23 blocks are indirectly lost in loss record 92 of 93
would imply the leak occurred 23 times, and presumably was once per operation.
--
You are receiving this mail because:
You are on the CC list for the issue.
9:12 a.m.
New subject: [Issue 9365] Mem leaks with Æ-DIR providers
https://bugs.openldap.org/show_bug.cgi?id=9365
--- Comment #5 from Quanah Gibson-Mount <quanah(a)openldap.org> ---
Added the missed init in:
Commits:
• 854771cd
by Howard Chu at 2020-10-12T16:11:36+00:00
Cleanup uninit'd vars
--
You are receiving this mail because:
You are on the CC list for the issue.
2:57 p.m.
New subject: [Issue 9365] Mem leaks with Æ-DIR providers
https://bugs.openldap.org/show_bug.cgi?id=9365
--- Comment #6 from Michael Ströder <michael(a)stroeder.com> ---
I've retested with latest RE24 with the commit and the output still contains
this:
==898== 47,104 bytes in 23 blocks are indirectly lost in loss record 95 of 96
==898== at 0x483BB65: calloc (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==898== by 0x49FA5F5: build_trtable
(/usr/src/debug/glibc-2.32-1.1.x86_64/posix/regexec.c:3403)
==898== by 0x49FCC04: transit_state
(/usr/src/debug/glibc-2.32-1.1.x86_64/posix/regexec.c:2252)
==898== by 0x49FCC04: check_matching
(/usr/src/debug/glibc-2.32-1.1.x86_64/posix/regexec.c:1120)
==898== by 0x49FCC04: re_search_internal
(/usr/src/debug/glibc-2.32-1.1.x86_64/posix/regexec.c:792)
==898== by 0x4A0126B: regexec@@GLIBC_2.3.4
(/usr/src/debug/glibc-2.32-1.1.x86_64/posix/regexec.c:218)
==898== by 0x54DEA84: sock_over_op
(/usr/src/debug/openldap2-2.4.54-14.1.x86_64/servers/slapd/back-sock/config.c:301)
==898== by 0x1BAA39: overlay_op_walk
(/usr/src/debug/openldap2-2.4.54-14.1.x86_64/servers/slapd/backover.c:661)
==898== by 0x1BABAA: over_op_func
(/usr/src/debug/openldap2-2.4.54-14.1.x86_64/servers/slapd/backover.c:730)
==898== by 0x168230: fe_op_bind
(/usr/src/debug/openldap2-2.4.54-14.1.x86_64/servers/slapd/bind.c:383)
==898== by 0x167840: do_bind
(/usr/src/debug/openldap2-2.4.54-14.1.x86_64/servers/slapd/bind.c:205)
==898== by 0x1482C1: connection_operation.lto_priv.0
(/usr/src/debug/openldap2-2.4.54-14.1.x86_64/servers/slapd/connection.c:1175)
==898== by 0x148E3A: connection_read_thread
(/usr/src/debug/openldap2-2.4.54-14.1.x86_64/servers/slapd/connection.c:1311)
==898== by 0x4860C41: ldap_int_thread_pool_wrapper.part.0
(/usr/src/debug/openldap2-2.4.54-14.1.x86_64/libraries/libldap_r/tpool.c:696)
For the tests I've disabled background jobs and all other replicas and I've
sent simple bind operation 59 times.
So to me the count of 23 does not seem to correlate with the number of
operations.
--
You are receiving this mail because:
You are on the CC list for the issue.
3:26 p.m.
New subject: [Issue 9365] Mem leaks with Æ-DIR providers
https://bugs.openldap.org/show_bug.cgi?id=9365
--- Comment #7 from Howard Chu <hyc(a)openldap.org> ---
(In reply to Michael Ströder from comment #6)
I've retested with latest RE24 with the commit and the output
still contains
this:
==898== 47,104 bytes in 23 blocks are indirectly lost in loss record 95 of 96
==898== at 0x483BB65: calloc (in
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==898== by 0x49FA5F5: build_trtable
(/usr/src/debug/glibc-2.32-1.1.x86_64/posix/regexec.c:3403)
For the tests I've disabled background jobs and all other
replicas and I've
sent simple bind operation 59 times.
So to me the count of 23 does not seem to correlate with the number of
operations.
There's nothing else in this output that indicates a leak. If this is just some
regex-internal structure that needs 23 allocs to init itself, then we have to
look elsewhere. Probably you'll need to enable the consumers and generate some
activity on the provider.
--
You are receiving this mail because:
You are on the CC list for the issue.
6:38 a.m.
New subject: [Issue 9365] Mem leaks with Æ-DIR providers
https://bugs.openldap.org/show_bug.cgi?id=9365
--- Comment #8 from Michael Ströder <michael(a)stroeder.com> ---
Unfortunately this issue is still present, e.g. when running 2.6.1 on Ubuntu
20.04 (Focal) and eventually even 22.04 (Jammy).
The customer provided jemalloc profiler call-graphs from the running production
env. Are these interesting to look at?
--
You are receiving this mail because:
You are on the CC list for the issue.
7:18 a.m.
New subject: [Issue 9365] Mem leaks with Æ-DIR providers
https://bugs.openldap.org/show_bug.cgi?id=9365
--- Comment #9 from Howard Chu <hyc(a)openldap.org> ---
(In reply to Michael Ströder from comment #8)
Unfortunately this issue is still present, e.g. when running 2.6.1 on
Ubuntu
20.04 (Focal) and eventually even 22.04 (Jammy).
The customer provided jemalloc profiler call-graphs from the running
production env. Are these interesting to look at?
Unlikely. Memory profiles of a running process won't reveal actual leaks,
though it may show some areas of heavy memory use. Generally won't know that
something has actually leaked unless it is still unfreed after shutdown. I
still recommend using this leak tracer https://github.com/hyc/mleak/ since it
performs well even in production.
--
You are receiving this mail because:
You are on the CC list for the issue.
8:12 a.m.
New subject: [Issue 9365] Mem leaks with Æ-DIR providers
https://bugs.openldap.org/show_bug.cgi?id=9365
--- Comment #10 from Ondřej Kuzník <ondra(a)mistotebe.net> ---
On Wed, Apr 27, 2022 at 01:38:48PM +0000, openldap-its(a)openldap.org wrote:
Unfortunately this issue is still present, e.g. when running 2.6.1 on
Ubuntu
20.04 (Focal) and eventually even 22.04 (Jammy).
The customer provided jemalloc profiler call-graphs from the running production
env. Are these interesting to look at?
If you have several profiles from the same run as the memory use is
going up, pprof with --base should be able to show who the additional
memory belongs to. That might help pinpoint the code responsible.
Thanks,
--
You are receiving this mail because:
You are on the CC list for the issue.
6:40 a.m.
New subject: [Issue 9365] Mem leaks with Æ-DIR providers
https://bugs.openldap.org/show_bug.cgi?id=9365
--- Comment #11 from Michael Ströder <michael(a)stroeder.com> ---
Some tests show that the fix for ITS#9924 might be the solution.
I've published Debian/Ubuntu packages with this back-port patch and will wait
what customers' admins are reporting.
--
You are receiving this mail because:
You are on the CC list for the issue.
9:33 a.m.
New subject: [Issue 9365] Mem leaks with Æ-DIR providers
https://bugs.openldap.org/show_bug.cgi?id=9365
Quanah Gibson-Mount <quanah(a)openldap.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
See Also| |https://bugs.openldap.org/s
| |how_bug.cgi?id=9924
--
You are receiving this mail because:
You are on the CC list for the issue.
176
days inactive
899
days old
12 comments
1 participants
participants (1)
-
openldap-its＠openldap.org