7:31 a.m.
https://bugs.openldap.org/show_bug.cgi?id=9924
Issue ID: 9924
Summary: Increased/RunAway memory usage slapo-deref
Product: OpenLDAP
Version: 2.5.13
Hardware: x86_64
OS: Linux
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: overlays
Assignee: bugs(a)openldap.org
Reporter: erikdewaard(a)gmail.com
Target Milestone: ---
Created attachment 916
--> https://bugs.openldap.org/attachment.cgi?id=916&action=edit
slapd.conf
Increased/RunAway memory usage slapo-deref
Running: 2.5.13
After enabling slapo-deref slapd memory usage increased and growing.
I can reproduce this on every consumer with deref enabled.
From:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
173229 ldap 20 0 26.6g 1.0g 941996 S 4.0 0.8 3674:19 slapd
To:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2745810 ldap 20 0 141.5g 115.8g 468940 S 3.0 92.5 312:42.93 slapd
How best to debug this?
I should probably recompile to get all symbols for slapd available.
#valgrind.sh
valgrind --leak-check=full \
--show-leak-kinds=all \
--extra-debuginfo-path=/usr/lib/debug/usr/lib64/openldap \
--allow-mismatched-debuginfo=yes \
--track-origins=yes \
--error-limit=no \
--verbose \
--log-file=valgrind-out.txt \
/usr/sbin/slapd -F /etc/openldap/slapd.d -u ldap -h "ldap:///
ldaps:/// ldapi:///"
#mleak.sh
LD_PRELOAD=/tmp/mleak/mleak.so \
/usr/sbin/slapd -F /etc/openldap/slapd.d -u ldap -h "ldap:/// ldaps:///
ldapi:///"
sent kill -2
#mleak_report.sh
./mdump /usr/sbin/slapd ml.*
./report.sh | more
fncdump: Cant open linux-vdso.so.1
Memory leaks (14480 total):
--
You are receiving this mail because:
You are on the CC list for the issue.
7:36 a.m.
New subject: [Issue 9924] Increased/RunAway memory usage slapo-deref
https://bugs.openldap.org/show_bug.cgi?id=9924
erikdewaard(a)gmail.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #916 is|0 |1
obsolete| |
--- Comment #1 from erikdewaard(a)gmail.com ---
Created attachment 917
--> https://bugs.openldap.org/attachment.cgi?id=917&action=edit
conf and valgrind output.
--
You are receiving this mail because:
You are on the CC list for the issue.
2:35 a.m.
New subject: [Issue 9924] Increased/RunAway memory usage slapo-deref
https://bugs.openldap.org/show_bug.cgi?id=9924
--- Comment #2 from Ondřej Kuzník <ondra(a)mistotebe.net> ---
Hi,
if you can isolate the traffic leading to the leak, would you mind preparing a
small script that reproduces the issue? Your configuration is quite involved so
rather than trying to fill in all the missing pieces to be able to run it, I
tried to isolate small bits of it but could not identify any leaks for searches
I've run (assuming it's a leak inside deref, which we should be able to confirm
once you've recompiled).
Thanks,
Ondrej
--
You are receiving this mail because:
You are on the CC list for the issue.
7:30 a.m.
New subject: [Issue 9924] Increased/RunAway memory usage slapo-deref
https://bugs.openldap.org/show_bug.cgi?id=9924
--- Comment #3 from Howard Chu <hyc(a)openldap.org> ---
You need to also use the valgrind option --keep-debuginfo=yes otherwise the
symbols for dynamically loaded modules are lost when they're unloaded at slapd
shutdown time.
--
You are receiving this mail because:
You are on the CC list for the issue.
8:05 a.m.
New subject: [Issue 9924] Increased/RunAway memory usage slapo-deref
https://bugs.openldap.org/show_bug.cgi?id=9924
Howard Chu <hyc(a)openldap.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Ever confirmed|0 |1
Status|UNCONFIRMED |IN_PROGRESS
--- Comment #4 from Howard Chu <hyc(a)openldap.org> ---
A fix is in https://git.openldap.org/openldap/openldap/-/merge_requests/568
please verify, thanks.
--
You are receiving this mail because:
You are on the CC list for the issue.
11:26 a.m.
New subject: [Issue 9924] Increased/RunAway memory usage slapo-deref
https://bugs.openldap.org/show_bug.cgi?id=9924
--- Comment #5 from Michael Ströder <michael(a)stroeder.com> ---
--- Comment #4 from Howard Chu <hyc(a)openldap.org> ---
A fix is in https://git.openldap.org/openldap/openldap/-/merge_requests/568
please verify, thanks.
I'd like to roll out this as back-port patch for Æ-DIR servers to check
whether this also fixes ITS#9365.
Your patch [1] applies cleanly to RE26 and seems to work on my local
test systems (with python-ldap0, ae-dir-tool, aehostd and web2ldap using
deref control). Anything else I should test before pushing this to
production?
[1]
https://code.stroeder.com/AE-DIR/debian-openldap-ms/src/branch/main/debia...
--
You are receiving this mail because:
You are on the CC list for the issue.
11:43 a.m.
New subject: [Issue 9924] Increased/RunAway memory usage slapo-deref
https://bugs.openldap.org/show_bug.cgi?id=9924
--- Comment #6 from Howard Chu <hyc(a)openldap.org> ---
(In reply to Michael Ströder from comment #5)
> --- Comment #4 from Howard Chu <hyc(a)openldap.org> ---
> A fix is in https://git.openldap.org/openldap/openldap/-/merge_requests/568
> please verify, thanks.
I'd like to roll out this as back-port patch for Æ-DIR servers to check
whether this also fixes ITS#9365.
Your patch [1] applies cleanly to RE26 and seems to work on my local
test systems (with python-ldap0, ae-dir-tool, aehostd and web2ldap using
deref control). Anything else I should test before pushing this to
production?
Triggering the leak requires first exhausting the per-thread tmpmem allocator,
so you need a fairly large search response set with a lot of returned deref
values. Have you verified that you can trigger the leak without the patch, and
that the leak is gone with the patch in place?
--
You are receiving this mail because:
You are on the CC list for the issue.
12:15 p.m.
New subject: [Issue 9924] Increased/RunAway memory usage slapo-deref
https://bugs.openldap.org/show_bug.cgi?id=9924
--- Comment #7 from Michael Ströder <michael(a)stroeder.com> ---
--- Comment #6 from Howard Chu <hyc(a)openldap.org> ---
Triggering the leak requires first exhausting the per-thread tmpmem allocator,
so you need a fairly large search response set with a lot of returned deref
values.
What does "fairly large" mean? A few thousand entries? Or a millions?
--
You are receiving this mail because:
You are on the CC list for the issue.
1:55 p.m.
New subject: [Issue 9924] Increased/RunAway memory usage slapo-deref
https://bugs.openldap.org/show_bug.cgi?id=9924
--- Comment #8 from erikdewaard(a)gmail.com ---
(In reply to Howard Chu from comment #4)
A fix is in
https://git.openldap.org/openldap/openldap/-/merge_requests/568
please verify, thanks.
Almost an hour in and looking very promising, verified.
#Without Patch
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1539782 ldap 20 0 26.2g 1.4g 729904 S 17.6 1.1 0:49.58 slapd
#With Patch
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3241557 ldap 20 0 25.6g 781948 734952 S 1.3 0.6 0:42.91 slapd
--
You are receiving this mail because:
You are on the CC list for the issue.
4:52 p.m.
New subject: [Issue 9924] Increased/RunAway memory usage slapo-deref
https://bugs.openldap.org/show_bug.cgi?id=9924
--- Comment #9 from Howard Chu <hyc(a)openldap.org> ---
(In reply to Michael Ströder from comment #7)
> --- Comment #6 from Howard Chu <hyc(a)openldap.org> ---
> Triggering the leak requires first exhausting the per-thread tmpmem allocator,
> so you need a fairly large search response set with a lot of returned deref
> values.
What does "fairly large" mean? A few thousand entries? Or a millions?
The tmpmem allocator can handle up to 1MB by default, so the size of a single
search response entry plus all of the deref values must consume more than 1MB
to trigger the leak.
--
You are receiving this mail because:
You are on the CC list for the issue.
4:59 p.m.
New subject: [Issue 9924] Increased/RunAway memory usage slapo-deref
https://bugs.openldap.org/show_bug.cgi?id=9924
--- Comment #10 from Howard Chu <hyc(a)openldap.org> ---
(In reply to Howard Chu from comment #3)
You need to also use the valgrind option --keep-debuginfo=yes
otherwise the
symbols for dynamically loaded modules are lost when they're unloaded at
slapd shutdown time.
(Reminded me that we worked over 14 years to get this option added to
valgrind... https://bugs.kde.org/show_bug.cgi?id=79362 )
--
You are receiving this mail because:
You are on the CC list for the issue.
6:34 a.m.
New subject: [Issue 9924] Increased/RunAway memory usage slapo-deref
https://bugs.openldap.org/show_bug.cgi?id=9924
--- Comment #11 from Michael Ströder <michael(a)stroeder.com> ---
On 9/30/22 01:52, openldap-its(a)openldap.org wrote:
https://bugs.openldap.org/show_bug.cgi?id=9924
The tmpmem allocator can handle up to 1MB by default, so the size of a single
search response entry plus all of the deref values must consume more than 1MB
to trigger the leak.
It seems an Æ-DIR reporting script can trigger this (with 4000 aeUser
entries referencing 4000 aePerson entries).
--
You are receiving this mail because:
You are on the CC list for the issue.
9:31 a.m.
New subject: [Issue 9924] Increased/RunAway memory usage slapo-deref
https://bugs.openldap.org/show_bug.cgi?id=9924
Quanah Gibson-Mount <quanah(a)openldap.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Keywords|needs_review |
Target Milestone|--- |2.5.14
--
You are receiving this mail because:
You are on the CC list for the issue.
9:33 a.m.
New subject: [Issue 9924] Increased/RunAway memory usage slapo-deref
https://bugs.openldap.org/show_bug.cgi?id=9924
Quanah Gibson-Mount <quanah(a)openldap.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
See Also| |https://bugs.openldap.org/s
| |how_bug.cgi?id=9365
--
You are receiving this mail because:
You are on the CC list for the issue.
9:39 a.m.
New subject: [Issue 9924] Increased/RunAway memory usage slapo-deref
https://bugs.openldap.org/show_bug.cgi?id=9924
Quanah Gibson-Mount <quanah(a)openldap.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|IN_PROGRESS |RESOLVED
Resolution|--- |FIXED
--- Comment #12 from Quanah Gibson-Mount <quanah(a)openldap.org> ---
head:
• e640ce28
by Howard Chu at 2022-09-29T21:44:25+00:00
ITS#9924 slapo-deref: plug memleak
RE26:
• 40fb9a90
by Howard Chu at 2022-10-03T16:35:10+00:00
ITS#9924 slapo-deref: plug memleak
RE25:
• 1f99099c
by Howard Chu at 2022-10-03T16:36:02+00:00
ITS#9924 slapo-deref: plug memleak
--
You are receiving this mail because:
You are on the CC list for the issue.
8:10 a.m.
New subject: [Issue 9924] Increased/RunAway memory usage slapo-deref
https://bugs.openldap.org/show_bug.cgi?id=9924
Quanah Gibson-Mount <quanah(a)openldap.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Assignee|bugs(a)openldap.org |hyc(a)openldap.org
--
You are receiving this mail because:
You are on the CC list for the issue.
5
days inactive
133
days old
15 comments
1 participants
participants (1)
-
openldap-its＠openldap.org