HI!
An Æ-DIR installation with self-compiled OpenLDAP 2.4.53 on Debian (now buster) has memory leak issues on the Æ-DIR providers. The read-only consumers do not have this issue. The provider config is more complex with more overlays.
slapd is automatically restarted (by monit) when memory consumption reaches 80%. Thus monitoring clearly shows a frequent saw tooth pattern.
I'm experimenting with valgrind to track down the issue:
valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes --verbose --log-file=valgrind-out.txt -- /opt/openldap-ms/libexec/slapd [...]
Does that make sense or are some other command-line options needed?
Ciao, Michael.
Michael Ströder wrote:
HI!
An Æ-DIR installation with self-compiled OpenLDAP 2.4.53 on Debian (now buster) has memory leak issues on the Æ-DIR providers. The read-only consumers do not have this issue. The provider config is more complex with more overlays.
slapd is automatically restarted (by monit) when memory consumption reaches 80%. Thus monitoring clearly shows a frequent saw tooth pattern.
I'm experimenting with valgrind to track down the issue:
valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes --verbose --log-file=valgrind-out.txt -- /opt/openldap-ms/libexec/slapd [...]
Does that make sense or are some other command-line options needed?
I've usually just used --leak-check=full, never needed any other options.
But valgrind runs about 50x slower than realtime, I'd recommend using https://github.com/hyc/mleak/ instead. It's faster than all other leak checkers, makes it practical to use in near production-level loads.
Howard Chu wrote:
Michael Ströder wrote:
HI!
An Æ-DIR installation with self-compiled OpenLDAP 2.4.53 on Debian (now buster) has memory leak issues on the Æ-DIR providers. The read-only consumers do not have this issue. The provider config is more complex with more overlays.
slapd is automatically restarted (by monit) when memory consumption reaches 80%. Thus monitoring clearly shows a frequent saw tooth pattern.
I'm experimenting with valgrind to track down the issue:
valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes --verbose --log-file=valgrind-out.txt -- /opt/openldap-ms/libexec/slapd [...]
Does that make sense or are some other command-line options needed?
I've usually just used --leak-check=full, never needed any other options.
But valgrind runs about 50x slower than realtime, I'd recommend using https://github.com/hyc/mleak/ instead. It's faster than all other leak checkers, makes it practical to use in near production-level loads.
One other consideration - both of these tools wait until the target process exits cleanly, before producing results. If the leaks are occurring in a loaded module, you won't get any symbol info because slapd unloads all modules before exiting. To get useful results you need to comment out the module unloading code (in slapd/module.c). And of course, compile with debug symbols enabled, and best to turn optimization off.
On 10/9/20 11:23 PM, Howard Chu wrote:
Howard Chu wrote:
Michael Ströder wrote:
An Æ-DIR installation with self-compiled OpenLDAP 2.4.53 on Debian (now buster) has memory leak issues on the Æ-DIR providers. [..] I'm experimenting with valgrind to track down the issue:
valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes --verbose --log-file=valgrind-out.txt -- /opt/openldap-ms/libexec/slapd [...]
I've usually just used --leak-check=full, never needed any other options.
But valgrind runs about 50x slower than realtime, I'd recommend using https://github.com/hyc/mleak/ instead. It's faster than all other leak checkers, makes it practical to use in near production-level loads.
One other consideration - both of these tools wait until the target process exits cleanly, before producing results. If the leaks are occurring in a loaded module, you won't get any symbol info because slapd unloads all modules before exiting.
Ah yes, I see the ??? in valgrind output.
Thanks for the hint.
To get useful results you need to comment out the module unloading code (in slapd/module.c).
Like this for RE24 branch?
https://build.opensuse.org/package/view_file/home:stroeder:branches:home:str...
And of course, compile with debug symbols enabled, and best to turn optimization off.
Yes, debug symbols are already packaged anyway.
Ciao, Michael.
Michael Ströder wrote:
On 10/9/20 11:23 PM, Howard Chu wrote:
Howard Chu wrote:
Michael Ströder wrote:
An Æ-DIR installation with self-compiled OpenLDAP 2.4.53 on Debian (now buster) has memory leak issues on the Æ-DIR providers. [..] I'm experimenting with valgrind to track down the issue:
valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes --verbose --log-file=valgrind-out.txt -- /opt/openldap-ms/libexec/slapd [...]
I've usually just used --leak-check=full, never needed any other options.
But valgrind runs about 50x slower than realtime, I'd recommend using https://github.com/hyc/mleak/ instead. It's faster than all other leak checkers, makes it practical to use in near production-level loads.
One other consideration - both of these tools wait until the target process exits cleanly, before producing results. If the leaks are occurring in a loaded module, you won't get any symbol info because slapd unloads all modules before exiting.
Ah yes, I see the ??? in valgrind output.
Thanks for the hint.
To get useful results you need to comment out the module unloading code (in slapd/module.c).
Like this for RE24 branch?
https://build.opensuse.org/package/view_file/home:stroeder:branches:home:str...
Yeah that should work.
Also if your valgrind is new enough, you could just use --keep-debuginfo=yes and not have to patch anything. That's a feature that I worked on 14 years ago that they finally supported recently. https://bugs.kde.org/show_bug.cgi?id=79362
On 10/10/20 12:55 PM, Howard Chu wrote:
Also if your valgrind is new enough, you could just use --keep-debuginfo=yes and not have to patch anything.
Ah great. This is really helpful. Thanks!
openSUSE Tumbleweed has valgrind 3.16, Debian buster has version 3.14.
Together with --fullpath-after= it produces even better output.
Ciao, Michael.
openldap-technical@openldap.org
-
Howard Chu -
Michael Ströder