We've seen the slapd's on our mail slaves trying to grow greater than the 2 gigabyte resident set size. I spent two days w/ valgrind looking for memory leaks, but I failed to find anything. I finally discovered that repeatedly binding and unbinding using a SASL/ GSSAPI bind would cause slapd's resident set size and vm size to grow. So I wrote a test program to SASL Bind, issue a search, and unbind. The test program would do this sequence 10000 times before it stopped. The test program always issues the same exact search trying to eliminate any interferance from a growing entry cache and/or idlcache.
I also tried this test w/ OpenLDAP 2.3.30 and got similar results.
Initial program size for slapd RSS = 8392 VMSIZE = 180180
"Bind, search, unbind" memory usage.
Searches Anonymous Bind SASL/GSSAPI Bind Completed RSS VMSIZE RSS VMSIZE 10000 9320 182760 12560 184916 20000 9320 182760 15404 187988 30000 9376 187888 18268 196188 40000 9376 187888 21080 199260 50000 9380 187888 23896 201308 60000 9380 187888 26708 204380 70000 9380 187888 29524 207452 80000 9380 187888 32336 210524 90000 9380 187888 35148 212572 100000 9380 187888 37972 215644
And when I only bind once for every 10,00 searches, I get a slapd memory usage like:
Searches Anonymous Bind SASL/GSSAPI Bind Completed RSS VMSIZE RSS VMSIZE 10000 9308 181736 9788 181856 20000 9308 181736 9788 181856 30000 9308 181736 9788 181856 40000 9316 181736 9788 181856 50000 9316 181736 9788 181856 60000 9316 181736 9796 181856 70000 9316 181736 9796 181856 80000 9316 181736 9800 181856 90000 9316 181736 9800 181856 100000 9316 181736 9800 181856
Current Software Versions Linux kernal 2.6.17.6 OpenLDAP 2.3.27 also tried 2.3.30 w/ similar results cyrus SASL 2.1.21 Heimdal 0.7.2 Berkeley 4.2
slapd.conf loglevel 768
include /etc/openldap/schema/core.schema include /etc/openldap/schema/cosine.schema include /etc/openldap/schema/inetorgperson.schema include /etc/openldap/schema/nis.schema include /etc/openldap/schema/local.schema include /etc/openldap/slapd.acl.prod # default settings sizelimit 350 timelimit 60 idletimeout 300 defaultsearchbase "dc=umich,dc=edu" pidfile /etc/openldap/slapd.slave.pid argsfile /etc/openldap/slapd.slave.args # Allow these features allow bind_v2 bind_anon_cred bind_anon_dn threads 32 include /etc/openldap/slapd.tls # SASL / GSSAPI / Kerberos sasl-realm UMICH.EDU
saslRegexp uid=simta/mx.umich.edu,cn=umich.edu,cn=gssapi,cn=auth cn=simta,ou=mail agents,ou=security,dc=umich,dc=edu
saslRegexp uid=(.*),cn=umich.edu,cn=gssapi,cn=auth uid=$1,ou=people,dc=umich,dc=edu
saslRegexp dc=edu,dc=umich,ou=security,ou=(.*),cn=(.*) cn=$2,ou=$1,ou=security,dc=umich,dc=edu
# Proxy authorization checks saslAuthzFrom in destination entry sasl-authz-policy from
include /etc/openldap/slapd.sasl.hostname
TLSCertificateFile /etc/openssl/private/ldap-dev_itd_umich_edu.crt TLSCertificateKeyFile /etc/openssl/private/ldap-dev.itd.umich.edu.key
# database settings database bdb suffix "dc=umich,dc=edu" rootdn xxxxxxxx rootpw xxxxxxxx
limits dn.exact="cn=batch update,ou=security,dc=umich,dc=edu" size.soft=unlimited size.hard=soft time.soft=unlimited time.hard=soft
limits users size.soft=350 size.hard=soft size.unchecked=1000 time.soft=60 time.hard=soft
limits anonymous size.soft=50 size.hard=soft size.unchecked=1000 time.soft=5 time.hard=soft
directory /var/slapd/tmp/db/db.slave cachesize 100000 checkpoint 2048 5 idlcachesize 20000 include /etc/openldap/slapd.index
# Monitor Backend database monitor
BerkeleyDB DB_CONFIG file set_lk_max_locks 25000 set_lk_max_objects 25000
set_cachesize 0 128000000 1
# Set transaction log buffer size to 2 megs set_lg_bsize 2097152
Any ideas?
Thanks, Paul Turgyan University of Michigan
--On Thursday, December 14, 2006 3:55 PM -0500 Paul Turgyan pturgyan@umich.edu wrote:
We've seen the slapd's on our mail slaves trying to grow greater than the 2 gigabyte resident set size. I spent two days w/ valgrind looking for memory leaks, but I failed to find anything. I finally discovered that repeatedly binding and unbinding using a SASL/ GSSAPI bind would cause slapd's resident set size and vm size to grow. So I wrote a test program to SASL Bind, issue a search, and unbind. The test program would do this sequence 10000 times before it stopped. The test program always issues the same exact search trying to eliminate any interferance from a growing entry cache and/or idlcache.
Interesting... I've been using SASL/GSSAPI with OpenLDAP for several years, and I agree I see memory growth in the loaded slapd, but it also stabilizes for me after a point. Have you looked at using any alternative memory allocators? I use Hoard instead of the default on my servers, and that has drastically improved memory allocation on my multi-CPU servers. You don't say how many CPU's you've got.
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
On Dec 14, 2006, at 4:14 PM, Quanah Gibson-Mount wrote:
--On Thursday, December 14, 2006 3:55 PM -0500 Paul Turgyan pturgyan@umich.edu wrote:
We've seen the slapd's on our mail slaves trying to grow greater than the 2 gigabyte resident set size. I spent two days w/ valgrind looking for memory leaks, but I failed to find anything. I finally discovered that repeatedly binding and unbinding using a SASL/ GSSAPI bind would cause slapd's resident set size and vm size to grow. So I wrote a test program to SASL Bind, issue a search, and unbind. The test program would do this sequence 10000 times before it stopped. The test program always issues the same exact search trying to eliminate any interferance from a growing entry cache and/or idlcache.
Interesting... I've been using SASL/GSSAPI with OpenLDAP for several years, and I agree I see memory growth in the loaded slapd, but it also stabilizes for me after a point. Have you looked at using any alternative memory allocators? I use Hoard instead of the default on my servers, and that has drastically improved memory allocation on my multi-CPU servers. You don't say how many CPU's you've got.
We have 2 cpu's w/ hyperthreading, so it kinda looks like 4 cpu's.
--On Thursday, December 14, 2006 4:27 PM -0500 Paul Turgyan pturgyan@umich.edu wrote:
Interesting... I've been using SASL/GSSAPI with OpenLDAP for several years, and I agree I see memory growth in the loaded slapd, but it also stabilizes for me after a point. Have you looked at using any alternative memory allocators? I use Hoard instead of the default on my servers, and that has drastically improved memory allocation on my multi-CPU servers. You don't say how many CPU's you've got.
We have 2 cpu's w/ hyperthreading, so it kinda looks like 4 cpu's.
It is well established that glibc's default malloc stuff has suffers badly from fragmentation on multiple CPU's. I highly advise using Hoard as your allocator and re-testing. ;)
You may want to read over:
http://www.openldap.org/lists/openldap-devel/200611/msg00004.html
and the other messages in this thread.
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
Paul Turgyan wrote:
We've seen the slapd's on our mail slaves trying to grow greater than the 2 gigabyte resident set size. I spent two days w/ valgrind looking for memory leaks, but I failed to find anything. I finally discovered that repeatedly binding and unbinding using a SASL/ GSSAPI bind would cause slapd's resident set size and vm size to grow. So I wrote a test program to SASL Bind, issue a search, and unbind. The test program would do this sequence 10000 times before it stopped. The test program always issues the same exact search trying to eliminate any interferance from a growing entry cache and/or idlcache.
If valgrind doesn't show anything then you're going to have to get more invasive. One possibility is to test again with a stripped down configuration. Just measure the memory use from a single Bind and Unbind, with no search operation in between. Or just query the rootDSE. You can test this with a practically empty slapd.conf file (since the config database always exists).
Change the slapd shutdown handler to just call exit() immediately. That will prevent anything from cleaning up. It will give valgrind plenty of leaks to complain about. Compare a session using a single anonymous bind vs a session using a single SASL/GSSAPI bind. That will show you where the discrepancy in memory use came from.
openldap-software@openldap.org
-
Howard Chu -
Paul Turgyan -
Quanah Gibson-Mount