Hello all,
I'm experiencing some problems with some OpenLDAP servers: the slapd process seems to always use more memory, eventually reaching a point where it has consumed all the available server memory and is killed by the OOM killer. The servers have 32gb of memory plus 32gb of swap space and are running Debian Lenny (with kernel 2.6.26-2-amd64), and we have compiled OpenLDAP 2.4.23 from source, and we're using Berkeley DB 4.6. The servers are dedicated to running OpenLDAP, so they don't have other processes that use a significant amount of memory.
This is a relatively high-volume environment, with 4 servers running with mirrormode to enable multi-master replication. The current database size is about 900mb. Is there any setting that could limit this memory usage? I don't feel this is "normal" considering our DB size and cache sizes.
These are my slapd.conf and DB_CONFIG files:
--------------------------------- slapd.conf --------------------------------- disallow bind_anon require authc
#== Schemas snipped ==
pidfile /usr/local/openldap/var/run/slapd.pid argsfile /usr/local/openldap/var/run/slapd.args
loglevel 16640
modulepath /usr/local/openldap/lib/ldap/ moduleload back_bdb moduleload ppolicy moduleload syncprov
threads 8
database config rootdn "cn=admin,cn=config" rootpw {SSHA}PASS
database monitor rootdn "cn=admin,cn=monitor" rootpw {SSHA}PASS
database bdb suffix "dc=corpldap,dc=mycompany" rootdn "cn=admin,dc=corpldap,dc=mycompany" rootpw {SSHA}PASS
directory /usr/local/openldap/var/corpldap-mycompany-data overlay ppolicy overlay syncprov ppolicy_hash_cleartext syncprov-checkpoint 100 10 syncprov-sessionlog 100 monitoring on lastmod on
checkpoint 512 10
cachesize 200000 idlcachesize 600000
#== Indexes and ACLs snipped ==
serverid 1 syncrepl rid=002 provider="ldap://server02:389" searchbase="dc=corpldap,dc=mycompany" type="refreshAndPersist" retry="30 10 60 15 600 +" bindmethod=simple binddn="cn=repuser3,ou=replica,dc=corpldap,dc=mycompany" credentials="PASSWORD"
#== other syncrepl entries snipped
mirrormode on
--------------------------------- DB_CONFIG ---------------------------------
set_lg_max 10485760 set_lg_regionmax 1048576 set_lg_bsize 2097152 set_lg_dir /var/ldap/corpldap-mycompany-log set_flags DB_LOG_AUTOREMOVE set_cachesize 0 2073741824 0 set_lk_max_objects 5000 set_lk_max_locks 5000 set_lk_max_lockers 5000
Diego Lima wrote:
Hello all,
I'm experiencing some problems with some OpenLDAP servers: the slapd process seems to always use more memory, eventually reaching a point where it has consumed all the available server memory and is killed by the OOM killer. The servers have 32gb of memory plus 32gb of swap space and are running Debian Lenny (with kernel 2.6.26-2-amd64), and we have compiled OpenLDAP 2.4.23 from source, and we're using Berkeley DB 4.6. The servers are dedicated to running OpenLDAP, so they don't have other processes that use a significant amount of memory.
This is a relatively high-volume environment, with 4 servers running with mirrormode to enable multi-master replication. The current database size is about 900mb. Is there any setting that could limit this memory usage? I don't feel this is "normal" considering our DB size and cache sizes.
Very likely ITS#6660, you need to update to 2.4.24.
--On Thursday, March 17, 2011 8:06 AM -0700 Howard Chu hyc@symas.com wrote:
This is a relatively high-volume environment, with 4 servers running with mirrormode to enable multi-master replication. The current database size is about 900mb. Is there any setting that could limit this memory usage? I don't feel this is "normal" considering our DB size and cache sizes.
Very likely ITS#6660, you need to update to 2.4.24.
Also make sure you use an alternative memory allocator if you are running on Linux, such as tcmalloc from the google perf tools package, or hoard.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
In regard to: Re: OpenLDAP Memory Usage, Quanah Gibson-Mount said (at...:
--On Thursday, March 17, 2011 8:06 AM -0700 Howard Chu hyc@symas.com wrote:
This is a relatively high-volume environment, with 4 servers running with mirrormode to enable multi-master replication. The current database size is about 900mb. Is there any setting that could limit this memory usage? I don't feel this is "normal" considering our DB size and cache sizes.
Very likely ITS#6660, you need to update to 2.4.24.
Also make sure you use an alternative memory allocator if you are running on Linux, such as tcmalloc from the google perf tools package, or hoard.
Can you expound a bit more on that, Quanah, or point me at some more complete information on why using a non-default memory allocator is beneficial for OpenLDAP on Linux?
I've been subscribed to the OpenLDAP devel and now technical mailing list for quite a while and I don't recall seeing anything stated in such strong terms about not using the default malloc/free on Linux.
Tim
--On Friday, March 18, 2011 6:29 PM -0500 Tim Mooney Tim.Mooney@ndsu.edu wrote:
In regard to: Re: OpenLDAP Memory Usage, Quanah Gibson-Mount said (at...:
--On Thursday, March 17, 2011 8:06 AM -0700 Howard Chu hyc@symas.com wrote:
This is a relatively high-volume environment, with 4 servers running with mirrormode to enable multi-master replication. The current database size is about 900mb. Is there any setting that could limit this memory usage? I don't feel this is "normal" considering our DB size and cache sizes.
Very likely ITS#6660, you need to update to 2.4.24.
Also make sure you use an alternative memory allocator if you are running on Linux, such as tcmalloc from the google perf tools package, or hoard.
Can you expound a bit more on that, Quanah, or point me at some more complete information on why using a non-default memory allocator is beneficial for OpenLDAP on Linux?
I've been subscribed to the OpenLDAP devel and now technical mailing list for quite a while and I don't recall seeing anything stated in such strong terms about not using the default malloc/free on Linux.
It's been discussed in detail several times, but not for a few years.
I suggest reading over:
http://www.openldap.org/pub/hyc/scale2007.pdf
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
In regard to: Re: OpenLDAP Memory Usage, Quanah Gibson-Mount said (at...:
--On Friday, March 18, 2011 6:29 PM -0500 Tim Mooney Tim.Mooney@ndsu.edu wrote:
In regard to: Re: OpenLDAP Memory Usage, Quanah Gibson-Mount said (at...:
--On Thursday, March 17, 2011 8:06 AM -0700 Howard Chu hyc@symas.com wrote:
This is a relatively high-volume environment, with 4 servers running with mirrormode to enable multi-master replication. The current database size is about 900mb. Is there any setting that could limit this memory usage? I don't feel this is "normal" considering our DB size and cache sizes.
Very likely ITS#6660, you need to update to 2.4.24.
Also make sure you use an alternative memory allocator if you are running on Linux, such as tcmalloc from the google perf tools package, or hoard.
Can you expound a bit more on that, Quanah, or point me at some more complete information on why using a non-default memory allocator is beneficial for OpenLDAP on Linux?
I've been subscribed to the OpenLDAP devel and now technical mailing list for quite a while and I don't recall seeing anything stated in such strong terms about not using the default malloc/free on Linux.
It's been discussed in detail several times, but not for a few years.
I suggest reading over:
Thanks Quanah, that was very helpful. Slides 23 and 33-35 (comparing the various memory allocators) were very useful, especially when coupled with the openldap-devel threads starting on August 24 2006.
I also found slide 26, about the problem with trying to benchmark OpenLDAP using slow bench mark tools, interesting. What tools are recommended these days, for benchmarking an OpenLDAP installation? The most promising ones I could find were SLAMD (which was panned in slide 26) and cbe4. Are there others that are noteworthy?
Thanks again,
Tim
Tim Mooney wrote:
In regard to: Re: OpenLDAP Memory Usage, Quanah Gibson-Mount said (at...:
I suggest reading over:
Thanks Quanah, that was very helpful. Slides 23 and 33-35 (comparing the various memory allocators) were very useful, especially when coupled with the openldap-devel threads starting on August 24 2006.
I also found slide 26, about the problem with trying to benchmark OpenLDAP using slow bench mark tools, interesting. What tools are recommended these days, for benchmarking an OpenLDAP installation? The most promising ones I could find were SLAMD (which was panned in slide 26) and cbe4. Are there others that are noteworthy?
SLAMD is still the most convenient to use, in terms of overall features. I just make sure that I have several times as much horsepower in the clients as in the target server, to generate sufficient load on the server. (This can make benching very powerful servers rather difficult. Sometimes I will also resort to simple timing of the OpenLDAP test clients when the client machines are otherwise too underpowered for SLAMD. I try to avoid that because then I'm stuck having to tabulate the results manually.)
openldap-technical@openldap.org
-
Diego Lima -
Howard Chu -
Quanah Gibson-Mount -
Tim Mooney