7:16 a.m.
Hello all,
I'm experiencing some problems with some OpenLDAP servers: the slapd
process seems to always use more memory, eventually reaching a point
where it has consumed all the available server memory and is killed by
the OOM killer. The servers have 32gb of memory plus 32gb of swap
space and are running Debian Lenny (with kernel 2.6.26-2-amd64), and
we have compiled OpenLDAP 2.4.23 from source, and we're using Berkeley
DB 4.6. The servers are dedicated to running OpenLDAP, so they don't
have other processes that use a significant amount of memory.
This is a relatively high-volume environment, with 4 servers running
with mirrormode to enable multi-master replication. The current
database size is about 900mb. Is there any setting that could limit
this memory usage? I don't feel this is "normal" considering our DB
size and cache sizes.
These are my slapd.conf and DB_CONFIG files:
--------------------------------- slapd.conf ---------------------------------
disallow bind_anon
require authc
#== Schemas snipped ==
pidfile /usr/local/openldap/var/run/slapd.pid
argsfile /usr/local/openldap/var/run/slapd.args
loglevel 16640
modulepath /usr/local/openldap/lib/ldap/
moduleload back_bdb
moduleload ppolicy
moduleload syncprov
threads 8
database config
rootdn "cn=admin,cn=config"
rootpw {SSHA}PASS
database monitor
rootdn "cn=admin,cn=monitor"
rootpw {SSHA}PASS
database bdb
suffix "dc=corpldap,dc=mycompany"
rootdn "cn=admin,dc=corpldap,dc=mycompany"
rootpw {SSHA}PASS
directory /usr/local/openldap/var/corpldap-mycompany-data
overlay ppolicy
overlay syncprov
ppolicy_hash_cleartext
syncprov-checkpoint 100 10
syncprov-sessionlog 100
monitoring on
lastmod on
checkpoint 512 10
cachesize 200000
idlcachesize 600000
#== Indexes and ACLs snipped ==
serverid 1
syncrepl rid=002
provider="ldap://server02:389"
searchbase="dc=corpldap,dc=mycompany"
type="refreshAndPersist"
retry="30 10 60 15 600 +"
bindmethod=simple
binddn="cn=repuser3,ou=replica,dc=corpldap,dc=mycompany"
credentials="PASSWORD"
#== other syncrepl entries snipped
mirrormode on
--------------------------------- DB_CONFIG ---------------------------------
set_lg_max 10485760
set_lg_regionmax 1048576
set_lg_bsize 2097152
set_lg_dir /var/ldap/corpldap-mycompany-log
set_flags DB_LOG_AUTOREMOVE
set_cachesize 0 2073741824 0
set_lk_max_objects 5000
set_lk_max_locks 5000
set_lk_max_lockers 5000
--
Diego Lima
http://www.diegolima.org
8:06 a.m.
Diego Lima wrote:
Hello all,
I'm experiencing some problems with some OpenLDAP servers: the slapd
process seems to always use more memory, eventually reaching a point
where it has consumed all the available server memory and is killed by
the OOM killer. The servers have 32gb of memory plus 32gb of swap
space and are running Debian Lenny (with kernel 2.6.26-2-amd64), and
we have compiled OpenLDAP 2.4.23 from source, and we're using Berkeley
DB 4.6. The servers are dedicated to running OpenLDAP, so they don't
have other processes that use a significant amount of memory.
This is a relatively high-volume environment, with 4 servers running
with mirrormode to enable multi-master replication. The current
database size is about 900mb. Is there any setting that could limit
this memory usage? I don't feel this is "normal" considering our DB
size and cache sizes.
Very likely ITS#6660, you need to update to 2.4.24.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
9:11 a.m.
--On Thursday, March 17, 2011 8:06 AM -0700 Howard Chu <hyc(a)symas.com>
wrote:
> This is a relatively high-volume environment, with 4 servers
running
> with mirrormode to enable multi-master replication. The current
> database size is about 900mb. Is there any setting that could limit
> this memory usage? I don't feel this is "normal" considering our DB
> size and cache sizes.
Very likely ITS#6660, you need to update to 2.4.24.
Also make sure you use an alternative memory allocator if you are running
on Linux, such as tcmalloc from the google perf tools package, or hoard.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
4:29 p.m.
In regard to: Re: OpenLDAP Memory Usage, Quanah Gibson-Mount said (at...:
--On Thursday, March 17, 2011 8:06 AM -0700 Howard Chu
<hyc(a)symas.com> wrote:
>> This is a relatively high-volume environment, with 4 servers running
>> with mirrormode to enable multi-master replication. The current
>> database size is about 900mb. Is there any setting that could limit
>> this memory usage? I don't feel this is "normal" considering our
DB
>> size and cache sizes.
>
> Very likely ITS#6660, you need to update to 2.4.24.
Also make sure you use an alternative memory allocator if you are running on
Linux, such as tcmalloc from the google perf tools package, or hoard.
Can you expound a bit more on that, Quanah, or point me at some more
complete information on why using a non-default memory allocator is
beneficial for OpenLDAP on Linux?
I've been subscribed to the OpenLDAP devel and now technical mailing list
for quite a while and I don't recall seeing anything stated in such strong
terms about not using the default malloc/free on Linux.
Tim
--
Tim Mooney Tim.Mooney(a)ndsu.edu
Enterprise Computing & Infrastructure 701-231-1076 (Voice)
Room 242-J6, IACC Building 701-231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164
12:17 p.m.
--On Friday, March 18, 2011 6:29 PM -0500 Tim Mooney <Tim.Mooney(a)ndsu.edu>
wrote:
In regard to: Re: OpenLDAP Memory Usage, Quanah Gibson-Mount said
(at...:
> --On Thursday, March 17, 2011 8:06 AM -0700 Howard Chu <hyc(a)symas.com>
> wrote:
>
>
>>> This is a relatively high-volume environment, with 4 servers running
>>> with mirrormode to enable multi-master replication. The current
>>> database size is about 900mb. Is there any setting that could limit
>>> this memory usage? I don't feel this is "normal" considering
our DB
>>> size and cache sizes.
>>
>> Very likely ITS#6660, you need to update to 2.4.24.
>
> Also make sure you use an alternative memory allocator if you are
> running on Linux, such as tcmalloc from the google perf tools package,
> or hoard.
Can you expound a bit more on that, Quanah, or point me at some more
complete information on why using a non-default memory allocator is
beneficial for OpenLDAP on Linux?
I've been subscribed to the OpenLDAP devel and now technical mailing list
for quite a while and I don't recall seeing anything stated in such strong
terms about not using the default malloc/free on Linux.
It's been discussed in detail several times, but not for a few years.
I suggest reading over:
<http://www.openldap.org/pub/hyc/scale2007.pdf>
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
4:34 p.m.
In regard to: Re: OpenLDAP Memory Usage, Quanah Gibson-Mount said (at...:
--On Friday, March 18, 2011 6:29 PM -0500 Tim Mooney
<Tim.Mooney(a)ndsu.edu>
wrote:
> In regard to: Re: OpenLDAP Memory Usage, Quanah Gibson-Mount said (at...:
>
>> --On Thursday, March 17, 2011 8:06 AM -0700 Howard Chu <hyc(a)symas.com>
>> wrote:
>>
>>
>>>> This is a relatively high-volume environment, with 4 servers running
>>>> with mirrormode to enable multi-master replication. The current
>>>> database size is about 900mb. Is there any setting that could limit
>>>> this memory usage? I don't feel this is "normal"
considering our DB
>>>> size and cache sizes.
>>>
>>> Very likely ITS#6660, you need to update to 2.4.24.
>>
>> Also make sure you use an alternative memory allocator if you are
>> running on Linux, such as tcmalloc from the google perf tools package,
>> or hoard.
>
> Can you expound a bit more on that, Quanah, or point me at some more
> complete information on why using a non-default memory allocator is
> beneficial for OpenLDAP on Linux?
>
> I've been subscribed to the OpenLDAP devel and now technical mailing list
> for quite a while and I don't recall seeing anything stated in such strong
> terms about not using the default malloc/free on Linux.
It's been discussed in detail several times, but not for a few years.
I suggest reading over:
<http://www.openldap.org/pub/hyc/scale2007.pdf>
Thanks Quanah, that was very helpful. Slides 23 and 33-35 (comparing
the various memory allocators) were very useful, especially when coupled
with the openldap-devel threads starting on August 24 2006.
I also found slide 26, about the problem with trying to benchmark OpenLDAP
using slow bench mark tools, interesting. What tools are recommended
these days, for benchmarking an OpenLDAP installation? The most promising
ones I could find were SLAMD (which was panned in slide 26) and cbe4.
Are there others that are noteworthy?
Thanks again,
Tim
--
Tim Mooney mooney(a)dogbert.cc.ndsu.NoDak.edu
Enterprise Computing & Infrastructure 701-231-1076 (Voice)
Room 242-J6, IACC Building 701-231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164
12:47 a.m.
Tim Mooney wrote:
In regard to: Re: OpenLDAP Memory Usage, Quanah Gibson-Mount said
(at...:
> I suggest reading over:
>
> <http://www.openldap.org/pub/hyc/scale2007.pdf>
Thanks Quanah, that was very helpful. Slides 23 and 33-35 (comparing
the various memory allocators) were very useful, especially when coupled
with the openldap-devel threads starting on August 24 2006.
I also found slide 26, about the problem with trying to benchmark OpenLDAP
using slow bench mark tools, interesting. What tools are recommended
these days, for benchmarking an OpenLDAP installation? The most promising
ones I could find were SLAMD (which was panned in slide 26) and cbe4.
Are there others that are noteworthy?
SLAMD is still the most convenient to use, in terms of overall features. I
just make sure that I have several times as much horsepower in the clients as
in the target server, to generate sufficient load on the server. (This can
make benching very powerful servers rather difficult. Sometimes I will also
resort to simple timing of the OpenLDAP test clients when the client machines
are otherwise too underpowered for SLAMD. I try to avoid that because then I'm
stuck having to tabulate the results manually.)
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
4276
days inactive
4281
days old
openldap-technical@openldap.org
6 comments
4 participants
participants (4)
-
Diego Lima -
Howard Chu -
Quanah Gibson-Mount -
Tim Mooney