12:15 a.m.
Hello (originally posted to -bugs list, but think this might be the correct list)
I have an openldap installation setup which appears to have a memory leak which I have not
been able to solve by upgrading software etc. which is why I'm submitting this, if
this is the wrong list, please let me know.
The setup is:
- OpenLDAP server running 2.4.26, compiled against db-5.2.28, heimdal 1.2.1.
- Server is SunOS servername 5.10 Generic_141445-09 i86pc i386 i86pc, 6GB of RAM and 5GB
of swap (in a VMWare ESX environment)
I have the following in my DB_CONFIG file:
set_cachesize 0 52428800 0
set_lg_regionmax 1048576
set_lg_max 10485760
set_lg_bsize 2097152
set_lg_dir /pack/openldap/var/openldap-crl-logs
In my slapd.conf I have the following entries (I stripped out some restricts and other
irrelevant lines)
include /pack/openldap/etc/openldap/schema/core.schema
include /pack/openldap/etc/openldap/schema/entrust.attributes.cfg
include /pack/openldap/etc/openldap/schema/entrust.objectclasses.cfg
idletimeout 300
pidfile /pack/openldap/var/run/slapd-crl.pid
argsfile /pack/openldap/var/run/slapd-crl.args
threads 4
database bdb
suffix "c=DK"
directory /pack/openldap/var/openldap-crl
index objectClass,entryCSN,entryUUID eq
checkpoint 128 1
dbnosync
overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 100
serverID 2
monitoring off
idletimeout 10
Just FYI, the server is a clone of another server currently running syncprov replication
between two nodes, hence the syncprov lines. This server has the same databases as the
production ones, but I just moved it to isolated area and removed all sync options from
slapd.
My database is rather big:
vs24n02z2.prl1:/pack/openldap/var/openldap-crl root$ du -sh *.bdb
961M dn2id.bdb
162M entryCSN.bdb
67M entryUUID.bdb
10G id2entry.bdb
4.0M objectClass.bdb
but I'd imagine this not beeing a problem (other than some performance issues, but
this is not the problem here).
Now, when I start up my slapd, it consumes around 6MB RAM, all is fine, I can perform
searches and updates without problems. But my memory consumption is going through the
roof!
If I perform a search through the part of the database having alot of data, I suddenly
find slapd using 500+ MB of memory...
In our production setup, we actually see that after a couple of weeks, slapd is consuming
3+ GB of memory, and then at some point hits a Out of memory, and crashes.
Our production setup is infact a openldap 2.4.22 with berkeley db 4.6.21, and my test
server (cloned in a vmware environment) has just upgraded this environment by this
recipe:
Using db-4.6.21
* db_checkpoint -1
* db_recover -h <path to bdb files>
install db-5.2.28
* Loop through all bdb files: db_upgrade <bdb file>
* install openldap 2.4.26 (compiled against db-5.2.28)
* start slapd
The server which updates the LDAP is performing many search and updates, which is probably
why memory consumption is exploding after restarts?
Now, my main concern is, how (if possible) can we procede with this? We really need to
find the problem as this causes production problems whenever the ldap crashes.
Regards
Thomas
8:35 a.m.
--On Thursday, August 25, 2011 9:15 AM +0200 Thomas Rasmussen
<rasmussen.thomas(a)gmail.com> wrote:
Now, my main concern is, how (if possible) can we procede with this?
We
really need to find the problem as this causes production problems
whenever the ldap crashes.
Use an alternative memory allocator such as tcmalloc.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
5:36 a.m.
On 26/08/2011, at 17.35, Quanah Gibson-Mount wrote:
--On Thursday, August 25, 2011 9:15 AM +0200 Thomas Rasmussen
<rasmussen.thomas(a)gmail.com> wrote:
> Now, my main concern is, how (if possible) can we procede with this? We
> really need to find the problem as this causes production problems
> whenever the ldap crashes.
Use an alternative memory allocator such as tcmalloc.
--Quanah
Is that truely the only solution to this problem? This is a major production system, and
changing memory allocator is not just something that we can do.
Regards
Thomas
9:54 a.m.
--On Tuesday, August 30, 2011 2:36 PM +0200 Thomas Rasmussen
<rasmussen.thomas(a)gmail.com> wrote:
Is that truly the only solution to this problem? This is a major
production system, and changing memory allocator is not just something
that we can do.
When I used Slowaris, it was really the only way to have a useful system.
The default memory allocator that ships with it is worthless. Pretty much
the same thing in Linux, too. It is very trivial to download and build
tcmalloc and set the environment variable to have slapd use it instead.
Probably about 10-15 minutes of your time.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
2:33 a.m.
On 30/08/2011, at 18.54, Quanah Gibson-Mount wrote:
--On Tuesday, August 30, 2011 2:36 PM +0200 Thomas Rasmussen
<rasmussen.thomas(a)gmail.com> wrote:
> Is that truly the only solution to this problem? This is a major
> production system, and changing memory allocator is not just something
> that we can do.
When I used Slowaris, it was really the only way to have a useful system. The default
memory allocator that ships with it is worthless. Pretty much the same thing in Linux,
too. It is very trivial to download and build tcmalloc and set the environment variable
to have slapd use it instead. Probably about 10-15 minutes of your time.
I have now given it a try, but still no luck, my SMF script does this:
LD_PRELOAD=/pack/google-perftools/lib/libtcmalloc_minimal.so /pack/openldap/libexec/slapd
-f /pack/openldap/etc/openldap/slapd-crl.conf -h ldap://0.0.0.0:389/ -l local1
After a restart and performing a ldapsearch slapd has allocated over 500MB of memory,
which is not exactly a good thing :-(
Regards
Thomas
10:29 a.m.
--On Wednesday, August 31, 2011 11:33 AM +0200 Thomas Rasmussen
<rasmussen.thomas(a)gmail.com> wrote:
After a restart and performing a ldapsearch slapd has allocated over
500MB of memory, which is not exactly a good thing :-(
How many total entries are in your database? What is your entrycache size?
I don't see that information in your stripped slapd.conf. Personally, I
suggest you never "strip" slapd.conf when sending it out, because you
aren't necessarily going to know what is relevant to what you are doing.
I think you are misunderstanding some of how OpenLDAP works. In addition
to the BDB cachesize (which you have configured as 50MB (which is likely
grossly small given the size of your database), there are also 3 additional
caches for back-bdb/back-hdb in slapd itself. Those are *not* filled until
you start doing searches. Those 3 caches are:
entrycache
idlcache
dn cache
My guess is it is growing to 500MB of memory because it is filling up the
DN cache. Given the 10GB size of your id2entry database, I'm guessing you
have a *lot* of DNs.
I.e., the only way you will know the total size of what the slapd process
*should* be is to start it, and then do an ldapsearch of the entire
database. Once that is finished, you'll get an approximate size of your DB
with most of the caches filled (although the idlcache may take more time to
fill).
Basically, I don't see 500MB as problematic, given the DB size you
reported. However, now that you have tcmalloc in place, the general size
of slapd should be significantly more stable.
Finally, I will note that using "db_upgrade" to upgrade an OpenLDAP
database has not, and it is unlikely every will be, supported. The *only*
supported method of migrating to a newer version of BDB is to slapcat the
original database and to slapadd it back in. So it is entirely possible
you have created a very interesting issue by doing a completely unsupported
operation.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
1:14 a.m.
On 31/08/2011, at 19.29, Quanah Gibson-Mount wrote:
--On Wednesday, August 31, 2011 11:33 AM +0200 Thomas Rasmussen
<rasmussen.thomas(a)gmail.com> wrote:
> After a restart and performing a ldapsearch slapd has allocated over
> 500MB of memory, which is not exactly a good thing :-(
How many total entries are in your database? What is your entrycache size? I don't
see that information in your stripped slapd.conf. Personally, I suggest you never
"strip" slapd.conf when sending it out, because you aren't necessarily going
to know what is relevant to what you are doing.
I have about 3 million DNs in the database, I was required by our customer to strip out
some of the information, but it was only the 'access' bits and the rootdn+rootpw
which is why I stripped some information.
I think you are misunderstanding some of how OpenLDAP works. In addition to the BDB
cachesize (which you have configured as 50MB (which is likely grossly small given the size
of your database), there are also 3 additional caches for back-bdb/back-hdb in slapd
itself. Those are *not* filled until you start doing searches. Those 3 caches are:
entrycache
idlcache
dn cache
OK, I was not aware that it would use memory for other caches that wasn't possible to
setup (or are they?)
My guess is it is growing to 500MB of memory because it is filling up the DN cache.
Given the 10GB size of your id2entry database, I'm guessing you have a *lot* of DNs.
I.e., the only way you will know the total size of what the slapd process *should* be is
to start it, and then do an ldapsearch of the entire database. Once that is finished,
you'll get an approximate size of your DB with most of the caches filled (although the
idlcache may take more time to fill).
I tried to perform an ldapsearch on cn=* on my database (on ldap compiled with gcc,
running BDB 5.2.x and tcmalloc enabled) and it allocated about 550MB of memory.
Basically, I don't see 500MB as problematic, given the DB size you reported.
However, now that you have tcmalloc in place, the general size of slapd should be
significantly more stable.
Finally, I will note that using "db_upgrade" to upgrade an OpenLDAP database
has not, and it is unlikely every will be, supported. The *only* supported method of
migrating to a newer version of BDB is to slapcat the original database and to slapadd it
back in. So it is entirely possible you have created a very interesting issue by doing a
completely unsupported operation.
Hmm, I found the db_upgrade process in the documentation of berkeley DB, so I'd
imagine that it was supported, but apparently not by openldap...
I will try to make a series of tests with my new-found knowledge about the way slapd will
eat memory... hopefully it will continue to behave (it seems to have stabilazed around
552MB)
Regards
Thomas
3:58 a.m.
On Thursday, 1 September 2011 10:14:12 Thomas Rasmussen wrote:
On 31/08/2011, at 19.29, Quanah Gibson-Mount wrote:
> --On Wednesday, August 31, 2011 11:33 AM +0200 Thomas Rasmussen
<rasmussen.thomas(a)gmail.com> wrote:
>> After a restart and performing a ldapsearch slapd has
allocated over
>> 500MB of memory, which is not exactly a good thing :-(
>
> How many total entries are in your database? What is your entrycache
> size? I don't see that information in your stripped slapd.conf.
> Personally, I suggest you never "strip" slapd.conf when sending it out,
> because you aren't necessarily going to know what is relevant to what
> you are doing.
I have about 3 million DNs in the database, I was required by our customer
to strip out some of the information, but it was only the 'access' bits
and the rootdn+rootpw which is why I stripped some information.
> I think you are misunderstanding some of how OpenLDAP works. In addition
> to the BDB cachesize (which you have configured as 50MB (which is likely
> grossly small given the size of your database), there are also 3
> additional caches for back-bdb/back-hdb in slapd itself. Those are
> *not* filled until you start doing searches. Those 3 caches are:
>
> entrycache
> idlcache
> dn cache
OK, I was not aware that it would use memory for other caches that wasn't
possible to setup (or are they?)
Have you read any documentation regarding tuning, at all?
$ man slapd-bdb|col -b|grep -A6 -E '^ *(idl|dn|)cache.*size'
cachesize <integer>
Specify the size in entries of the in-memory entry cache main‐
tained by the bdb or hdb backend database instance. The default
is 1000 entries.
cachefree <integer>
Specify the number of entries to free from the entry cache when
--
dncachesize <integer>
Specify the maximum number of DNs in the in-memory DN cache.
Ideally this cache should be large enough to contain the DNs of
every entry in the database. If set to a smaller value than the
cachesize it will be silently increased to equal the cachesize.
The default value is 0 which means unlimited, i.e. the DN cache
will grow without bound.
--
idlcachesize <integer>
Specify the size of the in-memory index cache, in index slots.
The default is zero. A larger value will speed up frequent
searches of indexed entries. An hdb database needs a large idl‐
cachesize for good search performance, typically three times the
cachesize (entry cache size) or larger.
Regards,
Buchan
4410
days inactive
4417
days old
openldap-technical@openldap.org
7 comments
3 participants
participants (3)
-
Buchan Milne -
Quanah Gibson-Mount -
Thomas Rasmussen