Quanah,
My initial attempt was using openldap2.4.11 since this is the stable version. At this version Berkeley DB 4.7 isn't supported and then I used the BDB 4.6 with all patches.
At this configuration, having the DB_CONFIG setup for 50MB, and even having the idlcache and cache sizes defined I continue to see the memory being allocated and never released by slapd.
Now I tried to evolve my OS to CentOS5.2 and then see if it could be an issue with my compilation using a community compiled openldap version that could have syncrepl available. Based in my issues I'm not using syncrepl at this time since I'm trying to check a stable sittuation for my DB.
What I'm observing is that under openldap version 2.3 and 2.4 if I have a DB, at least with BDB, that could be searched and generate indexes bigger than 3GB(max OS memory space per process), the search will make slapd load all indexes in memory and never release them.
This make that openldap would not be used for DBs which size would become bigger than 3GB. What would be the major part of cases.
Once a DB entrance is searched, it is loaded into memory and never more released. So, for example, if I made several binds to the same entrance in DB, slapd never allocates more memory and uses the already cached entrance.
But if I search for different entrances in DB, slapd always cache in memory and continues to do it until it reaches the 3GB barrier and then process cannot allocate more memory. slapd never release cache memory, respect the default idlcachesize and cachesize that is 1000, or even release cache memory.
So if there is not ldapsearch for a backup, that could speed this behavior, with the time if the queries are done randomly at certain time the cache will become of size 3GB and no more memory can be allocated.
My actual version now is:
2.3.27-8.el5_2.4 where the CentOS5.4 uses a built-in backend library for BDB4.4.
My DB_CONFIG is(without comments) :
set_cachesize 0 52428800 1 set_lg_regionmax 1048576 set_lg_max 10485760 set_lg_bsize 2097152 set_tmp_dir /var/tmp
And the slapd.conf I included(the indexes always existed) :
# The database directory MUST exist prior to running slapd AND # should only be accessible by the slapd and slap tools. # Mode 700 recommended. directory /var/openldap-data/bdb1
# Indices to maintain for this database index userid,pnnumber,submxid,maillogin,abookaliasid eq,pres index objectClass eq
#Cache values cachesize 1000 idlcachesize 1000
This issue, by other ITSs, appear to exist before. Not sure if this is really a configuration problem.
Running slapd with -d 64 showed the cachesize and idlcachesize did not return any complain.
Please let me know if you have some idea about this problem.
I will try to use the not yet stable 2.4.13 but I would like to have your feedback if this could really be caused by other problem or even if in openldap 2.4.13 there are some detection/solution for this kind of issue.
Regards,
Rodrigo.
--- On Wed, 1/21/09, Quanah Gibson-Mount quanah@zimbra.com wrote:
From: Quanah Gibson-Mount quanah@zimbra.com Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 To: rlvcosta@yahoo.com, openldap-its@openldap.org Date: Wednesday, January 21, 2009, 5:57 PM --On Tuesday, January 20, 2009 12:07 AM +0000 rlvcosta@yahoo.com wrote:
Pierangelo,
Executing some test I'm not sure anymore if this
is really a memory
leakage or if some cache issue.
I decided to install CentOS5.2 where I could have
OpenLdap 2.3.27
compiled by the distribution and then execute more
tests to check if some
library in RHEL4 would be causing some problem.
I then use a similar slapd.conf as in my original
e-mail just without
explicitly define an size for cache and idlcache
there. My DB_CONFIG file
has a cache defined as 50MB what is respected.
Looking at some other ITS I found the ITS#3938 what I
believe is very
similar to my problem.
I still have a DB with 3882998 entrances and where the
DB files at disk
is around 15GB. My machine has 12GB of memory where
for sure all the DB
cannot be cached into memory.
In any case what is happening is that every time a
transaction is done
with OpenLdap, even a distribution formal compilation,
the cache
continues to grow until consumes all memory. If the DB
is bigger than
machine memory then soon or later a memory out of
range will happen.
The cache is never release by OpenLdap and then if a
sequential search is
done and since DB is bigger than memory we can speed
up the problem.
I would suggest you use OpenLDAP 2.4, and set the slapd.conf cachesize, idlcachesize, and dncachesize. I would also review your settings for DB_CONFIG, and use a newer BDB version with OpenLDAP 2.4 (i.e., BDB 4.7 with all 3 patches). See if that helps resolve the issue you face. In OpenLDAP 2.3, there is no way to control the dncachesize.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc
Zimbra :: the leader in open source messaging and collaboration
-
rlvcosta＠yahoo.com