Quanah,
I follow your recommendation and compiled the Berkeley Db 4.7.25NC and openldap 2.4.13.
After that load my LDIF file to have 1million entraces in my DB just to test the dncachesize control.
During my initialization I have :
module back_bdb.la: null module registered line 29 (moduleload back_monitor.la) loaded module back_monitor.la module back_monitor.la: null module registered line 74 (database bdb) line 75 (suffix "ou=CONTENT,o=alcatel,c=fr") line 76 (rootdn "cn=admin,ou=CONTENT,o=alcatel,c=fr") line 80 (rootpw ***) line 86 (directory /var/openldap-data/bdb1) line 89 (index userid,pnnumber,submxid,maillogin,abookaliasid eq,pres) index uid 0x0006 index pnnumber 0x0006 index submxid 0x0006 index maillogin 0x0006 index abookaliasid 0x0006 line 90 (index objectClass eq) index objectClass 0x0004 line 93 (cachesize 1000) line 94 (idlcachesize 1000) line 95 (dncachesize 1000) line 105 (database bdb) line 106 (suffix "ou=INDEXES,o=alcatel,c=fr") line 107 (rootdn "cn=admin,ou=INDEXES,o=alcatel,c=fr") line 111 (rootpw ***) line 114 (directory /var/openldap-data/bdb2) line 117 (index weblogin,maillogin,mail,pnnumber eq,pres) index weblogin 0x0006 index maillogin 0x0006 index mail 0x0006 index pnnumber 0x0006 line 118 (index profileid,loginname eq,pres) index profileid 0x0006 index loginname 0x0006 line 119 (index objectClass eq) index objectClass 0x0004 line 122 (cachesize 1000) line 123 (idlcachesize 1000) line 124 (dncachesize 1000) line 129 (database monitor) slapd starting
See that I have 2 DBs and in each DB area in slapd.conf, read without error by slapd as seen above, I have the cachesize, idlcachesize and dncachesize specified with small values just to monitor the memory usage by slapd.
Unfortunately even with this configuration I still having slapd consuming memory without release it. So if a new entrance is queried by some LDAP client, slapd will consume more memory. Since my DB dn's are bigger than 3GB then soon or later the slapd will crash.
See my DB information :
7.2G dn2id.bdb 12G id2entry.bdb 110M maillogin.bdb 3.3M objectClass.bdb 108M pnnumber.bdb 1.3M submxid.bdb 323M uid.bdb
See my dn2id is much bigger than yours and maybe you just did not saw this issue happenig since your dn can be loaded in your memory and you in a 64 bits environment can use more than 3GB per process.
But in any case looks like there is a problem in slapd that do not respect the cache limitations imposed.
In this way we cannot use slapd for large databases. Please see my configuration above and let me know if a made a mistake(I hope so) since dncachesize is not documented at man pages.
Just as example, with the DB I loaded for tests with 1 million entrances, with the cache sizes definitions as in the slapd.conf described above and making a ldapsearch on all entrances from the CONTENT DB, in the end I have :
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3189 ldap 15 0 842m 724m 67m S 99 6.1 7:00.48 slapd
See the slapd process already consumed 842m and I just read 1 million entrances, even dncachesize is defined to be 1000.
I do not see anyway to control the memory usage by slapd.
Is this "disrespect" to slapd.conf directives a possible problem?
I hope there is a configuration problem with my slapd.conf where this could be work around.
Regards,
Rodrigo.
--- On Wed, 1/21/09, Quanah Gibson-Mount quanah@zimbra.com wrote:
From: Quanah Gibson-Mount quanah@zimbra.com Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4 To: rlvcosta@yahoo.com, openldap-its@openldap.org Date: Wednesday, January 21, 2009, 8:30 PM --On Wednesday, January 21, 2009 12:49 PM -0800 Rodrigo Costa rlvcosta@yahoo.com wrote:
Quanah,
That's is exactly the point. My system is a 32
bits and with 64 bits if
there are enough physical memory available even slapd
cache grows without
never release it will be difficult(or very long) to
cache enough
entrances before system consumes all memory.
With the 32 bits 3GB maximum, depend on OS too, this
issue appears more
often. My impression is if you ldapsearch all DB in a
32 bits HW/OS slapd
will consume all possible memory unless to keep
boundaries for its
cache(erase and overwrite).
As I noted, in 2.3, you cannot constrain the DN cachesize. You can do this with OL 2.4. You need to try a current RE2.4 release, and set the dncachesize. There was no dn cache at all in OL 2.1.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc
Zimbra :: the leader in open source messaging and collaboration
-
rlvcosta＠yahoo.com