Hello (originally posted to -bugs list, but think this might be the correct list)
I have an openldap installation setup which appears to have a memory leak which I have not been able to solve by upgrading software etc. which is why I'm submitting this, if this is the wrong list, please let me know.
The setup is: - OpenLDAP server running 2.4.26, compiled against db-5.2.28, heimdal 1.2.1. - Server is SunOS servername 5.10 Generic_141445-09 i86pc i386 i86pc, 6GB of RAM and 5GB of swap (in a VMWare ESX environment)
I have the following in my DB_CONFIG file: set_cachesize 0 52428800 0 set_lg_regionmax 1048576 set_lg_max 10485760 set_lg_bsize 2097152 set_lg_dir /pack/openldap/var/openldap-crl-logs
In my slapd.conf I have the following entries (I stripped out some restricts and other irrelevant lines)
include /pack/openldap/etc/openldap/schema/core.schema include /pack/openldap/etc/openldap/schema/entrust.attributes.cfg include /pack/openldap/etc/openldap/schema/entrust.objectclasses.cfg idletimeout 300 pidfile /pack/openldap/var/run/slapd-crl.pid argsfile /pack/openldap/var/run/slapd-crl.args threads 4 database bdb suffix "c=DK" directory /pack/openldap/var/openldap-crl index objectClass,entryCSN,entryUUID eq checkpoint 128 1 dbnosync overlay syncprov syncprov-checkpoint 100 10 syncprov-sessionlog 100 serverID 2 monitoring off idletimeout 10
Just FYI, the server is a clone of another server currently running syncprov replication between two nodes, hence the syncprov lines. This server has the same databases as the production ones, but I just moved it to isolated area and removed all sync options from slapd.
My database is rather big: vs24n02z2.prl1:/pack/openldap/var/openldap-crl root$ du -sh *.bdb 961M dn2id.bdb 162M entryCSN.bdb 67M entryUUID.bdb 10G id2entry.bdb 4.0M objectClass.bdb
but I'd imagine this not beeing a problem (other than some performance issues, but this is not the problem here).
Now, when I start up my slapd, it consumes around 6MB RAM, all is fine, I can perform searches and updates without problems. But my memory consumption is going through the roof!
If I perform a search through the part of the database having alot of data, I suddenly find slapd using 500+ MB of memory...
In our production setup, we actually see that after a couple of weeks, slapd is consuming 3+ GB of memory, and then at some point hits a Out of memory, and crashes.
Our production setup is infact a openldap 2.4.22 with berkeley db 4.6.21, and my test server (cloned in a vmware environment) has just upgraded this environment by this recipe:
Using db-4.6.21 * db_checkpoint -1 * db_recover -h <path to bdb files>
install db-5.2.28 * Loop through all bdb files: db_upgrade <bdb file> * install openldap 2.4.26 (compiled against db-5.2.28) * start slapd
The server which updates the LDAP is performing many search and updates, which is probably why memory consumption is exploding after restarts?
Now, my main concern is, how (if possible) can we procede with this? We really need to find the problem as this causes production problems whenever the ldap crashes.
Regards Thomas
--On Thursday, August 25, 2011 9:15 AM +0200 Thomas Rasmussen rasmussen.thomas@gmail.com wrote:
Now, my main concern is, how (if possible) can we procede with this? We really need to find the problem as this causes production problems whenever the ldap crashes.
Use an alternative memory allocator such as tcmalloc.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
On 26/08/2011, at 17.35, Quanah Gibson-Mount wrote:
--On Thursday, August 25, 2011 9:15 AM +0200 Thomas Rasmussen rasmussen.thomas@gmail.com wrote:
Now, my main concern is, how (if possible) can we procede with this? We really need to find the problem as this causes production problems whenever the ldap crashes.
Use an alternative memory allocator such as tcmalloc.
--Quanah
Is that truely the only solution to this problem? This is a major production system, and changing memory allocator is not just something that we can do.
Regards Thomas
--On Tuesday, August 30, 2011 2:36 PM +0200 Thomas Rasmussen rasmussen.thomas@gmail.com wrote:
Is that truly the only solution to this problem? This is a major production system, and changing memory allocator is not just something that we can do.
When I used Slowaris, it was really the only way to have a useful system. The default memory allocator that ships with it is worthless. Pretty much the same thing in Linux, too. It is very trivial to download and build tcmalloc and set the environment variable to have slapd use it instead. Probably about 10-15 minutes of your time.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
On 30/08/2011, at 18.54, Quanah Gibson-Mount wrote:
--On Tuesday, August 30, 2011 2:36 PM +0200 Thomas Rasmussen rasmussen.thomas@gmail.com wrote:
Is that truly the only solution to this problem? This is a major production system, and changing memory allocator is not just something that we can do.
When I used Slowaris, it was really the only way to have a useful system. The default memory allocator that ships with it is worthless. Pretty much the same thing in Linux, too. It is very trivial to download and build tcmalloc and set the environment variable to have slapd use it instead. Probably about 10-15 minutes of your time.
I have now given it a try, but still no luck, my SMF script does this:
LD_PRELOAD=/pack/google-perftools/lib/libtcmalloc_minimal.so /pack/openldap/libexec/slapd -f /pack/openldap/etc/openldap/slapd-crl.conf -h ldap://0.0.0.0:389/ -l local1
After a restart and performing a ldapsearch slapd has allocated over 500MB of memory, which is not exactly a good thing :-(
Regards Thomas
--On Wednesday, August 31, 2011 11:33 AM +0200 Thomas Rasmussen rasmussen.thomas@gmail.com wrote:
After a restart and performing a ldapsearch slapd has allocated over 500MB of memory, which is not exactly a good thing :-(
How many total entries are in your database? What is your entrycache size? I don't see that information in your stripped slapd.conf. Personally, I suggest you never "strip" slapd.conf when sending it out, because you aren't necessarily going to know what is relevant to what you are doing.
I think you are misunderstanding some of how OpenLDAP works. In addition to the BDB cachesize (which you have configured as 50MB (which is likely grossly small given the size of your database), there are also 3 additional caches for back-bdb/back-hdb in slapd itself. Those are *not* filled until you start doing searches. Those 3 caches are:
entrycache idlcache dn cache
My guess is it is growing to 500MB of memory because it is filling up the DN cache. Given the 10GB size of your id2entry database, I'm guessing you have a *lot* of DNs.
I.e., the only way you will know the total size of what the slapd process *should* be is to start it, and then do an ldapsearch of the entire database. Once that is finished, you'll get an approximate size of your DB with most of the caches filled (although the idlcache may take more time to fill).
Basically, I don't see 500MB as problematic, given the DB size you reported. However, now that you have tcmalloc in place, the general size of slapd should be significantly more stable.
Finally, I will note that using "db_upgrade" to upgrade an OpenLDAP database has not, and it is unlikely every will be, supported. The *only* supported method of migrating to a newer version of BDB is to slapcat the original database and to slapadd it back in. So it is entirely possible you have created a very interesting issue by doing a completely unsupported operation.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
On 31/08/2011, at 19.29, Quanah Gibson-Mount wrote:
--On Wednesday, August 31, 2011 11:33 AM +0200 Thomas Rasmussen rasmussen.thomas@gmail.com wrote:
After a restart and performing a ldapsearch slapd has allocated over 500MB of memory, which is not exactly a good thing :-(
How many total entries are in your database? What is your entrycache size? I don't see that information in your stripped slapd.conf. Personally, I suggest you never "strip" slapd.conf when sending it out, because you aren't necessarily going to know what is relevant to what you are doing.
I have about 3 million DNs in the database, I was required by our customer to strip out some of the information, but it was only the 'access' bits and the rootdn+rootpw which is why I stripped some information.
I think you are misunderstanding some of how OpenLDAP works. In addition to the BDB cachesize (which you have configured as 50MB (which is likely grossly small given the size of your database), there are also 3 additional caches for back-bdb/back-hdb in slapd itself. Those are *not* filled until you start doing searches. Those 3 caches are:
entrycache idlcache dn cache
OK, I was not aware that it would use memory for other caches that wasn't possible to setup (or are they?)
My guess is it is growing to 500MB of memory because it is filling up the DN cache. Given the 10GB size of your id2entry database, I'm guessing you have a *lot* of DNs.
I.e., the only way you will know the total size of what the slapd process *should* be is to start it, and then do an ldapsearch of the entire database. Once that is finished, you'll get an approximate size of your DB with most of the caches filled (although the idlcache may take more time to fill).
I tried to perform an ldapsearch on cn=* on my database (on ldap compiled with gcc, running BDB 5.2.x and tcmalloc enabled) and it allocated about 550MB of memory.
Basically, I don't see 500MB as problematic, given the DB size you reported. However, now that you have tcmalloc in place, the general size of slapd should be significantly more stable.
Finally, I will note that using "db_upgrade" to upgrade an OpenLDAP database has not, and it is unlikely every will be, supported. The *only* supported method of migrating to a newer version of BDB is to slapcat the original database and to slapadd it back in. So it is entirely possible you have created a very interesting issue by doing a completely unsupported operation.
Hmm, I found the db_upgrade process in the documentation of berkeley DB, so I'd imagine that it was supported, but apparently not by openldap...
I will try to make a series of tests with my new-found knowledge about the way slapd will eat memory... hopefully it will continue to behave (it seems to have stabilazed around 552MB)
Regards Thomas
On Thursday, 1 September 2011 10:14:12 Thomas Rasmussen wrote:
On 31/08/2011, at 19.29, Quanah Gibson-Mount wrote:
--On Wednesday, August 31, 2011 11:33 AM +0200 Thomas Rasmussen
rasmussen.thomas@gmail.com wrote:
After a restart and performing a ldapsearch slapd has allocated over 500MB of memory, which is not exactly a good thing :-(
How many total entries are in your database? What is your entrycache size? I don't see that information in your stripped slapd.conf. Personally, I suggest you never "strip" slapd.conf when sending it out, because you aren't necessarily going to know what is relevant to what you are doing.
I have about 3 million DNs in the database, I was required by our customer to strip out some of the information, but it was only the 'access' bits and the rootdn+rootpw which is why I stripped some information.
I think you are misunderstanding some of how OpenLDAP works. In addition to the BDB cachesize (which you have configured as 50MB (which is likely grossly small given the size of your database), there are also 3 additional caches for back-bdb/back-hdb in slapd itself. Those are *not* filled until you start doing searches. Those 3 caches are:
entrycache idlcache dn cache
OK, I was not aware that it would use memory for other caches that wasn't possible to setup (or are they?)
Have you read any documentation regarding tuning, at all?
$ man slapd-bdb|col -b|grep -A6 -E '^ *(idl|dn|)cache.*size' cachesize <integer> Specify the size in entries of the in-memory entry cache main‐ tained by the bdb or hdb backend database instance. The default is 1000 entries.
cachefree <integer> Specify the number of entries to free from the entry cache when -- dncachesize <integer> Specify the maximum number of DNs in the in-memory DN cache. Ideally this cache should be large enough to contain the DNs of every entry in the database. If set to a smaller value than the cachesize it will be silently increased to equal the cachesize. The default value is 0 which means unlimited, i.e. the DN cache will grow without bound. -- idlcachesize <integer> Specify the size of the in-memory index cache, in index slots. The default is zero. A larger value will speed up frequent searches of indexed entries. An hdb database needs a large idl‐ cachesize for good search performance, typically three times the cachesize (entry cache size) or larger.
Regards, Buchan
openldap-technical@openldap.org
-
Buchan Milne -
Quanah Gibson-Mount -
Thomas Rasmussen