On 31/08/2011, at 19.29, Quanah Gibson-Mount wrote:
--On Wednesday, August 31, 2011 11:33 AM +0200 Thomas Rasmussen rasmussen.thomas@gmail.com wrote:
After a restart and performing a ldapsearch slapd has allocated over 500MB of memory, which is not exactly a good thing :-(
How many total entries are in your database? What is your entrycache size? I don't see that information in your stripped slapd.conf. Personally, I suggest you never "strip" slapd.conf when sending it out, because you aren't necessarily going to know what is relevant to what you are doing.
I have about 3 million DNs in the database, I was required by our customer to strip out some of the information, but it was only the 'access' bits and the rootdn+rootpw which is why I stripped some information.
I think you are misunderstanding some of how OpenLDAP works. In addition to the BDB cachesize (which you have configured as 50MB (which is likely grossly small given the size of your database), there are also 3 additional caches for back-bdb/back-hdb in slapd itself. Those are *not* filled until you start doing searches. Those 3 caches are:
entrycache idlcache dn cache
OK, I was not aware that it would use memory for other caches that wasn't possible to setup (or are they?)
My guess is it is growing to 500MB of memory because it is filling up the DN cache. Given the 10GB size of your id2entry database, I'm guessing you have a *lot* of DNs.
I.e., the only way you will know the total size of what the slapd process *should* be is to start it, and then do an ldapsearch of the entire database. Once that is finished, you'll get an approximate size of your DB with most of the caches filled (although the idlcache may take more time to fill).
I tried to perform an ldapsearch on cn=* on my database (on ldap compiled with gcc, running BDB 5.2.x and tcmalloc enabled) and it allocated about 550MB of memory.
Basically, I don't see 500MB as problematic, given the DB size you reported. However, now that you have tcmalloc in place, the general size of slapd should be significantly more stable.
Finally, I will note that using "db_upgrade" to upgrade an OpenLDAP database has not, and it is unlikely every will be, supported. The *only* supported method of migrating to a newer version of BDB is to slapcat the original database and to slapadd it back in. So it is entirely possible you have created a very interesting issue by doing a completely unsupported operation.
Hmm, I found the db_upgrade process in the documentation of berkeley DB, so I'd imagine that it was supported, but apparently not by openldap...
I will try to make a series of tests with my new-found knowledge about the way slapd will eat memory... hopefully it will continue to behave (it seems to have stabilazed around 552MB)
Regards Thomas