I have tried openldap-2.4.16 (latest) BDB 4.7.25 with all patches from Oracle on Dell AMD64 RAM 4Gb, 8 CPUs, OS Solaris 10. According to ITS#5860 slapd should not allocate more memory than it was specified. In fact I have the same behaviour as it was with previous version (2.4.11). It correctly allocates 1.2 Gb via shared memory (for set_cachesize 300 MB). But slapd process continuosly grows in memory until 32-bit limit then crashes due to unavalable memory resources. I need to mention bdb database size is less then 1 GB. After restart it rapidly consumes 1.5 GB RSS in few minutes and then it grows beyond the limit during 24 hours. The more queries are processing the more memory it eats. I am using this database for logging which size grows. The question is: is it possible to use 32-bit openldap with relatively large database sizes that override 32-bit per-process memory space?
My slapd.conf contains:
threads 64 concurrency 50
cachesize 200000 dncachesize 400000 idlcachesize 600000
shm_key 10
DB_CONFIG:
set_cachesize 0 300097152 1 set_lk_max_locks 1000000 set_lk_max_lockers 1000000 set_lk_max_objects 1000000 set_lg_regionmax 1048576 set_lg_max 10485760 set_lg_bsize 2097152
ls -ls ./openldap-data total 1212876 -rw-r--r-- 1 slapd nobody 640 Apr 17 08:42 DB_CONFIG -rw------- 1 slapd nobody 8 Apr 16 10:50 __db.001 -rw-r--r-- 1 slapd nobody 4096 Apr 16 10:50 alock -rw------- 1 slapd nobody 6053888 Apr 17 09:50 cn.bdb -rw------- 1 slapd nobody 1024000 Apr 17 09:50 dhcpHWAddress.bdb -rw------- 1 slapd nobody 96583680 Apr 17 09:50 dn2id.bdb -rw------- 1 slapd nobody 515637248 Apr 17 09:50 id2entry.bdb -rw------- 1 slapd nobody 1343488 Apr 17 09:50 objectClass.bdb
Any help would be much appreciated.
Regards, Sergey.
--On Friday, April 17, 2009 12:17 PM +0300 sscdvp@gmail.com wrote:
I have tried openldap-2.4.16 (latest) BDB 4.7.25 with all patches from Oracle on Dell AMD64 RAM 4Gb, 8 CPUs, OS Solaris 10. According to ITS#5860 slapd should not allocate more memory than it was specified. In fact I have the same behaviour as it was with previous
version (2.4.11). It correctly allocates 1.2 Gb via shared memory (for set_cachesize 300 MB). But slapd process continuosly grows in memory until 32-bit limit then crashes due to unavalable memory resources. I need to
mention bdb database size is less then 1 GB. After restart it rapidly consumes 1.5 GB RSS in few minutes and then it grows beyond the limit during 24 hours. The more queries are processing the more memory it eats.
I am using this database for logging which size grows. The question is: is it possible to use 32-bit openldap with relatively large database sizes that override 32-bit per-process memory space?
My slapd.conf contains:
threads 64
Why do you have so many threads configured? How many cpu's do you have??
Any help would be much appreciated.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
sscdvp@gmail.com wrote:
I have tried openldap-2.4.16 (latest) BDB 4.7.25 with all patches from Oracle on Dell AMD64 RAM 4Gb, 8 CPUs, OS Solaris 10.
A system with 8 CPUs and only 4GB of RAM is seriously unbalanced. Heck, even my single-processor laptop has 4GB of RAM.
According to ITS#5860 slapd should not allocate more memory than it was specified. In fact I have the same behaviour as it was with previous version (2.4.11). It correctly allocates 1.2 Gb via shared memory (for set_cachesize 300 MB). But slapd process continuosly grows in memory until 32-bit limit then crashes due to unavalable memory resources. I need to mention bdb database size is less then 1 GB. After restart it rapidly consumes 1.5 GB RSS in few minutes and then it grows beyond the limit during 24 hours. The more queries are processing the more memory it eats.
I am using this database for logging which size grows. The question is: is it possible to use 32-bit openldap with relatively large database sizes that override 32-bit per-process memory space?
Yes, it's possible. But performance will always be poor since it will be continually paging data in and out of cache. You have AMD64 processors, why are you running a 32 bit OS?
My slapd.conf contains:
threads 64
256MB of thread call stack right there. Plus another 64MB for per-thread sl_malloc. Plus another 512MB for per-thread search stack.
This setting is too high for the amount of memory you have in your machine.
concurrency 50
cachesize 200000 dncachesize 400000 idlcachesize 600000
You should also set cachefree.
shm_key 10
DB_CONFIG:
set_cachesize 0 300097152 1
set_lk_max_locks 1000000 set_lk_max_lockers 1000000 set_lk_max_objects 1000000
Those settings are too high for the amount of memory you have in your machine.
Right away you've consumed about 2GB of RAM *before* taking any of back-bdb's cache sizes into account.
set_lg_regionmax 1048576 set_lg_max 10485760 set_lg_bsize 2097152
ls -ls ./openldap-data total 1212876
-rw-r--r-- 1 slapd nobody 640 Apr 17 08:42 DB_CONFIG -rw------- 1 slapd nobody 8 Apr 16 10:50 __db.001 -rw-r--r-- 1 slapd nobody 4096 Apr 16 10:50 alock -rw------- 1 slapd nobody 6053888 Apr 17 09:50 cn.bdb
-rw------- 1 slapd nobody 1024000 Apr 17 09:50 dhcpHWAddress.bdb -rw------- 1 slapd nobody 96583680 Apr 17 09:50 dn2id.bdb -rw------- 1 slapd nobody 515637248 Apr 17 09:50 id2entry.bdb -rw------- 1 slapd nobody 1343488 Apr 17 09:50 objectClass.bdb
Any help would be much appreciated.
Buy more RAM and use a 64 bit OS. You're running out of memory, but not due to ITS#5860.
2009/4/18, Howard Chu hyc@symas.com:
sscdvp@gmail.com wrote:
I have tried openldap-2.4.16 (latest) BDB 4.7.25 with all patches from Oracle on Dell AMD64 RAM 4Gb, 8 CPUs, OS Solaris 10.
A system with 8 CPUs and only 4GB of RAM is seriously unbalanced. Heck, even my single-processor laptop has 4GB of RAM.
According to ITS#5860 slapd should not allocate more memory than it was specified. In fact I have the same behaviour as it was with previous version (2.4.11). It correctly allocates 1.2 Gb via shared memory (for set_cachesize 300 MB). But slapd process continuosly grows in memory until 32-bit limit then crashes due to unavalable memory resources. I need to mention bdb database size is less then 1 GB. After restart it rapidly consumes 1.5 GB RSS in few minutes and then it grows beyond the limit during 24 hours. The more queries are processing the more memory it eats.
I am using this database for logging which size grows. The question is: is it possible to use 32-bit openldap with relatively large database sizes that override 32-bit per-process memory space?
Yes, it's possible. But performance will always be poor since it will be continually paging data in and out of cache. You have AMD64 processors, why are you running a 32 bit OS?
My slapd.conf contains:
threads 64
256MB of thread call stack right there. Plus another 64MB for per-thread sl_malloc. Plus another 512MB for per-thread search stack.
This setting is too high for the amount of memory you have in your machine.
concurrency 50
cachesize 200000 dncachesize 400000 idlcachesize 600000
You should also set cachefree.
shm_key 10
DB_CONFIG:
set_cachesize 0 300097152 1
set_lk_max_locks 1000000 set_lk_max_lockers 1000000 set_lk_max_objects 1000000
Those settings are too high for the amount of memory you have in your machine.
Right away you've consumed about 2GB of RAM *before* taking any of back-bdb's cache sizes into account.
set_lg_regionmax 1048576 set_lg_max 10485760 set_lg_bsize 2097152
ls -ls ./openldap-data total 1212876
-rw-r--r-- 1 slapd nobody 640 Apr 17 08:42 DB_CONFIG -rw------- 1 slapd nobody 8 Apr 16 10:50 __db.001 -rw-r--r-- 1 slapd nobody 4096 Apr 16 10:50 alock -rw------- 1 slapd nobody 6053888 Apr 17 09:50 cn.bdb
-rw------- 1 slapd nobody 1024000 Apr 17 09:50 dhcpHWAddress.bdb -rw------- 1 slapd nobody 96583680 Apr 17 09:50 dn2id.bdb -rw------- 1 slapd nobody 515637248 Apr 17 09:50 id2entry.bdb -rw------- 1 slapd nobody 1343488 Apr 17 09:50 objectClass.bdb
Any help would be much appreciated.
Buy more RAM and use a 64 bit OS. You're running out of memory, but not due to ITS#5860.
I have added more RAM (12 gigabytes) and switched to 64-bit openldap version (DB_CONFIG and slapd.conf options were not changed excepting of using "threads 2" now). But slapd process grows continuously in memory size (the size is 4 gigabytes now). I observed that that growth appears only when data in database is replaced with new one. My database presents application's activity logs which are completely overwritting every 12 hours by application. This process is done by using LDAP_DELETE operation on some attributes withing ldap_modify_ext call or periodically by purging the whole old entries by ldap_delete_ext call. What is recommended to avoid the cache trashing by already deleted information?
-
Howard Chu -
Quanah Gibson-Mount -
sscdvp＠gmail.com