[First moment after some minutes the consumer(slave) slapd start the search] Thread 5 (Thread -1498268784 (LWP 22749)): #0 0x0033f402 in __kernel_vsyscall () #1 0x004944d6 in epoll_wait () from /lib/libc.so.6 #2 0x0807334f in slapd_daemon_destroy () #3 0x0030b45b in start_thread () from /lib/libpthread.so.0 #4 0x00493e5e in clone () from /lib/libc.so.6 Thread 4 (Thread -1502467184 (LWP 22750)): #0 0x0033f402 in __kernel_vsyscall () #1 0x0030f256 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x00380230 in ldap_pvt_thread_cond_wait (cond=0x9da920c, mutex=0x9da91f4) at thr_posix.c:277 #3 0x0037eeeb in ldap_int_thread_pool_wrapper (xpool=0x9da91f0) at tpool.c:654 #4 0x0030b45b in start_thread () from /lib/libpthread.so.0 #5 0x00493e5e in clone () from /lib/libc.so.6 Thread 3 (Thread -1506665584 (LWP 22751)): #0 0x0033f402 in __kernel_vsyscall () #1 0x0030f256 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x00380230 in ldap_pvt_thread_cond_wait (cond=0x9da920c, mutex=0x9da91f4) at thr_posix.c:277 #3 0x0037eeeb in ldap_int_thread_pool_wrapper (xpool=0x9da91f0) at tpool.c:654 #4 0x0030b45b in start_thread () from /lib/libpthread.so.0 #5 0x00493e5e in clone () from /lib/libc.so.6 Thread 2 (Thread -1858077808 (LWP 22876)): #0 0x005e6420 in bdb_cache_lru_purge (bdb=0x9df0308) at cache.c:716 #1 0x005e6d30 in bdb_cache_find_id (op=0x913fe774, tid=0x9eb7288, id=405592, eip=0x9133e66c, flag=0, lock=0x9133e650) at cache.c:1053 #2 0x005d5df5 in bdb_search (op=0x913fe774, rs=0x913fe84c) at search.c:705 #3 0x080eb977 in cancel_extop () #4 0x080e71b1 in cancel_extop () #5 0x080e8c34 in cancel_extop () #6 0x0807752e in connection_client_stop () #7 0x0037ef5e in ldap_int_thread_pool_wrapper (xpool=0x9da91f0) at tpool.c:663 #8 0x0030b45b in start_thread () from /lib/libpthread.so.0 #9 0x00493e5e in clone () from /lib/libc.so.6 Thread 1 (Thread -1208891632 (LWP 22748)): #0 0x0033f402 in __kernel_vsyscall () #1 0x0030c557 in pthread_join () from /lib/libpthread.so.0 #2 0x0038012b in ldap_pvt_thread_join (thread=2796698512, thread_return=0x0) at thr_posix.c:197 #3 0x08074074 in slapd_daemon () #4 0x08058318 in main () Thread 2 (Thread -1858077808 (LWP 22876)): #0 0x005e6420 in bdb_cache_lru_purge (bdb=0x9df0308) at cache.c:716 #1 0x005e6d30 in bdb_cache_find_id (op=0x913fe774, tid=0x9eb7288, id=405592, eip=0x9133e66c, flag=0, lock=0x9133e650) at cache.c:1053 #2 0x005d5df5 in bdb_search (op=0x913fe774, rs=0x913fe84c) at search.c:705 #3 0x080eb977 in cancel_extop () #4 0x080e71b1 in cancel_extop () #5 0x080e8c34 in cancel_extop () #6 0x0807752e in connection_client_stop () #7 0x0037ef5e in ldap_int_thread_pool_wrapper (xpool=0x9da91f0) at tpool.c:663 #8 0x0030b45b in start_thread () from /lib/libpthread.so.0 #9 0x00493e5e in clone () from /lib/libc.so.6 Thread 1 (Thread -1208891632 (LWP 22748)): #0 0x0033f402 in __kernel_vsyscall () #1 0x0030c557 in pthread_join () from /lib/libpthread.so.0 #2 0x0038012b in ldap_pvt_thread_join (thread=2796698512, thread_return=0x0) at thr_posix.c:197 #3 0x08074074 in slapd_daemon () #4 0x08058318 in main () #0 0x0033f402 in __kernel_vsyscall () --------------------------------------------- [Second moment after more time passed the consumer(slave) slapd has around 188-200% CPU usage and search appears to continue since provider(master) slapd process continues to have CPU in use] Thread 5 (Thread -1497973872 (LWP 28856)): #0 0x0044e402 in __kernel_vsyscall () #1 0x005204d6 in epoll_wait () from /lib/libc.so.6 #2 0x0807334f in slapd_daemon_destroy () #3 0x0030b45b in start_thread () from /lib/libpthread.so.0 #4 0x0051fe5e in clone () from /lib/libc.so.6 Thread 4 (Thread -1502172272 (LWP 28857)): #0 0x0019db2c in __lock_get_internal () from /usr/lib/libdb-4.7.so #1 0xa71677a0 in ?? () #2 0x08a0c5e8 in ?? () #3 0x08a0c1e8 in ?? () #4 0x0019cf28 in __lock_put_internal () from /usr/lib/libdb-4.7.so #5 0x00000000 in ?? () Thread 3 (Thread -1506370672 (LWP 28858)): #0 0x0044e402 in __kernel_vsyscall () #1 0x0030f256 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x00b5e230 in ldap_pvt_thread_cond_wait (cond=0x896120c, mutex=0x89611f4) at thr_posix.c:277 #3 0x00b5ceeb in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:654 #4 0x0030b45b in start_thread () from /lib/libpthread.so.0 #1 0x0030f256 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x00b5e230 in ldap_pvt_thread_cond_wait (cond=0x896120c, mutex=0x89611f4) at thr_posix.c:277 #3 0x00b5ceeb in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:654 #4 0x0030b45b in start_thread () from /lib/libpthread.so.0 ---Type to continue, or q to quit--- #5 0x0051fe5e in clone () from /lib/libc.so.6 Thread 2 (Thread -1870660720 (LWP 29107)): #0 0x001e0b79 in __env_alloc () from /usr/lib/libdb-4.7.so #1 0xb63221b4 in ?? () #2 0xafe6c000 in ?? () #3 0x0000000d in ?? () #4 0x00000000 in ?? () Thread 1 (Thread -1208596720 (LWP 28855)): #0 0x0044e402 in __kernel_vsyscall () #1 0x0030c557 in pthread_join () from /lib/libpthread.so.0 #2 0x00b5e12b in ldap_pvt_thread_join (thread=2796993424, thread_return=0x0) at thr_posix.c:197 #3 0x08074074 in slapd_daemon () #4 0x08058318 in main () #0 0x0044e402 in __kernel_vsyscall () ------------------------------------------------------------------------------ [Third moment after more time passed the consumer(slave) slapd has around 188-200% CPU usage, and similar to second moment and search appears to continue since provider(master) slapd process continues to have CPU in use] Thread 5 (Thread -1497973872 (LWP 28856)): #0 0x0044e402 in __kernel_vsyscall () #1 0x005204d6 in epoll_wait () from /lib/libc.so.6 #2 0x0807334f in slapd_daemon_destroy () #3 0x0030b45b in start_thread () from /lib/libpthread.so.0 #4 0x0051fe5e in clone () from /lib/libc.so.6 Thread 4 (Thread -1502172272 (LWP 28857)): #0 0x0019c56c in __lock_getobj () from /usr/lib/libdb-4.7.so #1 0x0000023c in ?? () #2 0x00a3fdac in ?? () from /usr/lib/liblber-2-devel.so.0 #3 0x00000125 in ?? () #4 0x002148d2 in __os_id () from /usr/lib/libdb-4.7.so #5 0x00118365 in __db_tas_mutex_lock () from /usr/lib/libdb-4.7.so #6 0xa0a0bbf8 in ?? () #7 0xa66a953c in ?? () #8 0xaf1a80c0 in ?? () #9 0x0023c0c0 in ?? () from /usr/lib/libdb-4.7.so #10 0x08a0c5e8 in ?? () #11 0xa66a9428 in ?? () #12 0xa71899e0 in ?? () #13 0x0019d7d4 in __lock_get_internal () from /usr/lib/libdb-4.7.so #14 0xa714ec50 in ?? () #15 0x08a0c5e8 in ?? () #16 0xa714f648 in ?? () #17 0x0019cf28 in __lock_put_internal () from /usr/lib/libdb-4.7.so #18 0x00000000 in ?? () Thread 3 (Thread -1506370672 (LWP 28858)): #0 0x0044e402 in __kernel_vsyscall () #1 0x0030f256 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x00b5e230 in ldap_pvt_thread_cond_wait (cond=0x896120c, mutex=0x89611f4) at thr_posix.c:277 #3 0x00b5ceeb in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:654 #4 0x0030b45b in start_thread () from /lib/libpthread.so.0 #5 0x0051fe5e in clone () from /lib/libc.so.6 Thread 2 (Thread -1870660720 (LWP 29107)): #0 0x004b9712 in malloc () from /lib/libc.so.6 #1 0x00a3a822 in ber_memalloc_x (s=8, ctx=0x0) at memory.c:226 #2 0x00a3b290 in ber_strndup_x (s=0x36b5d2ce "domain,c=fr", l=7, ctx=0x0) at memory.c:675 #3 0x00b73081 in str2strval (str=0x36b5d2ce "domain,c=fr", stoplen=12, val=0x907fe674, next=0x907fe688, flags=0, retFlags=0x907fe684, ctx=0x0) at getdn.c:1531 #4 0x00b727b9 in ldap_bv2rdn_x (bv=0x907fe71c, rdn=0x907fe7ac, n_in=0x907fe7b0, flags=0, ctx=0x0) at getdn.c:1250 #5 0x00b719d3 in ldap_bv2dn_x (bvin=0x36bc2f98, dn=0x907fe82c, flags=0, ctx=0x0) at getdn.c:787 #6 0x08090386 in dnPretty () #7 0x08099af5 in ordered_value_pretty () #8 0x0809497d in slap_mods_check () #9 0x080eaa75 in cancel_extop () #10 0x080e7159 in cancel_extop () #11 0x080e8c34 in cancel_extop () #12 0x0807752e in connection_client_stop () #13 0x00b5cf5e in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:663 #14 0x0030b45b in start_thread () from /lib/libpthread.so.0 #15 0x0051fe5e in clone () from /lib/libc.so.6 Thread 1 (Thread -1208596720 (LWP 28855)): #0 0x0044e402 in __kernel_vsyscall () #1 0x0030c557 in pthread_join () from /lib/libpthread.so.0 #2 0x00b5e12b in ldap_pvt_thread_join (thread=2796993424, thread_return=0x0) at thr_posix.c:197 #3 0x08074074 in slapd_daemon () #4 0x08058318 in main () #0 0x0044e402 in __kernel_vsyscall () ---------------------------------------------------------------------------- [Forth moment after just a little more time passed the consumer(slave) slapd has exactly 200% CPU usage and search appears to finish since provider(master) slapd process has 0-2% CPU in use by top] Thread 5 (Thread -1497973872 (LWP 28856)): #0 0x0044e402 in __kernel_vsyscall () #1 0x005204d6 in epoll_wait () from /lib/libc.so.6 #2 0x0807334f in slapd_daemon_destroy () #3 0x0030b45b in start_thread () from /lib/libpthread.so.0 #4 0x0051fe5e in clone () from /lib/libc.so.6 Thread 4 (Thread -1502172272 (LWP 28857)): #0 0x08108a4a in UTF8bvnormalize () #1 0x080bb416 in nameUIDPretty () #2 0x0808fae3 in rdnValidate () #3 0x0808fbdf in rdnValidate () #4 0x0808fdcb in dnNormalize () #5 0x08099d9a in ordered_value_normalize () #6 0x08094b7f in slap_mods_check () #7 0x080eaa75 in cancel_extop () #8 0x080e7159 in cancel_extop () #9 0x080e8c34 in cancel_extop () #10 0x0807752e in connection_client_stop () #11 0x00b5cf5e in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:663 #12 0x0030b45b in start_thread () from /lib/libpthread.so.0 #13 0x0051fe5e in clone () from /lib/libc.so.6 Thread 3 (Thread -1506370672 (LWP 28858)): #0 0x0044e402 in __kernel_vsyscall () #1 0x0030f256 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x00b5e230 in ldap_pvt_thread_cond_wait (cond=0x896120c, mutex=0x89611f4) at thr_posix.c:277 #3 0x00b5ceeb in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:654 #4 0x0030b45b in start_thread () from /lib/libpthread.so.0 #5 0x0051fe5e in clone () from /lib/libc.so.6 Thread 2 (Thread -1870660720 (LWP 29107)): #0 0x0019e316 in __lock_get_internal () from /usr/lib/libdb-4.7.so #1 0xafb0ce38 in ?? () #2 0x08a0a098 in ?? () #3 0x08a09db8 in ?? () #4 0x0000057c in ?? () #5 0x000004a4 in ?? () #6 0xafaf40c0 in ?? () #7 0x00000000 in ?? () Thread 1 (Thread -1208596720 (LWP 28855)): #0 0x0044e402 in __kernel_vsyscall () #1 0x0030c557 in pthread_join () from /lib/libpthread.so.0 #3 0x00b5ceeb in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:654 #4 0x0030b45b in start_thread () from /lib/libpthread.so.0 #5 0x0051fe5e in clone () from /lib/libc.so.6 Thread 2 (Thread -1870660720 (LWP 29107)): #0 0x0019e316 in __lock_get_internal () from /usr/lib/libdb-4.7.so #1 0xafb0ce38 in ?? () #2 0x08a0a098 in ?? () #3 0x08a09db8 in ?? () #4 0x0000057c in ?? () #5 0x000004a4 in ?? () #6 0xafaf40c0 in ?? () #7 0x00000000 in ?? () Thread 1 (Thread -1208596720 (LWP 28855)): #0 0x0044e402 in __kernel_vsyscall () #1 0x0030c557 in pthread_join () from /lib/libpthread.so.0 ---Type to continue, or q to quit--- #2 0x00b5e12b in ldap_pvt_thread_join (thread=2796993424, thread_return=0x0) at thr_posix.c:197 #3 0x08074074 in slapd_daemon () #4 0x08058318 in main () #0 0x0044e402 in __kernel_vsyscall () ----------------------------------------------------------------------------- [Fifth moment just one more gdb information in a posterior moment from forfth moment - still with 200% CPU in consumer(slave)] Thread 5 (Thread -1497973872 (LWP 28856)): #0 0x0044e402 in __kernel_vsyscall () #1 0x005204d6 in epoll_wait () from /lib/libc.so.6 #2 0x0807334f in slapd_daemon_destroy () #3 0x0030b45b in start_thread () from /lib/libpthread.so.0 #4 0x0051fe5e in clone () from /lib/libc.so.6 Thread 4 (Thread -1502172272 (LWP 28857)): #0 0x0019c636 in __lock_getobj () from /usr/lib/libdb-4.7.so #1 0x0000023c in ?? () #2 0x00a3fdac in ?? () from /usr/lib/liblber-2-devel.so.0 #3 0x000003c7 in ?? () #4 0xa66a9428 in ?? () #5 0x08a0c5e8 in ?? () #6 0x00000000 in ?? () Thread 3 (Thread -1506370672 (LWP 28858)): #0 0x0044e402 in __kernel_vsyscall () #1 0x0030f256 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x00b5e230 in ldap_pvt_thread_cond_wait (cond=0x896120c, mutex=0x89611f4) at thr_posix.c:277 #3 0x00b5ceeb in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:654 #4 0x0030b45b in start_thread () from /lib/libpthread.so.0 #5 0x0051fe5e in clone () from /lib/libc.so.6 Thread 2 (Thread -1870660720 (LWP 29107)): #0 0x0019dae8 in __lock_get_internal () from /usr/lib/libdb-4.7.so #1 0xafb09800 in ?? () #2 0x08a0a098 in ?? () #3 0x08a09db8 in ?? () #4 0x0019cf28 in __lock_put_internal () from /usr/lib/libdb-4.7.so #5 0x00000000 in ?? () Thread 1 (Thread -1208596720 (LWP 28855)): #0 0x0044e402 in __kernel_vsyscall () #1 0x0030c557 in pthread_join () from /lib/libpthread.so.0 #2 0x00b5e12b in ldap_pvt_thread_join (thread=2796993424, thread_return=0x0) at thr_posix.c:197 #3 0x08074074 in slapd_daemon () #4 0x08058318 in main () #0 0x0044e402 in __kernel_vsyscall () ----------------------------------------------------------------------------- [Sixth moment just one more gdb information in a posterior moment from forfth moment - still with 200% CPU in consumer(slave) and 0-2% CPU usage in provider(master)] Thread 5 (Thread -1497973872 (LWP 28856)): #0 0x0044e402 in __kernel_vsyscall () #1 0x005204d6 in epoll_wait () from /lib/libc.so.6 #2 0x0807334f in slapd_daemon_destroy () #3 0x0030b45b in start_thread () from /lib/libpthread.so.0 #4 0x0051fe5e in clone () from /lib/libc.so.6 Thread 4 (Thread -1502172272 (LWP 28857)): #0 0x0030e4ea in __pthread_mutex_unlock_usercnt () from /lib/libpthread.so.0 #1 0x0030e880 in pthread_mutex_unlock () from /lib/libpthread.so.0 #2 0x00b5e2e7 in ldap_pvt_thread_mutex_unlock (mutex=0x132ce5c0) at thr_posix.c:308 #3 0x003ef608 in bdb_cache_lru_purge (bdb=0x89a8308) at cache.c:785 #4 0x003efd30 in bdb_cache_find_id (op=0xa6769774, tid=0x8a2c900, id=620396, eip=0xa66a966c, flag=0, lock=0xa66a9650) at cache.c:1053 #5 0x003dedf5 in bdb_search (op=0xa6769774, rs=0xa676984c) at search.c:705 #6 0x080eb977 in cancel_extop () #7 0x080e71b1 in cancel_extop () #8 0x080e8c34 in cancel_extop () #9 0x0807752e in connection_client_stop () #10 0x00b5cf5e in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:663 #11 0x0030b45b in start_thread () from /lib/libpthread.so.0 #12 0x0051fe5e in clone () from /lib/libc.so.6 Thread 3 (Thread -1506370672 (LWP 28858)): #0 0x0044e402 in __kernel_vsyscall () #1 0x0030f256 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x00b5e230 in ldap_pvt_thread_cond_wait (cond=0x896120c, mutex=0x89611f4) at thr_posix.c:277 #3 0x00b5ceeb in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:654 #4 0x0030b45b in start_thread () from /lib/libpthread.so.0 #5 0x0051fe5e in clone () from /lib/libc.so.6 Thread 2 (Thread -1870660720 (LWP 29107)): #0 0x0019c5a1 in __lock_getobj () from /usr/lib/libdb-4.7.so #1 0x000006b4 in ?? () #2 0x00a3fdac in ?? () from /usr/lib/liblber-2-devel.so.0 #3 0x000000d4 in ?? () #4 0x9073e428 in ?? () #5 0x08a0a098 in ?? () #6 0xb7ecf0b4 in ?? () #7 0xafaf40c0 in ?? () #8 0x08a09db8 in ?? () #9 0x00000000 in ?? () Thread 1 (Thread -1208596720 (LWP 28855)): #0 0x0044e402 in __kernel_vsyscall () #1 0x0030c557 in pthread_join () from /lib/libpthread.so.0 #2 0x00b5e12b in ldap_pvt_thread_join (thread=2796993424, thread_return=0x0) at thr_posix.c:197 #3 0x08074074 in slapd_daemon () #4 0x08058318 in main () #0 0x0044e402 in __kernel_vsyscall () ----------------------------------------------------------------------------- Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 0.2%us, 0.3%sy, 0.0%ni, 99.1%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 12217276k total, 11809900k used, 407376k free, 20144k buffers Swap: 4736848k total, 76k used, 4736772k free, 10657464k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23938 ldap 17 0 1320m 1.2g 261m S 0 10.6 23:01.42 slapd Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu(s): 50.0%us, 0.4%sy, 0.0%ni, 49.4%id, 0.1%wa, 0.1%hi, 0.1%si, 0.0%st Mem: 12217276k total, 11775852k used, 441424k free, 181572k buffers Swap: 4149328k total, 76k used, 4149252k free, 8831924k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 28855 ldap 18 0 2844m 2.7g 261m S 200 23.2 62:59.07 slapd This openldap was a HEAD compilation as : env CFLAGS="-ggdb" \ ./configure --enable-syslog --enable-proctitle --with-cyrus-sasl=yes --with-cyrus-sasl=yes \ --with-threads --with-tls --enable-slapd --enable-cleartext --enable-crypt --enable-spasswd \ --enable-dynamic --enable-dynacl --enable-modules --enable-rewrite --enable-backends=mod \ --enable-bdb=mod --enable-monitor=mod --enable-relay=mod --enable-hdb=mod --enable-ndb=mod \ --disable-ldbm --enable-shared --enable-dyngroup=mod --enable-ndb=no \ --enable-overlays=mod --enable-seqmod=mod --enable-syncprov=mod \ --sysconfdir=/etc --prefix=/usr --mandir=/usr/man make depend make make install My machines are(see as OS and Kernel version) : Linux brtldp12 2.6.18-92.1.22.el5PAE #1 SMP Tue Dec 16 12:36:25 EST 2008 i686 i686 i386 GNU/Linux