I randomly notice my OpenLDAP server freezes, and I can't udnerstand why. I have a few LDAP clients (ldapsearch, a legacy Java app. and ApacheDirectoryStudio), running from different systems, either locally on the OpenLDAP server, or on another OpenLDAP server, or on a remote workstation, and none manages to get an answer from OpenLDAP. The connection is established but each client gets stuck waiting for any result.
The last time, it occured only a few minutes after the OpenLDAP daemon was restarted, under very low load. I use OpenLDAP 2.4.23, over Red Hat Enterprise Linux Server release 5.4 (Tikanga), Kernel version 2.6.18-164.el5, amd64 architecture over VMWare. I also use BDB 4.8.30, and OpenLDAP's been compiled with the following options:
./configure --prefix=/opt/openldap-2.4.23 --enable-dynamic=yes --enable-dynlist=yes --enable-memberof=yes --enable-ppolicy=yes --enable-accesslog=yes --enable-sssvlv=yes
I noticed only 60 to 70 established connections on port 389 (according to netstat) when the freeze occurs.
The OpenLDAP server is set in syncrepl mirror mode replication, and the other OpenLDAP server (only used as a hot stand-by) works fine (but doesn't receive any load)
In the OpenLDAP accesslog, I usually see the client requests, but no lines for the responses.
Also, I noticed sometimes the "deferring operation: binding" message :
Jul 12 10:20:05 dev-ldap1 slapd[28525]: connection_input: conn=3377 deferring operation: binding Jul 12 10:20:07 dev-ldap1 slapd[28525]: conn=3378 fd=66 ACCEPT from IP=......:3823 (IP=0.0.0.0:389)
I don't know if this message is linked or not to the main problem so I'd like to know if this message is harmless or not.
The stack of the slapd process is as follows when the freeze occurs:
Thread 18 (Thread 0x40d4b940 (LWP 1893)): #0 0x00000031a7ad4018 in epoll_wait () from /lib64/libc.so.6 #1 0x000000000041d76e in ldap_pvt_sasl_mutex_dispose () #2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x4154c940 (LWP 1894)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002b710a3ad7fb in ldap_pvt_thread_rmutex_lock ()
#2 0x00000000004d4c74 in ldap_pvt_sasl_mutex_dispose () #3 0x0000000000481b4a in ldap_pvt_sasl_mutex_dispose () #4 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #5 0x00000000004de44d in ldap_pvt_sasl_mutex_dispose ()
#6 0x000000000042ee25 in ldap_pvt_sasl_mutex_dispose () #7 0x00000000004319c8 in ldap_pvt_sasl_mutex_dispose () #8 0x0000000000432749 in ldap_pvt_sasl_mutex_dispose () #9 0x00000000004ae91b in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #11 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #12 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose () #13 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#14 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #15 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose () #16 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose () #17 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#18 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper () #19 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #20 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x42771940 (LWP 1895)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002b710a833dd2 in __db_pthread_mutex_lock () #2 0x00002b710a833426 in __db_tas_mutex_lock () #3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c42c2 in __lock_get_pp () #5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose () #6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose () #7 0x00000000004b4183 in ldap_pvt_sasl_mutex_dispose ()
#8 0x00000000004b802e in ldap_pvt_sasl_mutex_dispose () #9 0x0000000000481a7a in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000482547 in ldap_pvt_sasl_mutex_dispose () #11 0x00000000004decab in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000481b4a in ldap_pvt_sasl_mutex_dispose () #13 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #14 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose () #15 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#16 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #17 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose () #18 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose () #19 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#20 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper () #21 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #22 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x42f72940 (LWP 1896)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002b710a833dd2 in __db_pthread_mutex_lock () #2 0x00002b710a833426 in __db_tas_mutex_lock () #3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c42c2 in __lock_get_pp () #5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose () #6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose () #7 0x0000000000492a80 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #9 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000422736 in ldap_pvt_sasl_mutex_dispose () #11 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #13 0x0000000000422ec7 in ldap_pvt_sasl_mutex_dispose () #14 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose () #15 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper () #17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x43773940 (LWP 1897)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002b710a833dd2 in __db_pthread_mutex_lock () #2 0x00002b710a833426 in __db_tas_mutex_lock () #3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c42c2 in __lock_get_pp () #5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose () #6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose () #7 0x0000000000492a80 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #9 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000422736 in ldap_pvt_sasl_mutex_dispose () #11 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #13 0x0000000000422ec7 in ldap_pvt_sasl_mutex_dispose () #14 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose () #15 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper () #17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x43f74940 (LWP 1898)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002b710a833dd2 in __db_pthread_mutex_lock () #2 0x00002b710a833426 in __db_tas_mutex_lock () #3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c42c2 in __lock_get_pp () #5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose () #6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose () #7 0x0000000000492a80 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #9 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000422736 in ldap_pvt_sasl_mutex_dispose () #11 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #13 0x0000000000422ec7 in ldap_pvt_sasl_mutex_dispose () #14 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose () #15 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper () #17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x44775940 (LWP 1902)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002b710a833dd2 in __db_pthread_mutex_lock () #2 0x00002b710a833426 in __db_tas_mutex_lock () #3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c42c2 in __lock_get_pp () #5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose () #6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose () #7 0x0000000000492a80 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #9 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000422736 in ldap_pvt_sasl_mutex_dispose () #11 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #13 0x0000000000422ec7 in ldap_pvt_sasl_mutex_dispose () #14 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose () #15 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper () #17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x41d4d940 (LWP 1947)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002b710a833dd2 in __db_pthread_mutex_lock () #2 0x00002b710a833426 in __db_tas_mutex_lock () #3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c42c2 in __lock_get_pp () #5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose () #6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose () #7 0x0000000000492a80 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #9 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000422736 in ldap_pvt_sasl_mutex_dispose () #11 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #13 0x0000000000422ec7 in ldap_pvt_sasl_mutex_dispose () #14 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose () #15 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper () #17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x44f76940 (LWP 1948)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002b710a3ad7fb in ldap_pvt_thread_rmutex_lock () #2 0x00000000004d4c74 in ldap_pvt_sasl_mutex_dispose () #3 0x0000000000481b4a in ldap_pvt_sasl_mutex_dispose ()
#4 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #5 0x00000000004de44d in ldap_pvt_sasl_mutex_dispose () #6 0x000000000042ee25 in ldap_pvt_sasl_mutex_dispose () #7 0x00000000004319c8 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000432749 in ldap_pvt_sasl_mutex_dispose () #9 0x00000000004ae91b in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #11 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#12 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose () #13 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #14 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #15 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose () #17 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose () #18 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper () #19 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#20 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x45777940 (LWP 1949)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002b710a833dd2 in __db_pthread_mutex_lock ()
#2 0x00002b710a833426 in __db_tas_mutex_lock () #3 0x00002b710a8c3b49 in __lock_get_internal () #4 0x00002b710a8c42c2 in __lock_get_pp () #5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose () #6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose ()
#7 0x00000000004b4183 in ldap_pvt_sasl_mutex_dispose () #8 0x00000000004b802e in ldap_pvt_sasl_mutex_dispose () #9 0x0000000000481a7a in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000482547 in ldap_pvt_sasl_mutex_dispose ()
#11 0x00000000004decab in ldap_pvt_sasl_mutex_dispose () #12 0x0000000000481b4a in ldap_pvt_sasl_mutex_dispose () #13 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #14 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose ()
#15 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #16 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #17 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose () #18 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#19 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose () #20 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper () #21 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #22 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x45f78940 (LWP 1950)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002b710a3ad7fb in ldap_pvt_thread_rmutex_lock () #2 0x00000000004d4c74 in ldap_pvt_sasl_mutex_dispose ()
#3 0x0000000000481b4a in ldap_pvt_sasl_mutex_dispose () #4 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #5 0x00000000004de44d in ldap_pvt_sasl_mutex_dispose () #6 0x000000000042ee25 in ldap_pvt_sasl_mutex_dispose ()
#7 0x00000000004319c8 in ldap_pvt_sasl_mutex_dispose () #8 0x0000000000432749 in ldap_pvt_sasl_mutex_dispose () #9 0x00000000004ae91b in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#11 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #12 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose () #13 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #14 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#15 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose () #16 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose () #17 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose () #18 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#19 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #20 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x46779940 (LWP 1951)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a3ad7fb in ldap_pvt_thread_rmutex_lock () #2 0x00000000004d4c74 in ldap_pvt_sasl_mutex_dispose () #3 0x0000000000481b4a in ldap_pvt_sasl_mutex_dispose () #4 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#5 0x00000000004de44d in ldap_pvt_sasl_mutex_dispose () #6 0x000000000042ee25 in ldap_pvt_sasl_mutex_dispose () #7 0x00000000004319c8 in ldap_pvt_sasl_mutex_dispose () #8 0x0000000000432749 in ldap_pvt_sasl_mutex_dispose ()
#9 0x00000000004ae91b in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #11 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #12 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #14 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #15 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose () #16 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#17 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose () #18 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper () #19 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #20 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x46f7a940 (LWP 1952)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002b710a833dd2 in __db_pthread_mutex_lock () #2 0x00002b710a833426 in __db_tas_mutex_lock () #3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c42c2 in __lock_get_pp () #5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose () #6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose () #7 0x0000000000492a80 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #9 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000422736 in ldap_pvt_sasl_mutex_dispose () #11 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #13 0x0000000000422ec7 in ldap_pvt_sasl_mutex_dispose () #14 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose () #15 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper () #17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x4777b940 (LWP 2149)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002b710a833dd2 in __db_pthread_mutex_lock () #2 0x00002b710a833426 in __db_tas_mutex_lock () #3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c44ea in __lock_vec () #5 0x00002b710a8c512b in __lock_vec_pp () #6 0x00000000004af514 in ldap_pvt_sasl_mutex_dispose () #7 0x00000000004af900 in ldap_pvt_sasl_mutex_dispose () #8 0x000000000048d323 in ldap_pvt_sasl_mutex_dispose ()
#9 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #11 0x00000000004de44d in ldap_pvt_sasl_mutex_dispose () #12 0x000000000042ee25 in ldap_pvt_sasl_mutex_dispose ()
#13 0x00000000004319c8 in ldap_pvt_sasl_mutex_dispose () #14 0x0000000000432749 in ldap_pvt_sasl_mutex_dispose () #15 0x00000000004ae91b in ldap_pvt_sasl_mutex_dispose () #16 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#17 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #18 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose () #19 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #20 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#21 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose () #22 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose () #23 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose () #24 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#25 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #26 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x47f7c940 (LWP 2150)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a3ad7fb in ldap_pvt_thread_rmutex_lock () #2 0x00000000004d4c74 in ldap_pvt_sasl_mutex_dispose () #3 0x0000000000481b4a in ldap_pvt_sasl_mutex_dispose () #4 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#5 0x00000000004de44d in ldap_pvt_sasl_mutex_dispose () #6 0x000000000042ee25 in ldap_pvt_sasl_mutex_dispose () #7 0x00000000004319c8 in ldap_pvt_sasl_mutex_dispose () #8 0x0000000000432749 in ldap_pvt_sasl_mutex_dispose ()
#9 0x00000000004ae91b in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #11 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #12 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #14 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #15 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose () #16 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#17 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose () #18 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper () #19 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #20 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x4877d940 (LWP 2151)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002b710a833dd2 in __db_pthread_mutex_lock () #2 0x00002b710a833426 in __db_tas_mutex_lock () #3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c42c2 in __lock_get_pp () #5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose () #6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose () #7 0x0000000000492a80 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #9 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000422736 in ldap_pvt_sasl_mutex_dispose () #11 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #13 0x0000000000422ec7 in ldap_pvt_sasl_mutex_dispose () #14 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose () #15 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper () #17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x48f7e940 (LWP 2152)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002b710a3ad7fb in ldap_pvt_thread_rmutex_lock () #2 0x00000000004d4c74 in ldap_pvt_sasl_mutex_dispose () #3 0x0000000000481b4a in ldap_pvt_sasl_mutex_dispose ()
#4 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #5 0x00000000004de44d in ldap_pvt_sasl_mutex_dispose () #6 0x000000000042ee25 in ldap_pvt_sasl_mutex_dispose () #7 0x00000000004319c8 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000432749 in ldap_pvt_sasl_mutex_dispose () #9 0x00000000004ae91b in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #11 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#12 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose () #13 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose () #14 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose () #15 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose () #17 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose () #18 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper () #19 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#20 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x2b710ab80bd0 (LWP 1892)): #0 0x00000031a86077e5 in pthread_join () from /lib64/libpthread.so.0 #1 0x000000000041b3e2 in ldap_pvt_sasl_mutex_dispose ()
#2 0x000000000040960e in ldap_pvt_sasl_mutex_dispose () #3 0x00000031a7a1d994 in __libc_start_main () from /lib64/libc.so.6 #4 0x00000000004080d9 in ldap_pvt_sasl_mutex_dispose () #5 0x00007fff4c26b148 in ?? ()
#6 0x0000000000000000 in ?? ()
Any idea of what's wrong ? Known bug ?
--On Tuesday, August 02, 2011 6:28 PM +0200 Cyril GROSJEAN cgrosjean@janua.fr wrote:
Any idea of what's wrong ? Known bug ?
Can you try with 2.4.26? There was an issue with 2.4.23 that may be the culprit.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
OK, I'll try to see if version 2.4.26 is the cure ...
Quanah Gibson-Mount a écrit :
--On Tuesday, August 02, 2011 6:28 PM +0200 Cyril GROSJEAN cgrosjean@janua.fr wrote:
Any idea of what's wrong ? Known bug ?
Can you try with 2.4.26? There was an issue with 2.4.23 that may be the culprit.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc.
Zimbra :: the leader in open source messaging and collaboration
I've just also tested with the standard OpenLDAP rpm packages on RHEL 6, but the problem remains. Those packages are:
openldap-2.4.23-12.el6.i686 openldap-servers-2.4.23-12.el6.i686
Since the slapd process still doesn't respond, what traces do you need to investigate ?
I've just tested with OpenLDAP 2.4.26, as you suggested, but the problem remains.
I've no load at all, I use mirror mode replication, and this time, the freeze occured when trying to modify a user's password: The modification is done from a usual Web interface from a Java application. The Java application is set to connect to OpenLDAP with a unique dedicated account, but in the case of a password change, we use the PROXYAUTH operation to do the modification with the user's rights, so that his pwdreset attribute is removed if previously set to TRUE
Below is the connection that brings the freeze. We never get an answer from operation 21:
Aug 4 13:43:28 rhvtq slapd-master1[23372]: conn=1201 fd=139 ACCEPT from IP=192.168.1.132:57374 (IP=192.168.1.132:1389) Aug 4 13:43:28 rhvtq slapd-master1[23372]: conn=1201 op=0 BIND dn="cn=yellow pages,ou=special users,dc=company" method=128 Aug 4 13:43:28 rhvtq slapd-master1[23372]: conn=1201 op=0 BIND dn="cn=yellow pages,ou=special users,dc=company" mech=SIMPLE ssf=0 Aug 4 13:43:28 rhvtq slapd-master1[23372]: conn=1201 op=0 RESULT tag=97 err=0 text= ... ... ... Aug 4 13:46:40 rhvtq slapd-master1[23372]: conn=1201 op=20 SEARCH RESULT tag=101 err=0 nentries=1 text= Aug 4 13:46:40 rhvtq slapd-master1[23372]: conn=1201 op=21 PROXYAUTHZ dn="employeeNumber=00296,ou=people,dc=company" Aug 4 13:46:40 rhvtq slapd-master1[23372]: conn=1201 op=21 MOD dn="employeeNumber=00296,ou=people,dc=company" Aug 4 13:46:40 rhvtq slapd-master1[23372]: conn=1201 op=21 MOD attr=userPassword
And here's the slapd pstack output :
Thread 18 (Thread 0x42018940 (LWP 23373)): #0 0x00000031a7ad4018 in epoll_wait () from /lib64/libc.so.6 #1 0x000000000041eb0b in ldap_pvt_sasl_mutex_dispose () #2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x42819940 (LWP 23374)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper () #2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x4301a940 (LWP 23375)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper () #2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x4381b940 (LWP 23376)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper () #2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x4401c940 (LWP 23377)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper () #2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x4481d940 (LWP 23381)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002afb09199e8b in ldap_pvt_thread_rmutex_lock () #2 0x00000000004d7fcf in ?? () #3 0x000000000048402a in ldap_pvt_sasl_mutex_dispose () #4 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose () #5 0x0000000000477c2f in ldap_pvt_sasl_mutex_dispose () #6 0x000000000047f2eb in ldap_pvt_sasl_mutex_dispose () #7 0x0000000000422713 in ldap_pvt_sasl_mutex_dispose () #8 0x00002afb0919af38 in ldap_int_thread_pool_wrapper () #9 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #10 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x4501e940 (LWP 23382)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002afb09622dd2 in __db_pthread_mutex_lock () #2 0x00002afb09622426 in __db_tas_mutex_lock () #3 0x00002afb096b2b49 in __lock_get_internal () #4 0x00002afb096b32c2 in __lock_get_pp () #5 0x00000000004b2c28 in ?? () #6 0x00000000004b3ab9 in ?? () #7 0x00000000004955ca in ldap_pvt_sasl_mutex_dispose () #8 0x00000000004840a2 in ldap_pvt_sasl_mutex_dispose () #9 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000423a49 in ldap_pvt_sasl_mutex_dispose () #11 0x00000000004840a2 in ldap_pvt_sasl_mutex_dispose () #12 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose () #13 0x0000000000424255 in ldap_pvt_sasl_mutex_dispose () #14 0x00000000004217e5 in ldap_pvt_sasl_mutex_dispose () #15 0x0000000000421dbf in ldap_pvt_sasl_mutex_dispose () #16 0x00002afb0919af38 in ldap_int_thread_pool_wrapper () #17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x4581f940 (LWP 25216)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper () #2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x46020940 (LWP 25217)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002afb09622dd2 in __db_pthread_mutex_lock () #2 0x00002afb09622426 in __db_tas_mutex_lock () #3 0x00002afb096b2b49 in __lock_get_internal () #4 0x00002afb096b32c2 in __lock_get_pp () #5 0x00000000004b2c28 in ?? () #6 0x00000000004b3ab9 in ?? () #7 0x00000000004b75e3 in ?? () #8 0x00000000004bb3ce in ?? () #9 0x0000000000483f5a in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000484a67 in ldap_pvt_sasl_mutex_dispose () #11 0x00000000004e226b in ?? () #12 0x000000000048402a in ldap_pvt_sasl_mutex_dispose () #13 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose () #14 0x000000000043d21e in ldap_pvt_sasl_mutex_dispose () #15 0x00000000004840a2 in ldap_pvt_sasl_mutex_dispose () #16 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose () #17 0x000000000043dc1f in ldap_pvt_sasl_mutex_dispose () #18 0x00000000004217e5 in ldap_pvt_sasl_mutex_dispose () #19 0x0000000000421dbf in ldap_pvt_sasl_mutex_dispose () #20 0x00002afb0919af38 in ldap_int_thread_pool_wrapper () #21 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #22 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x46821940 (LWP 25218)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002afb09622dd2 in __db_pthread_mutex_lock () #2 0x00002afb09622426 in __db_tas_mutex_lock () #3 0x00002afb096b2b49 in __lock_get_internal () #4 0x00002afb096b34ea in __lock_vec () #5 0x00002afb096b412b in __lock_vec_pp () #6 0x00000000004b3264 in ?? () #7 0x00000000004b3650 in ?? () #8 0x000000000048fe13 in ldap_pvt_sasl_mutex_dispose () #9 0x00000000004840a2 in ldap_pvt_sasl_mutex_dispose () #10 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose () #11 0x0000000000438847 in ldap_pvt_sasl_mutex_dispose () #12 0x00000000004840a2 in ldap_pvt_sasl_mutex_dispose () #13 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose () #14 0x0000000000438fb2 in ldap_pvt_sasl_mutex_dispose () #15 0x00000000004217e5 in ldap_pvt_sasl_mutex_dispose () #16 0x0000000000421dbf in ldap_pvt_sasl_mutex_dispose () #17 0x00002afb0919af38 in ldap_int_thread_pool_wrapper () #18 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #19 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x47022940 (LWP 25219)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper () #2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x47823940 (LWP 25220)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper () #2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x48024940 (LWP 25221)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper () #2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x48825940 (LWP 25222)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper () #2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x49026940 (LWP 25223)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper () #2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x49827940 (LWP 25224)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper () #2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x4a028940 (LWP 25225)): #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00002afb09199e8b in ldap_pvt_thread_rmutex_lock () #2 0x00000000004d7fcf in ?? () #3 0x000000000048402a in ldap_pvt_sasl_mutex_dispose () #4 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose () #5 0x00000000004e1a08 in ?? () #6 0x0000000000430285 in ldap_pvt_sasl_mutex_dispose () #7 0x0000000000430cc8 in ldap_pvt_sasl_mutex_dispose () #8 0x0000000000431a49 in ldap_pvt_sasl_mutex_dispose () #9 0x000000000043ce10 in ldap_pvt_sasl_mutex_dispose () #10 0x000000000043d4b4 in ldap_pvt_sasl_mutex_dispose () #11 0x00000000004840a2 in ldap_pvt_sasl_mutex_dispose () #12 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose () #13 0x000000000043dc1f in ldap_pvt_sasl_mutex_dispose () #14 0x00000000004217e5 in ldap_pvt_sasl_mutex_dispose () #15 0x0000000000421dbf in ldap_pvt_sasl_mutex_dispose () #16 0x00002afb0919af38 in ldap_int_thread_pool_wrapper () #17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x2afb0996fbd0 (LWP 23372)): #0 0x00000031a86077e5 in pthread_join () from /lib64/libpthread.so.0 #1 0x000000000041c16d in ldap_pvt_sasl_mutex_dispose () #2 0x000000000040997b in ldap_pvt_sasl_mutex_dispose () #3 0x00000031a7a1d994 in __libc_start_main () from /lib64/libc.so.6 #4 0x0000000000408389 in ldap_pvt_sasl_mutex_dispose () #5 0x00007fffd4f51358 in ?? () #6 0x0000000000000000 in ?? ()
I've also opened a case with Red Hat, they suggested me to test under RHEL 6, with their OpenLDAP rpm package (2.4.23-15.el6). So I'll do it right now but if you need more details on the RHEL 5.4 hung slapd process, I can send them since it's still frozen.
Quanah Gibson-Mount a écrit :
--On Tuesday, August 02, 2011 6:28 PM +0200 Cyril GROSJEAN cgrosjean@janua.fr wrote:
Any idea of what's wrong ? Known bug ?
Can you try with 2.4.26? There was an issue with 2.4.23 that may be the culprit.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc.
Zimbra :: the leader in open source messaging and collaboration
--On Thursday, August 04, 2011 2:23 PM +0200 Cyril Grosjean cgrosjean@janua.fr wrote:
I've just tested with OpenLDAP 2.4.26, as you suggested, but the problem remains.
I've no load at all, I use mirror mode replication, and this time, the freeze occurred when trying to modify a user's password: The modification is done from a usual Web interface from a Java application. The Java application is set to connect to OpenLDAP with a unique dedicated account, but in the case of a password change, we use the PROXYAUTH operation to do the modification with the user's rights, so that his pwdreset attribute is removed if previously set to TRUE
I've also opened a case with Red Hat, they suggested me to test under RHEL 6, with their OpenLDAP rpm package (2.4.23-15.el6). So I'll do it right now but if you need more details on the RHEL 5.4 hung slapd process, I can send them since it's still frozen.
Please build with debugging symbols and no optimization (-O0 and -g). Make sure when you install, you set STRIP="" so it doesn't strip the symbols out. Then get a full backtrace.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Before building OpenLDAP with debugging symbols and no optimization, we noticed some possible locks (with db_stat) on the bdb (4.8.30) back-ends, that we can reproduce with the main Java application that interacts with OpenLDAP. It uses to occur when we create a few users. So, it could possibly be a problem with the way our Java application works with OpenLDAP, even if once OpenLDAP is frozen, other usual LDAP clients such a ApacheDirectoryStudio or ldapsearch can't connect (bind request sent but no response) to OpenLDAP. Also, the same main Java application didn't use to freeze our previous (Sun DS 6) directory.
So, we now first intend to replace bdb and test with hdb or possibly some other database back-end, before testing with debugging symbols and no optimization. I'll let you know the results of of our tests ..
Our tests with hdb haven't improve the situation. Even after a few binds from our Java application or thanks to ldapsearch, OpenLDAP doesn't respond. We did those tests on RHEL 5.4 + OpenLDAP 2.4.26 + Berkeley DB 4.8.30. We also got the freeze with BDB 5.1.25, but didn't test with BDB 5.2.28 ..
So, we'll now build with debugging symbols and no optimization and follow your other recommandations ..
Here're the traces you required, for the 1st 4 OpenLDAP threads at least:
(gdb) attach 20884 Attaching to program: /opt/openldap-2.4.26/libexec/slapd, process 20884 Reading symbols from /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2...done. Loaded symbols for /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2 Reading symbols from /opt/openldap-2.4.26/lib/liblber-2.4.so.2...done. Loaded symbols for /opt/openldap-2.4.26/lib/liblber-2.4.so.2 Reading symbols from /lib64/libuuid.so.1...done. Loaded symbols for /lib64/libuuid.so.1 Reading symbols from /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so...done. Loaded symbols for /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so Reading symbols from /lib64/libpthread.so.0...done. [Thread debugging using libthread_db enabled] [New Thread 0x2b1c9a53dbd0 (LWP 20884)] [New Thread 0x48a36940 (LWP 21499)] [New Thread 0x40930940 (LWP 21498)] [New Thread 0x48235940 (LWP 21203)] [New Thread 0x47a34940 (LWP 21202)] [New Thread 0x47233940 (LWP 21201)] [New Thread 0x46a32940 (LWP 21200)] [New Thread 0x46231940 (LWP 21087)] [New Thread 0x45a30940 (LWP 21086)] [New Thread 0x4522f940 (LWP 21021)] [New Thread 0x44a2e940 (LWP 21020)] [New Thread 0x4422d940 (LWP 20891)] [New Thread 0x43a2c940 (LWP 20890)] [New Thread 0x4322b940 (LWP 20889)] [New Thread 0x42a2a940 (LWP 20888)] [New Thread 0x42229940 (LWP 20887)] [New Thread 0x41a28940 (LWP 20886)] [New Thread 0x41227940 (LWP 20885)] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /usr/lib64/libsasl2.so.2...done. Loaded symbols for /usr/lib64/libsasl2.so.2 Reading symbols from /lib64/libssl.so.6...done. Loaded symbols for /lib64/libssl.so.6 Reading symbols from /lib64/libcrypto.so.6...done. Loaded symbols for /lib64/libcrypto.so.6 Reading symbols from /lib64/libresolv.so.2...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libc.so.6...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libdl.so.2...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libcrypt.so.1...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done. Loaded symbols for /usr/lib64/libgssapi_krb5.so.2 Reading symbols from /usr/lib64/libkrb5.so.3...done. Loaded symbols for /usr/lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /usr/lib64/libk5crypto.so.3...done. Loaded symbols for /usr/lib64/libk5crypto.so.3 Reading symbols from /usr/lib64/libz.so.1...done. Loaded symbols for /usr/lib64/libz.so.1 Reading symbols from /usr/lib64/libkrb5support.so.0...done. Loaded symbols for /usr/lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libselinux.so.1...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libsepol.so.1...done. Loaded symbols for /lib64/libsepol.so.1 Reading symbols from /lib64/libnss_files.so.2...done. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /lib64/libnss_dns.so.2...done. Loaded symbols for /lib64/libnss_dns.so.2 Reading symbols from /usr/lib64/sasl2/liblogin.so.2...done. Loaded symbols for /usr/lib64/sasl2/liblogin.so.2 Reading symbols from /usr/lib64/sasl2/libanonymous.so.2...done. Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2 Reading symbols from /usr/lib64/sasl2/libplain.so.2...done. Loaded symbols for /usr/lib64/sasl2/libplain.so.2 Reading symbols from /usr/lib64/sasl2/libsasldb.so.2...done. Loaded symbols for /usr/lib64/sasl2/libsasldb.so.2 0x00000031a86077e5 in pthread_join () from /lib64/libpthread.so.0 (gdb) bt #0 0x00000031a86077e5 in pthread_join () from /lib64/libpthread.so.0 #1 0x000000000042aca4 in slapd_daemon () at daemon.c:2922 #2 0x0000000000409f34 in main (argc=7, argv=0x7ffff62c9618) at main.c:983 (gdb) detach Detaching from program: /opt/openldap-2.4.26/libexec/slapd, process 20884 (gdb) attach 21499 Attaching to program: /opt/openldap-2.4.26/libexec/slapd, process 21499 warning: process 21499 is a cloned process Reading symbols from /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2...done. Loaded symbols for /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2 Reading symbols from /opt/openldap-2.4.26/lib/liblber-2.4.so.2...done. Loaded symbols for /opt/openldap-2.4.26/lib/liblber-2.4.so.2 Reading symbols from /lib64/libuuid.so.1...done. Loaded symbols for /lib64/libuuid.so.1 Reading symbols from /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so...done. Loaded symbols for /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so Reading symbols from /lib64/libpthread.so.0...done. [Thread debugging using libthread_db enabled] [New Thread 0x2b1c9a53dbd0 (LWP 20884)] [New Thread 0x48a36940 (LWP 21499)] [New Thread 0x40930940 (LWP 21498)] [New Thread 0x48235940 (LWP 21203)] [New Thread 0x47a34940 (LWP 21202)] [New Thread 0x47233940 (LWP 21201)] [New Thread 0x46a32940 (LWP 21200)] [New Thread 0x46231940 (LWP 21087)] [New Thread 0x45a30940 (LWP 21086)] [New Thread 0x4522f940 (LWP 21021)] [New Thread 0x44a2e940 (LWP 21020)] [New Thread 0x4422d940 (LWP 20891)] [New Thread 0x43a2c940 (LWP 20890)] [New Thread 0x4322b940 (LWP 20889)] [New Thread 0x42a2a940 (LWP 20888)] [New Thread 0x42229940 (LWP 20887)] [New Thread 0x41a28940 (LWP 20886)] [New Thread 0x41227940 (LWP 20885)] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /usr/lib64/libsasl2.so.2...done. Loaded symbols for /usr/lib64/libsasl2.so.2 Reading symbols from /lib64/libssl.so.6...done. Loaded symbols for /lib64/libssl.so.6 Reading symbols from /lib64/libcrypto.so.6...done. Loaded symbols for /lib64/libcrypto.so.6 Reading symbols from /lib64/libresolv.so.2...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libc.so.6...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libdl.so.2...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libcrypt.so.1...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done. Loaded symbols for /usr/lib64/libgssapi_krb5.so.2 Reading symbols from /usr/lib64/libkrb5.so.3...done. Loaded symbols for /usr/lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /usr/lib64/libk5crypto.so.3...done. Loaded symbols for /usr/lib64/libk5crypto.so.3 Reading symbols from /usr/lib64/libz.so.1...done. Loaded symbols for /usr/lib64/libz.so.1 Reading symbols from /usr/lib64/libkrb5support.so.0...done. Loaded symbols for /usr/lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libselinux.so.1...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libsepol.so.1...done. Loaded symbols for /lib64/libsepol.so.1 Reading symbols from /lib64/libnss_files.so.2...done. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /lib64/libnss_dns.so.2...done. Loaded symbols for /lib64/libnss_dns.so.2 Reading symbols from /usr/lib64/sasl2/liblogin.so.2...done. Loaded symbols for /usr/lib64/sasl2/liblogin.so.2 Reading symbols from /usr/lib64/sasl2/libanonymous.so.2...done. Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2 Reading symbols from /usr/lib64/sasl2/libplain.so.2...done. Loaded symbols for /usr/lib64/sasl2/libplain.so.2 Reading symbols from /usr/lib64/sasl2/libsasldb.so.2...done. Loaded symbols for /usr/lib64/sasl2/libsasldb.so.2 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) bt #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b1c99d65f1b in ldap_pvt_thread_rmutex_lock (rmutex=<value optimized out>, owner=1218668864) at rmutex.c:129 #2 0x000000000052efb7 in accesslog_op_mod (op=0x48a34fe0, rs=0x48a35150) at accesslog.c:1889 #3 0x00000000004bbb2b in overlay_op_walk (op=0x48a34fe0, rs=0x48a35150, which=op_modify, oi=0x1ed31f10, on=0x1ed32d60) at backover.c:661 #4 0x00000000004bbdbd in over_op_func (op=0x48a34fe0, rs=0x48a35150, which=op_modify) at backover.c:723 #5 0x00000000004bbed3 in over_op_modify (op=0x48a34fe0, rs=0x48a35150) at backover.c:762 #6 0x000000000053c7af in ppolicy_bind_response (op=0x1fbf8d90, rs=0x48a35c30) at ppolicy.c:1173 #7 0x0000000000443a08 in slap_response_play (op=0x1fbf8d90, rs=0x48a35c30) at result.c:505 #8 0x0000000000443c2f in send_ldap_response (op=0x1fbf8d90, rs=0x48a35c30) at result.c:579 #9 0x0000000000444b14 in slap_send_ldap_result (op=0x1fbf8d90, rs=0x48a35c30) at result.c:851 #10 0x000000000050f2f3 in hdb_bind (op=0x1fbf8d90, rs=0x48a35c30) at bind.c:158 #11 0x00000000004bbbba in overlay_op_walk (op=0x1fbf8d90, rs=0x48a35c30, which=op_bind, oi=0x1ed31f10, on=0x0) at backover.c:671 #12 0x00000000004bbdbd in over_op_func (op=0x1fbf8d90, rs=0x48a35c30, which=op_bind) at backover.c:723 #13 0x00000000004bbe43 in over_op_bind (op=0x1fbf8d90, rs=0x48a35c30) at backover.c:738 #14 0x0000000000456411 in fe_op_bind (op=0x1fbf8d90, rs=0x48a35c30) at bind.c:383 #15 0x00000000004bbbba in overlay_op_walk (op=0x1fbf8d90, rs=0x48a35c30, which=op_bind, oi=0x1ed02760, on=0x0) at backover.c:671 #16 0x00000000004bbdbd in over_op_func (op=0x1fbf8d90, rs=0x48a35c30, which=op_bind) at backover.c:723 #17 0x00000000004bbe43 in over_op_bind (op=0x1fbf8d90, rs=0x48a35c30) at backover.c:738 #18 0x0000000000455b40 in do_bind (op=0x1fbf8d90, rs=0x48a35c30) at bind.c:205 #19 0x000000000042dbce in connection_operation (ctx=0x48a35d70, arg_v=0x1fbf8d90) at connection.c:1138 #20 0x000000000042e153 in connection_read_thread (ctx=0x48a35d70, argv=0x30) at connection.c:1274 #21 0x00002b1c99d66fc8 in ldap_int_thread_pool_wrapper (xpool=0x1ec9acb0) at tpool.c:685 #22 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #23 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 (gdb) detach Detaching from program: /opt/openldap-2.4.26/libexec/slapd, process 21499 (gdb) attach 21498 Attaching to program: /opt/openldap-2.4.26/libexec/slapd, process 21498 warning: process 21498 is a cloned process Reading symbols from /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2...done. Loaded symbols for /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2 Reading symbols from /opt/openldap-2.4.26/lib/liblber-2.4.so.2...done. Loaded symbols for /opt/openldap-2.4.26/lib/liblber-2.4.so.2 Reading symbols from /lib64/libuuid.so.1...done. Loaded symbols for /lib64/libuuid.so.1 Reading symbols from /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so...done. Loaded symbols for /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so Reading symbols from /lib64/libpthread.so.0...done. [Thread debugging using libthread_db enabled] [New Thread 0x2b1c9a53dbd0 (LWP 20884)] [New Thread 0x48a36940 (LWP 21499)] [New Thread 0x40930940 (LWP 21498)] [New Thread 0x48235940 (LWP 21203)] [New Thread 0x47a34940 (LWP 21202)] [New Thread 0x47233940 (LWP 21201)] [New Thread 0x46a32940 (LWP 21200)] [New Thread 0x46231940 (LWP 21087)] [New Thread 0x45a30940 (LWP 21086)] [New Thread 0x4522f940 (LWP 21021)] [New Thread 0x44a2e940 (LWP 21020)] [New Thread 0x4422d940 (LWP 20891)] [New Thread 0x43a2c940 (LWP 20890)] [New Thread 0x4322b940 (LWP 20889)] [New Thread 0x42a2a940 (LWP 20888)] [New Thread 0x42229940 (LWP 20887)] [New Thread 0x41a28940 (LWP 20886)] [New Thread 0x41227940 (LWP 20885)] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /usr/lib64/libsasl2.so.2...done. Loaded symbols for /usr/lib64/libsasl2.so.2 Reading symbols from /lib64/libssl.so.6...done. Loaded symbols for /lib64/libssl.so.6 Reading symbols from /lib64/libcrypto.so.6...done. Loaded symbols for /lib64/libcrypto.so.6 Reading symbols from /lib64/libresolv.so.2...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libc.so.6...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libdl.so.2...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libcrypt.so.1...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done. Loaded symbols for /usr/lib64/libgssapi_krb5.so.2 Reading symbols from /usr/lib64/libkrb5.so.3...done. Loaded symbols for /usr/lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /usr/lib64/libk5crypto.so.3...done. Loaded symbols for /usr/lib64/libk5crypto.so.3 Reading symbols from /usr/lib64/libz.so.1...done. Loaded symbols for /usr/lib64/libz.so.1 Reading symbols from /usr/lib64/libkrb5support.so.0...done. Loaded symbols for /usr/lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libselinux.so.1...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libsepol.so.1...done. Loaded symbols for /lib64/libsepol.so.1 Reading symbols from /lib64/libnss_files.so.2...done. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /lib64/libnss_dns.so.2...done. Loaded symbols for /lib64/libnss_dns.so.2 Reading symbols from /usr/lib64/sasl2/liblogin.so.2...done. Loaded symbols for /usr/lib64/sasl2/liblogin.so.2 Reading symbols from /usr/lib64/sasl2/libanonymous.so.2...done. Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2 Reading symbols from /usr/lib64/sasl2/libplain.so.2...done. Loaded symbols for /usr/lib64/sasl2/libplain.so.2 Reading symbols from /usr/lib64/sasl2/libsasldb.so.2...done. Loaded symbols for /usr/lib64/sasl2/libsasldb.so.2 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) bt #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b1c99d65f1b in ldap_pvt_thread_rmutex_lock (rmutex=<value optimized out>, owner=1083377984) at rmutex.c:129 #2 0x000000000052efb7 in accesslog_op_mod (op=0x4092efe0, rs=0x4092f150) at accesslog.c:1889 #3 0x00000000004bbb2b in overlay_op_walk (op=0x4092efe0, rs=0x4092f150, which=op_modify, oi=0x1ed31f10, on=0x1ed32d60) at backover.c:661 #4 0x00000000004bbdbd in over_op_func (op=0x4092efe0, rs=0x4092f150, which=op_modify) at backover.c:723 #5 0x00000000004bbed3 in over_op_modify (op=0x4092efe0, rs=0x4092f150) at backover.c:762 #6 0x000000000053c7af in ppolicy_bind_response (op=0x1f5bf360, rs=0x4092fc30) at ppolicy.c:1173 #7 0x0000000000443a08 in slap_response_play (op=0x1f5bf360, rs=0x4092fc30) at result.c:505 #8 0x0000000000443c2f in send_ldap_response (op=0x1f5bf360, rs=0x4092fc30) at result.c:579 #9 0x0000000000444b14 in slap_send_ldap_result (op=0x1f5bf360, rs=0x4092fc30) at result.c:851 #10 0x000000000050f2f3 in hdb_bind (op=0x1f5bf360, rs=0x4092fc30) at bind.c:158 #11 0x00000000004bbbba in overlay_op_walk (op=0x1f5bf360, rs=0x4092fc30, which=op_bind, oi=0x1ed31f10, on=0x0) at backover.c:671 #12 0x00000000004bbdbd in over_op_func (op=0x1f5bf360, rs=0x4092fc30, which=op_bind) at backover.c:723 #13 0x00000000004bbe43 in over_op_bind (op=0x1f5bf360, rs=0x4092fc30) at backover.c:738 #14 0x0000000000456411 in fe_op_bind (op=0x1f5bf360, rs=0x4092fc30) at bind.c:383 #15 0x00000000004bbbba in overlay_op_walk (op=0x1f5bf360, rs=0x4092fc30, which=op_bind, oi=0x1ed02760, on=0x0) at backover.c:671 #16 0x00000000004bbdbd in over_op_func (op=0x1f5bf360, rs=0x4092fc30, which=op_bind) at backover.c:723 #17 0x00000000004bbe43 in over_op_bind (op=0x1f5bf360, rs=0x4092fc30) at backover.c:738 #18 0x0000000000455b40 in do_bind (op=0x1f5bf360, rs=0x4092fc30) at bind.c:205 #19 0x000000000042dbce in connection_operation (ctx=0x4092fd70, arg_v=0x1f5bf360) at connection.c:1138 #20 0x000000000042e153 in connection_read_thread (ctx=0x4092fd70, argv=0x25) at connection.c:1274 #21 0x00002b1c99d66fc8 in ldap_int_thread_pool_wrapper (xpool=0x1ec9acb0) at tpool.c:685 #22 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #23 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 (gdb) detach Detaching from program: /opt/openldap-2.4.26/libexec/slapd, process 21498 (gdb) attach 21203 Attaching to program: /opt/openldap-2.4.26/libexec/slapd, process 21203 warning: process 21203 is a cloned process Reading symbols from /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2...done. Loaded symbols for /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2 Reading symbols from /opt/openldap-2.4.26/lib/liblber-2.4.so.2...done. Loaded symbols for /opt/openldap-2.4.26/lib/liblber-2.4.so.2 Reading symbols from /lib64/libuuid.so.1...done. Loaded symbols for /lib64/libuuid.so.1 Reading symbols from /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so...done. Loaded symbols for /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so Reading symbols from /lib64/libpthread.so.0...done. [Thread debugging using libthread_db enabled] [New Thread 0x2b1c9a53dbd0 (LWP 20884)] [New Thread 0x48a36940 (LWP 21499)] [New Thread 0x40930940 (LWP 21498)] [New Thread 0x48235940 (LWP 21203)] [New Thread 0x47a34940 (LWP 21202)] [New Thread 0x47233940 (LWP 21201)] [New Thread 0x46a32940 (LWP 21200)] [New Thread 0x46231940 (LWP 21087)] [New Thread 0x45a30940 (LWP 21086)] [New Thread 0x4522f940 (LWP 21021)] [New Thread 0x44a2e940 (LWP 21020)] [New Thread 0x4422d940 (LWP 20891)] [New Thread 0x43a2c940 (LWP 20890)] [New Thread 0x4322b940 (LWP 20889)] [New Thread 0x42a2a940 (LWP 20888)] [New Thread 0x42229940 (LWP 20887)] [New Thread 0x41a28940 (LWP 20886)] [New Thread 0x41227940 (LWP 20885)] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /usr/lib64/libsasl2.so.2...done. Loaded symbols for /usr/lib64/libsasl2.so.2 Reading symbols from /lib64/libssl.so.6...done. Loaded symbols for /lib64/libssl.so.6 Reading symbols from /lib64/libcrypto.so.6...done. Loaded symbols for /lib64/libcrypto.so.6 Reading symbols from /lib64/libresolv.so.2...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libc.so.6...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libdl.so.2...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libcrypt.so.1...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done. Loaded symbols for /usr/lib64/libgssapi_krb5.so.2 Reading symbols from /usr/lib64/libkrb5.so.3...done. Loaded symbols for /usr/lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /usr/lib64/libk5crypto.so.3...done. Loaded symbols for /usr/lib64/libk5crypto.so.3 Reading symbols from /usr/lib64/libz.so.1...done. Loaded symbols for /usr/lib64/libz.so.1 Reading symbols from /usr/lib64/libkrb5support.so.0...done. Loaded symbols for /usr/lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libselinux.so.1...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libsepol.so.1...done. Loaded symbols for /lib64/libsepol.so.1 Reading symbols from /lib64/libnss_files.so.2...done. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /lib64/libnss_dns.so.2...done. Loaded symbols for /lib64/libnss_dns.so.2 Reading symbols from /usr/lib64/sasl2/liblogin.so.2...done. Loaded symbols for /usr/lib64/sasl2/liblogin.so.2 Reading symbols from /usr/lib64/sasl2/libanonymous.so.2...done. Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2 Reading symbols from /usr/lib64/sasl2/libplain.so.2...done. Loaded symbols for /usr/lib64/sasl2/libplain.so.2 Reading symbols from /usr/lib64/sasl2/libsasldb.so.2...done. Loaded symbols for /usr/lib64/sasl2/libsasldb.so.2 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) bt #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b1c99d65f1b in ldap_pvt_thread_rmutex_lock (rmutex=<value optimized out>, owner=1210276160) at rmutex.c:129 #2 0x000000000052efb7 in accesslog_op_mod (op=0x48233fe0, rs=0x48234150) at accesslog.c:1889 #3 0x00000000004bbb2b in overlay_op_walk (op=0x48233fe0, rs=0x48234150, which=op_modify, oi=0x1ed31f10, on=0x1ed32d60) at backover.c:661 #4 0x00000000004bbdbd in over_op_func (op=0x48233fe0, rs=0x48234150, which=op_modify) at backover.c:723 #5 0x00000000004bbed3 in over_op_modify (op=0x48233fe0, rs=0x48234150) at backover.c:762 #6 0x000000000053c7af in ppolicy_bind_response (op=0x1fe91140, rs=0x48234c30) at ppolicy.c:1173 #7 0x0000000000443a08 in slap_response_play (op=0x1fe91140, rs=0x48234c30) at result.c:505 #8 0x0000000000443c2f in send_ldap_response (op=0x1fe91140, rs=0x48234c30) at result.c:579 #9 0x0000000000444b14 in slap_send_ldap_result (op=0x1fe91140, rs=0x48234c30) at result.c:851 #10 0x000000000050f2f3 in hdb_bind (op=0x1fe91140, rs=0x48234c30) at bind.c:158 #11 0x00000000004bbbba in overlay_op_walk (op=0x1fe91140, rs=0x48234c30, which=op_bind, oi=0x1ed31f10, on=0x0) at backover.c:671 #12 0x00000000004bbdbd in over_op_func (op=0x1fe91140, rs=0x48234c30, which=op_bind) at backover.c:723 #13 0x00000000004bbe43 in over_op_bind (op=0x1fe91140, rs=0x48234c30) at backover.c:738 #14 0x0000000000456411 in fe_op_bind (op=0x1fe91140, rs=0x48234c30) at bind.c:383 #15 0x00000000004bbbba in overlay_op_walk (op=0x1fe91140, rs=0x48234c30, which=op_bind, oi=0x1ed02760, on=0x0) at backover.c:671 #16 0x00000000004bbdbd in over_op_func (op=0x1fe91140, rs=0x48234c30, which=op_bind) at backover.c:723 #17 0x00000000004bbe43 in over_op_bind (op=0x1fe91140, rs=0x48234c30) at backover.c:738 #18 0x0000000000455b40 in do_bind (op=0x1fe91140, rs=0x48234c30) at bind.c:205 #19 0x000000000042dbce in connection_operation (ctx=0x48234d70, arg_v=0x1fe91140) at connection.c:1138 #20 0x000000000042e153 in connection_read_thread (ctx=0x48234d70, argv=0x64) at connection.c:1274 #21 0x00002b1c99d66fc8 in ldap_int_thread_pool_wrapper (xpool=0x1ec9acb0) at tpool.c:685 #22 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #23 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6 (gdb) detach Detaching from program: /opt/openldap-2.4.26/libexec/slapd, process 21203 (gdb) attach 21202 Attaching to program: /opt/openldap-2.4.26/libexec/slapd, process 21202 warning: process 21202 is a cloned process Reading symbols from /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2...done. Loaded symbols for /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2 Reading symbols from /opt/openldap-2.4.26/lib/liblber-2.4.so.2...done. Loaded symbols for /opt/openldap-2.4.26/lib/liblber-2.4.so.2 Reading symbols from /lib64/libuuid.so.1...done. Loaded symbols for /lib64/libuuid.so.1 Reading symbols from /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so...done. Loaded symbols for /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so Reading symbols from /lib64/libpthread.so.0...done. [Thread debugging using libthread_db enabled] [New Thread 0x2b1c9a53dbd0 (LWP 20884)] [New Thread 0x48a36940 (LWP 21499)] [New Thread 0x40930940 (LWP 21498)] [New Thread 0x48235940 (LWP 21203)] [New Thread 0x47a34940 (LWP 21202)] [New Thread 0x47233940 (LWP 21201)] [New Thread 0x46a32940 (LWP 21200)] [New Thread 0x46231940 (LWP 21087)] [New Thread 0x45a30940 (LWP 21086)] [New Thread 0x4522f940 (LWP 21021)] [New Thread 0x44a2e940 (LWP 21020)] [New Thread 0x4422d940 (LWP 20891)] [New Thread 0x43a2c940 (LWP 20890)] [New Thread 0x4322b940 (LWP 20889)] [New Thread 0x42a2a940 (LWP 20888)] [New Thread 0x42229940 (LWP 20887)] [New Thread 0x41a28940 (LWP 20886)] [New Thread 0x41227940 (LWP 20885)] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /usr/lib64/libsasl2.so.2...done. Loaded symbols for /usr/lib64/libsasl2.so.2 Reading symbols from /lib64/libssl.so.6...done. Loaded symbols for /lib64/libssl.so.6 Reading symbols from /lib64/libcrypto.so.6...done. Loaded symbols for /lib64/libcrypto.so.6 Reading symbols from /lib64/libresolv.so.2...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libc.so.6...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libdl.so.2...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libcrypt.so.1...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done. Loaded symbols for /usr/lib64/libgssapi_krb5.so.2 Reading symbols from /usr/lib64/libkrb5.so.3...done. Loaded symbols for /usr/lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /usr/lib64/libk5crypto.so.3...done. Loaded symbols for /usr/lib64/libk5crypto.so.3 Reading symbols from /usr/lib64/libz.so.1...done. Loaded symbols for /usr/lib64/libz.so.1 Reading symbols from /usr/lib64/libkrb5support.so.0...done. Loaded symbols for /usr/lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libselinux.so.1...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libsepol.so.1...done. Loaded symbols for /lib64/libsepol.so.1 Reading symbols from /lib64/libnss_files.so.2...done. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /lib64/libnss_dns.so.2...done. Loaded symbols for /lib64/libnss_dns.so.2 Reading symbols from /usr/lib64/sasl2/liblogin.so.2...done. Loaded symbols for /usr/lib64/sasl2/liblogin.so.2 Reading symbols from /usr/lib64/sasl2/libanonymous.so.2...done. Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2 Reading symbols from /usr/lib64/sasl2/libplain.so.2...done. Loaded symbols for /usr/lib64/sasl2/libplain.so.2 Reading symbols from /usr/lib64/sasl2/libsasldb.so.2...done. Loaded symbols for /usr/lib64/sasl2/libsasldb.so.2 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb) bt #0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b1c99d65f1b in ldap_pvt_thread_rmutex_lock (rmutex=<value optimized out>, owner=1201883456) at rmutex.c:129 #2 0x000000000052efb7 in accesslog_op_mod (op=0x47a32fe0, rs=0x47a33150) at accesslog.c:1889 #3 0x00000000004bbb2b in overlay_op_walk (op=0x47a32fe0, rs=0x47a33150, which=op_modify, oi=0x1ed31f10, on=0x1ed32d60) at backover.c:661 #4 0x00000000004bbdbd in over_op_func (op=0x47a32fe0, rs=0x47a33150, which=op_modify) at backover.c:723 #5 0x00000000004bbed3 in over_op_modify (op=0x47a32fe0, rs=0x47a33150) at backover.c:762 #6 0x000000000053c7af in ppolicy_bind_response (op=0x1fe6e000, rs=0x47a33c30) at ppolicy.c:1173 #7 0x0000000000443a08 in slap_response_play (op=0x1fe6e000, rs=0x47a33c30) at result.c:505 #8 0x0000000000443c2f in send_ldap_response (op=0x1fe6e000, rs=0x47a33c30) at result.c:579 #9 0x0000000000444b14 in slap_send_ldap_result (op=0x1fe6e000, rs=0x47a33c30) at result.c:851 #10 0x000000000050f2f3 in hdb_bind (op=0x1fe6e000, rs=0x47a33c30) at bind.c:158 #11 0x00000000004bbbba in overlay_op_walk (op=0x1fe6e000, rs=0x47a33c30, which=op_bind, oi=0x1ed31f10, on=0x0) at backover.c:671 #12 0x00000000004bbdbd in over_op_func (op=0x1fe6e000, rs=0x47a33c30, which=op_bind) at backover.c:723 #13 0x00000000004bbe43 in over_op_bind (op=0x1fe6e000, rs=0x47a33c30) at backover.c:738 #14 0x0000000000456411 in fe_op_bind (op=0x1fe6e000, rs=0x47a33c30) at bind.c:383 #15 0x00000000004bbbba in overlay_op_walk (op=0x1fe6e000, rs=0x47a33c30, which=op_bind, oi=0x1ed02760, on=0x0) at backover.c:671 #16 0x00000000004bbdbd in over_op_func (op=0x1fe6e000, rs=0x47a33c30, which=op_bind) at backover.c:723 #17 0x00000000004bbe43 in over_op_bind (op=0x1fe6e000, rs=0x47a33c30) at backover.c:738 #18 0x0000000000455b40 in do_bind (op=0x1fe6e000, rs=0x47a33c30) at bind.c:205 #19 0x000000000042dbce in connection_operation (ctx=0x47a33d70, arg_v=0x1fe6e000) at connection.c:1138 #20 0x000000000042e153 in connection_read_thread (ctx=0x47a33d70, argv=0x6a) at connection.c:1274 #21 0x00002b1c99d66fc8 in ldap_int_thread_pool_wrapper (xpool=0x1ec9acb0) at tpool.c:685 #22 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0 #23 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
I can provide you with the other threads backtrace if needed.
Quanah Gibson-Mount a écrit :
--On Thursday, August 04, 2011 2:23 PM +0200 Cyril Grosjean cgrosjean@janua.fr wrote:
I've just tested with OpenLDAP 2.4.26, as you suggested, but the problem remains.
I've no load at all, I use mirror mode replication, and this time, the freeze occurred when trying to modify a user's password: The modification is done from a usual Web interface from a Java application. The Java application is set to connect to OpenLDAP with a unique dedicated account, but in the case of a password change, we use the PROXYAUTH operation to do the modification with the user's rights, so that his pwdreset attribute is removed if previously set to TRUE
I've also opened a case with Red Hat, they suggested me to test under RHEL 6, with their OpenLDAP rpm package (2.4.23-15.el6). So I'll do it right now but if you need more details on the RHEL 5.4 hung slapd process, I can send them since it's still frozen.
Please build with debugging symbols and no optimization (-O0 and -g). Make sure when you install, you set STRIP="" so it doesn't strip the symbols out. Then get a full backtrace.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc.
Zimbra :: the leader in open source messaging and collaboration
--On Monday, August 08, 2011 5:31 PM +0200 Cyril Grosjean cgrosjean@janua.fr wrote:
Here're the traces you required, for the 1st 4 OpenLDAP threads at least:
Please follow:
http://www.openldap.org/faq/data/cache/59.html
Particularly use:
thread apply all bt full
I would at this point advise you to file an ITS, since you can regularly lock up slapd. In addition to the FULL backtrace, provide your full slapd.conf OR cn=config database, minus any passwords.
Thanks.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
I did the full backtrace and logged a new ITS:
http://www.openldap.org/its/index.cgi/Incoming?id=7013;selectid=7013
Please also notice I haven't been able to upload my materials, since there's no space left on the OpenLDAP FTP device.
Quanah Gibson-Mount a écrit :
--On Monday, August 08, 2011 5:31 PM +0200 Cyril Grosjean cgrosjean@janua.fr wrote:
Here're the traces you required, for the 1st 4 OpenLDAP threads at least:
Please follow:
http://www.openldap.org/faq/data/cache/59.html
Particularly use:
thread apply all bt full
I would at this point advise you to file an ITS, since you can regularly lock up slapd. In addition to the FULL backtrace, provide your full slapd.conf OR cn=config database, minus any passwords.
Thanks.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc.
Zimbra :: the leader in open source messaging and collaboration
openldap-technical@openldap.org
-
Cyril Grosjean -
Cyril GROSJEAN
-
Quanah Gibson-Mount