9:28 a.m.
I randomly notice my OpenLDAP server freezes, and I can't udnerstand why.
I have a few LDAP clients (ldapsearch, a legacy Java app. and
ApacheDirectoryStudio), running from different systems,
either locally on the OpenLDAP server, or on another OpenLDAP server, or on
a remote workstation, and none manages
to get an answer from OpenLDAP. The connection is established but each
client gets stuck waiting for any result.
The last time, it occured only a few minutes after the OpenLDAP daemon was
restarted, under very low load.
I use OpenLDAP 2.4.23, over Red Hat Enterprise Linux Server release 5.4
(Tikanga), Kernel version 2.6.18-164.el5, amd64 architecture over VMWare.
I also use BDB 4.8.30, and OpenLDAP's been compiled with the following
options:
./configure --prefix=/opt/openldap-2.4.23 --enable-dynamic=yes
--enable-dynlist=yes --enable-memberof=yes --enable-ppolicy=yes
--enable-accesslog=yes --enable-sssvlv=yes
I noticed only 60 to 70 established connections on port 389 (according to
netstat) when the freeze occurs.
The OpenLDAP server is set in syncrepl mirror mode replication, and the
other OpenLDAP server (only used as a hot stand-by) works fine (but doesn't
receive any load)
In the OpenLDAP accesslog, I usually see the client requests, but no lines
for the responses.
Also, I noticed sometimes the "deferring operation: binding" message :
Jul 12 10:20:05 dev-ldap1 slapd[28525]: connection_input: conn=3377
deferring operation: binding
Jul 12 10:20:07 dev-ldap1 slapd[28525]: conn=3378 fd=66 ACCEPT from
IP=......:3823 (IP=0.0.0.0:389)
I don't know if this message is linked or not to the main problem so I'd
like to know if this message is harmless or not.
The stack of the slapd process is as follows when the freeze occurs:
Thread 18 (Thread 0x40d4b940 (LWP 1893)):
#0 0x00000031a7ad4018 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000041d76e in ldap_pvt_sasl_mutex_dispose ()
#2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x4154c940 (LWP 1894)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a3ad7fb in ldap_pvt_thread_rmutex_lock ()
#2 0x00000000004d4c74 in ldap_pvt_sasl_mutex_dispose ()
#3 0x0000000000481b4a in ldap_pvt_sasl_mutex_dispose ()
#4 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#5 0x00000000004de44d in ldap_pvt_sasl_mutex_dispose ()
#6 0x000000000042ee25 in ldap_pvt_sasl_mutex_dispose ()
#7 0x00000000004319c8 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000432749 in ldap_pvt_sasl_mutex_dispose ()
#9 0x00000000004ae91b in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#11 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#12 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#14 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#15 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#17 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#18 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#19 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#20 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x42771940 (LWP 1895)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a833dd2 in __db_pthread_mutex_lock ()
#2 0x00002b710a833426 in __db_tas_mutex_lock ()
#3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c42c2 in __lock_get_pp ()
#5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose ()
#6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose ()
#7 0x00000000004b4183 in ldap_pvt_sasl_mutex_dispose ()
#8 0x00000000004b802e in ldap_pvt_sasl_mutex_dispose ()
#9 0x0000000000481a7a in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000482547 in ldap_pvt_sasl_mutex_dispose ()
#11 0x00000000004decab in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000481b4a in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#14 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose ()
#15 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#16 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#17 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose ()
#18 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#19 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#20 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#21 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#22 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x42f72940 (LWP 1896)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a833dd2 in __db_pthread_mutex_lock ()
#2 0x00002b710a833426 in __db_tas_mutex_lock ()
#3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c42c2 in __lock_get_pp ()
#5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose ()
#6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose ()
#7 0x0000000000492a80 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#9 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000422736 in ldap_pvt_sasl_mutex_dispose ()
#11 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000422ec7 in ldap_pvt_sasl_mutex_dispose ()
#14 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#15 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x43773940 (LWP 1897)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a833dd2 in __db_pthread_mutex_lock ()
#2 0x00002b710a833426 in __db_tas_mutex_lock ()
#3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c42c2 in __lock_get_pp ()
#5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose ()
#6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose ()
#7 0x0000000000492a80 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#9 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000422736 in ldap_pvt_sasl_mutex_dispose ()
#11 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000422ec7 in ldap_pvt_sasl_mutex_dispose ()
#14 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#15 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x43f74940 (LWP 1898)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a833dd2 in __db_pthread_mutex_lock ()
#2 0x00002b710a833426 in __db_tas_mutex_lock ()
#3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c42c2 in __lock_get_pp ()
#5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose ()
#6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose ()
#7 0x0000000000492a80 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#9 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000422736 in ldap_pvt_sasl_mutex_dispose ()
#11 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000422ec7 in ldap_pvt_sasl_mutex_dispose ()
#14 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#15 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x44775940 (LWP 1902)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a833dd2 in __db_pthread_mutex_lock ()
#2 0x00002b710a833426 in __db_tas_mutex_lock ()
#3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c42c2 in __lock_get_pp ()
#5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose ()
#6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose ()
#7 0x0000000000492a80 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#9 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000422736 in ldap_pvt_sasl_mutex_dispose ()
#11 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000422ec7 in ldap_pvt_sasl_mutex_dispose ()
#14 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#15 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x41d4d940 (LWP 1947)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a833dd2 in __db_pthread_mutex_lock ()
#2 0x00002b710a833426 in __db_tas_mutex_lock ()
#3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c42c2 in __lock_get_pp ()
#5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose ()
#6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose ()
#7 0x0000000000492a80 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#9 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000422736 in ldap_pvt_sasl_mutex_dispose ()
#11 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000422ec7 in ldap_pvt_sasl_mutex_dispose ()
#14 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#15 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x44f76940 (LWP 1948)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a3ad7fb in ldap_pvt_thread_rmutex_lock ()
#2 0x00000000004d4c74 in ldap_pvt_sasl_mutex_dispose ()
#3 0x0000000000481b4a in ldap_pvt_sasl_mutex_dispose ()
#4 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#5 0x00000000004de44d in ldap_pvt_sasl_mutex_dispose ()
#6 0x000000000042ee25 in ldap_pvt_sasl_mutex_dispose ()
#7 0x00000000004319c8 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000432749 in ldap_pvt_sasl_mutex_dispose ()
#9 0x00000000004ae91b in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#11 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#12 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#14 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#15 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#17 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#18 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#19 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#20 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x45777940 (LWP 1949)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a833dd2 in __db_pthread_mutex_lock ()
#2 0x00002b710a833426 in __db_tas_mutex_lock ()
#3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c42c2 in __lock_get_pp ()
#5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose ()
#6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose ()
#7 0x00000000004b4183 in ldap_pvt_sasl_mutex_dispose ()
#8 0x00000000004b802e in ldap_pvt_sasl_mutex_dispose ()
#9 0x0000000000481a7a in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000482547 in ldap_pvt_sasl_mutex_dispose ()
#11 0x00000000004decab in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000481b4a in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#14 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose ()
#15 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#16 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#17 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose ()
#18 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#19 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#20 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#21 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#22 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x45f78940 (LWP 1950)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a3ad7fb in ldap_pvt_thread_rmutex_lock ()
#2 0x00000000004d4c74 in ldap_pvt_sasl_mutex_dispose ()
#3 0x0000000000481b4a in ldap_pvt_sasl_mutex_dispose ()
#4 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#5 0x00000000004de44d in ldap_pvt_sasl_mutex_dispose ()
#6 0x000000000042ee25 in ldap_pvt_sasl_mutex_dispose ()
#7 0x00000000004319c8 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000432749 in ldap_pvt_sasl_mutex_dispose ()
#9 0x00000000004ae91b in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#11 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#12 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#14 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#15 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#17 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#18 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#19 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#20 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x46779940 (LWP 1951)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a3ad7fb in ldap_pvt_thread_rmutex_lock ()
#2 0x00000000004d4c74 in ldap_pvt_sasl_mutex_dispose ()
#3 0x0000000000481b4a in ldap_pvt_sasl_mutex_dispose ()
#4 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#5 0x00000000004de44d in ldap_pvt_sasl_mutex_dispose ()
#6 0x000000000042ee25 in ldap_pvt_sasl_mutex_dispose ()
#7 0x00000000004319c8 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000432749 in ldap_pvt_sasl_mutex_dispose ()
#9 0x00000000004ae91b in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#11 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#12 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#14 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#15 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#17 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#18 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#19 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#20 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x46f7a940 (LWP 1952)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a833dd2 in __db_pthread_mutex_lock ()
#2 0x00002b710a833426 in __db_tas_mutex_lock ()
#3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c42c2 in __lock_get_pp ()
#5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose ()
#6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose ()
#7 0x0000000000492a80 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#9 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000422736 in ldap_pvt_sasl_mutex_dispose ()
#11 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000422ec7 in ldap_pvt_sasl_mutex_dispose ()
#14 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#15 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x4777b940 (LWP 2149)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a833dd2 in __db_pthread_mutex_lock ()
#2 0x00002b710a833426 in __db_tas_mutex_lock ()
#3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c44ea in __lock_vec ()
#5 0x00002b710a8c512b in __lock_vec_pp ()
#6 0x00000000004af514 in ldap_pvt_sasl_mutex_dispose ()
#7 0x00000000004af900 in ldap_pvt_sasl_mutex_dispose ()
#8 0x000000000048d323 in ldap_pvt_sasl_mutex_dispose ()
#9 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#11 0x00000000004de44d in ldap_pvt_sasl_mutex_dispose ()
#12 0x000000000042ee25 in ldap_pvt_sasl_mutex_dispose ()
#13 0x00000000004319c8 in ldap_pvt_sasl_mutex_dispose ()
#14 0x0000000000432749 in ldap_pvt_sasl_mutex_dispose ()
#15 0x00000000004ae91b in ldap_pvt_sasl_mutex_dispose ()
#16 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#17 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#18 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose ()
#19 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#20 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#21 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose ()
#22 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#23 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#24 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#25 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#26 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x47f7c940 (LWP 2150)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a3ad7fb in ldap_pvt_thread_rmutex_lock ()
#2 0x00000000004d4c74 in ldap_pvt_sasl_mutex_dispose ()
#3 0x0000000000481b4a in ldap_pvt_sasl_mutex_dispose ()
#4 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#5 0x00000000004de44d in ldap_pvt_sasl_mutex_dispose ()
#6 0x000000000042ee25 in ldap_pvt_sasl_mutex_dispose ()
#7 0x00000000004319c8 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000432749 in ldap_pvt_sasl_mutex_dispose ()
#9 0x00000000004ae91b in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#11 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#12 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#14 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#15 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#17 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#18 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#19 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#20 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x4877d940 (LWP 2151)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a833dd2 in __db_pthread_mutex_lock ()
#2 0x00002b710a833426 in __db_tas_mutex_lock ()
#3 0x00002b710a8c3b49 in __lock_get_internal ()
#4 0x00002b710a8c42c2 in __lock_get_pp ()
#5 0x00000000004af398 in ldap_pvt_sasl_mutex_dispose ()
#6 0x00000000004b04ca in ldap_pvt_sasl_mutex_dispose ()
#7 0x0000000000492a80 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#9 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000422736 in ldap_pvt_sasl_mutex_dispose ()
#11 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000422ec7 in ldap_pvt_sasl_mutex_dispose ()
#14 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#15 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x48f7e940 (LWP 2152)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002b710a3ad7fb in ldap_pvt_thread_rmutex_lock ()
#2 0x00000000004d4c74 in ldap_pvt_sasl_mutex_dispose ()
#3 0x0000000000481b4a in ldap_pvt_sasl_mutex_dispose ()
#4 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#5 0x00000000004de44d in ldap_pvt_sasl_mutex_dispose ()
#6 0x000000000042ee25 in ldap_pvt_sasl_mutex_dispose ()
#7 0x00000000004319c8 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000432749 in ldap_pvt_sasl_mutex_dispose ()
#9 0x00000000004ae91b in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#11 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#12 0x000000000043be3e in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000481bc2 in ldap_pvt_sasl_mutex_dispose ()
#14 0x0000000000482127 in ldap_pvt_sasl_mutex_dispose ()
#15 0x000000000043c83f in ldap_pvt_sasl_mutex_dispose ()
#16 0x00000000004205a5 in ldap_pvt_sasl_mutex_dispose ()
#17 0x0000000000420b7f in ldap_pvt_sasl_mutex_dispose ()
#18 0x00002b710a3ae8a8 in ldap_int_thread_pool_wrapper ()
#19 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#20 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2b710ab80bd0 (LWP 1892)):
#0 0x00000031a86077e5 in pthread_join () from /lib64/libpthread.so.0
#1 0x000000000041b3e2 in ldap_pvt_sasl_mutex_dispose ()
#2 0x000000000040960e in ldap_pvt_sasl_mutex_dispose ()
#3 0x00000031a7a1d994 in __libc_start_main () from /lib64/libc.so.6
#4 0x00000000004080d9 in ldap_pvt_sasl_mutex_dispose ()
#5 0x00007fff4c26b148 in ?? ()
#6 0x0000000000000000 in ?? ()
Any idea of what's wrong ? Known bug ?
11:07 a.m.
--On Tuesday, August 02, 2011 6:28 PM +0200 Cyril GROSJEAN
<cgrosjean(a)janua.fr> wrote:
Any idea of what's wrong ? Known bug ?
Can you try with 2.4.26? There was an issue with 2.4.23 that may be the
culprit.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
10:13 p.m.
OK, I'll try to see if version 2.4.26 is the cure ...
Quanah Gibson-Mount a écrit :
--On Tuesday, August 02, 2011 6:28 PM +0200 Cyril GROSJEAN
<cgrosjean(a)janua.fr> wrote:
> Any idea of what's wrong ? Known bug ?
Can you try with 2.4.26? There was an issue with 2.4.23 that may be
the culprit.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
8:29 a.m.
I've just also tested with the standard OpenLDAP rpm packages on RHEL 6,
but the problem remains. Those packages are:
openldap-2.4.23-12.el6.i686
openldap-servers-2.4.23-12.el6.i686
Since the slapd process still doesn't respond, what traces do you need
to investigate ?
5:23 a.m.
I've just tested with OpenLDAP 2.4.26, as you suggested, but the problem
remains.
I've no load at all, I use mirror mode replication, and this time, the
freeze occured when trying to modify a user's password:
The modification is done from a usual Web interface from a Java
application. The Java application is set to connect to OpenLDAP
with a unique dedicated account, but in the case of a password change,
we use the PROXYAUTH operation to do the modification
with the user's rights, so that his pwdreset attribute is removed if
previously set to TRUE
Below is the connection that brings the freeze. We never get an answer
from operation 21:
Aug 4 13:43:28 rhvtq slapd-master1[23372]: conn=1201 fd=139 ACCEPT from
IP=192.168.1.132:57374 (IP=192.168.1.132:1389)
Aug 4 13:43:28 rhvtq slapd-master1[23372]: conn=1201 op=0 BIND
dn="cn=yellow pages,ou=special users,dc=company" method=128
Aug 4 13:43:28 rhvtq slapd-master1[23372]: conn=1201 op=0 BIND
dn="cn=yellow pages,ou=special users,dc=company" mech=SIMPLE ssf=0
Aug 4 13:43:28 rhvtq slapd-master1[23372]: conn=1201 op=0 RESULT tag=97
err=0 text=
...
...
...
Aug 4 13:46:40 rhvtq slapd-master1[23372]: conn=1201 op=20 SEARCH
RESULT tag=101 err=0 nentries=1 text=
Aug 4 13:46:40 rhvtq slapd-master1[23372]: conn=1201 op=21 PROXYAUTHZ
dn="employeeNumber=00296,ou=people,dc=company"
Aug 4 13:46:40 rhvtq slapd-master1[23372]: conn=1201 op=21 MOD
dn="employeeNumber=00296,ou=people,dc=company"
Aug 4 13:46:40 rhvtq slapd-master1[23372]: conn=1201 op=21 MOD
attr=userPassword
And here's the slapd pstack output :
Thread 18 (Thread 0x42018940 (LWP 23373)):
#0 0x00000031a7ad4018 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000041eb0b in ldap_pvt_sasl_mutex_dispose ()
#2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x42819940 (LWP 23374)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper ()
#2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x4301a940 (LWP 23375)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper ()
#2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x4381b940 (LWP 23376)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper ()
#2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x4401c940 (LWP 23377)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper ()
#2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x4481d940 (LWP 23381)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002afb09199e8b in ldap_pvt_thread_rmutex_lock ()
#2 0x00000000004d7fcf in ?? ()
#3 0x000000000048402a in ldap_pvt_sasl_mutex_dispose ()
#4 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose ()
#5 0x0000000000477c2f in ldap_pvt_sasl_mutex_dispose ()
#6 0x000000000047f2eb in ldap_pvt_sasl_mutex_dispose ()
#7 0x0000000000422713 in ldap_pvt_sasl_mutex_dispose ()
#8 0x00002afb0919af38 in ldap_int_thread_pool_wrapper ()
#9 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#10 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x4501e940 (LWP 23382)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002afb09622dd2 in __db_pthread_mutex_lock ()
#2 0x00002afb09622426 in __db_tas_mutex_lock ()
#3 0x00002afb096b2b49 in __lock_get_internal ()
#4 0x00002afb096b32c2 in __lock_get_pp ()
#5 0x00000000004b2c28 in ?? ()
#6 0x00000000004b3ab9 in ?? ()
#7 0x00000000004955ca in ldap_pvt_sasl_mutex_dispose ()
#8 0x00000000004840a2 in ldap_pvt_sasl_mutex_dispose ()
#9 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000423a49 in ldap_pvt_sasl_mutex_dispose ()
#11 0x00000000004840a2 in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000424255 in ldap_pvt_sasl_mutex_dispose ()
#14 0x00000000004217e5 in ldap_pvt_sasl_mutex_dispose ()
#15 0x0000000000421dbf in ldap_pvt_sasl_mutex_dispose ()
#16 0x00002afb0919af38 in ldap_int_thread_pool_wrapper ()
#17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x4581f940 (LWP 25216)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper ()
#2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x46020940 (LWP 25217)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002afb09622dd2 in __db_pthread_mutex_lock ()
#2 0x00002afb09622426 in __db_tas_mutex_lock ()
#3 0x00002afb096b2b49 in __lock_get_internal ()
#4 0x00002afb096b32c2 in __lock_get_pp ()
#5 0x00000000004b2c28 in ?? ()
#6 0x00000000004b3ab9 in ?? ()
#7 0x00000000004b75e3 in ?? ()
#8 0x00000000004bb3ce in ?? ()
#9 0x0000000000483f5a in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000484a67 in ldap_pvt_sasl_mutex_dispose ()
#11 0x00000000004e226b in ?? ()
#12 0x000000000048402a in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose ()
#14 0x000000000043d21e in ldap_pvt_sasl_mutex_dispose ()
#15 0x00000000004840a2 in ldap_pvt_sasl_mutex_dispose ()
#16 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose ()
#17 0x000000000043dc1f in ldap_pvt_sasl_mutex_dispose ()
#18 0x00000000004217e5 in ldap_pvt_sasl_mutex_dispose ()
#19 0x0000000000421dbf in ldap_pvt_sasl_mutex_dispose ()
#20 0x00002afb0919af38 in ldap_int_thread_pool_wrapper ()
#21 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#22 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x46821940 (LWP 25218)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002afb09622dd2 in __db_pthread_mutex_lock ()
#2 0x00002afb09622426 in __db_tas_mutex_lock ()
#3 0x00002afb096b2b49 in __lock_get_internal ()
#4 0x00002afb096b34ea in __lock_vec ()
#5 0x00002afb096b412b in __lock_vec_pp ()
#6 0x00000000004b3264 in ?? ()
#7 0x00000000004b3650 in ?? ()
#8 0x000000000048fe13 in ldap_pvt_sasl_mutex_dispose ()
#9 0x00000000004840a2 in ldap_pvt_sasl_mutex_dispose ()
#10 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose ()
#11 0x0000000000438847 in ldap_pvt_sasl_mutex_dispose ()
#12 0x00000000004840a2 in ldap_pvt_sasl_mutex_dispose ()
#13 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose ()
#14 0x0000000000438fb2 in ldap_pvt_sasl_mutex_dispose ()
#15 0x00000000004217e5 in ldap_pvt_sasl_mutex_dispose ()
#16 0x0000000000421dbf in ldap_pvt_sasl_mutex_dispose ()
#17 0x00002afb0919af38 in ldap_int_thread_pool_wrapper ()
#18 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#19 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x47022940 (LWP 25219)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper ()
#2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x47823940 (LWP 25220)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper ()
#2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x48024940 (LWP 25221)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper ()
#2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x48825940 (LWP 25222)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper ()
#2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x49026940 (LWP 25223)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper ()
#2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x49827940 (LWP 25224)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002afb0919af8d in ldap_int_thread_pool_wrapper ()
#2 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x4a028940 (LWP 25225)):
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00002afb09199e8b in ldap_pvt_thread_rmutex_lock ()
#2 0x00000000004d7fcf in ?? ()
#3 0x000000000048402a in ldap_pvt_sasl_mutex_dispose ()
#4 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose ()
#5 0x00000000004e1a08 in ?? ()
#6 0x0000000000430285 in ldap_pvt_sasl_mutex_dispose ()
#7 0x0000000000430cc8 in ldap_pvt_sasl_mutex_dispose ()
#8 0x0000000000431a49 in ldap_pvt_sasl_mutex_dispose ()
#9 0x000000000043ce10 in ldap_pvt_sasl_mutex_dispose ()
#10 0x000000000043d4b4 in ldap_pvt_sasl_mutex_dispose ()
#11 0x00000000004840a2 in ldap_pvt_sasl_mutex_dispose ()
#12 0x0000000000484607 in ldap_pvt_sasl_mutex_dispose ()
#13 0x000000000043dc1f in ldap_pvt_sasl_mutex_dispose ()
#14 0x00000000004217e5 in ldap_pvt_sasl_mutex_dispose ()
#15 0x0000000000421dbf in ldap_pvt_sasl_mutex_dispose ()
#16 0x00002afb0919af38 in ldap_int_thread_pool_wrapper ()
#17 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#18 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2afb0996fbd0 (LWP 23372)):
#0 0x00000031a86077e5 in pthread_join () from /lib64/libpthread.so.0
#1 0x000000000041c16d in ldap_pvt_sasl_mutex_dispose ()
#2 0x000000000040997b in ldap_pvt_sasl_mutex_dispose ()
#3 0x00000031a7a1d994 in __libc_start_main () from /lib64/libc.so.6
#4 0x0000000000408389 in ldap_pvt_sasl_mutex_dispose ()
#5 0x00007fffd4f51358 in ?? ()
#6 0x0000000000000000 in ?? ()
I've also opened a case with Red Hat, they suggested me to test under
RHEL 6, with their OpenLDAP rpm package (2.4.23-15.el6).
So I'll do it right now but if you need more details on the RHEL 5.4
hung slapd process, I can send them since it's still frozen.
Quanah Gibson-Mount a écrit :
--On Tuesday, August 02, 2011 6:28 PM +0200 Cyril GROSJEAN
<cgrosjean(a)janua.fr> wrote:
> Any idea of what's wrong ? Known bug ?
Can you try with 2.4.26? There was an issue with 2.4.23 that may be
the culprit.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
--
10:32 a.m.
--On Thursday, August 04, 2011 2:23 PM +0200 Cyril Grosjean
<cgrosjean(a)janua.fr> wrote:
I've just tested with OpenLDAP 2.4.26, as you suggested, but the problem
remains.
I've no load at all, I use mirror mode replication, and this time, the
freeze occurred when trying to modify a user's password:
The modification is done from a usual Web interface from a Java
application. The Java application is set to connect to OpenLDAP
with a unique dedicated account, but in the case of a password change, we
use the PROXYAUTH operation to do the modification
with the user's rights, so that his pwdreset attribute is removed if
previously set to TRUE
I've also opened a case with Red Hat, they suggested me to test under
RHEL 6, with their OpenLDAP rpm package (2.4.23-15.el6).
So I'll do it right now but if you need more details on the RHEL 5.4 hung
slapd process, I can send them since it's still frozen.
Please build with debugging symbols and no optimization (-O0 and -g). Make
sure when you install, you set STRIP="" so it doesn't strip the symbols
out. Then get a full backtrace.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
12:25 a.m.
Before building OpenLDAP with debugging symbols and no optimization, we
noticed some possible locks (with db_stat) on the bdb (4.8.30) back-ends,
that we can reproduce with the main Java application that interacts with
OpenLDAP. It uses to occur when we create a few users.
So, it could possibly be a problem with the way our Java application
works with OpenLDAP, even if once OpenLDAP is frozen, other usual LDAP
clients
such a ApacheDirectoryStudio or ldapsearch can't connect (bind request
sent but no response) to OpenLDAP. Also, the same main Java application
didn't
use to freeze our previous (Sun DS 6) directory.
So, we now first intend to replace bdb and test with hdb or possibly
some other database back-end, before testing with debugging symbols and
no optimization.
I'll let you know the results of of our tests ..
2:34 a.m.
Our tests with hdb haven't improve the situation. Even after a few binds
from our Java application or thanks to ldapsearch,
OpenLDAP doesn't respond. We did those tests on RHEL 5.4 + OpenLDAP 2.4.26
+ Berkeley DB 4.8.30.
We also got the freeze with BDB 5.1.25, but didn't test with BDB 5.2.28 ..
So, we'll now build with debugging symbols and no optimization and follow
your other recommandations ..
8:31 a.m.
Here're the traces you required, for the 1st 4 OpenLDAP threads at least:
(gdb) attach 20884
Attaching to program: /opt/openldap-2.4.26/libexec/slapd, process 20884
Reading symbols from /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2...done.
Loaded symbols for /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2
Reading symbols from /opt/openldap-2.4.26/lib/liblber-2.4.so.2...done.
Loaded symbols for /opt/openldap-2.4.26/lib/liblber-2.4.so.2
Reading symbols from /lib64/libuuid.so.1...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so...done.
Loaded symbols for /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so
Reading symbols from /lib64/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x2b1c9a53dbd0 (LWP 20884)]
[New Thread 0x48a36940 (LWP 21499)]
[New Thread 0x40930940 (LWP 21498)]
[New Thread 0x48235940 (LWP 21203)]
[New Thread 0x47a34940 (LWP 21202)]
[New Thread 0x47233940 (LWP 21201)]
[New Thread 0x46a32940 (LWP 21200)]
[New Thread 0x46231940 (LWP 21087)]
[New Thread 0x45a30940 (LWP 21086)]
[New Thread 0x4522f940 (LWP 21021)]
[New Thread 0x44a2e940 (LWP 21020)]
[New Thread 0x4422d940 (LWP 20891)]
[New Thread 0x43a2c940 (LWP 20890)]
[New Thread 0x4322b940 (LWP 20889)]
[New Thread 0x42a2a940 (LWP 20888)]
[New Thread 0x42229940 (LWP 20887)]
[New Thread 0x41a28940 (LWP 20886)]
[New Thread 0x41227940 (LWP 20885)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /usr/lib64/libsasl2.so.2...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libssl.so.6...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libz.so.1...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /usr/lib64/libkrb5support.so.0...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libselinux.so.1...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libsepol.so.1...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /lib64/libnss_files.so.2...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /lib64/libnss_dns.so.2...done.
Loaded symbols for /lib64/libnss_dns.so.2
Reading symbols from /usr/lib64/sasl2/liblogin.so.2...done.
Loaded symbols for /usr/lib64/sasl2/liblogin.so.2
Reading symbols from /usr/lib64/sasl2/libanonymous.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2
Reading symbols from /usr/lib64/sasl2/libplain.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libplain.so.2
Reading symbols from /usr/lib64/sasl2/libsasldb.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libsasldb.so.2
0x00000031a86077e5 in pthread_join () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00000031a86077e5 in pthread_join () from /lib64/libpthread.so.0
#1 0x000000000042aca4 in slapd_daemon () at daemon.c:2922
#2 0x0000000000409f34 in main (argc=7, argv=0x7ffff62c9618) at main.c:983
(gdb) detach
Detaching from program: /opt/openldap-2.4.26/libexec/slapd, process 20884
(gdb) attach 21499
Attaching to program: /opt/openldap-2.4.26/libexec/slapd, process 21499
warning: process 21499 is a cloned process
Reading symbols from /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2...done.
Loaded symbols for /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2
Reading symbols from /opt/openldap-2.4.26/lib/liblber-2.4.so.2...done.
Loaded symbols for /opt/openldap-2.4.26/lib/liblber-2.4.so.2
Reading symbols from /lib64/libuuid.so.1...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so...done.
Loaded symbols for /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so
Reading symbols from /lib64/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x2b1c9a53dbd0 (LWP 20884)]
[New Thread 0x48a36940 (LWP 21499)]
[New Thread 0x40930940 (LWP 21498)]
[New Thread 0x48235940 (LWP 21203)]
[New Thread 0x47a34940 (LWP 21202)]
[New Thread 0x47233940 (LWP 21201)]
[New Thread 0x46a32940 (LWP 21200)]
[New Thread 0x46231940 (LWP 21087)]
[New Thread 0x45a30940 (LWP 21086)]
[New Thread 0x4522f940 (LWP 21021)]
[New Thread 0x44a2e940 (LWP 21020)]
[New Thread 0x4422d940 (LWP 20891)]
[New Thread 0x43a2c940 (LWP 20890)]
[New Thread 0x4322b940 (LWP 20889)]
[New Thread 0x42a2a940 (LWP 20888)]
[New Thread 0x42229940 (LWP 20887)]
[New Thread 0x41a28940 (LWP 20886)]
[New Thread 0x41227940 (LWP 20885)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /usr/lib64/libsasl2.so.2...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libssl.so.6...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libz.so.1...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /usr/lib64/libkrb5support.so.0...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libselinux.so.1...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libsepol.so.1...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /lib64/libnss_files.so.2...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /lib64/libnss_dns.so.2...done.
Loaded symbols for /lib64/libnss_dns.so.2
Reading symbols from /usr/lib64/sasl2/liblogin.so.2...done.
Loaded symbols for /usr/lib64/sasl2/liblogin.so.2
Reading symbols from /usr/lib64/sasl2/libanonymous.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2
Reading symbols from /usr/lib64/sasl2/libplain.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libplain.so.2
Reading symbols from /usr/lib64/sasl2/libsasldb.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libsasldb.so.2
0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
(gdb) bt
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002b1c99d65f1b in ldap_pvt_thread_rmutex_lock (rmutex=<value
optimized out>, owner=1218668864) at rmutex.c:129
#2 0x000000000052efb7 in accesslog_op_mod (op=0x48a34fe0,
rs=0x48a35150) at accesslog.c:1889
#3 0x00000000004bbb2b in overlay_op_walk (op=0x48a34fe0, rs=0x48a35150,
which=op_modify, oi=0x1ed31f10, on=0x1ed32d60) at backover.c:661
#4 0x00000000004bbdbd in over_op_func (op=0x48a34fe0, rs=0x48a35150,
which=op_modify) at backover.c:723
#5 0x00000000004bbed3 in over_op_modify (op=0x48a34fe0, rs=0x48a35150)
at backover.c:762
#6 0x000000000053c7af in ppolicy_bind_response (op=0x1fbf8d90,
rs=0x48a35c30) at ppolicy.c:1173
#7 0x0000000000443a08 in slap_response_play (op=0x1fbf8d90,
rs=0x48a35c30) at result.c:505
#8 0x0000000000443c2f in send_ldap_response (op=0x1fbf8d90,
rs=0x48a35c30) at result.c:579
#9 0x0000000000444b14 in slap_send_ldap_result (op=0x1fbf8d90,
rs=0x48a35c30) at result.c:851
#10 0x000000000050f2f3 in hdb_bind (op=0x1fbf8d90, rs=0x48a35c30) at
bind.c:158
#11 0x00000000004bbbba in overlay_op_walk (op=0x1fbf8d90, rs=0x48a35c30,
which=op_bind, oi=0x1ed31f10, on=0x0) at backover.c:671
#12 0x00000000004bbdbd in over_op_func (op=0x1fbf8d90, rs=0x48a35c30,
which=op_bind) at backover.c:723
#13 0x00000000004bbe43 in over_op_bind (op=0x1fbf8d90, rs=0x48a35c30) at
backover.c:738
#14 0x0000000000456411 in fe_op_bind (op=0x1fbf8d90, rs=0x48a35c30) at
bind.c:383
#15 0x00000000004bbbba in overlay_op_walk (op=0x1fbf8d90, rs=0x48a35c30,
which=op_bind, oi=0x1ed02760, on=0x0) at backover.c:671
#16 0x00000000004bbdbd in over_op_func (op=0x1fbf8d90, rs=0x48a35c30,
which=op_bind) at backover.c:723
#17 0x00000000004bbe43 in over_op_bind (op=0x1fbf8d90, rs=0x48a35c30) at
backover.c:738
#18 0x0000000000455b40 in do_bind (op=0x1fbf8d90, rs=0x48a35c30) at
bind.c:205
#19 0x000000000042dbce in connection_operation (ctx=0x48a35d70,
arg_v=0x1fbf8d90) at connection.c:1138
#20 0x000000000042e153 in connection_read_thread (ctx=0x48a35d70,
argv=0x30) at connection.c:1274
#21 0x00002b1c99d66fc8 in ldap_int_thread_pool_wrapper
(xpool=0x1ec9acb0) at tpool.c:685
#22 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#23 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
(gdb) detach
Detaching from program: /opt/openldap-2.4.26/libexec/slapd, process 21499
(gdb) attach 21498
Attaching to program: /opt/openldap-2.4.26/libexec/slapd, process 21498
warning: process 21498 is a cloned process
Reading symbols from /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2...done.
Loaded symbols for /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2
Reading symbols from /opt/openldap-2.4.26/lib/liblber-2.4.so.2...done.
Loaded symbols for /opt/openldap-2.4.26/lib/liblber-2.4.so.2
Reading symbols from /lib64/libuuid.so.1...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so...done.
Loaded symbols for /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so
Reading symbols from /lib64/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x2b1c9a53dbd0 (LWP 20884)]
[New Thread 0x48a36940 (LWP 21499)]
[New Thread 0x40930940 (LWP 21498)]
[New Thread 0x48235940 (LWP 21203)]
[New Thread 0x47a34940 (LWP 21202)]
[New Thread 0x47233940 (LWP 21201)]
[New Thread 0x46a32940 (LWP 21200)]
[New Thread 0x46231940 (LWP 21087)]
[New Thread 0x45a30940 (LWP 21086)]
[New Thread 0x4522f940 (LWP 21021)]
[New Thread 0x44a2e940 (LWP 21020)]
[New Thread 0x4422d940 (LWP 20891)]
[New Thread 0x43a2c940 (LWP 20890)]
[New Thread 0x4322b940 (LWP 20889)]
[New Thread 0x42a2a940 (LWP 20888)]
[New Thread 0x42229940 (LWP 20887)]
[New Thread 0x41a28940 (LWP 20886)]
[New Thread 0x41227940 (LWP 20885)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /usr/lib64/libsasl2.so.2...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libssl.so.6...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libz.so.1...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /usr/lib64/libkrb5support.so.0...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libselinux.so.1...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libsepol.so.1...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /lib64/libnss_files.so.2...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /lib64/libnss_dns.so.2...done.
Loaded symbols for /lib64/libnss_dns.so.2
Reading symbols from /usr/lib64/sasl2/liblogin.so.2...done.
Loaded symbols for /usr/lib64/sasl2/liblogin.so.2
Reading symbols from /usr/lib64/sasl2/libanonymous.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2
Reading symbols from /usr/lib64/sasl2/libplain.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libplain.so.2
Reading symbols from /usr/lib64/sasl2/libsasldb.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libsasldb.so.2
0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
(gdb) bt
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002b1c99d65f1b in ldap_pvt_thread_rmutex_lock (rmutex=<value
optimized out>, owner=1083377984) at rmutex.c:129
#2 0x000000000052efb7 in accesslog_op_mod (op=0x4092efe0,
rs=0x4092f150) at accesslog.c:1889
#3 0x00000000004bbb2b in overlay_op_walk (op=0x4092efe0, rs=0x4092f150,
which=op_modify, oi=0x1ed31f10, on=0x1ed32d60) at backover.c:661
#4 0x00000000004bbdbd in over_op_func (op=0x4092efe0, rs=0x4092f150,
which=op_modify) at backover.c:723
#5 0x00000000004bbed3 in over_op_modify (op=0x4092efe0, rs=0x4092f150)
at backover.c:762
#6 0x000000000053c7af in ppolicy_bind_response (op=0x1f5bf360,
rs=0x4092fc30) at ppolicy.c:1173
#7 0x0000000000443a08 in slap_response_play (op=0x1f5bf360,
rs=0x4092fc30) at result.c:505
#8 0x0000000000443c2f in send_ldap_response (op=0x1f5bf360,
rs=0x4092fc30) at result.c:579
#9 0x0000000000444b14 in slap_send_ldap_result (op=0x1f5bf360,
rs=0x4092fc30) at result.c:851
#10 0x000000000050f2f3 in hdb_bind (op=0x1f5bf360, rs=0x4092fc30) at
bind.c:158
#11 0x00000000004bbbba in overlay_op_walk (op=0x1f5bf360, rs=0x4092fc30,
which=op_bind, oi=0x1ed31f10, on=0x0) at backover.c:671
#12 0x00000000004bbdbd in over_op_func (op=0x1f5bf360, rs=0x4092fc30,
which=op_bind) at backover.c:723
#13 0x00000000004bbe43 in over_op_bind (op=0x1f5bf360, rs=0x4092fc30) at
backover.c:738
#14 0x0000000000456411 in fe_op_bind (op=0x1f5bf360, rs=0x4092fc30) at
bind.c:383
#15 0x00000000004bbbba in overlay_op_walk (op=0x1f5bf360, rs=0x4092fc30,
which=op_bind, oi=0x1ed02760, on=0x0) at backover.c:671
#16 0x00000000004bbdbd in over_op_func (op=0x1f5bf360, rs=0x4092fc30,
which=op_bind) at backover.c:723
#17 0x00000000004bbe43 in over_op_bind (op=0x1f5bf360, rs=0x4092fc30) at
backover.c:738
#18 0x0000000000455b40 in do_bind (op=0x1f5bf360, rs=0x4092fc30) at
bind.c:205
#19 0x000000000042dbce in connection_operation (ctx=0x4092fd70,
arg_v=0x1f5bf360) at connection.c:1138
#20 0x000000000042e153 in connection_read_thread (ctx=0x4092fd70,
argv=0x25) at connection.c:1274
#21 0x00002b1c99d66fc8 in ldap_int_thread_pool_wrapper
(xpool=0x1ec9acb0) at tpool.c:685
#22 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#23 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
(gdb) detach
Detaching from program: /opt/openldap-2.4.26/libexec/slapd, process 21498
(gdb) attach 21203
Attaching to program: /opt/openldap-2.4.26/libexec/slapd, process 21203
warning: process 21203 is a cloned process
Reading symbols from /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2...done.
Loaded symbols for /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2
Reading symbols from /opt/openldap-2.4.26/lib/liblber-2.4.so.2...done.
Loaded symbols for /opt/openldap-2.4.26/lib/liblber-2.4.so.2
Reading symbols from /lib64/libuuid.so.1...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so...done.
Loaded symbols for /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so
Reading symbols from /lib64/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x2b1c9a53dbd0 (LWP 20884)]
[New Thread 0x48a36940 (LWP 21499)]
[New Thread 0x40930940 (LWP 21498)]
[New Thread 0x48235940 (LWP 21203)]
[New Thread 0x47a34940 (LWP 21202)]
[New Thread 0x47233940 (LWP 21201)]
[New Thread 0x46a32940 (LWP 21200)]
[New Thread 0x46231940 (LWP 21087)]
[New Thread 0x45a30940 (LWP 21086)]
[New Thread 0x4522f940 (LWP 21021)]
[New Thread 0x44a2e940 (LWP 21020)]
[New Thread 0x4422d940 (LWP 20891)]
[New Thread 0x43a2c940 (LWP 20890)]
[New Thread 0x4322b940 (LWP 20889)]
[New Thread 0x42a2a940 (LWP 20888)]
[New Thread 0x42229940 (LWP 20887)]
[New Thread 0x41a28940 (LWP 20886)]
[New Thread 0x41227940 (LWP 20885)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /usr/lib64/libsasl2.so.2...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libssl.so.6...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libz.so.1...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /usr/lib64/libkrb5support.so.0...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libselinux.so.1...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libsepol.so.1...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /lib64/libnss_files.so.2...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /lib64/libnss_dns.so.2...done.
Loaded symbols for /lib64/libnss_dns.so.2
Reading symbols from /usr/lib64/sasl2/liblogin.so.2...done.
Loaded symbols for /usr/lib64/sasl2/liblogin.so.2
Reading symbols from /usr/lib64/sasl2/libanonymous.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2
Reading symbols from /usr/lib64/sasl2/libplain.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libplain.so.2
Reading symbols from /usr/lib64/sasl2/libsasldb.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libsasldb.so.2
0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
(gdb) bt
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002b1c99d65f1b in ldap_pvt_thread_rmutex_lock (rmutex=<value
optimized out>, owner=1210276160) at rmutex.c:129
#2 0x000000000052efb7 in accesslog_op_mod (op=0x48233fe0,
rs=0x48234150) at accesslog.c:1889
#3 0x00000000004bbb2b in overlay_op_walk (op=0x48233fe0, rs=0x48234150,
which=op_modify, oi=0x1ed31f10, on=0x1ed32d60) at backover.c:661
#4 0x00000000004bbdbd in over_op_func (op=0x48233fe0, rs=0x48234150,
which=op_modify) at backover.c:723
#5 0x00000000004bbed3 in over_op_modify (op=0x48233fe0, rs=0x48234150)
at backover.c:762
#6 0x000000000053c7af in ppolicy_bind_response (op=0x1fe91140,
rs=0x48234c30) at ppolicy.c:1173
#7 0x0000000000443a08 in slap_response_play (op=0x1fe91140,
rs=0x48234c30) at result.c:505
#8 0x0000000000443c2f in send_ldap_response (op=0x1fe91140,
rs=0x48234c30) at result.c:579
#9 0x0000000000444b14 in slap_send_ldap_result (op=0x1fe91140,
rs=0x48234c30) at result.c:851
#10 0x000000000050f2f3 in hdb_bind (op=0x1fe91140, rs=0x48234c30) at
bind.c:158
#11 0x00000000004bbbba in overlay_op_walk (op=0x1fe91140, rs=0x48234c30,
which=op_bind, oi=0x1ed31f10, on=0x0) at backover.c:671
#12 0x00000000004bbdbd in over_op_func (op=0x1fe91140, rs=0x48234c30,
which=op_bind) at backover.c:723
#13 0x00000000004bbe43 in over_op_bind (op=0x1fe91140, rs=0x48234c30) at
backover.c:738
#14 0x0000000000456411 in fe_op_bind (op=0x1fe91140, rs=0x48234c30) at
bind.c:383
#15 0x00000000004bbbba in overlay_op_walk (op=0x1fe91140, rs=0x48234c30,
which=op_bind, oi=0x1ed02760, on=0x0) at backover.c:671
#16 0x00000000004bbdbd in over_op_func (op=0x1fe91140, rs=0x48234c30,
which=op_bind) at backover.c:723
#17 0x00000000004bbe43 in over_op_bind (op=0x1fe91140, rs=0x48234c30) at
backover.c:738
#18 0x0000000000455b40 in do_bind (op=0x1fe91140, rs=0x48234c30) at
bind.c:205
#19 0x000000000042dbce in connection_operation (ctx=0x48234d70,
arg_v=0x1fe91140) at connection.c:1138
#20 0x000000000042e153 in connection_read_thread (ctx=0x48234d70,
argv=0x64) at connection.c:1274
#21 0x00002b1c99d66fc8 in ldap_int_thread_pool_wrapper
(xpool=0x1ec9acb0) at tpool.c:685
#22 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#23 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
(gdb) detach
Detaching from program: /opt/openldap-2.4.26/libexec/slapd, process 21203
(gdb) attach 21202
Attaching to program: /opt/openldap-2.4.26/libexec/slapd, process 21202
warning: process 21202 is a cloned process
Reading symbols from /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2...done.
Loaded symbols for /opt/openldap-2.4.26/lib/libldap_r-2.4.so.2
Reading symbols from /opt/openldap-2.4.26/lib/liblber-2.4.so.2...done.
Loaded symbols for /opt/openldap-2.4.26/lib/liblber-2.4.so.2
Reading symbols from /lib64/libuuid.so.1...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so...done.
Loaded symbols for /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so
Reading symbols from /lib64/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x2b1c9a53dbd0 (LWP 20884)]
[New Thread 0x48a36940 (LWP 21499)]
[New Thread 0x40930940 (LWP 21498)]
[New Thread 0x48235940 (LWP 21203)]
[New Thread 0x47a34940 (LWP 21202)]
[New Thread 0x47233940 (LWP 21201)]
[New Thread 0x46a32940 (LWP 21200)]
[New Thread 0x46231940 (LWP 21087)]
[New Thread 0x45a30940 (LWP 21086)]
[New Thread 0x4522f940 (LWP 21021)]
[New Thread 0x44a2e940 (LWP 21020)]
[New Thread 0x4422d940 (LWP 20891)]
[New Thread 0x43a2c940 (LWP 20890)]
[New Thread 0x4322b940 (LWP 20889)]
[New Thread 0x42a2a940 (LWP 20888)]
[New Thread 0x42229940 (LWP 20887)]
[New Thread 0x41a28940 (LWP 20886)]
[New Thread 0x41227940 (LWP 20885)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /usr/lib64/libsasl2.so.2...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libssl.so.6...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libz.so.1...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /usr/lib64/libkrb5support.so.0...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libselinux.so.1...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libsepol.so.1...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /lib64/libnss_files.so.2...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /lib64/libnss_dns.so.2...done.
Loaded symbols for /lib64/libnss_dns.so.2
Reading symbols from /usr/lib64/sasl2/liblogin.so.2...done.
Loaded symbols for /usr/lib64/sasl2/liblogin.so.2
Reading symbols from /usr/lib64/sasl2/libanonymous.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libanonymous.so.2
Reading symbols from /usr/lib64/sasl2/libplain.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libplain.so.2
Reading symbols from /usr/lib64/sasl2/libsasldb.so.2...done.
Loaded symbols for /usr/lib64/sasl2/libsasldb.so.2
0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
(gdb) bt
#0 0x00000031a860ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002b1c99d65f1b in ldap_pvt_thread_rmutex_lock (rmutex=<value
optimized out>, owner=1201883456) at rmutex.c:129
#2 0x000000000052efb7 in accesslog_op_mod (op=0x47a32fe0,
rs=0x47a33150) at accesslog.c:1889
#3 0x00000000004bbb2b in overlay_op_walk (op=0x47a32fe0, rs=0x47a33150,
which=op_modify, oi=0x1ed31f10, on=0x1ed32d60) at backover.c:661
#4 0x00000000004bbdbd in over_op_func (op=0x47a32fe0, rs=0x47a33150,
which=op_modify) at backover.c:723
#5 0x00000000004bbed3 in over_op_modify (op=0x47a32fe0, rs=0x47a33150)
at backover.c:762
#6 0x000000000053c7af in ppolicy_bind_response (op=0x1fe6e000,
rs=0x47a33c30) at ppolicy.c:1173
#7 0x0000000000443a08 in slap_response_play (op=0x1fe6e000,
rs=0x47a33c30) at result.c:505
#8 0x0000000000443c2f in send_ldap_response (op=0x1fe6e000,
rs=0x47a33c30) at result.c:579
#9 0x0000000000444b14 in slap_send_ldap_result (op=0x1fe6e000,
rs=0x47a33c30) at result.c:851
#10 0x000000000050f2f3 in hdb_bind (op=0x1fe6e000, rs=0x47a33c30) at
bind.c:158
#11 0x00000000004bbbba in overlay_op_walk (op=0x1fe6e000, rs=0x47a33c30,
which=op_bind, oi=0x1ed31f10, on=0x0) at backover.c:671
#12 0x00000000004bbdbd in over_op_func (op=0x1fe6e000, rs=0x47a33c30,
which=op_bind) at backover.c:723
#13 0x00000000004bbe43 in over_op_bind (op=0x1fe6e000, rs=0x47a33c30) at
backover.c:738
#14 0x0000000000456411 in fe_op_bind (op=0x1fe6e000, rs=0x47a33c30) at
bind.c:383
#15 0x00000000004bbbba in overlay_op_walk (op=0x1fe6e000, rs=0x47a33c30,
which=op_bind, oi=0x1ed02760, on=0x0) at backover.c:671
#16 0x00000000004bbdbd in over_op_func (op=0x1fe6e000, rs=0x47a33c30,
which=op_bind) at backover.c:723
#17 0x00000000004bbe43 in over_op_bind (op=0x1fe6e000, rs=0x47a33c30) at
backover.c:738
#18 0x0000000000455b40 in do_bind (op=0x1fe6e000, rs=0x47a33c30) at
bind.c:205
#19 0x000000000042dbce in connection_operation (ctx=0x47a33d70,
arg_v=0x1fe6e000) at connection.c:1138
#20 0x000000000042e153 in connection_read_thread (ctx=0x47a33d70,
argv=0x6a) at connection.c:1274
#21 0x00002b1c99d66fc8 in ldap_int_thread_pool_wrapper
(xpool=0x1ec9acb0) at tpool.c:685
#22 0x00000031a86064a7 in start_thread () from /lib64/libpthread.so.0
#23 0x00000031a7ad3c2d in clone () from /lib64/libc.so.6
I can provide you with the other threads backtrace if needed.
Quanah Gibson-Mount a écrit :
--On Thursday, August 04, 2011 2:23 PM +0200 Cyril Grosjean
<cgrosjean(a)janua.fr> wrote:
>
> I've just tested with OpenLDAP 2.4.26, as you suggested, but the problem
> remains.
>
> I've no load at all, I use mirror mode replication, and this time, the
> freeze occurred when trying to modify a user's password:
> The modification is done from a usual Web interface from a Java
> application. The Java application is set to connect to OpenLDAP
> with a unique dedicated account, but in the case of a password
> change, we
> use the PROXYAUTH operation to do the modification
> with the user's rights, so that his pwdreset attribute is removed if
> previously set to TRUE
>
> I've also opened a case with Red Hat, they suggested me to test under
> RHEL 6, with their OpenLDAP rpm package (2.4.23-15.el6).
> So I'll do it right now but if you need more details on the RHEL 5.4
> hung
> slapd process, I can send them since it's still frozen.
Please build with debugging symbols and no optimization (-O0 and -g).
Make sure when you install, you set STRIP="" so it doesn't strip the
symbols out. Then get a full backtrace.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
8:43 a.m.
--On Monday, August 08, 2011 5:31 PM +0200 Cyril Grosjean
<cgrosjean(a)janua.fr> wrote:
Here're the traces you required, for the 1st 4 OpenLDAP threads at least:
Please follow:
<http://www.openldap.org/faq/data/cache/59.html>
Particularly use:
thread apply all bt full
I would at this point advise you to file an ITS, since you can regularly
lock up slapd. In addition to the FULL backtrace, provide your full
slapd.conf OR cn=config database, minus any passwords.
Thanks.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
9:27 a.m.
I did the full backtrace and logged a new ITS:
http://www.openldap.org/its/index.cgi/Incoming?id=7013;selectid=7013
Please also notice I haven't been able to upload my materials, since
there's no space left on the OpenLDAP FTP device.
Quanah Gibson-Mount a écrit :
--On Monday, August 08, 2011 5:31 PM +0200 Cyril Grosjean
<cgrosjean(a)janua.fr> wrote:
>
>
> Here're the traces you required, for the 1st 4 OpenLDAP threads at
> least:
Please follow:
<http://www.openldap.org/faq/data/cache/59.html>
Particularly use:
thread apply all bt full
I would at this point advise you to file an ITS, since you can
regularly lock up slapd. In addition to the FULL backtrace, provide
your full slapd.conf OR cn=config database, minus any passwords.
Thanks.
--Quanah
--
Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
4313
days inactive
4319
days old
openldap-technical@openldap.org
10 comments
3 participants
participants (3)
-
Cyril Grosjean -
Cyril GROSJEAN
-
Quanah Gibson-Mount