I'm getting failures on test039-glue-ldap-concurrency:
lt-slapd: bind.c:157: ldap_back_conn_delete: Assertion `!(*(&((lc))->lc_lcflags) & (((0x00000020U))))' failed.
Full testrun data:
wiki.suretec.org/testrun.tar.gz
Thanks.
ghenry@OpenLDAP.org wrote:
I'm getting failures on test039-glue-ldap-concurrency:
lt-slapd: bind.c:157: ldap_back_conn_delete: Assertion `!(*(&((lc))->lc_lcflags) & (((0x00000020U))))' failed.
Full testrun data:
wiki.suretec.org/testrun.tar.gz
That seems to be ITS#5894
Ciao, Michael.
RE24 synced from CVS half an hour ago hangs in test008.
openSUSE 11.1 packages: kernel-default-base-2.6.27.7-9.1 gcc-4.3-34.168 cyrus-sasl-2.1.22-182.1 BDB 4.5.21 (AFAIK it's patched) krb5-1.6.3-132.1
Built with -O4 optimizations.
----------------------------------------------------------------------------
Last lines of slapd.1.log:
bdb_idl_delete_key: db [c0f81f85] <= key_change 0 => key_change(DELETE,db) bdb_idl_delete_key: db [e46dc5bf] <= key_change 0 => key_change(DELETE,db) bdb_idl_delete_key: db [2a965451] <= key_change 0 => key_change(DELETE,db) bdb_idl_delete_key: db [4c90b4ba] <= key_change 0 <= index_entry_del( 219, "cn=James A Jones 2,ou=Alumni Association,ou=People,dc=example,dc=com" ) success ====> bdb_cache_delete( 219 ) bdb_delete: deleted id=000000db dn="cn=James A Jones 2,ou=Alumni Association,ou=People,dc=example,dc=com" send_ldap_result: conn=19 op=100 p=3 send_ldap_result: err=0 matched="" text="" send_ldap_response: msgid=101 tag=107 err=0 ber_flush2: 14 bytes to sd 35 conn=19 op=100 RESULT tag=107 err=0 text= connection_get(35) connection_get(35): got connid=19 connection_read(35): checking for input on id=19 ber_get_next ber_get_next: tag 0x30 len 5 contents: ber_get_next conn=19 op=101 do_unbind conn=19 op=101 UNBIND connection_close: conn=19 sd=35 conn=19 fd=35 closed
----------------------------------------------------------------------------
(gdb) bt all No symbol "all" in current context. (gdb) bt full #0 0xffffe430 in __kernel_vsyscall () No symbol table info available. #1 0x4024e747 in pthread_join () from /lib/libpthread.so.0 No symbol table info available. #2 0x40030064 in ldap_pvt_thread_join (thread=1087380368, thread_return=0x0) at thr_posix.c:197 No locals. #3 0x08065ac3 in slapd_daemon () at daemon.c:2658 listener_tid = 1087380368 rc = 0 #4 0x0805103e in main (argc=8, argv=0xbfa2dd14) at main.c:948 i = 137470840 no_detach = 1 rc = -12 urls = 0x82a2060 "ldap://localhost:9011/" username = 0x0 groupname = 0x0 sandbox = 0x0 syslogUser = 160 configfile = 0x82a2008 "/usr/src/michael/openldap/OPENLDAP_REL_ENG_2_4/openldap/tests/testrun/slapd.1.conf" configdir = 0x0 serverName = 0xbfa2edf9 "lt-slapd" scp = <value optimized out> scp_entry = <value optimized out> debug_unknowns = (char **) 0x0 syslog_unknowns = (char **) 0x0 slapd_pid_file_unlink = 1 slapd_args_file_unlink = 1 firstopt = <value optimized out> __PRETTY_FUNCTION__ = "main"
--On January 28, 2009 4:33:53 PM +0100 Michael Ströder michael@stroeder.com wrote:
(gdb) bt all No symbol "all" in current context.
Generally:
thr apply all bt
I think is what you are going for (i.e., get a full backtrace on all threads).
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount wrote:
--On January 28, 2009 4:33:53 PM +0100 Michael Ströder michael@stroeder.com wrote:
(gdb) bt all No symbol "all" in current context.
Generally:
thr apply all bt
I think is what you are going for (i.e., get a full backtrace on all threads).
Now gdb backtrace of hanging test008 with HEAD and compiled with only -O2:
Thread 18 (Thread 0x40d14b90 (LWP 26740)): #0 0xffffe430 in __kernel_vsyscall () #1 0x40579c26 in epoll_wait () from /lib/libc.so.6 #2 0x0806617f in slapd_daemon_task (ptr=0x0) at daemon.c:2291 #3 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #4 0x4057938e in clone () from /lib/libc.so.6
Thread 17 (Thread 0x41115b90 (LWP 26741)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 16 (Thread 0x41e18b90 (LWP 27131)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 15 (Thread 0x42b1bb90 (LWP 27136)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x4073a288, mutex=0x4073a270) at thr_posix.c:277 #3 0x0807a418 in send_ldap_ber (conn=0x4073a178, ber=0x42b19de4) at result.c:217 #4 0x0807c43b in slap_send_search_entry (op=0x8381ce0, rs=0x42b1b0c8) at result.c:1246 #5 0x0810b2ba in monitor_send_children (op=0x8381ce0, rs=0x42b1b0c8, e_parent=0x835f3e4, sub=1) at search.c:115 #6 0x0810b1cc in monitor_send_children (op=0x8381ce0, rs=0x42b1b0c8, e_parent=<value optimized out>, sub=1) at search.c:123 #7 0x0810b5dd in monitor_back_search (op=0x8381ce0, rs=0x42b1b0c8) at search.c:245 #8 0x0806bb73 in fe_op_search (op=0x8381ce0, rs=0x42b1b0c8) at search.c:366 #9 0x0806c3e8 in do_search (op=0x8381ce0, rs=0x42b1b0c8) at search.c:217 #10 0x080696bb in connection_operation (ctx=0x42b1b1cc, arg_v=0x8381ce0) at connection.c:1100 #11 0x0806a117 in connection_read_thread (ctx=0x42b1b1cc, argv=0x1c) at connection.c:1226 #12 0x4002f400 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:663 #13 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #14 0x4057938e in clone () from /lib/libc.so.6
Thread 14 (Thread 0x4381eb90 (LWP 27138)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 13 (Thread 0x43c1fb90 (LWP 27144)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 12 (Thread 0x44020b90 (LWP 27145)): ---Type <return> to continue, or q <return> to quit--- #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 11 (Thread 0x44421b90 (LWP 27146)): #0 0xffffe430 in __kernel_vsyscall () #1 0x402507b9 in __lll_lock_wait () from /lib/libpthread.so.0 #2 0x4024bcd1 in _L_lock_89 () from /lib/libpthread.so.0 #3 0x4024b5b2 in pthread_mutex_lock () from /lib/libpthread.so.0 #4 0x4002fffd in ldap_pvt_thread_mutex_lock (mutex=0x837a8c8) at thr_posix.c:296 #5 0x0815f27c in monitor_cache_lock (e=0x835f3e4) at cache.c:117 #6 0x0810b155 in monitor_send_children (op=0x83891a8, rs=0x444210c8, e_parent=<value optimized out>, sub=1) at search.c:144 #7 0x0810b1cc in monitor_send_children (op=0x83891a8, rs=0x444210c8, e_parent=<value optimized out>, sub=1) at search.c:123 #8 0x0810b5dd in monitor_back_search (op=0x83891a8, rs=0x444210c8) at search.c:245 #9 0x0806bb73 in fe_op_search (op=0x83891a8, rs=0x444210c8) at search.c:366 #10 0x0806c3e8 in do_search (op=0x83891a8, rs=0x444210c8) at search.c:217 #11 0x080696bb in connection_operation (ctx=0x444211cc, arg_v=0x83891a8) at connection.c:1100 #12 0x0806a117 in connection_read_thread (ctx=0x444211cc, argv=0x18) at connection.c:1226 #13 0x4002f400 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:663 #14 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #15 0x4057938e in clone () from /lib/libc.so.6
Thread 10 (Thread 0x45427b90 (LWP 27150)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 9 (Thread 0x45929b90 (LWP 27154)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 8 (Thread 0x45e2bb90 (LWP 27160)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 7 (Thread 0x4632db90 (LWP 27163)): #0 0xffffe430 in __kernel_vsyscall () #1 0x402507b9 in __lll_lock_wait () from /lib/libpthread.so.0 #2 0x4024bcd1 in _L_lock_89 () from /lib/libpthread.so.0 #3 0x4024b5b2 in pthread_mutex_lock () from /lib/libpthread.so.0 #4 0x4002fffd in ldap_pvt_thread_mutex_lock (mutex=0x837a8c8) at thr_posix.c:296 #5 0x0815f27c in monitor_cache_lock (e=0x835f3e4) at cache.c:117 #6 0x0810b155 in monitor_send_children (op=0x839b3a0, rs=0x4632d0c8, e_parent=<value optimized out>, sub=1) at search.c:144 #7 0x0810b1cc in monitor_send_children (op=0x839b3a0, rs=0x4632d0c8, e_parent=<value optimized out>, sub=1) at search.c:123 #8 0x0810b5dd in monitor_back_search (op=0x839b3a0, rs=0x4632d0c8) at search.c:245 #9 0x0806bb73 in fe_op_search (op=0x839b3a0, rs=0x4632d0c8) at search.c:366 #10 0x0806c3e8 in do_search (op=0x839b3a0, rs=0x4632d0c8) at search.c:217 #11 0x080696bb in connection_operation (ctx=0x4632d1cc, arg_v=0x839b3a0) at connection.c:1100 #12 0x0806a117 in connection_read_thread (ctx=0x4632d1cc, argv=0x1f) at connection.c:1226 #13 0x4002f400 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:663 #14 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #15 0x4057938e in clone () from /lib/libc.so.6
Thread 6 (Thread 0x4682fb90 (LWP 27166)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 5 (Thread 0x46d31b90 (LWP 27169)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 4 (Thread 0x47233b90 (LWP 27170)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 3 (Thread 0x47634b90 (LWP 27173)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 2 (Thread 0x47a35b90 (LWP 27174)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 1 (Thread 0x406f1930 (LWP 26725)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024a747 in pthread_join () from /lib/libpthread.so.0 #2 0x400301d4 in ldap_pvt_thread_join (thread=1087458192, thread_return=0x0) at thr_posix.c:197 #3 0x08063693 in slapd_daemon () at daemon.c:2658 #4 0x08050ee6 in main (argc=8, argv=0xbf90dc64) at main.c:948
Michael Ströder wrote:
Quanah Gibson-Mount wrote:
--On January 28, 2009 4:33:53 PM +0100 Michael Ströder michael@stroeder.com wrote:
(gdb) bt all No symbol "all" in current context.
Generally:
thr apply all bt
I think is what you are going for (i.e., get a full backtrace on all threads).
Now gdb backtrace of hanging test008 with HEAD and compiled with only -O2:
Thanks. Seems like this is a race condition due to the edge-triggered epoll - if the socket becomes writable before we wait for it, we won't wake up from the wait. Apparently we'll have to go back to level-triggered like before. Too bad we can't set different epoll behaviors for different events.
(Thread 15 is waiting to write again...)
Thread 18 (Thread 0x40d14b90 (LWP 26740)): #0 0xffffe430 in __kernel_vsyscall () #1 0x40579c26 in epoll_wait () from /lib/libc.so.6 #2 0x0806617f in slapd_daemon_task (ptr=0x0) at daemon.c:2291 #3 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #4 0x4057938e in clone () from /lib/libc.so.6
Thread 17 (Thread 0x41115b90 (LWP 26741)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 16 (Thread 0x41e18b90 (LWP 27131)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 15 (Thread 0x42b1bb90 (LWP 27136)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x4073a288, mutex=0x4073a270) at thr_posix.c:277 #3 0x0807a418 in send_ldap_ber (conn=0x4073a178, ber=0x42b19de4) at result.c:217 #4 0x0807c43b in slap_send_search_entry (op=0x8381ce0, rs=0x42b1b0c8) at result.c:1246 #5 0x0810b2ba in monitor_send_children (op=0x8381ce0, rs=0x42b1b0c8, e_parent=0x835f3e4, sub=1) at search.c:115 #6 0x0810b1cc in monitor_send_children (op=0x8381ce0, rs=0x42b1b0c8, e_parent=<value optimized out>, sub=1) at search.c:123 #7 0x0810b5dd in monitor_back_search (op=0x8381ce0, rs=0x42b1b0c8) at search.c:245 #8 0x0806bb73 in fe_op_search (op=0x8381ce0, rs=0x42b1b0c8) at search.c:366 #9 0x0806c3e8 in do_search (op=0x8381ce0, rs=0x42b1b0c8) at search.c:217 #10 0x080696bb in connection_operation (ctx=0x42b1b1cc, arg_v=0x8381ce0) at connection.c:1100 #11 0x0806a117 in connection_read_thread (ctx=0x42b1b1cc, argv=0x1c) at connection.c:1226 #12 0x4002f400 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:663 #13 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #14 0x4057938e in clone () from /lib/libc.so.6
Thread 14 (Thread 0x4381eb90 (LWP 27138)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 13 (Thread 0x43c1fb90 (LWP 27144)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 12 (Thread 0x44020b90 (LWP 27145)): ---Type<return> to continue, or q<return> to quit--- #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 11 (Thread 0x44421b90 (LWP 27146)): #0 0xffffe430 in __kernel_vsyscall () #1 0x402507b9 in __lll_lock_wait () from /lib/libpthread.so.0 #2 0x4024bcd1 in _L_lock_89 () from /lib/libpthread.so.0 #3 0x4024b5b2 in pthread_mutex_lock () from /lib/libpthread.so.0 #4 0x4002fffd in ldap_pvt_thread_mutex_lock (mutex=0x837a8c8) at thr_posix.c:296 #5 0x0815f27c in monitor_cache_lock (e=0x835f3e4) at cache.c:117 #6 0x0810b155 in monitor_send_children (op=0x83891a8, rs=0x444210c8, e_parent=<value optimized out>, sub=1) at search.c:144 #7 0x0810b1cc in monitor_send_children (op=0x83891a8, rs=0x444210c8, e_parent=<value optimized out>, sub=1) at search.c:123 #8 0x0810b5dd in monitor_back_search (op=0x83891a8, rs=0x444210c8) at search.c:245 #9 0x0806bb73 in fe_op_search (op=0x83891a8, rs=0x444210c8) at search.c:366 #10 0x0806c3e8 in do_search (op=0x83891a8, rs=0x444210c8) at search.c:217 #11 0x080696bb in connection_operation (ctx=0x444211cc, arg_v=0x83891a8) at connection.c:1100 #12 0x0806a117 in connection_read_thread (ctx=0x444211cc, argv=0x18) at connection.c:1226 #13 0x4002f400 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:663 #14 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #15 0x4057938e in clone () from /lib/libc.so.6
Thread 10 (Thread 0x45427b90 (LWP 27150)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 9 (Thread 0x45929b90 (LWP 27154)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 8 (Thread 0x45e2bb90 (LWP 27160)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 7 (Thread 0x4632db90 (LWP 27163)): #0 0xffffe430 in __kernel_vsyscall () #1 0x402507b9 in __lll_lock_wait () from /lib/libpthread.so.0 #2 0x4024bcd1 in _L_lock_89 () from /lib/libpthread.so.0 #3 0x4024b5b2 in pthread_mutex_lock () from /lib/libpthread.so.0 #4 0x4002fffd in ldap_pvt_thread_mutex_lock (mutex=0x837a8c8) at thr_posix.c:296 #5 0x0815f27c in monitor_cache_lock (e=0x835f3e4) at cache.c:117 #6 0x0810b155 in monitor_send_children (op=0x839b3a0, rs=0x4632d0c8, e_parent=<value optimized out>, sub=1) at search.c:144 #7 0x0810b1cc in monitor_send_children (op=0x839b3a0, rs=0x4632d0c8, e_parent=<value optimized out>, sub=1) at search.c:123 #8 0x0810b5dd in monitor_back_search (op=0x839b3a0, rs=0x4632d0c8) at search.c:245 #9 0x0806bb73 in fe_op_search (op=0x839b3a0, rs=0x4632d0c8) at search.c:366 #10 0x0806c3e8 in do_search (op=0x839b3a0, rs=0x4632d0c8) at search.c:217 #11 0x080696bb in connection_operation (ctx=0x4632d1cc, arg_v=0x839b3a0) at connection.c:1100 #12 0x0806a117 in connection_read_thread (ctx=0x4632d1cc, argv=0x1f) at connection.c:1226 #13 0x4002f400 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:663 #14 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #15 0x4057938e in clone () from /lib/libc.so.6
Thread 6 (Thread 0x4682fb90 (LWP 27166)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 5 (Thread 0x46d31b90 (LWP 27169)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 4 (Thread 0x47233b90 (LWP 27170)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 3 (Thread 0x47634b90 (LWP 27173)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 2 (Thread 0x47a35b90 (LWP 27174)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0x4024a1b5 in start_thread () from /lib/libpthread.so.0 #5 0x4057938e in clone () from /lib/libc.so.6
Thread 1 (Thread 0x406f1930 (LWP 26725)): #0 0xffffe430 in __kernel_vsyscall () #1 0x4024a747 in pthread_join () from /lib/libpthread.so.0 #2 0x400301d4 in ldap_pvt_thread_join (thread=1087458192, thread_return=0x0) at thr_posix.c:197 #3 0x08063693 in slapd_daemon () at daemon.c:2658 #4 0x08050ee6 in main (argc=8, argv=0xbf90dc64) at main.c:948
Michael Ströder wrote:
Quanah Gibson-Mount wrote:
--On January 28, 2009 4:33:53 PM +0100 Michael Ströder michael@stroeder.com wrote:
(gdb) bt all No symbol "all" in current context.
Generally:
thr apply all bt
I think is what you are going for (i.e., get a full backtrace on all threads).
Now gdb backtrace of hanging test008 with HEAD and compiled with only -O2:
OK, please try current HEAD again. It's reverted to the original epoll behavior (level-triggered) so this type of problem can no longer occur. Also I've found that setting the descriptor to edge-triggered after-the-fact works, so it still prevents the endless hangup reports from happening.
Howard Chu wrote:
Michael Ströder wrote:
Now gdb backtrace of hanging test008 with HEAD and compiled with only -O2:
OK, please try current HEAD again. It's reverted to the original epoll behavior (level-triggered) so this type of problem can no longer occur. Also I've found that setting the descriptor to edge-triggered after-the-fact works, so it still prevents the endless hangup reports from happening.
There was only a change to servers/slapd/daemon.c. Is that right? Still I see hangs in test008 with HEAD.
Ciao, Michael.
Thread 18 (Thread 0xb7641b90 (LWP 15588)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7a24c26 in epoll_wait () from /lib/libc.so.6 #2 0x0806617f in slapd_daemon_task (ptr=0x0) at daemon.c:2291 #3 0xb7cff1b5 in start_thread () from /lib/libpthread.so.0 #4 0xb7a2438e in clone () from /lib/libc.so.6
Thread 17 (Thread 0xb7240b90 (LWP 15589)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7d057b9 in __lll_lock_wait () from /lib/libpthread.so.0 #2 0xb7d00cd1 in _L_lock_89 () from /lib/libpthread.so.0 #3 0xb7d005b2 in pthread_mutex_lock () from /lib/libpthread.so.0 #4 0xb7efaffd in ldap_pvt_thread_mutex_lock (mutex=0x837a5f8) at thr_posix.c:296 #5 0x0815f2cc in monitor_cache_lock (e=0x835f3bc) at cache.c:117 #6 0x0810b1a5 in monitor_send_children (op=0x837d338, rs=0xb72400c8, e_parent=<value optimized out>, sub=1) at search.c:144 #7 0x0810b21c in monitor_send_children (op=0x837d338, rs=0xb72400c8, e_parent=<value optimized out>, sub=1) at search.c:123 #8 0x0810b62d in monitor_back_search (op=0x837d338, rs=0xb72400c8) at search.c:245 #9 0x0806bbc3 in fe_op_search (op=0x837d338, rs=0xb72400c8) at search.c:366 #10 0x0806c438 in do_search (op=0x837d338, rs=0xb72400c8) at search.c:217 #11 0x0806970b in connection_operation (ctx=0xb72401cc, arg_v=0x837d338) at connection.c:1100 #12 0x0806a167 in connection_read_thread (ctx=0xb72401cc, argv=0x1c) at connection.c:1226 #13 0xb7efa400 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:663 #14 0xb7cff1b5 in start_thread () from /lib/libpthread.so.0 #15 0xb7a2438e in clone () from /lib/libc.so.6
Thread 16 (Thread 0xb653db90 (LWP 15716)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7d02c35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0xb7efb094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0xb7efa472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0xb7cff1b5 in start_thread () from /lib/libpthread.so.0 #5 0xb7a2438e in clone () from /lib/libc.so.6
Thread 15 (Thread 0xb583ab90 (LWP 15736)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7d02c35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0xb7efb094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0xb7efa472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0xb7cff1b5 in start_thread () from /lib/libpthread.so.0 #5 0xb7a2438e in clone () from /lib/libc.so.6
Thread 14 (Thread 0xb4b37b90 (LWP 15757)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7d057b9 in __lll_lock_wait () from /lib/libpthread.so.0 #2 0xb7d00cd1 in _L_lock_89 () from /lib/libpthread.so.0 #3 0xb7d005b2 in pthread_mutex_lock () from /lib/libpthread.so.0 #4 0xb7efaffd in ldap_pvt_thread_mutex_lock (mutex=0x837a5f8) at thr_posix.c:296 #5 0x0815f2cc in monitor_cache_lock (e=0x835f3bc) at cache.c:117 #6 0x0810b1a5 in monitor_send_children (op=0x8385510, rs=0xb4b370c8, e_parent=<value optimized out>, sub=1) at search.c:144 #7 0x0810b21c in monitor_send_children (op=0x8385510, rs=0xb4b370c8, e_parent=<value optimized out>, sub=1) at search.c:123 #8 0x0810b62d in monitor_back_search (op=0x8385510, rs=0xb4b370c8) at search.c:245 #9 0x0806bbc3 in fe_op_search (op=0x8385510, rs=0xb4b370c8) at search.c:366 #10 0x0806c438 in do_search (op=0x8385510, rs=0xb4b370c8) at search.c:217 #11 0x0806970b in connection_operation (ctx=0xb4b371cc, arg_v=0x8385510) at connection.c:1100 #12 0x0806a167 in connection_read_thread (ctx=0xb4b371cc, argv=0x14) at connection.c:1226 #13 0xb7efa400 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:663 #14 0xb7cff1b5 in start_thread () from /lib/libpthread.so.0 #15 0xb7a2438e in clone () from /lib/libc.so.6
Thread 13 (Thread 0xb4635b90 (LWP 15775)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7d02c35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0xb7efb094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0xb7efa472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0xb7cff1b5 in start_thread () from /lib/libpthread.so.0 #5 0xb7a2438e in clone () from /lib/libc.so.6
Thread 12 (Thread 0xb3932b90 (LWP 15776)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7d057b9 in __lll_lock_wait () from /lib/libpthread.so.0 #2 0xb7d00cd1 in _L_lock_89 () from /lib/libpthread.so.0 #3 0xb7d005b2 in pthread_mutex_lock () from /lib/libpthread.so.0 #4 0xb7efaffd in ldap_pvt_thread_mutex_lock (mutex=0x837a5f8) at thr_posix.c:296 #5 0x0815f2cc in monitor_cache_lock (e=0x835f3bc) at cache.c:117 #6 0x0810b1a5 in monitor_send_children (op=0x838a7c0, rs=0xb39320c8, e_parent=<value optimized out>, sub=1) at search.c:144 #7 0x0810b21c in monitor_send_children (op=0x838a7c0, rs=0xb39320c8, e_parent=<value optimized out>, sub=1) at search.c:123 #8 0x0810b62d in monitor_back_search (op=0x838a7c0, rs=0xb39320c8) at search.c:245 #9 0x0806bbc3 in fe_op_search (op=0x838a7c0, rs=0xb39320c8) at search.c:366 #10 0x0806c438 in do_search (op=0x838a7c0, rs=0xb39320c8) at search.c:217 #11 0x0806970b in connection_operation (ctx=0xb39321cc, arg_v=0x838a7c0) at connection.c:1100 #12 0x0806a167 in connection_read_thread (ctx=0xb39321cc, argv=0x22) at connection.c:1226 #13 0xb7efa400 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:663 #14 0xb7cff1b5 in start_thread () from /lib/libpthread.so.0 #15 0xb7a2438e in clone () from /lib/libc.so.6
Thread 11 (Thread 0xb3430b90 (LWP 15781)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7d02c35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0xb7efb094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0xb7efa472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0xb7cff1b5 in start_thread () from /lib/libpthread.so.0 #5 0xb7a2438e in clone () from /lib/libc.so.6
Thread 10 (Thread 0xb272db90 (LWP 15824)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7d02c35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0xb7efb094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0xb7efa472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0xb7cff1b5 in start_thread () from /lib/libpthread.so.0 #5 0xb7a2438e in clone () from /lib/libc.so.6
Thread 9 (Thread 0xb1229b90 (LWP 15841)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7d02c35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0xb7efb094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0xb7efa472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0xb7cff1b5 in start_thread () from /lib/libpthread.so.0 #5 0xb7a2438e in clone () from /lib/libc.so.6
Thread 8 (Thread 0xb0e28b90 (LWP 15854)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7d02c35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0xb7efb094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0xb7efa472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0xb7cff1b5 in start_thread () from /lib/libpthread.so.0 #5 0xb7a2438e in clone () from /lib/libc.so.6
Thread 7 (Thread 0xb0825b90 (LWP 15857)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7d02c35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0xb7efb094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0xb7efa472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0xb7cff1b5 in start_thread () from /lib/libpthread.so.0 #5 0xb7a2438e in clone () from /lib/libc.so.6
Thread 6 (Thread 0xafb22b90 (LWP 15873)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7d02c35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0xb7efb094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0xb7efa472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0xb7cff1b5 in start_thread () from /lib/libpthread.so.0 #5 0xb7a2438e in clone () from /lib/libc.so.6
Thread 5 (Thread 0xaf721b90 (LWP 15885)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7d02c35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0xb7efb094 in ldap_pvt_thread_cond_wait (cond=0xb77b0610, mutex=0xb77b05f8) at thr_posix.c:277 #3 0x0807a468 in send_ldap_ber (conn=0xb77b0500, ber=0xaf71fde4) at result.c:217 #4 0x0807c48b in slap_send_search_entry (op=0x839d138, rs=0xaf7210c8) at result.c:1246 #5 0x0810b30a in monitor_send_children (op=0x839d138, rs=0xaf7210c8, e_parent=0x835f3bc, sub=1) at search.c:115 #6 0x0810b21c in monitor_send_children (op=0x839d138, rs=0xaf7210c8, e_parent=<value optimized out>, sub=1) at search.c:123 #7 0x0810b62d in monitor_back_search (op=0x839d138, rs=0xaf7210c8) at search.c:245 #8 0x0806bbc3 in fe_op_search (op=0x839d138, rs=0xaf7210c8) at search.c:366 #9 0x0806c438 in do_search (op=0x839d138, rs=0xaf7210c8) at search.c:217 #10 0x0806970b in connection_operation (ctx=0xaf7211cc, arg_v=0x839d138) at connection.c:1100 #11 0x0806a167 in connection_read_thread (ctx=0xaf7211cc, argv=0x1e) at connection.c:1226 #12 0xb7efa400 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:663 #13 0xb7cff1b5 in start_thread () from /lib/libpthread.so.0 #14 0xb7a2438e in clone () from /lib/libc.so.6
Thread 4 (Thread 0xaf11eb90 (LWP 15887)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7d057b9 in __lll_lock_wait () from /lib/libpthread.so.0 #2 0xb7d00cd1 in _L_lock_89 () from /lib/libpthread.so.0 #3 0xb7d005b2 in pthread_mutex_lock () from /lib/libpthread.so.0 #4 0xb7efaffd in ldap_pvt_thread_mutex_lock (mutex=0x837a5f8) at thr_posix.c:296 #5 0x0815f2cc in monitor_cache_lock (e=0x835f3bc) at cache.c:117 #6 0x0810b1a5 in monitor_send_children (op=0x83a4810, rs=0xaf11e0c8, e_parent=<value optimized out>, sub=1) at search.c:144 #7 0x0810b21c in monitor_send_children (op=0x83a4810, rs=0xaf11e0c8, e_parent=<value optimized out>, sub=1) at search.c:123 #8 0x0810b62d in monitor_back_search (op=0x83a4810, rs=0xaf11e0c8) at search.c:245 #9 0x0806bbc3 in fe_op_search (op=0x83a4810, rs=0xaf11e0c8) at search.c:366 #10 0x0806c438 in do_search (op=0x83a4810, rs=0xaf11e0c8) at search.c:217 #11 0x0806970b in connection_operation (ctx=0xaf11e1cc, arg_v=0x83a4810) at connection.c:1100 #12 0x0806a167 in connection_read_thread (ctx=0xaf11e1cc, argv=0x12) at connection.c:1226 #13 0xb7efa400 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:663 #14 0xb7cff1b5 in start_thread () from /lib/libpthread.so.0 #15 0xb7a2438e in clone () from /lib/libc.so.6
Thread 3 (Thread 0xaed1db90 (LWP 15897)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7d02c35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0xb7efb094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0xb7efa472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0xb7cff1b5 in start_thread () from /lib/libpthread.so.0 #5 0xb7a2438e in clone () from /lib/libc.so.6
Thread 2 (Thread 0xae91cb90 (LWP 15898)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7d02c35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0xb7efb094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4, mutex=0x82ba4dc) at thr_posix.c:277 #3 0xb7efa472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at tpool.c:654 #4 0xb7cff1b5 in start_thread () from /lib/libpthread.so.0 #5 0xb7a2438e in clone () from /lib/libc.so.6
Thread 1 (Thread 0xb7864720 (LWP 15573)): #0 0xffffe430 in __kernel_vsyscall () #1 0xb7cff747 in pthread_join () from /lib/libpthread.so.0 #2 0xb7efb1d4 in ldap_pvt_thread_join (thread=3076791184, thread_return=0x0) at thr_posix.c:197 #3 0x08063693 in slapd_daemon () at daemon.c:2663 #4 0x08050ee6 in main (argc=8, argv=0xbf852d54) at main.c:948
Michael Ströder wrote:
Howard Chu wrote:
Michael Ströder wrote:
Now gdb backtrace of hanging test008 with HEAD and compiled with only -O2:
OK, please try current HEAD again. It's reverted to the original epoll behavior (level-triggered) so this type of problem can no longer occur. Also I've found that setting the descriptor to edge-triggered after-the-fact works, so it still prevents the endless hangup reports from happening.
There was only a change to servers/slapd/daemon.c. Is that right?
At that time, right.
Still I see hangs in test008 with HEAD.
Still unable to reproduce it on any of my machines. But, see if the updated connection.c makes a difference.
Howard Chu wrote:
Michael Ströder wrote:
Howard Chu wrote:
Michael Ströder wrote:
Now gdb backtrace of hanging test008 with HEAD and compiled with only -O2:
OK, please try current HEAD again. It's reverted to the original epoll behavior (level-triggered) so this type of problem can no longer occur. Also I've found that setting the descriptor to edge-triggered after-the-fact works, so it still prevents the endless hangup reports from happening.
There was only a change to servers/slapd/daemon.c. Is that right?
At that time, right.
Still I see hangs in test008 with HEAD.
Still unable to reproduce it on any of my machines. But, see if the updated connection.c makes a difference.
It works now with ten iterations of test008.
I'm running this on a rather old laptop. Maybe your machines are too fast to reproduce it. ;-)
Ciao, Michael.
Michael Ströder wrote:
Howard Chu wrote:
Still unable to reproduce it on any of my machines. But, see if the updated connection.c makes a difference.
It works now with ten iterations of test008.
I'm running this on a rather old laptop. Maybe your machines are too fast to reproduce it. ;-)
Heh, wouldn't be the first time. I recall a while back another bug I was trying to track, that I could only catch by forcing my laptop CPU into its slowest speed. I guess I'll have to keep that in mind... ;)
Howard Chu wrote:
Michael Ströder wrote:
Howard Chu wrote:
Still unable to reproduce it on any of my machines. But, see if the updated connection.c makes a difference.
It works now with ten iterations of test008.
I'm running this on a rather old laptop. Maybe your machines are too fast to reproduce it. ;-)
Heh, wouldn't be the first time. I recall a while back another bug I was trying to track, that I could only catch by forcing my laptop CPU into its slowest speed. I guess I'll have to keep that in mind... ;)
My impression was it happens more often with more aggressive compiler optimization option. Not sure about this though.
Ciao, Michael.
Michael Ströder wrote:
Howard Chu wrote:
Michael Ströder wrote:
Howard Chu wrote:
Michael Ströder wrote:
Now gdb backtrace of hanging test008 with HEAD and compiled with only -O2:
OK, please try current HEAD again. It's reverted to the original epoll behavior (level-triggered) so this type of problem can no longer occur. Also I've found that setting the descriptor to edge-triggered after-the-fact works, so it still prevents the endless hangup reports from happening.
There was only a change to servers/slapd/daemon.c. Is that right?
At that time, right.
Still I see hangs in test008 with HEAD.
Still unable to reproduce it on any of my machines. But, see if the updated connection.c makes a difference.
It works now with ten iterations of test008.
Quanah, can you please check-in Howard's changes in HEAD to RE24?
Ciao, Michael.
--On Friday, January 30, 2009 1:33 PM +0100 Michael Ströder michael@stroeder.com wrote:
It works now with ten iterations of test008.
Quanah, can you please check-in Howard's changes in HEAD to RE24?
Sure, when I have the time to do so.
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
-
ghenry＠OpenLDAP.org -
Howard Chu -
Michael Ströder -
Quanah Gibson-Mount