After around 24 hours, backtrace is the same:
(gdb) thr apply all bt
Thread 5 (Thread 1106889024 (LWP 9361)): #0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b7819f27de9 in bdb_tool_trickle_task (ctx=<value optimized out>, ptr=<value optimized out>) at tools.c:1124 #2 0x00002b781715a2ca in ldap_int_thread_pool_wrapper (xpool=0x17e07fd0) at ../../../libraries/libldap_r/tpool.c:663 #3 0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0 #4 0x000000395e0d1e3d in clone () from /lib64/libc.so.6
Thread 4 (Thread 1115281728 (LWP 9362)): #0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b7819f28163 in bdb_tool_index_task (ctx=<value optimized out>, ptr=<value optimized out>) at tools.c:1147 #2 0x00002b781715a2ca in ldap_int_thread_pool_wrapper (xpool=0x17e07fd0) at ../../../libraries/libldap_r/tpool.c:663 #3 0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0 #4 0x000000395e0d1e3d in clone () from /lib64/libc.so.6
Thread 3 (Thread 1123674432 (LWP 9363)): #0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b7819f28163 in bdb_tool_index_task (ctx=<value optimized out>, ptr=<value optimized out>) at tools.c:1147 #2 0x00002b781715a2ca in ldap_int_thread_pool_wrapper (xpool=0x17e07fd0) at ../../../libraries/libldap_r/tpool.c:663 #3 0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0 #4 0x000000395e0d1e3d in clone () from /lib64/libc.so.6
Thread 2 (Thread 1132067136 (LWP 9360)): #0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b7819f28014 in bdb_tool_index_add (op=0x7fff93950310, txn=<value optimized out>, e=0x2b7b521d2068) at tools.c:475 #2 0x00002b7819f28970 in hdb_tool_entry_put (be=0x17f39240, e=0x2b7b521d2068, text=0x7fff93960cc0) at tools.c:545 #3 0x00000000004906f9 in slapadd (argc=<value optimized out>, argv=0x7fff93960ec8) at ../../../servers/slapd/slapadd.c:350 #4 0x0000000000416c59 in main (argc=6, argv=0x7fff93960ec8) at ../../../servers/slapd/main.c:391
Thread 1 (Thread 47794824997056 (LWP 9360)): #0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b7819f28014 in bdb_tool_index_add (op=0x7fff93950310, txn=<value optimized out>, e=0x2b7b521d2068) at tools.c:475 #2 0x00002b7819f28970 in hdb_tool_entry_put (be=0x17f39240, e=0x2b7b521d2068, text=0x7fff93960cc0) at tools.c:545 #3 0x00000000004906f9 in slapadd (argc=<value optimized out>, argv=0x7fff93960ec8) at ../../../servers/slapd/slapadd.c:350 #4 0x0000000000416c59 in main (argc=6, argv=0x7fff93960ec8) at ../../../servers/slapd/main.c:391 #0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
--On Sunday, March 22, 2009 2:37 AM +0000 quanah@zimbra.com wrote:
--On Saturday, March 21, 2009 7:09 PM -0700 Howard Chu hyc@symas.com wrote:
quanah@zimbra.com wrote:
Full_Name: Quanah Gibson-Mount Version: RE24 3/19/2009 OS: Linux 2.6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (75.111.29.239)
While testing various load scenarios, slapadd deadlocked on me.
The trace for thread 1 and 2 look identical, so I'm assuming they're the same thread. In that case I see 3 tool threads running. How many did you have configured? If 1 and 2 are really different threads (certainly their thread IDs look quite different), then there's something very wrong in the thread library...
print bdb_tool_index_tcount
(gdb) print bdb_tool_index_tcount $1 = 1
It looks like one thread has exited early. I'm not sure how that can happen since the only way for any index thread to exit is if slapd_shutdown is set. Is it set?
(gdb) print slapd_shutdown $2 = 0
It's configured with 4 tool threads.
--Quanah
(gdb) thr apply all bt
Thread 5 (Thread 1106889024 (LWP 9361)): # 0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 # 1 0x00002b7819f27de9 in bdb_tool_trickle_task (ctx=<value optimized # out>, ptr=<value optimized out>) at tools.c:1124 # 2 0x00002b781715a2ca in ldap_int_thread_pool_wrapper # (xpool=0x17e07fd0) at ../../../libraries/libldap_r/tpool.c:663 # 3 0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0 # 4 0x000000395e0d1e3d in clone () from /lib64/libc.so.6
Thread 4 (Thread 1115281728 (LWP 9362)): # 0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 # 1 0x00002b7819f28163 in bdb_tool_index_task (ctx=<value optimized # out>, ptr=<value optimized out>) at tools.c:1147 # 2 0x00002b781715a2ca in ldap_int_thread_pool_wrapper # (xpool=0x17e07fd0) at ../../../libraries/libldap_r/tpool.c:663 # 3 0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0 # 4 0x000000395e0d1e3d in clone () from /lib64/libc.so.6
Thread 3 (Thread 1123674432 (LWP 9363)): # 0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 # 1 0x00002b7819f28163 in bdb_tool_index_task (ctx=<value optimized # out>, ptr=<value optimized out>) at tools.c:1147 # 2 0x00002b781715a2ca in ldap_int_thread_pool_wrapper # (xpool=0x17e07fd0) at ../../../libraries/libldap_r/tpool.c:663 # 3 0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0 # 4 0x000000395e0d1e3d in clone () from /lib64/libc.so.6
Thread 2 (Thread 1132067136 (LWP 9360)): # 0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 # 1 0x00002b7819f28014 in bdb_tool_index_add (op=0x7fff93950310, # txn=<value optimized out>, e=0x2b7b521d2068) at tools.c:475 # 2 0x00002b7819f28970 in hdb_tool_entry_put (be=0x17f39240, # e=0x2b7b521d2068, text=0x7fff93960cc0) at tools.c:545 # 3 0x00000000004906f9 in slapadd (argc=<value optimized out>, argv=0x7fff93960ec8) at ../../../servers/slapd/slapadd.c:350 # 4 0x0000000000416c59 in main (argc=6, argv=0x7fff93960ec8) at ../../../servers/slapd/main.c:391
Thread 1 (Thread 47794824997056 (LWP 9360)): # 0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 # 1 0x00002b7819f28014 in bdb_tool_index_add (op=0x7fff93950310, # txn=<value optimized out>, e=0x2b7b521d2068) at tools.c:475 # 2 0x00002b7819f28970 in hdb_tool_entry_put (be=0x17f39240, # e=0x2b7b521d2068, text=0x7fff93960cc0) at tools.c:545 # 3 0x00000000004906f9 in slapadd (argc=<value optimized out>, argv=0x7fff93960ec8) at ../../../servers/slapd/slapadd.c:350 # 4 0x0000000000416c59 in main (argc=6, argv=0x7fff93960ec8) at ../../../servers/slapd/main.c:391 # 0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc
Zimbra :: the leader in open source messaging and collaboration
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration