Full_Name: gerard gobillard Version: 2.4.30 OS: RHEL 5.2 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (92.139.226.97)
Hello all,
Summary : on an Intel basic box (dual core) we create search base traffic with slamd in asynchronous mode (30 threads). When, after 1 or 2 seconds, Ctrl-C is sent to slamd (SIGINT), slapd gets stuck with main thread in epoll_wait() and all other threads in pthread_cond_wait() (in send_ldap_ber()). It may be necessary to repeat a few times before slapd gets stuck
Notes : no change to slapd.conf except RootDN.
We build OpenLDAP as follows :
CPPFLAGS="-I/usr/local/BerkeleyDB.4.7/include -g"
export CPPFLAGS
LDFLAGS="-L/usr/local/lib -L/usr/local/BerkeleyDB.4.7/lib -R/usr/local/BerkeleyDB.4.7/lib"
export LDFLAGS
LD_LIBRARY_PATH=/usr/local/BerkeleyDB.4.7/lib
./configure --prefix=/usr/slapd.2.4.23.Dbg --enable-spasswd=yes --enable-crypt=yes --disable-syslog --enable-debug --enable-ppolicy
make depend
make
make install
gstack:
[root@lalxsdmcomp db-5.2.36]# gstack 19444 Thread 18 (Thread 1099237696 (LWP 19445)): #0 0x0000003cf08d21d8 in epoll_wait () from /lib64/libc.so.6 #1 0x000000000042bf11 in slapd_daemon_task () #2 0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0 #3 0x0000003cf08d1ded in clone () from /lib64/libc.so.6 Thread 17 (Thread 1112529216 (LWP 19463)): #0 0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x0000000000444428 in send_ldap_ber () #2 0x00000000004477da in slap_send_search_entry () #3 0x00000000004cdbe6 in bdb_search () #4 0x00000000004334e4 in fe_op_search () #5 0x0000000000432d9b in do_search () #6 0x000000000042fec7 in connection_operation () #7 0x000000000056562c in ldap_int_thread_pool_wrapper () #8 0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003cf08d1ded in clone () from /lib64/libc.so.6 Thread 16 (Thread 1120921920 (LWP 19464)): #0 0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x0000000000444428 in send_ldap_ber () #2 0x00000000004477da in slap_send_search_entry () #3 0x00000000004cdbe6 in bdb_search () #4 0x00000000004334e4 in fe_op_search () #5 0x0000000000432d9b in do_search () #6 0x000000000042fec7 in connection_operation () #7 0x000000000056562c in ldap_int_thread_pool_wrapper () #8 0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003cf08d1ded in clone () from /lib64/libc.so.6 Thread 15 (Thread 1129314624 (LWP 19465)): #0 0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x0000000000444428 in send_ldap_ber () #2 0x00000000004477da in slap_send_search_entry () #3 0x00000000004cdbe6 in bdb_search () #4 0x00000000004334e4 in fe_op_search () #5 0x0000000000432d9b in do_search () #6 0x000000000042fec7 in connection_operation () #7 0x000000000056562c in ldap_int_thread_pool_wrapper () #8 0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003cf08d1ded in clone () from /lib64/libc.so.6 Thread 14 (Thread 1137707328 (LWP 19466)): #0 0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x0000000000444428 in send_ldap_ber () #2 0x00000000004477da in slap_send_search_entry () #3 0x00000000004cdbe6 in bdb_search () #4 0x00000000004334e4 in fe_op_search () #5 0x0000000000432d9b in do_search () #6 0x000000000042fec7 in connection_operation () #7 0x000000000056562c in ldap_int_thread_pool_wrapper () #8 0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003cf08d1ded in clone () from /lib64/libc.so.6 Thread 13 (Thread 1146100032 (LWP 19467)): #0 0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x0000000000444428 in send_ldap_ber () #2 0x00000000004477da in slap_send_search_entry () #3 0x00000000004cdbe6 in bdb_search () #4 0x00000000004334e4 in fe_op_search () #5 0x0000000000432d9b in do_search () #6 0x000000000042fec7 in connection_operation () #7 0x000000000056562c in ldap_int_thread_pool_wrapper () #8 0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003cf08d1ded in clone () from /lib64/libc.so.6 Thread 12 (Thread 1154492736 (LWP 19468)): #0 0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x0000000000444428 in send_ldap_ber () #2 0x00000000004477da in slap_send_search_entry () #3 0x00000000004cdbe6 in bdb_search () #4 0x00000000004334e4 in fe_op_search () #5 0x0000000000432d9b in do_search () #6 0x000000000042fec7 in connection_operation () #7 0x000000000056562c in ldap_int_thread_pool_wrapper () #8 0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003cf08d1ded in clone () from /lib64/libc.so.6 Thread 11 (Thread 1162885440 (LWP 19469)): #0 0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00000000004446a0 in send_ldap_ber () #2 0x00000000004477da in slap_send_search_entry () #3 0x00000000004cdbe6 in bdb_search () #4 0x00000000004334e4 in fe_op_search () #5 0x0000000000432d9b in do_search () #6 0x000000000042fec7 in connection_operation () #7 0x000000000056562c in ldap_int_thread_pool_wrapper () #8 0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003cf08d1ded in clone () from /lib64/libc.so.6 Thread 10 (Thread 1171278144 (LWP 19470)): #0 0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x0000000000444428 in send_ldap_ber () #2 0x00000000004477da in slap_send_search_entry () #3 0x00000000004cdbe6 in bdb_search () #4 0x00000000004334e4 in fe_op_search () #5 0x0000000000432d9b in do_search () #6 0x000000000042fec7 in connection_operation () #7 0x000000000056562c in ldap_int_thread_pool_wrapper () #8 0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003cf08d1ded in clone () from /lib64/libc.so.6 Thread 9 (Thread 1087490368 (LWP 19471)): #0 0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x0000000000444428 in send_ldap_ber () #2 0x00000000004477da in slap_send_search_entry () #3 0x00000000004cdbe6 in bdb_search () #4 0x00000000004334e4 in fe_op_search () #5 0x0000000000432d9b in do_search () #6 0x000000000042fec7 in connection_operation () #7 0x000000000056562c in ldap_int_thread_pool_wrapper () #8 0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003cf08d1ded in clone () from /lib64/libc.so.6 Thread 8 (Thread 1179670848 (LWP 19472)): #0 0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x0000000000444428 in send_ldap_ber () #2 0x00000000004477da in slap_send_search_entry () #3 0x00000000004cdbe6 in bdb_search () #4 0x00000000004334e4 in fe_op_search () #5 0x0000000000432d9b in do_search () #6 0x000000000042fec7 in connection_operation () #7 0x000000000056562c in ldap_int_thread_pool_wrapper () #8 0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003cf08d1ded in clone () from /lib64/libc.so.6 Thread 7 (Thread 1188063552 (LWP 19473)): #0 0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x0000000000444428 in send_ldap_ber () #2 0x00000000004477da in slap_send_search_entry () #3 0x00000000004cdbe6 in bdb_search () #4 0x00000000004334e4 in fe_op_search () #5 0x0000000000432d9b in do_search () #6 0x000000000042fec7 in connection_operation () #7 0x000000000056562c in ldap_int_thread_pool_wrapper () #8 0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003cf08d1ded in clone () from /lib64/libc.so.6 Thread 6 (Thread 1196456256 (LWP 19474)): #0 0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00000000004446a0 in send_ldap_ber () #2 0x00000000004477da in slap_send_search_entry () #3 0x00000000004cdbe6 in bdb_search () #4 0x00000000004334e4 in fe_op_search () #5 0x0000000000432d9b in do_search () #6 0x000000000042fec7 in connection_operation () #7 0x000000000056562c in ldap_int_thread_pool_wrapper () #8 0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003cf08d1ded in clone () from /lib64/libc.so.6 Thread 5 (Thread 1204848960 (LWP 19475)): #0 0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x0000000000444428 in send_ldap_ber () #2 0x00000000004477da in slap_send_search_entry () #3 0x00000000004cdbe6 in bdb_search () #4 0x00000000004334e4 in fe_op_search () #5 0x0000000000432d9b in do_search () #6 0x000000000042fec7 in connection_operation () #7 0x000000000056562c in ldap_int_thread_pool_wrapper () #8 0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003cf08d1ded in clone () from /lib64/libc.so.6 Thread 4 (Thread 1213241664 (LWP 19476)): #0 0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x0000000000444428 in send_ldap_ber () #2 0x00000000004477da in slap_send_search_entry () #3 0x00000000004cdbe6 in bdb_search () #4 0x00000000004334e4 in fe_op_search () #5 0x0000000000432d9b in do_search () #6 0x000000000042fec7 in connection_operation () #7 0x000000000056562c in ldap_int_thread_pool_wrapper () #8 0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003cf08d1ded in clone () from /lib64/libc.so.6 Thread 3 (Thread 1221634368 (LWP 19477)): #0 0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x0000000000444428 in send_ldap_ber () #2 0x00000000004477da in slap_send_search_entry () #3 0x00000000004cdbe6 in bdb_search () #4 0x00000000004334e4 in fe_op_search () #5 0x0000000000432d9b in do_search () #6 0x000000000042fec7 in connection_operation () #7 0x000000000056562c in ldap_int_thread_pool_wrapper () #8 0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003cf08d1ded in clone () from /lib64/libc.so.6 Thread 2 (Thread 1230027072 (LWP 19478)): #0 0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 () #1 0x00000000004446a0 in send_ldap_ber () #2 0x00000000004477da in slap_send_search_entry () #3 0x00000000004cdbe6 in bdb_search () #4 0x00000000004334e4 in fe_op_search () #5 0x0000000000432d9b in do_search () #6 0x000000000042fec7 in connection_operation () #7 0x000000000056562c in ldap_int_thread_pool_wrapper () #8 0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003cf08d1ded in clone () from /lib64/libc.so.6 Thread 1 (Thread 47813429243440 (LWP 19444)): #0 0x0000003cf14075c5 in pthread_join () from /lib64/libpthread.so.0 #1 0x000000000042cf32 in slapd_daemon () #2 0x000000000040d2a0 in main () #0 0x0000003cf14075c5 in pthread_join () from /lib64/libpthread.so.0
netstat:
[root@lalxsdmcomp db-5.2.36]# netstat -pan | grep :389 tcp 0 0 0.0.0.0:389 0.0.0.0:* LISTEN 19444/slapd tcp 28071 0 172.25.198.71:389 172.25.195.198:38743 CLOSE_WAIT 19444/slapd tcp 14026 0 172.25.198.71:389 172.25.195.198:38749 CLOSE_WAIT 19444/slapd tcp 57221 0 172.25.198.71:389 172.25.195.198:38751 CLOSE_WAIT 19444/slapd tcp 31304 0 172.25.198.71:389 172.25.195.198:38750 CLOSE_WAIT 19444/slapd tcp 38300 0 172.25.198.71:389 172.25.195.198:38745 CLOSE_WAIT 19444/slapd tcp 83048 0 172.25.198.71:389 172.25.195.198:38744 ESTABLISHED 19444/slapd tcp 3433 0 172.25.198.71:389 172.25.195.198:38747 CLOSE_WAIT 19444/slapd tcp 11058 0 172.25.198.71:389 172.25.195.198:38746 CLOSE_WAIT 19444/slapd tcp 47204 0 172.25.198.71:389 172.25.195.198:38757 CLOSE_WAIT 19444/slapd tcp 36392 0 172.25.198.71:389 172.25.195.198:38756 CLOSE_WAIT 19444/slapd tcp 2341 0 172.25.198.71:389 172.25.195.198:38759 CLOSE_WAIT 19444/slapd tcp 73183 0 172.25.198.71:389 172.25.195.198:38758 ESTABLISHED 19444/slapd tcp 71796 0 172.25.198.71:389 172.25.195.198:38753 CLOSE_WAIT 19444/slapd tcp 78209 0 172.25.198.71:389 172.25.195.198:38752 CLOSE_WAIT 19444/slapd tcp 35915 0 172.25.198.71:389 172.25.195.198:38755 CLOSE_WAIT 19444/slapd tcp 12701 0 172.25.198.71:389 172.25.195.198:38754 CLOSE_WAIT 19444/slapd tcp 20333 0 172.25.198.71:389 172.25.195.198:38765 CLOSE_WAIT 19444/slapd tcp 74817 0 172.25.198.71:389 172.25.195.198:38764 CLOSE_WAIT 19444/slapd tcp 73757 0 172.25.198.71:389 172.25.195.198:38767 CLOSE_WAIT 19444/slapd tcp 2705 0 172.25.198.71:389 172.25.195.198:38766 CLOSE_WAIT 19444/slapd tcp 36180 0 172.25.198.71:389 172.25.195.198:38761 CLOSE_WAIT 19444/slapd tcp 75029 0 172.25.198.71:389 172.25.195.198:38760 CLOSE_WAIT 19444/slapd tcp 49112 0 172.25.198.71:389 172.25.195.198:38763 CLOSE_WAIT 19444/slapd tcp 8726 0 172.25.198.71:389 172.25.195.198:38762 CLOSE_WAIT 19444/slapd tcp 44 0 172.25.198.71:389 172.25.195.198:38772 ESTABLISHED - tcp 4005 0 172.25.198.71:389 172.25.195.198:38769 CLOSE_WAIT 19444/slapd tcp 17789 0 172.25.198.71:389 172.25.195.198:38768 CLOSE_WAIT 19444/slapd tcp 9945 0 172.25.198.71:389 172.25.195.198:38770 CLOSE_WAIT 19444/slapd tcp 0 0 :::389 :::* LISTEN 19444/slapd
Provides attachments at https://online-storage.yuuwaa.com/s/HkSdr/94bc6ba666
slapd_stuck.log - openldap logs populBlits.ldif - ldif file to get a blitz population (that's what we used but it works with any other data) testWriteTO.pl - a Perl script to loop thru client disconnection