On 8/8/23 10:49 PM, Quanah Gibson-Mount wrote:
--On Tuesday, August 8, 2023 7:05 PM -0400 Brendan Kearney bpk678@gmail.com wrote:
list members,
i am implementing slapd 2.6.4, having had 2.4.x running for quite some time. i had 2 primary nodes with n-way multi primary replication working like a charm, and want to have 2.6.4 with 3 primary nodes. i keep running into some odd issues when i bring up all three nodes and they begin replicating between each of the nodes. i have failure messages showing the instances core-dumping and want to find why the core dumps occur.
Aug 08 16:21:20 server1 systemd[1]: slapd.service: Main process exited, code=dumped, status=11/SEGV Aug 08 16:21:20 server1 systemd[1]: slapd.service: Failed with result 'core-dump'.
i have enable "stats" log level, and don't get much more info. i can run one instance and seem to be working without core dumping, but if i turn up one or both of the other slapd instances, all 3 of the instances begin failing. should i bump the log level up or is there diagnostics i can run, to find the reason things are failing? any insight is greatly appreciated. if i need to provide more info, let me know what is needed.
First, upgrade to OpenLDAP 2.6.6. Then if you still get core dumps, install the debug symbol package (I'm assuming you're using a well built openldap such as what Symas or LTB project distributes), install gdb, and get a backtrace from the core file.
--Quanah
Quanah,
i am using prebuilt packages from fedora (38, fully updated), and 2.6.6 is not available yet, so i will have to wait for it. in reviewing logs, i did find some coredump messages. in all, there are 65 messages from 2 of the 3 servers. would these logs be of any value? one such log:
Process 2566 (slapd) of user 55 dumped core.
Module libbasicobjects.so.0 from rpm ding-libs-0.6.2-53.fc38.x86_64 Module libpath_utils.so.1 from rpm ding-libs-0.6.2-53.fc38.x86_64 Module libcollection.so.4 from rpm ding-libs-0.6.2-53.fc38.x86_64 Module libgssrpc.so.4 from rpm krb5-1.21-2.fc38.x86_64 Module libverto.so.1 from rpm libverto-0.3.2-5.fc38.x86_64 Module libini_config.so.5 from rpm ding-libs-0.6.2-53.fc38.x86_64 Module libref_array.so.1 from rpm ding-libs-0.6.2-53.fc38.x86_64 Module proxymech.so from rpm gssproxy-0.9.1-5.fc38.x86_64 Module libpcre2-8.so.0 from rpm pcre2-10.42-1.fc38.1.x86_64 Module libselinux.so.1 from rpm libselinux-3.5-1.fc38.x86_64 Module libkeyutils.so.1 from rpm keyutils-1.6.1-6.fc38.x86_64 Module libkrb5support.so.0 from rpm krb5-1.21-2.fc38.x86_64 Module libcom_err.so.2 from rpm e2fsprogs-1.46.5-4.fc38.x86_64 Module libk5crypto.so.3 from rpm krb5-1.21-2.fc38.x86_64 Module libkrb5.so.3 from rpm krb5-1.21-2.fc38.x86_64 Module libgssapi_krb5.so.2 from rpm krb5-1.21-2.fc38.x86_64 Module libgssapiv2.so from rpm cyrus-sasl-2.1.28-9.fc38.x86_64 Module libsasldb.so from rpm cyrus-sasl-2.1.28-9.fc38.x86_64 Module libanonymous.so from rpm cyrus-sasl-2.1.28-9.fc38.x86_64 Module libz.so.1 from rpm zlib-1.2.13-3.fc38.x86_64 Module libcrypto.so.3 from rpm openssl-3.0.9-2.fc38.x86_64 Module libssl.so.3 from rpm openssl-3.0.9-2.fc38.x86_64 Module libevent-2.1.so.7 from rpm libevent-2.1.12-8.fc38.x86_64 Module libltdl.so.7 from rpm libtool-2.4.7-6.fc38.x86_64 Module libcrypt.so.2 from rpm libxcrypt-4.4.36-1.fc38.x86_64 Module libsasl2.so.3 from rpm cyrus-sasl-2.1.28-9.fc38.x86_64 Module slapd from rpm openldap-2.6.4-1.fc38.x86_64 Stack trace of thread 2571: #0 0x00007f8d4f2bc787 _int_free (libc.so.6 + 0x9a787) #1 0x00007f8d4f2bf1ce free (libc.so.6 + 0x9d1ce) #2 0x000055c177e023f5 entry_clean (slapd + 0x533f5) #3 0x000055c177ec142f mdb_entry_release (slapd + 0x11242f) #4 0x000055c177e7ee0f overlay_entry_release_ov (slapd + 0xcfe0f) #5 0x00007f8d4e897391 syncprov_op_search (syncprov.so.2 + 0xb391) #6 0x000055c177e770da overlay_op_walk (slapd + 0xc80da) #7 0x000055c177e77294 over_op_func (slapd + 0xc8294) #8 0x000055c177e03b61 fe_op_search (slapd + 0x54b61) #9 0x000055c177e061e4 do_search (slapd + 0x571e4) #10 0x000055c177dfdfe5 connection_operation.lto_priv.0 (slapd + 0x4efe5) #11 0x000055c177dfebbb connection_read_thread (slapd + 0x4fbbb) #12 0x00007f8d4f988af6 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x4caf6) #13 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #14 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870)
Stack trace of thread 2566: #0 0x00007f8d4f2ab1d9 __futex_abstimed_wait_common (libc.so.6 + 0x891d9) #1 0x00007f8d4f2b05e3 __pthread_clockjoin_ex (libc.so.6 + 0x8e5e3) #2 0x000055c177dfb3fb slapd_daemon (slapd + 0x4c3fb) #3 0x000055c177ddcc99 main (slapd + 0x2dc99) #4 0x00007f8d4f249b4a __libc_start_call_main (libc.so.6 + 0x27b4a) #5 0x00007f8d4f249c0b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x27c0b) #6 0x000055c177ddd0e5 _start (slapd + 0x2e0e5)
Stack trace of thread 2568: #0 0x00007f8d4f334c72 epoll_wait (libc.so.6 + 0x112c72) #1 0x000055c177dfcc9b slapd_daemon_task (slapd + 0x4dc9b) #2 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #3 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870)
Stack trace of thread 2572: #0 0x00007f8d4f2ab1d9 __futex_abstimed_wait_common (libc.so.6 + 0x891d9) #1 0x00007f8d4f2adb79 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x8bb79) #2 0x00007f8d4f988ba7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x4cba7) #3 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #4 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870)
Stack trace of thread 2573: #0 0x00007f8d4f2ab1d9 __futex_abstimed_wait_common (libc.so.6 + 0x891d9) #1 0x00007f8d4f2adb79 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x8bb79) #2 0x00007f8d4f988ba7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x4cba7) #3 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #4 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870)
Stack trace of thread 2574: #0 0x00007f8d4f2ab1d9 __futex_abstimed_wait_common (libc.so.6 + 0x891d9) #1 0x00007f8d4f2adb79 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x8bb79) #2 0x00007f8d4f988ba7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x4cba7) #3 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #4 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870)
Stack trace of thread 2575: #0 0x00007f8d4f2ab1d9 __futex_abstimed_wait_common (libc.so.6 + 0x891d9) #1 0x00007f8d4f2adb79 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x8bb79) #2 0x00007f8d4f988ba7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x4cba7) #3 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #4 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870)
Stack trace of thread 2576: #0 0x00007f8d4f2ab1d9 __futex_abstimed_wait_common (libc.so.6 + 0x891d9) #1 0x00007f8d4f2adb79 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x8bb79) #2 0x00007f8d4f988ba7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x4cba7) #3 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #4 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870)
Stack trace of thread 2569: #0 0x00007f8d4f2ab430 __GI___lll_lock_wait (libc.so.6 + 0x89430) #1 0x00007f8d4f2b1e21 __pthread_mutex_lock@GLIBC_2.2.5 (libc.so.6 + 0x8fe21) #2 0x00007f8d4f96b17d ldap_connect_to_host (libldap.so.2 + 0x2f17d) #3 0x00007f8d4f9583a6 ldap_int_open_connection (libldap.so.2 + 0x1c3a6) #4 0x00007f8d4f96c8fd ldap_new_connection (libldap.so.2 + 0x308fd) #5 0x00007f8d4f957714 ldap_open_defconn (libldap.so.2 + 0x1b714) #6 0x00007f8d4f96d678 ldap_send_initial_request (libldap.so.2 + 0x31678) #7 0x00007f8d4f95d529 ldap_sasl_bind (libldap.so.2 + 0x21529) #8 0x00007f8d4f95d639 ldap_sasl_bind_s (libldap.so.2 + 0x21639) #9 0x000055c177df4383 slap_client_connect (slapd + 0x45383) #10 0x000055c177e66c14 do_syncrep1 (slapd + 0xb7c14) #11 0x000055c177e69d4c do_syncrepl.lto_priv.0 (slapd + 0xbad4c) #12 0x00007f8d4f988af6 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x4caf6) #13 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #14 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870)
Stack trace of thread 2570: #0 0x00007f8d4f32735d __poll (libc.so.6 + 0x10535d) #1 0x00007f8d4f35bb55 send_dg (libc.so.6 + 0x139b55) #2 0x00007f8d4f35c868 __res_context_send (libc.so.6 + 0x13a868) #3 0x00007f8d4f359b4a __res_context_query (libc.so.6 + 0x137b4a) #4 0x00007f8d4f35a97f __res_context_search (libc.so.6 + 0x13897f) #5 0x00007f8d4f352e37 gethostbyname3_context (libc.so.6 + 0x130e37) #6 0x00007f8d4f35356e _nss_dns_gethostbyname2_r (libc.so.6 + 0x13156e) #7 0x00007f8d4f345913 gethostbyname2_r@@GLIBC_2.2.5 (libc.so.6 + 0x123913) #8 0x00007f8d4f31c0af getaddrinfo (libc.so.6 + 0xfa0af) #9 0x00007f8d4f96b196 ldap_connect_to_host (libldap.so.2 + 0x2f196) #10 0x00007f8d4f9583a6 ldap_int_open_connection (libldap.so.2 + 0x1c3a6) #11 0x00007f8d4f96c8fd ldap_new_connection (libldap.so.2 + 0x308fd) #12 0x00007f8d4f957714 ldap_open_defconn (libldap.so.2 + 0x1b714) #13 0x00007f8d4f96d678 ldap_send_initial_request (libldap.so.2 + 0x31678) #14 0x00007f8d4f95d529 ldap_sasl_bind (libldap.so.2 + 0x21529) #15 0x00007f8d4f95d639 ldap_sasl_bind_s (libldap.so.2 + 0x21639) #16 0x000055c177df4383 slap_client_connect (slapd + 0x45383) #17 0x000055c177e66c14 do_syncrep1 (slapd + 0xb7c14) #18 0x000055c177e69d4c do_syncrepl.lto_priv.0 (slapd + 0xbad4c) #19 0x00007f8d4f988af6 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x4caf6) #20 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #21 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870) ELF object binary architecture: AMD x86-64
thanks,
brendan