list members,
i am implementing slapd 2.6.4, having had 2.4.x running for quite some time. i had 2 primary nodes with n-way multi primary replication working like a charm, and want to have 2.6.4 with 3 primary nodes. i keep running into some odd issues when i bring up all three nodes and they begin replicating between each of the nodes. i have failure messages showing the instances core-dumping and want to find why the core dumps occur.
Aug 08 16:21:20 server1 systemd[1]: slapd.service: Main process exited, code=dumped, status=11/SEGV Aug 08 16:21:20 server1 systemd[1]: slapd.service: Failed with result 'core-dump'.
i have enable "stats" log level, and don't get much more info. i can run one instance and seem to be working without core dumping, but if i turn up one or both of the other slapd instances, all 3 of the instances begin failing. should i bump the log level up or is there diagnostics i can run, to find the reason things are failing? any insight is greatly appreciated. if i need to provide more info, let me know what is needed.
thanks in advance,
brendan
--On Tuesday, August 8, 2023 7:05 PM -0400 Brendan Kearney bpk678@gmail.com wrote:
list members,
i am implementing slapd 2.6.4, having had 2.4.x running for quite some time. i had 2 primary nodes with n-way multi primary replication working like a charm, and want to have 2.6.4 with 3 primary nodes. i keep running into some odd issues when i bring up all three nodes and they begin replicating between each of the nodes. i have failure messages showing the instances core-dumping and want to find why the core dumps occur.
Aug 08 16:21:20 server1 systemd[1]: slapd.service: Main process exited, code=dumped, status=11/SEGV Aug 08 16:21:20 server1 systemd[1]: slapd.service: Failed with result 'core-dump'.
i have enable "stats" log level, and don't get much more info. i can run one instance and seem to be working without core dumping, but if i turn up one or both of the other slapd instances, all 3 of the instances begin failing. should i bump the log level up or is there diagnostics i can run, to find the reason things are failing? any insight is greatly appreciated. if i need to provide more info, let me know what is needed.
First, upgrade to OpenLDAP 2.6.6. Then if you still get core dumps, install the debug symbol package (I'm assuming you're using a well built openldap such as what Symas or LTB project distributes), install gdb, and get a backtrace from the core file.
--Quanah
On 8/8/23 10:49 PM, Quanah Gibson-Mount wrote:
--On Tuesday, August 8, 2023 7:05 PM -0400 Brendan Kearney bpk678@gmail.com wrote:
list members,
i am implementing slapd 2.6.4, having had 2.4.x running for quite some time. i had 2 primary nodes with n-way multi primary replication working like a charm, and want to have 2.6.4 with 3 primary nodes. i keep running into some odd issues when i bring up all three nodes and they begin replicating between each of the nodes. i have failure messages showing the instances core-dumping and want to find why the core dumps occur.
Aug 08 16:21:20 server1 systemd[1]: slapd.service: Main process exited, code=dumped, status=11/SEGV Aug 08 16:21:20 server1 systemd[1]: slapd.service: Failed with result 'core-dump'.
i have enable "stats" log level, and don't get much more info. i can run one instance and seem to be working without core dumping, but if i turn up one or both of the other slapd instances, all 3 of the instances begin failing. should i bump the log level up or is there diagnostics i can run, to find the reason things are failing? any insight is greatly appreciated. if i need to provide more info, let me know what is needed.
First, upgrade to OpenLDAP 2.6.6. Then if you still get core dumps, install the debug symbol package (I'm assuming you're using a well built openldap such as what Symas or LTB project distributes), install gdb, and get a backtrace from the core file.
--Quanah
Quanah,
i am using prebuilt packages from fedora (38, fully updated), and 2.6.6 is not available yet, so i will have to wait for it. in reviewing logs, i did find some coredump messages. in all, there are 65 messages from 2 of the 3 servers. would these logs be of any value? one such log:
Process 2566 (slapd) of user 55 dumped core.
Module libbasicobjects.so.0 from rpm ding-libs-0.6.2-53.fc38.x86_64 Module libpath_utils.so.1 from rpm ding-libs-0.6.2-53.fc38.x86_64 Module libcollection.so.4 from rpm ding-libs-0.6.2-53.fc38.x86_64 Module libgssrpc.so.4 from rpm krb5-1.21-2.fc38.x86_64 Module libverto.so.1 from rpm libverto-0.3.2-5.fc38.x86_64 Module libini_config.so.5 from rpm ding-libs-0.6.2-53.fc38.x86_64 Module libref_array.so.1 from rpm ding-libs-0.6.2-53.fc38.x86_64 Module proxymech.so from rpm gssproxy-0.9.1-5.fc38.x86_64 Module libpcre2-8.so.0 from rpm pcre2-10.42-1.fc38.1.x86_64 Module libselinux.so.1 from rpm libselinux-3.5-1.fc38.x86_64 Module libkeyutils.so.1 from rpm keyutils-1.6.1-6.fc38.x86_64 Module libkrb5support.so.0 from rpm krb5-1.21-2.fc38.x86_64 Module libcom_err.so.2 from rpm e2fsprogs-1.46.5-4.fc38.x86_64 Module libk5crypto.so.3 from rpm krb5-1.21-2.fc38.x86_64 Module libkrb5.so.3 from rpm krb5-1.21-2.fc38.x86_64 Module libgssapi_krb5.so.2 from rpm krb5-1.21-2.fc38.x86_64 Module libgssapiv2.so from rpm cyrus-sasl-2.1.28-9.fc38.x86_64 Module libsasldb.so from rpm cyrus-sasl-2.1.28-9.fc38.x86_64 Module libanonymous.so from rpm cyrus-sasl-2.1.28-9.fc38.x86_64 Module libz.so.1 from rpm zlib-1.2.13-3.fc38.x86_64 Module libcrypto.so.3 from rpm openssl-3.0.9-2.fc38.x86_64 Module libssl.so.3 from rpm openssl-3.0.9-2.fc38.x86_64 Module libevent-2.1.so.7 from rpm libevent-2.1.12-8.fc38.x86_64 Module libltdl.so.7 from rpm libtool-2.4.7-6.fc38.x86_64 Module libcrypt.so.2 from rpm libxcrypt-4.4.36-1.fc38.x86_64 Module libsasl2.so.3 from rpm cyrus-sasl-2.1.28-9.fc38.x86_64 Module slapd from rpm openldap-2.6.4-1.fc38.x86_64 Stack trace of thread 2571: #0 0x00007f8d4f2bc787 _int_free (libc.so.6 + 0x9a787) #1 0x00007f8d4f2bf1ce free (libc.so.6 + 0x9d1ce) #2 0x000055c177e023f5 entry_clean (slapd + 0x533f5) #3 0x000055c177ec142f mdb_entry_release (slapd + 0x11242f) #4 0x000055c177e7ee0f overlay_entry_release_ov (slapd + 0xcfe0f) #5 0x00007f8d4e897391 syncprov_op_search (syncprov.so.2 + 0xb391) #6 0x000055c177e770da overlay_op_walk (slapd + 0xc80da) #7 0x000055c177e77294 over_op_func (slapd + 0xc8294) #8 0x000055c177e03b61 fe_op_search (slapd + 0x54b61) #9 0x000055c177e061e4 do_search (slapd + 0x571e4) #10 0x000055c177dfdfe5 connection_operation.lto_priv.0 (slapd + 0x4efe5) #11 0x000055c177dfebbb connection_read_thread (slapd + 0x4fbbb) #12 0x00007f8d4f988af6 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x4caf6) #13 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #14 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870)
Stack trace of thread 2566: #0 0x00007f8d4f2ab1d9 __futex_abstimed_wait_common (libc.so.6 + 0x891d9) #1 0x00007f8d4f2b05e3 __pthread_clockjoin_ex (libc.so.6 + 0x8e5e3) #2 0x000055c177dfb3fb slapd_daemon (slapd + 0x4c3fb) #3 0x000055c177ddcc99 main (slapd + 0x2dc99) #4 0x00007f8d4f249b4a __libc_start_call_main (libc.so.6 + 0x27b4a) #5 0x00007f8d4f249c0b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x27c0b) #6 0x000055c177ddd0e5 _start (slapd + 0x2e0e5)
Stack trace of thread 2568: #0 0x00007f8d4f334c72 epoll_wait (libc.so.6 + 0x112c72) #1 0x000055c177dfcc9b slapd_daemon_task (slapd + 0x4dc9b) #2 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #3 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870)
Stack trace of thread 2572: #0 0x00007f8d4f2ab1d9 __futex_abstimed_wait_common (libc.so.6 + 0x891d9) #1 0x00007f8d4f2adb79 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x8bb79) #2 0x00007f8d4f988ba7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x4cba7) #3 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #4 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870)
Stack trace of thread 2573: #0 0x00007f8d4f2ab1d9 __futex_abstimed_wait_common (libc.so.6 + 0x891d9) #1 0x00007f8d4f2adb79 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x8bb79) #2 0x00007f8d4f988ba7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x4cba7) #3 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #4 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870)
Stack trace of thread 2574: #0 0x00007f8d4f2ab1d9 __futex_abstimed_wait_common (libc.so.6 + 0x891d9) #1 0x00007f8d4f2adb79 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x8bb79) #2 0x00007f8d4f988ba7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x4cba7) #3 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #4 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870)
Stack trace of thread 2575: #0 0x00007f8d4f2ab1d9 __futex_abstimed_wait_common (libc.so.6 + 0x891d9) #1 0x00007f8d4f2adb79 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x8bb79) #2 0x00007f8d4f988ba7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x4cba7) #3 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #4 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870)
Stack trace of thread 2576: #0 0x00007f8d4f2ab1d9 __futex_abstimed_wait_common (libc.so.6 + 0x891d9) #1 0x00007f8d4f2adb79 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x8bb79) #2 0x00007f8d4f988ba7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x4cba7) #3 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #4 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870)
Stack trace of thread 2569: #0 0x00007f8d4f2ab430 __GI___lll_lock_wait (libc.so.6 + 0x89430) #1 0x00007f8d4f2b1e21 __pthread_mutex_lock@GLIBC_2.2.5 (libc.so.6 + 0x8fe21) #2 0x00007f8d4f96b17d ldap_connect_to_host (libldap.so.2 + 0x2f17d) #3 0x00007f8d4f9583a6 ldap_int_open_connection (libldap.so.2 + 0x1c3a6) #4 0x00007f8d4f96c8fd ldap_new_connection (libldap.so.2 + 0x308fd) #5 0x00007f8d4f957714 ldap_open_defconn (libldap.so.2 + 0x1b714) #6 0x00007f8d4f96d678 ldap_send_initial_request (libldap.so.2 + 0x31678) #7 0x00007f8d4f95d529 ldap_sasl_bind (libldap.so.2 + 0x21529) #8 0x00007f8d4f95d639 ldap_sasl_bind_s (libldap.so.2 + 0x21639) #9 0x000055c177df4383 slap_client_connect (slapd + 0x45383) #10 0x000055c177e66c14 do_syncrep1 (slapd + 0xb7c14) #11 0x000055c177e69d4c do_syncrepl.lto_priv.0 (slapd + 0xbad4c) #12 0x00007f8d4f988af6 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x4caf6) #13 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #14 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870)
Stack trace of thread 2570: #0 0x00007f8d4f32735d __poll (libc.so.6 + 0x10535d) #1 0x00007f8d4f35bb55 send_dg (libc.so.6 + 0x139b55) #2 0x00007f8d4f35c868 __res_context_send (libc.so.6 + 0x13a868) #3 0x00007f8d4f359b4a __res_context_query (libc.so.6 + 0x137b4a) #4 0x00007f8d4f35a97f __res_context_search (libc.so.6 + 0x13897f) #5 0x00007f8d4f352e37 gethostbyname3_context (libc.so.6 + 0x130e37) #6 0x00007f8d4f35356e _nss_dns_gethostbyname2_r (libc.so.6 + 0x13156e) #7 0x00007f8d4f345913 gethostbyname2_r@@GLIBC_2.2.5 (libc.so.6 + 0x123913) #8 0x00007f8d4f31c0af getaddrinfo (libc.so.6 + 0xfa0af) #9 0x00007f8d4f96b196 ldap_connect_to_host (libldap.so.2 + 0x2f196) #10 0x00007f8d4f9583a6 ldap_int_open_connection (libldap.so.2 + 0x1c3a6) #11 0x00007f8d4f96c8fd ldap_new_connection (libldap.so.2 + 0x308fd) #12 0x00007f8d4f957714 ldap_open_defconn (libldap.so.2 + 0x1b714) #13 0x00007f8d4f96d678 ldap_send_initial_request (libldap.so.2 + 0x31678) #14 0x00007f8d4f95d529 ldap_sasl_bind (libldap.so.2 + 0x21529) #15 0x00007f8d4f95d639 ldap_sasl_bind_s (libldap.so.2 + 0x21639) #16 0x000055c177df4383 slap_client_connect (slapd + 0x45383) #17 0x000055c177e66c14 do_syncrep1 (slapd + 0xb7c14) #18 0x000055c177e69d4c do_syncrepl.lto_priv.0 (slapd + 0xbad4c) #19 0x00007f8d4f988af6 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x4caf6) #20 0x00007f8d4f2ae907 start_thread (libc.so.6 + 0x8c907) #21 0x00007f8d4f334870 __clone3 (libc.so.6 + 0x112870) ELF object binary architecture: AMD x86-64
thanks,
brendan
--On Wednesday, August 9, 2023 9:19 AM -0400 Brendan Kearney bpk678@gmail.com wrote:
Quanah,
i am using prebuilt packages from fedora (38, fully updated), and 2.6.6 is not available yet, so i will have to wait for it.
I didn't see anything in the stack trace indicating a core dump.
One option would be to start slapd on one of the systems from within gdb, and get a backtrace that way. I.e., something like:
gdb /path/to/slapd set args <args for slapd to execute with> run
For example, I get the args my slapd process is running with from the process stack:
/opt/symas/lib/slapd -d 0 -h ldap:/// ldaps:/// ldapi:///
Then stop slapd:
systemctl stop slapd
Start gdb:
gdb /opt/symas/lib/slapd
(gdb) set args -d 0 -h 'ldap:/// ldaps:/// ldapi:///' (gdb) run
Then wait for it to hit the issue, and then:
(gdb) thr apply all bt full
You may also want to set it to log to a file, so something like: gdb /path/to/slapd (gdb) set pagination off (gdb) set logging file /somewhere/gdb.txt (gdb) set logging on (gdb) set args ... (gdb) run (gdb) thr apply all bt full (gdb) set logging off (gdb) quit
Regards, Quanah
On 8/9/23 11:10 AM, Quanah Gibson-Mount wrote:
--On Wednesday, August 9, 2023 9:19 AM -0400 Brendan Kearney bpk678@gmail.com wrote:
Quanah,
i am using prebuilt packages from fedora (38, fully updated), and 2.6.6 is not available yet, so i will have to wait for it.
I didn't see anything in the stack trace indicating a core dump.
One option would be to start slapd on one of the systems from within gdb, and get a backtrace that way. I.e., something like:
gdb /path/to/slapd set args <args for slapd to execute with> run
For example, I get the args my slapd process is running with from the process stack:
/opt/symas/lib/slapd -d 0 -h ldap:/// ldaps:/// ldapi:///
Then stop slapd:
systemctl stop slapd
Start gdb:
gdb /opt/symas/lib/slapd
(gdb) set args -d 0 -h 'ldap:/// ldaps:/// ldapi:///' (gdb) run
Then wait for it to hit the issue, and then:
(gdb) thr apply all bt full
You may also want to set it to log to a file, so something like: gdb /path/to/slapd (gdb) set pagination off (gdb) set logging file /somewhere/gdb.txt (gdb) set logging on (gdb) set args ... (gdb) run (gdb) thr apply all bt full (gdb) set logging off (gdb) quit
Regards, Quanah
it took me a few tries to get things working, but i seem to have produced some output.
Starting program: /usr/sbin/slapd -u ldap -h 'ldap://192.168.88.1' -4 -d 0
This GDB supports auto-downloading debuginfo from the following URLs: https://debuginfod.fedoraproject.org/ Debuginfod has been enabled. To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7ffff32646c0 (LWP 22269)] [Thread 0x7ffff32646c0 (LWP 22269) exited] [New Thread 0x7ffff32646c0 (LWP 22270)] [New Thread 0x7fff527ff6c0 (LWP 22271)] [New Thread 0x7fff51ffe6c0 (LWP 22272)] [New Thread 0x7fff517fd6c0 (LWP 22273)] [New Thread 0x7fff50ffc6c0 (LWP 22274)]
Thread 5 "slapd" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fff51ffe6c0 (LWP 22272)] 0x00007ffff78bc787 in _int_free (av=0x0, p=p@entry=0x7ffff6091980 <syncprov_setup_accesslog+752>, have_lock=have_lock@entry=0) at malloc.c:4562 4562 __libc_lock_lock (av->mutex); Starting program: /usr/sbin/slapd -u ldap -h 'ldap://192.168.88.1' -4 -d 0 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7ffff32646c0 (LWP 22277)] [Thread 0x7ffff32646c0 (LWP 22277) exited] [New Thread 0x7ffff32646c0 (LWP 22278)] [New Thread 0x7fff527ff6c0 (LWP 22279)] [New Thread 0x7fff51ffe6c0 (LWP 22280)] [New Thread 0x7fff517fd6c0 (LWP 22281)] [New Thread 0x7fff50ffc6c0 (LWP 22282)]
Thread 4 "slapd" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fff527ff6c0 (LWP 22279)] 0x00007ffff78bc787 in _int_free (av=0x0, p=p@entry=0x7ffff6091980 <syncprov_setup_accesslog+752>, have_lock=have_lock@entry=0) at malloc.c:4562 4562 __libc_lock_lock (av->mutex); Starting program: /usr/sbin/slapd -u ldap -h 'ldap://192.168.88.1' -4 -d 0 Downloading separate debug info for system-supplied DSO at 0x7ffff7fc8000 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7ffff32646c0 (LWP 22290)] [Thread 0x7ffff32646c0 (LWP 22290) exited] [New Thread 0x7ffff32646c0 (LWP 22291)] [New Thread 0x7fff527ff6c0 (LWP 22292)] [New Thread 0x7fff51ffe6c0 (LWP 22293)] [New Thread 0x7fff517fd6c0 (LWP 22294)] [New Thread 0x7fff50ffc6c0 (LWP 22295)] Downloading separate debug info for /usr/lib64/krb5/plugins/authdata/sssd_pac_plugin.so
Thread 7 "slapd" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fff50ffc6c0 (LWP 22295)] 0x00007ffff78bc787 in _int_free (av=0x0, p=p@entry=0x7ffff6091980 <syncprov_setup_accesslog+752>, have_lock=have_lock@entry=0) at malloc.c:4562 4562 __libc_lock_lock (av->mutex);
Thread 7 (Thread 0x7fff50ffc6c0 (LWP 22295) "slapd"): #0 0x00007ffff78bc787 in _int_free (av=0x0, p=p@entry=0x7ffff6091980 <syncprov_setup_accesslog+752>, have_lock=have_lock@entry=0) at malloc.c:4562 __futex = 0x0 size = 10405003989086109824 fb = <optimized out> nextchunk = <optimized out> nextsize = <optimized out> nextinuse = <optimized out> prevsize = <optimized out> bck = <optimized out> fwd = <optimized out> __PRETTY_FUNCTION__ = "_int_free" #1 0x00007ffff78bf1ce in __GI___libc_free (mem=0x7ffff6091990 <syncprov_db_init>) at malloc.c:3367 ar_ptr = <optimized out> p = 0x7ffff6091980 <syncprov_setup_accesslog+752> err = 11 #2 0x00005555555a73f5 in entry_clean (e=0x555555932da0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/entry.c:465 __PRETTY_FUNCTION__ = "entry_clean" #3 0x000055555566642f in entry_free (e=0x555555932da0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/entry.c:481 No locals. #4 mdb_entry_return (e=0x555555932da0, op=0x7fff40106490) at back-mdb/id2entry.c:543 No locals. #5 mdb_entry_release (op=0x7fff40106490, e=0x555555932da0, rw=<optimized out>) at back-mdb/id2entry.c:592 mdb = <optimized out> moi = <optimized out> release = <optimized out> #6 0x0000555555623e0f in overlay_entry_release_ov (op=op@entry=0x7fff40106490, e=0x555555932da0, rw=rw@entry=0, on=0x0, on@entry=0x555555932da0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/backover.c:452 oi = 0x55555577ab10 be = 0x7fff50ffa5d0 db = {bd_info = 0x0, bd_self = 0x0, be_ctrls = '\000' <repeats 16 times>, "(f\020@\377\177\000\000c\000\000\000\000\000\000", <incomplete sequence \363>, be_flags = 3654763765825540, be_restrictops = 0, be_requires = 45, be_ssf_set = {sss_ssf = 1435701312, sss_transport = 21845, sss_tls = 14, sss_sasl = 0, sss_update_ssf = 1073767256, sss_update_transport = 32767, sss_update_tls = 14, sss_update_sasl = 0, sss_simple_bind = 1073767312}, be_suffix = 0x0, be_nsuffix = 0xffffffff00000001, be_schemadn = {bv_len = 0, bv_val = 0x1 <error: Cannot access memory at address 0x1>}, be_schemandn = {bv_len = 93824993966880, bv_val = 0x7ffff609a000 <generic_filter> "\207"}, be_rootdn = {bv_len = 15, bv_val = 0x7ffff609636d "(objectclass=*)"}, be_rootndn = {bv_len = 0, bv_val = 0x0}, be_rootpw = {bv_len = 0, bv_val = 0x0}, be_max_deref_depth = 0, be_def_limit = {lms_t_soft = 0, lms_t_hard = 0, lms_s_soft = 0, lms_s_hard = 0, lms_s_unchecked = 0, lms_s_pr = 0, lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits = 0x0, be_acl = 0x3000000, be_dfltaccess = ACL_NONE, be_extra_anlist = 0x7fff40006858, be_lastbind_precision = 1437273824, be_update_ndn = {bv_len = 140734267132816, bv_val = 0x3 <error: Cannot access memory at address 0x3>}, be_update_refs = 0x3, be_pcsn_st = {be_pcsn_list = {tqh_first = 0x7fff50ffa0c0, tqh_last = 0x7ffff7f49f58 <ber_memalloc_x+40>}, be_pcsn_mutex = {__data = {__lock = 1437273776, __count = 21845, __owner = 1073768536, __nusers = 32767, __kind = 1358930160, __spins = 32767, __elision = 0, __list = {__prev = 0x7ffff7f4bfb1 <ber_dupbv_x+145>, __next = 0x7fff50ffa0f0}}, __size = "\260\016\253UUU\000\000Xh\000@\377\177\000\000\360\240\377P\377\177\000\000\261\277\364\367\377\177\000\000\360\240\377P\377\177\000", __align = 93824997854896}}, be_pcsn_p = 0x7fff40006858, be_syncinfo = 0x555555ab0ee0, be_pb = 0x7fff40000b90, be_cf_ocs = 0x7fff50ffa140, be_private = 0x2fcab248a7e8d500, be_next = {stqe_next = 0x7fff50ffa210}} bi = 0x555555932da0 rc = 32768 #7 0x00007ffff6091391 in syncprov_op_search (op=<optimized out>, rs=0x7fff50ffb8d0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/overlays/syncprov.c:3381 e = 0x555555932da0 a = <optimized out> rc = <optimized out> sl = <optimized out> i = <optimized out> j = <optimized out> ctrls = <optimized out> on = 0x555555932da0 si = 0x555555932fa0 cb = <optimized out> gotstate = 0 changed = <optimized out> do_present = 0 sop = 0x7fff4010dbb0 ss = <optimized out> srs = 0x7fff400064a0 ctxcsn = 0x7fff40006828 i = <optimized out> sids = 0x7fff40006960 numcsns = <optimized out> mincsn = {bv_len = 40, bv_val = 0x7fff40006688 "20230802223107.484385Z#000000#002#000000"} maxcsn = <optimized out> minsid = <optimized out> maxsid = <optimized out> dirty = <optimized out> shortcut = <optimized out> #8 0x000055555561c0da in overlay_op_walk (op=op@entry=0x7fff40106490, rs=rs@entry=0x7fff50ffb8d0, which=which@entry=op_search, oi=0x55555577ab10, on=0x555555932da0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/backover.c:691 bi = 0x555555932da0 rc = 32768 #9 0x000055555561c294 in over_op_func (op=0x7fff40106490, rs=0x7fff50ffb8d0, which=op_search) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/backover.c:766 oi = <optimized out> on = <optimized out> be = 0x555555931040 db = {bd_info = 0x5555556f5b80 <slap_binfo+1440>, bd_self = 0x555555931040, be_ctrls = "\000\001\001\001\000\001\000\000\001\000\000\001\001\000\001\001\000\000\000\001\001\001\000\000\000\000\000\000\000\000\000\000\001", be_flags = 4757768, be_restrictops = 0, be_requires = 0, be_ssf_set = {sss_ssf = 0, sss_transport = 0, sss_tls = 0, sss_sasl = 0, sss_update_ssf = 0, sss_update_transport = 0, sss_update_tls = 0, sss_update_sasl = 0, sss_simple_bind = 0}, be_suffix = 0x555555935170, be_nsuffix = 0x5555559351a0, be_schemadn = {bv_len = 0, bv_val = 0x0}, be_schemandn = {bv_len = 0, bv_val = 0x0}, be_rootdn = {bv_len = 25, bv_val = 0x555555935380 "cn=Manager,dc=bpk2,dc=com"}, be_rootndn = {bv_len = 25, bv_val = 0x555555936490 "cn=manager,dc=bpk2,dc=com"}, be_rootpw = {bv_len = 38, bv_val = 0x5555559352b0 "{SSHA}tAHVQo521yiuLlTvbQ4QoHo9i5usOttD"}, be_max_deref_depth = 15, be_def_limit = {lms_t_soft = 3600, lms_t_hard = 0, lms_s_soft = -1, lms_s_hard = 0, lms_s_unchecked = -1, lms_s_pr = 0, lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits = 0x0, be_acl = 0x5555559351d0, be_dfltaccess = ACL_READ, be_extra_anlist = 0x0, be_lastbind_precision = 0, be_update_ndn = {bv_len = 0, bv_val = 0x0}, be_update_refs = 0x0, be_pcsn_st = {be_pcsn_list = {tqh_first = 0x0, tqh_last = 0x555555931180}, be_pcsn_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, be_pcsn_p = 0x555555931180, be_syncinfo = 0x555555937ad0, be_pb = 0x0, be_cf_ocs = 0x5555556fc5e0 <mdbocs.lto_priv+64>, be_private = 0x7ffff5be5010, be_next = {stqe_next = 0x0}} sc = <optimized out> cb = 0x7fff400067f8 rc = 32768 __PRETTY_FUNCTION__ = "over_op_func" #10 0x00005555555a8b61 in fe_op_search (op=0x7fff40106490, rs=0x7fff50ffb8d0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/search.c:426 bd = 0x555555753fe0 <slap_frontendDB> #11 0x00005555555ab1e4 in do_search (op=0x7fff40106490, rs=0x7fff50ffb8d0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/search.c:267 base = {bv_len = 14, bv_val = 0x7fff40003667 "dc=bpk2,dc=com"} siz = 2 off = 0 i = <optimized out> #12 0x00005555555a2fe5 in connection_operation (ctx=ctx@entry=0x7fff50ffbb70, arg_v=arg_v@entry=0x7fff40106490) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/connection.c:1115 rc = 80 cancel = <optimized out> op = 0x7fff40106490 rs = {sr_type = REP_RESULT, sr_tag = 0, sr_msgid = 0, sr_err = 0, sr_matched = 0x0, sr_text = 0x0, sr_ref = 0x0, sr_ctrls = 0x0, sr_un = {sru_search = {r_entry = 0x0, r_attr_flags = 0, r_operational_attrs = 0x0, r_attrs = 0x0, r_nentries = 0, r_v2ref = 0x0}, sru_sasl = {r_sasldata = 0x0}, sru_extended = {r_rspoid = 0x0, r_rspdata = 0x0}}, sr_flags = 0} tag = 99 opidx = SLAP_OP_SEARCH conn = 0x7ffff3268158 memctx = 0x7fff40000b90 memctx_null = 0x0 memsiz = <optimized out> __PRETTY_FUNCTION__ = "connection_operation" #13 0x00005555555a3bbb in connection_read_thread (ctx=0x7fff50ffbb70, argv=<optimized out>) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/connection.c:1267 rc = 0 cri = {op = 0x7fff40106490, func = 0x0, arg = 0x0, ctx = 0x7fff50ffbb70, nullop = <optimized out>} s = 19 #14 0x00007ffff7fa1af6 in ldap_int_thread_pool_wrapper (xpool=0x555555799e40) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/tpool.c:1059 pq = 0x555555799e40 pool = 0x555555799d30 task = 0x7fff4c000f10 work_list = <optimized out> ctx = {ltu_pq = 0x555555799e40, ltu_id = 140734552327872, ltu_key = {{ltk_key = 0x555555605030 <slap_sl_mem_init>, ltk_data = 0x7fff40000b90, ltk_free = 0x555555606ca0 <slap_sl_mem_destroy>}, {ltk_key = 0x555555599ca0 <operation_counter_init>, ltk_data = 0x7fff40003c60, ltk_free = 0x555555599620 <conn_counter_destroy>}, {ltk_key = 0x5555555b6b50 <slap_op_free>, ltk_data = 0x0, ltk_free = 0x5555555b0330 <slap_op_q_destroy>}, {ltk_key = 0x555555ae5290, ltk_data = 0x7fff401084a0, ltk_free = 0x55555565e580 <mdb_reader_free>}, {ltk_key = 0x5555556418d0 <search_stack>, ltk_data = 0x7fff371fa010, ltk_free = 0x555555641740 <search_stack_free>}, {ltk_key = 0x555555641800 <scope_chunk_get>, ltk_data = 0x7fff383fb010, ltk_free = 0x555555641700 <scope_chunk_free>}, {ltk_key = 0x0, ltk_data = 0x7fff4410ed90, ltk_free = 0x0}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0x0} <repeats 25 times>}} kctx = <optimized out> i = <optimized out> keyslot = <optimized out> hash = <optimized out> pool_lock = 0 freeme = 0 __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper" #15 0x00007ffff78ae907 in start_thread (arg=<optimized out>) at pthread_create.c:444 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737346463248, 7824176847303498060, 140734552327872, -128, 0, 140737272748720, 7824176847248972108, 7823950512947091788}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #16 0x00007ffff7934870 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals.
Thread 6 (Thread 0x7fff517fd6c0 (LWP 22294) "slapd"): #0 0x00007ffff78ab1d9 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555555799ea0) at futex-internal.c:57 sc_cancel_oldtype = 0 __arg6 = <optimized out> __arg3 = <optimized out> _a5 = <optimized out> _a2 = <optimized out> sc_ret = <optimized out> __arg4 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a3 = <optimized out> resultvar = <optimized out> __arg5 = <optimized out> __arg2 = <optimized out> _a4 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 #2 0x00007ffff78ab25f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139 No locals. #3 0x00007ffff78adb79 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x555555799e50, cond=0x555555799e78) at pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7ffff78ad8a0 <__condvar_cleanup_waiting>, __arg = 0x7fff517fcaf0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 664, cond = 0x555555799e78, mutex = 0x555555799e50, private = 0} err = <optimized out> g = 0 flags = <optimized out> g1_start = <optimized out> maxspin = 0 signals = <optimized out> result = 0 wseq = 664 seq = 332 private = 0 maxspin = <optimized out> err = <optimized out> result = <optimized out> wseq = <optimized out> g = <optimized out> seq = <optimized out> flags = <optimized out> private = <optimized out> signals = <optimized out> done = <optimized out> g1_start = <optimized out> spin = <optimized out> buffer = <optimized out> cbuffer = <optimized out> s = <optimized out> #4 ___pthread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at pthread_cond_wait.c:618 No locals. #5 0x00007ffff7fa0269 in ldap_pvt_thread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/thr_posix.c:294 No locals. #6 0x00007ffff7fa1ba7 in ldap_int_thread_pool_wrapper (xpool=0x555555799e40) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/tpool.c:1041 pq = 0x555555799e40 pool = 0x555555799d30 task = 0x0 work_list = <optimized out> ctx = {ltu_pq = 0x555555799e40, ltu_id = 140734560720576, ltu_key = {{ltk_key = 0x555555605030 <slap_sl_mem_init>, ltk_data = 0x7fff48000b90, ltk_free = 0x555555606ca0 <slap_sl_mem_destroy>}, {ltk_key = 0x555555599ca0 <operation_counter_init>, ltk_data = 0x7fff48106a00, ltk_free = 0x555555599620 <conn_counter_destroy>}, {ltk_key = 0x5555555b6b50 <slap_op_free>, ltk_data = 0x7fff481065b0, ltk_free = 0x5555555b0330 <slap_op_q_destroy>}, {ltk_key = 0x555555ae5290, ltk_data = 0x7fff48107920, ltk_free = 0x55555565e580 <mdb_reader_free>}, {ltk_key = 0x5555556418d0 <search_stack>, ltk_data = 0x7fff39bfe010, ltk_free = 0x555555641740 <search_stack_free>}, {ltk_key = 0x555555641800 <scope_chunk_get>, ltk_data = 0x7fff501fa010, ltk_free = 0x555555641700 <scope_chunk_free>}, {ltk_key = 0x0, ltk_data = 0x7fff4010e630, ltk_free = 0x0}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0x0} <repeats 25 times>}} kctx = <optimized out> i = <optimized out> keyslot = <optimized out> hash = <optimized out> pool_lock = 0 freeme = 0 __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper" #7 0x00007ffff78ae907 in start_thread (arg=<optimized out>) at pthread_create.c:444 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737346463248, 7824173549305485644, 140734560720576, -128, 0, 140737272748720, 7824173549250959692, 7823950512947091788}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007ffff7934870 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals.
Thread 5 (Thread 0x7fff51ffe6c0 (LWP 22293) "slapd"): #0 0x00007ffff78ab1d9 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555555799ea0) at futex-internal.c:57 sc_cancel_oldtype = 0 __arg6 = <optimized out> __arg3 = <optimized out> _a5 = <optimized out> _a2 = <optimized out> sc_ret = <optimized out> __arg4 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a3 = <optimized out> resultvar = <optimized out> __arg5 = <optimized out> __arg2 = <optimized out> _a4 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 #2 0x00007ffff78ab25f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139 No locals. #3 0x00007ffff78adb79 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x555555799e50, cond=0x555555799e78) at pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7ffff78ad8a0 <__condvar_cleanup_waiting>, __arg = 0x7fff51ffdaf0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 668, cond = 0x555555799e78, mutex = 0x555555799e50, private = 0} err = <optimized out> g = 0 flags = <optimized out> g1_start = <optimized out> maxspin = 0 signals = <optimized out> result = 0 wseq = 668 seq = 334 private = 0 maxspin = <optimized out> err = <optimized out> result = <optimized out> wseq = <optimized out> g = <optimized out> seq = <optimized out> flags = <optimized out> private = <optimized out> signals = <optimized out> done = <optimized out> g1_start = <optimized out> spin = <optimized out> buffer = <optimized out> cbuffer = <optimized out> s = <optimized out> #4 ___pthread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at pthread_cond_wait.c:618 No locals. #5 0x00007ffff7fa0269 in ldap_pvt_thread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/thr_posix.c:294 No locals. #6 0x00007ffff7fa1ba7 in ldap_int_thread_pool_wrapper (xpool=0x555555799e40) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/tpool.c:1041 pq = 0x555555799e40 pool = 0x555555799d30 task = 0x0 work_list = <optimized out> ctx = {ltu_pq = 0x555555799e40, ltu_id = 140734569113280, ltu_key = {{ltk_key = 0x555555605030 <slap_sl_mem_init>, ltk_data = 0x7fff3c000b90, ltk_free = 0x555555606ca0 <slap_sl_mem_destroy>}, {ltk_key = 0x555555599ca0 <operation_counter_init>, ltk_data = 0x7fff3c107580, ltk_free = 0x555555599620 <conn_counter_destroy>}, {ltk_key = 0x555555ae5290, ltk_data = 0x7fff3c10aa30, ltk_free = 0x55555565e580 <mdb_reader_free>}, {ltk_key = 0x5555556418d0 <search_stack>, ltk_data = 0x7fff3adff010, ltk_free = 0x555555641740 <search_stack_free>}, {ltk_key = 0x555555641800 <scope_chunk_get>, ltk_data = 0x7fff504fb010, ltk_free = 0x555555641700 <scope_chunk_free>}, {ltk_key = 0x5555555b6b50 <slap_op_free>, ltk_data = 0x7fff3c10a640, ltk_free = 0x5555555b0330 <slap_op_q_destroy>}, {ltk_key = 0x0, ltk_data = 0x7fff4410c000, ltk_free = 0x0}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0x0} <repeats 25 times>}} kctx = <optimized out> i = <optimized out> keyslot = <optimized out> hash = <optimized out> pool_lock = 0 freeme = 0 __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper" #7 0x00007ffff78ae907 in start_thread (arg=<optimized out>) at pthread_create.c:444 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737346463248, 7824174649353984332, 140734569113280, -128, 0, 140737272748720, 7824174649299458380, 7823950512947091788}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007ffff7934870 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals.
Thread 4 (Thread 0x7fff527ff6c0 (LWP 22292) "slapd"): #0 0x00007ffff78ab1d9 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555555799ea0) at futex-internal.c:57 sc_cancel_oldtype = 0 __arg6 = <optimized out> __arg3 = <optimized out> _a5 = <optimized out> _a2 = <optimized out> sc_ret = <optimized out> __arg4 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a3 = <optimized out> resultvar = <optimized out> __arg5 = <optimized out> __arg2 = <optimized out> _a4 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 #2 0x00007ffff78ab25f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139 No locals. #3 0x00007ffff78adb79 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x555555799e50, cond=0x555555799e78) at pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7ffff78ad8a0 <__condvar_cleanup_waiting>, __arg = 0x7fff527feaf0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 666, cond = 0x555555799e78, mutex = 0x555555799e50, private = 0} err = <optimized out> g = 0 flags = <optimized out> g1_start = <optimized out> maxspin = 0 signals = <optimized out> result = 0 wseq = 666 seq = 333 private = 0 maxspin = <optimized out> err = <optimized out> result = <optimized out> wseq = <optimized out> g = <optimized out> seq = <optimized out> flags = <optimized out> private = <optimized out> signals = <optimized out> done = <optimized out> g1_start = <optimized out> spin = <optimized out> buffer = <optimized out> cbuffer = <optimized out> s = <optimized out> #4 ___pthread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at pthread_cond_wait.c:618 No locals. #5 0x00007ffff7fa0269 in ldap_pvt_thread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/thr_posix.c:294 No locals. #6 0x00007ffff7fa1ba7 in ldap_int_thread_pool_wrapper (xpool=0x555555799e40) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/tpool.c:1041 pq = 0x555555799e40 pool = 0x555555799d30 task = 0x0 work_list = <optimized out> ctx = {ltu_pq = 0x555555799e40, ltu_id = 140734577505984, ltu_key = {{ltk_key = 0x555555605030 <slap_sl_mem_init>, ltk_data = 0x7fff44000b90, ltk_free = 0x555555606ca0 <slap_sl_mem_destroy>}, {ltk_key = 0x555555ae5290, ltk_data = 0x7fff441061d0, ltk_free = 0x55555565e580 <mdb_reader_free>}, {ltk_key = 0x555555599ca0 <operation_counter_init>, ltk_data = 0x7fff4410b6e0, ltk_free = 0x555555599620 <conn_counter_destroy>}, {ltk_key = 0x5555555b6b50 <slap_op_free>, ltk_data = 0x7fff4410b290, ltk_free = 0x5555555b0330 <slap_op_q_destroy>}, {ltk_key = 0x5555556418d0 <search_stack>, ltk_data = 0x7fff386fc010, ltk_free = 0x555555641740 <search_stack_free>}, {ltk_key = 0x555555641800 <scope_chunk_get>, ltk_data = 0x7fff398fd010, ltk_free = 0x555555641700 <scope_chunk_free>}, {ltk_key = 0x0, ltk_data = 0x7fff401102c0, ltk_free = 0x0}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0x0} <repeats 25 times>}} kctx = <optimized out> i = <optimized out> keyslot = <optimized out> hash = <optimized out> pool_lock = 0 freeme = 0 __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper" #7 0x00007ffff78ae907 in start_thread (arg=<optimized out>) at pthread_create.c:444 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737346463248, 7824171351355971916, 140734577505984, -128, 0, 140737272748720, 7824171351301445964, 7823950512947091788}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007ffff7934870 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals.
Thread 3 (Thread 0x7ffff32646c0 (LWP 22291) "slapd"): #0 0x00007ffff7934c72 in epoll_wait (epfd=6, events=events@entry=0x7ffff6dcd004, maxevents=65535, timeout=72000) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 sc_ret = -4 sc_cancel_oldtype = 0 sc_ret = <optimized out> #1 0x00005555555a1c9b in slapd_daemon_task (ptr=<optimized out>) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/daemon.c:2844 ns = <optimized out> at = <optimized out> nfds = <optimized out> revents = 0x7ffff6dcd004 tvp = <optimized out> cat = {tv_sec = 1691690555, tv_usec = 0} i = <optimized out> nwriters = <optimized out> now = <optimized out> tv = {tv_sec = 72, tv_usec = 0} tdelta = 1 rtask = <optimized out> l = <optimized out> last_idle_check = 1691690224 ebadf = 0 tid = <optimized out> ebuf = '\000' <repeats 127 times> loop = <optimized out> #2 0x00007ffff78ae907 in start_thread (arg=<optimized out>) at pthread_create.c:444 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737346463248, 7823958512641005900, 140737272760000, -128, 0, 140737488346336, 7823958512586479948, 7823950512947091788}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #3 0x00007ffff7934870 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals.
Thread 1 (Thread 0x7ffff7dbe140 (LWP 22284) "slapd"): #0 0x00007ffff78ab1d9 in __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=22291, futex_word=0x7ffff3264990) at futex-internal.c:57 sc_cancel_oldtype = 0 __arg6 = <optimized out> __arg3 = <optimized out> _a5 = <optimized out> _a2 = <optimized out> sc_ret = <optimized out> __arg4 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a3 = <optimized out> resultvar = <optimized out> __arg5 = <optimized out> __arg2 = <optimized out> _a4 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7ffff3264990, expected=22291, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at futex-internal.c:87 err = <optimized out> clockbit = 256 op = 265 #2 0x00007ffff78ab25f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffff3264990, expected=<optimized out>, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at futex-internal.c:139 No locals. #3 0x00007ffff78b05e3 in __pthread_clockjoin_ex (threadid=140737272760000, thread_return=thread_return@entry=0x0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, block=block@entry=true) at pthread_join_common.c:102 ret = <optimized out> _buffer = {__routine = 0x7ffff78b04c0 <cleanup>, __arg = 0x7ffff3264ce0, __canceltype = 1433401664, __prev = 0x0} tid = <optimized out> pd = 0x7ffff32646c0 self = <optimized out> result = 0 pd_result = <optimized out> #4 0x00007ffff78b04b3 in ___pthread_join (threadid=<optimized out>, thread_return=thread_return@entry=0x0) at pthread_join.c:24 No locals. #5 0x00007ffff7fa01f9 in ldap_pvt_thread_join (thread=<optimized out>, thread_return=thread_return@entry=0x0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/thr_posix.c:214 No locals. #6 0x00005555555a03fb in slapd_daemon () at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/daemon.c:3377 i = 0 rc = <optimized out> #7 0x0000555555581c99 in main (argc=<optimized out>, argv=0x7fffffffe1f8) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/main.c:869 i = <optimized out> no_detach = <optimized out> rc = 0 urls = <optimized out> username = <optimized out> groupname = <optimized out> sandbox = <optimized out> syslogUser = 160 pid = <optimized out> waitfds = {0, 0} g_argc = <optimized out> g_argv = 0x7fffffffe1f8 configfile = <optimized out> configdir = <optimized out> serverMode = 1 scp = <optimized out> scp_entry = <optimized out> serverNamePrefix = 0x5555556da917 "" l = <optimized out> slapd_pid_file_unlink = 1 slapd_args_file_unlink = 1 firstopt = <optimized out> __PRETTY_FUNCTION__ = "main"
On 8/10/23 2:08 PM, Brendan Kearney wrote:
On 8/9/23 11:10 AM, Quanah Gibson-Mount wrote:
--On Wednesday, August 9, 2023 9:19 AM -0400 Brendan Kearney bpk678@gmail.com wrote:
Quanah,
i am using prebuilt packages from fedora (38, fully updated), and 2.6.6 is not available yet, so i will have to wait for it.
I didn't see anything in the stack trace indicating a core dump.
One option would be to start slapd on one of the systems from within gdb, and get a backtrace that way. I.e., something like:
gdb /path/to/slapd set args <args for slapd to execute with> run
For example, I get the args my slapd process is running with from the process stack:
/opt/symas/lib/slapd -d 0 -h ldap:/// ldaps:/// ldapi:///
Then stop slapd:
systemctl stop slapd
Start gdb:
gdb /opt/symas/lib/slapd
(gdb) set args -d 0 -h 'ldap:/// ldaps:/// ldapi:///' (gdb) run
Then wait for it to hit the issue, and then:
(gdb) thr apply all bt full
You may also want to set it to log to a file, so something like: gdb /path/to/slapd (gdb) set pagination off (gdb) set logging file /somewhere/gdb.txt (gdb) set logging on (gdb) set args ... (gdb) run (gdb) thr apply all bt full (gdb) set logging off (gdb) quit
Regards, Quanah
it took me a few tries to get things working, but i seem to have produced some output.
Starting program: /usr/sbin/slapd -u ldap -h 'ldap://192.168.88.1' -4 -d 0
This GDB supports auto-downloading debuginfo from the following URLs: https://debuginfod.fedoraproject.org/ Debuginfod has been enabled. To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7ffff32646c0 (LWP 22269)] [Thread 0x7ffff32646c0 (LWP 22269) exited] [New Thread 0x7ffff32646c0 (LWP 22270)] [New Thread 0x7fff527ff6c0 (LWP 22271)] [New Thread 0x7fff51ffe6c0 (LWP 22272)] [New Thread 0x7fff517fd6c0 (LWP 22273)] [New Thread 0x7fff50ffc6c0 (LWP 22274)]
Thread 5 "slapd" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fff51ffe6c0 (LWP 22272)] 0x00007ffff78bc787 in _int_free (av=0x0, p=p@entry=0x7ffff6091980 <syncprov_setup_accesslog+752>, have_lock=have_lock@entry=0) at malloc.c:4562 4562 __libc_lock_lock (av->mutex); Starting program: /usr/sbin/slapd -u ldap -h 'ldap://192.168.88.1' -4 -d 0 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7ffff32646c0 (LWP 22277)] [Thread 0x7ffff32646c0 (LWP 22277) exited] [New Thread 0x7ffff32646c0 (LWP 22278)] [New Thread 0x7fff527ff6c0 (LWP 22279)] [New Thread 0x7fff51ffe6c0 (LWP 22280)] [New Thread 0x7fff517fd6c0 (LWP 22281)] [New Thread 0x7fff50ffc6c0 (LWP 22282)]
Thread 4 "slapd" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fff527ff6c0 (LWP 22279)] 0x00007ffff78bc787 in _int_free (av=0x0, p=p@entry=0x7ffff6091980 <syncprov_setup_accesslog+752>, have_lock=have_lock@entry=0) at malloc.c:4562 4562 __libc_lock_lock (av->mutex); Starting program: /usr/sbin/slapd -u ldap -h 'ldap://192.168.88.1' -4 -d 0 Downloading separate debug info for system-supplied DSO at 0x7ffff7fc8000 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7ffff32646c0 (LWP 22290)] [Thread 0x7ffff32646c0 (LWP 22290) exited] [New Thread 0x7ffff32646c0 (LWP 22291)] [New Thread 0x7fff527ff6c0 (LWP 22292)] [New Thread 0x7fff51ffe6c0 (LWP 22293)] [New Thread 0x7fff517fd6c0 (LWP 22294)] [New Thread 0x7fff50ffc6c0 (LWP 22295)] Downloading separate debug info for /usr/lib64/krb5/plugins/authdata/sssd_pac_plugin.so
Thread 7 "slapd" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fff50ffc6c0 (LWP 22295)] 0x00007ffff78bc787 in _int_free (av=0x0, p=p@entry=0x7ffff6091980 <syncprov_setup_accesslog+752>, have_lock=have_lock@entry=0) at malloc.c:4562 4562 __libc_lock_lock (av->mutex);
Thread 7 (Thread 0x7fff50ffc6c0 (LWP 22295) "slapd"): #0 0x00007ffff78bc787 in _int_free (av=0x0, p=p@entry=0x7ffff6091980 <syncprov_setup_accesslog+752>, have_lock=have_lock@entry=0) at malloc.c:4562 __futex = 0x0 size = 10405003989086109824 fb = <optimized out> nextchunk = <optimized out> nextsize = <optimized out> nextinuse = <optimized out> prevsize = <optimized out> bck = <optimized out> fwd = <optimized out> __PRETTY_FUNCTION__ = "_int_free" #1 0x00007ffff78bf1ce in __GI___libc_free (mem=0x7ffff6091990 <syncprov_db_init>) at malloc.c:3367 ar_ptr = <optimized out> p = 0x7ffff6091980 <syncprov_setup_accesslog+752> err = 11 #2 0x00005555555a73f5 in entry_clean (e=0x555555932da0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/entry.c:465 __PRETTY_FUNCTION__ = "entry_clean" #3 0x000055555566642f in entry_free (e=0x555555932da0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/entry.c:481 No locals. #4 mdb_entry_return (e=0x555555932da0, op=0x7fff40106490) at back-mdb/id2entry.c:543 No locals. #5 mdb_entry_release (op=0x7fff40106490, e=0x555555932da0, rw=<optimized out>) at back-mdb/id2entry.c:592 mdb = <optimized out> moi = <optimized out> release = <optimized out> #6 0x0000555555623e0f in overlay_entry_release_ov (op=op@entry=0x7fff40106490, e=0x555555932da0, rw=rw@entry=0, on=0x0, on@entry=0x555555932da0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/backover.c:452 oi = 0x55555577ab10 be = 0x7fff50ffa5d0 db = {bd_info = 0x0, bd_self = 0x0, be_ctrls = '\000' <repeats 16 times>, "(f\020@\377\177\000\000c\000\000\000\000\000\000", <incomplete sequence \363>, be_flags = 3654763765825540, be_restrictops = 0, be_requires = 45, be_ssf_set = {sss_ssf = 1435701312, sss_transport = 21845, sss_tls = 14, sss_sasl = 0, sss_update_ssf = 1073767256, sss_update_transport = 32767, sss_update_tls = 14, sss_update_sasl = 0, sss_simple_bind = 1073767312}, be_suffix = 0x0, be_nsuffix = 0xffffffff00000001, be_schemadn = {bv_len = 0, bv_val = 0x1 <error: Cannot access memory at address 0x1>}, be_schemandn = {bv_len = 93824993966880, bv_val = 0x7ffff609a000 <generic_filter> "\207"}, be_rootdn = {bv_len = 15, bv_val = 0x7ffff609636d "(objectclass=*)"}, be_rootndn = {bv_len = 0, bv_val = 0x0}, be_rootpw = {bv_len = 0, bv_val = 0x0}, be_max_deref_depth = 0, be_def_limit = {lms_t_soft = 0, lms_t_hard = 0, lms_s_soft = 0, lms_s_hard = 0, lms_s_unchecked = 0, lms_s_pr = 0, lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits = 0x0, be_acl = 0x3000000, be_dfltaccess = ACL_NONE, be_extra_anlist = 0x7fff40006858, be_lastbind_precision = 1437273824, be_update_ndn = {bv_len = 140734267132816, bv_val = 0x3 <error: Cannot access memory at address 0x3>}, be_update_refs = 0x3, be_pcsn_st = {be_pcsn_list = {tqh_first = 0x7fff50ffa0c0, tqh_last = 0x7ffff7f49f58 <ber_memalloc_x+40>}, be_pcsn_mutex = {__data = {__lock = 1437273776, __count = 21845, __owner = 1073768536, __nusers = 32767, __kind = 1358930160, __spins = 32767, __elision = 0, __list = {__prev = 0x7ffff7f4bfb1 <ber_dupbv_x+145>, __next = 0x7fff50ffa0f0}}, __size = "\260\016\253UUU\000\000Xh\000@\377\177\000\000\360\240\377P\377\177\000\000\261\277\364\367\377\177\000\000\360\240\377P\377\177\000", __align = 93824997854896}}, be_pcsn_p = 0x7fff40006858, be_syncinfo = 0x555555ab0ee0, be_pb = 0x7fff40000b90, be_cf_ocs = 0x7fff50ffa140, be_private = 0x2fcab248a7e8d500, be_next = {stqe_next = 0x7fff50ffa210}} bi = 0x555555932da0 rc = 32768 #7 0x00007ffff6091391 in syncprov_op_search (op=<optimized out>, rs=0x7fff50ffb8d0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/overlays/syncprov.c:3381 e = 0x555555932da0 a = <optimized out> rc = <optimized out> sl = <optimized out> i = <optimized out> j = <optimized out> ctrls = <optimized out> on = 0x555555932da0 si = 0x555555932fa0 cb = <optimized out> gotstate = 0 changed = <optimized out> do_present = 0 sop = 0x7fff4010dbb0 ss = <optimized out> srs = 0x7fff400064a0 ctxcsn = 0x7fff40006828 i = <optimized out> sids = 0x7fff40006960 numcsns = <optimized out> mincsn = {bv_len = 40, bv_val = 0x7fff40006688 "20230802223107.484385Z#000000#002#000000"} maxcsn = <optimized out> minsid = <optimized out> maxsid = <optimized out> dirty = <optimized out> shortcut = <optimized out> #8 0x000055555561c0da in overlay_op_walk (op=op@entry=0x7fff40106490, rs=rs@entry=0x7fff50ffb8d0, which=which@entry=op_search, oi=0x55555577ab10, on=0x555555932da0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/backover.c:691 bi = 0x555555932da0 rc = 32768 #9 0x000055555561c294 in over_op_func (op=0x7fff40106490, rs=0x7fff50ffb8d0, which=op_search) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/backover.c:766 oi = <optimized out> on = <optimized out> be = 0x555555931040 db = {bd_info = 0x5555556f5b80 <slap_binfo+1440>, bd_self = 0x555555931040, be_ctrls = "\000\001\001\001\000\001\000\000\001\000\000\001\001\000\001\001\000\000\000\001\001\001\000\000\000\000\000\000\000\000\000\000\001", be_flags = 4757768, be_restrictops = 0, be_requires = 0, be_ssf_set = {sss_ssf = 0, sss_transport = 0, sss_tls = 0, sss_sasl = 0, sss_update_ssf = 0, sss_update_transport = 0, sss_update_tls = 0, sss_update_sasl = 0, sss_simple_bind = 0}, be_suffix = 0x555555935170, be_nsuffix = 0x5555559351a0, be_schemadn = {bv_len = 0, bv_val = 0x0}, be_schemandn = {bv_len = 0, bv_val = 0x0}, be_rootdn = {bv_len = 25, bv_val = 0x555555935380 "cn=Manager,dc=bpk2,dc=com"}, be_rootndn = {bv_len = 25, bv_val = 0x555555936490 "cn=manager,dc=bpk2,dc=com"}, be_rootpw = {bv_len = 38, bv_val = 0x5555559352b0 "{SSHA}tAHVQo521yiuLlTvbQ4QoHo9i5usOttD"}, be_max_deref_depth = 15, be_def_limit = {lms_t_soft = 3600, lms_t_hard = 0, lms_s_soft = -1, lms_s_hard = 0, lms_s_unchecked = -1, lms_s_pr = 0, lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits = 0x0, be_acl = 0x5555559351d0, be_dfltaccess = ACL_READ, be_extra_anlist = 0x0, be_lastbind_precision = 0, be_update_ndn = {bv_len = 0, bv_val = 0x0}, be_update_refs = 0x0, be_pcsn_st = {be_pcsn_list = {tqh_first = 0x0, tqh_last = 0x555555931180}, be_pcsn_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, be_pcsn_p = 0x555555931180, be_syncinfo = 0x555555937ad0, be_pb = 0x0, be_cf_ocs = 0x5555556fc5e0 <mdbocs.lto_priv+64>, be_private = 0x7ffff5be5010, be_next = {stqe_next = 0x0}} sc = <optimized out> cb = 0x7fff400067f8 rc = 32768 __PRETTY_FUNCTION__ = "over_op_func" #10 0x00005555555a8b61 in fe_op_search (op=0x7fff40106490, rs=0x7fff50ffb8d0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/search.c:426 bd = 0x555555753fe0 <slap_frontendDB> #11 0x00005555555ab1e4 in do_search (op=0x7fff40106490, rs=0x7fff50ffb8d0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/search.c:267 base = {bv_len = 14, bv_val = 0x7fff40003667 "dc=bpk2,dc=com"} siz = 2 off = 0 i = <optimized out> #12 0x00005555555a2fe5 in connection_operation (ctx=ctx@entry=0x7fff50ffbb70, arg_v=arg_v@entry=0x7fff40106490) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/connection.c:1115 rc = 80 cancel = <optimized out> op = 0x7fff40106490 rs = {sr_type = REP_RESULT, sr_tag = 0, sr_msgid = 0, sr_err = 0, sr_matched = 0x0, sr_text = 0x0, sr_ref = 0x0, sr_ctrls = 0x0, sr_un = {sru_search = {r_entry = 0x0, r_attr_flags = 0, r_operational_attrs = 0x0, r_attrs = 0x0, r_nentries = 0, r_v2ref = 0x0}, sru_sasl = {r_sasldata = 0x0}, sru_extended = {r_rspoid = 0x0, r_rspdata = 0x0}}, sr_flags = 0} tag = 99 opidx = SLAP_OP_SEARCH conn = 0x7ffff3268158 memctx = 0x7fff40000b90 memctx_null = 0x0 memsiz = <optimized out> __PRETTY_FUNCTION__ = "connection_operation" #13 0x00005555555a3bbb in connection_read_thread (ctx=0x7fff50ffbb70, argv=<optimized out>) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/connection.c:1267 rc = 0 cri = {op = 0x7fff40106490, func = 0x0, arg = 0x0, ctx = 0x7fff50ffbb70, nullop = <optimized out>} s = 19 #14 0x00007ffff7fa1af6 in ldap_int_thread_pool_wrapper (xpool=0x555555799e40) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/tpool.c:1059 pq = 0x555555799e40 pool = 0x555555799d30 task = 0x7fff4c000f10 work_list = <optimized out> ctx = {ltu_pq = 0x555555799e40, ltu_id = 140734552327872, ltu_key = {{ltk_key = 0x555555605030 <slap_sl_mem_init>, ltk_data = 0x7fff40000b90, ltk_free = 0x555555606ca0 <slap_sl_mem_destroy>}, {ltk_key = 0x555555599ca0 <operation_counter_init>, ltk_data = 0x7fff40003c60, ltk_free = 0x555555599620 <conn_counter_destroy>}, {ltk_key = 0x5555555b6b50 <slap_op_free>, ltk_data = 0x0, ltk_free = 0x5555555b0330 <slap_op_q_destroy>}, {ltk_key = 0x555555ae5290, ltk_data = 0x7fff401084a0, ltk_free = 0x55555565e580 <mdb_reader_free>}, {ltk_key = 0x5555556418d0 <search_stack>, ltk_data = 0x7fff371fa010, ltk_free = 0x555555641740 <search_stack_free>}, {ltk_key = 0x555555641800 <scope_chunk_get>, ltk_data = 0x7fff383fb010, ltk_free = 0x555555641700 <scope_chunk_free>}, {ltk_key = 0x0, ltk_data = 0x7fff4410ed90, ltk_free = 0x0}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0x0} <repeats 25 times>}} kctx = <optimized out> i = <optimized out> keyslot = <optimized out> hash = <optimized out> pool_lock = 0 freeme = 0 __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper" #15 0x00007ffff78ae907 in start_thread (arg=<optimized out>) at pthread_create.c:444 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737346463248, 7824176847303498060, 140734552327872, -128, 0, 140737272748720, 7824176847248972108, 7823950512947091788}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #16 0x00007ffff7934870 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals.
Thread 6 (Thread 0x7fff517fd6c0 (LWP 22294) "slapd"): #0 0x00007ffff78ab1d9 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555555799ea0) at futex-internal.c:57 sc_cancel_oldtype = 0 __arg6 = <optimized out> __arg3 = <optimized out> _a5 = <optimized out> _a2 = <optimized out> sc_ret = <optimized out> __arg4 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a3 = <optimized out> resultvar = <optimized out> __arg5 = <optimized out> __arg2 = <optimized out> _a4 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 #2 0x00007ffff78ab25f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139 No locals. #3 0x00007ffff78adb79 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x555555799e50, cond=0x555555799e78) at pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7ffff78ad8a0 <__condvar_cleanup_waiting>, __arg = 0x7fff517fcaf0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 664, cond = 0x555555799e78, mutex = 0x555555799e50, private = 0} err = <optimized out> g = 0 flags = <optimized out> g1_start = <optimized out> maxspin = 0 signals = <optimized out> result = 0 wseq = 664 seq = 332 private = 0 maxspin = <optimized out> err = <optimized out> result = <optimized out> wseq = <optimized out> g = <optimized out> seq = <optimized out> flags = <optimized out> private = <optimized out> signals = <optimized out> done = <optimized out> g1_start = <optimized out> spin = <optimized out> buffer = <optimized out> cbuffer = <optimized out> s = <optimized out> #4 ___pthread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at pthread_cond_wait.c:618 No locals. #5 0x00007ffff7fa0269 in ldap_pvt_thread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/thr_posix.c:294 No locals. #6 0x00007ffff7fa1ba7 in ldap_int_thread_pool_wrapper (xpool=0x555555799e40) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/tpool.c:1041 pq = 0x555555799e40 pool = 0x555555799d30 task = 0x0 work_list = <optimized out> ctx = {ltu_pq = 0x555555799e40, ltu_id = 140734560720576, ltu_key = {{ltk_key = 0x555555605030 <slap_sl_mem_init>, ltk_data = 0x7fff48000b90, ltk_free = 0x555555606ca0 <slap_sl_mem_destroy>}, {ltk_key = 0x555555599ca0 <operation_counter_init>, ltk_data = 0x7fff48106a00, ltk_free = 0x555555599620 <conn_counter_destroy>}, {ltk_key = 0x5555555b6b50 <slap_op_free>, ltk_data = 0x7fff481065b0, ltk_free = 0x5555555b0330 <slap_op_q_destroy>}, {ltk_key = 0x555555ae5290, ltk_data = 0x7fff48107920, ltk_free = 0x55555565e580 <mdb_reader_free>}, {ltk_key = 0x5555556418d0 <search_stack>, ltk_data = 0x7fff39bfe010, ltk_free = 0x555555641740 <search_stack_free>}, {ltk_key = 0x555555641800 <scope_chunk_get>, ltk_data = 0x7fff501fa010, ltk_free = 0x555555641700 <scope_chunk_free>}, {ltk_key = 0x0, ltk_data = 0x7fff4010e630, ltk_free = 0x0}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0x0} <repeats 25 times>}} kctx = <optimized out> i = <optimized out> keyslot = <optimized out> hash = <optimized out> pool_lock = 0 freeme = 0 __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper" #7 0x00007ffff78ae907 in start_thread (arg=<optimized out>) at pthread_create.c:444 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737346463248, 7824173549305485644, 140734560720576, -128, 0, 140737272748720, 7824173549250959692, 7823950512947091788}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007ffff7934870 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals.
Thread 5 (Thread 0x7fff51ffe6c0 (LWP 22293) "slapd"): #0 0x00007ffff78ab1d9 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555555799ea0) at futex-internal.c:57 sc_cancel_oldtype = 0 __arg6 = <optimized out> __arg3 = <optimized out> _a5 = <optimized out> _a2 = <optimized out> sc_ret = <optimized out> __arg4 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a3 = <optimized out> resultvar = <optimized out> __arg5 = <optimized out> __arg2 = <optimized out> _a4 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 #2 0x00007ffff78ab25f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139 No locals. #3 0x00007ffff78adb79 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x555555799e50, cond=0x555555799e78) at pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7ffff78ad8a0 <__condvar_cleanup_waiting>, __arg = 0x7fff51ffdaf0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 668, cond = 0x555555799e78, mutex = 0x555555799e50, private = 0} err = <optimized out> g = 0 flags = <optimized out> g1_start = <optimized out> maxspin = 0 signals = <optimized out> result = 0 wseq = 668 seq = 334 private = 0 maxspin = <optimized out> err = <optimized out> result = <optimized out> wseq = <optimized out> g = <optimized out> seq = <optimized out> flags = <optimized out> private = <optimized out> signals = <optimized out> done = <optimized out> g1_start = <optimized out> spin = <optimized out> buffer = <optimized out> cbuffer = <optimized out> s = <optimized out> #4 ___pthread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at pthread_cond_wait.c:618 No locals. #5 0x00007ffff7fa0269 in ldap_pvt_thread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/thr_posix.c:294 No locals. #6 0x00007ffff7fa1ba7 in ldap_int_thread_pool_wrapper (xpool=0x555555799e40) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/tpool.c:1041 pq = 0x555555799e40 pool = 0x555555799d30 task = 0x0 work_list = <optimized out> ctx = {ltu_pq = 0x555555799e40, ltu_id = 140734569113280, ltu_key = {{ltk_key = 0x555555605030 <slap_sl_mem_init>, ltk_data = 0x7fff3c000b90, ltk_free = 0x555555606ca0 <slap_sl_mem_destroy>}, {ltk_key = 0x555555599ca0 <operation_counter_init>, ltk_data = 0x7fff3c107580, ltk_free = 0x555555599620 <conn_counter_destroy>}, {ltk_key = 0x555555ae5290, ltk_data = 0x7fff3c10aa30, ltk_free = 0x55555565e580 <mdb_reader_free>}, {ltk_key = 0x5555556418d0 <search_stack>, ltk_data = 0x7fff3adff010, ltk_free = 0x555555641740 <search_stack_free>}, {ltk_key = 0x555555641800 <scope_chunk_get>, ltk_data = 0x7fff504fb010, ltk_free = 0x555555641700 <scope_chunk_free>}, {ltk_key = 0x5555555b6b50 <slap_op_free>, ltk_data = 0x7fff3c10a640, ltk_free = 0x5555555b0330 <slap_op_q_destroy>}, {ltk_key = 0x0, ltk_data = 0x7fff4410c000, ltk_free = 0x0}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0x0} <repeats 25 times>}} kctx = <optimized out> i = <optimized out> keyslot = <optimized out> hash = <optimized out> pool_lock = 0 freeme = 0 __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper" #7 0x00007ffff78ae907 in start_thread (arg=<optimized out>) at pthread_create.c:444 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737346463248, 7824174649353984332, 140734569113280, -128, 0, 140737272748720, 7824174649299458380, 7823950512947091788}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007ffff7934870 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals.
Thread 4 (Thread 0x7fff527ff6c0 (LWP 22292) "slapd"): #0 0x00007ffff78ab1d9 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555555799ea0) at futex-internal.c:57 sc_cancel_oldtype = 0 __arg6 = <optimized out> __arg3 = <optimized out> _a5 = <optimized out> _a2 = <optimized out> sc_ret = <optimized out> __arg4 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a3 = <optimized out> resultvar = <optimized out> __arg5 = <optimized out> __arg2 = <optimized out> _a4 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 #2 0x00007ffff78ab25f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139 No locals. #3 0x00007ffff78adb79 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x555555799e50, cond=0x555555799e78) at pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7ffff78ad8a0 <__condvar_cleanup_waiting>, __arg = 0x7fff527feaf0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 666, cond = 0x555555799e78, mutex = 0x555555799e50, private = 0} err = <optimized out> g = 0 flags = <optimized out> g1_start = <optimized out> maxspin = 0 signals = <optimized out> result = 0 wseq = 666 seq = 333 private = 0 maxspin = <optimized out> err = <optimized out> result = <optimized out> wseq = <optimized out> g = <optimized out> seq = <optimized out> flags = <optimized out> private = <optimized out> signals = <optimized out> done = <optimized out> g1_start = <optimized out> spin = <optimized out> buffer = <optimized out> cbuffer = <optimized out> s = <optimized out> #4 ___pthread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at pthread_cond_wait.c:618 No locals. #5 0x00007ffff7fa0269 in ldap_pvt_thread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/thr_posix.c:294 No locals. #6 0x00007ffff7fa1ba7 in ldap_int_thread_pool_wrapper (xpool=0x555555799e40) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/tpool.c:1041 pq = 0x555555799e40 pool = 0x555555799d30 task = 0x0 work_list = <optimized out> ctx = {ltu_pq = 0x555555799e40, ltu_id = 140734577505984, ltu_key = {{ltk_key = 0x555555605030 <slap_sl_mem_init>, ltk_data = 0x7fff44000b90, ltk_free = 0x555555606ca0 <slap_sl_mem_destroy>}, {ltk_key = 0x555555ae5290, ltk_data = 0x7fff441061d0, ltk_free = 0x55555565e580 <mdb_reader_free>}, {ltk_key = 0x555555599ca0 <operation_counter_init>, ltk_data = 0x7fff4410b6e0, ltk_free = 0x555555599620 <conn_counter_destroy>}, {ltk_key = 0x5555555b6b50 <slap_op_free>, ltk_data = 0x7fff4410b290, ltk_free = 0x5555555b0330 <slap_op_q_destroy>}, {ltk_key = 0x5555556418d0 <search_stack>, ltk_data = 0x7fff386fc010, ltk_free = 0x555555641740 <search_stack_free>}, {ltk_key = 0x555555641800 <scope_chunk_get>, ltk_data = 0x7fff398fd010, ltk_free = 0x555555641700 <scope_chunk_free>}, {ltk_key = 0x0, ltk_data = 0x7fff401102c0, ltk_free = 0x0}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0x0} <repeats 25 times>}} kctx = <optimized out> i = <optimized out> keyslot = <optimized out> hash = <optimized out> pool_lock = 0 freeme = 0 __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper" #7 0x00007ffff78ae907 in start_thread (arg=<optimized out>) at pthread_create.c:444 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737346463248, 7824171351355971916, 140734577505984, -128, 0, 140737272748720, 7824171351301445964, 7823950512947091788}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007ffff7934870 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals.
Thread 3 (Thread 0x7ffff32646c0 (LWP 22291) "slapd"): #0 0x00007ffff7934c72 in epoll_wait (epfd=6, events=events@entry=0x7ffff6dcd004, maxevents=65535, timeout=72000) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 sc_ret = -4 sc_cancel_oldtype = 0 sc_ret = <optimized out> #1 0x00005555555a1c9b in slapd_daemon_task (ptr=<optimized out>) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/daemon.c:2844 ns = <optimized out> at = <optimized out> nfds = <optimized out> revents = 0x7ffff6dcd004 tvp = <optimized out> cat = {tv_sec = 1691690555, tv_usec = 0} i = <optimized out> nwriters = <optimized out> now = <optimized out> tv = {tv_sec = 72, tv_usec = 0} tdelta = 1 rtask = <optimized out> l = <optimized out> last_idle_check = 1691690224 ebadf = 0 tid = <optimized out> ebuf = '\000' <repeats 127 times> loop = <optimized out> #2 0x00007ffff78ae907 in start_thread (arg=<optimized out>) at pthread_create.c:444 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737346463248, 7823958512641005900, 140737272760000, -128, 0, 140737488346336, 7823958512586479948, 7823950512947091788}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #3 0x00007ffff7934870 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals.
Thread 1 (Thread 0x7ffff7dbe140 (LWP 22284) "slapd"): #0 0x00007ffff78ab1d9 in __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=22291, futex_word=0x7ffff3264990) at futex-internal.c:57 sc_cancel_oldtype = 0 __arg6 = <optimized out> __arg3 = <optimized out> _a5 = <optimized out> _a2 = <optimized out> sc_ret = <optimized out> __arg4 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a3 = <optimized out> resultvar = <optimized out> __arg5 = <optimized out> __arg2 = <optimized out> _a4 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7ffff3264990, expected=22291, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at futex-internal.c:87 err = <optimized out> clockbit = 256 op = 265 #2 0x00007ffff78ab25f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffff3264990, expected=<optimized out>, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at futex-internal.c:139 No locals. #3 0x00007ffff78b05e3 in __pthread_clockjoin_ex (threadid=140737272760000, thread_return=thread_return@entry=0x0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, block=block@entry=true) at pthread_join_common.c:102 ret = <optimized out> _buffer = {__routine = 0x7ffff78b04c0 <cleanup>, __arg = 0x7ffff3264ce0, __canceltype = 1433401664, __prev = 0x0} tid = <optimized out> pd = 0x7ffff32646c0 self = <optimized out> result = 0 pd_result = <optimized out> #4 0x00007ffff78b04b3 in ___pthread_join (threadid=<optimized out>, thread_return=thread_return@entry=0x0) at pthread_join.c:24 No locals. #5 0x00007ffff7fa01f9 in ldap_pvt_thread_join (thread=<optimized out>, thread_return=thread_return@entry=0x0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/thr_posix.c:214 No locals. #6 0x00005555555a03fb in slapd_daemon () at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/daemon.c:3377 i = 0 rc = <optimized out> #7 0x0000555555581c99 in main (argc=<optimized out>, argv=0x7fffffffe1f8) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/main.c:869 i = <optimized out> no_detach = <optimized out> rc = 0 urls = <optimized out> username = <optimized out> groupname = <optimized out> sandbox = <optimized out> syslogUser = 160 pid = <optimized out> waitfds = {0, 0} g_argc = <optimized out> g_argv = 0x7fffffffe1f8 configfile = <optimized out> configdir = <optimized out> serverMode = 1 scp = <optimized out> scp_entry = <optimized out> serverNamePrefix = 0x5555556da917 "" l = <optimized out> slapd_pid_file_unlink = 1 slapd_args_file_unlink = 1 firstopt = <optimized out> __PRETTY_FUNCTION__ = "main"
seeing the message about the accesslog, i stop all slapd instances and removed the module. i wasnt using it, and did not have it leveraged anywhere, so i could just remove the ldif file from the filesystem and start services. in the 3 node cluster, node 2 was the one working, so i started that one first. nodes 1 and 3 were started subsequently and there were no coredumps.
interestingly, things did not go well beyond that. the replication seems to be started, but node 1 and 3 cannot be used. the DIT does not seem to be present, and all sorts of LDAP/Kerberos integrated services have issues. the 3 nodes are in a load balanced pool behind a VIP and "least connections" is the balancing method. any hit to node 1 or 3 results in missing data, bad auth, all the bad things.
on node 3 i tried to restart krb5kdc and kadmin, since i am using LDAP to backend Kerberos. i figured maybe they were in a weird state, but neither would restart. i had to stop slapd on node 1 and 3 to get things back up and functional.
to see what was going on, i tried modifying the log levels, and pointing to alternate ports on the VIP. port 389 load balances across all active nodes. ports 390, 391 and 392 point to the individual pool members only, so i can hit and test each instance. with all 3 nodes running only node 2 would take the modify operation and update the log levels. this update to node 2 was replicated to the other nodes and they moved to the new log level, but other wise, i cannot access the DIT on nodes 1 and 3.
it seems that something is still gumming up the works, even though the seemingly bad accesslog module was removed and services were restarted. if i start up node 1 and 3 and try to search with "dc=bpk2,dc=com" as the base, node 1 gives me back 24 numEntries, node 2 gives me back 164259 entries, and node 3 gives me back 25 numEntries.
clearly, something is out of whack with the replication now. what means can i employ to get things back up and running?
thanks,
brendan
--On Friday, August 11, 2023 10:24 AM -0400 Brendan Kearney bpk678@gmail.com wrote:
seeing the message about the accesslog, i stop all slapd instances and removed the module. i wasnt using it, and did not have it leveraged anywhere, so i could just remove the ldif file from the filesystem and start services. in the 3 node cluster, node 2 was the one working, so i started that one first. nodes 1 and 3 were started subsequently and there were no coredumps.
There's no information on how you've configured your servers, or if your configuration is even valid.
--Quanah
On 8/11/23 12:15 PM, Quanah Gibson-Mount wrote:
--On Friday, August 11, 2023 10:24 AM -0400 Brendan Kearney bpk678@gmail.com wrote:
seeing the message about the accesslog, i stop all slapd instances and removed the module. i wasnt using it, and did not have it leveraged anywhere, so i could just remove the ldif file from the filesystem and start services. in the 3 node cluster, node 2 was the one working, so i started that one first. nodes 1 and 3 were started subsequently and there were no coredumps.
There's no information on how you've configured your servers, or if your configuration is even valid.
--Quanah
thats fair. what info is appropriate, and at what depth? high level, it's a 3 node cluster, with syncprov, refint, memberof, dynlist, back_meta and sssvlv modules loaded. the cn=config database is only a olcDatabaseConfig, not olcDatabaseConfig + olcMdbConfig like dc=bpk2,dc=com (if that matters). only the syncprov overlay is used in cn=config. both cn=config and dc=bpk2,dc=com are intended to be replicated between the cluster members.
let me know what specifics would help and i can send those details over.
thanks,
brendan
--On Friday, August 11, 2023 2:10 PM -0400 Brendan Kearney bpk678@gmail.com wrote:
thats fair. what info is appropriate, and at what depth? high level, it's a 3 node cluster, with syncprov, refint, memberof, dynlist, back_meta and sssvlv modules loaded. the cn=config database is only a olcDatabaseConfig, not olcDatabaseConfig + olcMdbConfig like dc=bpk2,dc=com (if that matters). only the syncprov overlay is used in cn=config. both cn=config and dc=bpk2,dc=com are intended to be replicated between the cluster members.
let me know what specifics would help and i can send those details over.
Generally an export of the cn=config DB, minus any passwords of the different providers. If the config is identical across all 3, then just one copy of the config.
--Quanah
openldap-technical@openldap.org
-
Brendan Kearney -
Quanah Gibson-Mount