On 8/10/23 2:08 PM, Brendan Kearney wrote:
On 8/9/23 11:10 AM, Quanah Gibson-Mount wrote:
--On Wednesday, August 9, 2023 9:19 AM -0400 Brendan Kearney bpk678@gmail.com wrote:
Quanah,
i am using prebuilt packages from fedora (38, fully updated), and 2.6.6 is not available yet, so i will have to wait for it.
I didn't see anything in the stack trace indicating a core dump.
One option would be to start slapd on one of the systems from within gdb, and get a backtrace that way. I.e., something like:
gdb /path/to/slapd set args <args for slapd to execute with> run
For example, I get the args my slapd process is running with from the process stack:
/opt/symas/lib/slapd -d 0 -h ldap:/// ldaps:/// ldapi:///
Then stop slapd:
systemctl stop slapd
Start gdb:
gdb /opt/symas/lib/slapd
(gdb) set args -d 0 -h 'ldap:/// ldaps:/// ldapi:///' (gdb) run
Then wait for it to hit the issue, and then:
(gdb) thr apply all bt full
You may also want to set it to log to a file, so something like: gdb /path/to/slapd (gdb) set pagination off (gdb) set logging file /somewhere/gdb.txt (gdb) set logging on (gdb) set args ... (gdb) run (gdb) thr apply all bt full (gdb) set logging off (gdb) quit
Regards, Quanah
it took me a few tries to get things working, but i seem to have produced some output.
Starting program: /usr/sbin/slapd -u ldap -h 'ldap://192.168.88.1' -4 -d 0
This GDB supports auto-downloading debuginfo from the following URLs: https://debuginfod.fedoraproject.org/ Debuginfod has been enabled. To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7ffff32646c0 (LWP 22269)] [Thread 0x7ffff32646c0 (LWP 22269) exited] [New Thread 0x7ffff32646c0 (LWP 22270)] [New Thread 0x7fff527ff6c0 (LWP 22271)] [New Thread 0x7fff51ffe6c0 (LWP 22272)] [New Thread 0x7fff517fd6c0 (LWP 22273)] [New Thread 0x7fff50ffc6c0 (LWP 22274)]
Thread 5 "slapd" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fff51ffe6c0 (LWP 22272)] 0x00007ffff78bc787 in _int_free (av=0x0, p=p@entry=0x7ffff6091980 <syncprov_setup_accesslog+752>, have_lock=have_lock@entry=0) at malloc.c:4562 4562 __libc_lock_lock (av->mutex); Starting program: /usr/sbin/slapd -u ldap -h 'ldap://192.168.88.1' -4 -d 0 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7ffff32646c0 (LWP 22277)] [Thread 0x7ffff32646c0 (LWP 22277) exited] [New Thread 0x7ffff32646c0 (LWP 22278)] [New Thread 0x7fff527ff6c0 (LWP 22279)] [New Thread 0x7fff51ffe6c0 (LWP 22280)] [New Thread 0x7fff517fd6c0 (LWP 22281)] [New Thread 0x7fff50ffc6c0 (LWP 22282)]
Thread 4 "slapd" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fff527ff6c0 (LWP 22279)] 0x00007ffff78bc787 in _int_free (av=0x0, p=p@entry=0x7ffff6091980 <syncprov_setup_accesslog+752>, have_lock=have_lock@entry=0) at malloc.c:4562 4562 __libc_lock_lock (av->mutex); Starting program: /usr/sbin/slapd -u ldap -h 'ldap://192.168.88.1' -4 -d 0 Downloading separate debug info for system-supplied DSO at 0x7ffff7fc8000 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7ffff32646c0 (LWP 22290)] [Thread 0x7ffff32646c0 (LWP 22290) exited] [New Thread 0x7ffff32646c0 (LWP 22291)] [New Thread 0x7fff527ff6c0 (LWP 22292)] [New Thread 0x7fff51ffe6c0 (LWP 22293)] [New Thread 0x7fff517fd6c0 (LWP 22294)] [New Thread 0x7fff50ffc6c0 (LWP 22295)] Downloading separate debug info for /usr/lib64/krb5/plugins/authdata/sssd_pac_plugin.so
Thread 7 "slapd" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fff50ffc6c0 (LWP 22295)] 0x00007ffff78bc787 in _int_free (av=0x0, p=p@entry=0x7ffff6091980 <syncprov_setup_accesslog+752>, have_lock=have_lock@entry=0) at malloc.c:4562 4562 __libc_lock_lock (av->mutex);
Thread 7 (Thread 0x7fff50ffc6c0 (LWP 22295) "slapd"): #0 0x00007ffff78bc787 in _int_free (av=0x0, p=p@entry=0x7ffff6091980 <syncprov_setup_accesslog+752>, have_lock=have_lock@entry=0) at malloc.c:4562 __futex = 0x0 size = 10405003989086109824 fb = <optimized out> nextchunk = <optimized out> nextsize = <optimized out> nextinuse = <optimized out> prevsize = <optimized out> bck = <optimized out> fwd = <optimized out> __PRETTY_FUNCTION__ = "_int_free" #1 0x00007ffff78bf1ce in __GI___libc_free (mem=0x7ffff6091990 <syncprov_db_init>) at malloc.c:3367 ar_ptr = <optimized out> p = 0x7ffff6091980 <syncprov_setup_accesslog+752> err = 11 #2 0x00005555555a73f5 in entry_clean (e=0x555555932da0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/entry.c:465 __PRETTY_FUNCTION__ = "entry_clean" #3 0x000055555566642f in entry_free (e=0x555555932da0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/entry.c:481 No locals. #4 mdb_entry_return (e=0x555555932da0, op=0x7fff40106490) at back-mdb/id2entry.c:543 No locals. #5 mdb_entry_release (op=0x7fff40106490, e=0x555555932da0, rw=<optimized out>) at back-mdb/id2entry.c:592 mdb = <optimized out> moi = <optimized out> release = <optimized out> #6 0x0000555555623e0f in overlay_entry_release_ov (op=op@entry=0x7fff40106490, e=0x555555932da0, rw=rw@entry=0, on=0x0, on@entry=0x555555932da0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/backover.c:452 oi = 0x55555577ab10 be = 0x7fff50ffa5d0 db = {bd_info = 0x0, bd_self = 0x0, be_ctrls = '\000' <repeats 16 times>, "(f\020@\377\177\000\000c\000\000\000\000\000\000", <incomplete sequence \363>, be_flags = 3654763765825540, be_restrictops = 0, be_requires = 45, be_ssf_set = {sss_ssf = 1435701312, sss_transport = 21845, sss_tls = 14, sss_sasl = 0, sss_update_ssf = 1073767256, sss_update_transport = 32767, sss_update_tls = 14, sss_update_sasl = 0, sss_simple_bind = 1073767312}, be_suffix = 0x0, be_nsuffix = 0xffffffff00000001, be_schemadn = {bv_len = 0, bv_val = 0x1 <error: Cannot access memory at address 0x1>}, be_schemandn = {bv_len = 93824993966880, bv_val = 0x7ffff609a000 <generic_filter> "\207"}, be_rootdn = {bv_len = 15, bv_val = 0x7ffff609636d "(objectclass=*)"}, be_rootndn = {bv_len = 0, bv_val = 0x0}, be_rootpw = {bv_len = 0, bv_val = 0x0}, be_max_deref_depth = 0, be_def_limit = {lms_t_soft = 0, lms_t_hard = 0, lms_s_soft = 0, lms_s_hard = 0, lms_s_unchecked = 0, lms_s_pr = 0, lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits = 0x0, be_acl = 0x3000000, be_dfltaccess = ACL_NONE, be_extra_anlist = 0x7fff40006858, be_lastbind_precision = 1437273824, be_update_ndn = {bv_len = 140734267132816, bv_val = 0x3 <error: Cannot access memory at address 0x3>}, be_update_refs = 0x3, be_pcsn_st = {be_pcsn_list = {tqh_first = 0x7fff50ffa0c0, tqh_last = 0x7ffff7f49f58 <ber_memalloc_x+40>}, be_pcsn_mutex = {__data = {__lock = 1437273776, __count = 21845, __owner = 1073768536, __nusers = 32767, __kind = 1358930160, __spins = 32767, __elision = 0, __list = {__prev = 0x7ffff7f4bfb1 <ber_dupbv_x+145>, __next = 0x7fff50ffa0f0}}, __size = "\260\016\253UUU\000\000Xh\000@\377\177\000\000\360\240\377P\377\177\000\000\261\277\364\367\377\177\000\000\360\240\377P\377\177\000", __align = 93824997854896}}, be_pcsn_p = 0x7fff40006858, be_syncinfo = 0x555555ab0ee0, be_pb = 0x7fff40000b90, be_cf_ocs = 0x7fff50ffa140, be_private = 0x2fcab248a7e8d500, be_next = {stqe_next = 0x7fff50ffa210}} bi = 0x555555932da0 rc = 32768 #7 0x00007ffff6091391 in syncprov_op_search (op=<optimized out>, rs=0x7fff50ffb8d0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/overlays/syncprov.c:3381 e = 0x555555932da0 a = <optimized out> rc = <optimized out> sl = <optimized out> i = <optimized out> j = <optimized out> ctrls = <optimized out> on = 0x555555932da0 si = 0x555555932fa0 cb = <optimized out> gotstate = 0 changed = <optimized out> do_present = 0 sop = 0x7fff4010dbb0 ss = <optimized out> srs = 0x7fff400064a0 ctxcsn = 0x7fff40006828 i = <optimized out> sids = 0x7fff40006960 numcsns = <optimized out> mincsn = {bv_len = 40, bv_val = 0x7fff40006688 "20230802223107.484385Z#000000#002#000000"} maxcsn = <optimized out> minsid = <optimized out> maxsid = <optimized out> dirty = <optimized out> shortcut = <optimized out> #8 0x000055555561c0da in overlay_op_walk (op=op@entry=0x7fff40106490, rs=rs@entry=0x7fff50ffb8d0, which=which@entry=op_search, oi=0x55555577ab10, on=0x555555932da0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/backover.c:691 bi = 0x555555932da0 rc = 32768 #9 0x000055555561c294 in over_op_func (op=0x7fff40106490, rs=0x7fff50ffb8d0, which=op_search) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/backover.c:766 oi = <optimized out> on = <optimized out> be = 0x555555931040 db = {bd_info = 0x5555556f5b80 <slap_binfo+1440>, bd_self = 0x555555931040, be_ctrls = "\000\001\001\001\000\001\000\000\001\000\000\001\001\000\001\001\000\000\000\001\001\001\000\000\000\000\000\000\000\000\000\000\001", be_flags = 4757768, be_restrictops = 0, be_requires = 0, be_ssf_set = {sss_ssf = 0, sss_transport = 0, sss_tls = 0, sss_sasl = 0, sss_update_ssf = 0, sss_update_transport = 0, sss_update_tls = 0, sss_update_sasl = 0, sss_simple_bind = 0}, be_suffix = 0x555555935170, be_nsuffix = 0x5555559351a0, be_schemadn = {bv_len = 0, bv_val = 0x0}, be_schemandn = {bv_len = 0, bv_val = 0x0}, be_rootdn = {bv_len = 25, bv_val = 0x555555935380 "cn=Manager,dc=bpk2,dc=com"}, be_rootndn = {bv_len = 25, bv_val = 0x555555936490 "cn=manager,dc=bpk2,dc=com"}, be_rootpw = {bv_len = 38, bv_val = 0x5555559352b0 "{SSHA}tAHVQo521yiuLlTvbQ4QoHo9i5usOttD"}, be_max_deref_depth = 15, be_def_limit = {lms_t_soft = 3600, lms_t_hard = 0, lms_s_soft = -1, lms_s_hard = 0, lms_s_unchecked = -1, lms_s_pr = 0, lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits = 0x0, be_acl = 0x5555559351d0, be_dfltaccess = ACL_READ, be_extra_anlist = 0x0, be_lastbind_precision = 0, be_update_ndn = {bv_len = 0, bv_val = 0x0}, be_update_refs = 0x0, be_pcsn_st = {be_pcsn_list = {tqh_first = 0x0, tqh_last = 0x555555931180}, be_pcsn_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, be_pcsn_p = 0x555555931180, be_syncinfo = 0x555555937ad0, be_pb = 0x0, be_cf_ocs = 0x5555556fc5e0 <mdbocs.lto_priv+64>, be_private = 0x7ffff5be5010, be_next = {stqe_next = 0x0}} sc = <optimized out> cb = 0x7fff400067f8 rc = 32768 __PRETTY_FUNCTION__ = "over_op_func" #10 0x00005555555a8b61 in fe_op_search (op=0x7fff40106490, rs=0x7fff50ffb8d0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/search.c:426 bd = 0x555555753fe0 <slap_frontendDB> #11 0x00005555555ab1e4 in do_search (op=0x7fff40106490, rs=0x7fff50ffb8d0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/search.c:267 base = {bv_len = 14, bv_val = 0x7fff40003667 "dc=bpk2,dc=com"} siz = 2 off = 0 i = <optimized out> #12 0x00005555555a2fe5 in connection_operation (ctx=ctx@entry=0x7fff50ffbb70, arg_v=arg_v@entry=0x7fff40106490) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/connection.c:1115 rc = 80 cancel = <optimized out> op = 0x7fff40106490 rs = {sr_type = REP_RESULT, sr_tag = 0, sr_msgid = 0, sr_err = 0, sr_matched = 0x0, sr_text = 0x0, sr_ref = 0x0, sr_ctrls = 0x0, sr_un = {sru_search = {r_entry = 0x0, r_attr_flags = 0, r_operational_attrs = 0x0, r_attrs = 0x0, r_nentries = 0, r_v2ref = 0x0}, sru_sasl = {r_sasldata = 0x0}, sru_extended = {r_rspoid = 0x0, r_rspdata = 0x0}}, sr_flags = 0} tag = 99 opidx = SLAP_OP_SEARCH conn = 0x7ffff3268158 memctx = 0x7fff40000b90 memctx_null = 0x0 memsiz = <optimized out> __PRETTY_FUNCTION__ = "connection_operation" #13 0x00005555555a3bbb in connection_read_thread (ctx=0x7fff50ffbb70, argv=<optimized out>) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/connection.c:1267 rc = 0 cri = {op = 0x7fff40106490, func = 0x0, arg = 0x0, ctx = 0x7fff50ffbb70, nullop = <optimized out>} s = 19 #14 0x00007ffff7fa1af6 in ldap_int_thread_pool_wrapper (xpool=0x555555799e40) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/tpool.c:1059 pq = 0x555555799e40 pool = 0x555555799d30 task = 0x7fff4c000f10 work_list = <optimized out> ctx = {ltu_pq = 0x555555799e40, ltu_id = 140734552327872, ltu_key = {{ltk_key = 0x555555605030 <slap_sl_mem_init>, ltk_data = 0x7fff40000b90, ltk_free = 0x555555606ca0 <slap_sl_mem_destroy>}, {ltk_key = 0x555555599ca0 <operation_counter_init>, ltk_data = 0x7fff40003c60, ltk_free = 0x555555599620 <conn_counter_destroy>}, {ltk_key = 0x5555555b6b50 <slap_op_free>, ltk_data = 0x0, ltk_free = 0x5555555b0330 <slap_op_q_destroy>}, {ltk_key = 0x555555ae5290, ltk_data = 0x7fff401084a0, ltk_free = 0x55555565e580 <mdb_reader_free>}, {ltk_key = 0x5555556418d0 <search_stack>, ltk_data = 0x7fff371fa010, ltk_free = 0x555555641740 <search_stack_free>}, {ltk_key = 0x555555641800 <scope_chunk_get>, ltk_data = 0x7fff383fb010, ltk_free = 0x555555641700 <scope_chunk_free>}, {ltk_key = 0x0, ltk_data = 0x7fff4410ed90, ltk_free = 0x0}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0x0} <repeats 25 times>}} kctx = <optimized out> i = <optimized out> keyslot = <optimized out> hash = <optimized out> pool_lock = 0 freeme = 0 __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper" #15 0x00007ffff78ae907 in start_thread (arg=<optimized out>) at pthread_create.c:444 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737346463248, 7824176847303498060, 140734552327872, -128, 0, 140737272748720, 7824176847248972108, 7823950512947091788}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #16 0x00007ffff7934870 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals.
Thread 6 (Thread 0x7fff517fd6c0 (LWP 22294) "slapd"): #0 0x00007ffff78ab1d9 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555555799ea0) at futex-internal.c:57 sc_cancel_oldtype = 0 __arg6 = <optimized out> __arg3 = <optimized out> _a5 = <optimized out> _a2 = <optimized out> sc_ret = <optimized out> __arg4 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a3 = <optimized out> resultvar = <optimized out> __arg5 = <optimized out> __arg2 = <optimized out> _a4 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 #2 0x00007ffff78ab25f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139 No locals. #3 0x00007ffff78adb79 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x555555799e50, cond=0x555555799e78) at pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7ffff78ad8a0 <__condvar_cleanup_waiting>, __arg = 0x7fff517fcaf0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 664, cond = 0x555555799e78, mutex = 0x555555799e50, private = 0} err = <optimized out> g = 0 flags = <optimized out> g1_start = <optimized out> maxspin = 0 signals = <optimized out> result = 0 wseq = 664 seq = 332 private = 0 maxspin = <optimized out> err = <optimized out> result = <optimized out> wseq = <optimized out> g = <optimized out> seq = <optimized out> flags = <optimized out> private = <optimized out> signals = <optimized out> done = <optimized out> g1_start = <optimized out> spin = <optimized out> buffer = <optimized out> cbuffer = <optimized out> s = <optimized out> #4 ___pthread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at pthread_cond_wait.c:618 No locals. #5 0x00007ffff7fa0269 in ldap_pvt_thread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/thr_posix.c:294 No locals. #6 0x00007ffff7fa1ba7 in ldap_int_thread_pool_wrapper (xpool=0x555555799e40) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/tpool.c:1041 pq = 0x555555799e40 pool = 0x555555799d30 task = 0x0 work_list = <optimized out> ctx = {ltu_pq = 0x555555799e40, ltu_id = 140734560720576, ltu_key = {{ltk_key = 0x555555605030 <slap_sl_mem_init>, ltk_data = 0x7fff48000b90, ltk_free = 0x555555606ca0 <slap_sl_mem_destroy>}, {ltk_key = 0x555555599ca0 <operation_counter_init>, ltk_data = 0x7fff48106a00, ltk_free = 0x555555599620 <conn_counter_destroy>}, {ltk_key = 0x5555555b6b50 <slap_op_free>, ltk_data = 0x7fff481065b0, ltk_free = 0x5555555b0330 <slap_op_q_destroy>}, {ltk_key = 0x555555ae5290, ltk_data = 0x7fff48107920, ltk_free = 0x55555565e580 <mdb_reader_free>}, {ltk_key = 0x5555556418d0 <search_stack>, ltk_data = 0x7fff39bfe010, ltk_free = 0x555555641740 <search_stack_free>}, {ltk_key = 0x555555641800 <scope_chunk_get>, ltk_data = 0x7fff501fa010, ltk_free = 0x555555641700 <scope_chunk_free>}, {ltk_key = 0x0, ltk_data = 0x7fff4010e630, ltk_free = 0x0}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0x0} <repeats 25 times>}} kctx = <optimized out> i = <optimized out> keyslot = <optimized out> hash = <optimized out> pool_lock = 0 freeme = 0 __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper" #7 0x00007ffff78ae907 in start_thread (arg=<optimized out>) at pthread_create.c:444 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737346463248, 7824173549305485644, 140734560720576, -128, 0, 140737272748720, 7824173549250959692, 7823950512947091788}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007ffff7934870 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals.
Thread 5 (Thread 0x7fff51ffe6c0 (LWP 22293) "slapd"): #0 0x00007ffff78ab1d9 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555555799ea0) at futex-internal.c:57 sc_cancel_oldtype = 0 __arg6 = <optimized out> __arg3 = <optimized out> _a5 = <optimized out> _a2 = <optimized out> sc_ret = <optimized out> __arg4 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a3 = <optimized out> resultvar = <optimized out> __arg5 = <optimized out> __arg2 = <optimized out> _a4 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 #2 0x00007ffff78ab25f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139 No locals. #3 0x00007ffff78adb79 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x555555799e50, cond=0x555555799e78) at pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7ffff78ad8a0 <__condvar_cleanup_waiting>, __arg = 0x7fff51ffdaf0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 668, cond = 0x555555799e78, mutex = 0x555555799e50, private = 0} err = <optimized out> g = 0 flags = <optimized out> g1_start = <optimized out> maxspin = 0 signals = <optimized out> result = 0 wseq = 668 seq = 334 private = 0 maxspin = <optimized out> err = <optimized out> result = <optimized out> wseq = <optimized out> g = <optimized out> seq = <optimized out> flags = <optimized out> private = <optimized out> signals = <optimized out> done = <optimized out> g1_start = <optimized out> spin = <optimized out> buffer = <optimized out> cbuffer = <optimized out> s = <optimized out> #4 ___pthread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at pthread_cond_wait.c:618 No locals. #5 0x00007ffff7fa0269 in ldap_pvt_thread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/thr_posix.c:294 No locals. #6 0x00007ffff7fa1ba7 in ldap_int_thread_pool_wrapper (xpool=0x555555799e40) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/tpool.c:1041 pq = 0x555555799e40 pool = 0x555555799d30 task = 0x0 work_list = <optimized out> ctx = {ltu_pq = 0x555555799e40, ltu_id = 140734569113280, ltu_key = {{ltk_key = 0x555555605030 <slap_sl_mem_init>, ltk_data = 0x7fff3c000b90, ltk_free = 0x555555606ca0 <slap_sl_mem_destroy>}, {ltk_key = 0x555555599ca0 <operation_counter_init>, ltk_data = 0x7fff3c107580, ltk_free = 0x555555599620 <conn_counter_destroy>}, {ltk_key = 0x555555ae5290, ltk_data = 0x7fff3c10aa30, ltk_free = 0x55555565e580 <mdb_reader_free>}, {ltk_key = 0x5555556418d0 <search_stack>, ltk_data = 0x7fff3adff010, ltk_free = 0x555555641740 <search_stack_free>}, {ltk_key = 0x555555641800 <scope_chunk_get>, ltk_data = 0x7fff504fb010, ltk_free = 0x555555641700 <scope_chunk_free>}, {ltk_key = 0x5555555b6b50 <slap_op_free>, ltk_data = 0x7fff3c10a640, ltk_free = 0x5555555b0330 <slap_op_q_destroy>}, {ltk_key = 0x0, ltk_data = 0x7fff4410c000, ltk_free = 0x0}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0x0} <repeats 25 times>}} kctx = <optimized out> i = <optimized out> keyslot = <optimized out> hash = <optimized out> pool_lock = 0 freeme = 0 __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper" #7 0x00007ffff78ae907 in start_thread (arg=<optimized out>) at pthread_create.c:444 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737346463248, 7824174649353984332, 140734569113280, -128, 0, 140737272748720, 7824174649299458380, 7823950512947091788}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007ffff7934870 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals.
Thread 4 (Thread 0x7fff527ff6c0 (LWP 22292) "slapd"): #0 0x00007ffff78ab1d9 in __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555555799ea0) at futex-internal.c:57 sc_cancel_oldtype = 0 __arg6 = <optimized out> __arg3 = <optimized out> _a5 = <optimized out> _a2 = <optimized out> sc_ret = <optimized out> __arg4 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a3 = <optimized out> resultvar = <optimized out> __arg5 = <optimized out> __arg2 = <optimized out> _a4 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at futex-internal.c:87 err = <optimized out> clockbit = 256 op = 393 #2 0x00007ffff78ab25f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x555555799ea0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at futex-internal.c:139 No locals. #3 0x00007ffff78adb79 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x555555799e50, cond=0x555555799e78) at pthread_cond_wait.c:503 spin = 0 buffer = {__routine = 0x7ffff78ad8a0 <__condvar_cleanup_waiting>, __arg = 0x7fff527feaf0, __canceltype = 0, __prev = 0x0} cbuffer = {wseq = 666, cond = 0x555555799e78, mutex = 0x555555799e50, private = 0} err = <optimized out> g = 0 flags = <optimized out> g1_start = <optimized out> maxspin = 0 signals = <optimized out> result = 0 wseq = 666 seq = 333 private = 0 maxspin = <optimized out> err = <optimized out> result = <optimized out> wseq = <optimized out> g = <optimized out> seq = <optimized out> flags = <optimized out> private = <optimized out> signals = <optimized out> done = <optimized out> g1_start = <optimized out> spin = <optimized out> buffer = <optimized out> cbuffer = <optimized out> s = <optimized out> #4 ___pthread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at pthread_cond_wait.c:618 No locals. #5 0x00007ffff7fa0269 in ldap_pvt_thread_cond_wait (cond=cond@entry=0x555555799e78, mutex=mutex@entry=0x555555799e50) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/thr_posix.c:294 No locals. #6 0x00007ffff7fa1ba7 in ldap_int_thread_pool_wrapper (xpool=0x555555799e40) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/tpool.c:1041 pq = 0x555555799e40 pool = 0x555555799d30 task = 0x0 work_list = <optimized out> ctx = {ltu_pq = 0x555555799e40, ltu_id = 140734577505984, ltu_key = {{ltk_key = 0x555555605030 <slap_sl_mem_init>, ltk_data = 0x7fff44000b90, ltk_free = 0x555555606ca0 <slap_sl_mem_destroy>}, {ltk_key = 0x555555ae5290, ltk_data = 0x7fff441061d0, ltk_free = 0x55555565e580 <mdb_reader_free>}, {ltk_key = 0x555555599ca0 <operation_counter_init>, ltk_data = 0x7fff4410b6e0, ltk_free = 0x555555599620 <conn_counter_destroy>}, {ltk_key = 0x5555555b6b50 <slap_op_free>, ltk_data = 0x7fff4410b290, ltk_free = 0x5555555b0330 <slap_op_q_destroy>}, {ltk_key = 0x5555556418d0 <search_stack>, ltk_data = 0x7fff386fc010, ltk_free = 0x555555641740 <search_stack_free>}, {ltk_key = 0x555555641800 <scope_chunk_get>, ltk_data = 0x7fff398fd010, ltk_free = 0x555555641700 <scope_chunk_free>}, {ltk_key = 0x0, ltk_data = 0x7fff401102c0, ltk_free = 0x0}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0x0} <repeats 25 times>}} kctx = <optimized out> i = <optimized out> keyslot = <optimized out> hash = <optimized out> pool_lock = 0 freeme = 0 __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper" #7 0x00007ffff78ae907 in start_thread (arg=<optimized out>) at pthread_create.c:444 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737346463248, 7824171351355971916, 140734577505984, -128, 0, 140737272748720, 7824171351301445964, 7823950512947091788}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #8 0x00007ffff7934870 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals.
Thread 3 (Thread 0x7ffff32646c0 (LWP 22291) "slapd"): #0 0x00007ffff7934c72 in epoll_wait (epfd=6, events=events@entry=0x7ffff6dcd004, maxevents=65535, timeout=72000) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 sc_ret = -4 sc_cancel_oldtype = 0 sc_ret = <optimized out> #1 0x00005555555a1c9b in slapd_daemon_task (ptr=<optimized out>) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/daemon.c:2844 ns = <optimized out> at = <optimized out> nfds = <optimized out> revents = 0x7ffff6dcd004 tvp = <optimized out> cat = {tv_sec = 1691690555, tv_usec = 0} i = <optimized out> nwriters = <optimized out> now = <optimized out> tv = {tv_sec = 72, tv_usec = 0} tdelta = 1 rtask = <optimized out> l = <optimized out> last_idle_check = 1691690224 ebadf = 0 tid = <optimized out> ebuf = '\000' <repeats 127 times> loop = <optimized out> #2 0x00007ffff78ae907 in start_thread (arg=<optimized out>) at pthread_create.c:444 ret = <optimized out> pd = <optimized out> out = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737346463248, 7823958512641005900, 140737272760000, -128, 0, 140737488346336, 7823958512586479948, 7823950512947091788}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> #3 0x00007ffff7934870 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 No locals.
Thread 1 (Thread 0x7ffff7dbe140 (LWP 22284) "slapd"): #0 0x00007ffff78ab1d9 in __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=22291, futex_word=0x7ffff3264990) at futex-internal.c:57 sc_cancel_oldtype = 0 __arg6 = <optimized out> __arg3 = <optimized out> _a5 = <optimized out> _a2 = <optimized out> sc_ret = <optimized out> __arg4 = <optimized out> __arg1 = <optimized out> _a6 = <optimized out> _a3 = <optimized out> resultvar = <optimized out> __arg5 = <optimized out> __arg2 = <optimized out> _a4 = <optimized out> _a1 = <optimized out> #1 __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7ffff3264990, expected=22291, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128, cancel=cancel@entry=true) at futex-internal.c:87 err = <optimized out> clockbit = 256 op = 265 #2 0x00007ffff78ab25f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7ffff3264990, expected=<optimized out>, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at futex-internal.c:139 No locals. #3 0x00007ffff78b05e3 in __pthread_clockjoin_ex (threadid=140737272760000, thread_return=thread_return@entry=0x0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, block=block@entry=true) at pthread_join_common.c:102 ret = <optimized out> _buffer = {__routine = 0x7ffff78b04c0 <cleanup>, __arg = 0x7ffff3264ce0, __canceltype = 1433401664, __prev = 0x0} tid = <optimized out> pd = 0x7ffff32646c0 self = <optimized out> result = 0 pd_result = <optimized out> #4 0x00007ffff78b04b3 in ___pthread_join (threadid=<optimized out>, thread_return=thread_return@entry=0x0) at pthread_join.c:24 No locals. #5 0x00007ffff7fa01f9 in ldap_pvt_thread_join (thread=<optimized out>, thread_return=thread_return@entry=0x0) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/libraries/libldap/thr_posix.c:214 No locals. #6 0x00005555555a03fb in slapd_daemon () at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/daemon.c:3377 i = 0 rc = <optimized out> #7 0x0000555555581c99 in main (argc=<optimized out>, argv=0x7fffffffe1f8) at /usr/src/debug/openldap-2.6.4-1.fc38.x86_64/openldap-2.6.4/servers/slapd/main.c:869 i = <optimized out> no_detach = <optimized out> rc = 0 urls = <optimized out> username = <optimized out> groupname = <optimized out> sandbox = <optimized out> syslogUser = 160 pid = <optimized out> waitfds = {0, 0} g_argc = <optimized out> g_argv = 0x7fffffffe1f8 configfile = <optimized out> configdir = <optimized out> serverMode = 1 scp = <optimized out> scp_entry = <optimized out> serverNamePrefix = 0x5555556da917 "" l = <optimized out> slapd_pid_file_unlink = 1 slapd_args_file_unlink = 1 firstopt = <optimized out> __PRETTY_FUNCTION__ = "main"
seeing the message about the accesslog, i stop all slapd instances and removed the module. i wasnt using it, and did not have it leveraged anywhere, so i could just remove the ldif file from the filesystem and start services. in the 3 node cluster, node 2 was the one working, so i started that one first. nodes 1 and 3 were started subsequently and there were no coredumps.
interestingly, things did not go well beyond that. the replication seems to be started, but node 1 and 3 cannot be used. the DIT does not seem to be present, and all sorts of LDAP/Kerberos integrated services have issues. the 3 nodes are in a load balanced pool behind a VIP and "least connections" is the balancing method. any hit to node 1 or 3 results in missing data, bad auth, all the bad things.
on node 3 i tried to restart krb5kdc and kadmin, since i am using LDAP to backend Kerberos. i figured maybe they were in a weird state, but neither would restart. i had to stop slapd on node 1 and 3 to get things back up and functional.
to see what was going on, i tried modifying the log levels, and pointing to alternate ports on the VIP. port 389 load balances across all active nodes. ports 390, 391 and 392 point to the individual pool members only, so i can hit and test each instance. with all 3 nodes running only node 2 would take the modify operation and update the log levels. this update to node 2 was replicated to the other nodes and they moved to the new log level, but other wise, i cannot access the DIT on nodes 1 and 3.
it seems that something is still gumming up the works, even though the seemingly bad accesslog module was removed and services were restarted. if i start up node 1 and 3 and try to search with "dc=bpk2,dc=com" as the base, node 1 gives me back 24 numEntries, node 2 gives me back 164259 entries, and node 3 gives me back 25 numEntries.
clearly, something is out of whack with the replication now. what means can i employ to get things back up and running?
thanks,
brendan