Hi!

From the stack trace it seems your binary is either extremelyoptimized, or the stack got severely damanged.

As Quanah said, running a binary with debug symbols might help.

Ulrich

From: Suresh Veliveli <Suresh.Veliveli@georgetown.edu>
Sent: Monday, November 11, 2024 7:09 PM
To: openldap-technical@openldap.org
Subject: [EXT] Master crashing when restarting Replicas

Hello,

We have a single master with multiple replicas. Sometimes when replicas are restarted the master is crashing. 

Openldap version - 2.6.7

Backend Database - mdb

(gdb) core core.slapd.3003.d7d7f844eec3408ba1dfc465ee8306a1.16323.1731327494000000
[New LWP 16333]
[New LWP 16741]
[New LWP 16327]
[New LWP 16328]
[New LWP 16329]
[New LWP 16360]
[New LWP 16325]
[New LWP 17359]
[New LWP 16794]
[New LWP 16323]
[New LWP 16324]
[New LWP 17765]
[New LWP 16998]
[New LWP 16334]
[New LWP 17766]
[New LWP 16999]
[New LWP 16361]
[New LWP 16795]

Core was generated by `/var/services/openldap/libexec/slapd -h ldap://*:389 ldaps://*:636 -f /var/serv'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00000000004389bb in ?? ()
[Current thread is 1 (LWP 16333)]
(gdb) bt full
#0  0x00000000004389bb in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) q

Nov 11 07:18:14 aaa-prod-master-1 kernel: slapd[16333]: segfault at 18 ip 00000000004389bb sp 00007f79ccaf8650 error 4 in slapd[419000+12d000] likely on CPU 1 (core 1, socket 0)
Nov 11 07:18:14 aaa-prod-master-1 kernel: Code: 89 e8 b9 0e 00 00 00 4c 89 e7 f3 48 ab 48 89 e8 48 8b ad 90 01 00 00 8b 90 c0 00 00 00 85 d2 75 35 48 8b 10 4c 89 e6 4c 89 f7 <8b> 52 18 c7 80 c0 00 00 00 01 00 00 00 48 8b 05 91 82 1b 00 89 94
Nov 11 07:18:14 aaa-prod-master-1 systemd[1]: Created slice Slice /system/systemd-coredump.
Nov 11 07:18:14 aaa-prod-master-1 systemd[1]: Started Process Core Dump (PID 349553/UID 0).
Nov 11 07:18:17 aaa-prod-master-1 systemd-coredump[349554]: Process 16323 (slapd) of user 3003 dumped core.#012#012Stack trace of thread 16333:#012#0  0x00000000004389bb n/a (slapd + 0x389bb)#012#1  0x000000000043a6f4 connection_closing (slapd + 0x3a6f4)#012#2  0x000000000043b5ce n/a (slapd + 0x3b5ce)#012#3  0x00007f9340de7d1a ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43d1a)#012#4  0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#5  0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 16741:#012#0  0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1  0x00007f9340488fa0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x88fa0)#012#2  0x00007f9340de7db7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43db7)#012#3  0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#4  0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 16327:#012#0  0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1  0x00007f9340488fa0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x88fa0)#012#2  0x00007f9340de7db7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43db7)#012#3  0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#4  0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 16328:#012#0  0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1  0x00007f9340488fa0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x88fa0)#012#2  0x00007f9340de7db7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43db7)#012#3  0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#4  0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 16329:#012#0  0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1  0x00007f9340488fa0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x88fa0)#012#2  0x00007f9340de7db7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43db7)#012#3  0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#4  0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 16360:#012#0  0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1  0x00007f9340488fa0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x88fa0)#012#2  0x00007f9340de7db7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43db7)#012#3  0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#4  0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 16325:#012#0  0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1  0x00007f9340488fa0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x88fa0)#012#2  0x00007f9340de7db7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43db7)#012#3  0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#4  0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 17359:#012#0  0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1  0x00007f9340488fa0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x88fa0)#012#2  0x00007f9340de7db7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43db7)#012#3  0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#4  0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 16794:#012#0  0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1  0x00007f9340488fa0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x88fa0)#012#2  0x00007f9340de7db7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43db7)#012#3  0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#4  0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 16323:#012#0  0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1  0x00007f934048b6d3 __pthread_clockjoin_ex (libc.so.6 + 0x8b6d3)#012#2  0x00000000004381ca slapd_daemon (slapd + 0x381ca)#012#3  0x000000000041e47f main (slapd + 0x1e47f)#012#4  0x00007f9340429590 __libc_start_call_main (libc.so.6 + 0x29590)#012#5  0x00007f9340429640 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29640)#012#6  0x000000000041e855 _start (slapd + 0x1e855Nov 11 07:18:17 aaa-prod-master-1 systemd[1]: systemd-coredump@0-349553-0.service: Deactivated successfully.

Nov 11 07:18:17 aaa-prod-master-1 systemd[1]: systemd-coredump@0-349553-0.service: Consumed 3.288s CPU time.
Nov 11 07:18:18 aaa-prod-master-1 systemd[1]: slapd.master.service: Main process exited, code=dumped, status=11/SEGV
Nov 11 07:18:18 aaa-prod-master-1 systemd[1]: slapd.master.service: Failed with result 'core-dump'.
Nov 11 07:18:18 aaa-prod-master-1 systemd[1]: slapd.master.service: Consumed 14h 31min 2.574s CPU time.

Any thoughts?

Thanks,
--

Suresh Veliveli

Sr. Systems Engineer

Georgetown University

University Information Services | Security Infrastructure and Policy-Identity and Collaboration