Hi!
From the stack trace it seems your binary is either extremelyoptimized, or the stack got severely damanged. As Quanah said, running a binary with debug symbols might help.
Ulrich From: Suresh Veliveli Suresh.Veliveli@georgetown.edu Sent: Monday, November 11, 2024 7:09 PM To: openldap-technical@openldap.org Subject: [EXT] Master crashing when restarting Replicas
Hello,
We have a single master with multiple replicas. Sometimes when replicas are restarted the master is crashing.
Openldap version - 2.6.7 Backend Database - mdb
(gdb) core core.slapd.3003.d7d7f844eec3408ba1dfc465ee8306a1.16323.1731327494000000 [New LWP 16333] [New LWP 16741] [New LWP 16327] [New LWP 16328] [New LWP 16329] [New LWP 16360] [New LWP 16325] [New LWP 17359] [New LWP 16794] [New LWP 16323] [New LWP 16324] [New LWP 17765] [New LWP 16998] [New LWP 16334] [New LWP 17766] [New LWP 16999] [New LWP 16361] [New LWP 16795] Core was generated by `/var/services/openldap/libexec/slapd -h ldap://*:389 ldaps://*:636 -f /var/serv'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00000000004389bb in ?? () [Current thread is 1 (LWP 16333)] (gdb) bt full #0 0x00000000004389bb in ?? () No symbol table info available. #1 0x0000000000000000 in ?? () No symbol table info available. (gdb) q
Nov 11 07:18:14 aaa-prod-master-1 kernel: slapd[16333]: segfault at 18 ip 00000000004389bb sp 00007f79ccaf8650 error 4 in slapd[419000+12d000] likely on CPU 1 (core 1, socket 0) Nov 11 07:18:14 aaa-prod-master-1 kernel: Code: 89 e8 b9 0e 00 00 00 4c 89 e7 f3 48 ab 48 89 e8 48 8b ad 90 01 00 00 8b 90 c0 00 00 00 85 d2 75 35 48 8b 10 4c 89 e6 4c 89 f7 <8b> 52 18 c7 80 c0 00 00 00 01 00 00 00 48 8b 05 91 82 1b 00 89 94 Nov 11 07:18:14 aaa-prod-master-1 systemd[1]: Created slice Slice /system/systemd-coredump. Nov 11 07:18:14 aaa-prod-master-1 systemd[1]: Started Process Core Dump (PID 349553/UID 0). Nov 11 07:18:17 aaa-prod-master-1 systemd-coredump[349554]: Process 16323 (slapd) of user 3003 dumped core.#012#012Stack trace of thread 16333:#012#0 0x00000000004389bb n/a (slapd + 0x389bb)#012#1 0x000000000043a6f4 connection_closing (slapd + 0x3a6f4)#012#2 0x000000000043b5ce n/a (slapd + 0x3b5ce)#012#3 0x00007f9340de7d1a ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43d1a)#012#4 0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#5 0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 16741:#012#0 0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1 0x00007f9340488fa0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x88fa0)#012#2 0x00007f9340de7db7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43db7)#012#3 0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#4 0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 16327:#012#0 0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1 0x00007f9340488fa0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x88fa0)#012#2 0x00007f9340de7db7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43db7)#012#3 0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#4 0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 16328:#012#0 0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1 0x00007f9340488fa0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x88fa0)#012#2 0x00007f9340de7db7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43db7)#012#3 0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#4 0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 16329:#012#0 0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1 0x00007f9340488fa0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x88fa0)#012#2 0x00007f9340de7db7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43db7)#012#3 0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#4 0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 16360:#012#0 0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1 0x00007f9340488fa0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x88fa0)#012#2 0x00007f9340de7db7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43db7)#012#3 0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#4 0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 16325:#012#0 0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1 0x00007f9340488fa0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x88fa0)#012#2 0x00007f9340de7db7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43db7)#012#3 0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#4 0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 17359:#012#0 0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1 0x00007f9340488fa0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x88fa0)#012#2 0x00007f9340de7db7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43db7)#012#3 0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#4 0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 16794:#012#0 0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1 0x00007f9340488fa0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x88fa0)#012#2 0x00007f9340de7db7 ldap_int_thread_pool_wrapper (libldap.so.2 + 0x43db7)#012#3 0x00007f9340489c02 start_thread (libc.so.6 + 0x89c02)#012#4 0x00007f934050ec40 __clone3 (libc.so.6 + 0x10ec40)#012#012Stack trace of thread 16323:#012#0 0x00007f934048679a __futex_abstimed_wait_common (libc.so.6 + 0x8679a)#012#1 0x00007f934048b6d3 __pthread_clockjoin_ex (libc.so.6 + 0x8b6d3)#012#2 0x00000000004381ca slapd_daemon (slapd + 0x381ca)#012#3 0x000000000041e47f main (slapd + 0x1e47f)#012#4 0x00007f9340429590 __libc_start_call_main (libc.so.6 + 0x29590)#012#5 0x00007f9340429640 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29640)#012#6 0x000000000041e855 _start (slapd + 0x1e855Nov 11 07:18:17 aaa-prod-master-1 systemd[1]: systemd-coredump@0-349553-0.service: Deactivated successfully. Nov 11 07:18:17 aaa-prod-master-1 systemd[1]: systemd-coredump@0-349553-0.service: Consumed 3.288s CPU time. Nov 11 07:18:18 aaa-prod-master-1 systemd[1]: slapd.master.service: Main process exited, code=dumped, status=11/SEGV Nov 11 07:18:18 aaa-prod-master-1 systemd[1]: slapd.master.service: Failed with result 'core-dump'. Nov 11 07:18:18 aaa-prod-master-1 systemd[1]: slapd.master.service: Consumed 14h 31min 2.574s CPU time.
Any thoughts?
Thanks, -- Suresh Veliveli Sr. Systems Engineer Georgetown University University Information Services | Security Infrastructure and Policy-Identity and Collaboration