Hi fellows
my OpenLDAP keeps on dying sporadically. Unfortunately it doesn't leave ANY trace of why it crashes. It stays alive for a while .... takes a couple of queries as expected ... and then suddenly it dies after another request. This could mean working perfectly for a day or working only for a couple of minutes before it crashes. The requests are all known to work (Dovecot auth and system PAM auth). At first I thought it could be related to newsyslog(8) ... I did some testing - unfortunately this doesn't change anything at all. It keeps on dying sporadically even if newsyslog is completely deactivated for the slapd(8) log file.
I experenced this behauviour since I first used OpenLDAP approximately 5 years ago. My OpenLDAP configurations have been changed many times and quite a lot over those 5 years. Unfortunately I was never able to figure out what the sudden crashes caused - so I wrote a supervisor script , which periodically checks if the slapd(8) is still running. If slapd(8) is not running, then my supervisor script starts it automatically again. Of course with every major change in the configuration of OpenLDAP, I also tried to debug this problem of the sudden crashes - but as I said: even with all debug levels activated - slapd(8) didn't want to share its root of the pain. This workaround was the only way I could use OpenLDAP at all over the past five years.
Two days ago I got to the point where my workaround came to its capacity boundary. The time between the slapd(8) crashes has become so little, so my script can't handle them anylonger. I would have two options now:
a) Fix my supervisor script, so it detects and starts slapd(8) even if the time between crashes is less than 3 sec. b) I ask for help in order to discover this mysterious cause of the crashes and FINALLY fix it FOREVER.
So this time I set my foot down and go for b). This is why I decided to contact you. The following link containes a more detailed debugging of the problem:
https://forums.freebsd.org/threads/openldap-slapd-dies-sporadically.47634/
And this is the current / latest configuration I deploy on my servers. Also, it doesn't make a difference whether I have monitoring activated or not. It keeps on dieing the same as before. The very same ocours to the "ldapab.schema" - it doesn't make a difference for the crashes to happen whether it is commented out or activated. Logging hapens through syslogd(8).
# ============================================================ #
#======================================# # slapd.conf # #======================================#
# LDAP Attribut Schema Definitions: include /usr/local/etc/openldap/schema/core.schema include /usr/local/etc/openldap/schema/cosine.schema include /usr/local/etc/openldap/schema/inetorgperson.schema include /usr/local/etc/openldap/schema/nis.schema include /usr/local/etc/openldap/schema/ldapab.schema include /usr/local/etc/openldap/schema/mail.schema #include /usr/local/etc/openldap/schema/qouta.schema
# Logging loglevel 256 # If "logfile" is activated, then newsyslog must restart slapd after rotation #logfile /var/log/slapd.log
# Important Files pidfile /var/run/openldap/slapd.pid argsfile /var/run/openldap/slapd.args
# Allow LDAPv2 client connections. This is required for iOS suppoprt via SSL. #allow bind_v2
# Time limits #idletimeout 0 #writetimeout 0 #timelimit 3600
# Dynamic Modules: modulepath /usr/local/libexec/openldap moduleload back_mdb moduleload back_monitor
# Access from 127.0.0.1 without encryption access to dn.subtree="dc=MyDomain,dc=local" by peername.ip=127.0.0.1 write by * none break
# Access from other than 127.0.0.1 requires TLS # encryption with min. 128 bit encryption access to dn.subtree="dc=MyDomain,dc=local" by ssf=128 write by * none
# TLS-Certificate TLSCertificateFile /etc/ssl/PKI/CA/Signing-CA/Certs/WM-01.MyDomain.Local.crt TLSCertificateKeyFile /etc/ssl/PKI/CA/Signing-CA/Private/WM-01.MyDomain.Local.key TLSVerifyClient allow
# Restrict write access for password attributes to root user only by dn="uid=dovecot,ou=systemuser,ou=mail,dc=MyDomain,dc=Local" access to attrs=userPassword by self write by anonymous auth by * none
# All the other atributes can be read by everyone else access to * by * read
# ================================== # # Internal Monitoring # # ================================== #
# DB type database monitor
# Name of Administrator rootdn "cn=monitoring,cn=Monitor"
# Password of Administrators rootpw {SSHA}SomeHash
# ACL for moitoring access to dn.subtree="cn=Monitor" by dn.exact="cn=admin,dc=MyDomain,dc=local" write by peername.ip=127.0.0.1 read by users read by * none
# ================================== # # Lightning Memory-Mapped Database # # ================================== #
# Datenbanktyp database mdb
# DB location on FS directory /var/db/openldap-data
# Datenbank-Pfad suffix "dc=MyDomain,dc=local"
# Name of Administrator rootdn "cn=admin,dc=MyDomain,dc=local"
# Password of Administrators rootpw {SSHA}SomeHash
# Max size of DB in Byte [ 5368709120 Byte => 5 GB ] maxsize 5368709120
# Index search related attributes index objectClass eq index uid eq index uidNumber eq index uniqueMember eq index gidNumber eq index cn eq index memberUid eq index mailAccountStatus eq index mailAddress eq index associatedDomain eq
# ============================================================= #
I compiled OpenLDAP from FreeBSD ports with following options: - DYNAMIC_BACKENDS - MDB - PPOLICY - SYNCPROV - TCP_WRAPPERS
Please let me know what kind of further information I could deliver to you in order to detect the root of the problem. Thank you
Best regards Leander
Leander Schäfer wrote:
my OpenLDAP keeps on dying sporadically.
Which OpenLDAP version is this?
Please note that OpenLDAP project does not know details about binary distribution releases.
Did you try with your own build from source using a recent OpenLDAP release?
Unfortunately it doesn't leave ANY trace of why it crashes.
I'm not familiar with FreeBSD. But try this to get more details:
1. Run with ulimit -c unlimited to get a core dump file.
2. Provide gdb stack trace like described herein: http://www.openldap.org/faq/data/cache/59.html
Ciao, Michael.
Hi Michael, Hi Ulrich,
Thanks for your reply. I'm running Version 2.4.40. As I said, I do not run binary version. I always compile OpenLDAP from sources / latest ports tree.
The maximum core file size "ulimit -c" was already set to unlimited. I'm using bash-4.3.30
root@FreeBASD # ulimit -a socket buffer size (bytes, -b) unlimited core file size (blocks, -c) unlimited data seg size (kbytes, -d) 33554432 file size (blocks, -f) unlimited max locked memory (kbytes, -l) 131072 max memory size (kbytes, -m) 3012064 open files (-n) 87651 pipe size (512 bytes, -p) 1 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 7592 virtual memory (kbytes, -v) unlimited swap size (kbytes, -w) unlimited
root@FreeBSD # /usr/local/libexec/slapd -d -1 -f /usr/local/etc/openldap/slapd.conf -u ldap -g ldap -h "ldapi://%2fvar%2frun%2fopenldap%2fldapi/ ldap:/// ldaps:///"
# =================================================================== #
[...] I have removed the previous lines for this demo [...]
553ded5a mailUidNumber 0040: 2e e8 e7 5a 9a fe a0 8c 02 96 e9 58 48 e0 49 10 ...Z.......XH.I. 0050: cd 10 08 6c 20 1f 9d bc ae dd 9a 4a 79 7e f2 3c ...l ......Jy~.< 0060: 25 a2 72 fe ac cc d0 09 eb 62 d2 bd 95 c8 50 7f %.r......b....P. 0020: b0 86 e3 1d 11 32 2d 8b fd 57 a6 a4 ce a2 ee 2f .....2-..W...../ 0030: 52 59 da 4d 7b d1 5c 0c 22 34 29 86 c2 9c 80 72 RY.M{.."4)....r 0040: ca 94 4d 69 2e ..Mi. 0010: 73 2c 6f 75 3d 6d 61 69 6c 2c 64 63 3d 4e 65 74 s,ou=mail,dc=MyD 0020: 4f 63 65 61 6e 2c 64 63 3d 4c 6f 63 61 6c 0a 01 omain,dc=Local.. 0090: 8e 5f 68 e0 0a 31 26 07 da 21 c6 cd 27 0e 17 2b ._h..1&..!..'..+ 00a0: fb 53 5e 0a 84 74 50 b8 74 13 a5 fa e2 02 9a ee .S^..tP.t....... 00b0: 5e ee 8e 6c b2 d3 b6 6e 82 6d 01 ab eb 81 25 bd ^..l...n.m....%. 00c0: f1 05 16 5b 7f 9e bb 76 7c ae ba a2 24 73 89 78 ...[...v|...$s.x 0070: ca 00 ec a3 7b 97 78 19 fe aa 56 fc a1 a5 9a 1e ....{.x...V..... 0080: 65 f0 04 b7 04 08 af 7a 82 ef 77 e......z..w 553ded5a mailGidNumberldap_read: want=8, got=8 553ded5a mailQuotaStorage 0000: 30 71 02 01 0b 63 6c 04 0q...cl. ldap_read: want=107, got=107 00d0: 6c ae 39 eb 15 85 4a f9 c1 6f 65 ca 4f c6 db 14 l.9...J..oe.O... ldap_write: want=14, written=14 00e0: c1 f4 fe b8 b5 a4 a3 75 96 b1 9b 9b 8f d5 d6 e5 .......u........ 00f0: 8d 3c 75 4c 50 cc 9d 85 cf bb a1 d8 50 21 93 fa .<uLP.......P!.. 0100: 38 ee 89 46 45 4e 06 17 7a 8c 4b 83 51 95 a9 ca 8..FEN..z.K.Q... 0110: 71 8c d0 b9 59 1a 14 f4 10 8d b9 bc 80 d5 cb e9 q...Y........... 0120: 46 03 a2 ce 59 49 0b db fc ea a3 3b fa cd a1 99 F...YI.....;.... 0030: 02 0a 01 00 02 01 00 02 01 78 01 01 00 a0 5a a3 .........x....Z. 0040: 1a 04 0b 6f 62 6a 65 63 74 43 6c 61 73 73 04 0b ...objectClass.. 0130: 14 90 d6 0a 55 da 84 b1 42 fe af 8d 14 92 ce 27 ....U...B......' 0050: 6d 61 69 6c 41 63 63 6f 75 6e 74 a3 1b 04 11 6d mailAccount....m 0060: 61 69 6c 41 63 63 6f 75 6e 74 53 74 61 74 75 73 ailAccountStatus 0000: 30 0c 02 01 06 65 07 0a 01 00 04 00 04 00 0....e........ 0070: 04 06 61 63 74 69 76 65 a3 1f 04 0b 6d 61 69 6c ..active....mail 0080: 41 64 64 72 65 73 73 04 10 69 6e 66 6f 40 6e 65 Address..info@So 0090: 74 6f 63 65 61 6e 2e 64 65 30 6f 04 14 6d 61 69 meDom.tld0o..mai 00a0: 6c 53 74 6f 72 61 67 65 44 69 72 65 63 74 6f 72 lStorageDirector 553ded5a mailQuotaMessages553ded5a 00b0: 79 04 14 6d 61 69 6c 53 74 6f 72 61 67 65 44 69 y..mailStorageDi 553ded5a conn=1008 op=5 SRCH base="ou=accounts,ou=mail,dc=MyDomain,dc=Local" scope=2 deref=0 filter="(&(objectClass=mailAccount)(mailAccountStatus=active)(mailAddress=info@somedom.tld))" 00c0: 72 65 63 74 6f 72 79 04 0d 6d 61 69 6c 55 69 64 rectory..mailUid 0000: 28 6f 75 3d 61 63 63 6f 75 6e 74 73 2c 6f 75 3d (ou=accounts,ou= 553ded5a conn=1008 op=4 SEARCH RESULT tag=101 err=0 nentries=0 text= 00d0: 4e 75 6d 62 65 72 04 0d 6d 61 69 6c 47 69 64 4e Number..mailGidN 553ded5a conn=1008 op=5 SRCH attr=mailStorageDirectory mailStorageDirectory mailUidNumber mailGidNumber mailQuotaStorage mailQuotaMessages ldap_read: want=8, got=8 0000: 30 82 01 0d 02 01 09 63 0......c ldap_read: want=265, got=265 0000: 82 01 06 04 28 6f 75 3d 61 63 63 6f 75 6e 74 73 ....(ou=accounts 0010: 2c 6f 75 3d 6d 61 69 6c 2c 64 63 3d 4e 65 74 4f ,ou=mail,dc=MyDo 553ded5a ==> limits_get: conn=1008 op=5 self="uid=dovecot,ou=systemuser,ou=mail,dc=mydomain,dc=local" this="ou=accounts,ou=mail,dc=mydomain,dc=local" 00e0: 75 6d 62 65 72 04 10 6d 61 69 6c 51 75 6f 74 61 umber..mailQuota Segmentation fault
# =================================================================== #
... and a second time the same debugging (-d -1) # =================================================================== #
[...]
553df2b8 conn=1000 op=6 SRCH base="ou=accounts,ou=mail,dc=MyDomain,dc=Local" scope=2 deref=0 filter="(&(objectClass=mailAccount)(mailAccountStatus=active)(mailAddress=info@somedom.tld))" 0010: 70 7b e7 97 2e b3 3e 24 f8 bc 52 c5 ce 0d b8 fb p{....>$..R..... 0020: 89 cb 07 f8 d3 34 0a 39 9a fc 0d 0d b9 bb d4 13 .....4.9........ 0030: 8d 1f 56 87 c0 9c 10 2b a2 57 ee 3b 4b 93 6a 7c ..V....+.W.;K.j| TLS trace: SSL_accept:SSLv3 read finished A 553df2b8 conn=1000 op=6 SRCH attr=mailStorageDirectory mailStorageDirectory mailUidNumber mailGidNumber mailQuotaStorage mailQuotaMessages 553df2b8 connection_read(20): checking for input on id=1006 553df2b8 ==> limits_get: conn=1000 op=6 self="uid=dovecot,ou=systemuser,ou=mail,dc=mydomain,dc=local" this="ou=accounts,ou=mail,dc=mydomain,dc=local" 0040: 54 2b b2 49 7e f7 ce 89 19 96 fe 83 86 77 5a 3a T+.I~........wZ: Segmentation fault
# =================================================================== #
I'm currently trying to figure out a way to add "-O0 -g" gdb(1) support for the slapd(8C) command, since I'm compiling OpenLDAP sources from ports tree. I'll post it as soon as I got it. But maybe you can already assume some pre-diagnosis from the above output?
@Ulrich: - Yes, I have tried several hardware plattforms, different physical, VmWare and now the latest XenServer. Same results all over. - A detailed log debug can be seen on: https://forums.freebsd.org/threads/openldap-slapd-dies-sporadically.47634/ - Be warned from the above diagnosis in the forum: It is NOT the assumed newsyslog(8) which causes the crashes! - What exactly do you mean by "exists cleanly"?
Best regards Leander
Am 27.04.15 um 08:59 schrieb Ulrich Windl:
Hi!
You should tell about the operation you do when slapd crashes, maybe as well the last log messages you see. Also: Did you try different hardware and different compilers to run and compile slapd? What about doing an "-O0 -g" build and enable core dumps? Or are you telling slapd exits "cleanly"?
Regards, Ulrich
Am 26.04.15 um 13:48 schrieb Michael Ströder:
Leander Schäfer wrote:
my OpenLDAP keeps on dying sporadically.
Which OpenLDAP version is this?
Please note that OpenLDAP project does not know details about binary distribution releases.
Did you try with your own build from source using a recent OpenLDAP release?
Unfortunately it doesn't leave ANY trace of why it crashes.
I'm not familiar with FreeBSD. But try this to get more details:
Run with ulimit -c unlimited to get a core dump file.
Provide gdb stack trace like described herein:
http://www.openldap.org/faq/data/cache/59.html
Ciao, Michael.
Leander Schäferinfo@netocean.de schrieb am 27.04.2015 um 11:02 in
Nachricht 553DFB24.5050201@netocean.de:
[...]
I'm currently trying to figure out a way to add "-O0 -g" gdb(1) support for the slapd(8C) command, since I'm compiling OpenLDAP sources from ports tree. I'll post it as soon as I got it. But maybe you can already assume some pre-diagnosis from the above output?
No, but you should have a core file after segmentation fault. What about something like "gdb /your_slapd /your/core" and "bt" and "info threads"? Probably not very useful unless you have debugging symbols in the binary.
Regards, Ulrich
I'll deliver the gdb(1) results as soon as I'm able to compile with relevant options. Meanwhile, here is some more debugging results about the break down of 'slapd -d -1 [...]'
# ======================================================================== #
[...]
553dfcc9 filter: (&(associatedDomain= somedom.tld)(!(destinationIndicator=*))) ber_scanf fmt ({M}}) ber: ber_dump: buf=0x94b40f020 ptr=0x94b40f09a end=0x94b40f0ae len=20 0000: 00 12 04 10 61 73 73 6f 63 69 61 74 65 64 44 6f ....associatedDo 0010: 6d 61 69 6e main 553dfcc9 attrs:553dfcc9 associatedDomain553dfcc9 553dfcc9 conn=1007 op=331 SRCH base="ou=domains,ou=mail,dc=MyDomain,dc=Local" scope=2 deref=0 filter="(&(associatedDomain=somedom.tld)(!(destinationIndicator=*)))" ber_get_next: tag 0x30 len 105 contents: ber_dump: buf=0x943d502b0 ptr=0x943d502b0 end=0x943d50319 len=105 0000: 02 01 0c 63 64 04 28 6f 75 3d 61 63 63 6f 75 6e ...cd.(ou=accoun 0010: 74 73 2c 6f 75 3d 6d 61 69 6c 2c 64 63 3d 4e 65 ts,ou=mail,dc=Ne 0020: 74 4f 63 65 61 6e 2c 64 63 3d 4c 6f 63 61 6c 0a tOcean,dc=Local. 0030: 01 02 0a 01 00 02 01 00 02 01 0a 01 01 00 a3 1c ................ 0040: 04 0b 6d 61 69 6c 41 64 64 72 65 73 73 04 0d 40 ..mailAddress..@ 0050: 6e 65 74 6f 63 65 61 6e 2e 64 65 20 30 0b 04 09 somedom.tld 0... 0060: 6d 61 69 6c 41 6c 69 61 73 mailAlias 553dfcc9 conn=1007 op=331 SRCH attr=associatedDomain 553dfcc9 op tag 0x63, time 1430125769 ber_get_next 553dfcc9 ==> limits_get: conn=1007 op=331 self="[anonymous]" this="ou=domains,ou=mail,dc=mydomain,dc=local" tls_read: want=5 error=Resource temporarily unavailable ldap_read: want=8 error=Resource temporarily unavailable Segmentation fault
# ======================================================================== #
[...]
553dfd7c daemon: select: listen=6 active_threads=0 tvp=NULL 553dfd7c conn=1008 op=2 SRCH attr=associatedDomain TLS trace: SSL_accept:SSLv3 read client key exchange A TLS trace: SSL_accept:SSLv3 read finished A tls_read: want=5, got=5 553dfd7c ==> limits_get: conn=1008 op=2 self="[anonymous]" this="ou=domains,ou=mail,dc=mydomain,dc=local" 553dfd7c daemon: select: listen=7 active_threads=0 tvp=NULL 553dfd7c daemon: select: listen=8 active_threads=0 tvp=NULL 553dfd7c daemon: select: listen=9 active_threads=0 tvp=NULL 553dfd7c => mdb_equality_candidates (associatedDomain) Segmentation fault
# ======================================================================== #
[...]
553dfdf3 => mdb_list_candidates 0xa1 553dfdf3 => mdb_filter_candidates 553dfdf3 EQUALITY 553dfdf3 => mdb_equality_candidates (objectClass) 553dfdf3 => key_read 553dfdf3 mdb_idl_fetch_key: [b49d1940] 553dfdf3 <= mdb_index_read: failed (-30798) 553dfdf3 <= mdb_equality_candidates: id=0, first=0, last=0 553dfdf3 <= mdb_filter_candidates: id=0 first=0 last=0 553dfdf3 => mdb_filter_candidates 553dfdf3 EQUALITY 553dfdf3 conn=1002 op=1131 SRCH attr=associatedDomain 553dfdf3 => mdb_equality_candidates (mailAddress) 553dfdf3 => key_read 553dfdf3 mdb_idl_fetch_key: [a4af5673] 553dfdf3 ==> limits_get: conn=1002 op=1131 self="[anonymous]" this="ou=domains,ou=mail,dc=mydomain,dc=local" 553dfdf3 <= mdb_index_read: failed (-30798) 553dfdf3 <= mdb_equality_candidates: id=0, first=0, last=0 553dfdf3 <= mdb_filter_candidates: id=0 first=0 last=0 553dfdf3 <= mdb_list_candidates: id=0 first=0 last=0 553dfdf3 <= mdb_filter_candidates: id=0 first=0 last=0 Segmentation fault
Am 27.04.15 um 11:02 schrieb Leander Schäfer:
Hi Michael, Hi Ulrich,
Thanks for your reply. I'm running Version 2.4.40. As I said, I do not run binary version. I always compile OpenLDAP from sources / latest ports tree.
The maximum core file size "ulimit -c" was already set to unlimited. I'm using bash-4.3.30
root@FreeBASD # ulimit -a socket buffer size (bytes, -b) unlimited core file size (blocks, -c) unlimited data seg size (kbytes, -d) 33554432 file size (blocks, -f) unlimited max locked memory (kbytes, -l) 131072 max memory size (kbytes, -m) 3012064 open files (-n) 87651 pipe size (512 bytes, -p) 1 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 7592 virtual memory (kbytes, -v) unlimited swap size (kbytes, -w) unlimited
root@FreeBSD # /usr/local/libexec/slapd -d -1 -f /usr/local/etc/openldap/slapd.conf -u ldap -g ldap -h "ldapi://%2fvar%2frun%2fopenldap%2fldapi/ ldap:/// ldaps:///"
# =================================================================== #
[...] I have removed the previous lines for this demo [...]
553ded5a mailUidNumber 0040: 2e e8 e7 5a 9a fe a0 8c 02 96 e9 58 48 e0 49 10 ...Z.......XH.I. 0050: cd 10 08 6c 20 1f 9d bc ae dd 9a 4a 79 7e f2 3c ...l ......Jy~.< 0060: 25 a2 72 fe ac cc d0 09 eb 62 d2 bd 95 c8 50 7f %.r......b....P. 0020: b0 86 e3 1d 11 32 2d 8b fd 57 a6 a4 ce a2 ee 2f .....2-..W...../ 0030: 52 59 da 4d 7b d1 5c 0c 22 34 29 86 c2 9c 80 72 RY.M{.."4)....r 0040: ca 94 4d 69 2e ..Mi. 0010: 73 2c 6f 75 3d 6d 61 69 6c 2c 64 63 3d 4e 65 74 s,ou=mail,dc=MyD 0020: 4f 63 65 61 6e 2c 64 63 3d 4c 6f 63 61 6c 0a 01 omain,dc=Local.. 0090: 8e 5f 68 e0 0a 31 26 07 da 21 c6 cd 27 0e 17 2b ._h..1&..!..'..+ 00a0: fb 53 5e 0a 84 74 50 b8 74 13 a5 fa e2 02 9a ee .S^..tP.t....... 00b0: 5e ee 8e 6c b2 d3 b6 6e 82 6d 01 ab eb 81 25 bd ^..l...n.m....%. 00c0: f1 05 16 5b 7f 9e bb 76 7c ae ba a2 24 73 89 78 ...[...v|...$s.x 0070: ca 00 ec a3 7b 97 78 19 fe aa 56 fc a1 a5 9a 1e ....{.x...V..... 0080: 65 f0 04 b7 04 08 af 7a 82 ef 77 e......z..w 553ded5a mailGidNumberldap_read: want=8, got=8 553ded5a mailQuotaStorage 0000: 30 71 02 01 0b 63 6c 04 0q...cl. ldap_read: want=107, got=107 00d0: 6c ae 39 eb 15 85 4a f9 c1 6f 65 ca 4f c6 db 14 l.9...J..oe.O... ldap_write: want=14, written=14 00e0: c1 f4 fe b8 b5 a4 a3 75 96 b1 9b 9b 8f d5 d6 e5 .......u........ 00f0: 8d 3c 75 4c 50 cc 9d 85 cf bb a1 d8 50 21 93 fa .<uLP.......P!.. 0100: 38 ee 89 46 45 4e 06 17 7a 8c 4b 83 51 95 a9 ca 8..FEN..z.K.Q... 0110: 71 8c d0 b9 59 1a 14 f4 10 8d b9 bc 80 d5 cb e9 q...Y........... 0120: 46 03 a2 ce 59 49 0b db fc ea a3 3b fa cd a1 99 F...YI.....;.... 0030: 02 0a 01 00 02 01 00 02 01 78 01 01 00 a0 5a a3 .........x....Z. 0040: 1a 04 0b 6f 62 6a 65 63 74 43 6c 61 73 73 04 0b ...objectClass.. 0130: 14 90 d6 0a 55 da 84 b1 42 fe af 8d 14 92 ce 27 ....U...B......' 0050: 6d 61 69 6c 41 63 63 6f 75 6e 74 a3 1b 04 11 6d mailAccount....m 0060: 61 69 6c 41 63 63 6f 75 6e 74 53 74 61 74 75 73 ailAccountStatus 0000: 30 0c 02 01 06 65 07 0a 01 00 04 00 04 00 0....e........ 0070: 04 06 61 63 74 69 76 65 a3 1f 04 0b 6d 61 69 6c ..active....mail 0080: 41 64 64 72 65 73 73 04 10 69 6e 66 6f 40 6e 65 Address..info@So 0090: 74 6f 63 65 61 6e 2e 64 65 30 6f 04 14 6d 61 69 meDom.tld0o..mai 00a0: 6c 53 74 6f 72 61 67 65 44 69 72 65 63 74 6f 72 lStorageDirector 553ded5a mailQuotaMessages553ded5a 00b0: 79 04 14 6d 61 69 6c 53 74 6f 72 61 67 65 44 69 y..mailStorageDi 553ded5a conn=1008 op=5 SRCH base="ou=accounts,ou=mail,dc=MyDomain,dc=Local" scope=2 deref=0 filter="(&(objectClass=mailAccount)(mailAccountStatus=active)(mailAddress=info@somedom.tld))" 00c0: 72 65 63 74 6f 72 79 04 0d 6d 61 69 6c 55 69 64 rectory..mailUid 0000: 28 6f 75 3d 61 63 63 6f 75 6e 74 73 2c 6f 75 3d (ou=accounts,ou= 553ded5a conn=1008 op=4 SEARCH RESULT tag=101 err=0 nentries=0 text= 00d0: 4e 75 6d 62 65 72 04 0d 6d 61 69 6c 47 69 64 4e Number..mailGidN 553ded5a conn=1008 op=5 SRCH attr=mailStorageDirectory mailStorageDirectory mailUidNumber mailGidNumber mailQuotaStorage mailQuotaMessages ldap_read: want=8, got=8 0000: 30 82 01 0d 02 01 09 63 0......c ldap_read: want=265, got=265 0000: 82 01 06 04 28 6f 75 3d 61 63 63 6f 75 6e 74 73 ....(ou=accounts 0010: 2c 6f 75 3d 6d 61 69 6c 2c 64 63 3d 4e 65 74 4f ,ou=mail,dc=MyDo 553ded5a ==> limits_get: conn=1008 op=5 self="uid=dovecot,ou=systemuser,ou=mail,dc=mydomain,dc=local" this="ou=accounts,ou=mail,dc=mydomain,dc=local" 00e0: 75 6d 62 65 72 04 10 6d 61 69 6c 51 75 6f 74 61 umber..mailQuota Segmentation fault
# =================================================================== #
... and a second time the same debugging (-d -1) # =================================================================== #
[...]
553df2b8 conn=1000 op=6 SRCH base="ou=accounts,ou=mail,dc=MyDomain,dc=Local" scope=2 deref=0 filter="(&(objectClass=mailAccount)(mailAccountStatus=active)(mailAddress=info@somedom.tld))" 0010: 70 7b e7 97 2e b3 3e 24 f8 bc 52 c5 ce 0d b8 fb p{....>$..R..... 0020: 89 cb 07 f8 d3 34 0a 39 9a fc 0d 0d b9 bb d4 13 .....4.9........ 0030: 8d 1f 56 87 c0 9c 10 2b a2 57 ee 3b 4b 93 6a 7c ..V....+.W.;K.j| TLS trace: SSL_accept:SSLv3 read finished A 553df2b8 conn=1000 op=6 SRCH attr=mailStorageDirectory mailStorageDirectory mailUidNumber mailGidNumber mailQuotaStorage mailQuotaMessages 553df2b8 connection_read(20): checking for input on id=1006 553df2b8 ==> limits_get: conn=1000 op=6 self="uid=dovecot,ou=systemuser,ou=mail,dc=mydomain,dc=local" this="ou=accounts,ou=mail,dc=mydomain,dc=local" 0040: 54 2b b2 49 7e f7 ce 89 19 96 fe 83 86 77 5a 3a T+.I~........wZ: Segmentation fault
# =================================================================== #
I'm currently trying to figure out a way to add "-O0 -g" gdb(1) support for the slapd(8C) command, since I'm compiling OpenLDAP sources from ports tree. I'll post it as soon as I got it. But maybe you can already assume some pre-diagnosis from the above output?
@Ulrich:
- Yes, I have tried several hardware plattforms, different physical,
VmWare and now the latest XenServer. Same results all over.
- A detailed log debug can be seen on:
https://forums.freebsd.org/threads/openldap-slapd-dies-sporadically.47634/
- Be warned from the above diagnosis in the forum: It is NOT the
assumed newsyslog(8) which causes the crashes!
- What exactly do you mean by "exists cleanly"?
Best regards Leander
Am 27.04.15 um 08:59 schrieb Ulrich Windl:
Hi!
You should tell about the operation you do when slapd crashes, maybe as well the last log messages you see. Also: Did you try different hardware and different compilers to run and compile slapd? What about doing an "-O0 -g" build and enable core dumps? Or are you telling slapd exits "cleanly"?
Regards, Ulrich
Am 26.04.15 um 13:48 schrieb Michael Ströder:
Leander Schäfer wrote:
my OpenLDAP keeps on dying sporadically.
Which OpenLDAP version is this?
Please note that OpenLDAP project does not know details about binary distribution releases.
Did you try with your own build from source using a recent OpenLDAP release?
Unfortunately it doesn't leave ANY trace of why it crashes.
I'm not familiar with FreeBSD. But try this to get more details:
Run with ulimit -c unlimited to get a core dump file.
Provide gdb stack trace like described herein:
http://www.openldap.org/faq/data/cache/59.html
Ciao, Michael.
--On Monday, April 27, 2015 12:02 PM +0200 Leander Schäfer info@netocean.de wrote:
Hi Michael, Hi Ulrich,
Thanks for your reply. I'm running Version 2.4.40. As I said, I do not run binary version. I always compile OpenLDAP from sources / latest ports tree.
Please compile and run current RE24 and see if you can reproduce. There are multiple segfault fixes present.
OpenLDAP 2.4.41 Engineering Fixed libldap segfault in ldap_sync_initialize (ITS#8001) Fixed slapd segfault when using matched values control (ITS#8046) Fixed slapo-collect segfault (ITS#7797) Fixed slapo-syncprov segfault on disconnect/abandon (ITS#5452,ITS#8012) Fixed slapo-syncprov segfault on disconnect/abandon (ITS#8043)
--Quanah
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
To be frank, I highly doubt it. Those problems have been there since I first used OpenLDAP five years ago. Neither do I use sync - so none except the collect fixes looks like it would apply in the first place. But I'll give it a quick shot. Can you please provide me a link, cause I wasn't able to find "current RE24" on the official website nor on the FTP mirror.
Thanks
Am 27.04.15 um 18:32 schrieb Quanah Gibson-Mount:
--On Monday, April 27, 2015 12:02 PM +0200 Leander Schäfer info@netocean.de wrote:
Hi Michael, Hi Ulrich,
Thanks for your reply. I'm running Version 2.4.40. As I said, I do not run binary version. I always compile OpenLDAP from sources / latest ports tree.
Please compile and run current RE24 and see if you can reproduce. There are multiple segfault fixes present.
OpenLDAP 2.4.41 Engineering Fixed libldap segfault in ldap_sync_initialize (ITS#8001) Fixed slapd segfault when using matched values control (ITS#8046) Fixed slapo-collect segfault (ITS#7797) Fixed slapo-syncprov segfault on disconnect/abandon (ITS#5452,ITS#8012) Fixed slapo-syncprov segfault on disconnect/abandon (ITS#8043)
--Quanah
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc.
Zimbra :: the leader in open source messaging and collaboration
Leander Schäfer wrote:
Can you please provide me a link, cause I wasn't able to find "current RE24" on the official website nor on the FTP mirror.
Use git or this link to checkout snapshot of the RE24 branch:
http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=snapshot;h=refs/he...
Ciao, Michael.
Ok, here is the first result running the debugging mode with gdb(1)
Procedure overview:
(gdb) run (gdb) bt full (gdb) thread apply all bt (gdb) generate-core-file
This came up:
candidates = Error accessing memory address 0x7ffffeafb6f0: Bad address.
# ================================================== #
root@FreeBSD [~]$ gdb --args /usr/local/libexec/slapd -d -1 -f /usr/local/etc/openldap/slapd.conf -u ldap -g ldap -h "ldapi://%2fvar%2frun%2fopenldap%2fldapi/ ldap:/// ldaps:///" GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... (gdb) run Starting program: /usr/local/libexec/slapd -d -1 -f /usr/local/etc/openldap/slapd.conf -u ldap -g ldap -h ldapi://%2fvar%2frun%2fopenldap%2fldapi/\ ldap:///\ ldaps:/// [New LWP 101138] [New Thread 802806400 (LWP 101138/slapd)]
[...]
553e8a87 conn=1006 op=2 SRCH attr=mailAlias 553e8a87 send_ldap_result: err=0 matched="" text="" 0010: 51 bd aa 7d 3f 1c 50 fb 25 f8 59 9e 9d 9a ba 0f Q..}?.P.%.Y..... 0020: d0 07 aa 95 ac 1c e7 3e 81 f6 e6 0b 6d 09 94 9b .......>....m... 0730: 1b 51 e3 08 4b 38 ec f1 ee 8c 0f 35 cd 55 eb 80 .Q..K8.....5.U.. 553e8a87 ==> limits_get: conn=1006 op=2 self="[anonymous]" this="ou=accounts,ou=mail,dc=mydomain,dc=local" 0740: 83 e2 3b b5 13 fd 08 51 13 25 d9 7d 57 9f 6b e9 ..;....Q.%.}W.k. [New Thread 943c11800 (LWP 100198/slapd)]
Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 943c11800 (LWP 100198/slapd)] mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404 404 search.c: No such file or directory. in search.c Current language: auto; currently minimal (gdb) bt full #0 mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404 mdb = (struct mdb_info *) 0x80290a000 id = 0 cursor = 0 nsubs = 128 ncand = 0 cscope = 0 lastid = 18446744073709551615 candidates = Error accessing memory address 0x7ffffeafb6f0: Bad address. (gdb) thread apply all bt [New Thread 943c15000 (LWP 101255/slapd)] [New Thread 943c14c00 (LWP 101213/slapd)] [New Thread 943c14800 (LWP 101202/slapd)] [New Thread 943c14400 (LWP 100898/slapd)] [New Thread 943c14000 (LWP 100884/slapd)] [New Thread 943c13c00 (LWP 100647/slapd)] [New Thread 943c13800 (LWP 100619/slapd)] [New Thread 943c13400 (LWP 100577/slapd)] [New Thread 943c13000 (LWP 100531/slapd)] [New Thread 943c12c00 (LWP 100515/slapd)] [New Thread 943c12800 (LWP 100347/slapd)] [New Thread 943c12400 (LWP 100311/slapd)] [New Thread 943c12000 (LWP 100296/slapd)] [New Thread 943c11c00 (LWP 100268/slapd)] [New Thread 943c11400 (LWP 100165/slapd)] [New Thread 802807800 (LWP 100103/slapd)]
Thread 19 (Thread 802807800 (LWP 100103/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3 #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7 #3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7 #4 0x0000000800bfd48f in lutil_debug () from /usr/local/lib/liblber-2.4.so.2 #5 0x000000000043b96f in slapd_daemon_task (ptr=0x8028afb08) at daemon.c:2530 #6 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3 #7 0x0000000000000000 in ?? ()
Thread 18 (Thread 943c11400 (LWP 100165/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3 #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7 #3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7 #4 0x0000000800bfcc66 in ber_error_print () from /usr/local/lib/liblber-2.4.so.2 #5 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2 #6 0x0000000800bfcf4a in ber_log_bprint () from /usr/local/lib/liblber-2.4.so.2 #7 0x0000000800c0017e in ber_int_sb_write () from /usr/local/lib/liblber-2.4.so.2 #8 0x00000008009e1905 in ldap_start_tls_s () from /usr/local/lib/libldap_r-2.4.so.2 #9 0x0000000801807e84 in BIO_read () from /lib/libcrypto.so.7 #10 0x000000080146ce97 in ssl3_read_n () from /usr/lib/libssl.so.7 #11 0x000000080146db61 in ssl3_read_bytes () from /usr/lib/libssl.so.7 #12 0x0000000801470aa0 in ssl3_read () from /usr/lib/libssl.so.7 #13 0x00000008009e16b8 in ldap_start_tls_s () from /usr/local/lib/libldap_r-2.4.so.2 #14 0x0000000800c00126 in ber_int_sb_write () from /usr/local/lib/liblber-2.4.so.2 #15 0x0000000800bff0f2 in ber_int_sb_read () from /usr/local/lib/liblber-2.4.so.2 #16 0x0000000800bfc720 in ber_get_next () from /usr/local/lib/liblber-2.4.so.2 #17 0x0000000000444cf5 in connection_input (conn=0x8034099c0, cri=0x7fffff3fcc18) at connection.c:1572 #18 0x0000000000444ac5 in connection_read (s=28, cri=0x7fffff3fcc18) at connection.c:1460 #19 0x0000000000442291 in connection_read_thread (ctx=0x7fffff3fcc80, argv=0x1c) at connection.c:1284 #20 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #21 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3 #22 0x0000000000000000 in ?? ()
Thread 17 (Thread 943c11c00 (LWP 100268/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3 #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7 #3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7 #4 0x0000000800bfcc66 in ber_error_print () from /usr/local/lib/liblber-2.4.so.2 #5 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2 #6 0x0000000800bfcf4a in ber_log_bprint () from /usr/local/lib/liblber-2.4.so.2 #7 0x0000000800c0028e in ber_int_sb_write () from /usr/local/lib/liblber-2.4.so.2 #8 0x00000008009e1895 in ldap_start_tls_s () from /usr/local/lib/libldap_r-2.4.so.2 #9 0x0000000801807f6a in BIO_write () from /lib/libcrypto.so.7 #10 0x00000008017d36cc in BIO_f_buffer () from /lib/libcrypto.so.7 #11 0x00000008014659e7 in ssl3_accept () from /usr/lib/libssl.so.7 #12 0x0000000801470dd7 in ssl23_accept () from /usr/lib/libssl.so.7 #13 0x00000008009e0cd9 in ldap_start_tls_s () from /usr/local/lib/libldap_r-2.4.so.2 #14 0x00000008009dee4b in ldap_pvt_tls_accept () from /usr/local/lib/libldap_r-2.4.so.2 #15 0x0000000000444703 in connection_read (s=35, cri=0x7ffffe3fac18) at connection.c:1372 #16 0x0000000000442291 in connection_read_thread (ctx=0x7ffffe3fac80, argv=0x23) at connection.c:1284 #17 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #18 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3 #19 0x0000000000000000 in ?? ()
Thread 16 (Thread 943c12000 (LWP 100296/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3 #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7 #3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7 #4 0x0000000800bfcc66 in ber_error_print () from /usr/local/lib/liblber-2.4.so.2 #5 0x0000000800bfd2a9 in ber_dump () from /usr/local/lib/liblber-2.4.so.2 #6 0x0000000800bfd1cf in ber_log_dump () from /usr/local/lib/liblber-2.4.so.2 #7 0x0000000800bf9266 in ber_scanf () from /usr/local/lib/liblber-2.4.so.2 #8 0x0000000000472692 in do_bind (op=0x94601cc00, rs=0x7ffffdbf9b60) at bind.c:102 #9 0x0000000000443398 in connection_operation (ctx=0x7ffffdbf9c80, arg_v=0x94601cc00) at connection.c:1155 #10 0x000000000044235b in connection_read_thread (ctx=0x7ffffdbf9c80, argv=0x1d) at connection.c:1291 #11 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #12 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3 #13 0x0000000000000000 in ?? ()
Thread 15 (Thread 943c12400 (LWP 100311/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3 ---Type <return> to continue, or q <return> to quit--- #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7 #3 0x0000000801dfc087 in fflush () from /lib/libc.so.7 #4 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2 #5 0x0000000800bfcf4a in ber_log_bprint () from /usr/local/lib/liblber-2.4.so.2 #6 0x0000000800c0028e in ber_int_sb_write () from /usr/local/lib/liblber-2.4.so.2 #7 0x00000008009e1895 in ldap_start_tls_s () from /usr/local/lib/libldap_r-2.4.so.2 #8 0x0000000801807f6a in BIO_write () from /lib/libcrypto.so.7 #9 0x00000008017d36cc in BIO_f_buffer () from /lib/libcrypto.so.7 #10 0x00000008014659e7 in ssl3_accept () from /usr/lib/libssl.so.7 #11 0x0000000801470dd7 in ssl23_accept () from /usr/lib/libssl.so.7 #12 0x00000008009e0cd9 in ldap_start_tls_s () from /usr/local/lib/libldap_r-2.4.so.2 #13 0x00000008009dee4b in ldap_pvt_tls_accept () from /usr/local/lib/libldap_r-2.4.so.2 #14 0x0000000000444703 in connection_read (s=37, cri=0x7ffffd3f8c18) at connection.c:1372 #15 0x0000000000442291 in connection_read_thread (ctx=0x7ffffd3f8c80, argv=0x25) at connection.c:1284 #16 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #17 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3 #18 0x0000000000000000 in ?? ()
Thread 14 (Thread 943c12800 (LWP 100347/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3 #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7 #3 0x0000000801dfc087 in fflush () from /lib/libc.so.7 #4 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2 #5 0x0000000800bfcf4a in ber_log_bprint () from /usr/local/lib/liblber-2.4.so.2 #6 0x0000000800c0028e in ber_int_sb_write () from /usr/local/lib/liblber-2.4.so.2 #7 0x00000008009e1895 in ldap_start_tls_s () from /usr/local/lib/libldap_r-2.4.so.2 #8 0x0000000801807f6a in BIO_write () from /lib/libcrypto.so.7 #9 0x00000008017d36cc in BIO_f_buffer () from /lib/libcrypto.so.7 #10 0x00000008014659e7 in ssl3_accept () from /usr/lib/libssl.so.7 #11 0x0000000801470dd7 in ssl23_accept () from /usr/lib/libssl.so.7 #12 0x00000008009e0cd9 in ldap_start_tls_s () from /usr/local/lib/libldap_r-2.4.so.2 #13 0x00000008009dee4b in ldap_pvt_tls_accept () from /usr/local/lib/libldap_r-2.4.so.2 #14 0x0000000000444703 in connection_read (s=36, cri=0x7ffffcbf7c18) at connection.c:1372 #15 0x0000000000442291 in connection_read_thread (ctx=0x7ffffcbf7c80, argv=0x24) at connection.c:1284 #16 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #17 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3 #18 0x0000000000000000 in ?? ()
Thread 13 (Thread 943c12c00 (LWP 100515/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3 #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7 #3 0x0000000801dfc087 in fflush () from /lib/libc.so.7 #4 0x0000000800bfcee0 in ber_pvt_log_printf () from /usr/local/lib/liblber-2.4.so.2 #5 0x0000000800bfbe78 in ber_flush2 () from /usr/local/lib/liblber-2.4.so.2 #6 0x0000000000460d1a in send_ldap_ber (op=0x946c6ac00, ber=0x7ffffc266168) at result.c:339 #7 0x000000000045c9e6 in send_ldap_response (op=0x946c6ac00, rs=0x7ffffc3f6bd0) at result.c:705 #8 0x000000000045d25c in slap_send_ldap_result (op=0x946c6ac00, rs=0x7ffffc3f6bd0) at result.c:861 #9 0x0000000802c1e61b in mdb_search (op=0x946c6ac00, rs=0x7ffffc3f6bd0) at search.c:1164 #10 0x00000000004473ad in fe_op_search (op=0x946c6ac00, rs=0x7ffffc3f6bd0) at search.c:402 #11 0x0000000000446bc2 in do_search (op=0x946c6ac00, rs=0x7ffffc3f6bd0) at search.c:247 #12 0x0000000000443398 in connection_operation (ctx=0x7ffffc3f6c80, arg_v=0x946c6ac00) at connection.c:1155 #13 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #14 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3 #15 0x0000000000000000 in ?? ()
Thread 12 (Thread 943c13000 (LWP 100531/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3 #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7 #3 0x0000000801dfc087 in fflush () from /lib/libc.so.7 #4 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2 #5 0x0000000800bfd2b5 in ber_dump () from /usr/local/lib/liblber-2.4.so.2 #6 0x0000000800bfd1cf in ber_log_dump () from /usr/local/lib/liblber-2.4.so.2 #7 0x0000000800bfca96 in ber_get_next () from /usr/local/lib/liblber-2.4.so.2 #8 0x0000000000444cf5 in connection_input (conn=0x8034097b0, cri=0x7ffffbbf5c18) at connection.c:1572 #9 0x0000000000444ac5 in connection_read (s=27, cri=0x7ffffbbf5c18) at connection.c:1460 #10 0x0000000000442291 in connection_read_thread (ctx=0x7ffffbbf5c80, argv=0x1b) at connection.c:1284 #11 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #12 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3 #13 0x0000000000000000 in ?? ()
Thread 11 (Thread 943c13400 (LWP 100577/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3 #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7 ---Type <return> to continue, or q <return> to quit--- #3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7 #4 0x0000000800bfcc66 in ber_error_print () from /usr/local/lib/liblber-2.4.so.2 #5 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2 #6 0x0000000800bfcf4a in ber_log_bprint () from /usr/local/lib/liblber-2.4.so.2 #7 0x0000000800c0028e in ber_int_sb_write () from /usr/local/lib/liblber-2.4.so.2 #8 0x00000008009e1895 in ldap_start_tls_s () from /usr/local/lib/libldap_r-2.4.so.2 #9 0x0000000801807f6a in BIO_write () from /lib/libcrypto.so.7 #10 0x00000008017d36cc in BIO_f_buffer () from /lib/libcrypto.so.7 #11 0x00000008014659e7 in ssl3_accept () from /usr/lib/libssl.so.7 #12 0x0000000801470dd7 in ssl23_accept () from /usr/lib/libssl.so.7 #13 0x00000008009e0cd9 in ldap_start_tls_s () from /usr/local/lib/libldap_r-2.4.so.2 #14 0x00000008009dee4b in ldap_pvt_tls_accept () from /usr/local/lib/libldap_r-2.4.so.2 #15 0x0000000000444703 in connection_read (s=38, cri=0x7ffffb3f4c18) at connection.c:1372 #16 0x0000000000442291 in connection_read_thread (ctx=0x7ffffb3f4c80, argv=0x26) at connection.c:1284 #17 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #18 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3 #19 0x0000000000000000 in ?? ()
Thread 10 (Thread 943c13800 (LWP 100619/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3 #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7 #3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7 #4 0x0000000800bfd48f in lutil_debug () from /usr/local/lib/liblber-2.4.so.2 #5 0x000000000045c3a0 in send_ldap_response (op=0x947811400, rs=0x7ffffabf3b60) at result.c:609 #6 0x000000000045d25c in slap_send_ldap_result (op=0x947811400, rs=0x7ffffabf3b60) at result.c:861 #7 0x0000000802c1e61b in mdb_search (op=0x947811400, rs=0x7ffffabf3b60) at search.c:1164 #8 0x00000000004473ad in fe_op_search (op=0x947811400, rs=0x7ffffabf3b60) at search.c:402 #9 0x0000000000446bc2 in do_search (op=0x947811400, rs=0x7ffffabf3b60) at search.c:247 #10 0x0000000000443398 in connection_operation (ctx=0x7ffffabf3c80, arg_v=0x947811400) at connection.c:1155 #11 0x000000000044235b in connection_read_thread (ctx=0x7ffffabf3c80, argv=0x17) at connection.c:1291 #12 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #13 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3 #14 0x0000000000000000 in ?? ()
Thread 9 (Thread 943c13c00 (LWP 100647/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3 #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7 #3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7 #4 0x0000000800bfcc66 in ber_error_print () from /usr/local/lib/liblber-2.4.so.2 #5 0x0000000800bfd158 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2 #6 0x0000000800bfd2b5 in ber_dump () from /usr/local/lib/liblber-2.4.so.2 #7 0x0000000800bfd1cf in ber_log_dump () from /usr/local/lib/liblber-2.4.so.2 #8 0x0000000800bfca96 in ber_get_next () from /usr/local/lib/liblber-2.4.so.2 #9 0x0000000000444cf5 in connection_input (conn=0x80340a200, cri=0x7ffffa3f2c18) at connection.c:1572 #10 0x0000000000444ac5 in connection_read (s=32, cri=0x7ffffa3f2c18) at connection.c:1460 #11 0x0000000000442291 in connection_read_thread (ctx=0x7ffffa3f2c80, argv=0x20) at connection.c:1284 #12 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #13 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3 #14 0x0000000000000000 in ?? ()
Thread 8 (Thread 943c14000 (LWP 100884/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3 #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7 #3 0x0000000801dfc087 in fflush () from /lib/libc.so.7 #4 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2 #5 0x0000000800bfd2b5 in ber_dump () from /usr/local/lib/liblber-2.4.so.2 #6 0x0000000800bfd1cf in ber_log_dump () from /usr/local/lib/liblber-2.4.so.2 #7 0x0000000800bf9266 in ber_scanf () from /usr/local/lib/liblber-2.4.so.2 #8 0x0000000000445bc1 in do_search (op=0x94801c400, rs=0x7ffff9bf1b60) at search.c:72 #9 0x0000000000443398 in connection_operation (ctx=0x7ffff9bf1c80, arg_v=0x94801c400) at connection.c:1155 #10 0x000000000044235b in connection_read_thread (ctx=0x7ffff9bf1c80, argv=0x19) at connection.c:1291 #11 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #12 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3 #13 0x0000000000000000 in ?? ()
Thread 7 (Thread 943c14400 (LWP 100898/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3 #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7 #3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7 #4 0x0000000800bfcc66 in ber_error_print () from /usr/local/lib/liblber-2.4.so.2 #5 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2 #6 0x0000000800bfcf4a in ber_log_bprint () from /usr/local/lib/liblber-2.4.so.2 #7 0x0000000800c0017e in ber_int_sb_write () from /usr/local/lib/liblber-2.4.so.2 ---Type <return> to continue, or q <return> to quit--- #8 0x0000000800bff0f2 in ber_int_sb_read () from /usr/local/lib/liblber-2.4.so.2 #9 0x0000000800bfca1a in ber_get_next () from /usr/local/lib/liblber-2.4.so.2 #10 0x0000000000444cf5 in connection_input (conn=0x8034095a0, cri=0x7ffff93f0c18) at connection.c:1572 #11 0x0000000000444ac5 in connection_read (s=26, cri=0x7ffff93f0c18) at connection.c:1460 #12 0x0000000000442291 in connection_read_thread (ctx=0x7ffff93f0c80, argv=0x1a) at connection.c:1284 #13 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #14 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3 #15 0x0000000000000000 in ?? ()
Thread 6 (Thread 943c14800 (LWP 101202/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3 #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7 #3 0x0000000801dfc087 in fflush () from /lib/libc.so.7 #4 0x0000000800bfcee0 in ber_pvt_log_printf () from /usr/local/lib/liblber-2.4.so.2 #5 0x0000000800c00167 in ber_int_sb_write () from /usr/local/lib/liblber-2.4.so.2 #6 0x0000000800bff0f2 in ber_int_sb_read () from /usr/local/lib/liblber-2.4.so.2 #7 0x0000000800bfca1a in ber_get_next () from /usr/local/lib/liblber-2.4.so.2 #8 0x0000000000444cf5 in connection_input (conn=0x803408940, cri=0x7ffff8befc18) at connection.c:1572 #9 0x0000000000444ac5 in connection_read (s=20, cri=0x7ffff8befc18) at connection.c:1460 #10 0x0000000000442291 in connection_read_thread (ctx=0x7ffff8befc80, argv=0x14) at connection.c:1284 #11 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #12 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3 #13 0x0000000000000000 in ?? ()
Thread 5 (Thread 943c14c00 (LWP 101213/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3 #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7 #3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7 #4 0x0000000800bfcc66 in ber_error_print () from /usr/local/lib/liblber-2.4.so.2 #5 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2 #6 0x0000000800bfd2b5 in ber_dump () from /usr/local/lib/liblber-2.4.so.2 #7 0x0000000800bfd1cf in ber_log_dump () from /usr/local/lib/liblber-2.4.so.2 #8 0x0000000800bf9266 in ber_scanf () from /usr/local/lib/liblber-2.4.so.2 #9 0x0000000000445bc1 in do_search (op=0x948c6ac00, rs=0x7ffff83eeb60) at search.c:72 #10 0x0000000000443398 in connection_operation (ctx=0x7ffff83eec80, arg_v=0x948c6ac00) at connection.c:1155 #11 0x000000000044235b in connection_read_thread (ctx=0x7ffff83eec80, argv=0x1e) at connection.c:1291 #12 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #13 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3 #14 0x0000000000000000 in ?? ()
Thread 4 (Thread 943c15000 (LWP 101255/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801dfc0a9 in fflush () from /lib/libc.so.7 #2 0x0000000800bfcee0 in ber_pvt_log_printf () from /usr/local/lib/liblber-2.4.so.2 #3 0x0000000800c00167 in ber_int_sb_write () from /usr/local/lib/liblber-2.4.so.2 #4 0x0000000800bff0f2 in ber_int_sb_read () from /usr/local/lib/liblber-2.4.so.2 #5 0x0000000800bfc720 in ber_get_next () from /usr/local/lib/liblber-2.4.so.2 #6 0x0000000000444cf5 in connection_input (conn=0x803408b50, cri=0x7ffff7bedc18) at connection.c:1572 #7 0x0000000000444ac5 in connection_read (s=21, cri=0x7ffff7bedc18) at connection.c:1460 #8 0x0000000000442291 in connection_read_thread (ctx=0x7ffff7bedc80, argv=0x15) at connection.c:1284 #9 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #10 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3 #11 0x0000000000000000 in ?? ()
Thread 3 (Thread 943c11800 (LWP 100198/slapd)): #0 mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404 #1 0x00000000004473ad in fe_op_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:402 #2 0x0000000000446bc2 in do_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:247 #3 0x0000000000443398 in connection_operation (ctx=0x7ffffebfbc80, arg_v=0x94581c400) at connection.c:1155 #4 0x000000000044235b in connection_read_thread (ctx=0x7ffffebfbc80, argv=0x18) at connection.c:1291 #5 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from /usr/local/lib/libldap_r-2.4.so.2 #6 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3 #7 0x0000000000000000 in ?? ()
Thread 2 (Thread 802806400 (LWP 100636/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x000000000043a770 in slapd_daemon () at daemon.c:2929 #2 0x0000000000416b5a in main (argc=11, argv=0x7fffffffe568) at main.c:1012 404 in search.c (gdb) generate-core-file warning: Memory read failed for corefile section, 5368709120 bytes at 0x0000000803c00000
Killed
root@FreeBSD [~] $ ls -lach ./ -rw-r--r-- 1 root wheel 3.6G 27 Apr 22:01 core.38120
# ================================================== #
Seems like something went wrong here. Am I using gdb wrong?
Am 27.04.15 um 19:04 schrieb Michael Ströder:
Leander Schäfer wrote:
Can you please provide me a link, cause I wasn't able to find "current RE24" on the official website nor on the FTP mirror.
Use git or this link to checkout snapshot of the RE24 branch:
http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=snapshot;h=refs/he...
Ciao, Michael.
Leander Schäfer wrote:
Ok, here is the first result running the debugging mode with gdb(1)
Procedure overview:
(gdb) run (gdb) bt full (gdb) thread apply all bt (gdb) generate-core-file
No need for a core file if you're just running slapd inside gdb.
This came up:
candidates = Error accessing memory address 0x7ffffeafb6f0: Bad address.
# ================================================== #
root@FreeBSD [~]$ gdb --args /usr/local/libexec/slapd -d -1 -f /usr/local/etc/openldap/slapd.conf -u ldap -g ldap -h "ldapi://%2fvar%2frun%2fopenldap%2fldapi/ ldap:/// ldaps:///" GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... (gdb) run Starting program: /usr/local/libexec/slapd -d -1 -f /usr/local/etc/openldap/slapd.conf -u ldap -g ldap -h ldapi://%2fvar%2frun%2fopenldap%2fldapi/\ ldap:///\ ldaps:/// [New LWP 101138] [New Thread 802806400 (LWP 101138/slapd)]
[...]
553e8a87 conn=1006 op=2 SRCH attr=mailAlias 553e8a87 send_ldap_result: err=0 matched="" text="" 0010: 51 bd aa 7d 3f 1c 50 fb 25 f8 59 9e 9d 9a ba 0f Q..}?.P.%.Y..... 0020: d0 07 aa 95 ac 1c e7 3e 81 f6 e6 0b 6d 09 94 9b .......>....m... 0730: 1b 51 e3 08 4b 38 ec f1 ee 8c 0f 35 cd 55 eb 80 .Q..K8.....5.U.. 553e8a87 ==> limits_get: conn=1006 op=2 self="[anonymous]" this="ou=accounts,ou=mail,dc=mydomain,dc=local" 0740: 83 e2 3b b5 13 fd 08 51 13 25 d9 7d 57 9f 6b e9 ..;....Q.%.}W.k. [New Thread 943c11800 (LWP 100198/slapd)]
Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 943c11800 (LWP 100198/slapd)] mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404 404 search.c: No such file or directory. in search.c Current language: auto; currently minimal (gdb) bt full #0 mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404 mdb = (struct mdb_info *) 0x80290a000 id = 0 cursor = 0 nsubs = 128 ncand = 0 cscope = 0 lastid = 18446744073709551615 candidates = Error accessing memory address 0x7ffffeafb6f0: Bad address. (gdb) thread apply all bt [New Thread 943c15000 (LWP 101255/slapd)] [New Thread 943c14c00 (LWP 101213/slapd)] [New Thread 943c14800 (LWP 101202/slapd)] [New Thread 943c14400 (LWP 100898/slapd)] [New Thread 943c14000 (LWP 100884/slapd)] [New Thread 943c13c00 (LWP 100647/slapd)] [New Thread 943c13800 (LWP 100619/slapd)] [New Thread 943c13400 (LWP 100577/slapd)] [New Thread 943c13000 (LWP 100531/slapd)] [New Thread 943c12c00 (LWP 100515/slapd)] [New Thread 943c12800 (LWP 100347/slapd)] [New Thread 943c12400 (LWP 100311/slapd)] [New Thread 943c12000 (LWP 100296/slapd)] [New Thread 943c11c00 (LWP 100268/slapd)] [New Thread 943c11400 (LWP 100165/slapd)] [New Thread 802807800 (LWP 100103/slapd)]
Thread 19 (Thread 802807800 (LWP 100103/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3 #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7 #3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7 #4 0x0000000800bfd48f in lutil_debug () from /usr/local/lib/liblber-2.4.so.2 #5 0x000000000043b96f in slapd_daemon_task (ptr=0x8028afb08) at daemon.c:2530 #6 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
Seems like something went wrong here. Am I using gdb wrong?
Looks like your liblber was installed without debug symbols. Most of these stack traces look invalid.
Am 27.04.15 um 19:04 schrieb Michael Ströder:
Leander Schäfer wrote:
Can you please provide me a link, cause I wasn't able to find "current RE24" on the official website nor on the FTP mirror.
Use git or this link to checkout snapshot of the RE24 branch:
http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=snapshot;h=refs/he...
Assuming you compiled the latest snapshot, the SEGV at back-mdb/search.c:404 makes not much sense, it's a return statement.
Also, as back-mdb didn't exist 5 years ago, this cannot be the same issue you've been running into all the time.
Perhaps you've hit a stack overrun. Generally slapd uses 8MB stacks on 64bit machines. It seems from your ulimit output that 8MB should be fine, so that also seems unlikely.
What was the full LDAP search request that was running at the moment of the crash? Mainly interested in seeing the search filter, and how complex it was, as well as the depth of the DIT.
Howard Chu wrote:
Assuming you compiled the latest snapshot, the SEGV at back-mdb/search.c:404 makes not much sense, it's a return statement.
Also, as back-mdb didn't exist 5 years ago, this cannot be the same issue you've been running into all the time.
Perhaps you've hit a stack overrun. Generally slapd uses 8MB stacks on 64bit machines. It seems from your ulimit output that 8MB should be fine, so that also seems unlikely.
Ah, yes, this is a known issue with FreeBSD.
http://www.openldap.org/lists/openldap-bugs/200506/msg00174.html
Howard Chu wrote:
Howard Chu wrote:
Assuming you compiled the latest snapshot, the SEGV at back-mdb/search.c:404 makes not much sense, it's a return statement.
Also, as back-mdb didn't exist 5 years ago, this cannot be the same issue you've been running into all the time.
Perhaps you've hit a stack overrun. Generally slapd uses 8MB stacks on 64bit machines. It seems from your ulimit output that 8MB should be fine, so that also seems unlikely.
Ah, yes, this is a known issue with FreeBSD.
http://www.openldap.org/lists/openldap-bugs/200506/msg00174.html
Furthermore, in the intervening 10 years, the FreeBSD developers have not yet fixed this issue.
http://lists.freebsd.org/pipermail/freebsd-current/2014-August/051646.html
Also, I just checked. It looks like slapd is successfully linked to "libthr.so.3"
root@FreeBSD [~]$ ldd /usr/local/libexec/slapd /usr/local/libexec/slapd: libldap_r-2.4.so.2 => /usr/local/lib/libldap_r-2.4.so.2 (0x8009a7000) liblber-2.4.so.2 => /usr/local/lib/liblber-2.4.so.2 (0x800bf5000) libltdl.so.7 => /usr/local/lib/libltdl.so.7 (0x800e03000) libcrypt.so.5 => /lib/libcrypt.so.5 (0x80100c000) libwrap.so.6 => /usr/lib/libwrap.so.6 (0x80122c000) libssl.so.7 => /usr/lib/libssl.so.7 (0x801435000) libcrypto.so.7 => /lib/libcrypto.so.7 (0x8016a0000) libthr.so.3 => /lib/libthr.so.3 (0x801a93000) libc.so.7 => /lib/libc.so.7 (0x801cb8000)
root@FreeBSD [~]$ ls -lach /lib/libthr.so.3 -r--r--r-- 1 root wheel 103K 18 Jan 15:36 /lib/libthr.so.3
So as all the other Applications using "libthr.so.3" slapd should also be able to use it?! I'm not 100% convinced yet, that this diagnosis is going into the right direction here. I think I'll provide a second gdb output, since it always seems to break down at a different point and action.
Am 28.04.15 um 02:49 schrieb Howard Chu:
Howard Chu wrote:
Howard Chu wrote:
Assuming you compiled the latest snapshot, the SEGV at back-mdb/search.c:404 makes not much sense, it's a return statement.
Also, as back-mdb didn't exist 5 years ago, this cannot be the same issue you've been running into all the time.
Perhaps you've hit a stack overrun. Generally slapd uses 8MB stacks on 64bit machines. It seems from your ulimit output that 8MB should be fine, so that also seems unlikely.
Ah, yes, this is a known issue with FreeBSD.
http://www.openldap.org/lists/openldap-bugs/200506/msg00174.html
Furthermore, in the intervening 10 years, the FreeBSD developers have not yet fixed this issue.
http://lists.freebsd.org/pipermail/freebsd-current/2014-August/051646.html
Am 28.04.15 um 02:32 schrieb Howard Chu:
Leander Schäfer wrote:
Ok, here is the first result running the debugging mode with gdb(1)
Procedure overview:
(gdb) run (gdb) bt full (gdb) thread apply all bt (gdb) generate-core-file
No need for a core file if you're just running slapd inside gdb.
I thought the core file would be required to have a better base of knowledge about the internal happening?! But ofcourse I'm happy if it is not required, cause it doesn't seem to create it properly in the first place.
This came up:
candidates = Error accessing memory address 0x7ffffeafb6f0: Bad address.
# ================================================== #
root@FreeBSD [~]$ gdb --args /usr/local/libexec/slapd -d -1 -f /usr/local/etc/openldap/slapd.conf -u ldap -g ldap -h "ldapi://%2fvar%2frun%2fopenldap%2fldapi/ ldap:/// ldaps:///" GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... (gdb) run Starting program: /usr/local/libexec/slapd -d -1 -f /usr/local/etc/openldap/slapd.conf -u ldap -g ldap -h ldapi://%2fvar%2frun%2fopenldap%2fldapi/\ ldap:///\ ldaps:/// [New LWP 101138] [New Thread 802806400 (LWP 101138/slapd)]
[...]
553e8a87 conn=1006 op=2 SRCH attr=mailAlias 553e8a87 send_ldap_result: err=0 matched="" text="" 0010: 51 bd aa 7d 3f 1c 50 fb 25 f8 59 9e 9d 9a ba 0f Q..}?.P.%.Y..... 0020: d0 07 aa 95 ac 1c e7 3e 81 f6 e6 0b 6d 09 94 9b .......>....m... 0730: 1b 51 e3 08 4b 38 ec f1 ee 8c 0f 35 cd 55 eb 80 .Q..K8.....5.U.. 553e8a87 ==> limits_get: conn=1006 op=2 self="[anonymous]" this="ou=accounts,ou=mail,dc=mydomain,dc=local" 0740: 83 e2 3b b5 13 fd 08 51 13 25 d9 7d 57 9f 6b e9 ..;....Q.%.}W.k. [New Thread 943c11800 (LWP 100198/slapd)]
Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 943c11800 (LWP 100198/slapd)] mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404 404 search.c: No such file or directory. in search.c Current language: auto; currently minimal (gdb) bt full #0 mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404 mdb = (struct mdb_info *) 0x80290a000 id = 0 cursor = 0 nsubs = 128 ncand = 0 cscope = 0 lastid = 18446744073709551615 candidates = Error accessing memory address 0x7ffffeafb6f0: Bad address. (gdb) thread apply all bt [New Thread 943c15000 (LWP 101255/slapd)] [New Thread 943c14c00 (LWP 101213/slapd)] [New Thread 943c14800 (LWP 101202/slapd)] [New Thread 943c14400 (LWP 100898/slapd)] [New Thread 943c14000 (LWP 100884/slapd)] [New Thread 943c13c00 (LWP 100647/slapd)] [New Thread 943c13800 (LWP 100619/slapd)] [New Thread 943c13400 (LWP 100577/slapd)] [New Thread 943c13000 (LWP 100531/slapd)] [New Thread 943c12c00 (LWP 100515/slapd)] [New Thread 943c12800 (LWP 100347/slapd)] [New Thread 943c12400 (LWP 100311/slapd)] [New Thread 943c12000 (LWP 100296/slapd)] [New Thread 943c11c00 (LWP 100268/slapd)] [New Thread 943c11400 (LWP 100165/slapd)] [New Thread 802807800 (LWP 100103/slapd)]
Thread 19 (Thread 802807800 (LWP 100103/slapd)): #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3 #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3 #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7 #3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7 #4 0x0000000800bfd48f in lutil_debug () from /usr/local/lib/liblber-2.4.so.2 #5 0x000000000043b96f in slapd_daemon_task (ptr=0x8028afb08) at daemon.c:2530 #6 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
Seems like something went wrong here. Am I using gdb wrong?
Looks like your liblber was installed without debug symbols. Most of these stack traces look invalid.
What does this mean? How did you see this / what indicated this to you? Is it required to fix this liblber issue for a better debug result, or is it ok for first diagnosis?
Am 27.04.15 um 19:04 schrieb Michael Ströder:
Leander Schäfer wrote:
Can you please provide me a link, cause I wasn't able to find "current RE24" on the official website nor on the FTP mirror.
Use git or this link to checkout snapshot of the RE24 branch:
http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=snapshot;h=refs/he...
Assuming you compiled the latest snapshot, the SEGV at back-mdb/search.c:404 makes not much sense, it's a return statement.
Also, as back-mdb didn't exist 5 years ago, this cannot be the same issue you've been running into all the time.
^ Pretty much possible. But again - the exit points are always different. I could do the same thing with BDB and yet it would exit on different points.
Perhaps you've hit a stack overrun. Generally slapd uses 8MB stacks on 64bit machines. It seems from your ulimit output that 8MB should be fine, so that also seems unlikely.
Since this exceeds my current knowledge, I decided to asked the developement experts on FreeBSD Forums: https://forums.freebsd.org/threads/thread-stack-size-segmentation-fault.5141...
What was the full LDAP search request that was running at the moment of the crash? Mainly interested in seeing the search filter, and how complex it was, as well as the depth of the DIT.
The search filter looks quite simple to me: ==> 553e8a87 conn=1006 op=2 SRCH base="ou=accounts,ou=mail,dc=MyDomain,dc=Local" scope=2 deref=0 filter="(mailAddress=root@wm-01.mydomain.local)"
Here is a bit more of the debug output.
[...]
553e8a87 daemon: waked 0670: 63 04 aa 27 74 94 4c 86 1d d7 3f a1 2f af 22 0f c..'t.L...?./.". 553e8a87 mailAlias553e8a87 0680: 41 88 ed e0 c6 d0 4e 92 ed 2f b1 40 51 ae a8 77 A.....N../.@Q..w ber_get_next: tag 0x30 len 12 contents: 0690: fe b9 b7 18 ae 33 df a4 27 50 0e 3d 8d 17 31 ee .....3..'P.=..1. 06a0: 21 e9 ed c7 a2 90 fa 6f 3e b3 87 ff d8 6b d4 e8 !......o>....k.. 0030: c8 57 d6 0e 59 3c 7b ef b9 db fe 64 f9 4d 02 10 .W..Y<{....d.M.. 0040: af 62 1a 21 b3 68 08 50 31 1e 6b 09 8f da 88 84 .b.!.h.P1.k..... 553e8a87 conn=1006 op=2 SRCH base="ou=accounts,ou=mail,dc=MyDomain,dc=Local" scope=2 deref=0 filter="(mailAddress=root@wm-01.mydomain.local)" 0010: 6d 61 69 6c 2c 64 63 3d 4e 65 74 4f 63 65 61 6e mail,dc=Mydomain 0020: 2c 64 63 3d 4c 6f 63 61 6c 0a 01 02 0a 01 00 02 ,dc=Local....... 0030: 01 00 02 01 0a 01 01 00 a3 29 04 0b 6d 61 69 6c .........)..mail 0040: 41 64 64 72 65 73 73 04 1a 72 6f 6f 74 40 77 6d Address..root@wm ldap_read: want=8 error=Resource temporarily unavailable ber_dump: buf=0x947c19070 ptr=0x947c19070 end=0x947c1907c len=12 553e8a87 conn=1011 op=1 do_bind ber_scanf fmt ({imt) ber: 553e8a87 <= mdb_list_candidates: id=0 first=0 last=0 553e8a87 daemon: select: listen=10 active_threads=0 tvp=NULL 0440: 2d 30 31 2e 4e 65 74 4f 63 65 61 6e 2e 4c 6f 63 -01.Mydomain.Loc 0450: 61 6c 2f 50 4b 49 2f 43 41 2f 53 69 67 6e 69 6e al/PKI/CA/Signin 0000: aa 58 54 9e 30 18 bb 75 df 8e 62 e4 62 b0 5e 39 .XT.0..u..b.b.^9 553e8a87 send_ldap_response: msgid=3 tag=101 err=0 ber_flush2: 14 bytes to sd 19 TLS trace: SSL_accept:SSLv3 write finished A 06b0: aa 09 24 e0 e3 6a c8 f2 51 06 ca 80 0c 3c c3 f8 ..$..j..Q....<.. 06c0: 32 df 78 c7 1d 90 62 d0 a3 34 cd 63 18 fa 96 1f 2.x...b..4.c.... tls_write: want=226, written=226 06d0: 4d 3e 23 5a 7f 8f c0 59 13 58 e7 c0 ad 15 a0 53 M>#Z...Y.X.....S 06e0: 7d 48 35 a8 6c 5f d8 f2 b3 cc e2 dd 0e b0 54 02 }H5.l_........T. 553e8a87 <= mdb_filter_candidates: id=0 first=0 last=0 553e8a87 mdb_search_candidates: id=0 first=0 last=0 553e8a87 mdb_search: no candidates 0050: 35 53 8d a9 94 ee bc 81 ab 9e ca af 51 dc 86 18 5S..........Q... 0060: f2 83 8b 13 23 1c b3 18 6e 9e 90 ce 07 7b 31 c1 ....#...n....{1. 0070: 95 f3 7c 1e 85 14 4b 54 4b 52 8b ..|...KTKR. ldap_read: want=91, got=91 ldap_read: want=8, got=8 ber_dump: buf=0x947171180 ptr=0x947171180 end=0x9471711f6 len=118 0000: 02 01 03 63 71 04 28 6f 75 3d 61 63 63 6f 75 6e ...cq.(ou=accoun 0000: 16 03 03 00 aa 04 00 00 a6 00 00 01 2c 00 a0 2b ............,..+ ber_dump: buf=0x948179180 ptr=0x948179183 end=0x9481791f6 len=115 0000: 63 71 04 28 6f 75 3d 61 63 63 6f 75 6e 74 73 2c cq.(ou=accounts, 0010: 6f 75 3d 6d 61 69 6c 2c 64 63 3d 4e 65 74 4f 63 ou=mail,dc=NetOc 553e8a87 send_ldap_result: conn=1005 op=2 p=3 553e8a87 conn=1006 op=2 SRCH attr=mailAlias 553e8a87 send_ldap_result: err=0 matched="" text="" 0010: 51 bd aa 7d 3f 1c 50 fb 25 f8 59 9e 9d 9a ba 0f Q..}?.P.%.Y..... 0020: d0 07 aa 95 ac 1c e7 3e 81 f6 e6 0b 6d 09 94 9b .......>....m... 0730: 1b 51 e3 08 4b 38 ec f1 ee 8c 0f 35 cd 55 eb 80 .Q..K8.....5.U.. 553e8a87 ==> limits_get: conn=1006 op=2 self="[anonymous]" this="ou=accounts,ou=mail,dc=mydomain,dc=local" 0740: 83 e2 3b b5 13 fd 08 51 13 25 d9 7d 57 9f 6b e9 ..;....Q.%.}W.k. [New Thread 943c11800 (LWP 100198/slapd)]
Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 943c11800 (LWP 100198/slapd)] mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404 404 search.c: No such file or directory. in search.c Current language: auto; currently minimal (gdb) bt full #0 mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404 mdb = (struct mdb_info *) 0x80290a000 id = 0 cursor = 0 nsubs = 128 ncand = 0 cscope = 0 lastid = 18446744073709551615 candidates = Error accessing memory address 0x7ffffeafb6f0: Bad address. (gdb)
--On Tuesday, April 28, 2015 2:14 PM +0200 Leander Schäfer info@netocean.de wrote:
Looks like your liblber was installed without debug symbols. Most of these stack traces look invalid.
What does this mean? How did you see this / what indicated this to you? Is it required to fix this liblber issue for a better debug result, or is it ok for first diagnosis?
It means either or both of:
You didn't compile with CFLAGS="-g" (debugging) and/or when you did install, you didn't do:
make install STRIP=""
so that the debugging symbols don't get stripped out.
I generally recommend compiling slapd with:
CFLAGS="-g -O0"
I.e., debugging symbols, and no attempts at optimizing the code. Generally my experience with various compilers has shown that at least in the OpenLDAP Case, higher optimization levels are not beneficial and simply make debugging significantly harder.
--Quanah
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Here is another (3rd) output of 'slapd -d -1 [...]' debugging. It looks a little more different than the previous two.
# ======================================================================== #
[...]
553dfbce => acl_get: [1] attr entry ldap_write: want=14, written=14 553dfbce => acl_mask: access to entry "ou=domains,ou=mail,dc=mydomain,dc=local", attr "entry" requested 553dfbce daemon: activity on 2 descriptors 553dfbce daemon: waked 553dfbce connection_get(25): got connid=1011 0040: 3f 8e 7f 94 2d 99 3e 60 41 93 73 b3 0e de d3 96 ?...-.>`A.s..... 0050: 3f 93 74 5c 06 a4 c3 18 21 ec dd bd 87 5e 84 ed ?.t....!....^.. 0560: 4c 6f 63 61 6c 87 04 c0 a8 32 65 87 04 7f 00 00 Local....2e..... 0060: a3 a1 03 5a cb 52 1c 75 db e9 bb ab 0d 5e 2d 97 ...Z.R.u.....^-. 0070: 93 0e 73 71 62 20 93 ef 76 f0 b8 6e 44 73 1d f4 ..sqb ..v..nDs.. 553dfbce ==> limits_get: conn=1006 op=2 self="[anonymous]" this="ou=accounts,ou=mail,dc=mydomain,dc=local" 0080: c3 49 7f 6e 49 bd e4 e0 7d 70 8b 12 46 39 f1 2b .I.nI...}p..F9.+ 0000: 30 0c 02 01 03 65 07 0a 01 00 04 00 04 00 0....e........ ber_dump: buf=0x945c75180 ptr=0x945c75183 end=0x945c751f6 len=115 Segmentation fault
# ======================================================================== #
openldap-technical@openldap.org