9:17 a.m.
Hi fellows
my OpenLDAP keeps on dying sporadically. Unfortunately it doesn't leave
ANY trace of why it crashes. It stays alive for a while .... takes a
couple of queries as expected ... and then suddenly it dies after
another request. This could mean working perfectly for a day or working
only for a couple of minutes before it crashes. The requests are all
known to work (Dovecot auth and system PAM auth). At first I thought it
could be related to newsyslog(8) ... I did some testing - unfortunately
this doesn't change anything at all. It keeps on dying sporadically even
if newsyslog is completely deactivated for the slapd(8) log file.
I experenced this behauviour since I first used OpenLDAP approximately 5
years ago. My OpenLDAP configurations have been changed many times and
quite a lot over those 5 years. Unfortunately I was never able to figure
out what the sudden crashes caused - so I wrote a supervisor script ,
which periodically checks if the slapd(8) is still running. If slapd(8)
is not running, then my supervisor script starts it automatically again.
Of course with every major change in the configuration of OpenLDAP, I
also tried to debug this problem of the sudden crashes - but as I said:
even with all debug levels activated - slapd(8) didn't want to share its
root of the pain. This workaround was the only way I could use OpenLDAP
at all over the past five years.
Two days ago I got to the point where my workaround came to its capacity
boundary. The time between the slapd(8) crashes has become so little, so
my script can't handle them anylonger. I would have two options now:
a) Fix my supervisor script, so it detects and starts slapd(8) even if
the time between crashes is less than 3 sec.
b) I ask for help in order to discover this mysterious cause of the
crashes and FINALLY fix it FOREVER.
So this time I set my foot down and go for b). This is why I decided to
contact you. The following link containes a more detailed debugging of
the problem:
https://forums.freebsd.org/threads/openldap-slapd-dies-sporadically.47634/
And this is the current / latest configuration I deploy on my servers.
Also, it doesn't make a difference whether I have monitoring activated
or not. It keeps on dieing the same as before. The very same ocours to
the "ldapab.schema" - it doesn't make a difference for the crashes to
happen whether it is commented out or activated. Logging hapens through
syslogd(8).
# ============================================================ #
#======================================#
# slapd.conf #
#======================================#
# LDAP Attribut Schema Definitions:
include /usr/local/etc/openldap/schema/core.schema
include /usr/local/etc/openldap/schema/cosine.schema
include /usr/local/etc/openldap/schema/inetorgperson.schema
include /usr/local/etc/openldap/schema/nis.schema
include /usr/local/etc/openldap/schema/ldapab.schema
include /usr/local/etc/openldap/schema/mail.schema
#include /usr/local/etc/openldap/schema/qouta.schema
# Logging
loglevel 256
# If "logfile" is activated, then newsyslog must restart slapd after
rotation
#logfile /var/log/slapd.log
# Important Files
pidfile /var/run/openldap/slapd.pid
argsfile /var/run/openldap/slapd.args
# Allow LDAPv2 client connections. This is required for iOS suppoprt via
SSL.
#allow bind_v2
# Time limits
#idletimeout 0
#writetimeout 0
#timelimit 3600
# Dynamic Modules:
modulepath /usr/local/libexec/openldap
moduleload back_mdb
moduleload back_monitor
# Access from 127.0.0.1 without encryption
access to dn.subtree="dc=MyDomain,dc=local"
by peername.ip=127.0.0.1 write
by * none break
# Access from other than 127.0.0.1 requires TLS
# encryption with min. 128 bit encryption
access to dn.subtree="dc=MyDomain,dc=local"
by ssf=128 write
by * none
# TLS-Certificate
TLSCertificateFile /etc/ssl/PKI/CA/Signing-CA/Certs/WM-01.MyDomain.Local.crt
TLSCertificateKeyFile
/etc/ssl/PKI/CA/Signing-CA/Private/WM-01.MyDomain.Local.key
TLSVerifyClient allow
# Restrict write access for password attributes to root user only
by dn="uid=dovecot,ou=systemuser,ou=mail,dc=MyDomain,dc=Local"
access to attrs=userPassword
by self write
by anonymous auth
by * none
# All the other atributes can be read by everyone else
access to *
by * read
# ================================== #
# Internal Monitoring #
# ================================== #
# DB type
database monitor
# Name of Administrator
rootdn "cn=monitoring,cn=Monitor"
# Password of Administrators
rootpw {SSHA}SomeHash
# ACL for moitoring
access to dn.subtree="cn=Monitor"
by dn.exact="cn=admin,dc=MyDomain,dc=local" write
by peername.ip=127.0.0.1 read
by users read
by * none
# ================================== #
# Lightning Memory-Mapped Database #
# ================================== #
# Datenbanktyp
database mdb
# DB location on FS
directory /var/db/openldap-data
# Datenbank-Pfad
suffix "dc=MyDomain,dc=local"
# Name of Administrator
rootdn "cn=admin,dc=MyDomain,dc=local"
# Password of Administrators
rootpw {SSHA}SomeHash
# Max size of DB in Byte [ 5368709120 Byte => 5 GB ]
maxsize 5368709120
# Index search related attributes
index objectClass eq
index uid eq
index uidNumber eq
index uniqueMember eq
index gidNumber eq
index cn eq
index memberUid eq
index mailAccountStatus eq
index mailAddress eq
index associatedDomain eq
# ============================================================= #
I compiled OpenLDAP from FreeBSD ports with following options:
- DYNAMIC_BACKENDS
- MDB
- PPOLICY
- SYNCPROV
- TCP_WRAPPERS
Please let me know what kind of further information I could deliver to
you in order to detect the root of the problem.
Thank you
Best regards
Leander
4:48 a.m.
Leander Schäfer wrote:
my OpenLDAP keeps on dying sporadically.
Which OpenLDAP version is this?
Please note that OpenLDAP project does not know details about binary
distribution releases.
Did you try with your own build from source using a recent OpenLDAP release?
Unfortunately it doesn't leave ANY
trace of why it crashes.
I'm not familiar with FreeBSD. But try this to get more details:
1. Run with ulimit -c unlimited to get a core dump file.
2. Provide gdb stack trace like described herein:
http://www.openldap.org/faq/data/cache/59.html
Ciao, Michael.
2:02 a.m.
Hi Michael,
Hi Ulrich,
Thanks for your reply. I'm running Version 2.4.40. As I said, I do not
run binary version. I always compile OpenLDAP from sources / latest
ports tree.
The maximum core file size "ulimit -c" was already set to unlimited. I'm
using bash-4.3.30
root@FreeBASD # ulimit -a
socket buffer size (bytes, -b) unlimited
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) 33554432
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) 131072
max memory size (kbytes, -m) 3012064
open files (-n) 87651
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 7592
virtual memory (kbytes, -v) unlimited
swap size (kbytes, -w) unlimited
root@FreeBSD # /usr/local/libexec/slapd -d -1 -f
/usr/local/etc/openldap/slapd.conf -u ldap -g ldap -h
"ldapi://%2fvar%2frun%2fopenldap%2fldapi/ ldap:/// ldaps:///"
# =================================================================== #
[...]
I have removed the previous lines for this demo
[...]
553ded5a mailUidNumber 0040: 2e e8 e7 5a 9a fe a0 8c 02 96 e9 58 48
e0 49 10 ...Z.......XH.I.
0050: cd 10 08 6c 20 1f 9d bc ae dd 9a 4a 79 7e f2 3c ...l
......Jy~.<
0060: 25 a2 72 fe ac cc d0 09 eb 62 d2 bd 95 c8 50 7f %.r......b....P.
0020: b0 86 e3 1d 11 32 2d 8b fd 57 a6 a4 ce a2 ee 2f .....2-..W...../
0030: 52 59 da 4d 7b d1 5c 0c 22 34 29 86 c2 9c 80 72 RY.M{.\."4)....r
0040: ca 94 4d 69 2e ..Mi.
0010: 73 2c 6f 75 3d 6d 61 69 6c 2c 64 63 3d 4e 65 74 s,ou=mail,dc=MyD
0020: 4f 63 65 61 6e 2c 64 63 3d 4c 6f 63 61 6c 0a 01 omain,dc=Local..
0090: 8e 5f 68 e0 0a 31 26 07 da 21 c6 cd 27 0e 17 2b ._h..1&..!..'..+
00a0: fb 53 5e 0a 84 74 50 b8 74 13 a5 fa e2 02 9a ee .S^..tP.t.......
00b0: 5e ee 8e 6c b2 d3 b6 6e 82 6d 01 ab eb 81 25 bd ^..l...n.m....%.
00c0: f1 05 16 5b 7f 9e bb 76 7c ae ba a2 24 73 89 78 ...[...v|...$s.x
0070: ca 00 ec a3 7b 97 78 19 fe aa 56 fc a1 a5 9a 1e ....{.x...V.....
0080: 65 f0 04 b7 04 08 af 7a 82 ef 77 e......z..w
553ded5a mailGidNumberldap_read: want=8, got=8
553ded5a mailQuotaStorage 0000: 30 71 02 01 0b 63 6c
04 0q...cl.
ldap_read: want=107, got=107
00d0: 6c ae 39 eb 15 85 4a f9 c1 6f 65 ca 4f c6 db 14 l.9...J..oe.O...
ldap_write: want=14, written=14
00e0: c1 f4 fe b8 b5 a4 a3 75 96 b1 9b 9b 8f d5 d6 e5 .......u........
00f0: 8d 3c 75 4c 50 cc 9d 85 cf bb a1 d8 50 21 93 fa .<uLP.......P!..
0100: 38 ee 89 46 45 4e 06 17 7a 8c 4b 83 51 95 a9 ca 8..FEN..z.K.Q...
0110: 71 8c d0 b9 59 1a 14 f4 10 8d b9 bc 80 d5 cb e9 q...Y...........
0120: 46 03 a2 ce 59 49 0b db fc ea a3 3b fa cd a1 99 F...YI.....;....
0030: 02 0a 01 00 02 01 00 02 01 78 01 01 00 a0 5a a3 .........x....Z.
0040: 1a 04 0b 6f 62 6a 65 63 74 43 6c 61 73 73 04 0b ...objectClass..
0130: 14 90 d6 0a 55 da 84 b1 42 fe af 8d 14 92 ce 27 ....U...B......'
0050: 6d 61 69 6c 41 63 63 6f 75 6e 74 a3 1b 04 11 6d mailAccount....m
0060: 61 69 6c 41 63 63 6f 75 6e 74 53 74 61 74 75 73 ailAccountStatus
0000: 30 0c 02 01 06 65 07 0a 01 00 04 00 04 00 0....e........
0070: 04 06 61 63 74 69 76 65 a3 1f 04 0b 6d 61 69 6c ..active....mail
0080: 41 64 64 72 65 73 73 04 10 69 6e 66 6f 40 6e 65 Address..info@So
0090: 74 6f 63 65 61 6e 2e 64 65 30 6f 04 14 6d 61 69 meDom.tld0o..mai
00a0: 6c 53 74 6f 72 61 67 65 44 69 72 65 63 74 6f 72 lStorageDirector
553ded5a mailQuotaMessages553ded5a
00b0: 79 04 14 6d 61 69 6c 53 74 6f 72 61 67 65 44 69 y..mailStorageDi
553ded5a conn=1008 op=5 SRCH
base="ou=accounts,ou=mail,dc=MyDomain,dc=Local" scope=2 deref=0
filter="(&(objectClass=mailAccount)(mailAccountStatus=active)(mailAddress=info(a)somedom.tld))"
00c0: 72 65 63 74 6f 72 79 04 0d 6d 61 69 6c 55 69 64 rectory..mailUid
0000: 28 6f 75 3d 61 63 63 6f 75 6e 74 73 2c 6f 75 3d (ou=accounts,ou=
553ded5a conn=1008 op=4 SEARCH RESULT tag=101 err=0 nentries=0 text=
00d0: 4e 75 6d 62 65 72 04 0d 6d 61 69 6c 47 69 64 4e Number..mailGidN
553ded5a conn=1008 op=5 SRCH attr=mailStorageDirectory
mailStorageDirectory mailUidNumber mailGidNumber mailQuotaStorage
mailQuotaMessages
ldap_read: want=8, got=8
0000: 30 82 01 0d 02 01 09 63 0......c
ldap_read: want=265, got=265
0000: 82 01 06 04 28 6f 75 3d 61 63 63 6f 75 6e 74 73 ....(ou=accounts
0010: 2c 6f 75 3d 6d 61 69 6c 2c 64 63 3d 4e 65 74 4f ,ou=mail,dc=MyDo
553ded5a ==> limits_get: conn=1008 op=5
self="uid=dovecot,ou=systemuser,ou=mail,dc=mydomain,dc=local"
this="ou=accounts,ou=mail,dc=mydomain,dc=local"
00e0: 75 6d 62 65 72 04 10 6d 61 69 6c 51 75 6f 74 61 umber..mailQuota
Segmentation fault
# =================================================================== #
... and a second time the same debugging (-d -1)
# =================================================================== #
[...]
553df2b8 conn=1000 op=6 SRCH
base="ou=accounts,ou=mail,dc=MyDomain,dc=Local" scope=2 deref=0
filter="(&(objectClass=mailAccount)(mailAccountStatus=active)(mailAddress=info(a)somedom.tld))"
0010: 70 7b e7 97 2e b3 3e 24 f8 bc 52 c5 ce 0d b8 fb p{....>$..R.....
0020: 89 cb 07 f8 d3 34 0a 39 9a fc 0d 0d b9 bb d4 13 .....4.9........
0030: 8d 1f 56 87 c0 9c 10 2b a2 57 ee 3b 4b 93 6a 7c ..V....+.W.;K.j|
TLS trace: SSL_accept:SSLv3 read finished A
553df2b8 conn=1000 op=6 SRCH attr=mailStorageDirectory
mailStorageDirectory mailUidNumber mailGidNumber mailQuotaStorage
mailQuotaMessages
553df2b8 connection_read(20): checking for input on id=1006
553df2b8 ==> limits_get: conn=1000 op=6
self="uid=dovecot,ou=systemuser,ou=mail,dc=mydomain,dc=local"
this="ou=accounts,ou=mail,dc=mydomain,dc=local"
0040: 54 2b b2 49 7e f7 ce 89 19 96 fe 83 86 77 5a 3a T+.I~........wZ:
Segmentation fault
# =================================================================== #
I'm currently trying to figure out a way to add "-O0 -g" gdb(1) support
for the slapd(8C) command, since I'm compiling OpenLDAP sources from
ports tree. I'll post it as soon as I got it. But maybe you can already
assume some pre-diagnosis from the above output?
@Ulrich:
- Yes, I have tried several hardware plattforms, different physical,
VmWare and now the latest XenServer. Same results all over.
- A detailed log debug can be seen on:
https://forums.freebsd.org/threads/openldap-slapd-dies-sporadically.47634/
- Be warned from the above diagnosis in the forum: It is NOT the
assumed newsyslog(8) which causes the crashes!
- What exactly do you mean by "exists cleanly"?
Best regards
Leander
Am 27.04.15 um 08:59 schrieb Ulrich Windl:
Hi!
You should tell about the operation you do when slapd crashes, maybe as well
the last log messages you see. Also: Did you try different hardware and
different compilers to run and compile slapd? What about doing an "-O0 -g"
build and enable core dumps? Or are you telling slapd exits "cleanly"?
Regards,
Ulrich
Am 26.04.15 um 13:48 schrieb Michael Ströder:
Leander Schäfer wrote:
> my OpenLDAP keeps on dying sporadically.
Which OpenLDAP version is this?
Please note that OpenLDAP project does not know details about binary
distribution releases.
Did you try with your own build from source using a recent OpenLDAP
release?
> Unfortunately it doesn't leave ANY
> trace of why it crashes.
I'm not familiar with FreeBSD. But try this to get more details:
1. Run with ulimit -c unlimited to get a core dump file.
2. Provide gdb stack trace like described herein:
http://www.openldap.org/faq/data/cache/59.html
Ciao, Michael.
2:12 a.m.
New subject: Antw: Re: OpenLDAP keeps on dying sporadically
>> Leander Schäfer<info(a)netocean.de> schrieb am 27.04.2015
um 11:02 in
Nachricht
<553DFB24.5050201(a)netocean.de>:
[...]
I'm currently trying to figure out a way to add "-O0
-g" gdb(1) support
for the slapd(8C) command, since I'm compiling OpenLDAP sources from
ports tree. I'll post it as soon as I got it. But maybe you can already
assume some pre-diagnosis from the above output?
No, but you should have a core file after segmentation fault. What about
something like "gdb /your_slapd /your/core" and "bt" and "info
threads"?
Probably not very useful unless you have debugging symbols in the binary.
Regards,
Ulrich
2:17 a.m.
I'll deliver the gdb(1) results as soon as I'm able to compile with
relevant options. Meanwhile, here is some more debugging results about
the break down of 'slapd -d -1 [...]'
# ======================================================================== #
[...]
553dfcc9 filter: (&(associatedDomain=
somedom.tld)(!(destinationIndicator=*)))
ber_scanf fmt ({M}}) ber:
ber_dump: buf=0x94b40f020 ptr=0x94b40f09a end=0x94b40f0ae len=20
0000: 00 12 04 10 61 73 73 6f 63 69 61 74 65 64 44 6f ....associatedDo
0010: 6d 61 69 6e main
553dfcc9 attrs:553dfcc9 associatedDomain553dfcc9
553dfcc9 conn=1007 op=331 SRCH
base="ou=domains,ou=mail,dc=MyDomain,dc=Local" scope=2 deref=0
filter="(&(associatedDomain=somedom.tld)(!(destinationIndicator=*)))"
ber_get_next: tag 0x30 len 105 contents:
ber_dump: buf=0x943d502b0 ptr=0x943d502b0 end=0x943d50319 len=105
0000: 02 01 0c 63 64 04 28 6f 75 3d 61 63 63 6f 75 6e ...cd.(ou=accoun
0010: 74 73 2c 6f 75 3d 6d 61 69 6c 2c 64 63 3d 4e 65 ts,ou=mail,dc=Ne
0020: 74 4f 63 65 61 6e 2c 64 63 3d 4c 6f 63 61 6c 0a tOcean,dc=Local.
0030: 01 02 0a 01 00 02 01 00 02 01 0a 01 01 00 a3 1c ................
0040: 04 0b 6d 61 69 6c 41 64 64 72 65 73 73 04 0d 40 ..mailAddress..@
0050: 6e 65 74 6f 63 65 61 6e 2e 64 65 20 30 0b 04 09 somedom.tld 0...
0060: 6d 61 69 6c 41 6c 69 61 73 mailAlias
553dfcc9 conn=1007 op=331 SRCH attr=associatedDomain
553dfcc9 op tag 0x63, time 1430125769
ber_get_next
553dfcc9 ==> limits_get: conn=1007 op=331 self="[anonymous]"
this="ou=domains,ou=mail,dc=mydomain,dc=local"
tls_read: want=5 error=Resource temporarily unavailable
ldap_read: want=8 error=Resource temporarily unavailable
Segmentation fault
# ======================================================================== #
[...]
553dfd7c daemon: select: listen=6 active_threads=0 tvp=NULL
553dfd7c conn=1008 op=2 SRCH attr=associatedDomain
TLS trace: SSL_accept:SSLv3 read client key exchange A
TLS trace: SSL_accept:SSLv3 read finished A
tls_read: want=5, got=5
553dfd7c ==> limits_get: conn=1008 op=2 self="[anonymous]"
this="ou=domains,ou=mail,dc=mydomain,dc=local"
553dfd7c daemon: select: listen=7 active_threads=0 tvp=NULL
553dfd7c daemon: select: listen=8 active_threads=0 tvp=NULL
553dfd7c daemon: select: listen=9 active_threads=0 tvp=NULL
553dfd7c => mdb_equality_candidates (associatedDomain)
Segmentation fault
# ======================================================================== #
[...]
553dfdf3 => mdb_list_candidates 0xa1
553dfdf3 => mdb_filter_candidates
553dfdf3 EQUALITY
553dfdf3 => mdb_equality_candidates (objectClass)
553dfdf3 => key_read
553dfdf3 mdb_idl_fetch_key: [b49d1940]
553dfdf3 <= mdb_index_read: failed (-30798)
553dfdf3 <= mdb_equality_candidates: id=0, first=0, last=0
553dfdf3 <= mdb_filter_candidates: id=0 first=0 last=0
553dfdf3 => mdb_filter_candidates
553dfdf3 EQUALITY
553dfdf3 conn=1002 op=1131 SRCH attr=associatedDomain
553dfdf3 => mdb_equality_candidates (mailAddress)
553dfdf3 => key_read
553dfdf3 mdb_idl_fetch_key: [a4af5673]
553dfdf3 ==> limits_get: conn=1002 op=1131 self="[anonymous]"
this="ou=domains,ou=mail,dc=mydomain,dc=local"
553dfdf3 <= mdb_index_read: failed (-30798)
553dfdf3 <= mdb_equality_candidates: id=0, first=0, last=0
553dfdf3 <= mdb_filter_candidates: id=0 first=0 last=0
553dfdf3 <= mdb_list_candidates: id=0 first=0 last=0
553dfdf3 <= mdb_filter_candidates: id=0 first=0 last=0
Segmentation fault
Am 27.04.15 um 11:02 schrieb Leander Schäfer:
Hi Michael,
Hi Ulrich,
Thanks for your reply. I'm running Version 2.4.40. As I said, I do not
run binary version. I always compile OpenLDAP from sources / latest
ports tree.
The maximum core file size "ulimit -c" was already set to unlimited.
I'm using bash-4.3.30
root@FreeBASD # ulimit -a
socket buffer size (bytes, -b) unlimited
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) 33554432
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) 131072
max memory size (kbytes, -m) 3012064
open files (-n) 87651
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 7592
virtual memory (kbytes, -v) unlimited
swap size (kbytes, -w) unlimited
root@FreeBSD # /usr/local/libexec/slapd -d -1 -f
/usr/local/etc/openldap/slapd.conf -u ldap -g ldap -h
"ldapi://%2fvar%2frun%2fopenldap%2fldapi/ ldap:/// ldaps:///"
# =================================================================== #
[...]
I have removed the previous lines for this demo
[...]
553ded5a mailUidNumber 0040: 2e e8 e7 5a 9a fe a0 8c 02 96 e9 58
48 e0 49 10 ...Z.......XH.I.
0050: cd 10 08 6c 20 1f 9d bc ae dd 9a 4a 79 7e f2 3c ...l
......Jy~.<
0060: 25 a2 72 fe ac cc d0 09 eb 62 d2 bd 95 c8 50 7f
%.r......b....P.
0020: b0 86 e3 1d 11 32 2d 8b fd 57 a6 a4 ce a2 ee 2f
.....2-..W...../
0030: 52 59 da 4d 7b d1 5c 0c 22 34 29 86 c2 9c 80 72
RY.M{.\."4)....r
0040: ca 94 4d 69 2e ..Mi.
0010: 73 2c 6f 75 3d 6d 61 69 6c 2c 64 63 3d 4e 65 74
s,ou=mail,dc=MyD
0020: 4f 63 65 61 6e 2c 64 63 3d 4c 6f 63 61 6c 0a 01
omain,dc=Local..
0090: 8e 5f 68 e0 0a 31 26 07 da 21 c6 cd 27 0e 17 2b
._h..1&..!..'..+
00a0: fb 53 5e 0a 84 74 50 b8 74 13 a5 fa e2 02 9a ee
.S^..tP.t.......
00b0: 5e ee 8e 6c b2 d3 b6 6e 82 6d 01 ab eb 81 25 bd
^..l...n.m....%.
00c0: f1 05 16 5b 7f 9e bb 76 7c ae ba a2 24 73 89 78
...[...v|...$s.x
0070: ca 00 ec a3 7b 97 78 19 fe aa 56 fc a1 a5 9a 1e
....{.x...V.....
0080: 65 f0 04 b7 04 08 af 7a 82 ef 77 e......z..w
553ded5a mailGidNumberldap_read: want=8, got=8
553ded5a mailQuotaStorage 0000: 30 71 02 01 0b 63 6c
04 0q...cl.
ldap_read: want=107, got=107
00d0: 6c ae 39 eb 15 85 4a f9 c1 6f 65 ca 4f c6 db 14
l.9...J..oe.O...
ldap_write: want=14, written=14
00e0: c1 f4 fe b8 b5 a4 a3 75 96 b1 9b 9b 8f d5 d6 e5
.......u........
00f0: 8d 3c 75 4c 50 cc 9d 85 cf bb a1 d8 50 21 93 fa
.<uLP.......P!..
0100: 38 ee 89 46 45 4e 06 17 7a 8c 4b 83 51 95 a9 ca
8..FEN..z.K.Q...
0110: 71 8c d0 b9 59 1a 14 f4 10 8d b9 bc 80 d5 cb e9
q...Y...........
0120: 46 03 a2 ce 59 49 0b db fc ea a3 3b fa cd a1 99
F...YI.....;....
0030: 02 0a 01 00 02 01 00 02 01 78 01 01 00 a0 5a a3
.........x....Z.
0040: 1a 04 0b 6f 62 6a 65 63 74 43 6c 61 73 73 04 0b
...objectClass..
0130: 14 90 d6 0a 55 da 84 b1 42 fe af 8d 14 92 ce 27
....U...B......'
0050: 6d 61 69 6c 41 63 63 6f 75 6e 74 a3 1b 04 11 6d
mailAccount....m
0060: 61 69 6c 41 63 63 6f 75 6e 74 53 74 61 74 75 73
ailAccountStatus
0000: 30 0c 02 01 06 65 07 0a 01 00 04 00 04 00 0....e........
0070: 04 06 61 63 74 69 76 65 a3 1f 04 0b 6d 61 69 6c
..active....mail
0080: 41 64 64 72 65 73 73 04 10 69 6e 66 6f 40 6e 65
Address..info@So
0090: 74 6f 63 65 61 6e 2e 64 65 30 6f 04 14 6d 61 69
meDom.tld0o..mai
00a0: 6c 53 74 6f 72 61 67 65 44 69 72 65 63 74 6f 72
lStorageDirector
553ded5a mailQuotaMessages553ded5a
00b0: 79 04 14 6d 61 69 6c 53 74 6f 72 61 67 65 44 69
y..mailStorageDi
553ded5a conn=1008 op=5 SRCH
base="ou=accounts,ou=mail,dc=MyDomain,dc=Local" scope=2 deref=0
filter="(&(objectClass=mailAccount)(mailAccountStatus=active)(mailAddress=info(a)somedom.tld))"
00c0: 72 65 63 74 6f 72 79 04 0d 6d 61 69 6c 55 69 64
rectory..mailUid
0000: 28 6f 75 3d 61 63 63 6f 75 6e 74 73 2c 6f 75 3d
(ou=accounts,ou=
553ded5a conn=1008 op=4 SEARCH RESULT tag=101 err=0 nentries=0 text=
00d0: 4e 75 6d 62 65 72 04 0d 6d 61 69 6c 47 69 64 4e
Number..mailGidN
553ded5a conn=1008 op=5 SRCH attr=mailStorageDirectory
mailStorageDirectory mailUidNumber mailGidNumber mailQuotaStorage
mailQuotaMessages
ldap_read: want=8, got=8
0000: 30 82 01 0d 02 01 09 63 0......c
ldap_read: want=265, got=265
0000: 82 01 06 04 28 6f 75 3d 61 63 63 6f 75 6e 74 73
....(ou=accounts
0010: 2c 6f 75 3d 6d 61 69 6c 2c 64 63 3d 4e 65 74 4f
,ou=mail,dc=MyDo
553ded5a ==> limits_get: conn=1008 op=5
self="uid=dovecot,ou=systemuser,ou=mail,dc=mydomain,dc=local"
this="ou=accounts,ou=mail,dc=mydomain,dc=local"
00e0: 75 6d 62 65 72 04 10 6d 61 69 6c 51 75 6f 74 61
umber..mailQuota
Segmentation fault
# =================================================================== #
... and a second time the same debugging (-d -1)
# =================================================================== #
[...]
553df2b8 conn=1000 op=6 SRCH
base="ou=accounts,ou=mail,dc=MyDomain,dc=Local" scope=2 deref=0
filter="(&(objectClass=mailAccount)(mailAccountStatus=active)(mailAddress=info(a)somedom.tld))"
0010: 70 7b e7 97 2e b3 3e 24 f8 bc 52 c5 ce 0d b8 fb
p{....>$..R.....
0020: 89 cb 07 f8 d3 34 0a 39 9a fc 0d 0d b9 bb d4 13
.....4.9........
0030: 8d 1f 56 87 c0 9c 10 2b a2 57 ee 3b 4b 93 6a 7c
..V....+.W.;K.j|
TLS trace: SSL_accept:SSLv3 read finished A
553df2b8 conn=1000 op=6 SRCH attr=mailStorageDirectory
mailStorageDirectory mailUidNumber mailGidNumber mailQuotaStorage
mailQuotaMessages
553df2b8 connection_read(20): checking for input on id=1006
553df2b8 ==> limits_get: conn=1000 op=6
self="uid=dovecot,ou=systemuser,ou=mail,dc=mydomain,dc=local"
this="ou=accounts,ou=mail,dc=mydomain,dc=local"
0040: 54 2b b2 49 7e f7 ce 89 19 96 fe 83 86 77 5a 3a
T+.I~........wZ:
Segmentation fault
# =================================================================== #
I'm currently trying to figure out a way to add "-O0 -g" gdb(1)
support for the slapd(8C) command, since I'm compiling OpenLDAP
sources from ports tree. I'll post it as soon as I got it. But maybe
you can already assume some pre-diagnosis from the above output?
@Ulrich:
- Yes, I have tried several hardware plattforms, different physical,
VmWare and now the latest XenServer. Same results all over.
- A detailed log debug can be seen on:
https://forums.freebsd.org/threads/openldap-slapd-dies-sporadically.47634/
- Be warned from the above diagnosis in the forum: It is NOT the
assumed newsyslog(8) which causes the crashes!
- What exactly do you mean by "exists cleanly"?
Best regards
Leander
Am 27.04.15 um 08:59 schrieb Ulrich Windl:
> Hi!
>
> You should tell about the operation you do when slapd crashes, maybe
> as well
> the last log messages you see. Also: Did you try different hardware and
> different compilers to run and compile slapd? What about doing an
> "-O0 -g"
> build and enable core dumps? Or are you telling slapd exits "cleanly"?
>
> Regards,
> Ulrich
>
Am 26.04.15 um 13:48 schrieb Michael Ströder:
> Leander Schäfer wrote:
>> my OpenLDAP keeps on dying sporadically.
>
> Which OpenLDAP version is this?
>
> Please note that OpenLDAP project does not know details about binary
> distribution releases.
>
> Did you try with your own build from source using a recent OpenLDAP
> release?
>
>> Unfortunately it doesn't leave ANY
>> trace of why it crashes.
>
> I'm not familiar with FreeBSD. But try this to get more details:
>
> 1. Run with ulimit -c unlimited to get a core dump file.
>
> 2. Provide gdb stack trace like described herein:
> http://www.openldap.org/faq/data/cache/59.html
>
> Ciao, Michael.
>
9:32 a.m.
--On Monday, April 27, 2015 12:02 PM +0200 Leander Schäfer
<info(a)netocean.de> wrote:
Hi Michael,
Hi Ulrich,
Thanks for your reply. I'm running Version 2.4.40. As I said, I do not
run binary version. I always compile OpenLDAP from sources / latest ports
tree.
Please compile and run current RE24 and see if you can reproduce. There
are multiple segfault fixes present.
OpenLDAP 2.4.41 Engineering
Fixed libldap segfault in ldap_sync_initialize (ITS#8001)
Fixed slapd segfault when using matched values control (ITS#8046)
Fixed slapo-collect segfault (ITS#7797)
Fixed slapo-syncprov segfault on disconnect/abandon
(ITS#5452,ITS#8012)
Fixed slapo-syncprov segfault on disconnect/abandon (ITS#8043)
--Quanah
--
Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
9:51 a.m.
To be frank, I highly doubt it. Those problems have been there since I
first used OpenLDAP five years ago. Neither do I use sync - so none
except the collect fixes looks like it would apply in the first place.
But I'll give it a quick shot. Can you please provide me a link, cause I
wasn't able to find "current RE24" on the official website nor on the
FTP mirror.
Thanks
Am 27.04.15 um 18:32 schrieb Quanah Gibson-Mount:
--On Monday, April 27, 2015 12:02 PM +0200 Leander Schäfer
<info(a)netocean.de> wrote:
> Hi Michael,
> Hi Ulrich,
>
> Thanks for your reply. I'm running Version 2.4.40. As I said, I do not
> run binary version. I always compile OpenLDAP from sources / latest
> ports
> tree.
Please compile and run current RE24 and see if you can reproduce.
There are multiple segfault fixes present.
OpenLDAP 2.4.41 Engineering
Fixed libldap segfault in ldap_sync_initialize (ITS#8001)
Fixed slapd segfault when using matched values control (ITS#8046)
Fixed slapo-collect segfault (ITS#7797)
Fixed slapo-syncprov segfault on disconnect/abandon
(ITS#5452,ITS#8012)
Fixed slapo-syncprov segfault on disconnect/abandon (ITS#8043)
--Quanah
--
Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
10:04 a.m.
Leander Schäfer wrote:
Can you please provide me a link, cause I wasn't able to find
"current RE24" on the official website nor on the FTP mirror.
Use git or this link to checkout snapshot of the RE24 branch:
http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=snapshot;h=refs...
Ciao, Michael.
1:51 p.m.
Ok, here is the first result running the debugging mode with gdb(1)
> Procedure overview:
(gdb) run
(gdb) bt full
(gdb) thread apply all bt
(gdb) generate-core-file
> This came up:
candidates = Error accessing memory address
0x7ffffeafb6f0: Bad address.
# ================================================== #
root@FreeBSD [~]$ gdb --args /usr/local/libexec/slapd -d -1 -f
/usr/local/etc/openldap/slapd.conf -u ldap -g ldap -h
"ldapi://%2fvar%2frun%2fopenldap%2fldapi/ ldap:/// ldaps:///"
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
(gdb) run
Starting program: /usr/local/libexec/slapd -d -1 -f
/usr/local/etc/openldap/slapd.conf -u ldap -g ldap -h
ldapi://%2fvar%2frun%2fopenldap%2fldapi/\ ldap:///\ ldaps:///
[New LWP 101138]
[New Thread 802806400 (LWP 101138/slapd)]
[...]
553e8a87 conn=1006 op=2 SRCH attr=mailAlias
553e8a87 send_ldap_result: err=0 matched="" text=""
0010: 51 bd aa 7d 3f 1c 50 fb 25 f8 59 9e 9d 9a ba 0f Q..}?.P.%.Y.....
0020: d0 07 aa 95 ac 1c e7 3e 81 f6 e6 0b 6d 09 94 9b .......>....m...
0730: 1b 51 e3 08 4b 38 ec f1 ee 8c 0f 35 cd 55 eb 80 .Q..K8.....5.U..
553e8a87 ==> limits_get: conn=1006 op=2 self="[anonymous]"
this="ou=accounts,ou=mail,dc=mydomain,dc=local"
0740: 83 e2 3b b5 13 fd 08 51 13 25 d9 7d 57 9f 6b e9 ..;....Q.%.}W.k.
[New Thread 943c11800 (LWP 100198/slapd)]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 943c11800 (LWP 100198/slapd)]
mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404
404 search.c: No such file or directory.
in search.c
Current language: auto; currently minimal
(gdb) bt full
#0 mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404
mdb = (struct mdb_info *) 0x80290a000
id = 0
cursor = 0
nsubs = 128
ncand = 0
cscope = 0
lastid = 18446744073709551615
candidates = Error accessing memory address 0x7ffffeafb6f0: Bad
address.
(gdb) thread apply all bt
[New Thread 943c15000 (LWP 101255/slapd)]
[New Thread 943c14c00 (LWP 101213/slapd)]
[New Thread 943c14800 (LWP 101202/slapd)]
[New Thread 943c14400 (LWP 100898/slapd)]
[New Thread 943c14000 (LWP 100884/slapd)]
[New Thread 943c13c00 (LWP 100647/slapd)]
[New Thread 943c13800 (LWP 100619/slapd)]
[New Thread 943c13400 (LWP 100577/slapd)]
[New Thread 943c13000 (LWP 100531/slapd)]
[New Thread 943c12c00 (LWP 100515/slapd)]
[New Thread 943c12800 (LWP 100347/slapd)]
[New Thread 943c12400 (LWP 100311/slapd)]
[New Thread 943c12000 (LWP 100296/slapd)]
[New Thread 943c11c00 (LWP 100268/slapd)]
[New Thread 943c11400 (LWP 100165/slapd)]
[New Thread 802807800 (LWP 100103/slapd)]
Thread 19 (Thread 802807800 (LWP 100103/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3
#2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7
#3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7
#4 0x0000000800bfd48f in lutil_debug () from
/usr/local/lib/liblber-2.4.so.2
#5 0x000000000043b96f in slapd_daemon_task (ptr=0x8028afb08) at
daemon.c:2530
#6 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
#7 0x0000000000000000 in ?? ()
Thread 18 (Thread 943c11400 (LWP 100165/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3
#2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7
#3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7
#4 0x0000000800bfcc66 in ber_error_print () from
/usr/local/lib/liblber-2.4.so.2
#5 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2
#6 0x0000000800bfcf4a in ber_log_bprint () from
/usr/local/lib/liblber-2.4.so.2
#7 0x0000000800c0017e in ber_int_sb_write () from
/usr/local/lib/liblber-2.4.so.2
#8 0x00000008009e1905 in ldap_start_tls_s () from
/usr/local/lib/libldap_r-2.4.so.2
#9 0x0000000801807e84 in BIO_read () from /lib/libcrypto.so.7
#10 0x000000080146ce97 in ssl3_read_n () from /usr/lib/libssl.so.7
#11 0x000000080146db61 in ssl3_read_bytes () from /usr/lib/libssl.so.7
#12 0x0000000801470aa0 in ssl3_read () from /usr/lib/libssl.so.7
#13 0x00000008009e16b8 in ldap_start_tls_s () from
/usr/local/lib/libldap_r-2.4.so.2
#14 0x0000000800c00126 in ber_int_sb_write () from
/usr/local/lib/liblber-2.4.so.2
#15 0x0000000800bff0f2 in ber_int_sb_read () from
/usr/local/lib/liblber-2.4.so.2
#16 0x0000000800bfc720 in ber_get_next () from
/usr/local/lib/liblber-2.4.so.2
#17 0x0000000000444cf5 in connection_input (conn=0x8034099c0,
cri=0x7fffff3fcc18) at connection.c:1572
#18 0x0000000000444ac5 in connection_read (s=28, cri=0x7fffff3fcc18) at
connection.c:1460
#19 0x0000000000442291 in connection_read_thread (ctx=0x7fffff3fcc80,
argv=0x1c) at connection.c:1284
#20 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from
/usr/local/lib/libldap_r-2.4.so.2
#21 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
#22 0x0000000000000000 in ?? ()
Thread 17 (Thread 943c11c00 (LWP 100268/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3
#2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7
#3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7
#4 0x0000000800bfcc66 in ber_error_print () from
/usr/local/lib/liblber-2.4.so.2
#5 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2
#6 0x0000000800bfcf4a in ber_log_bprint () from
/usr/local/lib/liblber-2.4.so.2
#7 0x0000000800c0028e in ber_int_sb_write () from
/usr/local/lib/liblber-2.4.so.2
#8 0x00000008009e1895 in ldap_start_tls_s () from
/usr/local/lib/libldap_r-2.4.so.2
#9 0x0000000801807f6a in BIO_write () from /lib/libcrypto.so.7
#10 0x00000008017d36cc in BIO_f_buffer () from /lib/libcrypto.so.7
#11 0x00000008014659e7 in ssl3_accept () from /usr/lib/libssl.so.7
#12 0x0000000801470dd7 in ssl23_accept () from /usr/lib/libssl.so.7
#13 0x00000008009e0cd9 in ldap_start_tls_s () from
/usr/local/lib/libldap_r-2.4.so.2
#14 0x00000008009dee4b in ldap_pvt_tls_accept () from
/usr/local/lib/libldap_r-2.4.so.2
#15 0x0000000000444703 in connection_read (s=35, cri=0x7ffffe3fac18) at
connection.c:1372
#16 0x0000000000442291 in connection_read_thread (ctx=0x7ffffe3fac80,
argv=0x23) at connection.c:1284
#17 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from
/usr/local/lib/libldap_r-2.4.so.2
#18 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
#19 0x0000000000000000 in ?? ()
Thread 16 (Thread 943c12000 (LWP 100296/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3
#2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7
#3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7
#4 0x0000000800bfcc66 in ber_error_print () from
/usr/local/lib/liblber-2.4.so.2
#5 0x0000000800bfd2a9 in ber_dump () from /usr/local/lib/liblber-2.4.so.2
#6 0x0000000800bfd1cf in ber_log_dump () from
/usr/local/lib/liblber-2.4.so.2
#7 0x0000000800bf9266 in ber_scanf () from /usr/local/lib/liblber-2.4.so.2
#8 0x0000000000472692 in do_bind (op=0x94601cc00, rs=0x7ffffdbf9b60) at
bind.c:102
#9 0x0000000000443398 in connection_operation (ctx=0x7ffffdbf9c80,
arg_v=0x94601cc00) at connection.c:1155
#10 0x000000000044235b in connection_read_thread (ctx=0x7ffffdbf9c80,
argv=0x1d) at connection.c:1291
#11 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from
/usr/local/lib/libldap_r-2.4.so.2
#12 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
#13 0x0000000000000000 in ?? ()
Thread 15 (Thread 943c12400 (LWP 100311/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3
---Type <return> to continue, or q <return> to quit---
#2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7
#3 0x0000000801dfc087 in fflush () from /lib/libc.so.7
#4 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2
#5 0x0000000800bfcf4a in ber_log_bprint () from
/usr/local/lib/liblber-2.4.so.2
#6 0x0000000800c0028e in ber_int_sb_write () from
/usr/local/lib/liblber-2.4.so.2
#7 0x00000008009e1895 in ldap_start_tls_s () from
/usr/local/lib/libldap_r-2.4.so.2
#8 0x0000000801807f6a in BIO_write () from /lib/libcrypto.so.7
#9 0x00000008017d36cc in BIO_f_buffer () from /lib/libcrypto.so.7
#10 0x00000008014659e7 in ssl3_accept () from /usr/lib/libssl.so.7
#11 0x0000000801470dd7 in ssl23_accept () from /usr/lib/libssl.so.7
#12 0x00000008009e0cd9 in ldap_start_tls_s () from
/usr/local/lib/libldap_r-2.4.so.2
#13 0x00000008009dee4b in ldap_pvt_tls_accept () from
/usr/local/lib/libldap_r-2.4.so.2
#14 0x0000000000444703 in connection_read (s=37, cri=0x7ffffd3f8c18) at
connection.c:1372
#15 0x0000000000442291 in connection_read_thread (ctx=0x7ffffd3f8c80,
argv=0x25) at connection.c:1284
#16 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from
/usr/local/lib/libldap_r-2.4.so.2
#17 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
#18 0x0000000000000000 in ?? ()
Thread 14 (Thread 943c12800 (LWP 100347/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3
#2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7
#3 0x0000000801dfc087 in fflush () from /lib/libc.so.7
#4 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2
#5 0x0000000800bfcf4a in ber_log_bprint () from
/usr/local/lib/liblber-2.4.so.2
#6 0x0000000800c0028e in ber_int_sb_write () from
/usr/local/lib/liblber-2.4.so.2
#7 0x00000008009e1895 in ldap_start_tls_s () from
/usr/local/lib/libldap_r-2.4.so.2
#8 0x0000000801807f6a in BIO_write () from /lib/libcrypto.so.7
#9 0x00000008017d36cc in BIO_f_buffer () from /lib/libcrypto.so.7
#10 0x00000008014659e7 in ssl3_accept () from /usr/lib/libssl.so.7
#11 0x0000000801470dd7 in ssl23_accept () from /usr/lib/libssl.so.7
#12 0x00000008009e0cd9 in ldap_start_tls_s () from
/usr/local/lib/libldap_r-2.4.so.2
#13 0x00000008009dee4b in ldap_pvt_tls_accept () from
/usr/local/lib/libldap_r-2.4.so.2
#14 0x0000000000444703 in connection_read (s=36, cri=0x7ffffcbf7c18) at
connection.c:1372
#15 0x0000000000442291 in connection_read_thread (ctx=0x7ffffcbf7c80,
argv=0x24) at connection.c:1284
#16 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from
/usr/local/lib/libldap_r-2.4.so.2
#17 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
#18 0x0000000000000000 in ?? ()
Thread 13 (Thread 943c12c00 (LWP 100515/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3
#2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7
#3 0x0000000801dfc087 in fflush () from /lib/libc.so.7
#4 0x0000000800bfcee0 in ber_pvt_log_printf () from
/usr/local/lib/liblber-2.4.so.2
#5 0x0000000800bfbe78 in ber_flush2 () from /usr/local/lib/liblber-2.4.so.2
#6 0x0000000000460d1a in send_ldap_ber (op=0x946c6ac00,
ber=0x7ffffc266168) at result.c:339
#7 0x000000000045c9e6 in send_ldap_response (op=0x946c6ac00,
rs=0x7ffffc3f6bd0) at result.c:705
#8 0x000000000045d25c in slap_send_ldap_result (op=0x946c6ac00,
rs=0x7ffffc3f6bd0) at result.c:861
#9 0x0000000802c1e61b in mdb_search (op=0x946c6ac00, rs=0x7ffffc3f6bd0)
at search.c:1164
#10 0x00000000004473ad in fe_op_search (op=0x946c6ac00,
rs=0x7ffffc3f6bd0) at search.c:402
#11 0x0000000000446bc2 in do_search (op=0x946c6ac00, rs=0x7ffffc3f6bd0)
at search.c:247
#12 0x0000000000443398 in connection_operation (ctx=0x7ffffc3f6c80,
arg_v=0x946c6ac00) at connection.c:1155
#13 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from
/usr/local/lib/libldap_r-2.4.so.2
#14 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
#15 0x0000000000000000 in ?? ()
Thread 12 (Thread 943c13000 (LWP 100531/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3
#2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7
#3 0x0000000801dfc087 in fflush () from /lib/libc.so.7
#4 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2
#5 0x0000000800bfd2b5 in ber_dump () from /usr/local/lib/liblber-2.4.so.2
#6 0x0000000800bfd1cf in ber_log_dump () from
/usr/local/lib/liblber-2.4.so.2
#7 0x0000000800bfca96 in ber_get_next () from
/usr/local/lib/liblber-2.4.so.2
#8 0x0000000000444cf5 in connection_input (conn=0x8034097b0,
cri=0x7ffffbbf5c18) at connection.c:1572
#9 0x0000000000444ac5 in connection_read (s=27, cri=0x7ffffbbf5c18) at
connection.c:1460
#10 0x0000000000442291 in connection_read_thread (ctx=0x7ffffbbf5c80,
argv=0x1b) at connection.c:1284
#11 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from
/usr/local/lib/libldap_r-2.4.so.2
#12 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
#13 0x0000000000000000 in ?? ()
Thread 11 (Thread 943c13400 (LWP 100577/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3
#2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7
---Type <return> to continue, or q <return> to quit---
#3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7
#4 0x0000000800bfcc66 in ber_error_print () from
/usr/local/lib/liblber-2.4.so.2
#5 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2
#6 0x0000000800bfcf4a in ber_log_bprint () from
/usr/local/lib/liblber-2.4.so.2
#7 0x0000000800c0028e in ber_int_sb_write () from
/usr/local/lib/liblber-2.4.so.2
#8 0x00000008009e1895 in ldap_start_tls_s () from
/usr/local/lib/libldap_r-2.4.so.2
#9 0x0000000801807f6a in BIO_write () from /lib/libcrypto.so.7
#10 0x00000008017d36cc in BIO_f_buffer () from /lib/libcrypto.so.7
#11 0x00000008014659e7 in ssl3_accept () from /usr/lib/libssl.so.7
#12 0x0000000801470dd7 in ssl23_accept () from /usr/lib/libssl.so.7
#13 0x00000008009e0cd9 in ldap_start_tls_s () from
/usr/local/lib/libldap_r-2.4.so.2
#14 0x00000008009dee4b in ldap_pvt_tls_accept () from
/usr/local/lib/libldap_r-2.4.so.2
#15 0x0000000000444703 in connection_read (s=38, cri=0x7ffffb3f4c18) at
connection.c:1372
#16 0x0000000000442291 in connection_read_thread (ctx=0x7ffffb3f4c80,
argv=0x26) at connection.c:1284
#17 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from
/usr/local/lib/libldap_r-2.4.so.2
#18 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
#19 0x0000000000000000 in ?? ()
Thread 10 (Thread 943c13800 (LWP 100619/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3
#2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7
#3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7
#4 0x0000000800bfd48f in lutil_debug () from
/usr/local/lib/liblber-2.4.so.2
#5 0x000000000045c3a0 in send_ldap_response (op=0x947811400,
rs=0x7ffffabf3b60) at result.c:609
#6 0x000000000045d25c in slap_send_ldap_result (op=0x947811400,
rs=0x7ffffabf3b60) at result.c:861
#7 0x0000000802c1e61b in mdb_search (op=0x947811400, rs=0x7ffffabf3b60)
at search.c:1164
#8 0x00000000004473ad in fe_op_search (op=0x947811400,
rs=0x7ffffabf3b60) at search.c:402
#9 0x0000000000446bc2 in do_search (op=0x947811400, rs=0x7ffffabf3b60)
at search.c:247
#10 0x0000000000443398 in connection_operation (ctx=0x7ffffabf3c80,
arg_v=0x947811400) at connection.c:1155
#11 0x000000000044235b in connection_read_thread (ctx=0x7ffffabf3c80,
argv=0x17) at connection.c:1291
#12 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from
/usr/local/lib/libldap_r-2.4.so.2
#13 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
#14 0x0000000000000000 in ?? ()
Thread 9 (Thread 943c13c00 (LWP 100647/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3
#2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7
#3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7
#4 0x0000000800bfcc66 in ber_error_print () from
/usr/local/lib/liblber-2.4.so.2
#5 0x0000000800bfd158 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2
#6 0x0000000800bfd2b5 in ber_dump () from /usr/local/lib/liblber-2.4.so.2
#7 0x0000000800bfd1cf in ber_log_dump () from
/usr/local/lib/liblber-2.4.so.2
#8 0x0000000800bfca96 in ber_get_next () from
/usr/local/lib/liblber-2.4.so.2
#9 0x0000000000444cf5 in connection_input (conn=0x80340a200,
cri=0x7ffffa3f2c18) at connection.c:1572
#10 0x0000000000444ac5 in connection_read (s=32, cri=0x7ffffa3f2c18) at
connection.c:1460
#11 0x0000000000442291 in connection_read_thread (ctx=0x7ffffa3f2c80,
argv=0x20) at connection.c:1284
#12 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from
/usr/local/lib/libldap_r-2.4.so.2
#13 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
#14 0x0000000000000000 in ?? ()
Thread 8 (Thread 943c14000 (LWP 100884/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3
#2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7
#3 0x0000000801dfc087 in fflush () from /lib/libc.so.7
#4 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2
#5 0x0000000800bfd2b5 in ber_dump () from /usr/local/lib/liblber-2.4.so.2
#6 0x0000000800bfd1cf in ber_log_dump () from
/usr/local/lib/liblber-2.4.so.2
#7 0x0000000800bf9266 in ber_scanf () from /usr/local/lib/liblber-2.4.so.2
#8 0x0000000000445bc1 in do_search (op=0x94801c400, rs=0x7ffff9bf1b60)
at search.c:72
#9 0x0000000000443398 in connection_operation (ctx=0x7ffff9bf1c80,
arg_v=0x94801c400) at connection.c:1155
#10 0x000000000044235b in connection_read_thread (ctx=0x7ffff9bf1c80,
argv=0x19) at connection.c:1291
#11 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from
/usr/local/lib/libldap_r-2.4.so.2
#12 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
#13 0x0000000000000000 in ?? ()
Thread 7 (Thread 943c14400 (LWP 100898/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3
#2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7
#3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7
#4 0x0000000800bfcc66 in ber_error_print () from
/usr/local/lib/liblber-2.4.so.2
#5 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2
#6 0x0000000800bfcf4a in ber_log_bprint () from
/usr/local/lib/liblber-2.4.so.2
#7 0x0000000800c0017e in ber_int_sb_write () from
/usr/local/lib/liblber-2.4.so.2
---Type <return> to continue, or q <return> to quit---
#8 0x0000000800bff0f2 in ber_int_sb_read () from
/usr/local/lib/liblber-2.4.so.2
#9 0x0000000800bfca1a in ber_get_next () from
/usr/local/lib/liblber-2.4.so.2
#10 0x0000000000444cf5 in connection_input (conn=0x8034095a0,
cri=0x7ffff93f0c18) at connection.c:1572
#11 0x0000000000444ac5 in connection_read (s=26, cri=0x7ffff93f0c18) at
connection.c:1460
#12 0x0000000000442291 in connection_read_thread (ctx=0x7ffff93f0c80,
argv=0x1a) at connection.c:1284
#13 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from
/usr/local/lib/libldap_r-2.4.so.2
#14 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
#15 0x0000000000000000 in ?? ()
Thread 6 (Thread 943c14800 (LWP 101202/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3
#2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7
#3 0x0000000801dfc087 in fflush () from /lib/libc.so.7
#4 0x0000000800bfcee0 in ber_pvt_log_printf () from
/usr/local/lib/liblber-2.4.so.2
#5 0x0000000800c00167 in ber_int_sb_write () from
/usr/local/lib/liblber-2.4.so.2
#6 0x0000000800bff0f2 in ber_int_sb_read () from
/usr/local/lib/liblber-2.4.so.2
#7 0x0000000800bfca1a in ber_get_next () from
/usr/local/lib/liblber-2.4.so.2
#8 0x0000000000444cf5 in connection_input (conn=0x803408940,
cri=0x7ffff8befc18) at connection.c:1572
#9 0x0000000000444ac5 in connection_read (s=20, cri=0x7ffff8befc18) at
connection.c:1460
#10 0x0000000000442291 in connection_read_thread (ctx=0x7ffff8befc80,
argv=0x14) at connection.c:1284
#11 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from
/usr/local/lib/libldap_r-2.4.so.2
#12 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
#13 0x0000000000000000 in ?? ()
Thread 5 (Thread 943c14c00 (LWP 101213/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3
#2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7
#3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7
#4 0x0000000800bfcc66 in ber_error_print () from
/usr/local/lib/liblber-2.4.so.2
#5 0x0000000800bfd002 in ber_bprint () from /usr/local/lib/liblber-2.4.so.2
#6 0x0000000800bfd2b5 in ber_dump () from /usr/local/lib/liblber-2.4.so.2
#7 0x0000000800bfd1cf in ber_log_dump () from
/usr/local/lib/liblber-2.4.so.2
#8 0x0000000800bf9266 in ber_scanf () from /usr/local/lib/liblber-2.4.so.2
#9 0x0000000000445bc1 in do_search (op=0x948c6ac00, rs=0x7ffff83eeb60)
at search.c:72
#10 0x0000000000443398 in connection_operation (ctx=0x7ffff83eec80,
arg_v=0x948c6ac00) at connection.c:1155
#11 0x000000000044235b in connection_read_thread (ctx=0x7ffff83eec80,
argv=0x1e) at connection.c:1291
#12 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from
/usr/local/lib/libldap_r-2.4.so.2
#13 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
#14 0x0000000000000000 in ?? ()
Thread 4 (Thread 943c15000 (LWP 101255/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x0000000801dfc0a9 in fflush () from /lib/libc.so.7
#2 0x0000000800bfcee0 in ber_pvt_log_printf () from
/usr/local/lib/liblber-2.4.so.2
#3 0x0000000800c00167 in ber_int_sb_write () from
/usr/local/lib/liblber-2.4.so.2
#4 0x0000000800bff0f2 in ber_int_sb_read () from
/usr/local/lib/liblber-2.4.so.2
#5 0x0000000800bfc720 in ber_get_next () from
/usr/local/lib/liblber-2.4.so.2
#6 0x0000000000444cf5 in connection_input (conn=0x803408b50,
cri=0x7ffff7bedc18) at connection.c:1572
#7 0x0000000000444ac5 in connection_read (s=21, cri=0x7ffff7bedc18) at
connection.c:1460
#8 0x0000000000442291 in connection_read_thread (ctx=0x7ffff7bedc80,
argv=0x15) at connection.c:1284
#9 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from
/usr/local/lib/libldap_r-2.4.so.2
#10 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
#11 0x0000000000000000 in ?? ()
Thread 3 (Thread 943c11800 (LWP 100198/slapd)):
#0 mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404
#1 0x00000000004473ad in fe_op_search (op=0x94581c400,
rs=0x7ffffebfbb60) at search.c:402
#2 0x0000000000446bc2 in do_search (op=0x94581c400, rs=0x7ffffebfbb60)
at search.c:247
#3 0x0000000000443398 in connection_operation (ctx=0x7ffffebfbc80,
arg_v=0x94581c400) at connection.c:1155
#4 0x000000000044235b in connection_read_thread (ctx=0x7ffffebfbc80,
argv=0x18) at connection.c:1291
#5 0x00000008009b7f9a in ldap_pvt_thread_pool_submit () from
/usr/local/lib/libldap_r-2.4.so.2
#6 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
#7 0x0000000000000000 in ?? ()
Thread 2 (Thread 802806400 (LWP 100636/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x000000000043a770 in slapd_daemon () at daemon.c:2929
#2 0x0000000000416b5a in main (argc=11, argv=0x7fffffffe568) at main.c:1012
404 in search.c
(gdb) generate-core-file
warning: Memory read failed for corefile section, 5368709120 bytes at
0x0000000803c00000
Killed
root@FreeBSD [~] $ ls -lach ./
-rw-r--r-- 1 root wheel 3.6G 27 Apr 22:01 core.38120
# ================================================== #
Seems like something went wrong here. Am I using gdb wrong?
Am 27.04.15 um 19:04 schrieb Michael Ströder:
Leander Schäfer wrote:
> Can you please provide me a link, cause I wasn't able to find
> "current RE24" on the official website nor on the FTP mirror.
Use git or this link to checkout snapshot of the RE24 branch:
http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=snapshot;h=refs...
Ciao, Michael.
5:32 p.m.
Leander Schäfer wrote:
Ok, here is the first result running the debugging mode with gdb(1)
>> Procedure overview:
(gdb) run
(gdb) bt full
(gdb) thread apply all bt
(gdb) generate-core-file
No need for a core file if you're just running slapd inside gdb.
>> This came up:
candidates = Error accessing memory address 0x7ffffeafb6f0: Bad address.
# ================================================== #
root@FreeBSD [~]$ gdb --args /usr/local/libexec/slapd -d -1 -f
/usr/local/etc/openldap/slapd.conf -u ldap -g ldap -h
"ldapi://%2fvar%2frun%2fopenldap%2fldapi/ ldap:/// ldaps:///"
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
(gdb) run
Starting program: /usr/local/libexec/slapd -d -1 -f
/usr/local/etc/openldap/slapd.conf -u ldap -g ldap -h
ldapi://%2fvar%2frun%2fopenldap%2fldapi/\ ldap:///\ ldaps:///
[New LWP 101138]
[New Thread 802806400 (LWP 101138/slapd)]
[...]
553e8a87 conn=1006 op=2 SRCH attr=mailAlias
553e8a87 send_ldap_result: err=0 matched="" text=""
0010: 51 bd aa 7d 3f 1c 50 fb 25 f8 59 9e 9d 9a ba 0f Q..}?.P.%.Y.....
0020: d0 07 aa 95 ac 1c e7 3e 81 f6 e6 0b 6d 09 94 9b .......>....m...
0730: 1b 51 e3 08 4b 38 ec f1 ee 8c 0f 35 cd 55 eb 80 .Q..K8.....5.U..
553e8a87 ==> limits_get: conn=1006 op=2 self="[anonymous]"
this="ou=accounts,ou=mail,dc=mydomain,dc=local"
0740: 83 e2 3b b5 13 fd 08 51 13 25 d9 7d 57 9f 6b e9 ..;....Q.%.}W.k.
[New Thread 943c11800 (LWP 100198/slapd)]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 943c11800 (LWP 100198/slapd)]
mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404
404 search.c: No such file or directory.
in search.c
Current language: auto; currently minimal
(gdb) bt full
#0 mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404
mdb = (struct mdb_info *) 0x80290a000
id = 0
cursor = 0
nsubs = 128
ncand = 0
cscope = 0
lastid = 18446744073709551615
candidates = Error accessing memory address 0x7ffffeafb6f0: Bad
address.
(gdb) thread apply all bt
[New Thread 943c15000 (LWP 101255/slapd)]
[New Thread 943c14c00 (LWP 101213/slapd)]
[New Thread 943c14800 (LWP 101202/slapd)]
[New Thread 943c14400 (LWP 100898/slapd)]
[New Thread 943c14000 (LWP 100884/slapd)]
[New Thread 943c13c00 (LWP 100647/slapd)]
[New Thread 943c13800 (LWP 100619/slapd)]
[New Thread 943c13400 (LWP 100577/slapd)]
[New Thread 943c13000 (LWP 100531/slapd)]
[New Thread 943c12c00 (LWP 100515/slapd)]
[New Thread 943c12800 (LWP 100347/slapd)]
[New Thread 943c12400 (LWP 100311/slapd)]
[New Thread 943c12000 (LWP 100296/slapd)]
[New Thread 943c11c00 (LWP 100268/slapd)]
[New Thread 943c11400 (LWP 100165/slapd)]
[New Thread 802807800 (LWP 100103/slapd)]
Thread 19 (Thread 802807800 (LWP 100103/slapd)):
#0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
#1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3
#2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7
#3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7
#4 0x0000000800bfd48f in lutil_debug () from
/usr/local/lib/liblber-2.4.so.2
#5 0x000000000043b96f in slapd_daemon_task (ptr=0x8028afb08) at
daemon.c:2530
#6 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
Seems like something went wrong here. Am I using gdb wrong?
Looks like your liblber was installed without debug symbols. Most of
these stack traces look invalid.
Am 27.04.15 um 19:04 schrieb Michael Ströder:
> Leander Schäfer wrote:
>> Can you please provide me a link, cause I wasn't able to find
>> "current RE24" on the official website nor on the FTP mirror.
>
> Use git or this link to checkout snapshot of the RE24 branch:
>
>
http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=snapshot;h=refs...
Assuming you compiled the latest snapshot, the SEGV at
back-mdb/search.c:404 makes not much sense, it's a return statement.
Also, as back-mdb didn't exist 5 years ago, this cannot be the same
issue you've been running into all the time.
Perhaps you've hit a stack overrun. Generally slapd uses 8MB stacks on
64bit machines. It seems from your ulimit output that 8MB should be
fine, so that also seems unlikely.
What was the full LDAP search request that was running at the moment of
the crash? Mainly interested in seeing the search filter, and how
complex it was, as well as the depth of the DIT.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
5:42 p.m.
Howard Chu wrote:
Assuming you compiled the latest snapshot, the SEGV at
back-mdb/search.c:404 makes not much sense, it's a return statement.
Also, as back-mdb didn't exist 5 years ago, this cannot be the same
issue you've been running into all the time.
Perhaps you've hit a stack overrun. Generally slapd uses 8MB stacks on
64bit machines. It seems from your ulimit output that 8MB should be
fine, so that also seems unlikely.
Ah, yes, this is a known issue with FreeBSD.
http://www.openldap.org/lists/openldap-bugs/200506/msg00174.html
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
5:49 p.m.
Howard Chu wrote:
Howard Chu wrote:
> Assuming you compiled the latest snapshot, the SEGV at
> back-mdb/search.c:404 makes not much sense, it's a return statement.
>
> Also, as back-mdb didn't exist 5 years ago, this cannot be the same
> issue you've been running into all the time.
>
> Perhaps you've hit a stack overrun. Generally slapd uses 8MB stacks on
> 64bit machines. It seems from your ulimit output that 8MB should be
> fine, so that also seems unlikely.
Ah, yes, this is a known issue with FreeBSD.
http://www.openldap.org/lists/openldap-bugs/200506/msg00174.html
Furthermore, in the intervening 10 years, the FreeBSD developers have
not yet fixed this issue.
http://lists.freebsd.org/pipermail/freebsd-current/2014-August/051646.html
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
5:02 a.m.
Also, I just checked. It looks like slapd is successfully linked to
"libthr.so.3"
root@FreeBSD [~]$ ldd /usr/local/libexec/slapd
/usr/local/libexec/slapd:
libldap_r-2.4.so.2 => /usr/local/lib/libldap_r-2.4.so.2
(0x8009a7000)
liblber-2.4.so.2 => /usr/local/lib/liblber-2.4.so.2 (0x800bf5000)
libltdl.so.7 => /usr/local/lib/libltdl.so.7 (0x800e03000)
libcrypt.so.5 => /lib/libcrypt.so.5 (0x80100c000)
libwrap.so.6 => /usr/lib/libwrap.so.6 (0x80122c000)
libssl.so.7 => /usr/lib/libssl.so.7 (0x801435000)
libcrypto.so.7 => /lib/libcrypto.so.7 (0x8016a0000)
libthr.so.3 => /lib/libthr.so.3 (0x801a93000)
libc.so.7 => /lib/libc.so.7 (0x801cb8000)
root@FreeBSD [~]$ ls -lach /lib/libthr.so.3
-r--r--r-- 1 root wheel 103K 18 Jan 15:36 /lib/libthr.so.3
So as all the other Applications using "libthr.so.3" slapd should also
be able to use it?! I'm not 100% convinced yet, that this diagnosis is
going into the right direction here. I think I'll provide a second gdb
output, since it always seems to break down at a different point and action.
Am 28.04.15 um 02:49 schrieb Howard Chu:
Howard Chu wrote:
> Howard Chu wrote:
>> Assuming you compiled the latest snapshot, the SEGV at
>> back-mdb/search.c:404 makes not much sense, it's a return statement.
>>
>> Also, as back-mdb didn't exist 5 years ago, this cannot be the same
>> issue you've been running into all the time.
>>
>> Perhaps you've hit a stack overrun. Generally slapd uses 8MB stacks on
>> 64bit machines. It seems from your ulimit output that 8MB should be
>> fine, so that also seems unlikely.
>
> Ah, yes, this is a known issue with FreeBSD.
>
> http://www.openldap.org/lists/openldap-bugs/200506/msg00174.html
Furthermore, in the intervening 10 years, the FreeBSD developers have
not yet fixed this issue.
http://lists.freebsd.org/pipermail/freebsd-current/2014-August/051646.html
4:14 a.m.
Am 28.04.15 um 02:32 schrieb Howard Chu:
Leander Schäfer wrote:
> Ok, here is the first result running the debugging mode with gdb(1)
>
> >> Procedure overview:
> (gdb) run
> (gdb) bt full
> (gdb) thread apply all bt
> (gdb) generate-core-file
No need for a core file if you're just running slapd inside gdb.
I thought the
core file would be required to have a better base of
knowledge about the internal happening?! But ofcourse I'm happy if it is
not required, cause it doesn't seem to create it properly in the first
place.
>
>
> >> This came up:
> candidates = Error accessing memory address 0x7ffffeafb6f0: Bad address.
>
> # ================================================== #
>
> root@FreeBSD [~]$ gdb --args /usr/local/libexec/slapd -d -1 -f
> /usr/local/etc/openldap/slapd.conf -u ldap -g ldap -h
> "ldapi://%2fvar%2frun%2fopenldap%2fldapi/ ldap:/// ldaps:///"
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you
> are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for
> details.
> This GDB was configured as "amd64-marcel-freebsd"...
> (gdb) run
> Starting program: /usr/local/libexec/slapd -d -1 -f
> /usr/local/etc/openldap/slapd.conf -u ldap -g ldap -h
> ldapi://%2fvar%2frun%2fopenldap%2fldapi/\ ldap:///\ ldaps:///
> [New LWP 101138]
> [New Thread 802806400 (LWP 101138/slapd)]
>
> [...]
>
> 553e8a87 conn=1006 op=2 SRCH attr=mailAlias
> 553e8a87 send_ldap_result: err=0 matched="" text=""
> 0010: 51 bd aa 7d 3f 1c 50 fb 25 f8 59 9e 9d 9a ba 0f
> Q..}?.P.%.Y.....
> 0020: d0 07 aa 95 ac 1c e7 3e 81 f6 e6 0b 6d 09 94 9b
> .......>....m...
> 0730: 1b 51 e3 08 4b 38 ec f1 ee 8c 0f 35 cd 55 eb 80
> .Q..K8.....5.U..
> 553e8a87 ==> limits_get: conn=1006 op=2 self="[anonymous]"
> this="ou=accounts,ou=mail,dc=mydomain,dc=local"
> 0740: 83 e2 3b b5 13 fd 08 51 13 25 d9 7d 57 9f 6b e9
> ..;....Q.%.}W.k.
> [New Thread 943c11800 (LWP 100198/slapd)]
>
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 943c11800 (LWP 100198/slapd)]
> mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404
> 404 search.c: No such file or directory.
> in search.c
> Current language: auto; currently minimal
> (gdb) bt full
> #0 mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404
> mdb = (struct mdb_info *) 0x80290a000
> id = 0
> cursor = 0
> nsubs = 128
> ncand = 0
> cscope = 0
> lastid = 18446744073709551615
> candidates = Error accessing memory address 0x7ffffeafb6f0: Bad
> address.
> (gdb) thread apply all bt
> [New Thread 943c15000 (LWP 101255/slapd)]
> [New Thread 943c14c00 (LWP 101213/slapd)]
> [New Thread 943c14800 (LWP 101202/slapd)]
> [New Thread 943c14400 (LWP 100898/slapd)]
> [New Thread 943c14000 (LWP 100884/slapd)]
> [New Thread 943c13c00 (LWP 100647/slapd)]
> [New Thread 943c13800 (LWP 100619/slapd)]
> [New Thread 943c13400 (LWP 100577/slapd)]
> [New Thread 943c13000 (LWP 100531/slapd)]
> [New Thread 943c12c00 (LWP 100515/slapd)]
> [New Thread 943c12800 (LWP 100347/slapd)]
> [New Thread 943c12400 (LWP 100311/slapd)]
> [New Thread 943c12000 (LWP 100296/slapd)]
> [New Thread 943c11c00 (LWP 100268/slapd)]
> [New Thread 943c11400 (LWP 100165/slapd)]
> [New Thread 802807800 (LWP 100103/slapd)]
>
> Thread 19 (Thread 802807800 (LWP 100103/slapd)):
> #0 0x0000000801aa78cc in __error () from /lib/libthr.so.3
> #1 0x0000000801aa27f4 in pthread_mutex_destroy () from /lib/libthr.so.3
> #2 0x0000000801dfc237 in flockfile () from /lib/libc.so.7
> #3 0x0000000801dd7e64 in fputs () from /lib/libc.so.7
> #4 0x0000000800bfd48f in lutil_debug () from
> /usr/local/lib/liblber-2.4.so.2
> #5 0x000000000043b96f in slapd_daemon_task (ptr=0x8028afb08) at
> daemon.c:2530
> #6 0x0000000801a9c4f5 in pthread_create () from /lib/libthr.so.3
> Seems like something went wrong here. Am I using gdb wrong?
Looks like your liblber was installed without debug symbols. Most of
these stack traces look invalid.
What does this mean? How did you see this / what
indicated this to you?
Is it required to fix this liblber issue for a better debug result, or
is it ok for first diagnosis?
> Am 27.04.15 um 19:04 schrieb Michael Ströder:
>> Leander Schäfer wrote:
>>> Can you please provide me a link, cause I wasn't able to find
>>> "current RE24" on the official website nor on the FTP mirror.
>>
>> Use git or this link to checkout snapshot of the RE24 branch:
>>
>>
http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=snapshot;h=refs...
>>
Assuming you compiled the latest snapshot, the SEGV at
back-mdb/search.c:404 makes not much sense, it's a return statement.
Also, as back-mdb didn't exist 5 years ago, this cannot be the same
issue you've been running into all the time.
^ Pretty much possible. But again
- the exit points are always
different. I could do the same thing with BDB and yet it would exit on
different points.
Perhaps you've hit a stack overrun. Generally slapd uses 8MB stacks on
64bit machines. It seems from your ulimit output that 8MB should be
fine, so that also seems unlikely.
Since this exceeds my current knowledge, I
decided to asked the
developement experts on FreeBSD Forums:
https://forums.freebsd.org/threads/thread-stack-size-segmentation-fault.5...
What was the full LDAP search request that was running at the moment
of the crash? Mainly interested in seeing the search filter, and how
complex it was, as well as the depth of the DIT.
The search filter looks quite
simple to me:
==> 553e8a87 conn=1006 op=2 SRCH
base="ou=accounts,ou=mail,dc=MyDomain,dc=Local" scope=2 deref=0
filter="(mailAddress=root(a)wm-01.mydomain.local)"
Here is a bit more of the debug output.
[...]
553e8a87 daemon: waked
0670: 63 04 aa 27 74 94 4c 86 1d d7 3f a1 2f af 22 0f c..'t.L...?./.".
553e8a87 mailAlias553e8a87
0680: 41 88 ed e0 c6 d0 4e 92 ed 2f b1 40 51 ae a8 77 A.....N../.@Q..w
ber_get_next: tag 0x30 len 12 contents:
0690: fe b9 b7 18 ae 33 df a4 27 50 0e 3d 8d 17 31 ee .....3..'P.=..1.
06a0: 21 e9 ed c7 a2 90 fa 6f 3e b3 87 ff d8 6b d4 e8 !......o>....k..
0030: c8 57 d6 0e 59 3c 7b ef b9 db fe 64 f9 4d 02 10 .W..Y<{....d.M..
0040: af 62 1a 21 b3 68 08 50 31 1e 6b 09 8f da 88 84 .b.!.h.P1.k.....
553e8a87 conn=1006 op=2 SRCH
base="ou=accounts,ou=mail,dc=MyDomain,dc=Local" scope=2 deref=0
filter="(mailAddress=root(a)wm-01.mydomain.local)"
0010: 6d 61 69 6c 2c 64 63 3d 4e 65 74 4f 63 65 61 6e mail,dc=Mydomain
0020: 2c 64 63 3d 4c 6f 63 61 6c 0a 01 02 0a 01 00 02 ,dc=Local.......
0030: 01 00 02 01 0a 01 01 00 a3 29 04 0b 6d 61 69 6c .........)..mail
0040: 41 64 64 72 65 73 73 04 1a 72 6f 6f 74 40 77 6d Address..root@wm
ldap_read: want=8 error=Resource temporarily unavailable
ber_dump: buf=0x947c19070 ptr=0x947c19070 end=0x947c1907c len=12
553e8a87 conn=1011 op=1 do_bind
ber_scanf fmt ({imt) ber:
553e8a87 <= mdb_list_candidates: id=0 first=0 last=0
553e8a87 daemon: select: listen=10 active_threads=0 tvp=NULL
0440: 2d 30 31 2e 4e 65 74 4f 63 65 61 6e 2e 4c 6f 63 -01.Mydomain.Loc
0450: 61 6c 2f 50 4b 49 2f 43 41 2f 53 69 67 6e 69 6e al/PKI/CA/Signin
0000: aa 58 54 9e 30 18 bb 75 df 8e 62 e4 62 b0 5e 39 .XT.0..u..b.b.^9
553e8a87 send_ldap_response: msgid=3 tag=101 err=0
ber_flush2: 14 bytes to sd 19
TLS trace: SSL_accept:SSLv3 write finished A
06b0: aa 09 24 e0 e3 6a c8 f2 51 06 ca 80 0c 3c c3 f8 ..$..j..Q....<..
06c0: 32 df 78 c7 1d 90 62 d0 a3 34 cd 63 18 fa 96 1f 2.x...b..4.c....
tls_write: want=226, written=226
06d0: 4d 3e 23 5a 7f 8f c0 59 13 58 e7 c0 ad 15 a0 53 M>#Z...Y.X.....S
06e0: 7d 48 35 a8 6c 5f d8 f2 b3 cc e2 dd 0e b0 54 02 }H5.l_........T.
553e8a87 <= mdb_filter_candidates: id=0 first=0 last=0
553e8a87 mdb_search_candidates: id=0 first=0 last=0
553e8a87 mdb_search: no candidates
0050: 35 53 8d a9 94 ee bc 81 ab 9e ca af 51 dc 86 18 5S..........Q...
0060: f2 83 8b 13 23 1c b3 18 6e 9e 90 ce 07 7b 31 c1 ....#...n....{1.
0070: 95 f3 7c 1e 85 14 4b 54 4b 52 8b ..|...KTKR.
ldap_read: want=91, got=91
ldap_read: want=8, got=8
ber_dump: buf=0x947171180 ptr=0x947171180 end=0x9471711f6 len=118
0000: 02 01 03 63 71 04 28 6f 75 3d 61 63 63 6f 75 6e ...cq.(ou=accoun
0000: 16 03 03 00 aa 04 00 00 a6 00 00 01 2c 00 a0 2b ............,..+
ber_dump: buf=0x948179180 ptr=0x948179183 end=0x9481791f6 len=115
0000: 63 71 04 28 6f 75 3d 61 63 63 6f 75 6e 74 73 2c cq.(ou=accounts,
0010: 6f 75 3d 6d 61 69 6c 2c 64 63 3d 4e 65 74 4f 63 ou=mail,dc=NetOc
553e8a87 send_ldap_result: conn=1005 op=2 p=3
553e8a87 conn=1006 op=2 SRCH attr=mailAlias
553e8a87 send_ldap_result: err=0 matched="" text=""
0010: 51 bd aa 7d 3f 1c 50 fb 25 f8 59 9e 9d 9a ba 0f Q..}?.P.%.Y.....
0020: d0 07 aa 95 ac 1c e7 3e 81 f6 e6 0b 6d 09 94 9b .......>....m...
0730: 1b 51 e3 08 4b 38 ec f1 ee 8c 0f 35 cd 55 eb 80 .Q..K8.....5.U..
553e8a87 ==> limits_get: conn=1006 op=2 self="[anonymous]"
this="ou=accounts,ou=mail,dc=mydomain,dc=local"
0740: 83 e2 3b b5 13 fd 08 51 13 25 d9 7d 57 9f 6b e9 ..;....Q.%.}W.k.
[New Thread 943c11800 (LWP 100198/slapd)]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 943c11800 (LWP 100198/slapd)]
mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404
404 search.c: No such file or directory.
in search.c
Current language: auto; currently minimal
(gdb) bt full
#0 mdb_search (op=0x94581c400, rs=0x7ffffebfbb60) at search.c:404
mdb = (struct mdb_info *) 0x80290a000
id = 0
cursor = 0
nsubs = 128
ncand = 0
cscope = 0
lastid = 18446744073709551615
candidates = Error accessing memory address 0x7ffffeafb6f0: Bad
address.
(gdb)
11:43 a.m.
--On Tuesday, April 28, 2015 2:14 PM +0200 Leander Schäfer
<info(a)netocean.de> wrote:
> Looks like your liblber was installed without debug symbols. Most
of
> these stack traces look invalid.
What does this mean? How did you see this / what indicated this to you?
Is it required to fix this liblber issue for a better debug result, or is
it ok for first diagnosis?
It means either or both of:
You didn't compile with CFLAGS="-g" (debugging) and/or when you did
install, you didn't do:
make install STRIP=""
so that the debugging symbols don't get stripped out.
I generally recommend compiling slapd with:
CFLAGS="-g -O0"
I.e., debugging symbols, and no attempts at optimizing the code. Generally
my experience with various compilers has shown that at least in the
OpenLDAP Case, higher optimization levels are not beneficial and simply
make debugging significantly harder.
--Quanah
--
Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
2:09 a.m.
Here is another (3rd) output of 'slapd -d -1 [...]' debugging. It looks
a little more different than the previous two.
# ======================================================================== #
[...]
553dfbce => acl_get: [1] attr entry
ldap_write: want=14, written=14
553dfbce => acl_mask: access to entry
"ou=domains,ou=mail,dc=mydomain,dc=local", attr "entry" requested
553dfbce daemon: activity on 2 descriptors
553dfbce daemon: waked
553dfbce connection_get(25): got connid=1011
0040: 3f 8e 7f 94 2d 99 3e 60 41 93 73 b3 0e de d3 96 ?...-.>`A.s.....
0050: 3f 93 74 5c 06 a4 c3 18 21 ec dd bd 87 5e 84 ed ?.t\....!....^..
0560: 4c 6f 63 61 6c 87 04 c0 a8 32 65 87 04 7f 00 00 Local....2e.....
0060: a3 a1 03 5a cb 52 1c 75 db e9 bb ab 0d 5e 2d 97 ...Z.R.u.....^-.
0070: 93 0e 73 71 62 20 93 ef 76 f0 b8 6e 44 73 1d f4 ..sqb
..v..nDs..
553dfbce ==> limits_get: conn=1006 op=2 self="[anonymous]"
this="ou=accounts,ou=mail,dc=mydomain,dc=local"
0080: c3 49 7f 6e 49 bd e4 e0 7d 70 8b 12 46 39 f1 2b .I.nI...}p..F9.+
0000: 30 0c 02 01 03 65 07 0a 01 00 04 00 04 00 0....e........
ber_dump: buf=0x945c75180 ptr=0x945c75183 end=0x945c751f6 len=115
Segmentation fault
# ======================================================================== #
3143
days inactive
3146
days old
openldap-technical@openldap.org
15 comments
5 participants
participants (5)
-
Howard Chu -
Leander Schäfer -
Michael Ströder -
Quanah Gibson-Mount -
Ulrich Windl