Hi,
Summary:
A tls connection between a client and a 2.3.30 slapd hangs while the server is giving the certificate; but this does not happen if the server is run with -d 2 or higher, or if the client is the server itself.
Details:
(A seemingly similar issue has been reported before, without satisfactory reply, 4 years ago: http://www.openldap.org/lists/openldap-software/200210/msg00459.html)
My slapd is the Debian-etch-packaged 2.3.30.
If I run a ldapsearch -ZZ on the server, it runs fine.
If I try to run the same thing from a client through the network, then the connection hangs (I tried this from three clients). After I hit Enter, the client stays there and it appears to wait forever until interrupted.
If I use -d 1 on both the client and the server, then the client hangs after it has displayed
TLS trace: SSL_connect:before/connect initialization TLS trace: SSL_connect:SSLv2/v3 write client hello A TLS trace: SSL_connect:SSLv3 read server hello A TLS certificate verification: depth: 0, err: 0, subject: /C=GR/L=Athens/O=National Technical University of Athens/OU=ITIA Research Team/CN=www.itia.ntua.gr/emailAddress=sysadmins@itia.ntua.gr, issuer: /C=GR/L=Athens/O=National Technical University of Athens/OU=ITIA Research Team/CN=www.itia.ntua.gr/emailAddress=sysadmins@itia.ntua.gr TLS trace: SSL_connect:SSLv3 read server certificate A
and the server has displayed
TLS trace: SSL_accept:before/accept initialization TLS trace: SSL_accept:SSLv3 read client hello A TLS trace: SSL_accept:SSLv3 write server hello A TLS trace: SSL_accept:SSLv3 write certificate A TLS trace: SSL_accept:SSLv3 write certificate request A TLS trace: SSL_accept:error in SSLv3 flush data TLS trace: SSL_accept:error in SSLv3 flush data
But if I use -d 2 or higher on the server, then the connection succeeds. (Increasing -d on the client does not appear to affect it.)
Could you suggest how to search into it next? Because I'm lost. I thought it might have to do with networking hardware, but ping -f from one client to the server runs fine, with no packet loss, and nfs (which I've noticed to be most sensitive to networking hardware issues) also runs fine. The three clients where through Gigabit, 100 MBits, and 10 MBits. Two of them were running ubuntu-edgy-packaged ldap-utils 2.2.26, one was running debian-etch-packaged 2.3.30. All had identical behaviour.
--On Monday, February 26, 2007 3:59 PM +0200 Antonis Christofides anthony@itia.ntua.gr wrote:
Hi,
Summary:
A tls connection between a client and a 2.3.30 slapd hangs while the server is giving the certificate; but this does not happen if the server is run with -d 2 or higher, or if the client is the server itself.
Details:
(A seemingly similar issue has been reported before, without satisfactory reply, 4 years ago: http://www.openldap.org/lists/openldap-software/200210/msg00459.html)
My slapd is the Debian-etch-packaged 2.3.30.
Problems with SSL on Debian are well known, and it is due to the fact that they long ago patched OpenLDAP 2.1 to compile against GnuTLS (note, I don't say *work*, just compile).
When you use their 2.2 and 2.3 packages, and their libraries get loaded into the same user space as the 2.1 libraries (which are always installed), then SSL/TLS stop working. There is *nothing* the OpenLDAP folks can do about this.
My only advice to you is to not use the Debian packages. Build OpenLDAP yourself, or get a prebuilt distribution like Symas' CDS that installs into a completely separate location so that it is not polluted by the Debian packages.
In the meantime, Stanford U. & The Written Word have hired Symas corporation to create true integration between GnuTLS and OpenLDAP. It is anticipated that will be part of the OpenLDAP 2.4 release. Too late for Debian's etch release, but I assume this problem will finally go away for Debian in the release after that.
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
I wrote:
A tls connection between a client and a 2.3.30 slapd hangs while the server is giving the certificate; but this does not happen if the server is run with -d 2 or higher, or if the client is the server itself.
My slapd is the Debian-etch-packaged 2.3.30.
Quanah Gibson-Mount quanah@stanford.edu replied:
Problems with SSL on Debian are well known, and it is due to the fact that they long ago patched OpenLDAP 2.1 to compile against GnuTLS
Thanks for the information. I downloaded and compiled 2.3.32 and I have exactly the same symptom. I used ./configure --with-tls and had to install package libssl-dev for it to work.
Could it be a dependency on a Debian shared library? While investigating whether this is the case with gnutls, I found out that the slapd package does depend on the libgnutls package, but I can't find the library dependence using ldd, and when I try "lsof|grep gnutls", slapd (the Debian slapd) isn't listed there (I can see slapd if I run "lsof|grep libssl). So either the Debian slapd does not actually use gnutls, or I don't know what's going on here (which is very likely).
Here is the dependencies of the compiled slapd:
acheloos:/usr/lib# ldd /usr/local/libexec/slapd linux-gate.so.1 => (0xffffe000) libdb-4.2.so => /usr/lib/libdb-4.2.so (0xb7e7e000) libssl.so.0.9.8 => /usr/lib/i686/cmov/libssl.so.0.9.8 (0xb7e3f000) libcrypto.so.0.9.8 => /usr/lib/i686/cmov/libcrypto.so.0.9.8 (0xb7d04000) libresolv.so.2 => /lib/tls/i686/cmov/libresolv.so.2 (0xb7cf1000) libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7cdf000) libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7bae000) libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7baa000) libz.so.1 => /usr/lib/libz.so.1 (0xb7b96000) /lib/ld-linux.so.2 (0xb7f5d000)
--On Wednesday, February 28, 2007 1:29 PM +0200 Antonis Christofides anthony@itia.ntua.gr wrote:
I wrote:
A tls connection between a client and a 2.3.30 slapd hangs while the server is giving the certificate; but this does not happen if the server is run with -d 2 or higher, or if the client is the server itself.
My slapd is the Debian-etch-packaged 2.3.30.
Quanah Gibson-Mount quanah@stanford.edu replied:
Problems with SSL on Debian are well known, and it is due to the fact that they long ago patched OpenLDAP 2.1 to compile against GnuTLS
Thanks for the information. I downloaded and compiled 2.3.32 and I have exactly the same symptom. I used ./configure --with-tls and had to install package libssl-dev for it to work.
Could it be a dependency on a Debian shared library? While investigating whether this is the case with gnutls, I found out that the slapd package does depend on the libgnutls package, but I can't find the library dependence using ldd, and when I try "lsof|grep gnutls", slapd (the Debian slapd) isn't listed there (I can see slapd if I run "lsof|grep libssl). So either the Debian slapd does not actually use gnutls, or I don't know what's going on here (which is very likely).
The libraries compiled against GnuTLS are:
:/usr/lib> ldd libldap.so.2.0.130 liblber.so.2 => /usr/lib/liblber.so.2 (0xa7f16000) libresolv.so.2 => /lib/tls/libresolv.so.2 (0xa7f03000) libdl.so.2 => /lib/tls/libdl.so.2 (0xa7f00000) libcrypt.so.1 => /lib/tls/libcrypt.so.1 (0xa7ed3000) libsasl2.so.2 => /usr/lib/libsasl2.so.2 (0xa7ebe000) libgnutls.so.11 => /usr/lib/libgnutls.so.11 (0xa7e57000) libpthread.so.0 => /lib/tls/libpthread.so.0 (0xa7e48000) libc.so.6 => /lib/tls/libc.so.6 (0xa7d12000) /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x75555000) libtasn1.so.2 => /usr/lib/libtasn1.so.2 (0xa7d01000) libgcrypt.so.11 => /usr/lib/libgcrypt.so.11 (0xa7cb4000) libgpg-error.so.0 => /usr/lib/libgpg-error.so.0 (0xa7cb0000) libz.so.1 => /usr/lib/libz.so.1 (0xa7c9e000) libnsl.so.1 => /lib/tls/libnsl.so.1 (0xa7c8a000)
The problem comes when the user ID running slapd, and the user ID handling other things that load /usr/lib/libldap.so.* are the same, whether that is root or the ldap user. As soon as both sets of libraries get loaded into the same user space, problems ensue.
Now, I just ran my build of OpenLDAP 2.3.28+patches (pretty much 2.3.32) at -d 0, and did a TLS bind to it, and there was no problem. (ldapsearch -ZZZ). I also ran my build of OpenLDAP 2.3.33 at -d 0, and did a TLS bind to it, and there was no problem.
So I can't reproduce the problem you are reporting, and this is on a debian server where I'm sure that slapd is only linked against the software pieces I've built myself.
ldd slapd /usr/local/lib/libhoard.so => /usr/local/lib/libhoard.so (0x00002ba7e0e35000) /usr/lib/libdl.so => /usr/lib/libdl.so (0x00002ba7e102a000) libldap_r-2.3.so.0 => /usr/local/lib/libldap_r-2.3.so.0 (0x00002ba7e112d000) liblber-2.3.so.0 => /usr/local/lib/liblber-2.3.so.0 (0x00002ba7e1276000) libsasl2.so.2 => /usr/local/lib/libsasl2.so.2 (0x00002ba7e1385000) libssl.so.0.9.8 => /usr/local/lib/libssl.so.0.9.8 (0x00002ba7e149d000) libcrypto.so.0.9.8 => /usr/local/lib/libcrypto.so.0.9.8 (0x00002ba7e15e4000) libresolv.so.2 => /lib/libresolv.so.2 (0x00002ba7e185e000) libltdl.so.3 => /usr/local/lib/libltdl.so.3 (0x00002ba7e1972000) libwrap.so.0 => /lib/libwrap.so.0 (0x00002ba7e1a79000) libpthread.so.0 => /lib/libpthread.so.0 (0x00002ba7e1b83000) libc.so.6 => /lib/libc.so.6 (0x00002ba7e1c97000) libstdc++.so.6 => /usr/local/lib/libstdc++.so.6 (0x00002ba7e1ed6000) libm.so.6 => /lib/libm.so.6 (0x00002ba7e20d6000) libgcc_s.so.1 => /usr/local/lib/libgcc_s.so.1 (0x00002ba7e225c000) /lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00002ba7e0d1f000) libnsl.so.1 => /lib/libnsl.so.1 (0x00002ba7e236a000)
Which is most everything, as you can see. ;)
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
Quanah Gibson-Mount quanah@stanford.edu writes:
The libraries compiled against GnuTLS are:
:/usr/lib> ldd libldap.so.2.0.130
...
libgnutls.so.11 => /usr/lib/libgnutls.so.11 (0xa7e57000)
However, the Debian etch slapd and ldap utilities, such as ldapsearch, don't use this library. It's only some other things that do so, such as samba. slapd and ldap* use libldap[_r]-2.3 instead, which appears to not be using gnutls.
The problem comes when the user ID running slapd, and the user ID handling other things that load /usr/lib/libldap.so.* are the same, whether that is root or the ldap user. As soon as both sets of libraries get loaded into the same user space, problems ensue.
I stopped all services that were using libldap*, then started the compiled slapd (with -u openldap anyway, and nothing else runs as that user, though I'm not certain the libraries aren't opened before it switches priviliges), and still I had the same symptom. (And my compiled slapd is also believed to not use gnutls; it doesn't use libldap anyway.)
I therefore don't see any evidence that the problem is really gnutls-related. That you can't reproduce it doesn't quite help, you've compiled almost everything yourself.
You think I should also try to gradually compile things myself, trying to locate which library causes the problem? Needless to say I'd prefer a simpler investigation path, if one exists.
A debian bug reporting similar behaviour was filed two days ago: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=412706. I'll add my experience there.
--On Thursday, March 01, 2007 12:05 PM +0200 Antonis Christofides anthony@itia.ntua.gr wrote:
I stopped all services that were using libldap*, then started the compiled slapd (with -u openldap anyway, and nothing else runs as that user, though I'm not certain the libraries aren't opened before it switches priviliges), and still I had the same symptom. (And my compiled slapd is also believed to not use gnutls; it doesn't use libldap anyway.)
I therefore don't see any evidence that the problem is really gnutls-related. That you can't reproduce it doesn't quite help, you've compiled almost everything yourself.
You think I should also try to gradually compile things myself, trying to locate which library causes the problem? Needless to say I'd prefer a simpler investigation path, if one exists.
Well, I think the fact that I can't reproduce it points to one of two things:
(a) A misconfiguration of slapd on your part or (b) A problem in the packaging on the part of debian
Given that SSL tends to work just fine for you as long as you aren't doing the differing debug level, I doubt it is (a), which points to (b) as the problem. There's not a whole lot the OpenLDAP folks can do as the upstream providers about that. It would need to be resolved by the debian folks.
Building out everything yourself or not is obviously your choice. I guess it depends just how much time you want to spend tracking down where the issue resides.
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
openldap-software@openldap.org
-
Antonis Christofides -
Quanah Gibson-Mount