--On December 10, 2007 5:37:36 PM +0100 Fabian Steiner lists@fabis-site.net wrote:

Hello! We are experiencing exactly the same issues without having found any solution yet. Any suggestion highly appreciated!

Hopefully all of the Debian specific problems with TLS will go away once OpenLDAP 2.4 is released and integrated into Debian, since it has GnuTLS support. People often run into SSL/TLS issues on Debian because it has a hacked version of OpenLDAP 2.1 libraries linked against GnuTLS, that causes serious problems when the ldap libraries linked against OpenSSL are also loaded.

--Quanah

--

Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

Dear all,

does anyone have any clue for this issue ?

Many thanks in advance

Best regards

Denis Sacchet

Denis Sacchet wrote:

...

Quanah Gibson-Mount : Hopefully all of the Debian specific problems with TLS will go away once OpenLDAP 2.4 is released and integrated into Debian, since it has GnuTLS support. People often run into SSL/TLS issues on Debian because it has a hacked version of OpenLDAP 2.1 libraries linked against GnuTLS, that causes serious problems when the ldap libraries linked against OpenSSL are also loaded.

The mix between openssl and gnutls is my first asumption. I tried some trick in this sense ... I already have problems with /etc/ssl/certs directory, when I use CACertDirs with a lot of certificates in the directory, it was awfully long to get the result, so I switch to CaCertFile instead ... And the problem is only with pam_ldap and not with ldapsearch (or the contrary, I don't remember), and the pam_ldap was linked against gnutls and ldapsearch linked against openssl ...

Do you know when the Openldap 2.4 is planned to be available in Debian and also simply from openldap ?

...

Buchan Milne says : So, how are you sure this is TLS-specific, and not just lack of (say) BDB backend tuning resulting in deferred binds ?

Could you please explain a little bit more your suspicion ? I don't really see what you point out !

...

Howard Chu : The fact that a reboot is required indicates that any problem is not in any user-level code. Maybe your /dev/random has run out of entropy, or some other underlying system resource is gone. Maybe strace would help here.

For my current problem, I also have doubts about /dev/(u)random and entropy because it was very long to generate a 2048 bits private keys, as the hardware is a Via C7, I switch to /dev/hwrng which is a very good random number generator. And the problem is always here ...

To finish, I put some strace and syslog as requested :

http://www.ouba.org/strace.slapd.no.probleme.tls.200712070838 http://www.ouba.org/strace.slapd.probleme.tls.200712070809 http://www.ouba.org/syslog.slapd.tls.problem.200712070804

Thanks all for your help

Best regards

Denis Sacchet

I already have problems with /etc/ssl/certs directory, when I use CACertDirs with a lot of certificates in the directory, it was awfully long to get the result, so I switch to CaCertFile instead ... And the problem is only with pam_ldap and not with ldapsearch (or the contrary, I don't remember), and the pam_ldap was linked against gnutls and ldapsearch linked against openssl ...

We experienced the same problem. The culprit turned out to be libldap2 (opposed to libldap-2.3-0), which has GNUTLS support patched in by Debian. To the contrary, ldapsearch is linked against libldap-2.3-0, which uses OpenSSL.

More precisely, the problem is get_ca_list() in libraries/libldap/tls.c. If you profile a test program like

#include <stdlib.h> #include <dirent.h> #include <stdio.h> #include <sys/param.h> #include <sys/stat.h> #include <gnutls/gnutls.h> gnutls_certificate_credentials_t ca_list;

int main(int argc, char *argv[]) { DIR *d; struct dirent *dent; char ca_file[MAXPATHLEN]; struct stat s; char *dir;

if (argc != 2) exit(1); dir = argv[1]; d = opendir(dir); if (d == NULL) exit(1); gnutls_global_init(); gnutls_certificate_allocate_credentials(&ca_list); while ((dent = readdir(d)) != NULL) { snprintf(ca_file, sizeof ca_file, "%s/%s", dir, dent->d_name); stat(ca_file, &s); if (!S_ISREG(s.st_mode)) continue; gnutls_certificate_set_x509_trust_file(ca_list, ca_file, GNUTLS_X509_FMT_PEM); } closedir(d); }

you see unreasonable amounts of time being spent in memory and string manipulation routines.

Not being an TLS/SSL expert, I'm wondering why you need to add all those certificates in the first place. I thought the whole point of all those <subject hash>.<serial> links in /etc/openssl/certs (or whatever) was that a client could find a CA certificate simply by hashing the subject.

Another pitfall is that libnss-ldap ignores the tls_cacert{file,dir} directives in libnss-ldap.conf unless "ssl on" is also given. So, if you set the URL to ldaps://..., you end up in OpenLDAP using SSL, but with the parameters given in ldap.conf, not those in libnss-ldap.conf. I've reported this to PADL Soft.

Simon Josefsson wrote:

Edgar Fußef@math.uni-bonn.de writes:

...

...

I already have problems with /etc/ssl/certs directory, when I use CACertDirs with a lot of certificates in the directory, it was awfully long to get the result, so I switch to CaCertFile instead ... And the problem is only with pam_ldap and not with ldapsearch (or the contrary, I don't remember), and the pam_ldap was linked against gnutls and ldapsearch linked against openssl ...

We experienced the same problem. The culprit turned out to be libldap2 (opposed to libldap-2.3-0), which has GNUTLS support patched in by Debian. To the contrary, ldapsearch is linked against libldap-2.3-0, which uses OpenSSL.

More precisely, the problem is get_ca_list() in libraries/libldap/tls.c. If you profile a test program like

#include<stdlib.h> #include<dirent.h> #include<stdio.h> #include<sys/param.h> #include<sys/stat.h> #include<gnutls/gnutls.h> gnutls_certificate_credentials_t ca_list;

int main(int argc, char *argv[]) { DIR *d; struct dirent *dent; char ca_file[MAXPATHLEN]; struct stat s; char *dir;

if (argc != 2) exit(1); dir = argv[1]; d = opendir(dir); if (d == NULL) exit(1); gnutls_global_init(); gnutls_certificate_allocate_credentials(&ca_list); while ((dent = readdir(d)) != NULL) { snprintf(ca_file, sizeof ca_file, "%s/%s", dir, dent->d_name); stat(ca_file,&s); if (!S_ISREG(s.st_mode)) continue; gnutls_certificate_set_x509_trust_file(ca_list, ca_file, GNUTLS_X509_FMT_PEM); } closedir(d); }

you see unreasonable amounts of time being spent in memory and string manipulation routines.

Edgar, thanks for providing test code. I profiled and improved GnuTLS to execute your code faster, on my system the time was reduced from 40 seconds to 0.3 seconds. I found debugging and fixing this problem interesting so I blogged about it:

http://blog.josefsson.org/2008/02/27/real-world-performance-tuning-with-call...

Yes, callgrind is pretty cool, much more useful than old gprof-style profilers. Though as you note, the runtimes of any valgrind tool can be pretty extreme. My tweaked version of FunctionCheck is also useful when you can afford to compile an instrumented version of your code. Faster runtimes in exchange for extra compile time - frequently it's a worthwhile tradeoff.

http://highlandsun.com/hyc/#fncchk

...

Not being an TLS/SSL expert, I'm wondering why you need to add all those certificates in the first place. I thought the whole point of all those<subject hash>.<serial> links in /etc/openssl/certs (or whatever) was that a client could find a CA certificate simply by hashing the subject.

GnuTLS doesn't support hashed certificate directories. Further, TLS servers need to send a list of names of trusted certificates to clients, so the server has to open and parse all local trust roots anyway. Right now, this is done for clients too, since the relevant code in GnuTLS doesn't know whether it will be used as a client or server. I hope the new code will be fast enough so that it isn't a bottle-neck. I suppose that it could be optimized further, so that it isn't done for clients at all, but let's not optimize prematurely.

You're already approaching GnuTLS version 2.4. If optimizing now is premature, when will it actually be time to optimize?

Simon Josefsson wrote:

Howard Chuhyc@symas.com writes:

...

Yes, callgrind is pretty cool, much more useful than old gprof-style profilers. Though as you note, the runtimes of any valgrind tool can be pretty extreme. My tweaked version of FunctionCheck is also useful when you can afford to compile an instrumented version of your code. Faster runtimes in exchange for extra compile time - frequently it's a worthwhile tradeoff.

http://highlandsun.com/hyc/#fncchk

Interesting, does the code still run and is useful? The latest comment about it was from 2005. To be able to use it, I need to set up a useful benchmark environment first though.

Sure, works fine. After I got call arc timings in, it pretty much had everything I wanted from it so it hasn't changed since. Works on Linux and Solaris.

Dieter Kluenter a écrit :

You are connection to host ldap.domain.com

...

/C=FR/ST=Lorraine/L=Nancy/O=<hiddencompany>/OU=<hiddencompany>/CN=smtp.<hiddendomain>.com/emailAddress=it@<hiddendomain>.com,

but the certificate is issued to host smtp.domain.com, a certificate verification must fail.

In fact, the trace doesn't show it, but I have an alternate domain name in the certificate, and the CN is set to the FQDN of the server (without that, OpenLDAP doesn't start), so if you look at the successful connection, the host and certificate are the same, and it works ...

Thanks for your answer ...

Best regards

Denis

Howard Chu wrote:

Denis Sacchet wrote:

...

Hello,

I have a strange behaviour regarding TLS encryption with an LDAP server. Everything works like a charm for a while, and without any sign, the server begins to not respond for TLS traffic. As the server is partially open on internet, I force TLS, so it is very annoying for us.

A trace of ldapsearch when there is the problem :

Show the corresponding slapd debug output for the same situation.

Of course, I don't want to hijack the OP's thread but as our problems seem to be rather similar I can also provide the corresponding slapd log:

[...] slapd starting

...

...

slap_listener(ldap:///)ldap_pvt_gethostbyname_a: host=uranos, r=0

connection_get(11): got connid=0 connection_read(11): checking for input on id=0 ber_get_next ber_get_next: tag 0x30 len 29 contents: ber_get_next ber_get_next on fd 11 failed errno=11 (Resource temporarily unavailable) do_extended ber_scanf fmt ({m) ber: send_ldap_extended: err=0 oid= len=0 send_ldap_response: msgid=1 tag=120 err=0 ber_flush: 14 bytes to sd 11 connection_get(11): got connid=0 connection_read(11): checking for input on id=0 TLS trace: SSL_accept:before/accept initialization TLS trace: SSL_accept:SSLv3 read client hello A TLS trace: SSL_accept:SSLv3 write server hello A TLS trace: SSL_accept:SSLv3 write certificate A TLS trace: SSL_accept:SSLv3 write certificate request A TLS trace: SSL_accept:SSLv3 flush data TLS trace: SSL_accept:error in SSLv3 read client certificate A TLS trace: SSL_accept:error in SSLv3 read client certificate A connection_get(11): got connid=0 connection_read(11): checking for input on id=0 TLS trace: SSL3 alert write:fatal:handshake failure TLS trace: SSL_accept:error in SSLv3 read client certificate B TLS: can't accept. TLS: error:140890C7:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:peer did not return a certificate s3_srvr.c:2471 connection_read(11): TLS accept failure error=-1 id=0, closing

[...]

Thanks, Fabian

Howard Chu wrote:

Fabian Steiner wrote:

...

Of course, I don't want to hijack the OP's thread but as our problems seem to be rather similar I can also provide the corresponding slapd log:

This looks like a simple configuration error; you have slapd configured to require client certificates and the client didn't send one. Either you need to configure the client with a certificate, or you need to relax the requirement on the server. [...]

In fact, this was also our first assumption after having analyzed the output for the very first time but due to our configuration this should't happen:

[...] TLSCertificateFile /etc/ssl-certs/ldap.crt TLSCertificateKeyFile /etc/ssl-certs/ldap.key TLSCACertificateFile /etc/ssl-certs/ca.crt TLSVerifyClient never [...]

Moreover, this wouldn't explain why it /does/ work for some time (as far as our case is concerned it works as long as slapd isn't restarted). Once the problem has occured the server has to be rebooted in order to ensure a working setup again :-(

Thanks, Fabian

Howard Chu wrote:

The fact that a reboot is required indicates that any problem is not in any user-level code. Maybe your /dev/random has run out of entropy, or some other underlying system resource is gone. Maybe strace would help here.

Thanks for pointing me into the right direction. I can remember having problems with Apache and mod_ssl in the past as they lacked of entropy with no keyboard and mouse being attached to the machine. I have just installed rng-tools and hope that this might solve the issues.

I'll report on it as soon as I can confirm that this was the cause.

Cheers, Fabian