This feels like a lame question, but I'm out of ideas, so...
I'm moving samba service between a couple of FreeBSD systems (9.3 to 10.2), and I'm stuck on getting samba on the new machine to connect to our openldap server over ssl - frustrating since I've been running samba+ldap for 15 years or so; feel sure I'm missing something basic!
The smbd-to-ldap connection works fine with no encryption, but I get errors when using either TLS to port 389 ("Failed to issue the StartTLS instruction: Connect error") or SSL to 636.
However I'm pretty certain it's not a certificate or CA validation issue. All my other ldap clients on that server are working as expected, including a simple "ldapsearch -ZZ". Both openssl s_client and gnutls-cli show the ldap server certificate validating.
Maximum debugging on the ldap server gave me: connection_read(3): TLS accept failure error=-1 id=1042, closing conn=1042 fd=3 closed (TLS negotiation failure)
Running smbd under gbd shows ldap_start_tls_s (ldap_struct, NULL, NULL) in smb_ldap_start_tls is returning -11 (LDAP_CONNECT_ERROR). That doesn't give me any ideas, sadly!
Capturing the packet exchange between smbd and slapd, I'm seeing the (smbd) client sends a "decrypt error" (TLS alert code 51) to the ldap server after receiving the certificate, while the working "ldapsearch -ZZ" moves on to client key exchange etc.
Besides all my other ldap clients working successfully, I think I'd expect to see a TLS alert more like 42-48 if it were a cert validation problem.
I'm not sure where to try looking next for the cause, would welcome any suggestions...
Thanks, Graham
Replying to my own message here, but I continue to investigate my problem and can't explain what I see. I put together a small test program to connect to our ldap server using same parameters as smbd. Setting "ldap debug level = 1" in smb.conf, and the equivalent LDAP_DEBUG_TRACE in my test program shows the smbd output complaining of certificate signature failure.
smbd output:
[LDAP] ldap_simple_bind_s [LDAP] ldap_sasl_bind_s [LDAP] ldap_sasl_bind [LDAP] ldap_send_initial_request [LDAP] ldap_new_connection 1 1 0 [LDAP] ldap_int_open_connection [LDAP] ldap_connect_to_host: TCP ldap.spa.umn.edu:636 [LDAP] ldap_new_socket: 9 [LDAP] ldap_prepare_socket: 9 [LDAP] ldap_connect_to_host: Trying 128.101.220.24:636 [LDAP] ldap_pvt_connect: fd: 9 tm: -1 async: 0 [LDAP] attempting to connect: [LDAP] connect success [LDAP] TLS trace: SSL_connect:before/connect initialization [LDAP] TLS trace: SSL_connect:SSLv2/v3 write client hello A [LDAP] TLS trace: SSL_connect:SSLv3 read server hello A [LDAP] TLS certificate verification: depth: 3, err: 0, subject: /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root,[LDAP] issuer: /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root [LDAP] TLS certificate verification: depth: 2, err: 0, subject: /C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority,[LDAP] issuer: /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root [LDAP] TLS certificate verification: depth: 1, err: 0, subject: /C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA,[LDAP] issuer: /C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority [LDAP] TLS certificate verification: depth: 0, err: 7, subject: /C=US/postalCode=55455/ST=MN/L=Minneapolis/street=100 Union Street SE/O=University of Minnesota/OU=School of Physics and Astronomy/CN=ldap.spa.umn.edu,[LDAP] issuer: /C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA [LDAP] TLS certificate verification: Error, certificate signature failure [LDAP] TLS certificate verification: depth: 0, err: 7, subject: /C=US/postalCode=55455/ST=MN/L=Minneapolis/street=100 Union Street SE/O=University of Minnesota/OU=School of Physics and Astronomy/CN=ldap.spa.umn.edu,[LDAP] issuer: /C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA [LDAP] TLS trace: SSL_connect:SSLv3 read server certificate A [LDAP] TLS trace: SSL_connect:SSLv3 read server done A [LDAP] TLS trace: SSL_connect:SSLv3 write client key exchange A [LDAP] TLS trace: SSL_connect:error in error [LDAP] TLS trace: SSL_connect:error in error [LDAP] TLS: can't connect: .
But my test program on same machine gives:
ldap_simple_bind_s ldap_sasl_bind_s ldap_sasl_bind ldap_send_initial_request ldap_new_connection 1 1 0 ldap_int_open_connection ldap_connect_to_host: TCP ldap.spa.umn.edu:636 ldap_new_socket: 3 ldap_prepare_socket: 3 ldap_connect_to_host: Trying 128.101.220.24:636 ldap_pvt_connect: fd: 3 tm: -1 async: 0 attempting to connect: connect success TLS trace: SSL_connect:before/connect initialization TLS trace: SSL_connect:SSLv2/v3 write client hello A TLS trace: SSL_connect:SSLv3 read server hello A TLS certificate verification: depth: 3, err: 0, subject: /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root, issuer: /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root TLS certificate verification: depth: 2, err: 0, subject: /C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority, issuer: /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root TLS certificate verification: depth: 1, err: 0, subject: /C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA, issuer: /C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority TLS certificate verification: depth: 0, err: 0, subject: /C=US/postalCode=55455/ST=MN/L=Minneapolis/street=100 Union Street SE/O=University of Minnesota/OU=School of Physics and Astronomy/CN=ldap.spa.umn.edu, issuer: /C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA TLS trace: SSL_connect:SSLv3 read server certificate A TLS trace: SSL_connect:SSLv3 read server done A TLS trace: SSL_connect:SSLv3 write client key exchange A TLS trace: SSL_connect:SSLv3 write change cipher spec A TLS trace: SSL_connect:SSLv3 write finished A TLS trace: SSL_connect:SSLv3 flush data TLS trace: SSL_connect:SSLv3 read server session ticket A TLS trace: SSL_connect:SSLv3 read finished A ldap_open_defconn: successful ldap_send_server_request
Same certificate chain, but one case verifies and the other doesn't...
I also stepped through smbd with gdb and verified that the parameters to ldap_simple_bind_s are the same as my test case.
Wonder if anyone can venture a guess how this might occur?
G.
--On Friday, January 08, 2016 3:38 PM -0600 Graham Allan allan@physics.umn.edu wrote:
Replying to my own message here, but I continue to investigate my problem and can't explain what I see. I put together a small test program to connect to our ldap server using same parameters as smbd. Setting "ldap debug level = 1" in smb.conf, and the equivalent LDAP_DEBUG_TRACE in my test program shows the smbd output complaining of certificate signature failure.
smbd output:
[LDAP] ldap_simple_bind_s [LDAP] ldap_sasl_bind_s [LDAP] ldap_sasl_bind [LDAP] ldap_send_initial_request [LDAP] ldap_new_connection 1 1 0 [LDAP] ldap_int_open_connection [LDAP] ldap_connect_to_host: TCP ldap.spa.umn.edu:636 [LDAP] ldap_new_socket: 9 [LDAP] ldap_prepare_socket: 9 [LDAP] ldap_connect_to_host: Trying 128.101.220.24:636 [LDAP] ldap_pvt_connect: fd: 9 tm: -1 async: 0 [LDAP] attempting to connect: [LDAP] connect success [LDAP] TLS trace: SSL_connect:before/connect initialization [LDAP] TLS trace: SSL_connect:SSLv2/v3 write client hello A [LDAP] TLS trace: SSL_connect:SSLv3 read server hello A [LDAP] TLS certificate verification: depth: 3, err: 0, subject: /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root,[LDAP] issuer: /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root [LDAP] TLS certificate verification: depth: 2, err: 0, subject: /C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority,[LDAP] issuer: /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root [LDAP] TLS certificate verification: depth: 1, err: 0, subject: /C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA,[LDAP] issuer: /C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority [LDAP] TLS certificate verification: depth: 0, err: 7, subject: /C=US/postalCode=55455/ST=MN/L=Minneapolis/street=100 Union Street SE/O=University of Minnesota/OU=School of Physics and Astronomy/CN=ldap.spa.umn.edu,[LDAP] issuer: /C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA [LDAP] TLS certificate verification: Error, certificate signature failure [LDAP] TLS certificate verification: depth: 0, err: 7, subject: /C=US/postalCode=55455/ST=MN/L=Minneapolis/street=100 Union Street SE/O=University of Minnesota/OU=School of Physics and Astronomy/CN=ldap.spa.umn.edu,[LDAP] issuer: /C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA [LDAP] TLS trace: SSL_connect:SSLv3 read server certificate A [LDAP] TLS trace: SSL_connect:SSLv3 read server done A [LDAP] TLS trace: SSL_connect:SSLv3 write client key exchange A [LDAP] TLS trace: SSL_connect:error in error [LDAP] TLS trace: SSL_connect:error in error [LDAP] TLS: can't connect: .
Error in error is a pretty interesting. What SSL libs is samba linked to? What SSL libs is your test program linked to?
--Quanah
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
On 01/08/2016 03:45 PM, Quanah Gibson-Mount wrote:
Error in error is a pretty interesting. What SSL libs is samba linked to? What SSL libs is your test program linked to?
It did make me wonder! The failure right after "write client key exchange A" does seem to correlate with my wireshark capture (client sends a "decrypt error" (TLS alert code 51) to the ldap server after receiving the certificate).
The actual error from ldap_simple_bind_s is: error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib
This is a FreeBSD 10.2 system, which uses openssl 1.0.1p. Both smbd and my test should be linked to the same ldap and ssl libraries - here's ldd output:
ldaptest: libldap-2.4.so.2 => /usr/local/lib/libldap-2.4.so.2 (0x800820000) libc.so.7 => /lib/libc.so.7 (0x800a66000) liblber-2.4.so.2 => /usr/local/lib/liblber-2.4.so.2 (0x800e12000) libssl.so.7 => /usr/lib/libssl.so.7 (0x801020000) libcrypto.so.7 => /lib/libcrypto.so.7 (0x80128c000)
and
/usr/local/sbin/smbd: libldap-2.4.so.2 => /usr/local/lib/libldap-2.4.so.2 (0x80103f000) liblber-2.4.so.2 => /usr/local/lib/liblber-2.4.so.2 (0x801285000) libcrypt.so.5 => /lib/libcrypt.so.5 (0x801493000) libpam.so.5 => /usr/lib/libpam.so.5 (0x8016b3000) libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x8018bf000) libmd.so.6 => /lib/libmd.so.6 (0x801ac2000) librt.so.1 => /usr/lib/librt.so.1 (0x801cd2000) libthr.so.3 => /lib/libthr.so.3 (0x801ed8000) libpopt.so.0 => /usr/local/lib/libpopt.so.0 (0x8020fc000) libtalloc.so.2 => /usr/local/lib/libtalloc.so.2 (0x802308000) libtevent.so.0 => /usr/local/lib/libtevent.so.0 (0x802515000) libtdb.so.1 => /usr/local/lib/libtdb.so.1 (0x802723000) libz.so.6 => /lib/libz.so.6 (0x802938000) libc.so.7 => /lib/libc.so.7 (0x802b4e000) libssl.so.7 => /usr/lib/libssl.so.7 (0x802efa000) libcrypto.so.7 => /lib/libcrypto.so.7 (0x803166000) libelf.so.1 => /usr/lib/libelf.so.1 (0x80355a000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x80376f000) libintl.so.8 => /usr/local/lib/libintl.so.8 (0x80397d000)
Thanks for any ideas!
Graham
On Fri, 8 Jan 2016, Graham Allan wrote:
Replying to my own message here, but I continue to investigate my problem and can't explain what I see. I put together a small test program to connect to our ldap server using same parameters as smbd. Setting "ldap debug level = 1" in smb.conf, and the equivalent LDAP_DEBUG_TRACE in my test program shows the smbd output complaining of certificate signature failure.
smbd output:
...
[LDAP] TLS certificate verification: depth: 0, err: 7, subject: /C=US/postalCode=55455/ST=MN/L=Minneapolis/street=100 Union Street SE/O=University of Minnesota/OU=School of Physics and Astronomy/CN=ldap.spa.umn.edu,[LDAP] issuer: /C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA [LDAP] TLS certificate verification: Error, certificate signature failure
Some certs verify, another doesn't: so what's different about that cert? Different signature hash algorithm, sha256 perhaps?
...
But my test program on same machine gives:
...
TLS certificate verification: depth: 0, err: 0, subject: /C=US/postalCode=55455/ST=MN/L=Minneapolis/street=100 Union Street SE/O=University of Minnesota/OU=School of Physics and Astronomy/CN=ldap.spa.umn.edu, issuer: /C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA
...
Same certificate chain, but one case verifies and the other doesn't...
I also stepped through smbd with gdb and verified that the parameters to ldap_simple_bind_s are the same as my test case.
Wonder if anyone can venture a guess how this might occur?
Are smbd and your test program linked against the same libldap version and openssl version?
Philip Guenther
On 01/08/2016 04:03 PM, Philip Guenther wrote:
On Fri, 8 Jan 2016, Graham Allan wrote:
Replying to my own message here, but I continue to investigate my problem and can't explain what I see. I put together a small test program to connect to our ldap server using same parameters as smbd. Setting "ldap debug level = 1" in smb.conf, and the equivalent LDAP_DEBUG_TRACE in my test program shows the smbd output complaining of certificate signature failure.
smbd output:
...
[LDAP] TLS certificate verification: depth: 0, err: 7, subject: /C=US/postalCode=55455/ST=MN/L=Minneapolis/street=100 Union Street SE/O=University of Minnesota/OU=School of Physics and Astronomy/CN=ldap.spa.umn.edu,[LDAP] issuer: /C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA [LDAP] TLS certificate verification: Error, certificate signature failure
Some certs verify, another doesn't: so what's different about that cert? Different signature hash algorithm, sha256 perhaps?
The cert is sha256 as it happens, but both smbd and the test case are connecting to the same ldap server, so receive the same certificate. I'm calling the same ldap library functions with the same parameters, which is what makes this so odd.
The smbd code does potentially call a few other ldap_set_option settings, eg referral behaviour, timeouts, attempt to upgrade to LDAPv3, but I don't see much really happening there in gdb - FWIW I tested skipping over these calls with no difference in result.
Are smbd and your test program linked against the same libldap version and openssl version?
They are, yes (I just posted ldd output in response to Quanah's reply).
Thanks for the ideas,
Graham
openldap-technical@openldap.org
-
Graham Allan -
Philip Guenther -
Quanah Gibson-Mount