Not sure if this is an openldap issue but have to examine everything we can.
We revised our nss certificate store as part of addressing the expiration of our root cert.
It now has two certs, the end service cert and the intermediate. Basic client operations (ldapsearch) work fine; using -d1 shows that the appropriate service certificate is loaded and the the search is successful.
But if we run an 'openssl s_client -showcerts' against the host and port 636, we continue to see the expired root certificate even though it's not in the nss store configured chain. This is causing issues for some applications (mainly java based) so we're just trying to understand where the expired root would be coming from if it's not in the openldap server configuration.
Thanks,
Peter
relevant bits: #slapd.conf # TLS/ssl TLSCACertificatePath /etc/openldap/certs TLSCertificateFile DirectoryLdap
#sudo certutil -d /etc/openldap/certs -L
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
intermediate-2024 ,, PennGroupsLdap
#sudo certutil -d /etc/openldap/certs -L -n PennGroupsLdap Certificate: Data: Version: 3 (0x2) Serial Number: 2b:27:6c:70:ac:b4:5c:3d:11:05:17:d9:15:59:24:af Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=InCommon RSA Server CA,OU=InCommon,O=Internet2,L=Ann Arbo r,ST=MI,C=US" Validity: Not Before: Tue Feb 05 00:00:00 2019 Not After : Thu Feb 04 23:59:59 2021 Subject: "CN=penngroups-dev.net.isc.upenn.edu,OU=ISC: N&T - NES - Ide ntity and Access Management (IAM),O=University of Pennsylvania,ST REET=3451 Walnut Street,L=Philadelphia,ST=PA,postalCode=19104,C=U S" *snip* #sudo certutil -d /etc/openldap/certs -L -n intermediate-2024 Certificate: Data: Version: 3 (0x2) Serial Number: 47:20:d0:fa:85:46:1a:7e:17:a1:64:02:91:84:63:74 Signature Algorithm: PKCS #1 SHA-384 With RSA Encryption Issuer: "CN=USERTrust RSA Certification Authority,O=The USERTRUST Net work,L=Jersey City,ST=New Jersey,C=US" Validity: Not Before: Mon Oct 06 00:00:00 2014 Not After : Sat Oct 05 23:59:59 2024 Subject: "CN=InCommon RSA Server CA,OU=InCommon,O=Internet2,L=Ann Arb or,ST=MI,C=US" *snip*
gather the openssl s_client info; why are 4 certs (depth 0->3) presented instead of the expected 2 (dept h0->1)?
#openssl s_client -host localhost -port 636 -showcerts 2>local.certs >> local.certs
#grep -A1 "s:" local.certs 0 s:/C=US/postalCode=19104/ST=PA/L=Philadelphia/street=3451 Walnut Street/O=University of Pennsylvania/OU=ISC: N&T - NES - Identity and Access Management (IAM)/CN=penngroups-dev.net.isc.upenn.edu i:/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA -- 1 s:/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA i:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority -- 2 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root -- 3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
0 s:/C=US/postalCode=19104/ST=PA/L=Philadelphia/street=3451 Walnut Street/O=University of Pennsylvania/OU=ISC: N&T - NES - Identity and Access Management (IAM)/CN=penngroups-dev.net.isc.upenn.edu i:/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA -- 1 s:/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA i:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority -- 2 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root -- 3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
On 2020-06-02 17:16, phei@isc.upenn.edu wrote:
Not sure if this is an openldap issue but have to examine everything we can.
We revised our nss certificate store as part of addressing the expiration of our root cert.
It now has two certs, the end service cert and the intermediate. Basic client operations (ldapsearch) work fine; using -d1 shows that the appropriate service certificate is loaded and the the search is successful.
But if we run an 'openssl s_client -showcerts' against the host and port 636, we continue to see the expired root certificate even though it's not in the nss store configured chain. This is causing issues for some applications (mainly java based) so we're just trying to understand where the expired root would be coming from if it's not in the openldap server configuration.
To put it mildly, it is unlikely that it is openldap. It is most likely that it will be the underlying SSL support that you need to understand. You don't tell us whether you are using a distribution build, which distribution, or whether you are using GNUTLS or OpenSSL. As I remember with GNUTLS you have to put everything in one file. Just don't put the bad CA cert in the file. With OpenSSL it can be a bit tricky if you are just pointing to the cert directory. You would need to make sure that you purge the cert directory of any bad/expired certs.
A warning about my SSL advice. It has been awhile since I actually tested SSL client connections to OpenLDAP, I use GSSAPI/Kerberos for authentication and connection privacy.
I got bit by this on some of my web servers. I was able to solve it by downloading a new root cert bundle from the CA for my web servers. (https://www.ssls.com/) But, I never could get the new cert chain to work on my webkdc. I think it was because of a limitation of the perl LWP module. But, it also might have been because the expired root CA was in the ca-certificates package on debian/ubuntu and it was not an option to delete that package. (That package has since been patched. If you are on a debian system you might try just installing the ca-certificates update.)
It was never clear what the problem was and the support was abysmal. If its not a web server they just don't have a clue.
In the end I solved my problem by abandoning that CA and just going to https://letsencrypt.org/ and getting a free cert from them. I spend a couple hours on a chat with support and never resolved the issue. I spent about 15 minutes getting and installing a cert from Let's Encrypt which solved my problem.
I would recommend that you drop that CA. They handled this transition extremely poorly.
Bill
This RedHat 6, their openldap-servers rpm is built to use the nss certificate database.
Changing CA provider is not an option due to policy.
In following this article from the openldap faq
https://www.openldap.org/faq/data/cache/1514.html (this section may need updating, as the certutil now requires the secmod and password dbs to be present. So just linking to the shared object file doesn't work, you get the famously unhelpful "SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format error").
I'm able to see the modules; can see the certs in the private but not the generic:
[root@dev-pnldap1 certs]# certutil -d $PWD -U
slot: NSS User Private Key and Certificate Services token: NSS Certificate DB uri: pkcs11:token=NSS%20Certificate%20DB;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203
slot: NSS Internal Cryptographic Services token: NSS Generic Crypto Services uri: pkcs11:token=NSS%20Generic%20Crypto%20Services;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203
[root@dev-pnldap1 certs]# certutil -d $PWD -L -h "NSS Generic Crypto Services"
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
[root@dev-pnldap1 certs]# certutil -d $PWD -L -h "NSS Certificate DB"
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
DirectoryTestLdap u,u,u intermediate-2024 ,,
________________________________ From: Bill MacAllister bill@ca-zephyr.org Sent: Wednesday, June 3, 2020 1:27 AM To: Heinemann, Peter G phei@isc.upenn.edu Cc: openldap-technical@openldap.org openldap-technical@openldap.org Subject: Re: ssl certificate chain
On 2020-06-02 17:16, phei@isc.upenn.edu wrote:
Not sure if this is an openldap issue but have to examine everything we can.
We revised our nss certificate store as part of addressing the expiration of our root cert.
It now has two certs, the end service cert and the intermediate. Basic client operations (ldapsearch) work fine; using -d1 shows that the appropriate service certificate is loaded and the the search is successful.
But if we run an 'openssl s_client -showcerts' against the host and port 636, we continue to see the expired root certificate even though it's not in the nss store configured chain. This is causing issues for some applications (mainly java based) so we're just trying to understand where the expired root would be coming from if it's not in the openldap server configuration.
To put it mildly, it is unlikely that it is openldap. It is most likely that it will be the underlying SSL support that you need to understand. You don't tell us whether you are using a distribution build, which distribution, or whether you are using GNUTLS or OpenSSL. As I remember with GNUTLS you have to put everything in one file. Just don't put the bad CA cert in the file. With OpenSSL it can be a bit tricky if you are just pointing to the cert directory. You would need to make sure that you purge the cert directory of any bad/expired certs.
A warning about my SSL advice. It has been awhile since I actually tested SSL client connections to OpenLDAP, I use GSSAPI/Kerberos for authentication and connection privacy.
I got bit by this on some of my web servers. I was able to solve it by downloading a new root cert bundle from the CA for my web servers. (https://www.ssls.com/) But, I never could get the new cert chain to work on my webkdc. I think it was because of a limitation of the perl LWP module. But, it also might have been because the expired root CA was in the ca-certificates package on debian/ubuntu and it was not an option to delete that package. (That package has since been patched. If you are on a debian system you might try just installing the ca-certificates update.)
It was never clear what the problem was and the support was abysmal. If its not a web server they just don't have a clue.
In the end I solved my problem by abandoning that CA and just going to https://letsencrypt.org/ and getting a free cert from them. I spend a couple hours on a chat with support and never resolved the issue. I spent about 15 minutes getting and installing a cert from Let's Encrypt which solved my problem.
I would recommend that you drop that CA. They handled this transition extremely poorly.
Bill
-- Bill MacAllister bill@ca-zephyr.org
On Wed, Jun 03 2020 at 00:16:31 -0000, phei@isc.upenn.edu scribbled in "ssl certificate chain":
Not sure if this is an openldap issue but have to examine everything we can.
We revised our nss certificate store as part of addressing the expiration of our root cert.
It now has two certs, the end service cert and the intermediate.
Basic client operations (ldapsearch) work fine; using -d1 shows that the appropriate service certificate is loaded and the the search is successful.
But if we run an 'openssl s_client -showcerts' against the host and port 636, we continue to see the expired root certificate even though it's not in the nss store configured chain. This is causing issues for some applications (mainly java based) so we're just trying to understand where the expired root would be coming from if it's not in the openldap server configuration.
Thanks,
Peter
relevant bits: #slapd.conf # TLS/ssl TLSCACertificatePath /etc/openldap/certs TLSCertificateFile DirectoryLdap
#sudo certutil -d /etc/openldap/certs -L
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
intermediate-2024 ,, PennGroupsLdap
Hi,
Firstly a caveat: we don't make use of NSS for any of our directory systems, so I have no experience of using OpenLDAP with NSS.
Given that, the question I have is rather simple, but I wanted to check nonetheless -- is there a typo in the information you've provided above? The TLSCertificateFile option has the value "DirectoryLdap", but by my understanding should, going by the output from `certutil`, possibly rather be configured with a value of "PennGroupsLdap"?
That doesn't really answer the question about where the wrong certificate is appearing from, but I don't know how libnss will behave if given an undefined TLSCertificateFile value to select from.
Cheers.
Dameon.
phei@isc.upenn.edu wrote:
Not sure if this is an openldap issue but have to examine everything we can.
We revised our nss certificate store as part of addressing the expiration of our root cert.
It now has two certs, the end service cert and the intermediate. Basic client operations (ldapsearch) work fine; using -d1 shows that the appropriate service certificate is loaded and the the search is successful.
What is the output from ldapsearch -d -1 ?
That's part of our puzzle. Happy to send more output if it would be helpful.
ldapsearch connects fine:
connect success TLS: certdb config: configDir='/etc/openldap/certs' tokenDescription='ldap(0)' certPrefix='' keyPrefix='' flags=readOnly TLS: using moznss security dir /etc/openldap/certs prefix . TLS: certificate [CN=directory.upenn.edu,OU=ISC: N&T - NES - Identity and Access Management (IAM),O=University of Pennsylvania,STREET=3451 Walnut Street,L=Philadelphia,ST=PA,postalCode=19104,C=US] is valid TLS certificate verification: subject: CN=directory.upenn.edu,OU=ISC: N&T - NES - Identity and Access Management (IAM),O=University of Pennsylvania,STREET=3451 Walnut Street,L=Philadelphia,ST=PA,postalCode=19104,C=US, issuer: CN=InCommon RSA Server CA,OU=InCommon,O=Internet2,L=Ann Arbor,ST=MI,C=US, cipher: AES-256, security level: high, secret key bits: 256, total key bits: 256, cache hits: 0, cache misses: 0, cache not reusable: 0 ldap_open_defconn: successful
even when there's an expired cert in the chain:
head pd-ldap1.certs (from this command: openssl s_client -host pd-ldap1.net.isc.upenn.edu -port 636 -showcerts 2>pd-ldap1.certs >> pd-ldap1.certs)
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root verify error:num=10:certificate has expired notAfter=May 30 10:48:38 2020 GMT verify return:0 DONE CTED(00000003) --- Certificate chain 0 s:/C=US/postalCode=19104/ST=PA/L=Philadelphia/street=3451 Walnut Street/O=University of Pennsylvania/OU=ISC: N&T - NES - Identity and Access Management (IAM)/CN=directory.upenn.edu i:/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA [0 phei@pi-haproxy2 ~]$ head -20 pd-ldap1.certs depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root verify error:num=10:certificate has expired notAfter=May 30 10:48:38 2020 GMT verify return:0 DONE CTED(00000003) ________________________________ From: Howard Chu hyc@symas.com Sent: Wednesday, June 3, 2020 9:43 AM To: Heinemann, Peter G phei@isc.upenn.edu; openldap-technical@openldap.org openldap-technical@openldap.org Subject: Re: ssl certificate chain
phei@isc.upenn.edu wrote:
Not sure if this is an openldap issue but have to examine everything we can.
We revised our nss certificate store as part of addressing the expiration of our root cert.
It now has two certs, the end service cert and the intermediate. Basic client operations (ldapsearch) work fine; using -d1 shows that the appropriate service certificate is loaded and the the search is successful.
What is the output from ldapsearch -d -1 ?
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Heinemann, Peter G wrote:
That's part of our puzzle. Happy to send more output if it would be helpful.
Yes, I wanted to see the entire output with debuglevel set to -1, for the connection establishment and TLS handshake. That includes the hex packet dumps of the network traffic.
The fact that it connects fine even with an expired cert implies a bug in the MozNSS cert validation functions.
ldapsearch connects fine:
connect success TLS: certdb config: configDir='/etc/openldap/certs' tokenDescription='ldap(0)' certPrefix='' keyPrefix='' flags=readOnly TLS: using moznss security dir /etc/openldap/certs prefix . TLS: certificate [CN=directory.upenn.edu,OU=ISC: N&T - NES - Identity and Access Management (IAM),O=University of Pennsylvania,STREET=3451 Walnut Street,L=Philadelphia,ST=PA,postalCode=19104,C=US] is valid TLS certificate verification: subject: CN=directory.upenn.edu,OU=ISC: N&T - NES - Identity and Access Management (IAM),O=University of Pennsylvania,STREET=3451 Walnut Street,L=Philadelphia,ST=PA,postalCode=19104,C=US, issuer: CN=InCommon RSA Server CA,OU=InCommon,O=Internet2,L=Ann Arbor,ST=MI,C=US, cipher: AES-256, security level: high, secret key bits: 256, total key bits: 256, cache hits: 0, cache misses: 0, cache not reusable: 0 ldap_open_defconn: successful
even when there's an expired cert in the chain:
head pd-ldap1.certs (from this command: openssl s_client -host pd-ldap1.net.isc.upenn.edu -port 636 -showcerts 2>pd-ldap1.certs >> pd-ldap1.certs)
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root verify error:num=10:certificate has expired notAfter=May 30 10:48:38 2020 GMT verify return:0 DONE CTED(00000003)
Certificate chain 0 s:/C=US/postalCode=19104/ST=PA/L=Philadelphia/street=3451 Walnut Street/O=University of Pennsylvania/OU=ISC: N&T - NES - Identity and Access Management (IAM)/CN=directory.upenn.edu i:/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA [0 phei@pi-haproxy2 ~]$ head -20 pd-ldap1.certs depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root verify error:num=10:certificate has expired notAfter=May 30 10:48:38 2020 GMT verify return:0 DONE CTED(00000003)
*From:* Howard Chu hyc@symas.com *Sent:* Wednesday, June 3, 2020 9:43 AM *To:* Heinemann, Peter G phei@isc.upenn.edu; openldap-technical@openldap.org openldap-technical@openldap.org *Subject:* Re: ssl certificate chain phei@isc.upenn.edu wrote:
Not sure if this is an openldap issue but have to examine everything we can.
We revised our nss certificate store as part of addressing the expiration of our root cert.
It now has two certs, the end service cert and the intermediate. Basic client operations (ldapsearch) work fine; using -d1 shows that the appropriate service certificate is loaded and the the search is successful.
What is the output from ldapsearch -d -1 ?
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Seems this is https://www.agwa.name/blog/post/fixing_the_addtrust_root_expiration.
Regards, Leonid.
On Wed, Jun 3, 2020 at 7:11 PM Howard Chu hyc@symas.com wrote:
Heinemann, Peter G wrote:
That's part of our puzzle. Happy to send more output if it would be helpful.
Yes, I wanted to see the entire output with debuglevel set to -1, for the connection establishment and TLS handshake. That includes the hex packet dumps of the network traffic.
The fact that it connects fine even with an expired cert implies a bug in the MozNSS cert validation functions.
ldapsearch connects fine:
connect success TLS: certdb config: configDir='/etc/openldap/certs' tokenDescription='ldap(0)' certPrefix='' keyPrefix='' flags=readOnly TLS: using moznss security dir /etc/openldap/certs prefix . TLS: certificate [CN=directory.upenn.edu,OU=ISC: N&T - NES - Identity and Access Management (IAM),O=University of Pennsylvania,STREET=3451 Walnut Street,L=Philadelphia,ST=PA,postalCode=19104,C=US] is valid TLS certificate verification: subject: CN=directory.upenn.edu,OU=ISC: N&T - NES - Identity and Access Management (IAM),O=University of Pennsylvania,STREET=3451 Walnut Street,L=Philadelphia,ST=PA,postalCode=19104,C=US, issuer: CN=InCommon RSA Server CA,OU=InCommon,O=Internet2,L=Ann Arbor,ST=MI,C=US, cipher: AES-256, security level: high, secret key bits: 256, total key bits: 256, cache hits: 0, cache misses: 0, cache not reusable: 0 ldap_open_defconn: successful
even when there's an expired cert in the chain:
head pd-ldap1.certs (from this command: openssl s_client -host pd-ldap1.net.isc.upenn.edu -port 636 -showcerts 2>pd-ldap1.certs >> pd-ldap1.certs)
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root verify error:num=10:certificate has expired notAfter=May 30 10:48:38 2020 GMT verify return:0 DONE CTED(00000003)
Certificate chain 0 s:/C=US/postalCode=19104/ST=PA/L=Philadelphia/street=3451 Walnut Street/O=University of Pennsylvania/OU=ISC: N&T - NES - Identity and Access Management (IAM)/CN=directory.upenn.edu i:/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA [0 phei@pi-haproxy2 ~]$ head -20 pd-ldap1.certs depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root verify error:num=10:certificate has expired notAfter=May 30 10:48:38 2020 GMT verify return:0 DONE CTED(00000003)
*From:* Howard Chu hyc@symas.com *Sent:* Wednesday, June 3, 2020 9:43 AM *To:* Heinemann, Peter G phei@isc.upenn.edu; openldap-technical@openldap.org openldap-technical@openldap.org *Subject:* Re: ssl certificate chain
phei@isc.upenn.edu wrote:
Not sure if this is an openldap issue but have to examine everything we can.
We revised our nss certificate store as part of addressing the expiration of our root cert.
It now has two certs, the end service cert and the intermediate. Basic client operations (ldapsearch) work fine; using -d1 shows that the appropriate service certificate is loaded and the the search is successful.
What is the output from ldapsearch -d -1 ?
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
--On Wednesday, June 3, 2020 8:15 PM +0300 Леонид Юрьев leo@yuriev.ru wrote:
Seems this is https://www.agwa.name/blog/post/fixing_the_addtrust_root_expiration.
Since RHEL6 is in use here, specifically see the linked tweet for Fedora/RHEL in the above post.
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
openldap-technical@openldap.org
-
Bill MacAllister -
Dameon Wagner -
Heinemann, Peter G -
Howard Chu -
phei＠isc.upenn.edu
-
Quanah Gibson-Mount -
Леонид Юрьев