On 1/16/20 8:05 PM, Quanah Gibson-Mount wrote:
--On Thursday, January 16, 2020 9:03 PM +0000 Prentice Bisbal pbisbal@princeton.edu wrote:
One of my coworkers just noticed that replication is broken between our primary and secondary LDAP servers. It appears to have been broken for about 1 week now. Nothing has changed relative to the LDAP configuration on either of our servers, so this is an odd thing to suddenly happen. When I look at the consumer with some debugging on, I see these messages (/usr/sbin/slapd -d 1638 was used to get these messages):
It looks like the consumer host/voltron-b.pppl.gov,cn=pppl.gov,cn=gssapi,cn=auth,is being rejected as not being authorized, but this has been working for years w/o issue. Any idea what has changed and how I may fix it?
Well, the error came from cyrus-sasl rather than OpenLDAP. This would indicate to me that the not authorized came from the KDC. Have you checked to ensure the keys in the keytab file haven't expired inside the KDC?
That's exactly what I suspected. We're using AD for our Kerberos Client, and one of our AD admins insists that it couldn't be expired credentials. I did use a utility called msktutil to make sure the kerberos tickets in /etc/krb5.keytab were up to date, but I'm still getting that error. Any ideas on how to prove/disprove what you suggest, so I can go back to my AD admins with more information?
-- Prentice
--On Friday, January 17, 2020 1:12 PM -0500 Prentice Bisbal pbisbal@pppl.gov wrote:
Well, the error came from cyrus-sasl rather than OpenLDAP. This would indicate to me that the not authorized came from the KDC. Have you checked to ensure the keys in the keytab file haven't expired inside the KDC?
That's exactly what I suspected. We're using AD for our Kerberos Client, and one of our AD admins insists that it couldn't be expired credentials. I did use a utility called msktutil to make sure the kerberos tickets in /etc/krb5.keytab were up to date, but I'm still getting that error. Any ideas on how to prove/disprove what you suggest, so I can go back to my AD admins with more information?
Hi Prentice,
Unfortunately I have no experience using AD as a KDC. So I can't really offer any further debugging advice.
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
On 2020-01-17 10:12, Prentice Bisbal wrote:
On 1/16/20 8:05 PM, Quanah Gibson-Mount wrote: That's exactly what I suspected. We're using AD for our Kerberos Client, and one of our AD admins insists that it couldn't be expired credentials. I did use a utility called msktutil to make sure the kerberos tickets in /etc/krb5.keytab were up to date, but I'm still getting that error. Any ideas on how to prove/disprove what you suggest, so I can go back to my AD admins with more information?
The tools that I use to investigate problems like this are klist, k5start, ldapsearch, and ldapwhoami. I usually start by looking at the keytab itself with klist just to make sure that the keytab itself is not damaged.
Next I make sure I can generate a ticket cache.
k5start -f /etc/krb5.keytab host/<fqdn> -- klist
Next I use the keytab for authentication to a simple service. For us this is a remctl help command.
k5start -f /etc/krb5.keytab host/<fqdn> -- remctl <host> help
If this fails there is no point in trying to authenticate to the LDAP server. Do the normal checks for Kerberos failures, i.e. forward and reverse DNS entries for hosts involved, system clocks, and krb5.conf.
If the keytab appears to work for the simple service I would proceed to testing searches against AD.
k5start -f /etc/krb5.keytab host/<fqdn> -- ldapsearch -h <ad> -b <basedn> cn=<hostid>
The <hostid> might and might not be the same as the principal that you specified when the keytab was created. In AD the <hostid> is limited to 20 characters. When we generate a keytab we define the ensure that the <hostid> does not exceed this limit by truncated as needed and adding a sequence number to ensure uniqueness. The <hostid> is probably in the keytab file.
I suppose that the ldapsearch agains <ad> will be affected by the ACLs implemented there. If this fails you will need to talk to the AD admin it figure that out. I got our AD administrators to create a separate OU for our keytabs and give us full access to that OU.
You really do want to see what the <ad> entry for the keytab is to ensure that the entry is self consistent. The critical attributes are userPrincipalName, dNSHostName, and servicePrincipalName. Make sure these are consistent. In our case if we see a problem here we just delete the entry and re-create it. We usually use wallet developed at Stanford for managing keytabs, but when we hit this type of problem we use ldapdelete to remove the offending entry.
If the <ad> ldapsearch works then I would proceed to trying an authentication against the OpenLDAP server using ldapwhoami.
k5start -f /etc/krb5.keytab host/<fqdn> -- ldapwhoami -h <ol>
In addition to any error messages look at the LDAP log entries. Specifically look at the authz entries to make sure that the principal to dn mapping is what you expect.
If you get stuck let me know what you are seeing.
Bill
On 1/21/20 2:58 PM, Bill MacAllister wrote:
On 2020-01-17 10:12, Prentice Bisbal wrote:
On 1/16/20 8:05 PM, Quanah Gibson-Mount wrote: That's exactly what I suspected. We're using AD for our Kerberos Client, and one of our AD admins insists that it couldn't be expired credentials. I did use a utility called msktutil to make sure the kerberos tickets in /etc/krb5.keytab were up to date, but I'm still getting that error. Any ideas on how to prove/disprove what you suggest, so I can go back to my AD admins with more information?
The tools that I use to investigate problems like this are klist, k5start, ldapsearch, and ldapwhoami. I usually start by looking at the keytab itself with klist just to make sure that the keytab itself is not damaged.
Next I make sure I can generate a ticket cache.
k5start -f /etc/krb5.keytab host/<fqdn> -- klist
Next I use the keytab for authentication to a simple service. For us this is a remctl help command.
k5start -f /etc/krb5.keytab host/<fqdn> -- remctl <host> help
If this fails there is no point in trying to authenticate to the LDAP server. Do the normal checks for Kerberos failures, i.e. forward and reverse DNS entries for hosts involved, system clocks, and krb5.conf.
If the keytab appears to work for the simple service I would proceed to testing searches against AD.
k5start -f /etc/krb5.keytab host/<fqdn> -- ldapsearch -h <ad> -b <basedn> cn=<hostid>
The <hostid> might and might not be the same as the principal that you specified when the keytab was created. In AD the <hostid> is limited to 20 characters. When we generate a keytab we define the ensure that the <hostid> does not exceed this limit by truncated as needed and adding a sequence number to ensure uniqueness. The <hostid> is probably in the keytab file.
I suppose that the ldapsearch agains <ad> will be affected by the ACLs implemented there. If this fails you will need to talk to the AD admin it figure that out. I got our AD administrators to create a separate OU for our keytabs and give us full access to that OU.
You really do want to see what the <ad> entry for the keytab is to ensure that the entry is self consistent. The critical attributes are userPrincipalName, dNSHostName, and servicePrincipalName. Make sure these are consistent. In our case if we see a problem here we just delete the entry and re-create it. We usually use wallet developed at Stanford for managing keytabs, but when we hit this type of problem we use ldapdelete to remove the offending entry.
If the <ad> ldapsearch works then I would proceed to trying an authentication against the OpenLDAP server using ldapwhoami.
k5start -f /etc/krb5.keytab host/<fqdn> -- ldapwhoami -h <ol>
In addition to any error messages look at the LDAP log entries. Specifically look at the authz entries to make sure that the principal to dn mapping is what you expect.
If you get stuck let me know what you are seeing.
Bill
Thanks for the tips/assistance. I'm going to give these commands a try.
-- Prentice
Bill,
See comments in-line.
On 1/21/20 2:58 PM, Bill MacAllister wrote:
On 2020-01-17 10:12, Prentice Bisbal wrote:
On 1/16/20 8:05 PM, Quanah Gibson-Mount wrote: That's exactly what I suspected. We're using AD for our Kerberos Client, and one of our AD admins insists that it couldn't be expired credentials. I did use a utility called msktutil to make sure the kerberos tickets in /etc/krb5.keytab were up to date, but I'm still getting that error. Any ideas on how to prove/disprove what you suggest, so I can go back to my AD admins with more information?
The tools that I use to investigate problems like this are klist, k5start, ldapsearch, and ldapwhoami. I usually start by looking at the keytab itself with klist just to make sure that the keytab itself is not damaged.
Next I make sure I can generate a ticket cache.
k5start -f /etc/krb5.keytab host/<fqdn> -- klist
This works.
Next I use the keytab for authentication to a simple service. For us this is a remctl help command.
k5start -f /etc/krb5.keytab host/<fqdn> -- remctl <host> help
I used ldapsearch for my command, and it worked without using -x for plain authentication, indicating kerberos auth was working
If this fails there is no point in trying to authenticate to the LDAP server. Do the normal checks for Kerberos failures, i.e. forward and reverse DNS entries for hosts involved, system clocks, and krb5.conf.
If the keytab appears to work for the simple service I would proceed to testing searches against AD.
k5start -f /etc/krb5.keytab host/<fqdn> -- ldapsearch -h <ad> -b <basedn> cn=<hostid>
Searches worked (no authentication issues, and the server returned referrals), but I don't know the proper basedn or cn to use in this case, so...
The <hostid> might and might not be the same as the principal that you specified when the keytab was created. In AD the <hostid> is limited to 20 characters. When we generate a keytab we define the ensure that the <hostid> does not exceed this limit by truncated as needed and adding a sequence number to ensure uniqueness. The <hostid> is probably in the keytab file.
I suppose that the ldapsearch agains <ad> will be affected by the ACLs implemented there. If this fails you will need to talk to the AD admin it figure that out. I got our AD administrators to create a separate OU for our keytabs and give us full access to that OU.
You really do want to see what the <ad> entry for the keytab is to ensure that the entry is self consistent. The critical attributes are userPrincipalName, dNSHostName, and servicePrincipalName. Make sure these are consistent. In our case if we see a problem here we just delete the entry and re-create it. We usually use wallet developed at Stanford for managing keytabs, but when we hit this type of problem we use ldapdelete to remove the offending entry.
If the <ad> ldapsearch works then I would proceed to trying an authentication against the OpenLDAP server using ldapwhoami.
k5start -f /etc/krb5.keytab host/<fqdn> -- ldapwhoami -h <ol>
This worked, too.
In addition to any error messages look at the LDAP log entries. Specifically look at the authz entries to make sure that the principal to dn mapping is what you expect.
If you get stuck let me know what you are seeing. Bill
It looks like Kerberos itself is working, but slapd is not getting a valid TGT when starting up. Could it be something wrong with the way slad is starting?
On 2020-01-21 13:14, Prentice Bisbal wrote:
It looks like Kerberos itself is working, but slapd is not getting a valid TGT when starting up. Could it be something wrong with the way slad is starting?
How slapd gets started really depends on the OS. I use Debian and setting the ticket cache pointer is done in /etc/default/slapd. The ticket cache must be maintained by an external process. We use k5start for that.
Oh, and I forgot. If you are using Ubuntu apparmor is enable by default. You might what to put it in complain mode to figure out what is going on.
aa-complain /usr/sbin/slapd
apparmor by default is coming to debian as well in the next release I believe.
Bill
openldap-technical@openldap.org
-
Bill MacAllister -
Prentice Bisbal -
Quanah Gibson-Mount