ldap_sasl_interactive_bind_s failed (-2) using openldap v2.3.43 when trying to replicate via syncrepl.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi guys,
I have a configuration that consists of 3 ldap servers. One is the provider and there are 2 consumers. I am using syncrepl to do the synchronization. simple and anonymous binds are totally disabled and Kerberos must be used via SASL (GSSAPI) and TLS to connect to the LDAP server.
distro: centos 5.4 openldap 2.3.43 cyrus-sasl 2.1.22
Other things: - - clocks are all in sync - - hostnames all have forward and reverse mappings and all dns servers in /etc/resolv.conf respond with proper entries on the consumer and both providers.
Here's the catch, the two providers are configured the same (except for hostnames/ips) and the first one works perfect. What is really frustrating is the lack of logging that is available to tell me what the problem is. I've tried loglevel -1 and it gave me even less info in regards to the SASL authentication than leaving it off.
The affected consumer is giving me:
Mar 31 22:41:00 ZZZ slapd[2442]: slapd starting Mar 31 22:41:02 ZZZ slapd[2442]: do_syncrep1: rid 010 ldap_sasl_interactive_bind_s failed (-2) Mar 31 22:41:02 ZZZ slapd[2442]: do_syncrepl: rid 010 quitting
On the "Provider":
Mar 31 17:49:54 XXX slapd[3494]: conn=212 fd=21 ACCEPT from IP=10.130.1.230:60288 (IP=0.0.0.0:389) Mar 31 17:49:54 XXX slapd[3494]: conn=212 op=0 STARTTLS Mar 31 17:49:54 XXX slapd[3494]: conn=212 op=0 RESULT oid= err=0 text= Mar 31 17:49:54 XXX slapd[3494]: conn=212 fd=21 TLS established tls_ssf=256 ssf=256 Mar 31 17:49:54 XXX slapd[3494]: conn=212 op=1 UNBIND Mar 31 17:49:54 XXX slapd[3494]: conn=212 fd=21 closed
This is what's REALLY weird - from the affected/broken box, ZZZ, after I kinit, I can do an LDAP search or ldapwhoami, no problems! So, kerberos and GSSAPI via SASL is working fine. ie:
ldapsearch -H ldaps://XXX/ -Y GSSAPI -> will dump the entries. or ldapwhoami -H ldaps://XXX/ -Y GSSAPI -> shows me that proper creds
If I destroy the credentials, it doesn't work as would be expected.
ON the working consumer, the behaviour is that I can ldapsearch and ldapwhoami properly after I kinit and when I start ldap it will authenticate properly with the provider via SASL GSSAPI and replicates the DB. If I kdestroy the credentials and start it, I get the same error that I'm struggling with on the box that doesn't work ->ldap_sasl_interactive_bind_s failed (-2) This behaviour leads me to believe that for some reason the ldap server on the box that doesn't work is having problems transmitting the kerberos credentials to the provider, whereas the ldapsearch and ldapwhoami binaries are not having problems.
There are some suspicious differences between the consumer that works and the broken one. The provider and consumer that works both have TLDs that match - '.com' and the consumer whose synrepl process won't authenticate is part of the .eu TLD. However, as you can see below in the krb5.conf files, the .com and .eu TLDS are always mapped to the same authentication realm. PLUS, again, ldapsearch and ldapwhoami WORK. It's just the syncrepl process that isn't quite getting the auth right.
This is the provider's pertinent configs:
slapd.conf: overlay syncprov syncprov-checkpoint 100 10 syncprov-sessionlog 100
This is the consumer's pertinent configs (WORKS ON one, not on the other) slapd.conf: syncrepl rid=10 provider=ldap://xxx.XXX.com starttls=yes type=refreshOnly interval=00:00:01:00 searchbase="dc=XXX,dc=com" schemachecking=off bindmethod=sasl saslmech=GSSAPI
krb5.conf [same as provider and kerb server]: [libdefaults] default_realm = BOUNCE.AAA.COM encrypt = true allow_weak_crypto = false clockskew = 600 dns_lookup_realm = false dns_lookup_kdc = false ticket_lifetime = 8h forwardable = no proxiable = no
[realms] BOUNCE.AAA.COM = { kdc = XXX.com kdc = YYY.com kdc = ZZZ.eu admin_server = XXX.com }
[domain_realm] .com = BOUNCE.AAA.COM .eu = BOUNCE.AAA.COM
All help is greatly appreciated! This has been going on for days and I've already yanked out most of my hair. Thank you.
Kris.
PGP Key: 4CC63A18 PGP Server: pool.sks-keyservers.net
I should provide one more piece of information that might be important. I just noticed in /var/log/messages:
slapd: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Unknown code krb5 194)
Looking that error code up @ http://www.lineox.net/lineox/docs/HTML/info2html/krb5-admin/krb5-admin.info.... yeilds:
194. KRB5_FCC_PERM: Credentials cache file permissions incorrect
However, the credential cache file has the appropriate perms. LDAP runs as user root. So, it should have no problems reading the /etc/krb5.keytab and the credential cache file under /tmp
ls -la /tmp/krb5cc_0 -rw------- 1 root root 569 Mar 31 18:11 krb5cc_0
Thanks again!
Kris.
PGP Key: 4CC63A18 PGP Server: pool.sks-keyservers.net
On 2010-03-31, at 6:20 PM, Kristian Kostecky wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi guys,
I have a configuration that consists of 3 ldap servers. One is the provider and there are 2 consumers. I am using syncrepl to do the synchronization. simple and anonymous binds are totally disabled and Kerberos must be used via SASL (GSSAPI) and TLS to connect to the LDAP server.
distro: centos 5.4 openldap 2.3.43 cyrus-sasl 2.1.22
Other things:
- clocks are all in sync
- hostnames all have forward and reverse mappings and all dns servers in /etc/resolv.conf respond with proper entries on the consumer and both providers.
Here's the catch, the two providers are configured the same (except for hostnames/ips) and the first one works perfect. What is really frustrating is the lack of logging that is available to tell me what the problem is. I've tried loglevel -1 and it gave me even less info in regards to the SASL authentication than leaving it off.
The affected consumer is giving me:
Mar 31 22:41:00 ZZZ slapd[2442]: slapd starting Mar 31 22:41:02 ZZZ slapd[2442]: do_syncrep1: rid 010 ldap_sasl_interactive_bind_s failed (-2) Mar 31 22:41:02 ZZZ slapd[2442]: do_syncrepl: rid 010 quitting
On the "Provider":
Mar 31 17:49:54 XXX slapd[3494]: conn=212 fd=21 ACCEPT from IP=10.130.1.230:60288 (IP=0.0.0.0:389) Mar 31 17:49:54 XXX slapd[3494]: conn=212 op=0 STARTTLS Mar 31 17:49:54 XXX slapd[3494]: conn=212 op=0 RESULT oid= err=0 text= Mar 31 17:49:54 XXX slapd[3494]: conn=212 fd=21 TLS established tls_ssf=256 ssf=256 Mar 31 17:49:54 XXX slapd[3494]: conn=212 op=1 UNBIND Mar 31 17:49:54 XXX slapd[3494]: conn=212 fd=21 closed
This is what's REALLY weird - from the affected/broken box, ZZZ, after I kinit, I can do an LDAP search or ldapwhoami, no problems! So, kerberos and GSSAPI via SASL is working fine. ie:
ldapsearch -H ldaps://XXX/ -Y GSSAPI -> will dump the entries. or ldapwhoami -H ldaps://XXX/ -Y GSSAPI -> shows me that proper creds
If I destroy the credentials, it doesn't work as would be expected.
ON the working consumer, the behaviour is that I can ldapsearch and ldapwhoami properly after I kinit and when I start ldap it will authenticate properly with the provider via SASL GSSAPI and replicates the DB. If I kdestroy the credentials and start it, I get the same error that I'm struggling with on the box that doesn't work ->ldap_sasl_interactive_bind_s failed (-2) This behaviour leads me to believe that for some reason the ldap server on the box that doesn't work is having problems transmitting the kerberos credentials to the provider, whereas the ldapsearch and ldapwhoami binaries are not having problems.
There are some suspicious differences between the consumer that works and the broken one. The provider and consumer that works both have TLDs that match - '.com' and the consumer whose synrepl process won't authenticate is part of the .eu TLD. However, as you can see below in the krb5.conf files, the .com and .eu TLDS are always mapped to the same authentication realm. PLUS, again, ldapsearch and ldapwhoami WORK. It's just the syncrepl process that isn't quite getting the auth right.
This is the provider's pertinent configs:
slapd.conf: overlay syncprov syncprov-checkpoint 100 10 syncprov-sessionlog 100
This is the consumer's pertinent configs (WORKS ON one, not on the other) slapd.conf: syncrepl rid=10 provider=ldap://xxx.XXX.com starttls=yes type=refreshOnly interval=00:00:01:00 searchbase="dc=XXX,dc=com" schemachecking=off bindmethod=sasl saslmech=GSSAPI
krb5.conf [same as provider and kerb server]: [libdefaults] default_realm = BOUNCE.AAA.COM encrypt = true allow_weak_crypto = false clockskew = 600 dns_lookup_realm = false dns_lookup_kdc = false ticket_lifetime = 8h forwardable = no proxiable = no
[realms] BOUNCE.AAA.COM = { kdc = XXX.com kdc = YYY.com kdc = ZZZ.eu admin_server = XXX.com }
[domain_realm] .com = BOUNCE.AAA.COM .eu = BOUNCE.AAA.COM
All help is greatly appreciated! This has been going on for days and I've already yanked out most of my hair. Thank you.
Kris.
PGP Key: 4CC63A18 PGP Server: pool.sks-keyservers.net
-----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
iEYEARECAAYFAkuzyroACgkQ2C/J5/UUQWEuUACdH/BhiZgTXFWbNMXS7Q99k8Rg VY8An3YWKcpnkxVYvZMlelkT0TIpYuAP =O9KI -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
My apologies. Problem solved.
I straced the slapd process and noticed that there were all sorts of SELINUX policies preventing the process from reading /tmp. I've enabled /tmp access and all works now. Thanks!
Kris.
PGP Key: 4CC63A18 PGP Server: pool.sks-keyservers.net
On 2010-03-31, at 6:20 PM, Kristian Kostecky wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi guys,
I have a configuration that consists of 3 ldap servers. One is the provider and there are 2 consumers. I am using syncrepl to do the synchronization. simple and anonymous binds are totally disabled and Kerberos must be used via SASL (GSSAPI) and TLS to connect to the LDAP server.
distro: centos 5.4 openldap 2.3.43 cyrus-sasl 2.1.22
Other things:
- clocks are all in sync
- hostnames all have forward and reverse mappings and all dns servers in /etc/resolv.conf respond with proper entries on the consumer and both providers.
Here's the catch, the two providers are configured the same (except for hostnames/ips) and the first one works perfect. What is really frustrating is the lack of logging that is available to tell me what the problem is. I've tried loglevel -1 and it gave me even less info in regards to the SASL authentication than leaving it off.
The affected consumer is giving me:
Mar 31 22:41:00 ZZZ slapd[2442]: slapd starting Mar 31 22:41:02 ZZZ slapd[2442]: do_syncrep1: rid 010 ldap_sasl_interactive_bind_s failed (-2) Mar 31 22:41:02 ZZZ slapd[2442]: do_syncrepl: rid 010 quitting
On the "Provider":
Mar 31 17:49:54 XXX slapd[3494]: conn=212 fd=21 ACCEPT from IP=10.130.1.230:60288 (IP=0.0.0.0:389) Mar 31 17:49:54 XXX slapd[3494]: conn=212 op=0 STARTTLS Mar 31 17:49:54 XXX slapd[3494]: conn=212 op=0 RESULT oid= err=0 text= Mar 31 17:49:54 XXX slapd[3494]: conn=212 fd=21 TLS established tls_ssf=256 ssf=256 Mar 31 17:49:54 XXX slapd[3494]: conn=212 op=1 UNBIND Mar 31 17:49:54 XXX slapd[3494]: conn=212 fd=21 closed
This is what's REALLY weird - from the affected/broken box, ZZZ, after I kinit, I can do an LDAP search or ldapwhoami, no problems! So, kerberos and GSSAPI via SASL is working fine. ie:
ldapsearch -H ldaps://XXX/ -Y GSSAPI -> will dump the entries. or ldapwhoami -H ldaps://XXX/ -Y GSSAPI -> shows me that proper creds
If I destroy the credentials, it doesn't work as would be expected.
ON the working consumer, the behaviour is that I can ldapsearch and ldapwhoami properly after I kinit and when I start ldap it will authenticate properly with the provider via SASL GSSAPI and replicates the DB. If I kdestroy the credentials and start it, I get the same error that I'm struggling with on the box that doesn't work ->ldap_sasl_interactive_bind_s failed (-2) This behaviour leads me to believe that for some reason the ldap server on the box that doesn't work is having problems transmitting the kerberos credentials to the provider, whereas the ldapsearch and ldapwhoami binaries are not having problems.
There are some suspicious differences between the consumer that works and the broken one. The provider and consumer that works both have TLDs that match - '.com' and the consumer whose synrepl process won't authenticate is part of the .eu TLD. However, as you can see below in the krb5.conf files, the .com and .eu TLDS are always mapped to the same authentication realm. PLUS, again, ldapsearch and ldapwhoami WORK. It's just the syncrepl process that isn't quite getting the auth right.
This is the provider's pertinent configs:
slapd.conf: overlay syncprov syncprov-checkpoint 100 10 syncprov-sessionlog 100
This is the consumer's pertinent configs (WORKS ON one, not on the other) slapd.conf: syncrepl rid=10 provider=ldap://xxx.XXX.com starttls=yes type=refreshOnly interval=00:00:01:00 searchbase="dc=XXX,dc=com" schemachecking=off bindmethod=sasl saslmech=GSSAPI
krb5.conf [same as provider and kerb server]: [libdefaults] default_realm = BOUNCE.AAA.COM encrypt = true allow_weak_crypto = false clockskew = 600 dns_lookup_realm = false dns_lookup_kdc = false ticket_lifetime = 8h forwardable = no proxiable = no
[realms] BOUNCE.AAA.COM = { kdc = XXX.com kdc = YYY.com kdc = ZZZ.eu admin_server = XXX.com }
[domain_realm] .com = BOUNCE.AAA.COM .eu = BOUNCE.AAA.COM
All help is greatly appreciated! This has been going on for days and I've already yanked out most of my hair. Thank you.
Kris.
PGP Key: 4CC63A18 PGP Server: pool.sks-keyservers.net
-----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
iEYEARECAAYFAkuzyroACgkQ2C/J5/UUQWEuUACdH/BhiZgTXFWbNMXS7Q99k8Rg VY8An3YWKcpnkxVYvZMlelkT0TIpYuAP =O9KI -----END PGP SIGNATURE-----
openldap-technical@openldap.org
-
Kristian Kostecky