I've been noticing various data discrepancies between our LDAP master and LDAP consumers. We are running OpenLDAP v2.4.44. We have two masters running "mirromode TRUE" and all updates go through a VIP that points to the first one unless it's not available (doesn't happen very often except for during patches and restarts). We have 13 consumers that replicate through that same VIP.
Here's an example of our syncrepl for a client:
syncrepl rid=221 type=refreshAndPersist schemachecking=on provider="ldap://ldapmastervip.rutgers.edu/" bindmethod=sasl saslmech=EXTERNAL starttls=yes tls_reqcert=demand tls_protocol_min="3.1" searchbase="dc=rutgers,dc=edu" attrs="*,+" retry="10 10 20 +" logbase="cn=accesslog" logfilter="(&(objectClass=auditWriteObject)(reqResult=0))" syncdata=accesslog network-timeout=30 keepalive=180:3:60
I check the contextCSN attributes on all the instances every day and they are all in sync (except during any major changes, of course). But I occasionally notice discrepancies in the data.... even though the contextCSNs and entryCSNs are the same. For example (note hostnames have been changed):
$ ldapsearch ... -H ldap://ldapmaster.rutgers.edu uid=XXXX postalAddress createTimestamp modifyTimestamp entryCSN dn: uid=XXXX,ou=People,dc=rutgers,dc=edu createTimestamp: 20121220100700Z postalAddress: Business And Science Bldg$227 Penn Street$Camden, NJ 081021656 entryCSN: 20180505002024.083133Z#000000#001#000000 modifyTimestamp: 20180505002024Z
$ ldapsearch ... -H ldap://ldapconsumer3.rutgers.edu uid=XXXX postalAddress createTimestamp modifyTimestamp entryCSN dn: uid=XXXX,ou=People,dc=rutgers,dc=edu createTimestamp: 20121220100700Z postalAddress: BUSINESS AND SCIENCE BLDG$227 PENN STREET$CAMDEN, NJ 081021656 entryCSN: 20180505002024.083133Z#000000#001#000000 modifyTimestamp: 20180505002024Z
So I'm trying to figure out why this happens (config issue, bug, ???) and second, if I can't use the contextCSN to report that everything is fine, what else can I do besides trying to compare ldif dumps.
thanks, ds
On Sep 5, 2018, at 16:49, Dave Steiner steiner@rutgers.edu wrote:
But I occasionally notice discrepancies in the data.... even though the contextCSNs and entryCSNs are the same. For example (note hostnames have been changed):
$ ldapsearch ... -H ldap://ldapmaster.rutgers.edu uid=XXXX postalAddress createTimestamp modifyTimestamp entryCSN postalAddress: Business And Science Bldg$227 Penn Street$Camden, NJ 081021656
$ ldapsearch ... -H ldap://ldapconsumer3.rutgers.edu uid=XXXX postalAddress createTimestamp modifyTimestamp entryCSN dn: uid=XXXX,ou=People,dc=rutgers,dc=edu postalAddress: BUSINESS AND SCIENCE BLDG$227 PENN STREET$CAMDEN, NJ 081021656
You do realize that as far as LDAP is concerned, those two postalAddress values are identical. The matching rules for postalAddress are case insensitive.
Assuming that the mixed case one is what you prefer and the upper case one is what you started with - one has to ask, how was it changed on the one server and make certain that it was not done in a way that would not be sent through the syncrepl process.
- Frank
--On Wednesday, September 05, 2018 5:49 PM -0400 Dave Steiner steiner@rutgers.edu wrote:
Hi Dave,
$ ldapsearch ... -H ldap://ldapmaster.rutgers.edu uid=XXXX postalAddress createTimestamp modifyTimestamp entryCSN dn: uid=XXXX,ou=People,dc=rutgers,dc=edu createTimestamp: 20121220100700Z postalAddress: Business And Science Bldg$227 Penn Street$Camden, NJ 081021656 entryCSN: 20180505002024.083133Z#000000#001#000000 modifyTimestamp: 20180505002024Z
$ ldapsearch ... -H ldap://ldapconsumer3.rutgers.edu uid=XXXX postalAddress createTimestamp modifyTimestamp entryCSN dn: uid=XXXX,ou=People,dc=rutgers,dc=edu createTimestamp: 20121220100700Z postalAddress: BUSINESS AND SCIENCE BLDG$227 PENN STREET$CAMDEN, NJ 081021656 entryCSN: 20180505002024.083133Z#000000#001#000000 modifyTimestamp: 20180505002024Z
Those entries appear identical to me. I assume you believe they are not identical because the case doesn't match. However, postalAddress uses caseIgnore normalization rules, so they are in fact identical as far as LDAP is concerned.
Also, given the entry was created in 2012 and there's no indication of when the postalAddress attribute was modified, it's impossible to determine when/how the divergence in capitalization occurred. You also don't provide any information about the underlying database in use, which could be very relevant, history wise.
So I'm trying to figure out why this happens (config issue, bug, ???) and second, if I can't use the contextCSN to report that everything is fine, what else can I do besides trying to compare ldif dumps.
So far, everything appears fine from a technical standpoint (the values match per LDAP SYNTAX rules).
I would suggest the following if you would like the case to match:
dn: ... changetype: modify replace: postalAddress postalAddress: ...
That should correct the value on the providers and consumers to match case.
Warm regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
On 9/6/2018 2:48 PM, Quanah Gibson-Mount wrote:
--On Wednesday, September 05, 2018 5:49 PM -0400 Dave Steiner steiner@rutgers.edu wrote:
Hi Dave,
$ ldapsearch ... -H ldap://ldapmaster.rutgers.edu uid=XXXX postalAddress createTimestamp modifyTimestamp entryCSN dn: uid=XXXX,ou=People,dc=rutgers,dc=edu createTimestamp: 20121220100700Z postalAddress: Business And Science Bldg$227 Penn Street$Camden, NJ 081021656 entryCSN: 20180505002024.083133Z#000000#001#000000 modifyTimestamp: 20180505002024Z
$ ldapsearch ... -H ldap://ldapconsumer3.rutgers.edu uid=XXXX postalAddress createTimestamp modifyTimestamp entryCSN dn: uid=XXXX,ou=People,dc=rutgers,dc=edu createTimestamp: 20121220100700Z postalAddress: BUSINESS AND SCIENCE BLDG$227 PENN STREET$CAMDEN, NJ 081021656 entryCSN: 20180505002024.083133Z#000000#001#000000 modifyTimestamp: 20180505002024Z
Those entries appear identical to me. I assume you believe they are not identical because the case doesn't match. However, postalAddress uses caseIgnore normalization rules, so they are in fact identical as far as LDAP is concerned.
Also, given the entry was created in 2012 and there's no indication of when the postalAddress attribute was modified, it's impossible to determine when/how the divergence in capitalization occurred. You also don't provide any information about the underlying database in use, which could be very relevant, history wise.
Ok, bad example... Here's another example that's rather different:
$ searchn -H ldap://ldapmaster.rutgers.edu uid=XXXX mail createTimestamp modifyTimestamp entryCSN dn: uid=XXXX,ou=People,dc=rutgers,dc=edu createTimestamp: 20180504140010Z mail: xxxxxxxxx.xxxx@rutgers.edu entryCSN: 20180831205041.549496Z#000000#001#000000 modifyTimestamp: 20180831205041Z
$ searchn -H ldap://ldapconsumer1.rutgers.edu uid=XXXX mail createTimestamp modifyTimestamp entryCSN dn: uid=XXXX,ou=People,dc=rutgers,dc=edu createTimestamp: 20180504140010Z mail: yyyyy@rutgers.edu entryCSN: 20180831205041.549496Z#000000#001#000000 modifyTimestamp: 20180831205041Z
So I'm trying to figure out why this happens (config issue, bug, ???) and second, if I can't use the contextCSN to report that everything is fine, what else can I do besides trying to compare ldif dumps.
So far, everything appears fine from a technical standpoint (the values match per LDAP SYNTAX rules).
I would suggest the following if you would like the case to match:
dn: ... changetype: modify replace: postalAddress postalAddress: ...
That should correct the value on the providers and consumers to match case.
Warm regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.symas.com&data=02%7C01%7Csteiner%40oit.rutgers.edu%7Ca13f8a75c55b4fdbdb6308d614294c60%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C636718564940835797&sdata=pVe%2FLnPDf4Civ13qlr1pXPCb0icy%2BzTZUEAcnViaTCo%3D&reserved=0
--On Thursday, September 06, 2018 5:43 PM -0400 Dave Steiner steiner@rutgers.edu wrote:
Ok, bad example... Here's another example that's rather different:
$ searchn -H ldap://ldapmaster.rutgers.edu uid=XXXX mail createTimestamp modifyTimestamp entryCSN dn: uid=XXXX,ou=People,dc=rutgers,dc=edu createTimestamp: 20180504140010Z mail: xxxxxxxxx.xxxx@rutgers.edu entryCSN: 20180831205041.549496Z#000000#001#000000 modifyTimestamp: 20180831205041Z
$ searchn -H ldap://ldapconsumer1.rutgers.edu uid=XXXX mail createTimestamp modifyTimestamp entryCSN dn: uid=XXXX,ou=People,dc=rutgers,dc=edu createTimestamp: 20180504140010Z mail: yyyyy@rutgers.edu entryCSN: 20180831205041.549496Z#000000#001#000000 modifyTimestamp: 20180831205041Z
Hi Dave,
That's certainly a more interesting case. Do you know when the discrepancy arose? Clearly the entry was modified on 8/31, but you'd have to parse the accesslog to see what the actual change was that was made to the entry. Generally to track down replication related issues, there are a number of various bits that need to be examined in the logs (provider and consumer) along with the data in the accesslog. One would also need to have knowledge on the version of OpenLDAP that's in use since it may be subject to bugs that could cause such issues (ITS#8444 for example, fixed in 2.4.46).
Warm regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
On 9/7/2018 12:59 PM, Quanah Gibson-Mount wrote:
--On Thursday, September 06, 2018 5:43 PM -0400 Dave Steiner steiner@rutgers.edu wrote:
Ok, bad example... Here's another example that's rather different:
$ searchn -H ldap://ldapmaster.rutgers.edu uid=XXXX mail createTimestamp modifyTimestamp entryCSN dn: uid=XXXX,ou=People,dc=rutgers,dc=edu createTimestamp: 20180504140010Z mail: xxxxxxxxx.xxxx@rutgers.edu entryCSN: 20180831205041.549496Z#000000#001#000000 modifyTimestamp: 20180831205041Z
$ searchn -H ldap://ldapconsumer1.rutgers.edu uid=XXXX mail createTimestamp modifyTimestamp entryCSN dn: uid=XXXX,ou=People,dc=rutgers,dc=edu createTimestamp: 20180504140010Z mail: yyyyy@rutgers.edu entryCSN: 20180831205041.549496Z#000000#001#000000 modifyTimestamp: 20180831205041Z
Hi Dave,
That's certainly a more interesting case. Do you know when the discrepancy arose? Clearly the entry was modified on 8/31, but you'd have to parse the accesslog to see what the actual change was that was made to the entry. Generally to track down replication related issues, there are a number of various bits that need to be examined in the logs (provider and consumer) along with the data in the accesslog. One would also need to have knowledge on the version of OpenLDAP that's in use since it may be subject to bugs that could cause such issues (ITS#8444 for example, fixed in 2.4.46).
Warm regards, Quanah
Hi Quanah,
We are running 2.4.44 so there maybe something fixed between that and the latest version....
So I've look at the ldap_audit.log and this person initially set their mail attribute at 20180813215911Z and then changed it to a more readable version at 20180813220044Z. This makes sense given our account activation flow for new students. So it seems that the second change didn't get out to the particular consumer. Unfortunately this is long enough ago that it won't be in the accesslog or in the ldap_access.log files on the consumers. I'll see if I can find a more recent case.
I checked all 13 of our consumers and while the modifyTimestamp and entryCSN are all consistent with the master(s), the mail attribute was not updated on 3 of the consumers. It's correct on the other 10.
Thanks for all your help, ds
Dave Steiner steiner@rutgers.edu schrieb am 07.09.2018 um 21:10 in Nachricht
44b66794-4e1b-8de1-57f3-55f280c85358@oit.rutgers.edu:
On 9/7/2018 12:59 PM, Quanah Gibson-Mount wrote:
--On Thursday, September 06, 2018 5:43 PM -0400 Dave Steiner steiner@rutgers.edu wrote:
Ok, bad example... Here's another example that's rather different:
$ searchn -H ldap://ldapmaster.rutgers.edu uid=XXXX mail createTimestamp modifyTimestamp entryCSN dn: uid=XXXX,ou=People,dc=rutgers,dc=edu createTimestamp: 20180504140010Z mail: xxxxxxxxx.xxxx@rutgers.edu entryCSN: 20180831205041.549496Z#000000#001#000000 modifyTimestamp: 20180831205041Z
$ searchn -H ldap://ldapconsumer1.rutgers.edu uid=XXXX mail createTimestamp modifyTimestamp entryCSN dn: uid=XXXX,ou=People,dc=rutgers,dc=edu createTimestamp: 20180504140010Z mail: yyyyy@rutgers.edu entryCSN: 20180831205041.549496Z#000000#001#000000 modifyTimestamp: 20180831205041Z
Hi Dave,
That's certainly a more interesting case. Do you know when the discrepancy arose? Clearly the entry was modified on 8/31, but you'd have to parse the accesslog to see what the actual change was that was made to the entry. Generally to track down replication related issues, there are a number of various bits that need to be examined in the logs (provider and consumer) along with the data in the accesslog. One would also need to have knowledge
on the version of OpenLDAP that's in use since it may be subject to bugs
that
could cause such issues (ITS#8444 for example, fixed in 2.4.46).
Warm regards, Quanah
Hi Quanah,
We are running 2.4.44 so there maybe something fixed between that and the latest version....
So I've look at the ldap_audit.log and this person initially set their mail attribute at 20180813215911Z and then changed it to a more readable version at 20180813220044Z. This makes sense given our account activation flow for new
students. So it seems that the second change didn't get out to the particular consumer. Unfortunately this is long enough ago that it won't be in the accesslog or in the ldap_access.log files on the consumers. I'll see if I can find a more recent case.
Hi!
What comes to my mind is this: For the email example the CSN does not necessarily refect the cange of the emmail attribute, especially if you are using password policy. Maybe the incorrect sync happened in the past with a buggy version of OpenLDAP?
Regards, Ulrich
I checked all 13 of our consumers and while the modifyTimestamp and entryCSN are all consistent with the master(s), the mail attribute was not updated on 3 of the consumers. It's correct on the other 10.
Thanks for all your help, ds
openldap-technical@openldap.org
-
Dave Steiner -
Frank Swasey -
Quanah Gibson-Mount -
Ulrich Windl