1:49 p.m.
I've been noticing various data discrepancies between our LDAP master and LDAP
consumers. We are running OpenLDAP v2.4.44. We have two masters running
"mirromode TRUE" and all updates go through a VIP that points to the first one
unless it's not available (doesn't happen very often except for during patches
and restarts). We have 13 consumers that replicate through that same VIP.
Here's an example of our syncrepl for a client:
syncrepl rid=221
type=refreshAndPersist
schemachecking=on
provider="ldap://ldapmastervip.rutgers.edu/"
bindmethod=sasl
saslmech=EXTERNAL
starttls=yes
tls_reqcert=demand
tls_protocol_min="3.1"
searchbase="dc=rutgers,dc=edu"
attrs="*,+"
retry="10 10 20 +"
logbase="cn=accesslog"
logfilter="(&(objectClass=auditWriteObject)(reqResult=0))"
syncdata=accesslog
network-timeout=30
keepalive=180:3:60
I check the contextCSN attributes on all the instances every day and they are
all in sync (except during any major changes, of course). But I occasionally
notice discrepancies in the data.... even though the contextCSNs and entryCSNs
are the same. For example (note hostnames have been changed):
$ ldapsearch ... -H ldap://ldapmaster.rutgers.edu uid=XXXX postalAddress
createTimestamp modifyTimestamp entryCSN
dn: uid=XXXX,ou=People,dc=rutgers,dc=edu
createTimestamp: 20121220100700Z
postalAddress: Business And Science Bldg$227 Penn Street$Camden, NJ 081021656
entryCSN: 20180505002024.083133Z#000000#001#000000
modifyTimestamp: 20180505002024Z
$ ldapsearch ... -H ldap://ldapconsumer3.rutgers.edu uid=XXXX postalAddress
createTimestamp modifyTimestamp entryCSN
dn: uid=XXXX,ou=People,dc=rutgers,dc=edu
createTimestamp: 20121220100700Z
postalAddress: BUSINESS AND SCIENCE BLDG$227 PENN STREET$CAMDEN, NJ 081021656
entryCSN: 20180505002024.083133Z#000000#001#000000
modifyTimestamp: 20180505002024Z
So I'm trying to figure out why this happens (config issue, bug, ???) and
second, if I can't use the contextCSN to report that everything is fine, what
else can I do besides trying to compare ldif dumps.
thanks,
ds
--
Dave Steiner steiner(a)rutgers.edu
IdM, Enterprise Application Services ASB101; 848.445.5433
Rutgers University, Office of Information Technology
11:39 a.m.
On Sep 5, 2018, at 16:49, Dave Steiner <steiner(a)rutgers.edu>
wrote:
But I occasionally notice discrepancies in the data.... even though the contextCSNs and
entryCSNs are the same. For example (note hostnames have been changed):
$ ldapsearch ... -H ldap://ldapmaster.rutgers.edu uid=XXXX postalAddress createTimestamp
modifyTimestamp entryCSN
postalAddress: Business And Science Bldg$227 Penn Street$Camden, NJ 081021656
$ ldapsearch ... -H ldap://ldapconsumer3.rutgers.edu uid=XXXX postalAddress
createTimestamp modifyTimestamp entryCSN
dn: uid=XXXX,ou=People,dc=rutgers,dc=edu
postalAddress: BUSINESS AND SCIENCE BLDG$227 PENN STREET$CAMDEN, NJ 081021656
You do realize that as far as LDAP is concerned, those two postalAddress values are
identical. The matching rules for postalAddress are case insensitive.
Assuming that the mixed case one is what you prefer and the upper case one is what you
started with - one has to ask, how was it changed on the one server and make certain that
it was not done in a way that would not be sent through the syncrepl process.
- Frank
11:48 a.m.
--On Wednesday, September 05, 2018 5:49 PM -0400 Dave Steiner
<steiner(a)rutgers.edu> wrote:
Hi Dave,
$ ldapsearch ... -H ldap://ldapmaster.rutgers.edu uid=XXXX
postalAddress
createTimestamp modifyTimestamp entryCSN
dn: uid=XXXX,ou=People,dc=rutgers,dc=edu
createTimestamp: 20121220100700Z
postalAddress: Business And Science Bldg$227 Penn Street$Camden, NJ
081021656
entryCSN: 20180505002024.083133Z#000000#001#000000
modifyTimestamp: 20180505002024Z
$ ldapsearch ... -H ldap://ldapconsumer3.rutgers.edu uid=XXXX
postalAddress createTimestamp modifyTimestamp entryCSN
dn: uid=XXXX,ou=People,dc=rutgers,dc=edu
createTimestamp: 20121220100700Z
postalAddress: BUSINESS AND SCIENCE BLDG$227 PENN STREET$CAMDEN, NJ
081021656
entryCSN: 20180505002024.083133Z#000000#001#000000
modifyTimestamp: 20180505002024Z
Those entries appear identical to me. I assume you believe they are not
identical because the case doesn't match. However, postalAddress uses
caseIgnore normalization rules, so they are in fact identical as far as
LDAP is concerned.
Also, given the entry was created in 2012 and there's no indication of when
the postalAddress attribute was modified, it's impossible to determine
when/how the divergence in capitalization occurred. You also don't provide
any information about the underlying database in use, which could be very
relevant, history wise.
So I'm trying to figure out why this happens (config issue, bug,
???) and
second, if I can't use the contextCSN to report that everything is fine,
what else can I do besides trying to compare ldif dumps.
So far, everything appears fine from a technical standpoint (the values
match per LDAP SYNTAX rules).
I would suggest the following if you would like the case to match:
dn: ...
changetype: modify
replace: postalAddress
postalAddress: ...
That should correct the value on the providers and consumers to match case.
Warm regards,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
1:43 p.m.
On 9/6/2018 2:48 PM, Quanah Gibson-Mount wrote:
--On Wednesday, September 05, 2018 5:49 PM -0400 Dave Steiner
<steiner(a)rutgers.edu> wrote:
Hi Dave,
> $ ldapsearch ... -H ldap://ldapmaster.rutgers.edu uid=XXXX postalAddress
> createTimestamp modifyTimestamp entryCSN
> dn: uid=XXXX,ou=People,dc=rutgers,dc=edu
> createTimestamp: 20121220100700Z
> postalAddress: Business And Science Bldg$227 Penn Street$Camden, NJ
> 081021656
> entryCSN: 20180505002024.083133Z#000000#001#000000
> modifyTimestamp: 20180505002024Z
>
> $ ldapsearch ... -H ldap://ldapconsumer3.rutgers.edu uid=XXXX
> postalAddress createTimestamp modifyTimestamp entryCSN
> dn: uid=XXXX,ou=People,dc=rutgers,dc=edu
> createTimestamp: 20121220100700Z
> postalAddress: BUSINESS AND SCIENCE BLDG$227 PENN STREET$CAMDEN, NJ
> 081021656
> entryCSN: 20180505002024.083133Z#000000#001#000000
> modifyTimestamp: 20180505002024Z
Those entries appear identical to me. I assume you believe they are not
identical because the case doesn't match. However, postalAddress uses
caseIgnore normalization rules, so they are in fact identical as far as LDAP
is concerned.
Also, given the entry was created in 2012 and there's no indication of when
the postalAddress attribute was modified, it's impossible to determine
when/how the divergence in capitalization occurred. You also don't provide
any information about the underlying database in use, which could be very
relevant, history wise.
Ok, bad example... Here's another example that's rather different:
$ searchn -H ldap://ldapmaster.rutgers.edu uid=XXXX mail createTimestamp
modifyTimestamp entryCSN
dn: uid=XXXX,ou=People,dc=rutgers,dc=edu
createTimestamp: 20180504140010Z
mail: xxxxxxxxx.xxxx(a)rutgers.edu
entryCSN: 20180831205041.549496Z#000000#001#000000
modifyTimestamp: 20180831205041Z
$ searchn -H ldap://ldapconsumer1.rutgers.edu uid=XXXX mail createTimestamp
modifyTimestamp entryCSN
dn: uid=XXXX,ou=People,dc=rutgers,dc=edu
createTimestamp: 20180504140010Z
mail: yyyyy(a)rutgers.edu
entryCSN: 20180831205041.549496Z#000000#001#000000
modifyTimestamp: 20180831205041Z
> So I'm trying to figure out why this happens (config issue, bug, ???) and
> second, if I can't use the contextCSN to report that everything is fine,
> what else can I do besides trying to compare ldif dumps.
So far, everything appears fine from a technical standpoint (the values match
per LDAP SYNTAX rules).
I would suggest the following if you would like the case to match:
dn: ...
changetype: modify
replace: postalAddress
postalAddress: ...
That should correct the value on the providers and consumers to match case.
Warm regards,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.symas...
9:59 a.m.
--On Thursday, September 06, 2018 5:43 PM -0400 Dave Steiner
<steiner(a)rutgers.edu> wrote:
Ok, bad example... Here's another example that's rather
different:
$ searchn -H ldap://ldapmaster.rutgers.edu uid=XXXX mail createTimestamp
modifyTimestamp entryCSN
dn: uid=XXXX,ou=People,dc=rutgers,dc=edu
createTimestamp: 20180504140010Z
mail: xxxxxxxxx.xxxx(a)rutgers.edu
entryCSN: 20180831205041.549496Z#000000#001#000000
modifyTimestamp: 20180831205041Z
$ searchn -H ldap://ldapconsumer1.rutgers.edu uid=XXXX mail
createTimestamp modifyTimestamp entryCSN
dn: uid=XXXX,ou=People,dc=rutgers,dc=edu
createTimestamp: 20180504140010Z
mail: yyyyy(a)rutgers.edu
entryCSN: 20180831205041.549496Z#000000#001#000000
modifyTimestamp: 20180831205041Z
Hi Dave,
That's certainly a more interesting case. Do you know when the discrepancy
arose? Clearly the entry was modified on 8/31, but you'd have to parse the
accesslog to see what the actual change was that was made to the entry.
Generally to track down replication related issues, there are a number of
various bits that need to be examined in the logs (provider and consumer)
along with the data in the accesslog. One would also need to have
knowledge on the version of OpenLDAP that's in use since it may be subject
to bugs that could cause such issues (ITS#8444 for example, fixed in
2.4.46).
Warm regards,
Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
12:10 p.m.
On 9/7/2018 12:59 PM, Quanah Gibson-Mount wrote:
--On Thursday, September 06, 2018 5:43 PM -0400 Dave Steiner
<steiner(a)rutgers.edu> wrote:
> Ok, bad example... Here's another example that's rather different:
>
> $ searchn -H ldap://ldapmaster.rutgers.edu uid=XXXX mail createTimestamp
> modifyTimestamp entryCSN
> dn: uid=XXXX,ou=People,dc=rutgers,dc=edu
> createTimestamp: 20180504140010Z
> mail: xxxxxxxxx.xxxx(a)rutgers.edu
> entryCSN: 20180831205041.549496Z#000000#001#000000
> modifyTimestamp: 20180831205041Z
>
> $ searchn -H ldap://ldapconsumer1.rutgers.edu uid=XXXX mail
> createTimestamp modifyTimestamp entryCSN
> dn: uid=XXXX,ou=People,dc=rutgers,dc=edu
> createTimestamp: 20180504140010Z
> mail: yyyyy(a)rutgers.edu
> entryCSN: 20180831205041.549496Z#000000#001#000000
> modifyTimestamp: 20180831205041Z
Hi Dave,
That's certainly a more interesting case. Do you know when the discrepancy
arose? Clearly the entry was modified on 8/31, but you'd have to parse the
accesslog to see what the actual change was that was made to the entry.
Generally to track down replication related issues, there are a number of
various bits that need to be examined in the logs (provider and consumer)
along with the data in the accesslog. One would also need to have knowledge
on the version of OpenLDAP that's in use since it may be subject to bugs that
could cause such issues (ITS#8444 for example, fixed in 2.4.46).
Warm regards,
Quanah
Hi Quanah,
We are running 2.4.44 so there maybe something fixed between that and the latest
version....
So I've look at the ldap_audit.log and this person initially set their mail
attribute at 20180813215911Z and then changed it to a more readable version at
20180813220044Z. This makes sense given our account activation flow for new
students. So it seems that the second change didn't get out to the particular
consumer. Unfortunately this is long enough ago that it won't be in the
accesslog or in the ldap_access.log files on the consumers. I'll see if I can
find a more recent case.
I checked all 13 of our consumers and while the modifyTimestamp and entryCSN are
all consistent with the master(s), the mail attribute was not updated on 3 of
the consumers. It's correct on the other 10.
Thanks for all your help,
ds
11:59 p.m.
New subject: Antw: Re: Replication issue? Data is different between master and consumer with same entryCSNs
>> Dave Steiner <steiner(a)rutgers.edu> schrieb am
07.09.2018 um 21:10 in Nachricht
<44b66794-4e1b-8de1-57f3-55f280c85358(a)oit.rutgers.edu>:
On 9/7/2018 12:59 PM, Quanah Gibson-Mount wrote:
> --On Thursday, September 06, 2018 5:43 PM -0400 Dave Steiner
> <steiner(a)rutgers.edu> wrote:
>
>> Ok, bad example... Here's another example that's rather different:
>>
>> $ searchn -H ldap://ldapmaster.rutgers.edu uid=XXXX mail createTimestamp
>> modifyTimestamp entryCSN
>> dn: uid=XXXX,ou=People,dc=rutgers,dc=edu
>> createTimestamp: 20180504140010Z
>> mail: xxxxxxxxx.xxxx(a)rutgers.edu
>> entryCSN: 20180831205041.549496Z#000000#001#000000
>> modifyTimestamp: 20180831205041Z
>>
>> $ searchn -H ldap://ldapconsumer1.rutgers.edu uid=XXXX mail
>> createTimestamp modifyTimestamp entryCSN
>> dn: uid=XXXX,ou=People,dc=rutgers,dc=edu
>> createTimestamp: 20180504140010Z
>> mail: yyyyy(a)rutgers.edu
>> entryCSN: 20180831205041.549496Z#000000#001#000000
>> modifyTimestamp: 20180831205041Z
>
> Hi Dave,
>
> That's certainly a more interesting case. Do you know when the discrepancy
> arose? Clearly the entry was modified on 8/31, but you'd have to parse the
> accesslog to see what the actual change was that was made to the entry.
> Generally to track down replication related issues, there are a number of
> various bits that need to be examined in the logs (provider and consumer)
> along with the data in the accesslog. One would also need to have knowledge
> on the version of OpenLDAP that's in use since it may be subject to bugs
that
> could cause such issues (ITS#8444 for example, fixed in 2.4.46).
>
> Warm regards,
> Quanah
Hi Quanah,
We are running 2.4.44 so there maybe something fixed between that and the
latest
version....
So I've look at the ldap_audit.log and this person initially set their mail
attribute at 20180813215911Z and then changed it to a more readable version
at
20180813220044Z. This makes sense given our account activation flow for new
students. So it seems that the second change didn't get out to the
particular
consumer. Unfortunately this is long enough ago that it won't be in the
accesslog or in the ldap_access.log files on the consumers. I'll see if I
can
find a more recent case.
Hi!
What comes to my mind is this: For the email example the CSN does not necessarily refect
the cange of the emmail attribute, especially if you are using password policy. Maybe the
incorrect sync happened in the past with a buggy version of OpenLDAP?
Regards,
Ulrich
I checked all 13 of our consumers and while the modifyTimestamp and entryCSN
are
all consistent with the master(s), the mail attribute was not updated on 3
of
the consumers. It's correct on the other 10.
Thanks for all your help,
ds
1914
days inactive
1919
days old
openldap-technical@openldap.org
6 comments
4 participants
participants (4)
-
Dave Steiner -
Frank Swasey -
Quanah Gibson-Mount -
Ulrich Windl