Hi,
due to the "DEL don't get replicated" issue, I have inconsistent data on the consumers.
I set up a test pair with a provider with current master data and a consumer with current slave data. The data set differs in a few entries which DEL were not replicated.
Is there any way to get the two server on sync again other then completely delete the database and refill (by slapadd from master or full resync) on the consumer?
I tried (with various combinations of 2.4.26, 2.4.28 and current pre 2.4.30) to start the consumer with "-c rid=xxx,csn=" which starts a full sync, but the (on the master not existing) objects don't get deleted (on the slave).
Marc
On Thu, 23 Feb 2012, Marc Patermann wrote:
The data set differs in a few entries which DEL were not replicated.
I tried (with various combinations of 2.4.26, 2.4.28 and current pre 2.4.30) to start the consumer with "-c rid=xxx,csn=" which starts a full sync, but the (on the master not existing) objects don't get deleted (on the slave).
Yeah, it's an impossible condition so it's not handled the best...
I'm sure that the most correct answer would be to slapcat the master and slapadd -q the replica. Since you should have multiple replicas, this shouldn't be all that high-impact.
With that said, the Net::LDAP CPAN module comes with some interesting contrib scripts such as "ldifdiff"; I'll confess to using that once or twice in similar situations (take a master and a replica slapcat). This won't be practical if you have high data churn, but many LDAP installations are read-mostly, so...
Hi,
Aaron Richton schrieb (23.02.2012 15:10 Uhr):
On Thu, 23 Feb 2012, Marc Patermann wrote:
The data set differs in a few entries which DEL were not replicated.
I tried (with various combinations of 2.4.26, 2.4.28 and current pre 2.4.30) to start the consumer with "-c rid=xxx,csn=" which starts a full sync, but the (on the master not existing) objects don't get deleted (on the slave).
Yeah, it's an impossible condition so it's not handled the best...
There is - however - a state where the consumer checks every entry for existence:
"slapd[23595]: nonpresent_callback: rid=402 present UUID a79a831e-f18b-1030-9aca-21c3336314b4, dn ..."
After this the consumer is exactly like the provider and all previous changes and DEL and synced.
But I'm not sure to always ensure how to achieve this state.
I deleted the database and put current master data in the provider and two day old slave data in the consumer. I started the provider and the consumer. This worked two times today, but later I was not able to reproduce it in all cases. This is a bit frustrating. :(
Is there any rock solid practice to get the consumer in the "check present" state?
Marc
Marc Patermann wrote:
Hi,
Aaron Richton schrieb (23.02.2012 15:10 Uhr):
On Thu, 23 Feb 2012, Marc Patermann wrote:
The data set differs in a few entries which DEL were not replicated.
I tried (with various combinations of 2.4.26, 2.4.28 and current pre 2.4.30) to start the consumer with "-c rid=xxx,csn=" which starts a full sync, but the (on the master not existing) objects don't get deleted (on the slave).
Yeah, it's an impossible condition so it's not handled the best...
There is - however - a state where the consumer checks every entry for existence:
"slapd[23595]: nonpresent_callback: rid=402 present UUID a79a831e-f18b-1030-9aca-21c3336314b4, dn ..."
After this the consumer is exactly like the provider and all previous changes and DEL and synced.
But I'm not sure to always ensure how to achieve this state.
I deleted the database and put current master data in the provider and two day old slave data in the consumer. I started the provider and the consumer. This worked two times today, but later I was not able to reproduce it in all cases. This is a bit frustrating. :(
Is there any rock solid practice to get the consumer in the "check present" state?
Read RFC4533, there's no reason to take stabs in the dark.
Howard Chu schrieb (24.02.2012 11:09 Uhr):
Marc Patermann wrote:
Aaron Richton schrieb (23.02.2012 15:10 Uhr):
On Thu, 23 Feb 2012, Marc Patermann wrote:
The data set differs in a few entries which DEL were not replicated.
I tried (with various combinations of 2.4.26, 2.4.28 and current pre 2.4.30) to start the consumer with "-c rid=xxx,csn=" which starts a full sync, but the (on the master not existing) objects don't get deleted (on the slave).
Yeah, it's an impossible condition so it's not handled the best...
There is - however - a state where the consumer checks every entry for existence:
"slapd[23595]: nonpresent_callback: rid=402 present UUID a79a831e-f18b-1030-9aca-21c3336314b4, dn ..."
After this the consumer is exactly like the provider and all previous changes and DEL and synced.
But I'm not sure to always ensure how to achieve this state.
I deleted the database and put current master data in the provider and two day old slave data in the consumer. I started the provider and the consumer. This worked two times today, but later I was not able to reproduce it in all cases. This is a bit frustrating. :(
Is there any rock solid practice to get the consumer in the "check present" state?
Read RFC4533, there's no reason to take stabs in the dark.
I'm sure slapd respects all this. :)
In parts of my test I made a mistake by providing a cookie (damn /etc/sysconfig/openldap). With this sorted out, this is what I get (both pre 2.4.30 code):
- slapadd the provider with current data - slapadd the consumer with older data -> present check, exact sync - ldap changes on provider - reset and slapadd the consumer with older data -> no present check, servers not in sync only changes made after the last provider start are synced - reset and slapadd the consumer with older data and restart provider -> present check, exact sync
I then checked against 2.4.26 on the provider and the behavior changed:
- slapadd the provider with current data - slapadd the consumer with older data -> present check, exact sync - ldap changes on provider - reset and slapadd the consumer with older data -> present check, exact sync
Is this the way it is supposed to happen? I think this is a bug in current code.
Marc
Marc Patermann wrote:
Howard Chu schrieb (24.02.2012 11:09 Uhr):
Marc Patermann wrote:
Aaron Richton schrieb (23.02.2012 15:10 Uhr):
On Thu, 23 Feb 2012, Marc Patermann wrote:
The data set differs in a few entries which DEL were not replicated.
I tried (with various combinations of 2.4.26, 2.4.28 and current pre 2.4.30) to start the consumer with "-c rid=xxx,csn=" which starts a full sync, but the (on the master not existing) objects don't get deleted (on the slave).
Yeah, it's an impossible condition so it's not handled the best...
There is - however - a state where the consumer checks every entry for existence:
"slapd[23595]: nonpresent_callback: rid=402 present UUID a79a831e-f18b-1030-9aca-21c3336314b4, dn ..."
After this the consumer is exactly like the provider and all previous changes and DEL and synced.
But I'm not sure to always ensure how to achieve this state.
I deleted the database and put current master data in the provider and two day old slave data in the consumer. I started the provider and the consumer. This worked two times today, but later I was not able to reproduce it in all cases. This is a bit frustrating. :(
Is there any rock solid practice to get the consumer in the "check present" state?
Read RFC4533, there's no reason to take stabs in the dark.
I'm sure slapd respects all this. :)
In parts of my test I made a mistake by providing a cookie (damn /etc/sysconfig/openldap). With this sorted out, this is what I get (both pre 2.4.30 code):
- slapadd the provider with current data
- slapadd the consumer with older data
-> present check, exact sync
- ldap changes on provider
- reset and slapadd the consumer with older data
-> no present check, servers not in sync only changes made after the last provider start are synced
- reset and slapadd the consumer with older data and restart provider
-> present check, exact sync
I then checked against 2.4.26 on the provider and the behavior changed:
- slapadd the provider with current data
- slapadd the consumer with older data
-> present check, exact sync
- ldap changes on provider
- reset and slapadd the consumer with older data
-> present check, exact sync
Is this the way it is supposed to happen? I think this is a bug in current code.
In the complete absence of any config information from you, it's impossible to determine. If you have a syncprov sessionlog, then if you didn't restart the provider, it may not cause a full presence check when a consumer connects.
Hi,
Howard Chu schrieb (24.02.2012 21:36 Uhr):
Marc Patermann wrote:
With this sorted out, this is what I get (both pre 2.4.30 code):
- slapadd the provider with current data
- slapadd the consumer with older data
-> present check, exact sync
- ldap changes on provider
- reset and slapadd the consumer with older data
-> no present check, servers not in sync only changes made after the last provider start are synced
- reset and slapadd the consumer with older data and restart provider
-> present check, exact sync
I then checked against 2.4.26 on the provider and the behavior changed:
- slapadd the provider with current data
- slapadd the consumer with older data
-> present check, exact sync
- ldap changes on provider
- reset and slapadd the consumer with older data
-> present check, exact sync
Is this the way it is supposed to happen? I think this is a bug in current code.
In the complete absence of any config information from you, it's impossible to determine. If you have a syncprov sessionlog, then if you didn't restart the provider, it may not cause a full presence check when a consumer connects.
So, the sessionlog keeps me from getting the full sync in version >2.4.26? I set sessionlog. Here is the provider config:
modulepath /usr/lib/openldap/modules moduleload accesslog.la moduleload dynlist.la moduleload memberof.la moduleload refint.la moduleload unique.la include /etc/openldap/schema/core.schema include /etc/openldap/schema/cosine.schema include /etc/openldap/schema/inetorgperson.schema include /etc/openldap/schema/nis.schema include /etc/openldap/schema/authldap.schema include /etc/openldap/schema/openldap.schema include /etc/openldap/schema/dyngroup.schema include /etc/openldap/schema/dnszone.schema include /etc/openldap/schema/dhcp.schema include /etc/openldap/schema/ofdconf.schema include /etc/openldap/schema/nagios.schema include /etc/openldap/schema/yast.schema include /etc/openldap/schema/ofdaddon.schema include /etc/openldap/ofd.acl pidfile /var/run/slapd/slapd.pid argsfile /var/run/slapd/slapd.args threads 350 concurrency 350 loglevel config stats stats2 sync defaultsearchbase "ou=humans,ou=foo" sizelimit -1 timelimit 4000 TLSCACertificateFile /etc/openldap/ssl/ldapca.pem TLSCertificateFile /etc/openldap/ssl/ldapcert.pem TLSCertificateKeyFile /etc/openldap/ssl/ldapkey.pem authz-policy to authz-regexp uid=human,cn=[^,]*,cn=auth dn:"cn=human,ou=mgr,ou=foo" authz-regexp uid=cyrus,cn=[^,]*,cn=auth "ldap:///ou=humans,ou=foo??sub?(uid=cyrus)" authz-regexp uid=([^,]*),cn=[^,]*,cn=auth "ldap:///ou=humans,ou=foo??sub?(maildrop=$1@*)" authz-regexp "cn=human,ou=mgr,ou=foo" dn.subtree="ou=humans,ou=foo" database config rootdn cn=config rootpw secret database bdb suffix "ou=foo" rootdn "cn=gen.man,ou=mgr,ou=foo" directory /var/lib/ldap/main-data checkpoint 4096 5 cachesize 25000 idlcachesize 45000 dbconfig set_cachesize 2 0 0 dbconfig set_lg_dir /var/log/bdb/main dbconfig set_lg_regionmax 262144 dbconfig set_lg_bsize 2097152 dbconfig set_flags DB_LOG_AUTOREMOVE index objectClass eq index cn eq,sub index version eq index entryCSN,entryUUID eq index default eq index uid index mail sub,eq index sn sub,eq index givenName sub,eq index maildrop,ou sub,eq index relativeDomainName eq,sub index dhcpHWAddress index dhcpClassData index dhcpOption index zoneName index uidNumber,gidNumber eq include /etc/openldap/linux.acl overlay glue overlay accesslog logdb "cn=log" logops writes logpurge 10+00:00 1+00:00 overlay dynlist dynlist-attrset groupOfURLs memberURL overlay refint refint_attributes member memberOf refint_nothing "cn=dummy,ou=foo" overlay syncprov syncprov-checkpoint 100 10 syncprov-sessionlog 100 database hdb suffix "cn=log" directory /var/lib/ldap/log-data rootdn "cn=gen.man,ou=mgr,ou=foo" checkpoint 10240 5 dbconfig set_cachesize 0 367001600 0 dbconfig set_lg_dir /var/log/bdb/log dbconfig set_lg_regionmax 262144 dbconfig set_lg_bsize 2097152 dbconfig set_flags DB_LOG_AUTOREMOVE index objectClass eq index cn eq,sub index reqStart eq overlay syncprov syncprov-nopresent TRUE syncprov-checkpoint 100 10 syncprov-sessionlog 100 database monitor access to dn.subtree="cn=monitor" by * read
Marc
openldap-technical@openldap.org
-
Aaron Richton -
Howard Chu -
Marc Patermann -
Michael Ströder