Hello,
I had thought I tested this beforehand but I seem to be able to reliably reproduce the following situation:
We have an installation where the provider server has information that is replicated to downstream replicas using the syncrepl protocol. The account used to replicate is allowed to see records where certain attributes meet specific values, a silly example is an attribute is set
dn: uid=somerecord,ou=people,dc=ucsc,dc=edu replicateMe: TRUE ...
When an account has that attribute set it then replicates to the downstream replica, however if later we set replicateMe to FALSE so that the account used for replication can no longer see the entry, it seems to be orphaned and is not removed in the replica.
We are using OpenLDAP 2.4.26 and I have the syncprov sessionlog set to 500 and the replica is set to refreshAndPersist.
Is this something that is simply not supported? or would a case like this be expected to work and I've either got a configuration issue or a bug?
Thanks Jeffrey
Jeffrey Crawford wrote:
Hello,
I had thought I tested this beforehand but I seem to be able to reliably reproduce the following situation:
We have an installation where the provider server has information that is replicated to downstream replicas using the syncrepl protocol. The account used to replicate is allowed to see records where certain attributes meet specific values, a silly example is an attribute is set
dn: uid=somerecord,ou=people,dc=ucsc,dc=edu replicateMe: TRUE ...
When an account has that attribute set it then replicates to the downstream replica, however if later we set replicateMe to FALSE so that the account used for replication can no longer see the entry, it seems to be orphaned and is not removed in the replica.
We are using OpenLDAP 2.4.26 and I have the syncprov sessionlog set to 500 and the replica is set to refreshAndPersist.
Is this something that is simply not supported? or would a case like this be expected to work and I've either got a configuration issue or a bug?
Visibility changes due to ACL rules are not detected. syncprov only checks an entry against the search parameters of the original sync search operation, i.e., the base, scope, and filter. If an entry matches these params before the modification, and no longer matches after the operation, syncprov will send a delete message for that entry. (Likewise if an entry doesn't match before, but matches after, syncprov will send an Add for the entry.)
Thanks Howard,
Let me make sure I understand your response. I'm not changing any ACL's, they are staying the same. Just the attributes in the record are changing. Are you saying that syncprov looks at the account that is bound and sends deletes if a record would become invisible after a modification? If that's the case it doesn't seem to be working right.
If syncprov is being used and looks at which user is bound would it have trouble if there are multiple replicas and replica accounts binding to get replication data? we have two distinct downstream replicas that each have different criteria for which records get replicated to them.
Jeffrey
On Thu, May 31, 2012 at 6:38 PM, Howard Chu hyc@symas.com wrote:
Jeffrey Crawford wrote:
Hello,
I had thought I tested this beforehand but I seem to be able to reliably reproduce the following situation:
We have an installation where the provider server has information that is replicated to downstream replicas using the syncrepl protocol. The account used to replicate is allowed to see records where certain attributes meet specific values, a silly example is an attribute is set
dn: uid=somerecord,ou=people,dc=**ucsc,dc=edu replicateMe: TRUE ...
When an account has that attribute set it then replicates to the downstream replica, however if later we set replicateMe to FALSE so that the account used for replication can no longer see the entry, it seems to be orphaned and is not removed in the replica.
We are using OpenLDAP 2.4.26 and I have the syncprov sessionlog set to 500 and the replica is set to refreshAndPersist.
Is this something that is simply not supported? or would a case like this be expected to work and I've either got a configuration issue or a bug?
Visibility changes due to ACL rules are not detected. syncprov only checks an entry against the search parameters of the original sync search operation, i.e., the base, scope, and filter. If an entry matches these params before the modification, and no longer matches after the operation, syncprov will send a delete message for that entry. (Likewise if an entry doesn't match before, but matches after, syncprov will send an Add for the entry.)
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/**project/http://www.openldap.org/project/
On 1/6/2012 8:54 πμ, Jeffrey Crawford wrote:
Are you saying that syncprov looks at the account that is bound and sends deletes if a record would become invisible after a modification?
I understand the opposite: syncprov will only send add/delete message based on base/scope/filter and not on ACL-visibility. So in essence Howard says that ACL-based filtering in replication does not result in proper results to consumers.
This is tricky! (I didn't know either.) It means that we should *not* design our replication based on ACL-filtering (which, unfortunately, we have done too), but, on the contrary, that we should design our DIT so that it can cover our replication needs based on consumer base/scope/filter configuration, and we should design/adapt our ACLs with the above rule in mind.
Please confirm the above thoughts.
Thanks, Nick
Humm and taking this one step further I'm guessing that the replication account probably needs to see at least the entryUUID and entryCSN for all accounts to make sure that it can see the records it needs to delete. Okay at least I have some direction to go on now.
Jeffrey
On Fri, Jun 1, 2012 at 9:06 AM, Nick Milas nick@eurobjects.com wrote:
On 1/6/2012 8:54 πμ, Jeffrey Crawford wrote:
Are you saying that syncprov looks at the account that is bound and sends
deletes if a record would become invisible after a modification?
I understand the opposite: syncprov will only send add/delete message based on base/scope/filter and not on ACL-visibility. So in essence Howard says that ACL-based filtering in replication does not result in proper results to consumers.
This is tricky! (I didn't know either.) It means that we should *not* design our replication based on ACL-filtering (which, unfortunately, we have done too), but, on the contrary, that we should design our DIT so that it can cover our replication needs based on consumer base/scope/filter configuration, and we should design/adapt our ACLs with the above rule in mind.
Please confirm the above thoughts.
Thanks, Nick
Ok I think I got this to work I didn't add a filter to the syncrepl parameter so I'm using ACL's as before, however I changed the acls to allow the replica account access to the attributes entry and entryUUID only on every item in the directory, now setting attributes to values so that they no longer match the access to the rest of the acls seem to do what I wan't however I'm not quite sure how. so in short the provider looks something like this:
olcAccess: to dn.subtree="dc=example,dc=org" attrs=entry,entryUUID by dn.base="cn=replica,ou=admins,dc=example,dc=org" ssf=128 read by * none break olcAccess: to dn.subtree="dc=example,dc=org" filter="(replicateMe=TRUE)" attrs=@inetOrgPerson,@posixAccount,entryCSN,contextCSN,createTimestamp,modifyTimestamp,structuralObjectClass by dn.base="cn=replica,ou=admins,dc=example,dc=org" ssf=128 read by * none break
And the replicas are configured as so:
dn: olcDatabase={1}hdb,cn=config changeType: modify replace: olcSyncrepl olcSyncrepl: rid=1 provider=ldap://ldap-!!!!ENV!!!!-1.example.org:1389/ starttls=critical tls_reqcert=never bindmethod=simple retry="60 10 900 92 86400 3" binddn="cn=replica,ou=admins,dc=example,dc=org" credentials=!!!!PW!!!! schemachecking=off searchbase="dc=example,dc=org" type=refreshAndPersist olcSyncrepl: rid=2 provider=ldap://ldap-!!!!ENV!!!!-2.example.org:1389/ starttls=critical tls_reqcert=never bindmethod=simple retry="60 10 900 92 86400 3" binddn="cn=replica,ou=admins,dc=example,dc=org" credentials=!!!!PW!!!! schemachecking=off searchbase="dc=example,dc=org" type=refreshAndPersist
now the following will add a record to the replica: dn: uid=user,ou=people,dc=example,dc=org changeType: modify replace: replicateMe replicateMe: TRUE
And the following will take it away (delete): dn: uid=user,ou=people,dc=example,dc=org changeType: modify replace: replicateMe replicateMe: FALSE
However I don't know enough about the underlying code to be sure the following is working as designed. Meaning I don't want to be relying on a bug that might get fixed later.
Thanks Jeffrey
access:
On Fri, Jun 1, 2012 at 9:26 AM, Jeffrey Crawford jeffreyc@ucsc.edu wrote:
Humm and taking this one step further I'm guessing that the replication account probably needs to see at least the entryUUID and entryCSN for all accounts to make sure that it can see the records it needs to delete. Okay at least I have some direction to go on now.
Jeffrey
On Fri, Jun 1, 2012 at 9:06 AM, Nick Milas nick@eurobjects.com wrote:
On 1/6/2012 8:54 πμ, Jeffrey Crawford wrote:
Are you saying that syncprov looks at the account that is bound and sends deletes if a record would become invisible after a modification?
I understand the opposite: syncprov will only send add/delete message based on base/scope/filter and not on ACL-visibility. So in essence Howard says that ACL-based filtering in replication does not result in proper results to consumers.
This is tricky! (I didn't know either.) It means that we should *not* design our replication based on ACL-filtering (which, unfortunately, we have done too), but, on the contrary, that we should design our DIT so that it can cover our replication needs based on consumer base/scope/filter configuration, and we should design/adapt our ACLs with the above rule in mind.
Please confirm the above thoughts.
Thanks, Nick
-- I fly because it releases my mind from the tyranny of petty things . . .
— Antoine de Saint-Exupéry
Jeffrey E. Crawford ITS Application Administrator (IDM) 831-459-4365 jeffreyc@ucsc.edu
-- I fly because it releases my mind from the tyranny of petty things . . .
— Antoine de Saint-Exupéry
Jeffrey E. Crawford ITS Application Administrator (IDM) 831-459-4365 jeffreyc@ucsc.edu
On 1/6/2012 4:38 πμ, Howard Chu wrote:
Visibility changes due to ACL rules are not detected. syncprov only checks an entry against the search parameters of the original sync search operation, i.e., the base, scope, and filter. If an entry matches these params before the modification, and no longer matches after the operation, syncprov will send a delete message for that entry. (Likewise if an entry doesn't match before, but matches after, syncprov will send an Add for the entry.)
I would like a clarification on this, please:
Since the syncprov mechanism does a search based on base/scope/filter from *a particular binddn* account, doesn't this mean that if visibility *by that same binddn* of some entry (due to ACL restrictions) changes after a modification, then effectively the same search (based on the same base/scope/filter) will produce different results, which means that the syncprov mechanism *should* generate an add/delete message accordingly?
In other words, syncprov does not produce messages based on the differences between the results of standard ldapsearch'es? And if it does not, shouldn't it?
Why syncprov should ignore ACL-based visibility? This seems unnatural and does not assist conceptualization. At least it seems confusing to me. Can you please provide more details on the syncprov mechanism with regard to this?
Please advise!
Thanks, Nick
On 2/6/2012 11:18 πμ, Nick Milas wrote:
In other words, syncprov does not produce messages based on the differences between the results of standard ldapsearch'es? And if it does not, shouldn't it?
My tests (with v2.4.31 on both provider and consumer) show that syncrepl (refreshAndPersist) works correctly when replicating based on ACL restrictions. OpenLDAP consumer deletes correctly an entry from a branch when the entry is moved to another, invisible by the consumer binddn, branch, and it re-creates it correctly when it is moved back to a visible (based on ACL) branch.
So the answer above is yes, syncprov *does* produce update messages based on the differences between the results of standard ldapsearch'es.
BUT, I had problems in the past when replicating based on ACLs: There might be scenarios - though I never had time to test exhaustively - where replication stalls (I even had some crashes) when the consumer binddn had -inadvertently- partial only privileges on some branches of the provider (probably on entry/children pseudo-attrs only). I ceased to have problems when I made sure that there existed *no* privileges *at all* on branches / entries where the consumer binddn should NOT have access (e.g. by explicitly declaring "by <consumer binddn> none").
I would like to invest some time to test such scenarios more (however, not feasible in the immediate future).
Any other info by the developers might be insightful.
Regards, Nick
openldap-technical@openldap.org
-
Howard Chu -
Jeffrey Crawford -
Nick Milas