-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 09/07/2010 03:26 PM, andrew.findlay@skills-1st.co.uk wrote:

On Tue, Sep 07, 2010 at 05:09:07AM -0700, Howard Chu wrote:

...

...

We've talked about doing this isolation in the first refresh upon slapd startup. That might still be a good idea.

It would certainly help to keep the apparent promises made by things like the uniqueness overlay. Alternatively you could take the view that the data will converge eventually and that is all that the LDAP standards promise.

We've hit a similar problem and decided to go this way, i.e. allowing the replication to bypass the uniqueness constraints.

In RFC 4533, there are only two claims regarding DIT consistency:

<quote> 1.2 Intended Usage Upon completion of each synchronization stage of the operation, all information to construct a synchronized client copy of the content has been provided to the client [...]. Except for transient inconsistencies due to concurrent operation (or other) processing at the server, the client copy is an accurate reflection of the content held by the server. Transient inconsistencies will be resolved by subsequent synchronization operations.

[...]

This protocol is not intended to be used in applications requiring transactional data consistency. </quote>

There is no claim that the DIT should stay consistent wrt. any structural restrictions during one synchronization operation. It might be worth noting that trying to replicate the data through as one transaction, like Andrew suggested (quoted below), would still have been impossible without a change in the overlay to allow it to pass through.

...

...

Doing it on every refresh seems far more problematic, because without some type of multi-version concurrency control, that means making the server non-responsive until the refresh completes.

That may not be a problem with refresh-and-persist, as in normal circumstances I would expect updates to arrive at the consumer in the same order they hit the supplier (so this bug could not trigger). More difficult for scheduled refresh mode though. Could the consumer server simply write-lock every entry involved in the refresh while it processes the list, and then commit the whole lot in one DB transaction?

Andrew

I have put a preliminary version of patches that modify the unique overlay here ftp://ftp.openldap.org/incoming/ondrej-kuznik-20101109-unique_bypass_v1.tgz

They add a new configuration attribute olcUniqueAllowManageBypass (it is prohibitively long for a name, though) that, if set to TRUE, triggers the uniqueness checks not to be performed if the operation has manage privilegies on the entry. There are three separate patches, configuration code regarding the new attribute, the checks in unique_{add,modify,modrdn} and manpage modifications.

Some things that should be sorted out before this is complete from my point of view: 1. While there might be a way to find out whether an operation comes from replication or is really issued by a rootdn, I have not found any. As a workaround, I used the "manage" privilege to the entry as a trigger, since this privilege already allows to change the structural objectclass of an entry, operation otherwise prohibited by RFC 4512. 2. When performing a modifyDN operation, the privilege is checked against the entry to be changed instead of the new entry. I found this approach more appropriate than modifying the entry returned by overlay_entry_get_ov as I do not yet understand the implications of doing that and do not know whether it is possible and ok to create a phony entry by hand just for the call to access_allowed. This is clearly marked as a "FIXME" comment in the patch. 3. The entry attribute used for acl checking might be too broad. However, deriving the attributes needed for each domain seemed too complex. At least when this is the first version of a patch for which I have no indication whether even the general idea is considered worth pursuing. 4. The code is similar in all three cases so the first attempt was to move it into unique_search, but there is not enough information available there and would have to be provided by each of the three functions anyway.

Howard and/or others, do you consider this solution valid for this ITS? If yes, could you help me address the things that should be sorted out?

Last, but not least, the IPR stuff: The attached modifications to OpenLDAP Software are subject to the following notice: Copyright 2010 Acision Redistribution and use in source and binary forms, with or without modification, are permitted only as authorized by the OpenLDAP Public License.

Regards, Ondrej Kuznik

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.