I have a custom overlay that registers a callback to rewrite attribute values for searches. I noticed that it was not working for refreshAndPersist sync searches for queued responses, and the reason for this is that there are no callbacks registered in syncprov_qtask() [1]. I'd see the initial responses come through with rewritten attributes, but later responses would not be rewritten.
I had some success adding the callbacks back, but it was a bit of a minefield. Before investigating further, I was hoping to understand why the decision was made to not set callbacks in syncprov_qtask(). For now I'm operating under the assumption that this is simply not supported.
So, why aren't callbacks supported for queued sync responses?
I noticed that valsort doesn't handle syncrepl responses [2], and I wonder if the reason is because the results may be inconsistent, as I am experiencing.
[1] https://git.openldap.org/openldap/openldap/-/blob/OPENLDAP_REL_ENG_2_5/serve... [2] https://git.openldap.org/openldap/openldap/-/blob/OPENLDAP_REL_ENG_2_5/serve...
On Fri, Sep 06, 2024 at 06:46:41PM +0000, Hawes, David wrote:
I have a custom overlay that registers a callback to rewrite attribute values for searches. I noticed that it was not working for refreshAndPersist sync searches for queued responses, and the reason for this is that there are no callbacks registered in syncprov_qtask() [1]. I'd see the initial responses come through with rewritten attributes, but later responses would not be rewritten.
I had some success adding the callbacks back, but it was a bit of a minefield. Before investigating further, I was hoping to understand why the decision was made to not set callbacks in syncprov_qtask(). For now I'm operating under the assumption that this is simply not supported.
So, why aren't callbacks supported for queued sync responses?
Hi David, replication scenarios generally assume that the underlying data is sent unchanged, so noone has considered this scenario. To confirm: when you set up e.g. rwm to munge DNs, the entries sent during persist phase are not transformed either?
I noticed that valsort doesn't handle syncrepl responses [2], and I wonder if the reason is because the results may be inconsistent, as I am experiencing.
Feel free to file an ITS with a way to reproduce, if you can propose a patch to achieve this (along with an enhancement to the test suite that makes sure it's tested going forward), that would be much appreciated.
Thanks,
So, why aren't callbacks supported for queued sync responses?
Hi David, replication scenarios generally assume that the underlying data is sent unchanged, so noone has considered this scenario. To confirm: when you set up e.g. rwm to munge DNs, the entries sent during persist phase are not transformed either?
That is the answer I expected, and it makes sense, but sync searches can be used for more than just replication. Overlays can and do modify the underlying data, and I've confirmed this with a simple searchEntryDN rwm-rewriteRule:
===== ldapsearch -b dc=vt,dc=edu -E sync=rp uid=dhawes 1.1 2>/dev/null # extended LDIF # # LDAPv3 # base <dc=vt,dc=edu> with scope subtree # filter: uid=dhawes # requesting: 1.1 #
# dhawes, rwmPeople, vt.edu dn: uid=dhawes,ou=rwmPeople,dc=vt,dc=edu control: 1.3.6.1.4.1.4203.1.9.1.2 false MBUKAQEEEKpW9nz7TBA+jm2HJeiBW80= # SyncState control, UUID aa56f67c-fb4c-103e-8e6d-8725e8815bcd added
# SyncInfo Received: refresh delete # cookie: rid=000,csn=20240910002202.039752Z#000000#000#000000 # refresh done, switching to persist stage
# dhawes, People, vt.edu dn: uid=dhawes,ou=People,dc=vt,dc=edu control: 1.3.6.1.4.1.4203.1.9.1.2 false MEsKAQIEEKpW9nz7TBA+jm2HJeiBW80ENHJpZD 0wMDAsY3NuPTIwMjQwOTEwMDA0NjE0Ljg1OTM5MlojMDAwMDAwIzAwMCMwMDAwMDA= # SyncState control, UUID aa56f67c-fb4c-103e-8e6d-8725e8815bcd modified # cookie: rid=000,csn=20240910004614.859392Z#000000#000#000000
# numResponses: 3 # numEntries: 2 # numPartial: 1 =====
As with my custom overlay, the results are inconsistent, which is my primary concern. If the callbacks were always called or never called for sync searches, I think that would make sense and do away with this inconsistency.
I noticed that valsort doesn't handle syncrepl responses [2], and I wonder if the reason is because the results may be inconsistent, as I am experiencing.
Feel free to file an ITS with a way to reproduce, if you can propose a patch to achieve this (along with an enhancement to the test suite that makes sure it's tested going forward), that would be much appreciated.
Were you specifically referencing valsort here, or callbacks in general?
Thanks, I appreciate the help.
On Tue, Sep 10, 2024 at 12:51:55AM +0000, Hawes, David wrote:
Hi David, replication scenarios generally assume that the underlying data is sent unchanged, so noone has considered this scenario. To confirm: when you set up e.g. rwm to munge DNs, the entries sent during persist phase are not transformed either?
That is the answer I expected, and it makes sense, but sync searches can be used for more than just replication. Overlays can and do modify the underlying data, and I've confirmed this with a simple searchEntryDN rwm-rewriteRule: [...] As with my custom overlay, the results are inconsistent, which is my primary concern. If the callbacks were always called or never called for sync searches, I think that would make sense and do away with this inconsistency.
Yes, I can see that, would you be able to file an ITS?
I noticed that valsort doesn't handle syncrepl responses [2], and I wonder if the reason is because the results may be inconsistent, as I am experiencing.
Feel free to file an ITS with a way to reproduce, if you can propose a patch to achieve this (along with an enhancement to the test suite that makes sure it's tested going forward), that would be much appreciated.
Were you specifically referencing valsort here, or callbacks in general?
More in terms of syncprov, passing the callbacks on the detached op, probably in syncprov_detach_op or similar. I would expect that fixes all overlays.
Thanks,
openldap-technical@openldap.org
-
Hawes, David -
Ondřej Kuzník