Hello
I still work on slapo-supann, an overlay that generates attributes from user-supplied attributes.
On a successful modify or add operation, I start an internal modify operation to add the generated attributes. It works fine, except that syncrepl sometimes miss this second operation: it only catch the first one. If I modify the entry again, it will get the missed second modification.
This is because my internal modify operation does not update entryCSN. syncrepl has no way to know there was a second update. But why entryCSN fails to be updated?
If I add op2.orm_no_opattrs = 0 and call slap_get_csn(), all operative attributes are updated. This probably fixes my problem, but modifiersName is changed to root, I preferd when I had the original user of the fist modify operation.
If I do not set op2.orm_no_opattrs = 0, but I call slap_get_csn() and I perform a Modification like fhis, then only entryCSN is updated. mods->sml_op = LDAP_MOD_REPLACE; mods->sml_flags = SLAP_MOD_INTERNAL; mods->sml_desc = slap_schema.si_ad_entryCSN; mods->sml_numvals = 1; mods->sml_values = csn;
I was happy about it, but the discovered that syncrepl consumer still misses the second modification, even if the master still has a more recent entryCSN than the replica. There must be a mising bit somewhere.
Any hint?
On Wed, May 27, 2026 at 04:09:33PM +0000, Emmanuel Dreyfus wrote:
I was happy about it, but the discovered that syncrepl consumer still misses the second modification, even if the master still has a more recent entryCSN than the replica. There must be a mising bit somewhere.
Following-up on myself...
I can force the sync by making another modification to the entry, or by restarting the replica.
The replica's syncrepl consumer is configured as refreshAndPersist. Once the initial synchronisation is done, I understand the master is supposed to push the changes to the replica. It seems it fails to do so with my internal modification.
Emmanuel Dreyfus wrote:
On Wed, May 27, 2026 at 04:09:33PM +0000, Emmanuel Dreyfus wrote:
I was happy about it, but the discovered that syncrepl consumer still misses the second modification, even if the master still has a more recent entryCSN than the replica. There must be a mising bit somewhere.
Following-up on myself...
I can force the sync by making another modification to the entry, or by restarting the replica.
The replica's syncrepl consumer is configured as refreshAndPersist. Once the initial synchronisation is done, I understand the master is supposed to push the changes to the replica. It seems it fails to do so with my internal modification.
Either your internal modification is somehow bypassing the overlay stack, or you have a flag set on the op that makes syncprov ignore it.
On Thu, May 28, 2026 at 02:25:31PM +0100, Howard Chu wrote:
Either your internal modification is somehow bypassing the overlay stack, or you have a flag set on the op that makes syncprov ignore it.
I am in a search callback that gets op and rs, I do this: Operation op2 = *op; SlapReply rs2 = { REP_RESULT };
cb.sc_response = supann_update_done_cb; cb.sc_private = NULL; op2.o_callback = &cb; op2.o_dn = op->o_bd->be_rootdn; op2.o_ndn = op->o_bd->be_rootndn; op2.o_protocol = LDAP_VERSION3; op2.o_tag = LDAP_REQ_MODIFY; op2.o_time = slap_get_time(); op2.o_req_dn = rs->sr_entry->e_name; op2.o_req_ndn = rs->sr_entry->e_name; op2.o_bd = select_backend(&op2.o_req_ndn, 1); /* populate op2.orm_modlis */
rc = op2.o_bd->be_modify(&op2, &rs2);
On Thu, May 28, 2026 at 02:25:31PM +0100, Howard Chu wrote:
Either your internal modification is somehow bypassing the overlay stack, or you have a flag set on the op that makes syncprov ignore it.
I ran with -d sync and I now wonder if the problem happens on the replica. On the master, two entryCSN are generated and processed.
The replica process both of them, but on the second it considers there were no changes.
6a1afc01.3928bcda 0x7a87752b7000 do_syncrep2: rid=217 cookie=rid=217,csn=20260530150225.941996Z#000000#000#000000 6a1afc01.3929495f 0x7a87752b7000 syncrepl_message_to_entry: rid=217 DN: uid=stemplar,o=org, UUID: e024968e-9f46-102c-990c-eda01653fa3a 6a1afc01.392fcbdb 0x7a87752b7000 syncrepl_entry: rid=217 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY) csn=20260530150225.941996Z#000000#000#000000 tid 0x7a87752b7000 6a1afc01.3931398a 0x7a87752b7000 syncrepl_entry: rid=217 be_search (0) 6a1afc01.393190b2 0x7a87752b7000 syncrepl_entry: rid=217 uid=stemplar,o=org 6a1afc01.39319c47 0x7a87752b7000 slap_queue_csn: queueing 0x7a8773eb7810 20260530150225.941996Z#000000#000#000000 6a1afc02.02efb83c 0x7a87752b7000 slap_graduate_commit_csn: removing 0x7a8773eb7810 20260530150225.941996Z#000000#000#000000 6a1afc02.02f066a7 0x7a87752b7000 syncrepl_entry: rid=217 be_modify uid=stemplar,o=org (0) 6a1afc02.02f1801d 0x7a87752b7000 slap_queue_csn: queueing 0x7a8773eb7770 20260530150225.941996Z#000000#000#000000 6a1afc02.03164918 0x7a87752b7000 slap_graduate_commit_csn: removing 0x7a8773eb7770 20260530150225.941996Z#000000#000#000000 6a1afc02.03174f8e 0x7a87752b7000 do_syncrep2: rid=217 etime=0.092888 6a1afc02.031ea3fc 0x7a87752b8c00 do_syncrep2: rid=217 cookie=rid=217,csn=20260530150225.958055Z#000000#000#000000 6a1afc02.031fd89d 0x7a87752b8c00 syncrepl_message_to_entry: rid=217 DN: uid=stemplar,o=org, UUID: e024968e-9f46-102c-990c-eda01653fa3a 6a1afc02.03270ec0 0x7a87752b8c00 syncrepl_entry: rid=217 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY) csn=20260530150225.958055Z#000000#000#000000 tid 0x7a87752b8c00 6a1afc02.0328edd9 0x7a87752b8c00 dn_callback : entries have identical CSN uid=stemplar,o=org 20260530150225.941996Z#000000#000#000000 6a1afc02.032a0579 0x7a87752b8c00 syncrepl_entry: rid=217 be_search (0) 6a1afc02.032a8f24 0x7a87752b8c00 syncrepl_entry: rid=217 uid=stemplar,o=org 6a1afc02.032b1809 0x7a87752b8c00 slap_queue_csn: queueing 0x7a87752948b0 20260530150225.958055Z#000000#000#000000 6a1afc02.032c0fb8 0x7a87752b8c00 syncrepl_entry: rid=217 entry unchanged, ignored (uid=stemplar,o=org) 6a1afc02.032cfe31 0x7a87752b8c00 slap_graduate_commit_csn: removing 0x7a87752948b0 20260530150225.958055Z#000000#000#000000 6a1afc02.032e6d2a 0x7a87752b8c00 slap_queue_csn: queueing 0x7a8775294810 20260530150225.958055Z#000000#000#000000 6a1afc02.034699d9 0x7a87752b8c00 slap_graduate_commit_csn: removing 0x7a8775294810 20260530150225.958055Z#000000#000#000000 6a1afc02.0350f04c 0x7a87752b8c00 do_syncrep2: rid=217 etime=0.003303
The second entryCSN is ignored. Searching the entry, on master: entryCSN: 20260530150225.958055Z#000000#000#000000
And on replica: entryCSN: 20260530150225.941996Z#000000#000#000000 I checked the syncrepl bindn user can read the complete entry from the master.
On Sat, May 30, 2026 at 03:42:33PM +0000, Emmanuel Dreyfus wrote:
I ran with -d sync and I now wonder if the problem happens on the replica.
Running tcpdump, I can check the two changes flowing from master to replica. The entires sent are identical, the only thing that changes is the csn at the end of the packet. 0x03c0: b597 4904 3472 6964 3d32 3137 2c63 736e ..I.4rid=217,csn 0x03d0: 3d32 3032 3630 3533 3131 3434 3230 382e =20260531144208. - 0x03e0: 3736 3631 3935 5a23 3030 3030 3030 2330 766195Z#000000#0 + 0x03e0: 3738 3735 3730 5a23 3030 3030 3030 2330 787570Z#000000#0 0x03f0: 3030 2330 3030 3030 30 00#000000
Hence the client behavior is fine, and master's behavior is the suspect again.
plain syncrepl and delta-syncrepl produce the same result, but delta-syncrepl lets me inspect the accesslog: the second modification is not recorded.
I now wonder if it is really legit to create a new modify operation in a search callback using this method: Operation op2 = *op; op2.o_dn = op->o_bd->be_rootdn; op2.o_ndn = op->o_bd->be_rootndn; op2.o_tag = LDAP_REQ_MODIFY; op2.o_time = slap_get_time(); (...)
It is used in several overlays, but perhaps only for not replicated data?
-
Emmanuel Dreyfus -
Howard Chu