https://bugs.openldap.org/show_bug.cgi?id=10653

Issue ID: 10653 Summary: MPR: writes made during a partition are discarded as "not new enough" by the refresh after it heals Product: OpenLDAP Version: 2.6.16 Hardware: x86_64 OS: Linux Status: UNCONFIRMED Keywords: needs_review Severity: normal Priority: --- Component: slapd Assignee: bugs@openldap.org Reporter: dns.spiros@gmail.com Target Milestone: ---

Created attachment 1240 --> https://bugs.openldap.org/attachment.cgi?id=1240&action=edit Attached is a patch, to be applied to the OpenLDAP repository using `git-am`, which contains the reproducer. Details are provided in the comment.

Version: 2.6.10 (also reproduced with 2.6.16 and 2.7.2)

In a multi-provider setup, writes acknowledged by a provider while it is partitioned from the others can be silently lost. When the partition heals and the other providers refresh from it, the refresh first applies a delete with the CSN of the latest delete; the adds and modifies that follow in the same refresh carry older CSNs and are discarded as "not new enough". Afterwards every provider reports the same contextCSN, so the missing changes are never requested again.

Configuration:

- 3 providers, serverID 1, 2 and 3, each one replicating from the other two - back-mdb, plain syncrepl (no accesslog/delta-syncrepl), type=refreshAndPersist, retry="1 +", "multiprovider on" - syncprov with "syncprov-sessionlog 2000000" and "syncprov-checkpoint 1000 10" - no other overlays

Steps:

1. Populate provider 1 and wait until the three databases are identical. 2. Isolate provider 1 from providers 2 and 3, closing the established syncrepl connections and refusing new ones, so the consumers keep retrying until the partition heals. 3. During the partition, add 20 entries on each side, one every 0.5s (probe-A-1 .. probe-A-20 on provider 1, probe-B-1 .. probe-B-20 on provider 2). 4. Still during the partition, on provider 1 delete cn=U1, add a member to cn=G2, delete cn=G3 and rename cn=U4; on provider 2 modify some of the same entries. 5. Add 5 more entries on each side (probe-A-21 .. 25, probe-B-21 .. 25). 6. Heal the partition and wait (120 seconds in my test).

Expected result:

Every write acknowledged on either side reaches all three providers.

Actual result (same on 2.6.10, 2.6.16 and 2.7.2):

- Providers 2 and 3 never receive probe-A-1 .. probe-A-20, i.e. all the adds made on provider 1 before its first delete. probe-A-21 .. 25, written after it, do arrive. Provider 1 has all the entries written on provider 2. - contextCSN is identical on all three providers, so nothing triggers a further resync: 120 seconds later (and hours later, on the larger cluster where I first saw it) the entries are still missing.

Log of provider 2 when its consumer of provider 1 (rid=001) reconnects:

do_syncrep1: rid=001 starting refresh (sending cookie=rid=001,sid=002,csn=20261028085339.694898Z#000000#001#000000;20261028085354.537828Z#000000#002#000000) do_syncrep2: rid=001 LDAP_RES_INTERMEDIATE - SYNC_ID_SET do_syncrep2: rid=001 cookie=rid=001,sid=001,csn=20261028085339.694898Z#000000#001#000000;,delcsn=20261028085353.358697Z#000000#001#000000 syncrepl_del_nonpresent: rid=001 using delcsn=20261028085353.358697Z#000000#001#000000 syncrepl_del_nonpresent: rid=001 be_delete cn=U1,ou=people,dc=example,dc=com (0) ... syncrepl_entry: rid=001 entry 'cn=probe-A-1,ou=people,dc=example,dc=com' csn=20261028085342.792509Z#000000#001#000000 not new enough, ignored ... syncrepl_entry: rid=001 entry 'cn=probe-A-20,ou=people,dc=example,dc=com' csn=20261028085352.788966Z#000000#001#000000 not new enough, ignored

The cookie sent by provider 2 still has the pre-partition CSN for SID 1 (...085339...). The delete set arrives first and is applied with delcsn=...085353.358697, the CSN of the delete of U1. From then on the adds of the same refresh, whose CSNs are between the two (085342 .. 085352), are rejected by the check in syncrepl_entry() against si_cookieState->cs_vals.

Notes:

- Possibly the same problem as ITS#10026 (Refresh handling can skip entries), or another way to trigger it. The change "syncprov: assume context is usually dirty" for ITS#10026, released in 2.6.14, does not change this result: 2.6.16 and 2.7.2 behave like 2.6.10. - If the partition holds the established connections instead of closing them (as an iptables DROP does), no refresh happens and no write is lost; that variant shows a different problem with concurrent delete and modify, which I am reporting separately. - I first saw this on a production-like cluster (3 providers and 1 read-only consumer, mTLS, SASL EXTERNAL, memberof and refint overlays), where 20-22 writes made on one provider during a partition never reached the other two; the setup above is the minimal one that still reproduces it.

Attached is a patch, to be applied to the OpenLDAP repository using `git-am`, which contains the reproducer. Details are provided in the comment.