https://bugs.openldap.org/show_bug.cgi?id=10604

Issue ID: 10604 Summary: Consumer silently loses a contiguous block of entries under sustained write load via standard syncrepl (refreshAndPersist), N-Way Multi-Master Product: OpenLDAP Version: 2.6.10 Hardware: x86_64 OS: Linux Status: UNCONFIRMED Keywords: needs_review Severity: normal Priority: --- Component: backends Assignee: bugs@openldap.org Reporter: dns.spiros@gmail.com Target Milestone: ---

Summary

(Architecture in brief: N-Way Multi-Master with 3 master nodes plus one read-only consumer in a hub-and-spoke arrangement, each node a separate VM with 4GB RAM / 2 vCPU on VMware Workstation - full detail under "Environment" below.) Under a sustained burst of Add operations on a single provider (roughly 500-1000+ operations at a rate of one every 5-50ms), one or more standard syncrepl consumers, in refreshAndPersist mode, silently fail to apply a contiguous block of entries - typically a few dozen, occasionally into the low hundreds. The missing entries are never retried: they are not present in the consumer's database, contextCSN on the consumer nonetheless advances to a value consistent with having processed them, and no error is logged. The condition has been observed to persist indefinitely until the affected slapd instance is restarted, at which point a full resync corrects it. The same underlying symptom (entries reported by a consumer as "<csn> not new enough, ignored" that were, per direct inspection, never actually applied by that consumer) has also been reproduced in a scenario involving an unclean shutdown ( kill -9 ) of two of three masters mid-write in an N-Way Multi-Master mesh, with one provider (captured live in its own log, loglevel: sync ) responding "nothing changed" / an empty-cookie refreshDelete to three simultaneous consumer sessions while its own local entry count was itself well behind the true dataset size.

We are not certain of the exact internal root cause. We are reporting this because the minimal reproduction below is clean, small, and - in our testing - reliably triggers the symptom at or above roughly 1000 sequential Add operations on a single provider/single consumer pair, with no crash, no network impairment, and no other variable we have been able to identify as causal (see "Ruled out" below).

Environment

slapd version: 2.6.10+dfsg-0ubuntu0.24.04.1 (Ubuntu 24.04 LTS package: slapd -VV → @(#) $OpenLDAP: slapd 2.6.10+dfsg-0ubuntu0.24.04.1 (Sep 23 2025 16:26:39) $ ) Note on distribution packaging: this is the Ubuntu-maintained package, not a build from upstream OpenLDAP source. We are aware that issues specific to a distributor's packaging are generally referred back to the distributor, and we have not yet reproduced this against a vanilla build from OpenLDAP Git/release source. We are reporting it here first because the symptom appears to sit in core slapo-syncprov / syncrepl logic rather than anything Ubuntu-specific in the packaging, but we cannot rule out a packaging-specific contribution and would welcome guidance on whether a from-source reproduction is a prerequisite before this can be considered further. Backend: mdb (LMDB) Topology: N-Way Multi-Master, 3 provider/master nodes + 1 olcReadOnly: TRUE consumer-only node, in a hub-and-spoke arrangement (full mesh - refreshAndPersist between every pair of masters - with the read-only node as a spoke carrying three independent syncrepl directives, one toward each master) Hardware: 4 separate virtual machines (one per node), each 4GB RAM / 2 vCPU, on VMware Workstation Replication mode: standard syncrepl , type=refreshAndPersist . delta-syncrepl is deliberately not used. TLS/auth: mutual TLS, bindmethod=sasl saslmech=EXTERNAL , certificate CN mapped via olcAuthzRegexp to a per-node identity DN.

Relevant olcSyncRepl directive (one per peer, rid varies):

olcSyncRepl: rid=NNN provider=ldaps://<peer>:636 bindmethod=sasl saslmech=EXTERNAL searchbase="<basedn>" type=refreshAndPersist retry="5 10 10 30 60 120 300 +" timeout=5 network-timeout=5 keepalive=10:5:2 tls_cacert=<path> tls_cert=<path> tls_key=<path tls_reqcert=demand attrs="*,+" exattrs="memberOf"

slapo-syncprov overlay parameters:

olcSpSessionlog: 2000000 olcSpCheckpoint: 1000 10

(tested at both the above and a much tighter 50 1 - see "Ruled out") Other overlays in the stack (order matters, listed innermost first): memberof (index {1}, olcMemberOfRefInt: FALSE , exattrs="memberOf" on all syncrepl directives so it is computed locally and not replicated), refint (index {2}, referential integrity on member / uniqueMember / manager / owner , explicitly not on memberOf ), nestgroup (index {3}, search-time only, no writes), ppolicy (index {4}). syncprov sits outermost (index {0}). Search/size limits: verified unlimited both per-identity ( olcLimits matching the mTLS peer DNs used by syncrepl ) and at the frontend database level ( olcSizeLimit / olcTimeLimit on olcDatabase= {-1}frontend,cn=config ). Ruled out as a factor (see below).

Minimal reproduction

This is the cleanest form in which we have triggered the symptom: 1. A single provider (one master), a single persistent refreshAndPersist consumer. The other two masters in the mesh are stopped for this reproduction, to remove any multi-provider interaction as a variable. 2. Database freshly initialized/empty on both sides (a clean redeploy, not a preexisting dataset).

3. A namespace never used in any previous run (a fresh, timestamp-based DN prefix each run), to rule out any interaction with residual state or CSN history from prior test iterations. 4. From the provider, over a single reused connection ( ldapadd -x -c ), add N entries of class inetOrgPerson , one every 5ms (results identical at 50ms - see "Ruled out"):

dn: uid=<prefix>.<i>,ou=People,<basedn> objectClass: top objectClass: inetOrgPerson uid: <prefix>.<i> cn: Test User <i> sn: Test userPassword: {SSHA}...

5. Wait for replication to settle (well past any configured retry interval), then count entries matching uid=<prefix>.* independently on the provider and on the consumer ( slapcat -a , not ldapsearch , to rule out any client-side search limit). Result across repeated runs: N=500 converges correctly in most runs, but not all (see "Notes on determinism" below). N=1000 and above have, in our testing, shown the symptom in the large majority of runs. When it occurs, the consumer is missing a contiguous range of entries (e.g. entries 474-504 of 2000 in one run; 1584-1604 of 2000 in a different run on a different consumer node; 299-320 of 1000 in a run using the never-before-used-namespace variant above). The size and position of the missing range vary between runs; its contiguity does not.

Direct log evidence

1. A consumer receiving, but never applying, an entry - later resolved by a different peer within the same run (this run happened to be an N-Way mesh, so a second delivery path existed; in the single-provider minimal reproduction above, no such second path exists):

syncrepl_message_to_entry: rid=101 DN: uid=crash.user.474,ou=people,dc

(no corresponding be_add / syncrepl_entry follows for rid=101 at this point - the message is received and converted, but not applied) Several minutes later, in the same log, from a different provider session on the same consumer:

syncrepl_message_to_entry: rid=103 DN: uid=crash.user.474,ou=people,dc syncrepl_entry: rid=103 uid=crash.user.474,ou=people,dc=example,dc=com syncrepl_entry: rid=103 be_add uid=crash.user.474,ou=people,dc=example do_syncrep2: rid=101 CSN too old, ignoring 20260921191238.368843Z#0000 do_syncrep2: rid=102 CSN too old, ignoring 20260921191238.368843Z#0000

2. A provider reporting "nothing changed" to three simultaneous consumers, including one whose own dataset was independently confirmed (via direct slapcat on that consumer, at the same moment) to be missing 163 of 500 expected entries:

conn=1000 op=1 syncprov_op_search: no change, skipping log replay conn=1000 op=1 syncprov_op_search: nothing changed, finishing up initi conn=1000 op=1 syncprov_sendinfo: refreshDelete cookie= conn=1001 op=1 syncprov_op_search: no change, skipping log replay conn=1001 op=1 syncprov_op_search: nothing changed, finishing up initi conn=1001 op=1 syncprov_sendinfo: refreshDelete cookie= conn=1002 op=1 syncprov_op_search: no change, skipping log replay conn=1002 op=1 syncprov_op_search: nothing changed, finishing up initi conn=1002 op=1 syncprov_sendinfo: refreshDelete cookie=

conn=1002 in this excerpt authenticated as the read-only consumer node (certificate CN mapped accordingly); the provider told it nothing had changed, and issued a refreshDelete with an empty cookie value. At the same wall-clock moment, slapcat -a run directly and locally on the provider itself returned 337 entries against an expected 500. The consumer's subsequent REFRESH_DELETE phase, taking the provider's "nothing changed / here is nothing" at face value, deleted entries the consumer had correctly received from a different, further-along provider - i.e. the incomplete provider's state propagated outward and actively destroyed correct data elsewhere in the mesh, rather than merely failing to advance it.

Ruled out

The following were each tested directly and found not to be causal: Unclean shutdown / kill -9 : the minimal reproduction above involves no crash of any kind. (A related, but distinct, amplification of the symptom has been observed specifically following kill -9 of two masters in a 3-master mesh - see log evidence #2 above - but the base symptom reproduces without any crash at all.) Network conditions: reproduces at full LAN speed with no induced latency/loss. A WAN emulation profile (25ms±1ms one-way, 0.1% loss via netem ) was used in some runs and disabled in others, with the same outcome either way. ss -ti on a stalled consumer-side connection during one incident showed a healthy TCP session (no retransmits, app_limited , RTT sub-millisecond) - the stall was not at the TCP layer. olcThreads : lowered from the default to 4 on all nodes; no change in outcome. Connection-level timeouts: timeout and network-timeout on the syncrepl directive (both apply only to initial connect/Bind, per slapd-config(5) , not to an established session); olcWriteTimeout (set to 30s server-wide); tcp-user-timeout was considered but not pursued once the TCP-layer health above was confirmed. None of the above affected the outcome. Clock skew: wall-clock time compared across all four nodes, dispatched in parallel via backgrounded SSH; maximum observed spread 123ms, most of which is attributable to SSH connection overhead itself. Reuse of DN namespace across test runs: reproduces with a namespace prefix that has never been used in any prior run (see "Minimal reproduction" above). olcLimits /search size limits: verified unlimited , both per-identity (the mTLS peer DNs used by the syncrepl sessions themselves) and at olcDatabase={-1}frontend,cn=config (i.e. cluster-default). No change in outcome with either configuration. Size of preexisting dataset: reproduces identically against a freshly initialized, otherwise-empty database.

Write rate: 5ms and 50ms between successive Add operations produced the same outcome at the same volume thresholds. syncprov-checkpoint interval: tightened from 1000 10 to 50 1 with the affected pair of masters restarted cleanly beforehand; this did not measurably change the outcome across repeated runs. (We had hypothesized, based on the documented behavior that contextCSN is updated in memory on every write but only persisted to the on-disk checkpoint periodically or on clean shutdown, that an unclean shutdown recovering from a stale on-disk checkpoint might explain the crash- correlated variant specifically. Direct testing did not support this; we record the negative result for completeness.)

Notes on determinism

The symptom is not 100% deterministic at any volume we tested. At 2000 sequential Add operations it has occurred in the substantial majority of our runs; at 1000 in many but not all; at 500 only in a minority of runs, with most runs at that volume converging correctly and instantly (sub-second). We have not identified what distinguishes a failing run from a succeeding one at the same volume and otherwise-identical configuration; every variable we varied to try to explain this (see "Ruled out") produced no change we could attribute to it with confidence. We flag this explicitly because it is the main reason we have not been able to narrow this report further before submitting it.

Related discussions we are aware of

An openldap-technical mailing list thread from 2008-2009 describes what appears to be the same class of symptom - entries reported by a consumer as superseded ("too old") that were, in fact, never applied - attributed there to out-of-order commit-queue processing on the provider under concurrent writes, with a partial fix referenced against 2.4.11/2.4.16 and a further related ITS (#6619) mentioned as still open as of a 2011 follow-up. We have not been able to confirm whether that specific issue is fully resolved in 2.6.x or whether what we are seeing is a distinct, related condition. ITS #9538 (April 2021, "Accesslog entryCSN ordering is not always monotonous") documents non-monotonic entryCSN ordering under

concurrent operations as a live concern in a more recent timeframe. We are aware of, and our architecture already follows, the guidance (Quanah Gibson-Mount, openldap-technical , 24 May 2024) that standard syncrepl is the recommended, safer replication mechanism for multi-provider environments in OpenLDAP 2.6+, in preference to delta-syncrepl . We note this because our minimal reproduction above is, if anything, a simpler case than the multi-provider scenario that guidance addresses (a single provider, a single consumer) and so does not appear to be excluded by it.

What we are asking

We are not confident enough in any single hypothesis to assert a root cause, and would welcome guidance on: 1. Whether this is a known, already-tracked issue (our searches of the mailing list and public ITS entries did not turn up an exact match, but we may have missed the right terms). 2. Whether a reproduction against a from-source build of current Git master (rather than the Ubuntu package) is a prerequisite for further triage, and if so we are willing to attempt it. 3. Any specific loglevel combination, beyond stats sync (already enabled throughout our testing), or any debugger/instrumentation approach the team would recommend to capture the provider-side state at the exact moment an entry is dropped, which we have not yet managed to capture directly (all evidence above is inferred from surrounding log lines, not a single line that directly shows the drop occurring). We can provide the complete Ansible automation used to build the test environment and drive these reproductions, full logs from any of the runs referenced above, and are happy to run additional targeted tests against specific hypotheses.