https://bugs.openldap.org/show_bug.cgi?id=10607
Issue ID: 10607
Summary: slapd: use-after-free in
config_back_delete/_modify/_modrdn with concurrent
cn=config writes
Product: OpenLDAP
Version: 2.6.13
Hardware: x86_64
OS: Linux
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: coding(a)markus-mazurczak.de
Target Milestone: ---
Created attachment 1217
--> https://bugs.openldap.org/attachment.cgi?id=1217&action=edit
Patch: re-resolve config entry after pause and write lock (against 2.6.13, -p1)
SUMMARY
=======
back-config resolves the target CfEntryInfo *before* it pauses the server.
Another cn=config write runs to completion inside that window and frees the
entry, so the first operation continues on freed memory. With two clients
writing to cn=config concurrently, slapd reliably crashes with SIGSEGV.
This is not a theoretical race. It is the blocking defect for using cn=config
as the control plane of a multi-tenant directory service: every tenant we
create or remove is a handful of cn=config writes, and two overlapping
provisioning flows take the server down.
ANALYSIS
========
config_back_modify, config_back_modrdn and config_back_delete all do:
ce = config_find_base( cfb->cb_root, &op->o_req_ndn, &last, op );
... cheap validation on ce ...
slap_pause_server();
ldap_pvt_thread_rdwr_wlock( &cfb->cb_rwlock );
... use ce ...
In openldap-2.6.13 (servers/slapd/bconfig.c):
function config_find_base slap_pause_server wlock
config_back_modify 6364 6439 6446
config_back_modrdn 6525 6668 6675
config_back_delete 6791 6825 6828
Between the lookup and the pause, this thread holds no lock on the config
tree. A second cn=config write that is already past its own pause completes,
reaches
ce->ce_entry->e_private = NULL;
entry_free( ce->ce_entry );
ch_free( ce ); /* bconfig.c:6946 */
and the first thread then dereferences the freed CfEntryInfo at bconfig.c:6835.
config_back_add is not affected: it resolves the entry after the lock, inside
config_add_internal.
The read paths are correct -- config_back_search and config_back_compare take
cb_rwlock for reading *before* calling config_find_base.
Note that the thread pool itself is fine. PAUSE_ARG(DO_PAUSE) subtracts
ltp_pause, so a second pauser first goes GO_IDLE, waits out the running pause
and only then pauses; assert(!pool->ltp_pause) in handle_pause() holds. The
problem is purely the stale pointer taken before the pause.
REPRODUCTION
============
Build slapd with AddressSanitizer (glibc; ASan does not work under musl):
CFLAGS="-g -O1 -fno-omit-frame-pointer -fsanitize=address" \
LDFLAGS="-fsanitize=address" \
./configure --enable-mdb=yes --enable-dynlist=yes --enable-spasswd \
--with-cyrus-sasl --with-tls=openssl --enable-crypt
make depend && make
Start slapd with a cn=config database, then run four concurrent loops that
each add an olcDatabase=mdb entry, add an olcOverlay=dynlist child, delete
olcRootPW, and delete both again. The crash appears within seconds.
AddressSanitizer output:
==15==ERROR: AddressSanitizer: heap-use-after-free
READ of size 4 at 0x5060000681a0 thread T11
#0 config_back_delete servers/slapd/bconfig.c:6835
#1 fe_op_delete servers/slapd/delete.c:181
#2 do_delete servers/slapd/delete.c:95
#3 connection_operation servers/slapd/connection.c:1137
freed by thread T9 here:
#0 free
#1 ber_memfree_x libraries/liblber/memory.c:152
#2 ch_free servers/slapd/ch_malloc.c:139
#3 config_back_delete servers/slapd/bconfig.c:6946
previously allocated by thread T2 here:
#0 calloc
#1 ber_memcalloc_x libraries/liblber/memory.c:283
#2 ch_calloc servers/slapd/ch_malloc.c:104
#3 config_add_internal servers/slapd/bconfig.c:5615
#4 config_back_add servers/slapd/bconfig.c:5849
Without the sanitizer, the stock 2.6.13 build dies with SIGSEGV after roughly
175 add/delete cycles under the same load. With the patch below applied, the
same load ran 14402 cycles in 150 seconds with no crash and a clean ASan run.
PATCH
=====
Resolve the entry again once the pause and the write lock are held, and answer
noSuchObject if it is gone. The pre-pause lookup is kept, so a request that is
going to fail anyway still does not pause the whole server.
The patch is attached as
0001-cn-config-resolve-entry-after-pause.patch and applies with -p1 to
openldap-2.6.13.
For config_back_modrdn the re-resolved entry's ce_type is compared with the
validated one; ixold is derived from op->o_req_ndn, so it stays consistent.
I am happy to adjust the approach -- taking cb_rwlock for reading around the
pre-pause validation as well would close the (much narrower) window in which
the early checks themselves touch a freed entry.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10605
Issue ID: 10605
Summary: slapo-auditlog creates the log file with insecure
permissions (0666)
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: overlays
Assignee: bugs(a)openldap.org
Reporter: dns.spiros(a)gmail.com
Target Milestone: ---
Created attachment 1216
--> https://bugs.openldap.org/attachment.cgi?id=1216&action=edit
patch implementing the described change
The auditlog overlay opens its log file with mode 0666:
fd = open(ad->ad_logfile, flags, 0666);
Combined with a typical process umask (e.g. 022), this commonly results in
a file that is world-readable (0644) - and, with a permissive umask, can
remain world-writable as literally requested.
This is a real concern for two independent reasons:
1. Confidentiality: the audit log records the full LDIF of every Add/
Modify/Delete/ModRDN operation, including attribute values. In our own
testing, this included userPassword hashes (SSHA) appearing in clear
LDIF form inside the log file. A world-readable audit log exposes this
to any local user, regardless of the LDAP-level ACLs protecting the
same data inside the directory itself.
2. Integrity: mode 0666 as requested permits world-write. An audit trail
that any local user can modify does not reliably serve its purpose -
tampering or truncation by an unprivileged user should not be possible.
Standard security guidance for audit logging (CIS Benchmarks, DISA STIG,
PCI-DSS Requirement 10.5, ISO/IEC 27001 Annex A.12.4) consistently
requires that audit logs be protected from unauthorized read and write
access - not left to whatever the deploying administrator's umask
happens to be.
Attached is a patch (plain git diff) changing the requested mode to 0640
(owner read/write, group read, no access for others), hardcoded rather
than left to umask, consistent with how other security-sensitive files
(e.g. private keys) are typically created.
Note: this is a behavior change for any existing deployment relying on
broader access to the raw log file (e.g. a non-group member reading it
directly) - such setups would need to adjust group membership or ACLs
accordingly after upgrading.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10606
Issue ID: 10606
Summary: LDAP_OPT_DEFBASE can't be reset to NULL
Product: OpenLDAP
Version: 2.6.14
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: libraries
Assignee: bugs(a)openldap.org
Reporter: ondra(a)mistotebe.net
Target Milestone: ---
If there is no ldo_defbase on the global context, it's impossible to unset a
connection's defbase to the default (libldap tries to strdup the NULL pointer
and checks for NULL, but that's the correct value here).
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10593
Issue ID: 10593
Summary: VLV response formatting can read past a stack buffer
and crash ldapsearch
Product: OpenLDAP
Version: 2.7.1
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: client tools
Assignee: bugs(a)openldap.org
Reporter: simon.pichugin(a)gmail.com
Target Milestone: ---
Created attachment 1209
--> https://bugs.openldap.org/attachment.cgi?id=1209&action=edit
full asan and search results
print_vlv() formats the server-provided VLV contextID into a fixed stack
buffer. If snprintf() truncates the output, its required length is passed to
the LDIF writer, which reads past the end of the buffer.
We reproduced this with an unmodified ldapsearch that explicitly requested
server-side sorting on cn and a VLV window. The server verified both request
controls, returned one valid entry, then returned SSS success and a valid VLV
success response with a large server-defined contextID. A 16 KiB contextID
produced an ASan stack-buffer-overflow. A 64 KiB contextID crashed a
non-sanitized, optimized client with SIGSEGV in all five runs.
This is different from the SSS case. A VLV contextID is an opaque value
selected by the server, and the VLV specification does not set a size limit.
The triggering response does not need to mismatch caller input or violate the
defined control sequence. The client must opt into VLV, and a 64 KiB contextID
is unusual.
Short ASan trace
ERROR: AddressSanitizer: stack-buffer-overflow
READ of size 1
#0 ldif_sput_wrap libraries/libldap/ldif.c:622
#1 ldif_put_wrap libraries/libldap/ldif.c:705
#2 tool_write_ldif clients/tools/common.c:2791
#3 print_vlv clients/tools/common.c:2264
#4 tool_print_ctrls clients/tools/common.c:2781
SUMMARY: AddressSanitizer: stack-buffer-overflow libraries/libldap/ldif.c:622
in ldif_sput_wrap
Short native trace
Program received signal SIGSEGV, Segmentation fault.
#0 ldif_sput_wrap libraries/libldap/ldif.c:579
#1 ldif_put_wrap libraries/libldap/ldif.c:705
#2 tool_write_ldif clients/tools/common.c:2791
#3 print_vlv clients/tools/common.c:2264
#4 tool_print_ctrls clients/tools/common.c:2781
#5 print_result clients/tools/ldapsearch.c:2426
#6 dosearch clients/tools/ldapsearch.c:1840
#7 main clients/tools/ldapsearch.c:1550
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10597
Issue ID: 10597
Summary: slapo-accesslog frees a shared `entryUUID` value from
operations that do not hold the mutex guarding it,
causing a double free and heap corruption. Introduced
in 2.6.14 by the fix for ITS#10482.
Product: OpenLDAP
Version: 2.6.14
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: overlays
Assignee: bugs(a)openldap.org
Reporter: bohdan.kmit(a)kiteworks.com
Target Milestone: ---
Created attachment 1212
--> https://bugs.openldap.org/attachment.cgi?id=1212&action=edit
Patch
# Symptom
`slapd` terminates with SIGSEGV under concurrent write and search load. glibc
reports the corruption at whatever the process allocates next, so the message
varies:
```
double free or corruption (out)
malloc(): unaligned fastbin chunk detected
malloc(): unaligned tcache chunk detected
free(): invalid pointer
```
Of ten cores collected from one instance, nine abort inside `malloc()` or
`calloc()` at unrelated allocation sites and carry no information about the
origin. One caught the faulting free itself:
```
#4 free ()
#5 accesslog_response (op=<optimized out>, rs=...) at accesslog.c:2034
#6 ...
#8 slap_send_ldap_result ()
```
(line 2034 in 2.6.15; the equivalent is 2176 in current master.)
# Analysis
`accesslog_response()` ends with:
```c
skip:
if ( !BER_BVISNULL( &li->li_uuid ) ) {
ber_memfree( li->li_uuid.bv_val );
BER_BVZERO( &li->li_uuid );
}
if ( lo->mask & LOG_OP_WRITES ) {
/* We haven't transitioned to li_log_mutex yet */
ldap_pvt_thread_mutex_unlock( &li->li_op_rmutex );
}
return SLAP_CB_CONTINUE;
```
`li_uuid` is a field of the shared `log_info` instance, not of the operation.
Access to it is asymmetric:
* Write operations (`bi_op_add`, `bi_op_delete`, `bi_op_modify`, `bi_op_modrdn`
→ `accesslog_op_mod()`) acquire `li_op_rmutex` and populate `li_uuid` with
`ber_dupbv()` while holding it. They still hold it on entry to
`accesslog_response()`, free under it at `skip:`, and release it immediately
after. This is correct.
* Non-write operations (`bi_op_bind`, `bi_op_compare`, `bi_op_search`,
`bi_extended` → `accesslog_op_misc()`) take no lock at all.
`accesslog_response()` acquires `li_op_rmutex` for them only part-way through
the function, and **all four `goto skip` sites precede that acquisition**
(master: 1656, 1661, 1671, 1683 versus the lock at 1693). Such an operation
therefore reaches `skip:` holding no lock, and frees a pointer it never set.
The `goto skip` conditions reachable this way are:
| site | condition |
|------|-----------------------------------------------------|
| 1656 | log database absent or not open |
| 1661 | `op->o_dont_replicate` |
| 1671 | `li_success` configured and the operation failed |
| 1683 | operation not in `li_ops` and no matching `logbase` |
Two consequences follow:
1. Two non-write operations completing concurrently both observe a non-NULL
`li_uuid` and both call `ber_memfree()` on it — a double free.
2. A non-write operation can free it in the window between a write operation
storing the value and that write operation taking ownership of it (master
1716–1718, under both mutexes). The write path then works with, and frees, a
dangling pointer.
Either way the allocator's free lists are corrupted, and the failure surfaces
later at an unrelated allocation, which is why almost every core points
somewhere innocent.
The exposure depends strongly on configuration. With `olcAccessLogOps: writes`,
searches are not in `li_ops`, so **every search** fails the test at 1673 and
takes `goto skip` at 1683. A read-heavy workload therefore executes the
unlocked free at close to search rate. Note that 1683 is unreachable for write
operations, since `accesslog_op_mod()` applies the same test before registering
the callback.
Access to the tree of related state is otherwise correct: every use of
`li_mincsn`, `li_sids` and `li_numcsns` in `log_old_lookup()`,
`accesslog_purge()`, `accesslog_response()` and `accesslog_db_root()` is
serialised by `li_log_mutex`, and no locked region in the file contains a
`return` or `goto` that bypasses its unlock. `li_uuid` is the only unserialised
mutation.
# Regression
The unconditional free was added by:
```
5c4e7f2f1a ITS#10482 slapo-accesslog: do not leak entryUUID (2026-03-31)
ab4e53e54b same, backported to the 2.6 branch
```
That commit consists solely of those four lines. It plugs a genuine leak but
places the release on a path reachable without the mutex.
# Affected versions
| release | affected |
|--------------|----------|
| 2.6.13 | no |
| 2.6.14 | yes |
| 2.6.15 | yes |
| 2.7.0, 2.7.1 | yes |
| master | yes |
Observed on 2.6.15, x86-64, glibc, back-mdb, with `syncprov` and `accesslog` on
the same database and delta-syncrepl consumers reading the accesslog.
# Reproduction
Configuration: `accesslog` over back-mdb with `olcAccessLogOps: writes` and
`olcAccessLogSuccess: TRUE`.
Load: six concurrent MODIFY streams cycling over four DNs (to maximise
same-target concurrency), two more spread over twenty DNs, one stream of ADDs
of existing entries (returning `entryAlreadyExists`), and twelve concurrent
search streams over the same subtree.
Result: SIGSEGV within **13–24 seconds**, repeatably — effectively one crash
per load application.
Two observations that isolate the mechanism:
* The **same write load with no search streams ran 300 seconds without a
fault.** Searches are required, because only non-write operations take the
unlocked path.
* Instances receiving the identical replicated write stream but carrying **no
local search traffic** ran 68 minutes without a fault, while an instance with
local searches plus the same replicated writes crashed.
`MALLOC_CHECK_` does not help: by the time of the second free the chunk has
typically been handed to another allocation, so it is a valid live chunk and
nothing is flagged at the free.
# Proposed fix
Free the value only on the path that owns it and already holds the mutex:
```diff
skip:
- if ( !BER_BVISNULL( &li->li_uuid ) ) {
- ber_memfree( li->li_uuid.bv_val );
- BER_BVZERO( &li->li_uuid );
- }
if ( lo->mask & LOG_OP_WRITES ) {
+ /* Only this path holds li_op_rmutex, which guards li_uuid */
+ if ( !BER_BVISNULL( &li->li_uuid ) ) {
+ ber_memfree( li->li_uuid.bv_val );
+ BER_BVZERO( &li->li_uuid );
+ }
/* We haven't transitioned to li_log_mutex yet */
ldap_pvt_thread_mutex_unlock( &li->li_op_rmutex );
}
```
Non-write operations never populate `li_uuid`, so they have nothing to release;
the leak ITS#10482 addressed remains fixed for the write path, which is the
only producer. Because every `goto skip` precedes the mutex acquisition, no
path can now reach the label holding the lock without releasing it.
# Related observation, not addressed by this patch
`li_old` has the same missing release. `accesslog_op_mod()` assigns `li->li_old
= entry_dup( e )` without freeing any previous value, `accesslog_response()`
moves it out only on the non-skip path, and neither `skip:` nor
`accesslog_db_destroy()` releases it. A write operation whose response takes
`goto skip` therefore strands an `Entry`, which the next write overwrites and
leaks. This requires `olcAccessLogOld` to be configured and is a leak only — no
unserialised free, so no corruption. Reported for completeness rather than
fixed here.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10598
Issue ID: 10598
Summary: Version 2.7 is both released in the past and a future
release
Product: website
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: website
Assignee: bugs(a)openldap.org
Reporter: dpa-openldap(a)aegee.org
Target Milestone: ---
The roadmap at https://www.openldap.org/software/roadmap.html contains:
Future Minor Releases
OpenLDAP 2.7 (Released August 2026)
2.7 cannot be at the same time a Future Release and Released in August 2026.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10603
Issue ID: 10603
Summary: slapd 2.6.10 aborts in mdb_opinfo_get (id2entry.c:828)
serving syncrepl refresh - thread-cached reader txn
still active, mdb_txn_renew returns EINVAL
Product: OpenLDAP
Version: 2.6.10
Hardware: x86_64
OS: Linux
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: vojtech(a)dusatko.org
Target Milestone: ---
Created attachment 1215
--> https://bugs.openldap.org/attachment.cgi?id=1215&action=edit
thread apply all bt (coredump + gdb)
Three-node multiprovider cluster (olcMultiProvider on {0}config and the data
DB),
back-mdb, replicating one suffix. Overlay stack on the data DB in insertion
order:
auditlog, memberof, refint, syncprov, ppolicy, unique. syncprov had no
checkpoint/sessionlog configured. Consumers use refreshAndPersist, retry "5 20
300 +".
Binary: Debian 13 (trixie) official package slapd 2.6.10+dfsg-1 from
deb.debian.org (build "May 29 2025 23:41:48", Debian OpenLDAP Maintainers);
symbols from slapd-dbgsym 2.6.10+dfsg-1 (debian-debug archive). No local
patches.
Symptom: whenever a provider slapd is (re)started and a consumer reconnects and
begins its refresh (present phase; consumer search is base=<suffix> scope=sub
deref=0 filter=(objectClass=*) attrs=* +), the provider aborts within seconds:
slapd: ../../../../../servers/slapd/back-mdb/id2entry.c:828: mdb_opinfo_get:
Assertion `!rc' failed.
Reproduced 5+ times over two days; effectively every provider restart followed
by
a consumer refresh crashes it. Environment is small and healthy: data.mdb 1.5
MB,
olcDbMaxSize 1 GiB, olcDbMaxReaders default, >20 GB free disk, no other process
opens the LMDB env (verified). The DB had been slapindex'ed offline (slapd
stopped) before the first occurrence; the crash also reproduces on a freshly
started slapd. Loglevel "stats acl filter" during the captured crash.
Analysis from the core: id2entry.c:828 is the mdb_txn_renew() of the thread's
cached reader transaction inside mdb_opinfo_get() (rdonly=1, renew=1). The
cached
txn retrieved via the thread-pool key has mt_flags = 0x20000 (MDB_TXN_RDONLY
set,
MDB_TXN_FINISHED NOT set), i.e. it was still active, not reset - so
mdb_txn_renew() returned EINVAL and the assert fired. The env is healthy:
me_flags = 0x30000000 (MDB_ENV_ACTIVE|MDB_ENV_TXKEY, no MDB_FATAL_ERROR).
The failing call is syncprov's internal findbase search nested on the same
worker
thread under the consumer's refresh search; the fresh stack-local mdb_op_info
(moi_ref = 0, reader flag) borrowed the thread's cached reader txn while it was
still in use - reentrant/unbalanced use of the per-thread cached read
transaction
(an outer op holding it active, or an earlier op missing its mdb_txn_reset).
Relevant locals at frame 4 (mdb_opinfo_get):
renew = 1
data = 0x7f3134105580 (thread-pool cached txn)
moi = 0x7f313b7ebc50 {moi_txn = 0x7f3134105580, moi_ref = 0,
moi_flag = 1}
moi->moi_txn->mt_flags = 0x20000 (RDONLY, not FINISHED ->
mdb_txn_renew EINVAL)
mdb->mi_dbenv->me_flags = 0x30000000 (no MDB_FATAL_ERROR)
Backtrace of the aborting thread (full "thread apply all bt" attached):
#4 mdb_opinfo_get (op=0x7f313b7fc070, mdb=0x7f31811e9010, rdonly=1,
moip=0x7f313b7ebb48) at back-mdb/id2entry.c:828
#5 mdb_search (op=0x7f313b7fc070, rs=0x7f313b7fc000) at back-mdb/search.c:449
#6 overlay_op_walk (op=0x7f313b7fc070, ...) at backover.c:706
#7 over_op_func (op=0x7f313b7fc070, ...) at backover.c:766
#8 syncprov_findbase (op=0x7f3134103be0, fc=0x7f313b7fc330) at
overlays/syncprov.c:530
#9 syncprov_op_search (op=0x7f3134103be0, rs=0x7f313b7fd9a0) at
overlays/syncprov.c:3175
#10 overlay_op_walk (op=0x7f3134103be0, ...) at backover.c:691
#11 over_op_func (op=0x7f3134103be0, ...) at backover.c:766
#12 fe_op_search (op=0x7f3134103be0, ...) at search.c:426
#13 do_search ... at search.c:267
#14 connection_operation ... at connection.c:1115
#15 connection_read_thread ... at connection.c:1267
Note the two distinct op pointers: the consumer's search op (0x7f3134103be0,
frames 8-15) and syncprov's internal findbase op (0x7f313b7fc070, frames 4-7)
on
the same worker thread. At crash time another worker thread was serving an
unrelated cn=config subtree search (config_back_search) and a third was blocked
in syslog(3) writing filter-level debug output, in case concurrency is
relevant.
slapd log tail before one abort:
conn=1355 fd=18 ACCEPT from IP=<consumer-ip>:48480 (IP=<provider-ip>:1389)
conn=1355 op=0 BIND dn="cn=admin,<suffix>" mech=SIMPLE ssf=0
conn=1355 op=1 SRCH base="<suffix>" scope=2 deref=0 filter="(objectClass=*)"
conn=1355 op=1 SRCH attr=* +
slapd: ../../../../../servers/slapd/back-mdb/id2entry.c:828: mdb_opinfo_get:
Assertion `!rc' failed.
Reproduction: restart slapd on a provider; wait for a consumer's syncrepl retry
to reconnect and start refresh (seconds to ~1 min). Ordinary application
traffic
only, no special load.
Timeline note: crashes began after a config deploy that (a) added a
slapo-unique
instance to the overlay stack, (b) added one new eq index followed by an
offline
slapindex, (c) changed loglevel to "stats acl filter". We cannot exclude the
changed overlay stack being a precondition rather than coincidence.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10604
Issue ID: 10604
Summary: Consumer silently loses a contiguous block of entries
under sustained write load via standard syncrepl
(refreshAndPersist), N-Way Multi-Master
Product: OpenLDAP
Version: 2.6.10
Hardware: x86_64
OS: Linux
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: backends
Assignee: bugs(a)openldap.org
Reporter: dns.spiros(a)gmail.com
Target Milestone: ---
Summary
(Architecture in brief: N-Way Multi-Master with 3 master nodes plus one
read-only consumer in a hub-and-spoke arrangement, each node a separate VM with
4GB RAM / 2 vCPU on VMware Workstation - full detail under "Environment"
below.)
Under a sustained burst of Add operations on a single provider (roughly
500-1000+ operations at a rate of one every 5-50ms), one or more standard
syncrepl consumers, in refreshAndPersist mode, silently fail to apply a
contiguous block of entries - typically a few dozen, occasionally into the low
hundreds. The missing entries are never retried: they are not present in the
consumer's database, contextCSN on the consumer nonetheless advances to a value
consistent with having processed them, and no error is logged.
The condition has been observed to persist indefinitely until the affected
slapd instance is restarted, at which point a full resync corrects it.
The same underlying symptom (entries reported by a consumer as "<csn> not new
enough, ignored" that were, per direct inspection, never actually applied by
that consumer) has also been reproduced in a scenario involving an unclean
shutdown ( kill -9 ) of two of three masters mid-write in an N-Way Multi-Master
mesh, with one provider (captured live in its own log, loglevel: sync )
responding "nothing changed" / an empty-cookie refreshDelete to three
simultaneous consumer sessions while its own local entry count was itself well
behind the true dataset size.
We are not certain of the exact internal root cause. We are reporting this
because the minimal reproduction below is clean, small, and - in our testing
- reliably triggers the symptom at or above roughly 1000 sequential Add
operations on a single provider/single consumer pair, with no crash, no network
impairment, and no other variable we have been able to identify as causal (see
"Ruled out" below).
Environment
slapd version: 2.6.10+dfsg-0ubuntu0.24.04.1 (Ubuntu 24.04 LTS package:
slapd -VV → @(#) $OpenLDAP: slapd 2.6.10+dfsg-0ubuntu0.24.04.1 (Sep 23 2025
16:26:39) $ )
Note on distribution packaging: this is the Ubuntu-maintained package,
not a build from upstream OpenLDAP source. We are aware that issues specific to
a distributor's packaging are generally referred back to the distributor, and
we have not yet reproduced this against a vanilla build from OpenLDAP
Git/release source. We are reporting it here first because the symptom appears
to sit in core slapo-syncprov / syncrepl logic rather than anything
Ubuntu-specific in the packaging, but we cannot rule out a packaging-specific
contribution and would welcome guidance on whether a from-source reproduction
is a prerequisite before this can be considered further.
Backend: mdb (LMDB)
Topology: N-Way Multi-Master, 3 provider/master nodes + 1 olcReadOnly:
TRUE consumer-only node, in a hub-and-spoke arrangement (full mesh -
refreshAndPersist between every pair of masters - with the read-only node as a
spoke carrying three independent syncrepl directives, one toward each master)
Hardware: 4 separate virtual machines (one per node), each 4GB RAM / 2
vCPU, on VMware Workstation
Replication mode: standard syncrepl , type=refreshAndPersist .
delta-syncrepl is deliberately not used.
TLS/auth: mutual TLS, bindmethod=sasl saslmech=EXTERNAL , certificate
CN mapped via olcAuthzRegexp to a per-node identity DN.
Relevant olcSyncRepl directive (one per peer, rid varies):
olcSyncRepl: rid=NNN provider=ldaps://<peer>:636 bindmethod=sasl
saslmech=EXTERNAL searchbase="<basedn>" type=refreshAndPersist retry="5 10 10
30 60 120 300 +" timeout=5 network-timeout=5 keepalive=10:5:2 tls_cacert=<path>
tls_cert=<path> tls_key=<path tls_reqcert=demand attrs="*,+" exattrs="memberOf"
slapo-syncprov overlay parameters:
olcSpSessionlog: 2000000
olcSpCheckpoint: 1000 10
(tested at both the above and a much tighter 50 1 - see "Ruled out")
Other overlays in the stack (order matters, listed innermost first):
memberof (index {1}, olcMemberOfRefInt: FALSE , exattrs="memberOf" on all
syncrepl directives so it is computed locally and not replicated), refint
(index {2}, referential integrity on member /
uniqueMember / manager / owner , explicitly not on memberOf ), nestgroup (index
{3}, search-time only, no writes), ppolicy (index {4}). syncprov sits outermost
(index {0}).
Search/size limits: verified unlimited both per-identity ( olcLimits
matching the mTLS peer DNs used by syncrepl ) and at the frontend database
level ( olcSizeLimit / olcTimeLimit on olcDatabase=
{-1}frontend,cn=config ). Ruled out as a factor (see below).
Minimal reproduction
This is the cleanest form in which we have triggered the symptom:
1. A single provider (one master), a single persistent refreshAndPersist
consumer. The other two masters in the mesh are stopped for this reproduction,
to remove any multi-provider interaction as a variable.
2. Database freshly initialized/empty on both sides (a clean redeploy, not
a preexisting dataset).
3. A namespace never used in any previous run (a fresh, timestamp-based DN
prefix each run), to rule out any interaction with residual state or CSN
history from prior test iterations.
4. From the provider, over a single reused connection ( ldapadd -x -c ),
add N entries of class inetOrgPerson , one every 5ms (results identical at 50ms
- see "Ruled out"):
dn: uid=<prefix>.<i>,ou=People,<basedn> objectClass: top
objectClass: inetOrgPerson
uid: <prefix>.<i> cn: Test User <i> sn: Test
userPassword: {SSHA}...
5. Wait for replication to settle (well past any configured retry
interval), then count entries matching uid=<prefix>.* independently on the
provider and on the consumer ( slapcat -a , not ldapsearch , to rule out any
client-side search limit).
Result across repeated runs: N=500 converges correctly in most runs, but not
all (see "Notes on determinism" below). N=1000 and above have, in our testing,
shown the symptom in the large majority of runs. When it occurs, the consumer
is missing a contiguous range of entries (e.g. entries 474-504 of 2000 in one
run; 1584-1604 of 2000 in a different run on a different consumer node; 299-320
of 1000 in a run using the never-before-used-namespace variant above). The size
and position of the missing range vary between runs; its contiguity does not.
Direct log evidence
1. A consumer receiving, but never applying, an entry - later resolved by
a different peer within the same run (this run happened to be an N-Way
mesh, so a second delivery path existed; in the single-provider minimal
reproduction above, no such second path exists):
syncrepl_message_to_entry: rid=101 DN: uid=crash.user.474,ou=people,dc
(no corresponding be_add / syncrepl_entry follows for rid=101 at this point -
the message is received and converted, but not applied)
Several minutes later, in the same log, from a different provider session on
the same consumer:
syncrepl_message_to_entry: rid=103 DN: uid=crash.user.474,ou=people,dc
syncrepl_entry: rid=103 uid=crash.user.474,ou=people,dc=example,dc=com
syncrepl_entry: rid=103 be_add uid=crash.user.474,ou=people,dc=example
do_syncrep2: rid=101 CSN too old, ignoring 20260921191238.368843Z#0000
do_syncrep2: rid=102 CSN too old, ignoring 20260921191238.368843Z#0000
2. A provider reporting "nothing changed" to three simultaneous consumers,
including one whose own dataset was independently confirmed (via direct slapcat
on that consumer, at the same moment) to be missing 163 of 500 expected
entries:
conn=1000 op=1 syncprov_op_search: no change, skipping log replay conn=1000
op=1 syncprov_op_search: nothing changed, finishing up initi conn=1000 op=1
syncprov_sendinfo: refreshDelete cookie=
conn=1001 op=1 syncprov_op_search: no change, skipping log replay conn=1001
op=1 syncprov_op_search: nothing changed, finishing up initi conn=1001 op=1
syncprov_sendinfo: refreshDelete cookie=
conn=1002 op=1 syncprov_op_search: no change, skipping log replay conn=1002
op=1 syncprov_op_search: nothing changed, finishing up initi conn=1002 op=1
syncprov_sendinfo: refreshDelete cookie=
conn=1002 in this excerpt authenticated as the read-only consumer node
(certificate CN mapped accordingly); the provider told it nothing had changed,
and issued a refreshDelete with an empty cookie value. At the same wall-clock
moment, slapcat -a run directly and locally on the provider itself returned 337
entries against an expected 500. The consumer's subsequent REFRESH_DELETE
phase, taking the provider's "nothing changed
/ here is nothing" at face value, deleted entries the consumer had correctly
received from a different, further-along provider - i.e. the incomplete
provider's state propagated outward and actively destroyed correct data
elsewhere in the mesh, rather than merely failing to advance it.
Ruled out
The following were each tested directly and found not to be causal:
Unclean shutdown / kill -9 : the minimal reproduction above involves no
crash of any kind. (A related, but distinct, amplification of the symptom has
been observed specifically following kill -9 of two masters in a 3-master mesh
- see log evidence #2 above - but the base symptom reproduces without any crash
at all.)
Network conditions: reproduces at full LAN speed with no induced
latency/loss. A WAN emulation profile (25ms±1ms one-way, 0.1% loss via netem )
was used in some runs and disabled in others, with the same outcome either way.
ss -ti on a stalled consumer-side connection during one incident showed a
healthy TCP session (no retransmits, app_limited , RTT sub-millisecond) - the
stall was not at the TCP layer.
olcThreads : lowered from the default to 4 on all nodes; no
change in outcome.
Connection-level timeouts: timeout and network-timeout on the syncrepl
directive (both apply only to initial connect/Bind, per slapd-config(5) , not
to an established session); olcWriteTimeout (set to 30s server-wide);
tcp-user-timeout was considered but not pursued once the TCP-layer health above
was confirmed. None of the above affected the outcome.
Clock skew: wall-clock time compared across all four nodes, dispatched
in parallel via backgrounded SSH; maximum observed spread 123ms, most of which
is attributable to SSH connection overhead itself.
Reuse of DN namespace across test runs: reproduces with a namespace
prefix that has never been used in any prior run (see "Minimal reproduction"
above).
olcLimits /search size limits: verified unlimited , both
per-identity (the mTLS peer DNs used by the syncrepl sessions themselves) and
at olcDatabase={-1}frontend,cn=config (i.e. cluster-default). No change in
outcome with either configuration.
Size of preexisting dataset: reproduces identically against a freshly
initialized, otherwise-empty database.
Write rate: 5ms and 50ms between successive Add operations produced the
same outcome at the same volume thresholds.
syncprov-checkpoint interval: tightened from 1000 10 to 50 1
with the affected pair of masters restarted cleanly beforehand; this did not
measurably change the outcome across repeated runs. (We had hypothesized, based
on the documented behavior that contextCSN is updated in memory on every write
but only persisted to the on-disk checkpoint periodically or on clean shutdown,
that an unclean shutdown recovering from a stale on-disk checkpoint might
explain the crash- correlated variant specifically. Direct testing did not
support this; we record the negative result for completeness.)
Notes on determinism
The symptom is not 100% deterministic at any volume we tested. At 2000
sequential Add operations it has occurred in the substantial majority of our
runs; at 1000 in many but not all; at 500 only in a minority of runs, with most
runs at that volume converging correctly and instantly (sub-second). We have
not identified what distinguishes a failing run from a succeeding one at the
same volume and otherwise-identical configuration; every variable we varied to
try to explain this (see "Ruled out") produced no change we could attribute to
it with confidence. We flag this explicitly because it is the main reason we
have not been able to narrow this report further before submitting it.
Related discussions we are aware of
An openldap-technical mailing list thread from 2008-2009 describes what
appears to be the same class of symptom - entries reported by a consumer as
superseded ("too old") that were, in fact, never applied - attributed there to
out-of-order commit-queue processing on the provider under concurrent writes,
with a partial fix referenced against 2.4.11/2.4.16 and a further related ITS
(#6619) mentioned as still open as of a 2011 follow-up. We have not been able
to confirm whether that specific issue is fully resolved in 2.6.x or whether
what we are seeing is a distinct, related condition.
ITS #9538 (April 2021, "Accesslog entryCSN ordering is not always
monotonous") documents non-monotonic entryCSN ordering under
concurrent operations as a live concern in a more recent timeframe.
We are aware of, and our architecture already follows, the guidance
(Quanah Gibson-Mount, openldap-technical , 24 May 2024) that standard syncrepl
is the recommended, safer replication mechanism for multi-provider environments
in OpenLDAP 2.6+, in preference to delta-syncrepl . We note this because our
minimal reproduction above is, if anything, a simpler case than the
multi-provider scenario that guidance addresses (a single provider, a single
consumer) and so does not appear to be excluded by it.
What we are asking
We are not confident enough in any single hypothesis to assert a root cause,
and would welcome guidance on:
1. Whether this is a known, already-tracked issue (our searches of the
mailing list and public ITS entries did not turn up an exact match, but we may
have missed the right terms).
2. Whether a reproduction against a from-source build of current Git
master (rather than the Ubuntu package) is a prerequisite for further triage,
and if so we are willing to attempt it.
3. Any specific loglevel combination, beyond stats sync (already enabled
throughout our testing), or any debugger/instrumentation approach the team
would recommend to capture the provider-side state at the exact moment an entry
is dropped, which we have not yet managed to capture directly (all evidence
above is inferred from surrounding log lines, not a single line that directly
shows the drop occurring).
We can provide the complete Ansible automation used to build the test
environment and drive these reproductions, full logs from any of the runs
referenced above, and are happy to run additional targeted tests against
specific hypotheses.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10601
Quanah Gibson-Mount <quanah(a)openldap.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |SUSPENDED
Keywords|needs_review |
Status|UNCONFIRMED |RESOLVED
Group|OpenLDAP-devs |
--- Comment #3 from Quanah Gibson-Mount <quanah(a)openldap.org> ---
patches welcome
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10600
Issue ID: 10600
Summary: Deferred referral work can be attached to the wrong
LDAP request
Product: OpenLDAP
Version: 2.7.1
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: libraries
Assignee: bugs(a)openldap.org
Reporter: simon.pichugin(a)gmail.com
Target Milestone: ---
When two operations receive referrals to the same connection while an anonymous
rebind is in progress, lconn_rebind_queue stores URL arrays without retaining
the request that owns each deferred referral. When the queue is drained, it
uses the request context performing the rebind, so a child search can be
attached to the wrong operation.
How to reproduce
Start two asynchronous searches on one LDAP handle. Have the origin return
referrals for both operations to the same target, timing the second referral so
it arrives while the target connection is performing the first anonymous bind.
Let the target accept the bind and return successful results for both referred
searches. Operation A receives its final result, while operation B has no
deliverable final result and reaches its caller-supplied timeout.
We reproduced this using public asynchronous APIs on one LDAP handle. The
target validated both child search IDs and base DNs. Operation A completed, but
operation B had no deliverable final result and timed out.
Short result
client_search_ids=1,2
origin_search_ids=1,2
target_verified_anonymous_bind id=4
origin_sent_b_referral_during_a_bind
target_child_search_ids=3,5
target_child_search_bases=dc=target-a,dc=target-b
operation_a_result=101 errno=0
operation_b_result=0 errno=-5 expected_timeout=-5
The test was generated using AI, so I can't share it here as per OpenLDAP's
Policy on AI Contributions. Hence, describing the process with words.
--
You are receiving this mail because:
You are on the CC list for the issue.