https://bugs.openldap.org/show_bug.cgi?id=10509
Issue ID: 10509
Summary: 2.7: Feature notes for announcement file
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: documentation
Assignee: bugs(a)openldap.org
Reporter: quanah(a)openldap.org
Target Milestone: ---
Add feature notes for the 2.7 announcement file
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10026
Issue ID: 10026
Summary: Refresh handling can skip entries (si_dirty not
managed properly)
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: overlays
Assignee: bugs(a)openldap.org
Reporter: ondra(a)mistotebe.net
Target Milestone: ---
Take MPR plain syncrepl with 3+ providers.
When a provider's own syncrepl session transitions to persist and a it starts a
new parallel session towards another host, that session always has to start as
a refresh. If that refresh serves entries to us, our handling of si_dirty is
not consistent:
- if the existing persist session serves some of these entries to us, we can
"forget" to pass the others to a newly connected consumer
- same if the refresh is abandoned and we start refreshing from a different
provider that might be behind what we were being served (again our consumers
could suffer)
- if we restart, si_dirty is forgotten and our consumers suffer even worse
We might need to be told (at the beginning of the refresh?) what the end state
we're going for is, so we can keep si_dirty on until then. And somehow persist
that knowledge in the DB...
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9343
Issue ID: 9343
Summary: Expand ppolicy policy configuration to allow URL
filter
Product: OpenLDAP
Version: 2.5
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: overlays
Assignee: bugs(a)openldap.org
Reporter: quanah(a)openldap.org
Target Milestone: ---
Currently, ppolicy only supports a single global default policy, and past that
any policies must be manually added to a given user entry if they are supposed
to have something other than the default policy.
Also, some sites want no default policy, and only a specific subset to have a
policy applied to them.
For both of these cases, it would be helpful if it were possible to configure a
policy to apply to a set of users via a URL similar to the way we handle
creating groups of users in dynlist
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10524
Issue ID: 10524
Summary: slaptest attempts to open a new database, instead of
creating it
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: nivanova(a)symas.com
Target Milestone: ---
The issue appears when one decides to convert a slapd.conf file to cn=config,
if slapd has never been started with said slapd.conf, and the slapd.conf
contains a database definition.
For the purpose of testing another issue, I wrote a brand new slapd.conf and
attempted to convert it into a brand new configuration directory, before
starting up slapd.
Example slapd.conf:
database config
rootdn cn=config
rootpw secret
database mdb
directory ./mdb_test
suffix dc=ldap,dc=example,dc=com
rootdn cn=admin,dc=ldap,dc=example,dc=com
rootpw secret
I create the mdb_test directory and the configuration directory (crash.d) and
run:
slapd -T test -f slap_mdb.conf -F crash.d
which fails:
mdb_db_open: database "dc=ldap,dc=example,dc=com" cannot be opened: No such
file or directory (2). Administrator intervention needed!
backend_startup_one (type=mdb, suffix="dc=ldap,dc=example,dc=com"): bi_db_open
failed! (2)
If I start slapd with the .conf file it starts up sucessfully and createds the
database. After that a slaptest run succeeds, because the database already
exists.
This is probably a rare use case, but I think we need to decide if this is
desired behavior and fix it if not.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9667
Issue ID: 9667
Summary: 2.6 to 2.7 upgrade documentation
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: documentation
Assignee: bugs(a)openldap.org
Reporter: quanah(a)openldap.org
Target Milestone: ---
Need to document any upgrade information for going from 2.6 to 2.7
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10358
Issue ID: 10358
Summary: syncrepl can revert an entry's CSN
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: ondra(a)mistotebe.net
Target Milestone: ---
Created attachment 1080
--> https://bugs.openldap.org/attachment.cgi?id=1080&action=edit
Debug log of an instance of this happening
There is a sequence of operations which can force a MPR node to apply changes
out of order (essentially reverting an operation). Currently investigating
which part of the code that should have prevented this has let it slip.
A sample log showing how this happened is attached.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10490
Issue ID: 10490
Summary: Bump library minor version(s)
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: libraries
Assignee: bugs(a)openldap.org
Reporter: ondra(a)mistotebe.net
Target Milestone: ---
Tracking bug for 2.7 branchpoint if we expose new interfaces in either library.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10054
Issue ID: 10054
Summary: Value size limited to 2,147,479,552 bytes
Product: LMDB
Version: unspecified
Hardware: x86_64
OS: Linux
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: louis(a)meilisearch.com
Target Milestone: ---
Hello,
According to the documentation[0], a database that is not using `MDB_DUPSORT`
can store values up to `0xffffffff` bytes (around 4GB).
In practice, under Linux, the actual limit is `0x7ffff000` though (2^31 - 4096,
so around 2GB).
This is due to the write loop in `mdb_page_flush`. The `wsize` value
determining how many bytes will be written can be as big as
`4096*dp->mp_pages`[1], and the number of overflow pages grows with the size of
the value put inside the DB.
The `wsize` is not split in smaller chunks in the case where there are many
overflow pages to write, and as a result the call to `pwrite`[2] does not
perform a full write, but only a "short" write of 2147479552 bytes (the maximum
allowed on a call to `pwrite` on Linux[3]).
This would be OK if the short write condition was handled by looping and
performing another `pwrite` with the rest of the data, but instead `EIO` is
returned[4].
There seems to be a related, but different issue on macOS when trying to
`pwrite` more the 2^31 bytes, that was already reported[5].
This issue was reported to me by a Meilisearch user because it causes their
database indexing to fail[6]. I had to investigate a bit because their setup
was peculiar (high number of documents in their database) and the `EIO` error
code is not very descriptive of the underlying issue.
I join a C reproducer of the issue that attempts to add a 2147479553 bytes
value to the DB and fails with `EIO` (decreasing `nb_items` to a smaller value
such as `2107479552` does succeed)[7].
Thank you for making LMDB!
Louis Dureuil.
[0]:
https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/lmdb.h#LL284…
[1]:
https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/mdb.c#LL3770…
[2]: https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/mdb.c#L3820
[3]:
https://stackoverflow.com/questions/70368651/why-cant-linux-write-more-than…
[4]: https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/mdb.c#L3840
[5]: https://bugs.openldap.org/show_bug.cgi?id=9736
[6]: https://github.com/meilisearch/meilisearch/issues/3654
[7]: https://github.com/dureuill/lmdb_3654/tree/main
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10526
Issue ID: 10526
Summary: back-mdb: mdb_idl_intersection is O(ID-gap), not O(n),
when a short candidate list is ANDed with a range
Product: OpenLDAP
Version: 2.5.20
Hardware: All
OS: Linux
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: backends
Assignee: bugs(a)openldap.org
Reporter: choeger(a)open-xchange.com
Target Milestone: ---
Created attachment 1157
--> https://bugs.openldap.org/attachment.cgi?id=1157&action=edit
the patch above
----------------------------------------------------------------------------
Note
The below report including the patch has been generated by Claude Opus 4.8.
I built packages based on the OpenLDAP-LTB project version v2.5.20 and verified
in our problematic production deployment.
This refers to the following thread:
https://lists.openldap.org/hyperkitty/list/openldap-technical@openldap.org/…
End Note
----------------------------------------------------------------------------
SUMMARY
-------
AND-filter searches can be orders of magnitude slower than the size of their
result set warrants. mdb_idl_intersection() falls back to a generic
element-at-a-time merge whose cost is proportional to the distance in ID space
between candidate IDs, not to the number of candidates. When one operand is a
range (any indexed term matching more than the IDL range threshold, e.g. a
common objectClass) and the other is a short list of IDs that are widely
separated, the merge walks the range one ID at a time across the entire gap.
This is common in practice: filters of the form
(&(objectClass=X)(someAttr=val)...) hit it whenever someAttr matches a small
number of entries whose IDs happen to be far apart and objectClass=X is stored
as a range that does not fully cover that span.
ENVIRONMENT
-----------
- OpenLDAP 2.5.19, slapd back-mdb.
- ~4.6M entries. An attribute is indexed eq and a given value normally matches
exactly one entry. A common objectClass value matches ~2,000,000 entries and
is therefore stored as an IDL range (it exceeds MDB_idl_db_max; see
servers/slapd/back-mdb/idl.c:467, "No room, convert to a range").
SYMPTOM
-------
Query: (&(objectClass=<common>)(<almost-unique-attr>=<value>)(<attr>=*))
The filter resolves to a single entry in all cases.
- When <almost-unique-attr>=<value> matches ONE entry: the search is very fast.
- When it matches TWO entries whose IDs are far apart (here 1759457 and
2731413, a gap of ~972,000), the same search becomes roughly 20x slower,
even though the AND still yields a single entry.
perf shows nearly all CPU in mdb_idl_next() and mdb_idl_intersection().
Enabling LDAP_DEBUG_FILTER and inspecting the per-term candidate counts
confirms the broad term is a range; gdb in mdb_idl_intersection shows
idmin/idmax equal to the two widely separated IDs.
Neither index_hash64 nor dropping the substring index changes the timing;
the cost is not in the index key lookups, it is in the candidate intersection.
ROOT CAUSE
----------
In servers/slapd/back-mdb/idl.c, mdb_idl_intersection() has fast paths for
both-ranges, for idmin==idmax, and for "range fully covers list". Anything
that misses those falls into a generic merge:
cursora = cursorb = idmin;
ida = mdb_idl_first( a, &cursora );
idb = mdb_idl_first( b, &cursorb );
cursorc = 0;
while( ida <= idmax || idb <= idmax ) {
if( ida == idb ) {
a[++cursorc] = ida;
ida = mdb_idl_next( a, &cursora );
idb = mdb_idl_next( b, &cursorb );
} else if ( ida < idb ) {
ida = mdb_idl_next( a, &cursora );
} else {
idb = mdb_idl_next( b, &cursorb );
}
}
For a range, mdb_idl_next() returns ++(*cursor), i.e. consecutive integers.
So when b is a range that does not fully cover list a (the "range fully covers
list" shortcut is missed because the non-matching list member lies outside the
range bounds), the loop advances idb one integer at a time from idmin to idmax.
With the two matches ~972,000 apart, that is ~972,000 iterations per search.
The single-match case is fast only because it lands on the idmin==idmax or
"range fully covers list" shortcut; it has nothing to do with the index.
The same loop is also O(gap) for the list-vs-list case (a populous list ANDed
with a short, widely-spread list): it walks the populous list across the gap.
mdb_idl_intersection() is reached for every non-first term of an AND filter via
list_candidates() in servers/slapd/back-mdb/filterindex.c.
REPRODUCTION (without specific data)
------------------------------------
1. Load a DB where some indexed attribute value V matches more than
MDB_idl_db_max entries (so V is stored as a range), and where the range's
[lo,hi] does not span the whole DB.
2. Pick another indexed attribute A and two entries that share a value W on A,
such that one of them is NOT in V's range and their IDs are far apart.
3. Time (&(A=W)) vs (&(V)(A=W)) repeatedly. The second is dramatically slower
despite returning fewer (or equal) entries.
A standalone harness that extracts the real 2.5.19 IDL routines, cross-checks a
fixed mdb_idl_intersection against brute-force set intersection, and benchmarks
the pathology is attached (idl_intersection_test.c).
PROPOSED FIX
------------
mdb_idl_intersection() never needs to iterate a range. After the existing
"range fully covers list" shortcut:
- If b is a range, the result is just the members of list a that fall within
[first,last]: iterate the (bounded) list, test each against the bounds.
O(|a|) instead of O(range width).
- If both a and b are lists, keep the sorted merge but, when the cursors
diverge, binary-search (mdb_idl_search) the lagging list forward to the
other's current value rather than stepping one element at a time. Each
catch-up becomes O(log n) instead of O(gap).
Result ordering (ascending) and the a[0] count convention are preserved, as is
the swap/copy-back behaviour. The in-place writes are safe: the write index
(cursorc, number of matches) never exceeds the read index.
Patch against OPENLDAP_REL_ENG_2_5_19 (applies cleanly to RE25 and master HEAD
as well); also attached as mdb-idl-intersection.patch:
--- a/servers/slapd/back-mdb/idl.c
+++ b/servers/slapd/back-mdb/idl.c
@@ -759,23 +759,49 @@
goto done;
}
- /* Fine, do the intersection one element at a time.
- * First advance to idmin in both IDLs.
+ /* If b is a range (and does not fully cover a, handled above), the
+ * intersection is just the elements of list a that fall within the
+ * range bounds. Walk the (bounded) list, never the (potentially huge)
+ * range: O(|a|), not O(range width).
*/
- cursora = cursorb = idmin;
- ida = mdb_idl_first( a, &cursora );
- idb = mdb_idl_first( b, &cursorb );
- cursorc = 0;
+ if ( MDB_IDL_IS_RANGE( b ) ) {
+ ID lo = MDB_IDL_RANGE_FIRST( b );
+ ID hi = MDB_IDL_RANGE_LAST( b );
+ cursorc = 0;
+ for ( ida = mdb_idl_first( a, &cursora );
+ ida != NOID;
+ ida = mdb_idl_next( a, &cursora ) ) {
+ if ( ida < lo )
+ continue;
+ if ( ida > hi )
+ break;
+ a[++cursorc] = ida;
+ }
+ a[0] = cursorc;
+ goto done;
+ }
- while( ida <= idmax || idb <= idmax ) {
- if( ida == idb ) {
+ /* Both a and b are lists. Sorted-merge intersection, but when the two
+ * cursors diverge, binary-search the lagging list forward to the
other's
+ * current value instead of stepping one element at a time. This keeps
a
+ * large gap in one list (e.g. two widely separated matches intersected
+ * against a populous list) from forcing a linear scan of the other
list
+ * across the whole gap: each catch-up is O(log n) rather than O(gap).
+ */
+ cursora = mdb_idl_search( a, idmin );
+ cursorb = mdb_idl_search( b, idmin );
+ cursorc = 0;
+ while ( cursora <= a[0] && cursorb <= b[0] ) {
+ ida = a[cursora];
+ idb = b[cursorb];
+ if ( ida == idb ) {
a[++cursorc] = ida;
- ida = mdb_idl_next( a, &cursora );
- idb = mdb_idl_next( b, &cursorb );
+ cursora++;
+ cursorb++;
} else if ( ida < idb ) {
- ida = mdb_idl_next( a, &cursora );
+ cursora = mdb_idl_search( a, idb );
} else {
- idb = mdb_idl_next( b, &cursorb );
+ cursorb = mdb_idl_search( b, ida );
}
}
a[0] = cursorc;
TESTING
-------
Using the attached harness (the patched function is byte-for-byte the shipped
code):
- Correctness: 400,000 randomized trials (two seeds) over zero/list/range
operands of varied sizes and gaps, in both argument orders, cross-checked
against brute-force set intersection. Zero mismatches.
- The reported case, list {1759457, 2731413} intersected with a 2,000,000-entry
range [1 .. 2731412] that does not cover the upper member, 1200 iterations:
original: 2.54 s, 1,166,348,400 mdb_idl_next calls, result count = 1
fixed: ~0 s, 1,200 mdb_idl_next calls, result count = 1
- Single-value baseline (original): ~0 s, 0 mdb_idl_next calls, confirming the
original is fast only for the single-match case.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9223
Bug ID: 9223
Summary: Add support for incremental backup
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: quanah(a)openldap.org
Target Milestone: ---
For LMDB 1.0, add support for incremental backups
--
You are receiving this mail because:
You are on the CC list for the bug.