https://bugs.openldap.org/show_bug.cgi?id=10514
Issue ID: 10514
Summary: another mutex bug in back-monitor
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: backends
Assignee: bugs(a)openldap.org
Reporter: hyc(a)openldap.org
Target Milestone: ---
Same as #1710 but introduced later in 127ac65c447b8d28ba3cd75e1b8cee28006d5472
Calling mutex_destroy on a locked mutex in cache.c:314 monitor_cache_remove().
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10516
Issue ID: 10516
Summary: dynamic backends don't get rebuilt
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: build
Assignee: bugs(a)openldap.org
Reporter: hyc(a)openldap.org
Target Milestone: ---
The makefile rules need a small fix.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10503
Quanah Gibson-Mount <quanah(a)openldap.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|TEST |FIXED
Status|RESOLVED |VERIFIED
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10526
Issue ID: 10526
Summary: back-mdb: mdb_idl_intersection is O(ID-gap), not O(n),
when a short candidate list is ANDed with a range
Product: OpenLDAP
Version: 2.5.20
Hardware: All
OS: Linux
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: backends
Assignee: bugs(a)openldap.org
Reporter: choeger(a)open-xchange.com
Target Milestone: ---
Created attachment 1157
--> https://bugs.openldap.org/attachment.cgi?id=1157&action=edit
the patch above
----------------------------------------------------------------------------
Note
The below report including the patch has been generated by Claude Opus 4.8.
I built packages based on the OpenLDAP-LTB project version v2.5.20 and verified
in our problematic production deployment.
This refers to the following thread:
https://lists.openldap.org/hyperkitty/list/openldap-technical@openldap.org/…
End Note
----------------------------------------------------------------------------
SUMMARY
-------
AND-filter searches can be orders of magnitude slower than the size of their
result set warrants. mdb_idl_intersection() falls back to a generic
element-at-a-time merge whose cost is proportional to the distance in ID space
between candidate IDs, not to the number of candidates. When one operand is a
range (any indexed term matching more than the IDL range threshold, e.g. a
common objectClass) and the other is a short list of IDs that are widely
separated, the merge walks the range one ID at a time across the entire gap.
This is common in practice: filters of the form
(&(objectClass=X)(someAttr=val)...) hit it whenever someAttr matches a small
number of entries whose IDs happen to be far apart and objectClass=X is stored
as a range that does not fully cover that span.
ENVIRONMENT
-----------
- OpenLDAP 2.5.19, slapd back-mdb.
- ~4.6M entries. An attribute is indexed eq and a given value normally matches
exactly one entry. A common objectClass value matches ~2,000,000 entries and
is therefore stored as an IDL range (it exceeds MDB_idl_db_max; see
servers/slapd/back-mdb/idl.c:467, "No room, convert to a range").
SYMPTOM
-------
Query: (&(objectClass=<common>)(<almost-unique-attr>=<value>)(<attr>=*))
The filter resolves to a single entry in all cases.
- When <almost-unique-attr>=<value> matches ONE entry: the search is very fast.
- When it matches TWO entries whose IDs are far apart (here 1759457 and
2731413, a gap of ~972,000), the same search becomes roughly 20x slower,
even though the AND still yields a single entry.
perf shows nearly all CPU in mdb_idl_next() and mdb_idl_intersection().
Enabling LDAP_DEBUG_FILTER and inspecting the per-term candidate counts
confirms the broad term is a range; gdb in mdb_idl_intersection shows
idmin/idmax equal to the two widely separated IDs.
Neither index_hash64 nor dropping the substring index changes the timing;
the cost is not in the index key lookups, it is in the candidate intersection.
ROOT CAUSE
----------
In servers/slapd/back-mdb/idl.c, mdb_idl_intersection() has fast paths for
both-ranges, for idmin==idmax, and for "range fully covers list". Anything
that misses those falls into a generic merge:
cursora = cursorb = idmin;
ida = mdb_idl_first( a, &cursora );
idb = mdb_idl_first( b, &cursorb );
cursorc = 0;
while( ida <= idmax || idb <= idmax ) {
if( ida == idb ) {
a[++cursorc] = ida;
ida = mdb_idl_next( a, &cursora );
idb = mdb_idl_next( b, &cursorb );
} else if ( ida < idb ) {
ida = mdb_idl_next( a, &cursora );
} else {
idb = mdb_idl_next( b, &cursorb );
}
}
For a range, mdb_idl_next() returns ++(*cursor), i.e. consecutive integers.
So when b is a range that does not fully cover list a (the "range fully covers
list" shortcut is missed because the non-matching list member lies outside the
range bounds), the loop advances idb one integer at a time from idmin to idmax.
With the two matches ~972,000 apart, that is ~972,000 iterations per search.
The single-match case is fast only because it lands on the idmin==idmax or
"range fully covers list" shortcut; it has nothing to do with the index.
The same loop is also O(gap) for the list-vs-list case (a populous list ANDed
with a short, widely-spread list): it walks the populous list across the gap.
mdb_idl_intersection() is reached for every non-first term of an AND filter via
list_candidates() in servers/slapd/back-mdb/filterindex.c.
REPRODUCTION (without specific data)
------------------------------------
1. Load a DB where some indexed attribute value V matches more than
MDB_idl_db_max entries (so V is stored as a range), and where the range's
[lo,hi] does not span the whole DB.
2. Pick another indexed attribute A and two entries that share a value W on A,
such that one of them is NOT in V's range and their IDs are far apart.
3. Time (&(A=W)) vs (&(V)(A=W)) repeatedly. The second is dramatically slower
despite returning fewer (or equal) entries.
A standalone harness that extracts the real 2.5.19 IDL routines, cross-checks a
fixed mdb_idl_intersection against brute-force set intersection, and benchmarks
the pathology is attached (idl_intersection_test.c).
PROPOSED FIX
------------
mdb_idl_intersection() never needs to iterate a range. After the existing
"range fully covers list" shortcut:
- If b is a range, the result is just the members of list a that fall within
[first,last]: iterate the (bounded) list, test each against the bounds.
O(|a|) instead of O(range width).
- If both a and b are lists, keep the sorted merge but, when the cursors
diverge, binary-search (mdb_idl_search) the lagging list forward to the
other's current value rather than stepping one element at a time. Each
catch-up becomes O(log n) instead of O(gap).
Result ordering (ascending) and the a[0] count convention are preserved, as is
the swap/copy-back behaviour. The in-place writes are safe: the write index
(cursorc, number of matches) never exceeds the read index.
Patch against OPENLDAP_REL_ENG_2_5_19 (applies cleanly to RE25 and master HEAD
as well); also attached as mdb-idl-intersection.patch:
--- a/servers/slapd/back-mdb/idl.c
+++ b/servers/slapd/back-mdb/idl.c
@@ -759,23 +759,49 @@
goto done;
}
- /* Fine, do the intersection one element at a time.
- * First advance to idmin in both IDLs.
+ /* If b is a range (and does not fully cover a, handled above), the
+ * intersection is just the elements of list a that fall within the
+ * range bounds. Walk the (bounded) list, never the (potentially huge)
+ * range: O(|a|), not O(range width).
*/
- cursora = cursorb = idmin;
- ida = mdb_idl_first( a, &cursora );
- idb = mdb_idl_first( b, &cursorb );
- cursorc = 0;
+ if ( MDB_IDL_IS_RANGE( b ) ) {
+ ID lo = MDB_IDL_RANGE_FIRST( b );
+ ID hi = MDB_IDL_RANGE_LAST( b );
+ cursorc = 0;
+ for ( ida = mdb_idl_first( a, &cursora );
+ ida != NOID;
+ ida = mdb_idl_next( a, &cursora ) ) {
+ if ( ida < lo )
+ continue;
+ if ( ida > hi )
+ break;
+ a[++cursorc] = ida;
+ }
+ a[0] = cursorc;
+ goto done;
+ }
- while( ida <= idmax || idb <= idmax ) {
- if( ida == idb ) {
+ /* Both a and b are lists. Sorted-merge intersection, but when the two
+ * cursors diverge, binary-search the lagging list forward to the
other's
+ * current value instead of stepping one element at a time. This keeps
a
+ * large gap in one list (e.g. two widely separated matches intersected
+ * against a populous list) from forcing a linear scan of the other
list
+ * across the whole gap: each catch-up is O(log n) rather than O(gap).
+ */
+ cursora = mdb_idl_search( a, idmin );
+ cursorb = mdb_idl_search( b, idmin );
+ cursorc = 0;
+ while ( cursora <= a[0] && cursorb <= b[0] ) {
+ ida = a[cursora];
+ idb = b[cursorb];
+ if ( ida == idb ) {
a[++cursorc] = ida;
- ida = mdb_idl_next( a, &cursora );
- idb = mdb_idl_next( b, &cursorb );
+ cursora++;
+ cursorb++;
} else if ( ida < idb ) {
- ida = mdb_idl_next( a, &cursora );
+ cursora = mdb_idl_search( a, idb );
} else {
- idb = mdb_idl_next( b, &cursorb );
+ cursorb = mdb_idl_search( b, ida );
}
}
a[0] = cursorc;
TESTING
-------
Using the attached harness (the patched function is byte-for-byte the shipped
code):
- Correctness: 400,000 randomized trials (two seeds) over zero/list/range
operands of varied sizes and gaps, in both argument orders, cross-checked
against brute-force set intersection. Zero mismatches.
- The reported case, list {1759457, 2731413} intersected with a 2,000,000-entry
range [1 .. 2731412] that does not cover the upper member, 1200 iterations:
original: 2.54 s, 1,166,348,400 mdb_idl_next calls, result count = 1
fixed: ~0 s, 1,200 mdb_idl_next calls, result count = 1
- Single-value baseline (original): ~0 s, 0 mdb_idl_next calls, confirming the
original is fast only for the single-match case.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10476
Quanah Gibson-Mount <quanah(a)openldap.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |VERIFIED
Resolution|TEST |FIXED
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10489
Issue ID: 10489
Summary: memory issue in liblber
Product: OpenLDAP
Version: 2.6.9
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: libraries
Assignee: bugs(a)openldap.org
Reporter: yutengsun1997(a)gmail.com
Target Milestone: ---
Dear OpenLDAP Security Team,
I reported a potential security issue two weeks ago, but have not received a
response.
I am writing to confirm whether I emailed the right place.
I can attach the email content if required.
Looking forward to your reply. Many thanks.
Regards,
Yt
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10496
Issue ID: 10496
Summary: Remove references to Mozilla NSS in lload.conf(5) man
page
Product: OpenLDAP
Version: 2.6.10
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: documentation
Assignee: bugs(a)openldap.org
Reporter: quanah(a)openldap.org
Target Milestone: ---
We removed Mozilla NSS support in OpenLDAP 2.5, remove references to it from
OpenLDAP 2.6 man pages.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10495
Issue ID: 10495
Summary: Environment variable for Systemd is ignored by
configure
Product: OpenLDAP
Version: 2.6.13
Hardware: All
OS: Linux
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: build
Assignee: bugs(a)openldap.org
Reporter: openldap.aftermost863(a)passinbox.com
Target Milestone: ---
Hello,
When using the configure script to generate the Makefile, the documentation in
the --help option states that the environment variable 'systemdsystemunitdir'
can be used to override the default system path for Systemd units.
However, this does not work: if I export the variable with a valid existing
path (e.g. 'export systemdsystemunitdir=/tmp/test') and then run ./configure,
the variable is found but is ignored in the output variables and is therefore
not used in the Makefile.
config.log:
...
## ---------------- ##
## Cache variables. ##
## ---------------- ##
...
ac_cv_env_systemdsystemunitdir_set=set
ac_cv_env_systemdsystemunitdir_value=/tmp/test
...
## ----------------- ##
## Output variables. ##
## ----------------- ##
...
systemdsystemunitdir='/usr/lib/systemd/system'
...
The configure script seems to override that variable on line 23942:
systemdsystemunitdir=
I don’t quite understand why this line is there, and it seems suspicious to me.
However, if I remove it and run ./configure again in the exact same shell,
config.log now shows the correct path in the output variables, and the Makefile
uses it correctly afterward:
...
## ---------------- ##
## Cache variables. ##
## ---------------- ##
...
ac_cv_env_systemdsystemunitdir_set=set
ac_cv_env_systemdsystemunitdir_value=/tmp/test
...
pkg_cv_systemdsystemunitdir=/tmp/test
## ----------------- ##
## Output variables. ##
## ----------------- ##
...
systemdsystemunitdir='/tmp/test'
I had this issue on both Debian 13 (using bash) and Arch Linux (using zsh).
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10480
Issue ID: 10480
Summary: Use after free in cn=config replication
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: backends
Assignee: bugs(a)openldap.org
Reporter: ondra(a)mistotebe.net
Target Milestone: ---
If a renumber needs to happen, cn=config frees e->e_name, but some code (e.g.
syncrepl) sets op->o_req_dn to point to the same and as such it can't be used
anymore. This causes a crash in syncrepl (if LDAP_DEBUG_SYNC is on) and
accesslog during cn=config replication.
Either syncrepl (and others) shouldn't do this or cn=config should check for
this case and adjust o_req_dn after the fact.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10524
Issue ID: 10524
Summary: slaptest attempts to open a new database, instead of
creating it
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: nivanova(a)symas.com
Target Milestone: ---
The issue appears when one decides to convert a slapd.conf file to cn=config,
if slapd has never been started with said slapd.conf, and the slapd.conf
contains a database definition.
For the purpose of testing another issue, I wrote a brand new slapd.conf and
attempted to convert it into a brand new configuration directory, before
starting up slapd.
Example slapd.conf:
database config
rootdn cn=config
rootpw secret
database mdb
directory ./mdb_test
suffix dc=ldap,dc=example,dc=com
rootdn cn=admin,dc=ldap,dc=example,dc=com
rootpw secret
I create the mdb_test directory and the configuration directory (crash.d) and
run:
slapd -T test -f slap_mdb.conf -F crash.d
which fails:
mdb_db_open: database "dc=ldap,dc=example,dc=com" cannot be opened: No such
file or directory (2). Administrator intervention needed!
backend_startup_one (type=mdb, suffix="dc=ldap,dc=example,dc=com"): bi_db_open
failed! (2)
If I start slapd with the .conf file it starts up sucessfully and createds the
database. After that a slaptest run succeeds, because the database already
exists.
This is probably a rare use case, but I think we need to decide if this is
desired behavior and fix it if not.
--
You are receiving this mail because:
You are on the CC list for the issue.