https://bugs.openldap.org/show_bug.cgi?id=10736
Issue ID: 10736
Summary: cloak: two cloak-attr values build a list loop, slapd
spins - HIGH
Product: OpenLDAP
Version: 2.6.16
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: contrib
Assignee: bugs(a)openldap.org
Reporter: jcc(a)security.souspilot.com
Target Milestone: ---
Overview:
The contrib overlay cloak (contrib/slapd-modules/cloak/cloak.c) keeps its
configured attributes in a linked list. In cloak_cfgen() the for-loop that
finds the insertion point has the condition "c->valx < 0 || i < c->valx,
*cip": the comma operator discards the left operand, so the condition is just
*cip, and the body "ci_next = *cip;" then leaves the new node pointing at the
first node. With two or more cloak-attr directives (the normal use) the list
is circular. Every walk of the list never ends: the configuration emit loop
for olcCloakAttribute (cloak.c:56) spins in config_back_db_open so slapd never
opens its listener, the search callback cloak_search_response_cb() spins on
the first search that returns an entry, and cloak_db_destroy() frees nodes
twice. No client action is needed for the start-up hang; a configuration with
a single cloak-attr works.
Steps to Reproduce:
1. Build cloak.so from contrib/slapd-modules/cloak of the same tree.
/tmp/cloak/slapd.conf (adjust the schema and module directories):
include /etc/ldap/schema/core.schema
include /etc/ldap/schema/cosine.schema
include /etc/ldap/schema/inetorgperson.schema
modulepath /usr/lib/ldap
moduleload back_mdb.so
moduleload /tmp/cloak/cloak.so
pidfile /tmp/cloak/slapd.pid
access to * by * read
database mdb
suffix "dc=example,dc=org"
rootdn "cn=admin,dc=example,dc=org"
rootpw secret
directory /tmp/cloak/db
overlay cloak
cloak-attr carLicense
cloak-attr description
/tmp/cloak/data.ldif:
dn: dc=example,dc=org
objectClass: dcObject
objectClass: organization
dc: example
o: Example
dn: cn=bob,dc=example,dc=org
objectClass: inetOrgPerson
cn: bob
sn: bob
2. Release build: load, start, and try a search after a few seconds:
mkdir -p /tmp/cloak/db
slapadd -f /tmp/cloak/slapd.conf -l /tmp/cloak/data.ldif
slapd -d stats -f /tmp/cloak/slapd.conf -h "ldap://127.0.0.1:1389/" &
sleep 6
timeout 4 ldapsearch -x -H ldap://127.0.0.1:1389 -b "" -s base
top -b -n 1 -p $(pgrep -x slapd)
3. AddressSanitizer build: run slapadd alone with the same configuration (the
list is built when the configuration is read and freed at exit):
rm -rf /tmp/cloak/db; mkdir -p /tmp/cloak/db
ASAN_OPTIONS=detect_leaks=0 slapadd -f /tmp/cloak/slapd.conf \
-l /tmp/cloak/data.ldif
4. Control: remove the second cloak-attr line.
A proof-of-concept script is available and can be provided on request.
Actual Results:
Step 2 (release 2.6.16): the search gets no answer; the slapd process is
running and one thread uses about 100 % CPU (99.5 % in one run, 99.8 % in an
independent reproduction). In the independent reproduction the listener was
never opened and the spin was in the olcCloakAttribute emit loop under
config_back_db_open. In the other run the listener came up, and the first
search that returned an entry (the rootDSE probe) never completed; the log
shows no SEARCH RESULT line for it.
Step 3 (AddressSanitizer build):
ERROR: AddressSanitizer: heap-use-after-free
READ of size 8
#0 cloak_db_destroy contrib/slapd-modules/cloak/cloak.c:309
#2 backend_stopdown_one
#6 slap_tool_destroy
#7 slapadd
freed by:
#3 cloak_db_destroy contrib/slapd-modules/cloak/cloak.c:310
Step 4 (control): slapd starts, searches are answered, and the sanitizer build
reports nothing.
Expected Results:
slapd starts with two or more cloak-attr values, searches return with the
listed attributes hidden, and shutdown frees each node once.
Build Date & Hardware:
OpenLDAP 2.6.16 (release tag OPENLDAP_REL_ENG_2_6_16), built from source with
GCC 12: release build and AddressSanitizer build, cloak built from the same
tree. Linux arm64 (aarch64), Ubuntu 24.04 runtime (glibc).
Additional Builds and Platforms:
2.7.2 and master: the source file is identical to 2.6.16 in the functions
named here, by reading; not run. Distribution packages, musl and x86-64: not
tested. Earlier versions: not determined.
Additional Information:
Type: endless loop (denial of service at start-up and on search) and
use-after-free / double free in cloak_db_destroy() (cloak.c:307-311).
Root cause: cloak_cfgen(), about lines 147-160 in 2.6.16:
for ( i = 0, cip = (cloak_info_t **)&on->on_bi.bi_private;
c->valx < 0 || i < c->valx, *cip;
i++, cip = &(*cip)->ci_next ) {
...
ci_next = *cip;
}
*cip = (cloak_info_t *)SLAP_CALLOC( 1, sizeof( cloak_info_t ) );
...
(*cip)->ci_next = ci_next;
The loop always runs to the end of the list and leaves ci_next set to the last
node visited, so the second node's ci_next points back at the first.
No fix was built or tested. Direction (untested): two changes are needed;
parentheses around the condition alone are not sufficient. Apply the index
test in the for condition without the comma, and take ci_next from *cip after
the loop instead of inside its body, for example
for ( i = 0, cip = (cloak_info_t **)&on->on_bi.bi_private;
*cip && ( c->valx < 0 || i < c->valx );
i++, cip = &(*cip)->ci_next )
/* find the insertion point */ ;
if ( c->valx >= 0 && i < c->valx ) {
/* report "invalid index" as before */
return 1;
}
ci_next = *cip;
contrib/slapd-modules is not part of a default build.
Assessment (not measured): both forms of the hang were seen (emit loop at
start-up, search callback on the first search); the conditions that select one
or the other were not isolated.
Credit: when a CVE is assigned for this issue, please credit Joao Charles
Carvalho.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10717
Issue ID: 10717
Summary: back-mdb: enabling multival on existing data drops
values - HIGH
Product: OpenLDAP
Version: 2.6.16
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: backends
Assignee: bugs(a)openldap.org
Reporter: jcc(a)security.souspilot.com
Target Milestone: ---
Overview:
When olcDbMultival (multival <attrs> <hi>,<lo>) is enabled or lowered on a
database whose entries already hold more than <hi> values of the attribute,
nothing migrates the existing data. On the first modify that rewrites such an
entry, mdb_entry_encode() classifies the attribute as SLAP_ATTR_BIG_MULTI and
leaves its values out of the entry blob, while mdb_id2entry_put() writes such
values to the separate table only when the entry is being added. A modify of
any other attribute therefore returns success and leaves the attribute with no
values.
The next modify of the entry aborts slapd in entry_schema_check() (assertion;
assertions are enabled in release builds), and the damaged entry is on disk,
so this repeats after every restart. The precondition is a configuration
change by an administrator; after it, ordinary write traffic causes the data
loss and any writer of an affected entry (self-write is enough) triggers the
abort.
Steps to Reproduce:
1) Start slapd 2.6.16 from this slapd.conf (adjust schema and module paths to
the install prefix), converted to cn=config so that the mdb database is
olcDatabase={1}mdb,cn=config:
include /etc/openldap/schema/core.schema
include /etc/openldap/schema/cosine.schema
include /etc/openldap/schema/inetorgperson.schema
modulepath /usr/lib/ldap
moduleload back_mdb
database config
rootdn cn=admin,cn=config
rootpw config
database mdb
suffix dc=example,dc=org
rootdn cn=admin,dc=example,dc=org
rootpw admin
directory /tmp/repro/db
maxsize 268435456
index objectClass eq
access to attrs=userPassword by self write by anonymous auth by * none
access to * by self write by users read by * none
2) Load the suffix, ou=people, ou=groups, a group
cn=biggrp,ou=groups,dc=example,dc=org (groupOfNames, description "team")
with 60 member values, and uid=carol,ou=people,dc=example,dc=org
(inetOrgPerson, userPassword carol-pass, telephoneNumber) with 60
description values. The data set used is in base.ldif (see Additional
Information); an equivalent one for the steps below:
{
printf 'dn: dc=example,dc=org\nobjectClass: dcObject\nobjectClass:
organization\no: Example\ndc: example\n\n'
for ou in people groups; do printf 'dn:
ou=%s,dc=example,dc=org\nobjectClass: organizationalUnit\nou: %s\n\n' $ou $ou;
done
printf 'dn: uid=carol,ou=people,dc=example,dc=org\nobjectClass:
inetOrgPerson\nuid: carol\ncn: Carol Clark\nsn: Clark\n'
printf 'telephoneNumber: +1 555 0100\nuserPassword: carol-pass\n'
for i in $(seq -w 1 60); do echo "description: note $i of carol's own
entry"; done; echo
printf 'dn: cn=biggrp,ou=groups,dc=example,dc=org\nobjectClass:
groupOfNames\ncn: biggrp\ndescription: team\n'
for i in $(seq -w 1 60); do echo "member:
uid=m$i,ou=people,dc=example,dc=org"; done
} > /tmp/repro/base.ldif
mkdir -p /tmp/repro/db; slapadd -f slapd.conf -l /tmp/repro/base.ldif
3) As the cn=config administrator apply:
dn: olcDatabase={1}mdb,cn=config
changetype: modify
add: olcDbMultival
olcDbMultival: member,description 50,20
4) Modify the group's description as the directory administrator, and as carol
modify her own telephoneNumber. Then count the values and let carol modify
her entry once more:
U=ldap://127.0.0.1:1389
ADM="-x -H $U -D cn=admin,dc=example,dc=org -w admin"
CAR="-x -H $U -D uid=carol,ou=people,dc=example,dc=org -w carol-pass"
GRP=cn=biggrp,ou=groups,dc=example,dc=org
CAROL=uid=carol,ou=people,dc=example,dc=org
mod() { printf 'dn: %s\nchangetype: modify\nreplace: %s\n%s: %s\n' "$2"
"$3" "$3" "$4" | ldapmodify $1; }
count() { ldapsearch $ADM -LLL -o ldif-wrap=no -b "$1" -s base "$2" | grep
-c "^$2:"; }
count $GRP member; count $CAROL description # 60 and 60
ldapmodify -x -H $U -D cn=admin,cn=config -w config -f enable-multival.ldif
count $GRP member; count $CAROL description # still 60 and 60
mod "$ADM" $GRP description "renamed team" # result 0
mod "$CAR" $CAROL telephoneNumber "+1 555 4242" # result 0
count $GRP member; count $CAROL description # 0 and 0
mod "$CAR" $CAROL telephoneNumber "+1 555 4243" # slapd aborts
5) Start slapd again on the same data directory and repeat the last command.
A proof-of-concept script is available and can be provided on request.
Actual Results:
Before and immediately after step 3 both attributes show 60 values. Both
modifies in step 4 return 0; afterwards member and description have 0 values.
The next self-modify gets no reply and slapd logs "schema_check.c:89:
entry_schema_check: Assertion 'a->a_vals[0].bv_val != NULL' failed" (line 82
on 2.7.2). After the restart the values are still gone and the same
self-modify aborts slapd a second time.
In an earlier run, not repeated with the commands above: the same loss when
the directive is added offline to slapd.d and slapd is restarted (60 members
-> 0, then the assertion); a Delete aborts in octetStringIndexer ("Assertion
'i > 0'") when the attribute is indexed.
Expected Results:
Existing values are migrated to the multival table on the first write (or the
configuration change is refused until the data is converted); no values are
lost and no assertion is reachable from on-disk data.
Build Date & Hardware:
OpenLDAP 2.6.16 (release tag, built from source), Linux aarch64, Ubuntu 24.04
userland, glibc 2.39.
Additional Builds and Platforms:
- 2.7.2 (release tag): reproduced in an earlier run; not repeated with the
commands above.
- master 7ae14ef466: the "adding && ec.multi" branch is unchanged, by reading.
- 2.5.20, Ubuntu package, Alpine/musl: not tested.
Additional Information:
Crash type: assertion failure (abort) in entry_schema_check(),
servers/slapd/schema_check.c:89. Root cause: back-mdb/id2entry.c:895-916
(mdb_entry_encode omits BIG_MULTI values) and id2entry.c:311 ("if (adding &&
ec.multi)", comment "Modifies handle them directly"). No fix was built or
tested. Directions (untested): migrate the values on the first write in
mdb_id2entry_put(); or refuse the change on a populated database and document
that multival needs a slapcat/slapadd reload; independently,
entry_schema_check() should return an error for an attribute without values
instead of asserting. Operator workaround: set multival only on an empty
database, or slapcat, change the setting and slapadd again. Removing the
setting afterwards does not repair a damaged entry. Where the abort was
measured with slapd as pid 1 of a container, the process stayed in shutdown
without serving; as an ordinary process it exits with the abort code. A
related problem in the same code is reported separately as "back-mdb:
oversized multival value fails modify with rc 80".
Credit: when a CVE is assigned for this issue, please credit Joao Charles
Carvalho.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10696
Issue ID: 10696
Summary: slapd: clients that do not read park every worker
thread - HIGH
Product: OpenLDAP
Version: 2.6.16
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: jcc(a)security.souspilot.com
Target Milestone: ---
Overview:
When a client does not read its replies, the operation that owns the
connection's writer slot blocks on the socket and every further operation on
that connection waits on c_write1_cv in send_ldap_ber()
(servers/slapd/result.c) with no timeout, each holding a worker thread. One
connection may run up to half of the configured threads, so two or three
connections fill the default pool of 16.
An anonymous client can do this in the default configuration with a few
kilobytes of pipelined searches whose answers are large (for example
cn=Subschema). olcWriteTimeout frees only the blocked writer, not the
operations queued behind it.
Steps to Reproduce:
1) Start slapd 2.6.16 with default settings (16 threads) where cn=Subschema is
readable anonymously, for example one back-mdb database with
access to * by * read
(the configuration used, with a monitor database to watch
2) Open 4 TCP connections without binding, with a small receive buffer
(SO_RCVBUF 4096). On each, write 20 base-scope SearchRequests of
cn=Subschema (filter (objectClass=*), attributes attributeTypes,
objectClasses, ldapSyntaxes, matchingRules, matchingRuleUse) back to back,
about 3 KB in total for all connections, and never read from the socket.
3) From another client, run 10 ordinary searches:
for i in $(seq 10); do
timeout 5 ldapsearch -x -H ldap://127.0.0.1:1389 -b dc=example,dc=org \
-s base 1.1 >/dev/null 2>&1 && echo ok || echo FAILED
done
4) Close the 4 connections and repeat step 3.
A second measured variant uses anonymous connections that send subtree
searches of an anonymously readable 6 MB tree, with a bound user probing the
server. A proof-of-concept script is available and can be provided on request.
Actual Results:
Step 3: 0 of 10 legitimate searches succeed while the connections are held.
Step 4: service recovers when they are closed. With olcWriteTimeout set,
reconnecting every 4 s keeps the outage going.
Second variant: 0 of 4 probes answered during the flood (worker pool at 16 of
16 active, or the monitor query itself unanswerable), 2 of 2 after it; slapd
never crashed.
Settings tried without stopping a reconnecting client (victim success 0 to 2
of 10): olcWriteTimeout with olcIdleTimeout, olcConnMaxPending=5,
olcConnMaxPendingAuth, olcThreads=64, olcListenerThreads, olcLimits, and all
combined. An ACL making cn=Subschema readable by users only stopped the
anonymous cn=Subschema variant (10 of 10).
Expected Results:
Operations waiting behind a stalled writer are bounded (timed out, or the
connection is closed) so that a few non-reading clients cannot hold the whole
worker pool.
Build Date & Hardware:
OpenLDAP 2.6.16 (release tag, built from source), Linux aarch64, Ubuntu 24.04
userland, glibc 2.39.
Additional Builds and Platforms:
- 2.7.2 (release tag): reproduces.
- Ubuntu 24.04 package 2.6.10: the defect reproduces unchanged.
- A vendor package of OpenLDAP 2.5.17 (x86-64, 8 threads): 2 anonymous
connections with 8 pipelined cn=Subschema searches each (2,048 bytes sent)
stop the whole server.
- A vendor build based on OpenLDAP 2.4.28: 15 to 16 non-reading anonymous
connections: no other client answered, 4 of 4 runs; recovery on close 8 of
8. One run, not independently reproduced.
- master, 2.5.20, Alpine/musl: not tested.
Additional Information:
No crash; slapd stays up and recovers when the connections close. Root cause:
result.c line 356 (wait on c_write1_cv with no timeout); the per-connection
cap of olcThreads/2 running operations is at connection.c line 1694. The
frontend answers rootDSE and cn=Subschema before limits_check(), so olcLimits
never applies to them. Fix built and tested on 2.6.16 (+53/-3 lines): when a
writer makes no progress for 1 s while other work waits for a thread, the
connection is closed ("writer blocked and worker pool exhausted"); it extends
the existing pool-pause rule near result.c line 409. The patched release build
on default settings kept the victim at 10 of 10 against all four attack forms
(unpatched: 0 to 1 of 10). Not built for 2.7.2; the test suite was not run.
The fix does not fix the related problem reported separately as "back-mdb:
search queued behind a writer pins its read txn", which was re-tested on the
patched binary with an identical outcome. An authenticated client that can
read large data triggers the same wait.
Suggested fix: Tested fix for 2.6.16: close a connection whose writer is
blocked while the worker pool is starved (daemon.c, result.c, proto-slap.h).
diff --git a/servers/slapd/daemon.c b/servers/slapd/daemon.c
index 04dd1ff..aa243d2 100644
--- a/servers/slapd/daemon.c
+++ b/servers/slapd/daemon.c
@@ -3582,8 +3582,12 @@ slap_wake_listener()
}
/* return 0 on timeout, 1 on writer ready
- * -1 on general error
+ * -1 on general error, -2 if the pool is starved
*/
+#ifndef SLAPD_WRITEWAIT_SLICE
+#define SLAPD_WRITEWAIT_SLICE 1000 /* ms */
+#endif
+
int
slapd_wait_writer( ber_socket_t sd )
{
@@ -3603,11 +3607,40 @@ slapd_wait_writer( ber_socket_t sd )
return select( 0, NULL, &writefds, NULL, tvp );
#else
struct pollfd fds;
- int timeout = global_writetimeout ? global_writetimeout * 1000 : -1;
+ int left = global_writetimeout ? global_writetimeout * 1000 : -1;
fds.fd = sd;
fds.events = POLLOUT;
- return poll( &fds, 1, timeout );
+ /* Wait in slices so that a client that never reads cannot keep
+ * this thread (and the writers queued behind it on c_write1_cv)
+ * forever while other clients' operations wait for a thread.
+ */
+ for (;;) {
+ int slice = ( left < 0 || left > SLAPD_WRITEWAIT_SLICE ) ?
+ SLAPD_WRITEWAIT_SLICE : left;
+ int rc = poll( &fds, 1, slice );
+ if ( rc != 0 )
+ return rc;
+ if ( left > 0 && ( left -= slice ) <= 0 )
+ return 0; /* writetimeout */
+ if ( slap_pool_starved() )
+ return -2; /* blocked writer is starving the pool
*/
+ }
#endif
}
+
+/* nonzero if worker threads are all busy and work is queued for them,
+ * or a pool pause is pending
+ */
+int
+slap_pool_starved( void )
+{
+ int pending = 0;
+
+ if ( ldap_pvt_thread_pool_pausing( &connection_pool ) > 0 )
+ return 1;
+ ldap_pvt_thread_pool_query( &connection_pool,
+ LDAP_PVT_THREAD_POOL_PARAM_PENDING, &pending );
+ return pending > 0;
+}
diff --git a/servers/slapd/proto-slap.h b/servers/slapd/proto-slap.h
index 7df2921..c533df9 100644
--- a/servers/slapd/proto-slap.h
+++ b/servers/slapd/proto-slap.h
@@ -878,6 +878,7 @@ LDAP_SLAPD_F (void) slapd_clr_write LDAP_P((ber_socket_t s,
int wake));
LDAP_SLAPD_F (void) slapd_set_read LDAP_P((ber_socket_t s, int wake));
LDAP_SLAPD_F (int) slapd_clr_read LDAP_P((ber_socket_t s, int wake));
LDAP_SLAPD_F (int) slapd_wait_writer( ber_socket_t sd );
+LDAP_SLAPD_F (int) slap_pool_starved( void );
LDAP_SLAPD_F (void) slapd_shutsock( ber_socket_t sd );
LDAP_SLAPD_V (volatile sig_atomic_t) slapd_abrupt_shutdown;
diff --git a/servers/slapd/result.c b/servers/slapd/result.c
index bdd8a22..84e928f 100644
--- a/servers/slapd/result.c
+++ b/servers/slapd/result.c
@@ -354,6 +354,20 @@ static long send_ldap_ber(
conn->c_writers++;
while ( conn->c_writers > 0 && conn->c_writing ) {
+ /* The current writer is blocked on a client that is not
reading
+ * and other clients are waiting for a thread: don't park
another
+ * thread here, drop the connection (this wakes all its
writers).
+ */
+ if ( conn->c_writewaiter && slap_pool_starved() ) {
+ conn->c_writers--;
+ ldap_pvt_thread_mutex_unlock( &conn->c_write1_mutex );
+ ldap_pvt_thread_mutex_lock( &conn->c_mutex );
+ if ( op->o_connid == conn->c_connid )
+ connection_closing( conn,
+ "writer blocked and worker pool
exhausted" );
+ ldap_pvt_thread_mutex_unlock( &conn->c_mutex );
+ return -1;
+ }
ldap_pvt_thread_cond_wait( &conn->c_write1_cv,
&conn->c_write1_mutex );
}
@@ -427,6 +441,8 @@ fail:
if ( err <= 0 ) {
if ( err == 0 )
close_reason = "writetimeout";
+ else if ( err == -2 )
+ close_reason = "writer blocked and worker pool
exhausted";
else
close_reason = "connection lost on writewait";
goto fail;
Credit: when a CVE is assigned for this issue, please credit Joao Charles
Carvalho.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10586
Howard Chu <hyc(a)openldap.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Group|OpenLDAP-devs |
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10587
Howard Chu <hyc(a)openldap.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Group|OpenLDAP-devs |
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10588
Howard Chu <hyc(a)openldap.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Group|OpenLDAP-devs |
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10592
Howard Chu <hyc(a)openldap.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Group|OpenLDAP-devs |
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10595
Howard Chu <hyc(a)openldap.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Group|OpenLDAP-devs |
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10602
Howard Chu <hyc(a)openldap.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Group|OpenLDAP-devs |
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10616
Howard Chu <hyc(a)openldap.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Group|OpenLDAP-devs |
--
You are receiving this mail because:
You are on the CC list for the issue.