https://bugs.openldap.org/show_bug.cgi?id=10274
Issue ID: 10274
Summary: Replication issue on MMR configuration
Product: OpenLDAP
Version: 2.5.14
Hardware: All
OS: Linux
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: falgon.comp(a)gmail.com
Target Milestone: ---
Created attachment 1036
--> https://bugs.openldap.org/attachment.cgi?id=1036&action=edit
In this attachment you will find 2 openldap configurations for 2 instances +
slamd conf exemple + 5 screenshots to show the issue and one text file to
explain what you see
Hello we are openning this issue further to the initial post in technical :
https://lists.openldap.org/hyperkitty/list/openldap-technical@openldap.org/…
Issue :
We are working on a project and we've come across an issue with the replication
after performance testing :
*Configuration :*
RHEL 8.6
OpenLDAP 2.5.14
*MMR-delta *configuration on multiple servers attached
300,000 users configured and used for tests
*olcLastBind: TRUE*
Use of SLAMD (performance shooting)
*Problem description:*
We are currently running performance and resilience tests on our infrastructure
using the SLAMD tool configured to perform BINDs and MODs on a defined range of
accounts.
We use a load balancer (VIP) to poll all of our servers equally. (but it is
possible to do performance tests directly on each of the directories)
With our current infrastructure we're able to perform approximately 300
MOD/BIND/s. Beyond that, we start to generate delays and can randomly come
across one issue.
However, when we run performance tests that exceed our write capacity, our
replication between servers can randomly create an incident with directories
being unable to catch up with their replication delay.
The directories update their contextCSNs, but extremely slowly (like freezing).
From then on, it's impossible for the directories to catch again. (even with no
incoming traffic)
A restart of the instance is required to perform a full refresh and solve the
incident.
We have enabled synchronization logs and have no error or refresh logs to
indicate a problem ( we can provide you with logs if necessary).
We suspect a write collision or a replication conflict but this is never write
in our sync logs.
We've run a lot of tests.
For example, when we run a performance test on a single live server, we don't
reproduce the problem.
Anothers examples: if we define different accounts ranges for each server with
SALMD, we don't reproduce the problem either.
If we use only one account for the range, we don't reproduce the problem
either.
______________________________________________________________________
I have add some screenshots on attachement to show you the issue and all the
explanations.
______________________________________________________________________
*Symptoms :*
One or more directories can no longer be replicated normally after performance
testing ends.
No apparent error logs.
Need a restart of instances to solve the problem.
*How to reproduce the problem:*
Have at least two servers in MMR mode
Set LastBind to TRUE
Perform a SLAMD shot from a LoadBalancer in bandwidth mode OR start multiple
SLAMD test on same time for each server with the same account range.
Exceed the maximum write capacity of the servers.
*SLAMD configuration :*
authrate.sh --hostname ${HOSTNAME} --port ${PORTSSL} \
--useSSL --trustStorePath ${CACERTJKS} \
--trustStorePassword ${CACERTJKSPW} --bindDN "${BINDDN}" \
--bindPassword ${BINDPW} --baseDN "${BASEDN}" \
--filter "(uid=[${RANGE}])" --credentials ${USERPW} \
--warmUpIntervals ${WARMUP} \
--numThreads ${NTHREADS} ${ARGS}
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10236
Issue ID: 10236
Summary: fragmentation makes mdb_page_alloc slow
Product: LMDB
Version: 0.9.31
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: aalekseyev(a)janestreet.com
Target Milestone: ---
Created attachment 1022
--> https://bugs.openldap.org/attachment.cgi?id=1022&action=edit
patch is relative to LMDB_0.9.31
It's a known problem that mdb_page_alloc can be slow
when the free list is large and fragmented. [1] [2] [3]
I'm not sure it's known *how* slow it can be.
In our workload we saw a fragmented freelist leading
to a pathological O(n^2) behavior.
To handle a multi-page allocation we iterate loading chunks of the
free list one by one, and at every iteration we do O(n) work to check
if the allocation can succeed.
Even small-ish allocations (tens of pages) are repeatedly hitting
this edge case, with free list growing to ~1000000, and the outer loop
taking ~2000 iterations (10^9 worth of work in total, just to allocate a
few pages).
Even though I'm sure there are ways to avoid hitting this pathological
scenario so much (avoid values larger than 4k, or fix whatever causes
fragmentation), it seems unacceptable to have a performance cliff this bad.
I made a patch to make the allocation take ~O(n*log(n)), by loading
and merging multiple chunks at once instead of doing it one-by-one.
I'd appreciate it if someone could review the patch (attached), improve it,
and/or come up with an alternative fix.
The code in `midl.c` is kinda meme-y, including a contribution from GPT-4o, but
it performs well enough to speed up our pathological workload by ~20x (which is
still ~3x away from the non-fragmented case).
Anyway, the main thing that warrants scrutiny is the change in `mdb.c`:
I understand very little about lmdb internals and I worry that loading
multiple pages at once instead of one-by-one might break something.
[1] issue #8664
[2]
https://lists.openldap.org/hyperkitty/list/openldap-bugs@openldap.org/threa…
[3]
https://lists.openldap.org/hyperkitty/list/openldap-technical@openldap.org/…
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10383
Issue ID: 10383
Summary: slapd-meta ignores olcDbIDAssertBind if olcDbURI
defined after it
Product: OpenLDAP
Version: 2.6.9
Hardware: x86_64
OS: Linux
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: david.frickert(a)protonmail.com
Target Milestone: ---
Hi,
We are using slapd-meta to connect an OpenLDAP server to another external LDAP
server and it works well on first configuration.
However, if we want to update any info, e.g. the external LDAP URI, we must
replace the olcDbURI attribute. This means that the ordering of the attributes
change and this attribute is now defined after olcDbIDAssertBind.
Didn't think this would be important, but after this change the "meta"
connection stops working and upon enabling debugging i can see that the
external LDAP server is responding with:
"ldap_bind: Inappropriate authentication (48)
additional info: Anonymous Simple Bind Disabled"
This seems to imply that the olcDbIDAssertBind attribute is being ignored,
likely due to being defined before olcDbURI (my assumption).
Is this intended? If so, what can we do to mitigate this problem?
Do we need to perform a replace on all attributes of the object to ensure
correct ordering, or is there any way to perform an in-place attribute
modification without making it shift its position in the object?
Example:
First configuration (OK):
# {0}uri, {2}meta, config
dn: olcMetaSub={0}uri,olcDatabase={2}meta,cn=config
objectClass: olcMetaTargetConfig
olcMetaSub: {0}uri
--> olcDbURI: ldap://REDACTED/ou=users,REDACTED
olcDbIDAssertBind: bindmethod=simple starttls=yes tls_reqcert=demand
binddn="REDACTED" credentials="REDACTED"
olcDbRewrite: {0}suffixmassage REDACTED REDACTED
olcDbKeepalive: 0:0:0
olcDbBindTimeout: 100000
olcDbCancel: abandon
After URI update (NOK):
# {0}uri, {2}meta, config
dn: olcMetaSub={0}uri,olcDatabase={2}meta,cn=config
objectClass: olcMetaTargetConfig
olcMetaSub: {0}uri
olcDbIDAssertBind: bindmethod=simple starttls=yes tls_reqcert=demand
binddn="REDACTED" credentials="REDACTED"
olcDbRewrite: {0}suffixmassage REDACTED REDACTED
olcDbKeepalive: 0:0:0
olcDbBindTimeout: 100000
olcDbCancel: abandon
--> olcDbURI: ldap://REDACTED/ou=users,REDACTED
The olcDbURI attribute is shifted to the bottom after a modify operation, and
seems to cause these issues.
Best Regards,
David
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10026
Issue ID: 10026
Summary: Refresh handling can skip entries (si_dirty not
managed properly)
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: overlays
Assignee: bugs(a)openldap.org
Reporter: ondra(a)mistotebe.net
Target Milestone: ---
Take MPR plain syncrepl with 3+ providers.
When a provider's own syncrepl session transitions to persist and a it starts a
new parallel session towards another host, that session always has to start as
a refresh. If that refresh serves entries to us, our handling of si_dirty is
not consistent:
- if the existing persist session serves some of these entries to us, we can
"forget" to pass the others to a newly connected consumer
- same if the refresh is abandoned and we start refreshing from a different
provider that might be behind what we were being served (again our consumers
could suffer)
- if we restart, si_dirty is forgotten and our consumers suffer even worse
We might need to be told (at the beginning of the refresh?) what the end state
we're going for is, so we can keep si_dirty on until then. And somehow persist
that knowledge in the DB...
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9343
Issue ID: 9343
Summary: Expand ppolicy policy configuration to allow URL
filter
Product: OpenLDAP
Version: 2.5
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: overlays
Assignee: bugs(a)openldap.org
Reporter: quanah(a)openldap.org
Target Milestone: ---
Currently, ppolicy only supports a single global default policy, and past that
any policies must be manually added to a given user entry if they are supposed
to have something other than the default policy.
Also, some sites want no default policy, and only a specific subset to have a
policy applied to them.
For both of these cases, it would be helpful if it were possible to configure a
policy to apply to a set of users via a URL similar to the way we handle
creating groups of users in dynlist
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9640
Issue ID: 9640
Summary: ACL privilege for MOD_INCREMENT
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: michael(a)stroeder.com
Target Milestone: ---
I'm using LDAP write operations with MOD_INCREMENT with pre-read-control for
uidNumber/gidNumber generation.
I'd like to limit write access to an Integer attribute "nextID" to
MOD_INCREMENT, ideally even restricting the de-/increment value.
(Uniqueness is achieved with slapo-unique anyway but still I'd like to avoid
users messing with this attribute).
IMHO the ideal solution would be a new privilege "i".
Example for limiting write access to increment by one and grant read access for
using read control:
access to
attrs=nextID
val=1
by group=... =ri
Example for decrementing by two without read:
access to
attrs=nextID
val=-2
by group=... =i
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10395
Issue ID: 10395
Summary: Support multiple readers on uncommitted changes
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: renault.cle(a)gmail.com
Target Milestone: ---
Created attachment 1086
--> https://bugs.openldap.org/attachment.cgi?id=1086&action=edit
A patch to support multiple readers on uncommitted changes
Hello,
The attached patch is not meant to be merged immediately into LMDB. Still, it
demonstrates how I added a helpful feature to the key-value store: reading
uncommitted changes from multiple transactions. I am conscious that the patch
still requires some work and uses non-C99 features, i.e., atomics are C11,
which could be a blocker for it to be merged upstream. I would also be
delighted to merge these changes under a flag/define to ensure we don't impact
other users with non-C99 stuff.
The main feature we need at Meilisearch is to read uncommitted changes from
multiple threads, compute parallel post-processing of different data structures
[1], and speed up the search requests. We could have done the post-processing
in a following transaction by opening multiple read transactions, but this
would mean that the post-processed data structure would not include newly
inserted or modified document IDs. Both data structures would be desync.
Regarding the design choice, I decided to follow the same design as the nested
write transactions: use the parent argument of the mdb_txn_begin [2], and allow
the MDB_RDONLY flag, which was disallowed when the parent argument was non-NULL
[3]. I find it clear enough that, by calling the mdb_txn_begin function with
these arguments, you can call it multiple times (I need to update the doc) to
obtain nested read-only transactions from the parent write transaction.
ret = mdb_txn_begin(env, parent_txn, MDB_RDONLY, new_nested_rtxn);
Note that this early proposal lacks security and error handling. The generated
transactions are fake-read-only and actually write transactions that share the
underlying parent allocations and data structures. This is unsafe and must be
changed or reviewed carefully, but most importantly, we need to add read-only
capabilities to these transactions to disallow writes. Using a Rust wrapper on
top of LMDB, I wrapped the fake read-only transactions into ReadTxn, which
disallows any writes at compile time. However, I haven't checked the conflict
database creations or openings.
The main issues I encountered were concurrent free of the main shared data
structures when the different threads owning the transactions were dropping the
transactions simultaneously. So, I decided to implement the equivalent of an
ARC to free resources only when the last nested transaction was freed.
I can share numbers about how this feature improves the post-processing by
4x-9x or from 1200s to 120s [1]. You can look at this PR, which I would be
happy to merge once an improved version of this patch lands on LMDB upstream.
I would be very happy if you could guide me a bit on how I could improve this
patch to make it mergeable into LMDB. We want to contribute useful features
like this to LMDB and not keep a deviant fork. LMDB works great; we are happy
about it, and its performance is predictable.
Have a lovely week,
kero
[1]: https://github.com/meilisearch/meilisearch/pull/5307
[2]:
https://github.com/LMDB/lmdb/blob/14d6629bc8a9fe40d8a6bee1bf71c45afe7576b6/…
[3]:
https://github.com/LMDB/lmdb/blob/14d6629bc8a9fe40d8a6bee1bf71c45afe7576b6/…
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10353
Issue ID: 10353
Summary: No TLS connection on Windows because of missing
ENOTCONN in socket.h
Product: OpenLDAP
Version: 2.6.10
Hardware: All
OS: Windows
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: libraries
Assignee: bugs(a)openldap.org
Reporter: julien.wadel(a)belledonne-communications.com
Target Milestone: ---
On Windows, the TLS connection cannot be done and we get the connection error:
Can't contact LDAP server.
=> Connections are done with WSAGetLastError().
After getting WSAEWOULDBLOCK, the connection is not restart because of the
state WSAENOTCONN that is not known.
OpenLDAP use ENOTCONN that is set to 126 by "ucrt/errno.h" while WSAENOTCONN
is 10057L.
Adding #define ENOTCONN WSAENOTCONN
like for EWOULDBLOCK resolve the issue.
Reference commit on external project:
https://gitlab.linphone.org/BC/public/external/openldap/-/commit/62fbfb12e8…
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9739
Issue ID: 9739
Summary: Undefined reference to ber_sockbuf_io_udp in 2.6.0
Product: OpenLDAP
Version: 2.6.0
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: build
Assignee: bugs(a)openldap.org
Reporter: simon.pichugin(a)gmail.com
Target Milestone: ---
While I was trying to build OpenLDAP 2.6 on Fedora Rawhide I've got the error
message:
/usr/bin/ld: ./.libs/libldap.so: undefined reference to
`ber_sockbuf_io_udp'
I've checked commits from https://bugs.openldap.org/show_bug.cgi?id=9673 and
found that 'ber_sockbuf_io_udp' was not added to
https://git.openldap.org/openldap/openldap/-/blob/master/libraries/liblber/…
I've asked on the project's mailing list and got a reply:
"That symbol only exists if OpenLDAP is built with LDAP_CONNECTIONLESS
defined, which is not a supported feature. Feel free to file a bug report
at https://bugs.openldap.org/"
https://lists.openldap.org/hyperkitty/list/openldap-technical@openldap.org/…
Hence, creating the bug.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9716
Issue ID: 9716
Summary: Fix default guide versions for head and 2.6
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: documentation
Assignee: bugs(a)openldap.org
Reporter: quanah(a)openldap.org
Target Milestone: ---
The preamble file needs to be fixed in openldap head and the 2.6 release branch
to be correct.
--
You are receiving this mail because:
You are on the CC list for the issue.