https://bugs.openldap.org/show_bug.cgi?id=10395
Issue ID: 10395
Summary: Support multiple readers on uncommitted changes
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: renault.cle(a)gmail.com
Target Milestone: ---
Created attachment 1086
--> https://bugs.openldap.org/attachment.cgi?id=1086&action=edit
A patch to support multiple readers on uncommitted changes
Hello,
The attached patch is not meant to be merged immediately into LMDB. Still, it
demonstrates how I added a helpful feature to the key-value store: reading
uncommitted changes from multiple transactions. I am conscious that the patch
still requires some work and uses non-C99 features, i.e., atomics are C11,
which could be a blocker for it to be merged upstream. I would also be
delighted to merge these changes under a flag/define to ensure we don't impact
other users with non-C99 stuff.
The main feature we need at Meilisearch is to read uncommitted changes from
multiple threads, compute parallel post-processing of different data structures
[1], and speed up the search requests. We could have done the post-processing
in a following transaction by opening multiple read transactions, but this
would mean that the post-processed data structure would not include newly
inserted or modified document IDs. Both data structures would be desync.
Regarding the design choice, I decided to follow the same design as the nested
write transactions: use the parent argument of the mdb_txn_begin [2], and allow
the MDB_RDONLY flag, which was disallowed when the parent argument was non-NULL
[3]. I find it clear enough that, by calling the mdb_txn_begin function with
these arguments, you can call it multiple times (I need to update the doc) to
obtain nested read-only transactions from the parent write transaction.
ret = mdb_txn_begin(env, parent_txn, MDB_RDONLY, new_nested_rtxn);
Note that this early proposal lacks security and error handling. The generated
transactions are fake-read-only and actually write transactions that share the
underlying parent allocations and data structures. This is unsafe and must be
changed or reviewed carefully, but most importantly, we need to add read-only
capabilities to these transactions to disallow writes. Using a Rust wrapper on
top of LMDB, I wrapped the fake read-only transactions into ReadTxn, which
disallows any writes at compile time. However, I haven't checked the conflict
database creations or openings.
The main issues I encountered were concurrent free of the main shared data
structures when the different threads owning the transactions were dropping the
transactions simultaneously. So, I decided to implement the equivalent of an
ARC to free resources only when the last nested transaction was freed.
I can share numbers about how this feature improves the post-processing by
4x-9x or from 1200s to 120s [1]. You can look at this PR, which I would be
happy to merge once an improved version of this patch lands on LMDB upstream.
I would be very happy if you could guide me a bit on how I could improve this
patch to make it mergeable into LMDB. We want to contribute useful features
like this to LMDB and not keep a deviant fork. LMDB works great; we are happy
about it, and its performance is predictable.
Have a lovely week,
kero
[1]: https://github.com/meilisearch/meilisearch/pull/5307
[2]:
https://github.com/LMDB/lmdb/blob/14d6629bc8a9fe40d8a6bee1bf71c45afe7576b6/…
[3]:
https://github.com/LMDB/lmdb/blob/14d6629bc8a9fe40d8a6bee1bf71c45afe7576b6/…
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10362
Issue ID: 10362
Summary: Normalised value confusions in replication
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: ondra(a)mistotebe.net
Target Milestone: ---
A lot of code expects a_nvals state to be consistent across modifies and
multiple instances of the same attribute (with asserts to that effect), but
replication doesn't seem to obey this:
Add this assert into syncrepl_diff_entry()[0] :
assert( ( old->a_nvals == NULL ) == ( new->a_nvals == NULL ) );
if ( old->a_nvals )
assert( ( old->a_nvals == old->a_vals ) == ( new->a_nvals == new->a_vals )
);
You'll see many tests fail this, the first of which is test043.
[0].
https://git.openldap.org/openldap/openldap/-/blob/14d47146b0442df85bd949b21…
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10403
Issue ID: 10403
Summary: Add a configuration directive that uses the OpenSSL
CONF API to allow openldap config files to set any
configuration supported by that API, and to get new
OpenSSL configuration capabilities through that API
with no changes in openldap.
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: stephen.wall(a)redcom.com
Target Milestone: ---
Created attachment 1090
--> https://bugs.openldap.org/attachment.cgi?id=1090&action=edit
Add a configuration directive that uses the OpenSSL CONF API to allow openldap
config files to set any configuration supported by that API, and to get new
OpenSSL configuration capabilities through th
I am submitting a patch to create a new directive for OpenLDAP config files
that uses the OpenSSL SSL_CONF API to allow configuration of any aspect of
OpenSSL that the API supports without adding specific directives to OpenLDAP
for them. When OpenSSL extends that API, all versions of OpenLDAP with thie
patch will also support those extensions with no additional code.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10364
Issue ID: 10364
Summary: back-asyncmeta should process a notice-of-disconnect
and close the target connection
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: backends
Assignee: bugs(a)openldap.org
Reporter: nivanova(a)symas.com
Target Milestone: ---
At present, back-asyncmeta discards all unsolicited messages, received on the
target connections. It should instead process Notice-of-Disconnection messages
and close the connections on which they were received.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10363
Issue ID: 10363
Summary: Implement a target connection time-to-live in
asyncmeta
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: backends
Assignee: bugs(a)openldap.org
Reporter: nivanova(a)symas.com
Target Milestone: ---
Implement a feature that limits the maximum time before a target connection is
reset, even if it is not idle - conn-ttl. To avoid too many target connections
timing out at once, a minimum reset interval per target should also be
configurable. So, a configuration of:
conn-ttl 5s 5s
would mean connections have a 5 seconds time-to-live, but a connection should
be reset no more frequently that once every 5 seconds per target.
This feature was requested by a customer.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10332
Issue ID: 10332
Summary: Add support for SSLKEYLOGFILE environment variable to
export keys for Wireshark decryption
Product: OpenLDAP
Version: 2.6.9
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: libraries
Assignee: bugs(a)openldap.org
Reporter: michael.osipov(a)siemens.com
Target Milestone: ---
Please add support to do the following:
SSLKEYLOGFILE=keylog.txt ldapsearch -H ldaps://...
Other libraries and tools support it to decrypt the TLS traffic with Wireshark
for analysis purposes.
Curl has a simple, but complete implementation:
https://github.com/curl/curl/blob/e008f71f435a39875d86885a96b2eb8968a60fd4/…
Maybe it can be reused if license allows that?!
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10311
Issue ID: 10311
Summary: Work with IETF LDAP working group to update password
hashing mechanism RFC
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: documentation
Assignee: bugs(a)openldap.org
Reporter: quanah(a)openldap.org
Target Milestone: ---
Work with the IETF ldap working group to update the RFC to make the suggested
hashing mechanism be what is currently "best practice" rather than a specific
hashing mechanism.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10286
Issue ID: 10286
Summary: ldap_pvt_gettime may result in "not new enough csn"
problems in multi-thread case.
Product: OpenLDAP
Version: 2.6.6
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: libraries
Assignee: bugs(a)openldap.org
Reporter: 971748261(a)qq.com
Target Milestone: ---
Created attachment 1041
--> https://bugs.openldap.org/attachment.cgi?id=1041&action=edit
the log of adding two entry which shows the time sequence.
I used openldap as krb5's database, and openldap was deploymented in
mirrormode.
I tried to add kerberos principals via kadmin.local -q "addprinc -randkey
principal.
slapd log showed that the entry of kadmin/admin was added earlier than the
entry of ossuser. But the csn of kadmin/admin was greater than ossuser.
In this case, when the two entry began to sync to the other slapd server,
ossuser was ignored because of "csn not new enough"
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10100
Issue ID: 10100
Summary: Non-sequential timestamps being logged on Windows
Product: OpenLDAP
Version: 2.6.6
Hardware: x86_64
OS: Windows
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: smckinney(a)symas.com
Target Milestone: ---
Presents as a dsync during replication. Consumer will log
```
650af021.2eadd901 0000000000001b40 slap_queue_csn: queueing 0000000002ac1620
20230920131409.992477Z#000000#001#000000
650af021.2eaed239 0000000000001b40 slap_graduate_commit_csn: removing
0000000002ac1620 20230920131409.992477Z#000000#001#000000
650af021.317b2a35 000000000000185c do_syncrep2: rid=102 CSN too old, ignoring
20230920131409.040136Z#000000#001#000000
(uid=slapd-test1-FOO1-6,ou=People,dc=example,dc=com)
```
The entry was not be added.
The provider will log messages using non-sequential timestamps. For example,
when grepping the CSN from above (in provider log):
```
# This:
650af021.3b3060d9 0000000000001ad8 conn=1001 op=1 syncprov_sendresp: to=002,
cookie=rid=102,sid=001,csn=20230920131409.992477Z#000000#001#000000
# and:
650af021.02648749 0000000000001810 slap_get_csn: conn=1003 op=7 generated new
csn=20230920131409.040136Z#000000#001#000000 manage=1
```
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10278
Issue ID: 10278
Summary: Move away from python-ldap0 in the test suite
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: test suite
Assignee: bugs(a)openldap.org
Reporter: ondra(a)mistotebe.net
Target Milestone: ---
While python-ldap0 has a superior API, it seems the module is no longer
receiving any development. We should move our python test suite over to another
module that can be supported long-term.
--
You are receiving this mail because:
You are on the CC list for the issue.