https://bugs.openldap.org/show_bug.cgi?id=10395
Issue ID: 10395
Summary: Support multiple readers on uncommitted changes
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: renault.cle(a)gmail.com
Target Milestone: ---
Created attachment 1086
--> https://bugs.openldap.org/attachment.cgi?id=1086&action=edit
A patch to support multiple readers on uncommitted changes
Hello,
The attached patch is not meant to be merged immediately into LMDB. Still, it
demonstrates how I added a helpful feature to the key-value store: reading
uncommitted changes from multiple transactions. I am conscious that the patch
still requires some work and uses non-C99 features, i.e., atomics are C11,
which could be a blocker for it to be merged upstream. I would also be
delighted to merge these changes under a flag/define to ensure we don't impact
other users with non-C99 stuff.
The main feature we need at Meilisearch is to read uncommitted changes from
multiple threads, compute parallel post-processing of different data structures
[1], and speed up the search requests. We could have done the post-processing
in a following transaction by opening multiple read transactions, but this
would mean that the post-processed data structure would not include newly
inserted or modified document IDs. Both data structures would be desync.
Regarding the design choice, I decided to follow the same design as the nested
write transactions: use the parent argument of the mdb_txn_begin [2], and allow
the MDB_RDONLY flag, which was disallowed when the parent argument was non-NULL
[3]. I find it clear enough that, by calling the mdb_txn_begin function with
these arguments, you can call it multiple times (I need to update the doc) to
obtain nested read-only transactions from the parent write transaction.
ret = mdb_txn_begin(env, parent_txn, MDB_RDONLY, new_nested_rtxn);
Note that this early proposal lacks security and error handling. The generated
transactions are fake-read-only and actually write transactions that share the
underlying parent allocations and data structures. This is unsafe and must be
changed or reviewed carefully, but most importantly, we need to add read-only
capabilities to these transactions to disallow writes. Using a Rust wrapper on
top of LMDB, I wrapped the fake read-only transactions into ReadTxn, which
disallows any writes at compile time. However, I haven't checked the conflict
database creations or openings.
The main issues I encountered were concurrent free of the main shared data
structures when the different threads owning the transactions were dropping the
transactions simultaneously. So, I decided to implement the equivalent of an
ARC to free resources only when the last nested transaction was freed.
I can share numbers about how this feature improves the post-processing by
4x-9x or from 1200s to 120s [1]. You can look at this PR, which I would be
happy to merge once an improved version of this patch lands on LMDB upstream.
I would be very happy if you could guide me a bit on how I could improve this
patch to make it mergeable into LMDB. We want to contribute useful features
like this to LMDB and not keep a deviant fork. LMDB works great; we are happy
about it, and its performance is predictable.
Have a lovely week,
kero
[1]: https://github.com/meilisearch/meilisearch/pull/5307
[2]:
https://github.com/LMDB/lmdb/blob/14d6629bc8a9fe40d8a6bee1bf71c45afe7576b6/…
[3]:
https://github.com/LMDB/lmdb/blob/14d6629bc8a9fe40d8a6bee1bf71c45afe7576b6/…
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10420
Issue ID: 10420
Summary: Add support for building on Haiku
Product: LMDB
Version: unspecified
Hardware: All
OS: Other
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: alizter(a)gmail.com
Target Milestone: ---
The following patch allows for lmdb to be built on Haiku:
```
diff --git a/vendor/ocaml-lmdb/mdb.c b/vendor/ocaml-lmdb/mdb.c
index 64e5ffd254..ffd8618ffa 100644
--- a/vendor/ocaml-lmdb/mdb.c
+++ b/vendor/ocaml-lmdb/mdb.c
@@ -174,6 +174,9 @@
# endif
#elif defined(__ANDROID__)
# define MDB_FDATASYNC fsync
+#elif defined(__HAIKU__)
+# define MDB_USE_POSIX_SEM 1
+# define MDB_FDATASYNC fsync
#endif
```
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10419
Issue ID: 10419
Summary: NetBSD does not define union semun nor set
_SEM_SEMUN_UNDEFINED
Product: LMDB
Version: unspecified
Hardware: All
OS: Other
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: alizter(a)gmail.com
Target Milestone: ---
NetBSD does not define union semun nor set _SEM_SEMUN_UNDEFINED. This means
that the union defined when _SEM_SEMUN_UNDEFINED is set is skipped leading to a
compilation error.
Adding this to `mdb.c` will fix this issue by setting this variable in NetBSD
when it is undefined (and it usually is, I can't speak for the future).
```
diff --git a/mdb.c b/mdb.c
index 3e3f529b98..64e5ffd254 100644
--- a/vendor/ocaml-lmdb/mdb.c
+++ b/vendor/ocaml-lmdb/mdb.c
@@ -176,6 +176,11 @@
# define MDB_FDATASYNC fsync
#endif
+/* NetBSD does not define union semun in sys/sem.h */
+#if defined(__NetBSD__) && !defined(_SEM_SEMUN_UNDEFINED)
+# define _SEM_SEMUN_UNDEFINED 1
+#endif
+
#ifndef _WIN32
#include <pthread.h>
#include <signal.h>
```
FreeBSD and OpenBSD don't suffer from this issue as they both correctly set
that variable.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10424
Issue ID: 10424
Summary: When using more than one syncrepl directive on a
single DB, the contextCSN should be stored accordingly
to the replication base
Product: OpenLDAP
Version: 2.6.10
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: overlays
Assignee: bugs(a)openldap.org
Reporter: elecharny(a)apache.org
Target Milestone: ---
Trying to replicate cn=config but parially, I tried to set many syncrepl
directives so that each oe of them just replicate the single element it is
configured for. Here is an exemple thet replicate the frontend database and
only it:
{0}rid=004
provider=ldaps://openldap1:10636
bindmethod=simple
binddn="cn=syncrepl,ou=accounts,dc=worteks,dc=com"
credentials=blah
tls_cacert=/opt/application/openldap/ssl/ca.crt
tls_cert=/opt/application/openldap/ssl/openldap-common.crt
tls_key=/opt/application/openldap/ssl/openldap-common.key
tls_reqcert=demand
tls_protocol_min=3.3
searchbase="olcDatabase={-1}frontend,cn=config"
scope="base"
type=refreshAndPersist
retry="30 +"
timeout=1
schemachecking=off
If I have this single syncrepl directive, all is ok, it replicates the frontend
data and only this entry.
Would I like to replicate, says, 'cn=config' with another syncrepl directive
like:
olcSyncrepl: {0}rid=001
provider=ldaps://openldap1:10636
bindmethod=simple
binddn="cn=syncrepl,ou=accounts,dc=worteks,dc=com"
credentials=blah
tls_cacert=/opt/application/openldap/ssl/ca.crt
tls_cert=/opt/application/openldap/ssl/openldap-common.crt
tls_key=/opt/application/openldap/ssl/openldap-common.key
tls_reqcert=demand
tls_protocol_min=3.3
searchbase="cn=config"
scope="base"
type=refreshAndPersist
retry="30 +"
timeout=1
schemachecking=off
where the search base is different, then suddenly I get some 'CSN too old'
errors, which make totally sense as we only have one single contextCSN stored
in the root entry (cn=config in my use case).
I know I'm really border line here (and the rational is that I want a partial
replication of cn=config because the two servers are a bit different), but I
would suggest that the contextCSN to be stored in the entry associated to the
searchBase, not at the root of the database.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10423
Issue ID: 10423
Summary: It is possible via cn=config to add an entry of one
database type with the object class of another,
causing OpenLDAP to crash
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: nivanova(a)symas.com
Target Milestone: ---
This happens specifically because the user by mistake attempts to create a meta
database by specifying the object class for back-ldap. Since these share a lot
of common attributes, schema checks do not fail. As a result, slapd initializes
a meta database (based on olcDatabase value), but attempts to manipulate an
initialized back-ldap database after that, and segfaults.
Like this:
dn: olcDatabase=meta,cn=config
objectClass: olcDatabaseConfig
objectClass: olcLDAPConfig
olcDatabase: meta
olcDbURI: "ldap://example.com"
While it seems this is only possible for the proxy databases, we cannot rely
that schema checks on attributes will fail for other database combinations, who
knows what common attributes can be added or supplied in the future, we we have
to fix this.
The issue is reproducible via slapadd and at runtime with ldapadd.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10416
Issue ID: 10416
Summary: Allow setting fractional timeouts in ldaprc
Product: OpenLDAP
Version: 2.6.10
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: enhancement
Priority: ---
Component: libraries
Assignee: bugs(a)openldap.org
Reporter: hyc(a)openldap.org
Target Milestone: ---
Created attachment 1099
--> https://bugs.openldap.org/attachment.cgi?id=1099&action=edit
Patch to libldap init.c
Currently the ldap.conf parser only accepts integer seconds, even though the
internal variable also accommodates microseconds. A simple patch is attached.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10388
Issue ID: 10388
Summary: ldif_parse_line2 is not compliant with RFC2849
Product: OpenLDAP
Version: 2.6.10
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: libraries
Assignee: bugs(a)openldap.org
Reporter: nivanova(a)symas.com
Target Milestone: ---
RFC2849:
Any
value that contains characters other than those defined as
"SAFE-CHAR", or begins with a character other than those
defined as "SAFE-INIT-CHAR", above, MUST be base-64 encoded.
Other values MAY be base-64 encoded.
SAFE-INIT-CHAR = %x01-09 / %x0B-0C / %x0E-1F /
%x21-39 / %x3B / %x3D-7F
; any value <= 127 except NUL, LF, CR,
; SPACE, colon (":", ASCII 58 decimal)
; and less-than ("<" , ASCII 60 decimal)
However, if the value is not base64 encoded, ldif_parse_line2 (and consequently
ldap_parse_ldif_record_x) uses isspace() to just skip any white-space
characters at the beginning of at attribute value, icl. tabs. According to the
RFC, however, such characters are acceptable. In addition, it does not return
an error on LF or CR, just skips them.
On the one hand, LDIFs are supposed to be human readable, and fixing this issue
may lead to unexpected problems (e.g a stray tab being added to the attribute
value). On the other hand, the standard does not exclude %x01-09 and %x0B-0C as
valid characters for a non-encoded value.
How should we proceed?
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10415
Issue ID: 10415
Summary: Allow limiting the buffers for data pending to be sent
to client
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: lloadd
Assignee: bugs(a)openldap.org
Reporter: ondra(a)mistotebe.net
Target Milestone: ---
In many scenarios it is more useful to limit the amount of data we buffer to be
written to the client's socket even if it means that sometimes a connection
might have to be dropped abruptly to enforce this.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9959
Issue ID: 9959
Summary: Expose the lloadd connection endpoints in cn=monitor
for identification
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: lloadd
Assignee: bugs(a)openldap.org
Reporter: ondra(a)mistotebe.net
Target Milestone: ---
ITS#9600 introduces a way to operate on lloadd connections, however it is
impossible to identify which socket this actually corresponds to. While we're
at it, might also expose the current bound identity there.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=8988
--- Comment #29 from Quanah Gibson-Mount <quanah(a)openldap.org> ---
• 8ba07ad7
by Howard Chu at 2025-12-16T17:33:38+00:00
ITS#8988 lmdb: make all keys and data 2-byte aligned
--
You are receiving this mail because:
You are on the CC list for the issue.