https://bugs.openldap.org/show_bug.cgi?id=10406
Issue ID: 10406
Summary: Write regression since 0.9.19
Product: LMDB
Version: 0.9.33
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: ben.alex(a)acegi.com.au
Target Milestone: ---
I maintain LmdbJava, a Java wrapper for LMDB. We have an extensive benchmark
suite covering read, cursor, and write pathways, with isolated testing of LMDB
versions from 0.9.17 through 0.9.33.
Results: https://lmdb-benchmark.lmdbjava.org/
==============================================
SUMMARY
==============================================
Read performance: 0.9.33 consistently outperforms 0.9.17 across all read
benchmarks.
Write performance: Consistent regression since 0.9.19:
----------------------------------
Tag ms/op vs 0.9.17
----------------------------------
LMDB_0.9.17 75.790 baseline
LMDB_0.9.18 74.909 -1.2%
LMDB_0.9.19 81.404 +7.4%
LMDB_0.9.20 83.459 +10.1%
LMDB_0.9.21 83.237 +9.8%
LMDB_0.9.22 83.027 +9.5%
LMDB_0.9.23 82.870 +9.3%
LMDB_0.9.24 82.952 +9.4%
LMDB_0.9.27 82.989 +9.5%
LMDB_0.9.28 83.025 +9.5%
LMDB_0.9.29 83.063 +9.6%
LMDB_0.9.30 83.237 +9.8%
LMDB_0.9.31 88.419 +16.7%
LMDB_0.9.33 83.304 +9.9%
Excluding 0.9.31 as an outlier, write performance degraded by approximately 9%
from 0.9.20 and has remained at that level through 0.9.33.
==============================================
BENCHMARK DETAILS
==============================================
The write benchmark sequentially inserts 1,000,000 entries (100-byte values,
4-byte integer keys) into a database opened with MDB_INTEGERKEY, using a single
cursor with MDB_APPEND flag. The benchmark includes 2 warmup iterations and 3
measurement iterations of 120 seconds each.
==============================================
ANALYSIS
==============================================
Reviewing the commit history, the regressions appear linked to correctness
fixes:
- 0.9.19: ITS#8406 (xcursor fixup after cursor_del)
- 0.9.20: ITS#8557 (cursor_last and C_EOF handling)
Both issues added cursor state management overhead in write-heavy code paths.
==============================================
REQUEST
==============================================
Could you please reproduce this regression using a native C benchmark to
confirm it's not JNI-specific? If confirmed, this may be a necessary
performance/correctness tradeoff given that most LMDB workloads are read-heavy.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10266
Issue ID: 10266
Summary: Adopt broader RFC4511 NoD interpretation on lloadd's
client side
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: lloadd
Assignee: bugs(a)openldap.org
Reporter: ondra(a)mistotebe.net
Target Milestone: ---
Server side, lloadd has long implemented a broad interpretation of NoD
unsolicited response handling: when the message is issued, no new requests are
accepted on the session however the client and server are both free to keep the
session open if there are any operations that have not resolved yet. The server
is still expected to close the connection as soon as no operations are still
pending.
This seems to interoperate with known clients. Those that want to will close
the session immediately, unaware of this possibility, those that also want to
interpret RFC 4511 this way can choose to wait for existing operations to
resolve.
This ticket is to track the lloadd's implementation of the client side of this
- when receiving a NoD message, we don't close the connection
immediately+unconditionally either but are willing to wait.
Related functionality:
- if connection was a bind connection processing a multi-stage SASL bind, the
bind should fail if/when the client attempts to progress it
- clients assigned to this connection through coherence at least 'connection'
are also marked closing
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10229
Issue ID: 10229
Summary: ldap_result, when invoked with MSG_RECEIVED and a
timeout value set to 0 (polling), does not return all
available messages until it is called again
Product: OpenLDAP
Version: 2.6.8
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: libraries
Assignee: bugs(a)openldap.org
Reporter: nivanova(a)symas.com
Target Milestone: ---
The issue is noticeable when ldap_result is used by the proxy back-ends. It has
not affected back-meta behavior, because when a first call is unsuccessful, it
retries with a small timeout. back-asyncmeta will also usually call it twice on
the same connection from different threads, although this is not a desired
behavior.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10250
Issue ID: 10250
Summary: syncrepl_diff_entry assumes attributes come in the
same order
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: ondra(a)mistotebe.net
Target Milestone: ---
When trying to diff an entry, syncrepl_diff_entry explicitly assumes attribute
come in the same order. That's not always the case and could cause it to report
a spurious rewrite of the attribute.
Normally this is ok, unless the rewrite itself (not) occurring has other
side-effects, when it could cause issues. (e.g. a DB with memberof
inconsistencies being mysteriously repaired in some scenarios, which is how it
was found).
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9888
Issue ID: 9888
Summary: When using cn=config replication, schema updates can
corrupt the index database(s)
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: quanah(a)openldap.org
Target Milestone: ---
Today I pushed a schema update out to the config node that holds schema that is
replicated to the providers and consumers. Post schema update, 2/11 servers
crashed in the mdb online indexing function. I fixed this by slapcat the db
and slapadd the db. This is important because it was later revealed that on
the 9/11 servers that did not crash or have their database reloaded, ldapsearch
would return the wrong attribute names for some attribute:value pairs in the
database, which caused mayhem in downstream systems and caused replication
issues between the nodes. The 2 nodes that were reloaded immediately after the
schema change had the only "good" copies of the database left.
To give an example, say an entry was something like:
dn: uid=joe,ou=people,dc=example,dc=com
uid: joe
sn: smith
cn: joe smith
givenName: joe
After the change, the broken servers could return something like:
dn: uid=joe,ou=people,dc=example,dc=com
uid: joe
posixGroup: smith
cn: joe smith
givenName joe
It's not clear how deeply this bug ran in the database. It for sure affected 2
attributes used by the person objectClass. Both of the "replacement"
attributes were not valid attributes for the person objectClasses in use.
Maybe related to the changes in ITS#9858?
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10308
Issue ID: 10308
Summary: Implement cn=monitor for back-asyncmeta
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: backends
Assignee: bugs(a)openldap.org
Reporter: nivanova(a)symas.com
Target Milestone: ---
Currently back-asyncmeta has no cn=monitor capabilities. It will be useful to
implement some, specifically to monitor targets and target connection states.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9909
Issue ID: 9909
Summary: slap* tools leak cn=config entries on shutdown
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: ondra(a)mistotebe.net
Target Milestone: ---
slap* tools set up their in-memory cn=config structures but cfb->cb_root is
never released on shutdown.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10346
Issue ID: 10346
Summary: mdb_env_copy2 on a database with a value larger than
(2GB-16) results in a corrupt copy
Product: LMDB
Version: 0.9.31
Hardware: x86_64
OS: Linux
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: mike.moritz(a)vertex.link
Target Milestone: ---
Created attachment 1072
--> https://bugs.openldap.org/attachment.cgi?id=1072&action=edit
reproduction source code
Running mdb_env_copy2 with compaction on a database with a value larger than
(2GB-16)bytes appears to complete successfully in that there are no errors, but
the copied database cannot be opened and throws an MDB_CORRUPTED error. Looking
at the copied database size, it appears that the value is either being skipped
or significantly truncated. Running mdb_env_copy2 without compaction also
completes successfully, and the copied database can be opened.
I initially encountered this while using py-lmdb with v0.9.31 of LMDB, but was
able to write up a simple script that uses the library directly. The source for
the script is attached, and the results below are from running it with the
latest from master.
Without compaction:
$ ./lmdb_repro test.lmdb $((2 * 1024 * 1024 * 1024 - 16 + 1)) testbak.lmdb
LMDB Version: LMDB 0.9.70: (December 19, 2015)
Set LMDB map size to 21474836330 bytes
Successfully inserted key with 2147483633 bytes of zero-filled data
Retrieved 2147483633 bytes of data
First 16 bytes (hex): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ...
Copying database to testbak.lmdb...
Database copy completed successfully.
Opening copied database and reading value...
Retrieved 2147483633 bytes of data from copied database
First 16 bytes from copy (hex): 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 ...
Data size matches between original and copy
With compaction:
$ ./lmdb_repro -c test.lmdb $((2 * 1024 * 1024 * 1024 - 16 + 1))
testbak.lmdb
LMDB Version: LMDB 0.9.70: (December 19, 2015)
Set LMDB map size to 21474836330 bytes
Successfully inserted key with 2147483633 bytes of zero-filled data
Retrieved 2147483633 bytes of data
First 16 bytes (hex): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ...
Copying database to testbak.lmdb (with compaction)...
Database copy completed successfully.
Opening copied database and reading value...
mdb_get (copy) failed: MDB_CORRUPTED: Located page was wrong type
Size difference on corrupt DB:
$ du -sh ./*
312K ./lmdb_repro
24K ./testbak.lmdb
2.1G ./test.lmdb
With compaction at the perceived max size:
$ ./lmdb_repro -c test.lmdb $((2 * 1024 * 1024 * 1024 - 16)) testbak.lmdb
LMDB Version: LMDB 0.9.70: (December 19, 2015)
Set LMDB map size to 21474836320 bytes
Successfully inserted key with 2147483632 bytes of zero-filled data
Retrieved 2147483632 bytes of data
First 16 bytes (hex): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ...
Copying database to testbak.lmdb (with compaction)...
Database copy completed successfully.
Opening copied database and reading value...
Retrieved 2147483632 bytes of data from copied database
First 16 bytes from copy (hex): 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 ...
Data size matches between original and copy
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10296
Issue ID: 10296
Summary: Force a Mac OS full flush
Product: LMDB
Version: unspecified
Hardware: All
OS: Mac OS
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: renault.cle(a)gmail.com
Target Milestone: ---
Created attachment 1045
--> https://bugs.openldap.org/attachment.cgi?id=1045&action=edit
Use a F_FULLFSYNC fcntl when committing on Mac OS
Hello Howard and Happy New Year,
As discussed in this issue [1], the LMDB durability is incorrect when
committing. I propose the following patch that uses fcntl with the F_FULLFSYNC
flag. The fcntl documentation is on this page [2].
Note that I kept the calls to msync/fsync for simplicity and because they don't
cost much but feel free to skip them on Mac OS.
Have a nice day,
kero
[1]: https://github.com/cberner/redb/pull/928#issuecomment-2567032808
[2]:
https://developer.apple.com/library/archive/documentation/System/Conceptual…
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10402
Issue ID: 10402
Summary: Feature request: parameter for mdb_env_copy to exclude
databases
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: jeffro256(a)tutanota.com
Target Milestone: ---
## Desired behavior
New overload or API modification to `mdb_env_copy{...}()` which allows passing
a list of names of databases to exclude from the environment copy.
## Example use-case
The Monero blockchain database [1] has the option to derived a "pruned"
database for space-saving purposes. This removes some ~60% of data which some
users may find non-essential. Currently, the pruning code [2] copies each
non-pruned table manually, specifying key ordering functions, DB flags, etc. It
then drops some entries from the relevant "prunable tables". This, however,
adds technical maintenance debt when databases are added/updated. A preferable
alternative would be to write high-level modification code once and use an
overload of `mdb_env_copy` which excludes copying database that we know in
advance we don't want to copy. Then our pruning utility would work agnostic to
database changes.
## Why
Adding this functionality ourselves would involve re-writing large portions of
`mdb_env_copyfd{0,1}` and `mdb_env_cwalk`, which requires either A) vendoring
LMDB, or B) possibly breaking in future updates.
You may understandably be of the opinion that the maintenance burden is an "us
problem", and not deem pursuing this feature request worth it, but hopefully
you seem the value in this utility. Thanks for y'alls hard work on LMDB.
## Links
[1]
https://github.com/monero-project/monero/blob/d32b5bfe18e2f5b979fa8dc3a8966…
[2]
https://github.com/monero-project/monero/blob/master/src/blockchain_utiliti…
--
You are receiving this mail because:
You are on the CC list for the issue.