https://bugs.openldap.org/show_bug.cgi?id=9223
Bug ID: 9223
Summary: Add support for incremental backup
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: quanah(a)openldap.org
Target Milestone: ---
For LMDB 1.0, add support for incremental backups
--
You are receiving this mail because:
You are on the CC list for the bug.
https://bugs.openldap.org/show_bug.cgi?id=9496
Issue ID: 9496
Summary: Some writes missing from database
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: igfoo(a)github.com
Target Milestone: ---
With the attached test program, some of my database writes appear not to
actually be written to the database. For example, a run may look like this:
$ ./run.sh
All done.
All finished
1802 test.txt
foo_200 is missing
bar_200 is missing
foo_404 is missing
bar_404 is missing
foo_407 is missing
bar_407 is missing
The script that I am using to run the program is below. This is using
mdb.master 52bc29ee2efccf09c650598635cd42a50b6ecffe on Linux, with an ext4
filesystem.
Is this an LMDB bug, or is there a bug in my code?
Thanks
Ian
#!/bin/sh
set -e
if ! [ -d lmdb ]
then
rm -rf lmdb
git clone https://github.com/LMDB/lmdb.git
INSTALL_DIR="`pwd`/inst"
cd lmdb/libraries/liblmdb
make install prefix="$INSTALL_DIR"
cd ../../..
fi
gcc -Wall -Werror -Iinst/include loop.c inst/lib/liblmdb.a -o loop -pthread
rm -f test.db test.db-lock
./loop
echo "All finished"
mdb_dump -np test.db > test.txt
wc -l test.txt
for i in `seq 100 999`
do
if ! grep -q "foo_$i" test.txt
then
echo "foo_$i is missing"
fi
if ! grep -q "bar_$i" test.txt
then
echo "bar_$i is missing"
fi
done
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9619
Issue ID: 9619
Summary: mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3
produces corrupt mdb file
Product: LMDB
Version: 0.9.29
Hardware: All
OS: Windows
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: kriszyp(a)gmail.com
Target Milestone: ---
When copying an LMDB database with mdb_env_copy2 with the MDB_CP_COMPACT with
mdb.master3, the resulting mdb file seems to be corrupt and when using it in
LMDB, I get segmentation faults. Copying without the compacting flag seems to
work fine. I apologize, I know this is not a very good issue report, as I
haven't had a chance to actually narrow this down to a more
reproducible/isolated case, or look for how to patch. I thought I would report
in case there are any ideas on what could cause this. The segmentation faults
always seem to be memory write faults (as opposed to try fault on trying to
read). Or perhaps the current backup/copying functionality is eventually going
to be replaced by incremental backup/copying anyway
(https://twitter.com/hyc_symas/status/1315651814096875520). I'll try to update
this if I get a chance to investigate more, but otherwise feel free to
ignore/consider low-priority since the work around is easy.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10395
Issue ID: 10395
Summary: Support multiple readers on uncommitted changes
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: renault.cle(a)gmail.com
Target Milestone: ---
Created attachment 1086
--> https://bugs.openldap.org/attachment.cgi?id=1086&action=edit
A patch to support multiple readers on uncommitted changes
Hello,
The attached patch is not meant to be merged immediately into LMDB. Still, it
demonstrates how I added a helpful feature to the key-value store: reading
uncommitted changes from multiple transactions. I am conscious that the patch
still requires some work and uses non-C99 features, i.e., atomics are C11,
which could be a blocker for it to be merged upstream. I would also be
delighted to merge these changes under a flag/define to ensure we don't impact
other users with non-C99 stuff.
The main feature we need at Meilisearch is to read uncommitted changes from
multiple threads, compute parallel post-processing of different data structures
[1], and speed up the search requests. We could have done the post-processing
in a following transaction by opening multiple read transactions, but this
would mean that the post-processed data structure would not include newly
inserted or modified document IDs. Both data structures would be desync.
Regarding the design choice, I decided to follow the same design as the nested
write transactions: use the parent argument of the mdb_txn_begin [2], and allow
the MDB_RDONLY flag, which was disallowed when the parent argument was non-NULL
[3]. I find it clear enough that, by calling the mdb_txn_begin function with
these arguments, you can call it multiple times (I need to update the doc) to
obtain nested read-only transactions from the parent write transaction.
ret = mdb_txn_begin(env, parent_txn, MDB_RDONLY, new_nested_rtxn);
Note that this early proposal lacks security and error handling. The generated
transactions are fake-read-only and actually write transactions that share the
underlying parent allocations and data structures. This is unsafe and must be
changed or reviewed carefully, but most importantly, we need to add read-only
capabilities to these transactions to disallow writes. Using a Rust wrapper on
top of LMDB, I wrapped the fake read-only transactions into ReadTxn, which
disallows any writes at compile time. However, I haven't checked the conflict
database creations or openings.
The main issues I encountered were concurrent free of the main shared data
structures when the different threads owning the transactions were dropping the
transactions simultaneously. So, I decided to implement the equivalent of an
ARC to free resources only when the last nested transaction was freed.
I can share numbers about how this feature improves the post-processing by
4x-9x or from 1200s to 120s [1]. You can look at this PR, which I would be
happy to merge once an improved version of this patch lands on LMDB upstream.
I would be very happy if you could guide me a bit on how I could improve this
patch to make it mergeable into LMDB. We want to contribute useful features
like this to LMDB and not keep a deviant fork. LMDB works great; we are happy
about it, and its performance is predictable.
Have a lovely week,
kero
[1]: https://github.com/meilisearch/meilisearch/pull/5307
[2]:
https://github.com/LMDB/lmdb/blob/14d6629bc8a9fe40d8a6bee1bf71c45afe7576b6/…
[3]:
https://github.com/LMDB/lmdb/blob/14d6629bc8a9fe40d8a6bee1bf71c45afe7576b6/…
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10428
Issue ID: 10428
Summary: Make larger key sizes available at compile time
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: marlowsd(a)gmail.com
Target Milestone: ---
I discovered that I can increase the key size to 1982 bytes with the following
patch
```
***************
*** 1053,1059 ****
#define METADATA(p) ((void *)((char *)(p) + PAGEHDRSZ))
/** ITS#7713, change PAGEBASE to handle 65536 byte pages */
! #define PAGEBASE ((MDB_DEVEL) ? PAGEHDRSZ : 0)
/** Number of nodes on a page */
#define NUMKEYS(p) ((MP_LOWER(p) - (PAGEHDRSZ-PAGEBASE)) >> 1)
--- 1054,1061 ----
#define METADATA(p) ((void *)((char *)(p) + PAGEHDRSZ))
/** ITS#7713, change PAGEBASE to handle 65536 byte pages */
! #define PAGEBASE PAGEHDRSZ
! // #define PAGEBASE ((MDB_DEVEL) ? PAGEHDRSZ : 0)
/** Number of nodes on a page */
#define NUMKEYS(p) ((MP_LOWER(p) - (PAGEHDRSZ-PAGEBASE)) >> 1)
```
and then passing `-DMDB_MAXKEYSIZE=0` at compile time. I didn't want to use
`MDB_DEVEL` because this is for a production use case.
Is this feature ready to be widely used? If so, could it be given a
compile-time (or run-time) flag please?
Background: this is for Glean
(https://github.com/facebookincubator/Glean/pull/663), this page explains our
DB structure https://glean.software/docs/implementation/db/.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10024
Issue ID: 10024
Summary: MDB_PREVSNAPSHOT broken
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: markus(a)objectbox.io
Target Milestone: ---
It seems that the patch #9496 had a negative side effect on MDB_PREVSNAPSHOT.
In certain cases, when opening the DB using MDB_PREVSNAPSHOT, the previous (2nd
latest) commit is not selected. Instead, reads show that the latest commit was
selected voiding the effect of MDB_PREVSNAPSHOT.
I observed this in our test cases a while back. Today, I was finally able to
reproduce it and debug into it.
When creating the transaction to read the data, I debugged into mdb_txn_renew0.
Here, ti (MDB_txninfo; env->me_txns) was non-NULL. However, ti->mti_txnid was 0
(!) and thus txn->mt_txnid was set to 0. That's the reason for always selecting
the first (index 0) meta page inside mdb_txn_renew0:
meta = env->me_metas[txn->mt_txnid & 1];
This line occurs twice (once for read txn and once for write txn; it affects
both txn types).
Thus, the chances of MDB_PREVSNAPSHOT selecting the correct meta page is 50-50.
It's only correct if the first meta page (index 0) is the older one.
I believe that this is related to #9496 because the patch, that was provided
there, removed the initialization of "env->me_txns->mti_txnid" in
mdb_env_open2. This would explain why txn->mt_txnid inside mdb_txn_renew0 was
set to 0.
I can confirm that adding back the following two lines back in fixes
MDB_PREVSNAPSHOT:
if (env->me_txns)
env->me_txns->mti_txnid = meta.mm_txnid;
The said patch including the removal of these two lines was applied in the
commit(s) "ITS#9496 fix mdb_env_open bug from #8704" (Howard Chu on 09.04.21).
I hope this information is useful to find a suitable fix. Please let me know if
you have questions. Also, I'd be happy to help confirming a potential fix with
our test suite.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9224
Bug ID: 9224
Summary: Add support for PREPARE/2-phase commit
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: quanah(a)openldap.org
Target Milestone: ---
In LMDB, add support for PREPARE/2-phase commits
--
You are receiving this mail because:
You are on the CC list for the bug.
https://bugs.openldap.org/show_bug.cgi?id=9769
Issue ID: 9769
Summary: Patch new feature batch get
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: rouzier(a)gmail.com
Target Milestone: ---
Created attachment 859
--> https://bugs.openldap.org/attachment.cgi?id=859&action=edit
New functionality mdb_cursor_get_batch
New functionality mdb_cursor_get_batch
mdb_cursor_get_batch retrieves a page worth of key/values.
This is to reduce the number of function calls when doing a scan of the
database.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10203
Issue ID: 10203
Summary: no pkgconfig file included for liblmdb
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: otto(a)drijf.net
Target Milestone: ---
liblmdb does not ship with a pkgconfig file. More and more build systems rely
on presense of a pkgconfig file, so it would be nice if liblmdb installed
oneone. An example:
prefix=/usr/local
exec_prefix=${prefix}
libdir=${prefix}/lib
includedir=${prefix}/include
Name: lmdb
Description: Lightning memory-mapped database: key-value data store
URL: https://www.symas.com/symas-embedded-database-lmdb
Version: 0.9.32
Libs: -L${libdir} -llmdb
Cflags: -I${includedir}
Thanks.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10402
Issue ID: 10402
Summary: Feature request: parameter for mdb_env_copy to exclude
databases
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: jeffro256(a)tutanota.com
Target Milestone: ---
## Desired behavior
New overload or API modification to `mdb_env_copy{...}()` which allows passing
a list of names of databases to exclude from the environment copy.
## Example use-case
The Monero blockchain database [1] has the option to derived a "pruned"
database for space-saving purposes. This removes some ~60% of data which some
users may find non-essential. Currently, the pruning code [2] copies each
non-pruned table manually, specifying key ordering functions, DB flags, etc. It
then drops some entries from the relevant "prunable tables". This, however,
adds technical maintenance debt when databases are added/updated. A preferable
alternative would be to write high-level modification code once and use an
overload of `mdb_env_copy` which excludes copying database that we know in
advance we don't want to copy. Then our pruning utility would work agnostic to
database changes.
## Why
Adding this functionality ourselves would involve re-writing large portions of
`mdb_env_copyfd{0,1}` and `mdb_env_cwalk`, which requires either A) vendoring
LMDB, or B) possibly breaking in future updates.
You may understandably be of the opinion that the maintenance burden is an "us
problem", and not deem pursuing this feature request worth it, but hopefully
you seem the value in this utility. Thanks for y'alls hard work on LMDB.
## Links
[1]
https://github.com/monero-project/monero/blob/d32b5bfe18e2f5b979fa8dc3a8966…
[2]
https://github.com/monero-project/monero/blob/master/src/blockchain_utiliti…
--
You are receiving this mail because:
You are on the CC list for the issue.