https://bugs.openldap.org/show_bug.cgi?id=9619
Issue ID: 9619
Summary: mdb_env_copy2 with MDB_CP_COMPACT in mdb.master3
produces corrupt mdb file
Product: LMDB
Version: 0.9.29
Hardware: All
OS: Windows
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: kriszyp(a)gmail.com
Target Milestone: ---
When copying an LMDB database with mdb_env_copy2 with the MDB_CP_COMPACT with
mdb.master3, the resulting mdb file seems to be corrupt and when using it in
LMDB, I get segmentation faults. Copying without the compacting flag seems to
work fine. I apologize, I know this is not a very good issue report, as I
haven't had a chance to actually narrow this down to a more
reproducible/isolated case, or look for how to patch. I thought I would report
in case there are any ideas on what could cause this. The segmentation faults
always seem to be memory write faults (as opposed to try fault on trying to
read). Or perhaps the current backup/copying functionality is eventually going
to be replaced by incremental backup/copying anyway
(https://twitter.com/hyc_symas/status/1315651814096875520). I'll try to update
this if I get a chance to investigate more, but otherwise feel free to
ignore/consider low-priority since the work around is easy.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9769
Issue ID: 9769
Summary: Patch new feature batch get
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: rouzier(a)gmail.com
Target Milestone: ---
Created attachment 859
--> https://bugs.openldap.org/attachment.cgi?id=859&action=edit
New functionality mdb_cursor_get_batch
New functionality mdb_cursor_get_batch
mdb_cursor_get_batch retrieves a page worth of key/values.
This is to reduce the number of function calls when doing a scan of the
database.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9291
Issue ID: 9291
Summary: Detection of corrupted database files
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: markus(a)objectbox.io
Target Milestone: ---
Let's assume we have to deal with a corrupted database for whatever reason
(e.g. broken hardware or file system). Current behavior seems to be mostly
undefined, which is understandable as it's not known what is broken (e.g. there
are no checksums).
For example, I'm seeing a SIGBUS in mdb_page_touch because the cursor's top
page (mp) is pointing to invalid memory (0x7f99cf004000) during a commit:
mdb_page_touch mdb.c:2772
mdb_page_search mdb.c:6595
mdb_freelist_save mdb.c:3575
mdb_txn_commit mdb.c:4060
Cursor data at that point: mc_snum = 1, mc_top = 0; myki[0] = 0
A SIGBUS is troublesome as it crashes the process, and I wonder if there are
other ways to detect such inconsistencies. If that be possible there could be
user-specific handling in place. E.g. a user might start a new database file.
This issue was reported by our users, which also provided DB files:
https://github.com/objectbox/objectbox-java/issues/859
I did not find a lot of consistency checks besides MDB_PAGE_NOTFOUND and
MDB_CORRUPTED. Also, I think there's no current way to thoroughly check a DB
file (e.g. like fsck for the DB file)?
My first idea other than checksums was to walk through the branch pages from
the root and check if the referenced pages are within reasonable bounds. Also
check the page content (e.g. nodes, flags). Additionally (optionally?), it
should be possible to check that the key values are actually sorted.
So, it boils down to 3 points in summary:
1.) If there no way to check the DB file for consistency yet(?), which approach
do you think would make sense? There might be two modes; one for a through
check through all data, and a quick check that does not take long and could be
e.g. done when opening the DB. Goal is to avoid process crashes and let users
handle the situation.
2.) In general, is it possible to add more consistency checks in regular DB
operations?
3.) Could the the particular situation (for which I provided the stack trace)
detected (e.g. is myki[0] = 0 legal here?)
I'd be happy to provide a patch if you provide some direction where you want to
take that.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10203
Issue ID: 10203
Summary: no pkgconfig file included for liblmdb
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: otto(a)drijf.net
Target Milestone: ---
liblmdb does not ship with a pkgconfig file. More and more build systems rely
on presense of a pkgconfig file, so it would be nice if liblmdb installed
oneone. An example:
prefix=/usr/local
exec_prefix=${prefix}
libdir=${prefix}/lib
includedir=${prefix}/include
Name: lmdb
Description: Lightning memory-mapped database: key-value data store
URL: https://www.symas.com/symas-embedded-database-lmdb
Version: 0.9.32
Libs: -L${libdir} -llmdb
Cflags: -I${includedir}
Thanks.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9388
Issue ID: 9388
Summary: mdb_stat for DupSort DBI shows incorrect data
Product: LMDB
Version: 0.9.26
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: AskAlexSharov(a)gmail.com
Target Milestone: ---
It doesn't include pages pages used for values.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10454
Issue ID: 10454
Summary: O_DSYNC is busted on macos
Product: LMDB
Version: unspecified
Hardware: All
OS: Mac OS
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: pyry.kovanen(a)gmail.com
Target Milestone: ---
LMDB relies on O_DSYNC for writing the meta page, unfortunately it doesn't work
on macos. Previous discovery by the tigerbeetle guys:
https://github.com/tigerbeetle/viewstamped-replication-made-famous#leaderbo…,
some more context at https://x.com/jorandirkgreef/status/1532314169604726784.
I discovered this during benchmarking and was wondering why lmdb writes were
twice as fast as macos as on linux.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9360
Issue ID: 9360
Summary: MDB_BAD_TXN: Transaction must abort, has a child, or
is invalid
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: spam(a)markandruth.co.uk
Target Milestone: ---
I have 2 python scripts writing to a database (lmdb 0.9.26, py-lmdb 0.98) and
5-10 lua processes (with lightningmdb module which uses lmdb 0.9.22) which are
long-running serving queries from the database.
The database seems fine, not corrupted, and the python writes still working all
the time. But periodically (perhaps 10-20% of the time), in a way I am unable
to reliably reproduce, when the lua starts up every time a query is issued txn
dbi_open returns "MDB_BAD_TXN: Transaction must abort, has a child, or is
invalid". A direct restart of the processes does not fix this issue, however
stopping lua+python and then starting again after a 5-20s wait usually fixes
the issue. This has been reproduced over multiple servers but I'm at a loss as
to how to debug this any further?
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9208
Bug ID: 9208
Summary: LMDB feature request: variant of mdb_env_copy{,fd2}
that takes transaction as parameter
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: github(a)nicwatson.org
Target Milestone: ---
The mdb_env_copy* functions create a read transaction themselves to run the
backup on. New variants of these functions (one for mdb_env_copy2 and one for
mdb_env_copyfd2) would have a transaction parameter. This transaction would be
used instead of creating a new transaction.
Application code could use these new functions to synchronize consistent live
backups across multiple LMDB instances (potentially across multiple hosts).
--
You are receiving this mail because:
You are on the CC list for the bug.
https://bugs.openldap.org/show_bug.cgi?id=10034
Issue ID: 10034
Summary: Assertion 'i < NUMKEYS(mp)' failed in
mdb_page_search_root()"
Product: LMDB
Version: 0.9.23
Hardware: Other
OS: Linux
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: 763280032(a)qq.com
Target Milestone: ---
We found that when lmdb is opened after OS startup and data is written to it,
lmdb will trigger abort probabilistically(Restart the OS 600 times will trigger
once);
We want to know what situation triggers this issue(Assertion 'i < NUMKEYS(mp)'
failed in mdb_page_search_root()); we want to know if there is a problem with
our usage;
Please Help Us
(gdb) x/8s 0x8baee988
0x8baee988: "8\373Բ\b\371Բmdb.c:5542: Assertion 'i < NUMKEYS(mp)' failed in
mdb_page_search_root()"
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9475
Issue ID: 9475
Summary: Add support for MAP_POPULATE
Product: LMDB
Version: unspecified
Hardware: All
OS: Linux
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: aa531811820(a)gmail.com
Target Milestone: ---
In some case (such as cloud computing platforms), the reading speed of large
files is very fast while the small files is very slow, and we have enough
memory, so we hope to prefetch the entire LMDB file into the memory during MMAP
through the MAP_POPULATE flag . According to our test, this is faster than
using readahead flag. Here are some test data:
# mmap with no readahead
read one sample: 0.2s
total time: 4800s
# mmap with readahead
read one sample: 0.0001s~0.03s
total time: 95.86s
# mmap with MAP_POPULATE
db init: 20s
read one sample: 0.0001s
total time: 78s
--
You are receiving this mail because:
You are on the CC list for the issue.