https://bugs.openldap.org/show_bug.cgi?id=9360
Issue ID: 9360
Summary: MDB_BAD_TXN: Transaction must abort, has a child, or
is invalid
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: spam(a)markandruth.co.uk
Target Milestone: ---
I have 2 python scripts writing to a database (lmdb 0.9.26, py-lmdb 0.98) and
5-10 lua processes (with lightningmdb module which uses lmdb 0.9.22) which are
long-running serving queries from the database.
The database seems fine, not corrupted, and the python writes still working all
the time. But periodically (perhaps 10-20% of the time), in a way I am unable
to reliably reproduce, when the lua starts up every time a query is issued txn
dbi_open returns "MDB_BAD_TXN: Transaction must abort, has a child, or is
invalid". A direct restart of the processes does not fix this issue, however
stopping lua+python and then starting again after a 5-20s wait usually fixes
the issue. This has been reproduced over multiple servers but I'm at a loss as
to how to debug this any further?
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9208
Bug ID: 9208
Summary: LMDB feature request: variant of mdb_env_copy{,fd2}
that takes transaction as parameter
Product: LMDB
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: github(a)nicwatson.org
Target Milestone: ---
The mdb_env_copy* functions create a read transaction themselves to run the
backup on. New variants of these functions (one for mdb_env_copy2 and one for
mdb_env_copyfd2) would have a transaction parameter. This transaction would be
used instead of creating a new transaction.
Application code could use these new functions to synchronize consistent live
backups across multiple LMDB instances (potentially across multiple hosts).
--
You are receiving this mail because:
You are on the CC list for the bug.
https://bugs.openldap.org/show_bug.cgi?id=10034
Issue ID: 10034
Summary: Assertion 'i < NUMKEYS(mp)' failed in
mdb_page_search_root()"
Product: LMDB
Version: 0.9.23
Hardware: Other
OS: Linux
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: 763280032(a)qq.com
Target Milestone: ---
We found that when lmdb is opened after OS startup and data is written to it,
lmdb will trigger abort probabilistically(Restart the OS 600 times will trigger
once);
We want to know what situation triggers this issue(Assertion 'i < NUMKEYS(mp)'
failed in mdb_page_search_root()); we want to know if there is a problem with
our usage;
Please Help Us
(gdb) x/8s 0x8baee988
0x8baee988: "8\373Բ\b\371Բmdb.c:5542: Assertion 'i < NUMKEYS(mp)' failed in
mdb_page_search_root()"
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9475
Issue ID: 9475
Summary: Add support for MAP_POPULATE
Product: LMDB
Version: unspecified
Hardware: All
OS: Linux
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: aa531811820(a)gmail.com
Target Milestone: ---
In some case (such as cloud computing platforms), the reading speed of large
files is very fast while the small files is very slow, and we have enough
memory, so we hope to prefetch the entire LMDB file into the memory during MMAP
through the MAP_POPULATE flag . According to our test, this is faster than
using readahead flag. Here are some test data:
# mmap with no readahead
read one sample: 0.2s
total time: 4800s
# mmap with readahead
read one sample: 0.0001s~0.03s
total time: 95.86s
# mmap with MAP_POPULATE
db init: 20s
read one sample: 0.0001s
total time: 78s
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9397
Issue ID: 9397
Summary: LMDB: A second process opening a file with
MDB_WRITEMAP can cause the first to SIGBUS
Product: LMDB
Version: 0.9.26
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: github(a)nicwatson.org
Target Milestone: ---
Created attachment 780
--> https://bugs.openldap.org/attachment.cgi?id=780&action=edit
Full reproduction of SIGBUS MDB_WRITEMAP issue (works on Linux only)
The fundamental problem is that a ftruncate() on Linux that makes a file
smaller will cause accesses past the new end of the file to SIGBUS (see the
mmap man page).
The sequence that causes a SIGBUS involves two processes.
1. The first process opens a new LMDB file with MDB_WRITEMAP.
2. The second process opens the same LMDB file with MDB_WRITEMAP and with an
explicit map_size smaller than the first process's map size.
* This causes an ftruncate that makes the underlying file *smaller*.
3. (Optional) The second process closes the environment and exits.
4. The first process opens a write transaction and writes a bunch of data.
5. The first process commits the transaction. This causes a memory read from
the mapped memory that's now past the end of the file. On Linux, this triggers
a SIGBUS.
Attached is code that fully reproduces the problem on Linux.
The most straightforward solution is to allow ftruncate to *reduce* the file
size if it is the only reader. Another possibility is check the file size and
ftruncate if necessary every time a write transaction is opened. A third
possibility is to catch the SIGBUS signal.
Repro note: I used clone() to create the subprocess to most straightforwardly
demonstrate that the problem is not due to inherited file descriptors. The
problem still manifests when the processes are completely independent.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9207
Bug ID: 9207
Summary: Remove Moznss compatibility layer
Product: OpenLDAP
Version: 2.5
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: ---
Component: libraries
Assignee: bugs(a)openldap.org
Reporter: quanah(a)openldap.org
Target Milestone: ---
For the 2.5 release, remove the MozNSS compatibility layer.
--
You are receiving this mail because:
You are on the CC list for the bug.
https://bugs.openldap.org/show_bug.cgi?id=10229
Issue ID: 10229
Summary: ldap_result, when invoked with MSG_RECEIVED and a
timeout value set to 0 (polling), does not return all
available messages until it is called again
Product: OpenLDAP
Version: 2.6.8
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: libraries
Assignee: bugs(a)openldap.org
Reporter: nivanova(a)symas.com
Target Milestone: ---
The issue is noticeable when ldap_result is used by the proxy back-ends. It has
not affected back-meta behavior, because when a first call is unsuccessful, it
retries with a small timeout. back-asyncmeta will also usually call it twice on
the same connection from different threads, although this is not a desired
behavior.
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=9888
Issue ID: 9888
Summary: When using cn=config replication, schema updates can
corrupt the index database(s)
Product: OpenLDAP
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: slapd
Assignee: bugs(a)openldap.org
Reporter: quanah(a)openldap.org
Target Milestone: ---
Today I pushed a schema update out to the config node that holds schema that is
replicated to the providers and consumers. Post schema update, 2/11 servers
crashed in the mdb online indexing function. I fixed this by slapcat the db
and slapadd the db. This is important because it was later revealed that on
the 9/11 servers that did not crash or have their database reloaded, ldapsearch
would return the wrong attribute names for some attribute:value pairs in the
database, which caused mayhem in downstream systems and caused replication
issues between the nodes. The 2 nodes that were reloaded immediately after the
schema change had the only "good" copies of the database left.
To give an example, say an entry was something like:
dn: uid=joe,ou=people,dc=example,dc=com
uid: joe
sn: smith
cn: joe smith
givenName: joe
After the change, the broken servers could return something like:
dn: uid=joe,ou=people,dc=example,dc=com
uid: joe
posixGroup: smith
cn: joe smith
givenName joe
It's not clear how deeply this bug ran in the database. It for sure affected 2
attributes used by the person objectClass. Both of the "replacement"
attributes were not valid attributes for the person objectClasses in use.
Maybe related to the changes in ITS#9858?
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10346
Issue ID: 10346
Summary: mdb_env_copy2 on a database with a value larger than
(2GB-16) results in a corrupt copy
Product: LMDB
Version: 0.9.31
Hardware: x86_64
OS: Linux
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: mike.moritz(a)vertex.link
Target Milestone: ---
Created attachment 1072
--> https://bugs.openldap.org/attachment.cgi?id=1072&action=edit
reproduction source code
Running mdb_env_copy2 with compaction on a database with a value larger than
(2GB-16)bytes appears to complete successfully in that there are no errors, but
the copied database cannot be opened and throws an MDB_CORRUPTED error. Looking
at the copied database size, it appears that the value is either being skipped
or significantly truncated. Running mdb_env_copy2 without compaction also
completes successfully, and the copied database can be opened.
I initially encountered this while using py-lmdb with v0.9.31 of LMDB, but was
able to write up a simple script that uses the library directly. The source for
the script is attached, and the results below are from running it with the
latest from master.
Without compaction:
$ ./lmdb_repro test.lmdb $((2 * 1024 * 1024 * 1024 - 16 + 1)) testbak.lmdb
LMDB Version: LMDB 0.9.70: (December 19, 2015)
Set LMDB map size to 21474836330 bytes
Successfully inserted key with 2147483633 bytes of zero-filled data
Retrieved 2147483633 bytes of data
First 16 bytes (hex): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ...
Copying database to testbak.lmdb...
Database copy completed successfully.
Opening copied database and reading value...
Retrieved 2147483633 bytes of data from copied database
First 16 bytes from copy (hex): 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 ...
Data size matches between original and copy
With compaction:
$ ./lmdb_repro -c test.lmdb $((2 * 1024 * 1024 * 1024 - 16 + 1))
testbak.lmdb
LMDB Version: LMDB 0.9.70: (December 19, 2015)
Set LMDB map size to 21474836330 bytes
Successfully inserted key with 2147483633 bytes of zero-filled data
Retrieved 2147483633 bytes of data
First 16 bytes (hex): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ...
Copying database to testbak.lmdb (with compaction)...
Database copy completed successfully.
Opening copied database and reading value...
mdb_get (copy) failed: MDB_CORRUPTED: Located page was wrong type
Size difference on corrupt DB:
$ du -sh ./*
312K ./lmdb_repro
24K ./testbak.lmdb
2.1G ./test.lmdb
With compaction at the perceived max size:
$ ./lmdb_repro -c test.lmdb $((2 * 1024 * 1024 * 1024 - 16)) testbak.lmdb
LMDB Version: LMDB 0.9.70: (December 19, 2015)
Set LMDB map size to 21474836320 bytes
Successfully inserted key with 2147483632 bytes of zero-filled data
Retrieved 2147483632 bytes of data
First 16 bytes (hex): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ...
Copying database to testbak.lmdb (with compaction)...
Database copy completed successfully.
Opening copied database and reading value...
Retrieved 2147483632 bytes of data from copied database
First 16 bytes from copy (hex): 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 ...
Data size matches between original and copy
--
You are receiving this mail because:
You are on the CC list for the issue.
https://bugs.openldap.org/show_bug.cgi?id=10296
Issue ID: 10296
Summary: Force a Mac OS full flush
Product: LMDB
Version: unspecified
Hardware: All
OS: Mac OS
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: bugs(a)openldap.org
Reporter: renault.cle(a)gmail.com
Target Milestone: ---
Created attachment 1045
--> https://bugs.openldap.org/attachment.cgi?id=1045&action=edit
Use a F_FULLFSYNC fcntl when committing on Mac OS
Hello Howard and Happy New Year,
As discussed in this issue [1], the LMDB durability is incorrect when
committing. I propose the following patch that uses fcntl with the F_FULLFSYNC
flag. The fcntl documentation is on this page [2].
Note that I kept the calls to msync/fsync for simplicity and because they don't
cost much but feel free to skip them on Mac OS.
Have a nice day,
kero
[1]: https://github.com/cberner/redb/pull/928#issuecomment-2567032808
[2]:
https://developer.apple.com/library/archive/documentation/System/Conceptual…
--
You are receiving this mail because:
You are on the CC list for the issue.