On 27.10.2020 18:52 Howard Chu wrote:
- LMDB causes crashes if database is corrupted
You can enable per-page checksums in LMDB 1.0, in which case you'll just get an error code if a page is corrupted (and the checksum fails to match). The DB will still be unusable if anything is corrupted.
That would fix the problem properly. Does it check that it is the correct transaction as well (e.g. by putting a transid into the page like btrfs)? Returning wrong results or MDB_CORRUPTED is something my application can handle (but not crashes obviously).
The txnid is part of the page header, which is one of the incompatible format changes from LMDB 0.9. This is what allows us to eliminate the dirty bit.
Not sure what you mean about the txnid being correct or not, but certainly it is included in the checksum.
A common problem that e.g. btrfs users encounter is that a disk drops some writes. If there was a page at the same location previously the checksum check succeeds. But btrfs stores the transid of the page in the page's parent, so it compares that as well (The error message is "btrfs parent transid verify failed on OFFSET wanted TRANSID found TRANSID"). I think ZFS stores the checksum of the page in the page's parent as well (idk if this would work with lmdbs b-tree).
I guess a simple check (that might already exist) is checking if page transid<=root/meta page transid. But that doesn't catch the cases where the root page was updated, but updates to other pages were dropped (the disk might also drop a complete "transaction" but not report an error, in which case the next transaction then writes the root page pointing to an incompletely written tree).