https://bugs.openldap.org/show_bug.cgi?id=10552
Issue ID: 10552 Summary: LMDB database getting corrupted Product: LMDB Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Keywords: needs_review Severity: normal Priority: --- Component: liblmdb Assignee: bugs@openldap.org Reporter: donxenapo@gmail.com Target Milestone: ---
We have deployed LMDB version 1.0.0 and we're getting corruption-type errors from some users. We don't have the exact number of affected users yet, but apparently deleting the database and recreating it seems to work around the issue. The error seems to happen either during a get or a put request, and once the database is corrupted, it will keep resulting in the same error for that user until the database is purged/recreated. Here are some examples of errors that we're getting:
During Put: LMDB Put Failed Error: MDB_PROBLEM: Unexpected problem - txn should abort LMDB Put Failed Error: MDB_CORRUPTED: Located page was wrong type LMDB Put Failed Error: MDB_PAGE_NOTFOUND: Requested page not found
During Get: LMDB Get Failed Error: MDB_PAGE_NOTFOUND: Requested page not found LMDB Get Failed Error: MDB_BAD_TXN: Transaction must abort, has a child, or is invalid
Sometimes it crashes during Put in memcpy, instead of returning an error. Call stack: memcpy(void) mdb_cursor_get mdb_txn_begin
We're still in the process of collecting corrupted databases from our users, so we don't have any corruption examples yet.
https://bugs.openldap.org/show_bug.cgi?id=10552
donxenapo@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- OS|All |Windows Hardware|All |x86_64
https://bugs.openldap.org/show_bug.cgi?id=10552
--- Comment #1 from Howard Chu hyc@openldap.org --- Will need some more information on what was being done just prior to the corruption. Does this happen during normal running, or after a power failure or OS crash?
Certainly, once corrupt data gets stored on disk it will remain corrupt from that point on, nothing will change after that.
https://bugs.openldap.org/show_bug.cgi?id=10552
--- Comment #2 from donxenapo@gmail.com --- We have no way of knowing what triggers it (maybe you can advice where to look?), we just see that this happens on a significant percent of our users. It seems to happen immediately after they have created the database: it either gets corrupted almost immediately after just a couple Put requests, or it never gets corrupted.
If it does get corrupted, purging and recreating it often fixes the issue.
We presume that the issue might have something to do with the fact that we're not using the exact 1.0.0 version, instead our lmdb.h version field says 0.9.90, which probably means we just got the master branch from github at some point? We're thinking to either downgrade to the official 0.9.35 tag or to upgrade to 1.0.0.
https://bugs.openldap.org/show_bug.cgi?id=10552
--- Comment #3 from donxenapo@gmail.com --- There was definitely no OS crashing and no power failures involved in any of the user reports, but there is a possibility that our process crashed during the transaction. This is the only flag we pass into LMDB that seems possibly relevant:
mdb_env_open(env.get(), location.c_str(), MDB_NOMETASYNC, 0644)
https://bugs.openldap.org/show_bug.cgi?id=10552
--- Comment #4 from Howard Chu hyc@openldap.org --- (In reply to donxenapo from comment #2)
We have no way of knowing what triggers it (maybe you can advice where to look?), we just see that this happens on a significant percent of our users. It seems to happen immediately after they have created the database: it either gets corrupted almost immediately after just a couple Put requests, or it never gets corrupted.
If it does get corrupted, purging and recreating it often fixes the issue.
We presume that the issue might have something to do with the fact that we're not using the exact 1.0.0 version, instead our lmdb.h version field says 0.9.90, which probably means we just got the master branch from github at some point? We're thinking to either downgrade to the official 0.9.35 tag or to upgrade to 1.0.0.
Yeah, that was your biggest mistake. "0.9.90" wasn't an actual version, it was just a placeholder in the development branch. It is definitely not production-quality code.
At this point we're about to release v0.9.36 and v1.0.1. You could use either the mdb.RE/0.9 or mdb.RE/1.0 branches instead of whatever you grabbed before. There's one relevant bugfix for Windows (ITS#10538) that you might want, otherwise 0.9.35 would be good enough for you.
https://bugs.openldap.org/show_bug.cgi?id=10552
Howard Chu hyc@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |INVALID Status|UNCONFIRMED |RESOLVED
https://bugs.openldap.org/show_bug.cgi?id=10552
--- Comment #5 from donxenapo@gmail.com --- Okay, then I guess it's important to note that we're using LMDB in a potentially non-conventional way where we set the _maximum_ database size to a very large value (we use 100GB) for a database that we expect to be ~1-5GB on average for most users. So we rely on the functionality that LMDB implemented in 1.0.0 (and we also have that in our 0.9.90) where the actual space used by the database on the disk grows proportionally to how much data we store in it. Looks like that functionality is not available in 0.9.35, so it would be non-trivial for us to downgrade to that version, hence we'd really prefer to change to a tag that does support this dynamic growing behavior.
So a couple questions here: is it actually intended behavior to allocate the database of size much larger than what a user might have available on their drive?In order to use that, should we stick to 1.0.0 and up?
https://bugs.openldap.org/show_bug.cgi?id=10552
--- Comment #6 from Howard Chu hyc@openldap.org --- (In reply to donxenapo from comment #5)
Okay, then I guess it's important to note that we're using LMDB in a potentially non-conventional way where we set the _maximum_ database size to a very large value (we use 100GB) for a database that we expect to be ~1-5GB on average for most users. So we rely on the functionality that LMDB implemented in 1.0.0 (and we also have that in our 0.9.90) where the actual space used by the database on the disk grows proportionally to how much data we store in it. Looks like that functionality is not available in 0.9.35, so it would be non-trivial for us to downgrade to that version, hence we'd really prefer to change to a tag that does support this dynamic growing behavior.
So a couple questions here: is it actually intended behavior to allocate the database of size much larger than what a user might have available on their drive?In order to use that, should we stick to 1.0.0 and up?
In that case yes, just use the current mdb.RE/1.0 branch. It's ready for release soon anyway.
https://bugs.openldap.org/show_bug.cgi?id=10552
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Keywords|needs_review | Status|RESOLVED |VERIFIED
https://bugs.openldap.org/show_bug.cgi?id=10552
--- Comment #7 from donxenapo@gmail.com --- We have deployed LMDB v1.0.0 from the current tag, and unfortunately looks like the corruption has not stopped. We added code that recreates the database once a corruption is detected, which lowered our crash rate pretty substantially, but the crashes are still there, mostly with
Error: MDB_PAGE_NOTFOUND: Requested page not found Error: MDB_PROBLEM: Unexpected problem - txn should abort etc
We got a couple user reports that at least some of these crashes happen right after they got a BSOD or a reboot, so looks like there is a higher chance to corrupt the database during an abnormal system shutdown.
One relevant flag we're using here is MDB_NOMETASYNC:
const auto error = mdb_env_open(env.get(), location.c_str(), MDB_NOMETASYNC, 0644);
and from what I understand, LMDB is supposed to handle abnormal shutdowns gracefully with this flag?
We have never reproduced this issue in-house, but in the wild it's in the top 10 of our crashes.
https://bugs.openldap.org/show_bug.cgi?id=10552
donxenapo@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Version|unspecified |1.0.0
https://bugs.openldap.org/show_bug.cgi?id=10552
donxenapo@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Resolution|INVALID |--- Status|VERIFIED |UNCONFIRMED
https://bugs.openldap.org/show_bug.cgi?id=10552
--- Comment #8 from Howard Chu hyc@openldap.org --- (In reply to donxenapo from comment #7)
We have deployed LMDB v1.0.0 from the current tag, and unfortunately looks like the corruption has not stopped.
I recommended you to use the 1.0 release branch. Not the 1.0.0 tag. We're about to release 1.0.1. 1.0.0 is obviously broken.
In version 1.0 Windows uses asynchronous overlapped writes by default. You may want to fall back to using standard synchronous writes instead. Compile LMDB with -DMDB_USE_WRITE_THROUGH=0.
You may want to reconsider NOMETASYNC, it doesn't really gain that much performance.
We added code that recreates the database once a corruption is detected, which lowered our crash rate pretty substantially, but the crashes are still there, mostly with
Error: MDB_PAGE_NOTFOUND: Requested page not found Error: MDB_PROBLEM: Unexpected problem - txn should abort etc
We got a couple user reports that at least some of these crashes happen right after they got a BSOD or a reboot, so looks like there is a higher chance to corrupt the database during an abnormal system shutdown.
One relevant flag we're using here is MDB_NOMETASYNC:
const auto error = mdb_env_open(env.get(), location.c_str(), MDB_NOMETASYNC, 0644);
and from what I understand, LMDB is supposed to handle abnormal shutdowns gracefully with this flag?
We have never reproduced this issue in-house, but in the wild it's in the top 10 of our crashes.
https://bugs.openldap.org/show_bug.cgi?id=10552
--- Comment #9 from Howard Chu hyc@openldap.org ---
We have never reproduced this issue in-house, but in the wild it's in the top 10 of our crashes.
One more thought - the default on Windows in LMDB 1.0 is to use write-through writes. Which means all writes go directly to the storage device, instead of sitting around in buffers that need to be flushed. The errors you're seeing are indicative of writes that got dropped, never made it to persistent storage. Most likely these are drives that lie about completing synchronous writes. And if you're using better quality drives in-house, that explains why you've never seen the problem yourselves.
This is unlikely to be resolved by any LMDB patch or even Windows patch, it sounds like flaky hardware.
-
openldap-its＠openldap.org