openldap-bugs August 2016

openldap-bugs@openldap.org

21 participants
50 discussions

Re: (ITS#8475) Feature request: MDB low durability transactions
by h.b.furuseth＠usit.uio.no 10 Aug '16

10 Aug '16

Nope, you're as confused as I was originally:-) LDMB doesn't know or care when a page was written. A page can be reused when the snapshot which _freed_ it is known to be durable and there are no older readers. (We could improve that by tracking page history better. Maybe later.) "Known to be durable" = sync datapages, write metapage, sync metapage, note that the metapage was synced. (We implicitly note that when writing next txn's metapage, since we must have synced first.) From a data safety point of view, txns which do all that are the real txns. Anything else is fluff, like no-sync txns if we implement them. Their metapages must go somewhere they *won't* be confused with durable ones. Think of such a fluffy commit as saving an intermediate stage of a real txn. That's irrelevant to a later write-txn wanting to not touch the last two durable snapshots. It's only relevant vs. oldest reader. So. 3rd metapage and variants - I've tried and Howard pointed out the flaws, Howard tried and I said here we go again. We do not need another round, but it's just as well to have it summarized here. (This discussion ignores MDB_NOSYNC and partly MDB_NOLOCK - if the user enables either, it's his responsibility to compensate.) -- Hallvard

1 0

Re: (ITS#8474) Securely Erase BerElement Buffer After Use
by alcronin＠cisco.com 09 Aug '16

09 Aug '16

SGksDQoNCkkgY2FuIHRha2UgYSBsb29rIGF0IHRoZSBzbGFwZCBzaWRlIGFuZCB0aG9zZSBvdGhl ciByZXF1ZXN0cyBhbmQgaW1wbGVtZW50IHRoZSBmaXggdGhlcmUgYXMgd2VsbC4gSWYgdGhlcmUg YXJlIGFueSBvdGhlciBhcmVhcyBsZXQgbWUga25vdyBhbmQgSSBjYW4gYWRkcmVzcyB0aGVtIHRv LiBJIHdpbGwgdXBkYXRlIHRoZSBwYXRjaCB3aGVuIEkgaGF2ZSB0aG9zZSBjaGFuZ2VzIHJlYWR5 Lg0KDQpUaGUgY29tcGlsZXJzIHdpbGwgdHJlYXQgdGhhdCAobWVtc2V0IGJlaGF2aW91cikgYXMg YW4gb3B0aW1pemF0aW9uIGFuZCB3b3VsZCBiZSBoZXNpdGFudCB0byBjaGFuZ2UgaXQuIFRoaXMg aXMgbW9zdGx5IHNlZW4gaW4gR0NDIGFuZCBhbnkgdmFyaWFudHMuIFRoZXJlIGFyZSBvdGhlciBv cHRpbWlzYXRpb25zIHN1Y2ggYXMgcmVtb3ZpbmcgdGhlIGNhbGwgdG8gdGhlIG1ldGhvZCBlbnRp cmVseSBpZiB0aGUgbmV4dCBvcGVyYXRpb24gaXMgYSBmcmVlIGFzIHRoaXMgd291bGQgYmUgc2Vl biBieSB0aGUgY29tcGlsZXIgYXMgYSBuby1vcC4gUmVtb3Zpbmcgb3B0aW1pc2F0aW9ucyBpcyBu b3QgaWRlYWwgYXMgaXQgY2FuIGFmZmVjdCBtb3JlIHRoYW4ganVzdCB0aGUgc2VjdGlvbiBvZiB0 aGUgY29kZSBhbmQgaXMgbm90IGEgY3Jvc3MgcGxhdGZvcm0gc29sdXRpb24uDQoNClRoYW5rcywN CkFsYW4NCg0KT24gMDgvMDgvMjAxNiwgMjE6MzcsICJIb3dhcmQgQ2h1IiA8aHljQHN5bWFzLmNv bT4gd3JvdGU6DQoNCiAgICBBbGFuIENyb25pbiAoYWxjcm9uaW4pIHdyb3RlOg0KICAgID4gSGkg SG93YXJkLA0KICAgID4NCiAgICA+IEkgY2FuIGNoYW5nZSB0aGUgY29kZSBzbyB0aGF0IGl0IG92 ZXJ3cml0ZXMgdGhlIG1lbW9yeSBmcm9tDQogICAgbGRhcF9mcmVlX3JlcXVlc3RfaW50KCkgaW5z dGVhZC4gVGhpcyB3aWxsIGNvdmVyIHRoZSBleGVjdXRpb24gdXNpbmcgYSBiaW5kDQogICAgKGJv dGggc3luY2hyb25vdXMgYW5kIGFzeW5jaHJvbm91cykuIElzIHRoZXJlIGFueSBvdGhlciBwbGFj ZXMgd2hlcmUgdGhlDQogICAgcGFzc3dvcmQgd291bGQgYmUgaW4gdGhlIGJ1ZmZlciB0aGF0IG5l ZWRzIHRvIGJlIGNsZWFyZWQ/IFRoZSByZWFzb24gZm9yDQogICAgaW5zZXJ0aW5nIGl0IGludG8g YmVyX2ZyZWVfYnVmKCkgd2FzIHRoYXQgaXQgd291bGQgY2xlYXIgYWxsIHBsYWNlcy4NCiAgICAN CiAgICBJIHNlZSwgeW91J3JlIGxvb2tpbmcgYXQgdGhlIGNsaWVudCBzaWRlLiBXaGF0IGFib3V0 IHRoZSBzbGFwZCBzaWRlPw0KICAgIA0KICAgIFRoZXJlIGFyZSAyIG1haW4gcGxhY2VzIHdoZXJl IHBhc3N3b3JkcyBvY2N1ciAtIEJpbmQsIGFuZCBQYXNzd29yZE1vZGlmeSBleG9wLiANCiAgICBU aGV5IG1pZ2h0IGFsc28gb2NjdXIgaW4gQWRkIGFuZCBNb2RpZnkgcmVxdWVzdHMuIEFkZCBpcyBu b3Qgc28gdW51c3VhbCwgDQogICAgTW9kaWZ5IHNob3VsZCBub3QgaGFwcGVuIHNpbmNlIGNsaWVu dHMgc2hvdWxkIHVzZSBQYXNzd29yZE1vZGlmeSBpbnN0ZWFkLg0KICAgIA0KICAgID4gU29tZSBj b21waWxlcnMgd2lsbCBvcHRpbWl6ZSBjYWxscyB0byBtZW1zZXQoKSBzbyB0aGF0IG9ubHkgdGhl IGZpcnN0DQogICAgY2hhcmFjdGVyIHdpbGwgYmUgY2hhbmdlZC4gTGVhdmluZyB0aGUgcmVzdCBv ZiB0aGUgc3RyaW5nIGludGFjdC4NCiAgICANCiAgICBUaGF0J3MgY2xlYXJseSBhIGNvbXBpbGVy IGJ1ZyB0aGVuLiBJZiB5b3UgaGF2ZSBleGFtcGxlcyBvZiBzdWNoIHNpdHVhdGlvbnMgDQogICAg eW91IHNob3VsZCByZXBvcnQgdGhhdCB0byB0aGUgcmVzcGVjdGl2ZSBjb21waWxlciBtYWludGFp bmVycy4NCiAgICANCiAgICA+IFRoaXMgaXMgd2h5DQogICAgdGhlIGl0ZXJhdGlvbiBpcyBpbiBw bGFjZS4NCiAgICA+DQogICAgPiBUaGFua3MgZm9yIGxvb2tpbmcgaW50byB0aGlzIHBhdGNoLg0K ICAgID4NCiAgICA+IEFsYW4NCiAgICA+DQogICAgPiBPbiAwNS8wOC8yMDE2LCAxOTo1MCwgIkhv d2FyZCBDaHUiIDxoeWNAc3ltYXMuY29tPiB3cm90ZToNCiAgICA+DQogICAgPiAgICAgIGFsY3Jv bmluQGNpc2NvLmNvbSB3cm90ZToNCiAgICA+ICAgICAgPiBGdWxsX05hbWU6IEFsYW4gQ3Jvbmlu DQogICAgPiAgICAgID4gVmVyc2lvbjogMi40LjQ0DQogICAgPiAgICAgID4gT1M6IFdpbmRvd3Mg OC4xDQogICAgPiAgICAgID4gVVJMOiBodHRwczovL2RsLmRyb3Bib3h1c2VyY29udGVudC5jb20v dS84MjM0MzQ3NS9TZWN1cmVseUVyYXNlQnVmZmVyLmRpZmYNCiAgICA+ICAgICAgPiBTdWJtaXNz aW9uIGZyb206IChOVUxMKSAoMjAwMTo0MjA6NDA0MToyMDAzOmY5NDI6NTlhNTo1NDVmOmIzMzQp DQogICAgPiAgICAgID4NCiAgICA+ICAgICAgPg0KICAgID4gICAgICA+IFRoZSBmb2xsb3dpbmcg cGF0Y2ggaXMgYSBtb2RpZmljYXRpb24gdG8gdGhlIE9wZW5MREFQIEJlckVsZW1lbnQgYnVmZmVy LiBUaGUNCiAgICA+ICAgICAgPiBidWZmZXIgY2FuIGJlIHVzZWQgdG8gY29udGFpbiB0aGUgTERB UCByZXF1ZXN0IGluY2x1ZGluZyB0aGUgcGFzc3dvcmQgZm9yDQogICAgPiAgICAgID4gYXV0aGVu dGljYXRpb24uIFdoaWxlIHRoaXMgaXMgZnJlZSdkIHdoZW4gaXQgaXMgbm8gbG9uZ2VyIG5lZWRl ZCwgdGhlIGNvbnRlbnRzDQogICAgPiAgICAgID4gb2YgdGhlIGJ1ZmZlciBpcyBub3Qgb3Zlcndy aXR0ZW4gZnJvbSBtZW1vcnkuIFRoaXMgY2FuIGxlYWQgdG8gc29tZW9uZSByZWFkaW5nDQogICAg PiAgICAgID4gdGhlIG1lbW9yeSBvZiB0aGUgcHJvY2VzcyBhbmQgZGV0ZXJtaW5pbmcgd2hhdCB0 aGUgcGFzc3dvcmQgaXMuIFRoZSBjaGFuZ2UNCiAgICA+ICAgICAgPiBpbmNsdWRlZCBpbiB0aGlz IHBhdGNoIHdpbGwgaXRlcmF0ZSBvdmVyIHRoZSBtZW1vcnkgYW5kIGNsZWFyIGl0LiBUaGlzIHdp bGwNCiAgICA+ICAgICAgPiByZW1vdmUgYW55IHRyYWNlIG9mIHRoZSBwYXNzd29yZCBieSB0aGUg dGltZSBleGVjdXRpb24gaXMgaGFuZGVkIGJhY2sgdG8gdGhlDQogICAgPiAgICAgID4gY2FsbGVy Lg0KICAgID4NCiAgICA+ICAgICAgV2h5IHdvdWxkIHlvdSBpbnNlcnQgYSBwZXJmb3JtYW5jZSBw ZXNzaW1pemF0aW9uIGludG8gZXZlcnkgdXNlIG9mIHRoZSBMQkVSDQogICAgPiAgICAgIGxpYnJh cnksIHJhdGhlciB0aGFuIGp1c3QgZXJhc2luZyB0aGUgcGFzc3dvcmQgZnJvbSBhIEJpbmQgcmVx dWVzdD8NCiAgICA+DQogICAgPiAgICAgIFdoeSB3b3VsZCB5b3UgdXNlIGFuIGV4cGxpY2l0bHkg Y29kZWQgbG9vcCBzZXR0aW5nIG9uZSBjaGFyYWN0ZXIgYXQgYSB0aW1lLA0KICAgID4gICAgICBp bnN0ZWFkIG9mIHVzaW5nIGxpYmMncyBtZW1zZXQoKSB3aGljaCBoYXMgcHJvYmFibHkgYmVlbiB3 ZWxsIG9wdGltaXplZD8NCiAgICA+DQogICAgPiAgICAgID4gVGhlIGF0dGFjaGVkIHBhdGNoIGZp bGUgaXMgZGVyaXZlZCBmcm9tIE9wZW5MREFQIFNvZnR3YXJlLiBBbGwgb2YgdGhlDQogICAgPiAg ICAgID4gbW9kaWZpY2F0aW9ucyB0byBPcGVuTERBUCBTb2Z0d2FyZSByZXByZXNlbnRlZCBpbiB0 aGUgZm9sbG93aW5nIHBhdGNoKGVzKSB3ZXJlDQogICAgPiAgICAgID4gZGV2ZWxvcGVkIGJ5IEFs YW4gQ3JvbmluIGFsY3JvbmluQGNpc2NvLmNvbS4gSSBoYXZlIG5vdCBhc3NpZ25lZCByaWdodHMg YW5kL29yDQogICAgPiAgICAgID4gaW50ZXJlc3QgaW4gdGhpcyB3b3JrIHRvIGFueSBwYXJ0eS4N CiAgICA+ICAgICAgPiBUaGUgYXR0YWNoZWQgZmlsZSBpcyBkZXJpdmVkIGZyb20gT3BlbkxEQVAg U29mdHdhcmUuIEFsbCBvZiB0aGUgbW9kaWZpY2F0aW9ucyB0bw0KICAgID4gICAgICA+IE9wZW5M REFQIFNvZnR3YXJlIHJlcHJlc2VudGVkIGluIHRoZSBmb2xsb3dpbmcgcGF0Y2goZXMpIHdlcmUg ZGV2ZWxvcGVkIGJ5IENpc2NvDQogICAgPiAgICAgID4gU3lzdGVtcywgSW5jLi4gQ2lzY28gU3lz dGVtcywgSW5jLiBoYXMgbm90IGFzc2lnbmVkIHJpZ2h0cyBhbmQvb3IgaW50ZXJlc3QgaW4NCiAg ICA+ICAgICAgPiB0aGlzIHdvcmsgdG8gYW55IHBhcnR5LiBJLCBBbGFuIENyb25pbiBhbSBhdXRo b3JpemVkIGJ5IENpc2NvIFN5c3RlbXMsIEluYy4sIG15DQogICAgPiAgICAgID4gZW1wbG95ZXIs IHRvIHJlbGVhc2UgdGhpcyB3b3JrIHVuZGVyIHRoZSBmb2xsb3dpbmcgdGVybXMuDQogICAgPiAg ICAgID4gQ2lzY28gU3lzdGVtcywgSW5jLiBoZXJlYnkgcGxhY2UgdGhlIGZvbGxvd2luZyBtb2Rp ZmljYXRpb25zIHRvIE9wZW5MREFQDQogICAgPiAgICAgID4gU29mdHdhcmUgKGFuZCBvbmx5IHRo ZXNlIG1vZGlmaWNhdGlvbnMpIGludG8gdGhlIHB1YmxpYyBkb21haW4uIEhlbmNlLCB0aGVzZQ0K ICAgID4gICAgICA+IG1vZGlmaWNhdGlvbnMgbWF5IGJlIGZyZWVseSB1c2VkIGFuZC9vciByZWRp c3RyaWJ1dGVkIGZvciBhbnkgcHVycG9zZSB3aXRoIG9yDQogICAgPiAgICAgID4gd2l0aG91dCBh dHRyaWJ1dGlvbiBhbmQvb3Igb3RoZXIgbm90aWNlLg0KICAgID4gICAgICA+DQogICAgPiAgICAg ID4NCiAgICA+DQogICAgPg0KICAgID4gICAgICAtLQ0KICAgID4gICAgICAgICAtLSBIb3dhcmQg Q2h1DQogICAgPiAgICAgICAgIENUTywgU3ltYXMgQ29ycC4gICAgICAgICAgIGh0dHA6Ly93d3cu c3ltYXMuY29tDQogICAgPiAgICAgICAgIERpcmVjdG9yLCBIaWdobGFuZCBTdW4gICAgIGh0dHA6 Ly9oaWdobGFuZHN1bi5jb20vaHljLw0KICAgID4gICAgICAgICBDaGllZiBBcmNoaXRlY3QsIE9w ZW5MREFQICBodHRwOi8vd3d3Lm9wZW5sZGFwLm9yZy9wcm9qZWN0Lw0KICAgID4NCiAgICA+DQog ICAgDQogICAgDQogICAgLS0gDQogICAgICAgLS0gSG93YXJkIENodQ0KICAgICAgIENUTywgU3lt YXMgQ29ycC4gICAgICAgICAgIGh0dHA6Ly93d3cuc3ltYXMuY29tDQogICAgICAgRGlyZWN0b3Is IEhpZ2hsYW5kIFN1biAgICAgaHR0cDovL2hpZ2hsYW5kc3VuLmNvbS9oeWMvDQogICAgICAgQ2hp ZWYgQXJjaGl0ZWN0LCBPcGVuTERBUCAgaHR0cDovL3d3dy5vcGVubGRhcC5vcmcvcHJvamVjdC8N CiAgICANCg0K

1 0

Re: (ITS#8474) Securely Erase BerElement Buffer After Use
by hyc＠symas.com 08 Aug '16

08 Aug '16

Alan Cronin (alcronin) wrote: > Hi Howard, > > I can change the code so that it overwrites the memory from ldap_free_request_int() instead. This will cover the execution using a bind (both synchronous and asynchronous). Is there any other places where the password would be in the buffer that needs to be cleared? The reason for inserting it into ber_free_buf() was that it would clear all places. I see, you're looking at the client side. What about the slapd side? There are 2 main places where passwords occur - Bind, and PasswordModify exop. They might also occur in Add and Modify requests. Add is not so unusual, Modify should not happen since clients should use PasswordModify instead. > Some compilers will optimize calls to memset() so that only the first character will be changed. Leaving the rest of the string intact. That's clearly a compiler bug then. If you have examples of such situations you should report that to the respective compiler maintainers. > This is why the iteration is in place. > > Thanks for looking into this patch. > > Alan > > On 05/08/2016, 19:50, "Howard Chu" <hyc(a)symas.com> wrote: > > alcronin(a)cisco.com wrote: > > Full_Name: Alan Cronin > > Version: 2.4.44 > > OS: Windows 8.1 > > URL: https://dl.dropboxusercontent.com/u/82343475/SecurelyEraseBuffer.diff > > Submission from: (NULL) (2001:420:4041:2003:f942:59a5:545f:b334) > > > > > > The following patch is a modification to the OpenLDAP BerElement buffer. The > > buffer can be used to contain the LDAP request including the password for > > authentication. While this is free'd when it is no longer needed, the contents > > of the buffer is not overwritten from memory. This can lead to someone reading > > the memory of the process and determining what the password is. The change > > included in this patch will iterate over the memory and clear it. This will > > remove any trace of the password by the time execution is handed back to the > > caller. > > Why would you insert a performance pessimization into every use of the LBER > library, rather than just erasing the password from a Bind request? > > Why would you use an explicitly coded loop setting one character at a time, > instead of using libc's memset() which has probably been well optimized? > > > The attached patch file is derived from OpenLDAP Software. All of the > > modifications to OpenLDAP Software represented in the following patch(es) were > > developed by Alan Cronin alcronin(a)cisco.com. I have not assigned rights and/or > > interest in this work to any party. > > The attached file is derived from OpenLDAP Software. All of the modifications to > > OpenLDAP Software represented in the following patch(es) were developed by Cisco > > Systems, Inc.. Cisco Systems, Inc. has not assigned rights and/or interest in > > this work to any party. I, Alan Cronin am authorized by Cisco Systems, Inc., my > > employer, to release this work under the following terms. > > Cisco Systems, Inc. hereby place the following modifications to OpenLDAP > > Software (and only these modifications) into the public domain. Hence, these > > modifications may be freely used and/or redistributed for any purpose with or > > without attribution and/or other notice. > > > > > > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8475) Feature request: MDB low durability transactions
by bentrask＠comcast.net 08 Aug '16

08 Aug '16

On 08/08/2016 05:41 AM, Hallvard Breien Furuseth wrote: > A transaction must not reuse data pages visible in the last snapshot > known to be durable, since that's how far back LDMB may need to revert > after abnormal termination. Like a crash after MDB_NOMETASYNC may do. > > Sync the data pages from a txn, write the metapage, eventually sync > that metapage, wait out any older read-only transactions, and *then* > you can reuse the pages the txn freed. Not before. So when you don't > sync, or a read-only txn won't die, LMDB degenerates to append-only. > > ...except if you sync the metapage and exit, next LMDB run may not > know you synced it and must assume the metapage isn't yet durable. > So it might not reuse pages visible to the _previous_ durable > metapage, until it syncs. I'm rather losing track at this point, > but I think it may mean twice as may not-yet-usable pages as one > might expect. Concretely: say the current write transaction is number 10, and a long-lived reader is on number 7. Currently, MDB will be unable to reuse any pages used in transactions 7+ until the reader ends. Now say a 3rd, durable root is added. For the sake of argument, no checksums are used and in the event of a crash, only the last durable state is recovered. Say the durable transaction is number 2. Pages used in transaction 2 need to be preserved, obviously. 7+ still need to be preserved for the slow reader. But pages from transactions 3-6 can be reused. Note that the last durable transaction is controlled purely by the single writer, so tracking it is actually easier than tracking which readers are where. If a crash happens before a durable root is fully synced, then there should be a second, older durable root that hasn't been reused yet. In that case MDB recovers the way it does currently. Does this make sense? Thanks for bearing with me.

1 0

Re: (ITS#8474) Securely Erase BerElement Buffer After Use
by alcronin＠cisco.com 08 Aug '16

08 Aug '16

SGkgSG93YXJkLA0KDQpJIGNhbiBjaGFuZ2UgdGhlIGNvZGUgc28gdGhhdCBpdCBvdmVyd3JpdGVz IHRoZSBtZW1vcnkgZnJvbSBsZGFwX2ZyZWVfcmVxdWVzdF9pbnQoKSBpbnN0ZWFkLiBUaGlzIHdp bGwgY292ZXIgdGhlIGV4ZWN1dGlvbiB1c2luZyBhIGJpbmQgKGJvdGggc3luY2hyb25vdXMgYW5k IGFzeW5jaHJvbm91cykuIElzIHRoZXJlIGFueSBvdGhlciBwbGFjZXMgd2hlcmUgdGhlIHBhc3N3 b3JkIHdvdWxkIGJlIGluIHRoZSBidWZmZXIgdGhhdCBuZWVkcyB0byBiZSBjbGVhcmVkPyBUaGUg cmVhc29uIGZvciBpbnNlcnRpbmcgaXQgaW50byBiZXJfZnJlZV9idWYoKSB3YXMgdGhhdCBpdCB3 b3VsZCBjbGVhciBhbGwgcGxhY2VzLg0KDQpTb21lIGNvbXBpbGVycyB3aWxsIG9wdGltaXplIGNh bGxzIHRvIG1lbXNldCgpIHNvIHRoYXQgb25seSB0aGUgZmlyc3QgY2hhcmFjdGVyIHdpbGwgYmUg Y2hhbmdlZC4gTGVhdmluZyB0aGUgcmVzdCBvZiB0aGUgc3RyaW5nIGludGFjdC4gVGhpcyBpcyB3 aHkgdGhlIGl0ZXJhdGlvbiBpcyBpbiBwbGFjZS4NCg0KVGhhbmtzIGZvciBsb29raW5nIGludG8g dGhpcyBwYXRjaC4NCg0KQWxhbg0KDQpPbiAwNS8wOC8yMDE2LCAxOTo1MCwgIkhvd2FyZCBDaHUi IDxoeWNAc3ltYXMuY29tPiB3cm90ZToNCg0KICAgIGFsY3JvbmluQGNpc2NvLmNvbSB3cm90ZToN CiAgICA+IEZ1bGxfTmFtZTogQWxhbiBDcm9uaW4NCiAgICA+IFZlcnNpb246IDIuNC40NA0KICAg ID4gT1M6IFdpbmRvd3MgOC4xDQogICAgPiBVUkw6IGh0dHBzOi8vZGwuZHJvcGJveHVzZXJjb250 ZW50LmNvbS91LzgyMzQzNDc1L1NlY3VyZWx5RXJhc2VCdWZmZXIuZGlmZg0KICAgID4gU3VibWlz c2lvbiBmcm9tOiAoTlVMTCkgKDIwMDE6NDIwOjQwNDE6MjAwMzpmOTQyOjU5YTU6NTQ1ZjpiMzM0 KQ0KICAgID4NCiAgICA+DQogICAgPiBUaGUgZm9sbG93aW5nIHBhdGNoIGlzIGEgbW9kaWZpY2F0 aW9uIHRvIHRoZSBPcGVuTERBUCBCZXJFbGVtZW50IGJ1ZmZlci4gVGhlDQogICAgPiBidWZmZXIg Y2FuIGJlIHVzZWQgdG8gY29udGFpbiB0aGUgTERBUCByZXF1ZXN0IGluY2x1ZGluZyB0aGUgcGFz c3dvcmQgZm9yDQogICAgPiBhdXRoZW50aWNhdGlvbi4gV2hpbGUgdGhpcyBpcyBmcmVlJ2Qgd2hl biBpdCBpcyBubyBsb25nZXIgbmVlZGVkLCB0aGUgY29udGVudHMNCiAgICA+IG9mIHRoZSBidWZm ZXIgaXMgbm90IG92ZXJ3cml0dGVuIGZyb20gbWVtb3J5LiBUaGlzIGNhbiBsZWFkIHRvIHNvbWVv bmUgcmVhZGluZw0KICAgID4gdGhlIG1lbW9yeSBvZiB0aGUgcHJvY2VzcyBhbmQgZGV0ZXJtaW5p bmcgd2hhdCB0aGUgcGFzc3dvcmQgaXMuIFRoZSBjaGFuZ2UNCiAgICA+IGluY2x1ZGVkIGluIHRo aXMgcGF0Y2ggd2lsbCBpdGVyYXRlIG92ZXIgdGhlIG1lbW9yeSBhbmQgY2xlYXIgaXQuIFRoaXMg d2lsbA0KICAgID4gcmVtb3ZlIGFueSB0cmFjZSBvZiB0aGUgcGFzc3dvcmQgYnkgdGhlIHRpbWUg ZXhlY3V0aW9uIGlzIGhhbmRlZCBiYWNrIHRvIHRoZQ0KICAgID4gY2FsbGVyLg0KICAgIA0KICAg IFdoeSB3b3VsZCB5b3UgaW5zZXJ0IGEgcGVyZm9ybWFuY2UgcGVzc2ltaXphdGlvbiBpbnRvIGV2 ZXJ5IHVzZSBvZiB0aGUgTEJFUiANCiAgICBsaWJyYXJ5LCByYXRoZXIgdGhhbiBqdXN0IGVyYXNp bmcgdGhlIHBhc3N3b3JkIGZyb20gYSBCaW5kIHJlcXVlc3Q/DQogICAgDQogICAgV2h5IHdvdWxk IHlvdSB1c2UgYW4gZXhwbGljaXRseSBjb2RlZCBsb29wIHNldHRpbmcgb25lIGNoYXJhY3RlciBh dCBhIHRpbWUsIA0KICAgIGluc3RlYWQgb2YgdXNpbmcgbGliYydzIG1lbXNldCgpIHdoaWNoIGhh cyBwcm9iYWJseSBiZWVuIHdlbGwgb3B0aW1pemVkPw0KICAgIA0KICAgID4gVGhlIGF0dGFjaGVk IHBhdGNoIGZpbGUgaXMgZGVyaXZlZCBmcm9tIE9wZW5MREFQIFNvZnR3YXJlLiBBbGwgb2YgdGhl DQogICAgPiBtb2RpZmljYXRpb25zIHRvIE9wZW5MREFQIFNvZnR3YXJlIHJlcHJlc2VudGVkIGlu IHRoZSBmb2xsb3dpbmcgcGF0Y2goZXMpIHdlcmUNCiAgICA+IGRldmVsb3BlZCBieSBBbGFuIENy b25pbiBhbGNyb25pbkBjaXNjby5jb20uIEkgaGF2ZSBub3QgYXNzaWduZWQgcmlnaHRzIGFuZC9v cg0KICAgID4gaW50ZXJlc3QgaW4gdGhpcyB3b3JrIHRvIGFueSBwYXJ0eS4NCiAgICA+IFRoZSBh dHRhY2hlZCBmaWxlIGlzIGRlcml2ZWQgZnJvbSBPcGVuTERBUCBTb2Z0d2FyZS4gQWxsIG9mIHRo ZSBtb2RpZmljYXRpb25zIHRvDQogICAgPiBPcGVuTERBUCBTb2Z0d2FyZSByZXByZXNlbnRlZCBp biB0aGUgZm9sbG93aW5nIHBhdGNoKGVzKSB3ZXJlIGRldmVsb3BlZCBieSBDaXNjbw0KICAgID4g U3lzdGVtcywgSW5jLi4gQ2lzY28gU3lzdGVtcywgSW5jLiBoYXMgbm90IGFzc2lnbmVkIHJpZ2h0 cyBhbmQvb3IgaW50ZXJlc3QgaW4NCiAgICA+IHRoaXMgd29yayB0byBhbnkgcGFydHkuIEksIEFs YW4gQ3JvbmluIGFtIGF1dGhvcml6ZWQgYnkgQ2lzY28gU3lzdGVtcywgSW5jLiwgbXkNCiAgICA+ IGVtcGxveWVyLCB0byByZWxlYXNlIHRoaXMgd29yayB1bmRlciB0aGUgZm9sbG93aW5nIHRlcm1z Lg0KICAgID4gQ2lzY28gU3lzdGVtcywgSW5jLiBoZXJlYnkgcGxhY2UgdGhlIGZvbGxvd2luZyBt b2RpZmljYXRpb25zIHRvIE9wZW5MREFQDQogICAgPiBTb2Z0d2FyZSAoYW5kIG9ubHkgdGhlc2Ug bW9kaWZpY2F0aW9ucykgaW50byB0aGUgcHVibGljIGRvbWFpbi4gSGVuY2UsIHRoZXNlDQogICAg PiBtb2RpZmljYXRpb25zIG1heSBiZSBmcmVlbHkgdXNlZCBhbmQvb3IgcmVkaXN0cmlidXRlZCBm b3IgYW55IHB1cnBvc2Ugd2l0aCBvcg0KICAgID4gd2l0aG91dCBhdHRyaWJ1dGlvbiBhbmQvb3Ig b3RoZXIgbm90aWNlLg0KICAgID4NCiAgICA+DQogICAgDQogICAgDQogICAgLS0gDQogICAgICAg LS0gSG93YXJkIENodQ0KICAgICAgIENUTywgU3ltYXMgQ29ycC4gICAgICAgICAgIGh0dHA6Ly93 d3cuc3ltYXMuY29tDQogICAgICAgRGlyZWN0b3IsIEhpZ2hsYW5kIFN1biAgICAgaHR0cDovL2hp Z2hsYW5kc3VuLmNvbS9oeWMvDQogICAgICAgQ2hpZWYgQXJjaGl0ZWN0LCBPcGVuTERBUCAgaHR0 cDovL3d3dy5vcGVubGRhcC5vcmcvcHJvamVjdC8NCiAgICANCg0K

1 0

Re: (ITS#8475) Feature request: MDB low durability transactions
by h.b.furuseth＠usit.uio.no 08 Aug '16

08 Aug '16

On 08/08/16 03:51, bentrask(a)comcast.net wrote: > The idea was that the two "floating" roots would reuse pages the way MDB > does now. The 3rd durable root would have its pages preserved > separately. I can see why this would cause up to a ~2X storage increase > as the two sets diverged, but I don't see why it would need to grow > unbounded. Apologies for this stupid question. A transaction must not reuse data pages visible in the last snapshot known to be durable, since that's how far back LDMB may need to revert after abnormal termination. Like a crash after MDB_NOMETASYNC may do. Sync the data pages from a txn, write the metapage, eventually sync that metapage, wait out any older read-only transactions, and *then* you can reuse the pages the txn freed. Not before. So when you don't sync, or a read-only txn won't die, LMDB degenerates to append-only. ...except if you sync the metapage and exit, next LMDB run may not know you synced it and must assume the metapage isn't yet durable. So it might not reuse pages visible to the _previous_ durable metapage, until it syncs. I'm rather losing track at this point, but I think it may mean twice as may not-yet-usable pages as one might expect.

1 0

Re: (ITS#8475) Feature request: MDB low durability transactions
by bentrask＠comcast.net 07 Aug '16

07 Aug '16

On 08/07/2016 09:29 PM, Howard Chu wrote: > Knowing whether or not the root pages are pristine still doesn't tell > you anything about whether the data pages are intact. The only way to > make any of these schemes work is to avoid overwriting/reusing any data > pages for the last N transactions. I.e., reverting to append-only > behavior. So the underlying question (which we have wrestled with > internally for quite some time) which you haven't asked or answered - > how many of these non-durable transactions will you support at any given > time? The idea was that the two "floating" roots would reuse pages the way MDB does now. The 3rd durable root would have its pages preserved separately. I can see why this would cause up to a ~2X storage increase as the two sets diverged, but I don't see why it would need to grow unbounded. Apologies for this stupid question.

1 0

Re: (ITS#8475) Feature request: MDB low durability transactions
by hyc＠symas.com 07 Aug '16

07 Aug '16

Ben Trask wrote: > On 08/07/2016 05:44 PM, Howard Chu wrote: >> The only way to guarantee integrity is with ordered writes. All SCSI >> devices support this feature, but e.g. the Linux kernel does not (and >> neither does SATA, and no idea about PCIe SSDs...). >> >> Lacking a portable mechanism for ordered writes, you have two choices >> for preserving integrity - append-only operation (which forces ordered >> writes anyway) or at least one synchronous write somewhere. >> >> Whenever you decide to reuse existing pages rather than operating as >> append-only, you create the possibility of overwriting some required >> data before it was safe to do so. Your 3-root checksum scheme *might* >> let you detect that the DB is corrupted, but it *won't* let you recover >> to a clean state. Given that writes occur in unpredictable order, >> without fsyncs there is no way you can guarantee that anything sane is >> on the disk. > > Consider three roots without any checksums. Each root has a simple flag > indicating whether it was written durably (fsync write barrier). During > recovery, non-durable roots are simply ignored/discarded. This is equivalent > to Hallvard's suggestion for volatile meta-pages. I think it's pretty clear > this is workable. > > From there, checksums just give you slightly stronger guarantees, although > they might not be worth the overhead (CPU/storage) and recovery complexity. Knowing whether or not the root pages are pristine still doesn't tell you anything about whether the data pages are intact. The only way to make any of these schemes work is to avoid overwriting/reusing any data pages for the last N transactions. I.e., reverting to append-only behavior. So the underlying question (which we have wrestled with internally for quite some time) which you haven't asked or answered - how many of these non-durable transactions will you support at any given time? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8475) Feature request: MDB low durability transactions
by bentrask＠comcast.net 07 Aug '16

07 Aug '16

On 08/07/2016 05:44 PM, Howard Chu wrote: > The only way to guarantee integrity is with ordered writes. All SCSI > devices support this feature, but e.g. the Linux kernel does not (and > neither does SATA, and no idea about PCIe SSDs...). > > Lacking a portable mechanism for ordered writes, you have two choices > for preserving integrity - append-only operation (which forces ordered > writes anyway) or at least one synchronous write somewhere. > > Whenever you decide to reuse existing pages rather than operating as > append-only, you create the possibility of overwriting some required > data before it was safe to do so. Your 3-root checksum scheme *might* > let you detect that the DB is corrupted, but it *won't* let you recover > to a clean state. Given that writes occur in unpredictable order, > without fsyncs there is no way you can guarantee that anything sane is > on the disk. Consider three roots without any checksums. Each root has a simple flag indicating whether it was written durably (fsync write barrier). During recovery, non-durable roots are simply ignored/discarded. This is equivalent to Hallvard's suggestion for volatile meta-pages. I think it's pretty clear this is workable. From there, checksums just give you slightly stronger guarantees, although they might not be worth the overhead (CPU/storage) and recovery complexity.

1 0

Re: (ITS#8475) Feature request: MDB low durability transactions
by hyc＠symas.com 07 Aug '16

07 Aug '16

bentrask(a)comcast.net wrote: > Thanks for the replies, Hallvard and Howard! > > I was mistaken in thinking that NOMETASYNC didn't guarantee integrity. > However, my proposal would allow fsync to be omitted entirely. > > I think my approach with three roots is better than a WAL because it > keeps the read and write paths simpler and more uniform. It also doesn't > force periodic fsyncs when the log wraps, or consume unbounded space. In > fact it's very similar to the basic design of MDB. > > You're right that you'd actually need to record the page's checksum in > the parent, rather than in the page itself. I guess this would hurt the > branching factor. And then it's turtles all the way down. What you're suggesting won't work. Trust me when I say we have spent far more time thinking about this question than you have. The only way to guarantee integrity is with ordered writes. All SCSI devices support this feature, but e.g. the Linux kernel does not (and neither does SATA, and no idea about PCIe SSDs...). Lacking a portable mechanism for ordered writes, you have two choices for preserving integrity - append-only operation (which forces ordered writes anyway) or at least one synchronous write somewhere. Whenever you decide to reuse existing pages rather than operating as append-only, you create the possibility of overwriting some required data before it was safe to do so. Your 3-root checksum scheme *might* let you detect that the DB is corrupted, but it *won't* let you recover to a clean state. Given that writes occur in unpredictable order, without fsyncs there is no way you can guarantee that anything sane is on the disk. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs August 2016