openldap-bugs August 2016

openldap-bugs@openldap.org

21 participants
50 discussions

Re: (ITS#8475) Feature request: MDB low durability transactions
by bentrask＠comcast.net 07 Aug '16

07 Aug '16

Thanks for the replies, Hallvard and Howard! I was mistaken in thinking that NOMETASYNC didn't guarantee integrity. However, my proposal would allow fsync to be omitted entirely. I think my approach with three roots is better than a WAL because it keeps the read and write paths simpler and more uniform. It also doesn't force periodic fsyncs when the log wraps, or consume unbounded space. In fact it's very similar to the basic design of MDB. You're right that you'd actually need to record the page's checksum in the parent, rather than in the page itself. I guess this would hurt the branching factor. Thanks again, Ben On 08/06/2016 12:42 PM, Hallvard Breien Furuseth wrote: > On 06. aug. 2016 17:38, bentrask(a)comcast.net wrote: >> Transaction commits are one of the few bottlenecks in MDB, because it >> has to >> fsync twice, sequentially. >> >> I think MDB could support mixed low and high durability transactions >> in the same >> database by adding per-page checksums and a third root page. The idea >> is that >> when committing a low-durability transaction, no fsyncs are performed. >> (...) > > Yesno. We can get rid of fsyncs, but not that way. Checksumming each > page isn't enough. We must know it's the right version of the page and > not e.g. a similar page from a previous aborted transaction. To commit > a branch or meta page, we'd need to scan its children and checksum the > page headers (thus including their checksum) of each. Expensive. > > IIRC there are three things we can do: > > - Use and fsync a WAL (write-ahead log) instead of the database pages. > That can be cheaper because it writes one contiguous region instead > of a lot of random-access pages. Requires recovery after a crash. > > - Volatile metapages which mdb_env_open() _always_ throws away if no > other environment is already open. They are lost of the application > crashes/exits without doing a final checkpoint. > > - Improve that a bit: Put them in a shared memory region, since that > won't survive a system crash (unlike if we put them in the lockfile). > That way they'll survive application crash provided something does > a checkpoint before next system crash. > > We've discussed these sometimes and there are caveats for some of them, > I don't quite remember. One issue is that a "system crash" isn't the > only thing which can lose unsynced pages. Another is unmounting and > re-mounting the disk (i.e. an USB disk). >

1 0

(ITS#8476) slapo-constraint: info per constraint_attribute
by michael＠stroeder.com 07 Aug '16

07 Aug '16

Full_Name: Version: OS: URL: Submission from: (NULL) (213.240.180.113) If your configuration has lots of overlapping constraints affecting the same attributes finding out which constraint was violated can be hard for the user. Therefore it would be nice if one could add a specific message to a constraint_attribute directive to be returned as LDAP diagnosticMessage if a specific constraint is violated. Ahem. Personally I'd add there what I have in the comments above the constraint_attribute directive. So this could also be handy for dynamic configuration with cn=config. ;-)

1 0

Re: (ITS#8475) Feature request: MDB low durability transactions
by h.b.furuseth＠usit.uio.no 06 Aug '16

06 Aug '16

On 06/08/16 18:56, hyc(a)symas.com wrote: > We already have low durability transactions, that's what MDB_NOMETASYNC is > for. It only does one fsync per commit instead of two. Haha, I didn't notice that since this reminded me of other discussions. The stuff I was talking about was for fewer fsyncs than one per commit.

1 0

Re: (ITS#8475) Feature request: MDB low durability transactions
by hyc＠symas.com 06 Aug '16

06 Aug '16

bentrask(a)comcast.net wrote: > Full_Name: Ben Trask > Version: > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (192.241.251.11) > > > Hi, > > Transaction commits are one of the few bottlenecks in MDB, because it has to > fsync twice, sequentially. > > I think MDB could support mixed low and high durability transactions in the same We already have low durability transactions, that's what MDB_NOMETASYNC is for. It only does one fsync per commit instead of two. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8475) Feature request: MDB low durability transactions
by h.b.furuseth＠usit.uio.no 06 Aug '16

06 Aug '16

On 06. aug. 2016 17:38, bentrask(a)comcast.net wrote: > Transaction commits are one of the few bottlenecks in MDB, because it has to > fsync twice, sequentially. > > I think MDB could support mixed low and high durability transactions in the same > database by adding per-page checksums and a third root page. The idea is that > when committing a low-durability transaction, no fsyncs are performed. (...) Yesno. We can get rid of fsyncs, but not that way. Checksumming each page isn't enough. We must know it's the right version of the page and not e.g. a similar page from a previous aborted transaction. To commit a branch or meta page, we'd need to scan its children and checksum the page headers (thus including their checksum) of each. Expensive. IIRC there are three things we can do: - Use and fsync a WAL (write-ahead log) instead of the database pages. That can be cheaper because it writes one contiguous region instead of a lot of random-access pages. Requires recovery after a crash. - Volatile metapages which mdb_env_open() _always_ throws away if no other environment is already open. They are lost of the application crashes/exits without doing a final checkpoint. - Improve that a bit: Put them in a shared memory region, since that won't survive a system crash (unlike if we put them in the lockfile). That way they'll survive application crash provided something does a checkpoint before next system crash. We've discussed these sometimes and there are caveats for some of them, I don't quite remember. One issue is that a "system crash" isn't the only thing which can lose unsynced pages. Another is unmounting and re-mounting the disk (i.e. an USB disk). -- Hallvard

1 0

(ITS#8475) Feature request: MDB low durability transactions
by bentrask＠comcast.net 06 Aug '16

06 Aug '16

Full_Name: Ben Trask Version: OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (192.241.251.11) Hi, Transaction commits are one of the few bottlenecks in MDB, because it has to fsync twice, sequentially. I think MDB could support mixed low and high durability transactions in the same database by adding per-page checksums and a third root page. The idea is that when committing a low-durability transaction, no fsyncs are performed. One of the two other roots is always consistent on-disk to serve as a "backstop" in the event of power loss/kernel panic. The two other pages are allowed to alternate, with consistency only ensured by checksums. During a restore after a crash, the non-durable roots (if any) get compared against the third durable root. Any new pages get checksum'd to ensure integrity. In the worst case, the durable root is used.

1 0

Re: (ITS#8474) Securely Erase BerElement Buffer After Use
by hyc＠symas.com 05 Aug '16

05 Aug '16

alcronin(a)cisco.com wrote: > Full_Name: Alan Cronin > Version: 2.4.44 > OS: Windows 8.1 > URL: https://dl.dropboxusercontent.com/u/82343475/SecurelyEraseBuffer.diff > Submission from: (NULL) (2001:420:4041:2003:f942:59a5:545f:b334) > > > The following patch is a modification to the OpenLDAP BerElement buffer. The > buffer can be used to contain the LDAP request including the password for > authentication. While this is free'd when it is no longer needed, the contents > of the buffer is not overwritten from memory. This can lead to someone reading > the memory of the process and determining what the password is. The change > included in this patch will iterate over the memory and clear it. This will > remove any trace of the password by the time execution is handed back to the > caller. Why would you insert a performance pessimization into every use of the LBER library, rather than just erasing the password from a Bind request? Why would you use an explicitly coded loop setting one character at a time, instead of using libc's memset() which has probably been well optimized? > The attached patch file is derived from OpenLDAP Software. All of the > modifications to OpenLDAP Software represented in the following patch(es) were > developed by Alan Cronin alcronin(a)cisco.com. I have not assigned rights and/or > interest in this work to any party. > The attached file is derived from OpenLDAP Software. All of the modifications to > OpenLDAP Software represented in the following patch(es) were developed by Cisco > Systems, Inc.. Cisco Systems, Inc. has not assigned rights and/or interest in > this work to any party. I, Alan Cronin am authorized by Cisco Systems, Inc., my > employer, to release this work under the following terms. > Cisco Systems, Inc. hereby place the following modifications to OpenLDAP > Software (and only these modifications) into the public domain. Hence, these > modifications may be freely used and/or redistributed for any purpose with or > without attribution and/or other notice. > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8474) Securely Erase BerElement Buffer After Use
by alcronin＠cisco.com 05 Aug '16

05 Aug '16

Full_Name: Alan Cronin Version: 2.4.44 OS: Windows 8.1 URL: https://dl.dropboxusercontent.com/u/82343475/SecurelyEraseBuffer.diff Submission from: (NULL) (2001:420:4041:2003:f942:59a5:545f:b334) The following patch is a modification to the OpenLDAP BerElement buffer. The buffer can be used to contain the LDAP request including the password for authentication. While this is free'd when it is no longer needed, the contents of the buffer is not overwritten from memory. This can lead to someone reading the memory of the process and determining what the password is. The change included in this patch will iterate over the memory and clear it. This will remove any trace of the password by the time execution is handed back to the caller. The attached patch file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Alan Cronin alcronin(a)cisco.com. I have not assigned rights and/or interest in this work to any party. The attached file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Cisco Systems, Inc.. Cisco Systems, Inc. has not assigned rights and/or interest in this work to any party. I, Alan Cronin am authorized by Cisco Systems, Inc., my employer, to release this work under the following terms. Cisco Systems, Inc. hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice.

1 0

(ITS#8473) unkown plugin type {0} preoperation
by dominik.spychalski＠googlemail.com 02 Aug '16

02 Aug '16

Full_Name: Dominik Spychalski Version: 2.4.44 OS: Ubuntu 14.04 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (141.12.66.96) After the configuration of a self written slapi-plugin in slapd.conf >plugin preoperation /../plugin.so plugin_init I'm converting the konfiguration File to the configuration directory structure >slaptest -F /etc/openldap/slapd.d -f /etc/openldap/slapd.conf When restarting slapd the Error "...unknown plugin type {0} preoperation..." occours. Line in /etc/openldap/slapd.d/cn=config/olcDatabase={-1}frontend.ldif: >olcPlugin: {0} preoperation /../plugin.so plugin_init When I modify the line above to "olcPlugin: preoperation /../plugin.so plugin_init" and recalculate the crc32 checksum everthing works fine.

1 0

(ITS#8472) slapd killed after ldapadd
by dominik.spychalski＠googlemail.com 02 Aug '16

02 Aug '16

Full_Name: Dominik Spychalski Version: 2.4.44 OS: Ubuntu 14.04 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (141.12.66.96) When trying to delete the olcDbIndex-Entry (to add it afterwards with additional indexes) in olcDatabase={1}mdb,cn=config: dn: olcDatabase={1}mdb,cn=config changeType: modify delete: olcDbIndex ldapadd -x -D cn=config -w *** -f del.ldif the slapd process gets killed with the error message: 57a08182 ch_calloc of 1 elems of 0 bytes failed slapd: ch_malloc.c:107: ch_calloc: Assertion `0' failed. If I try to add an additional olcDbIndex with other values, everything works fine.

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs August 2016