openldap-bugs

openldap-bugs@openldap.org

1 participants
21003 discussions

Re: (ITS#8505) LMDB vs. fork()
by lmb＠cloudflare.com 29 Sep '16

29 Sep '16

On 29 September 2016 at 03:39, Hallvard Breien Furuseth <h.b.furuseth(a)usit.uio.no> wrote: > Yes. Your concern seems to reduce to a doc bug and an > otherwise- harmless file descriptor leak which it is too late to fix > completely. So I ended up thinking of resource leaks in general. Why is it harmless in your opinion? > It'd be pretty intrusive of a _library_ to forbid the user to > fork() at all without exec(). >From my point of view, that is what the LMDB doc says. I can't use LMDB and fork(). We're probably approaching this from different angles though: I want to take something away that is de-facto possible, but was not documented to be that way. > No, I mean a partial env_close to be called _after_ the fork, in the > child. A child which does not want FD-leaks must in any case do a > close() loop or close the mdb_env_get_fd() descriptor. So maybe we > just as well should give him a more thorough "free resources in child" > call which cleans up a bit for exec()-less programs as well. > (It'll need an argument which says just what is safe to clean up - > e.g. it can't do much much if the parent is already multi-threaded.) Sorry, I misunderstood you here. If I understand you correctly, LMDB would be safe to use in these scenarios in an ideal world: 1. mdb_env_open(), fork(), exec() (without fd leaks I'd argue) 2. mdb_env_open(), pthread_create(), fork() 3. mdb_env_open(), fork() As far as I understand you are after 2 and 3, while I want 1. Case 2 seems unlikely, given that forking a multi-threaded program is so hard that is rarely makes sense [1]. Case 3 is simply a matter of reordering the calls. 1: http://thorstenball.com/blog/2014/10/13/why-threads-cant-fork/

1 0

Re: (ITS#8505) LMDB vs. fork()
by h.b.furuseth＠usit.uio.no 29 Sep '16

29 Sep '16

On 28/09/16 21:29, lmb(a)cloudflare.com wrote: > I think your proposal is going beyond what I was trying to do. Yes. Your concern seems to reduce to a doc bug and an otherwise- harmless file descriptor leak which it is too late to fix completely. So I ended up thinking of resource leaks in general. > From the documentation, it sounds like a process using LMDB can not > safely fork, under any circumstances. My idea was to specialise this > to saying that fork() followed immediately by exec() is safe, Yes. That doc bug should be fixed. > but that > a plain fork() is still not supported. If a child wants to use the > same db, it goes through the regular mdb_env_open sequence, rather > than somehow inheriting the environment across the fork. I think that > should address your pthread_exit concern, as well? It'd be pretty intrusive of a _library_ to forbid the user to fork() at all without exec(). A _program_ could specify that. So I prefer to protect the parent from pthread_exit in the child. (Note, this has nothing to do with file descriptor leaks). > On your point that a user may not call mdb_env_close before the fork: > the user might not want to, since s/he still wants to use the env in > the original process. No, I mean a partial env_close to be called _after_ the fork, in the child. A child which does not want FD-leaks must in any case do a close() loop or close the mdb_env_get_fd() descriptor. So maybe we just as well should give him a more thorough "free resources in child" call which cleans up a bit for exec()-less programs as well. (It'll need an argument which says just what is safe to clean up - e.g. it can't do much much if the parent is already multi-threaded.)

1 0

(ITS#8506) Authentication issue in ldap client machine after successfully integrating ldap in freenas OS
by caraghavvarma1＠gmail.com 29 Sep '16

29 Sep '16

Full_Name: Raghavendra varma C A Version: 3 OS: ubuntu server and freenas OS URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (123.63.153.45) Hi Team, After successfully integrating Ldap server in freenas OS(FreeBSD base OS), am able to view ldap users but i am not able to login to freenas os using ldap user credentials. Can you please let me know should i configure any additional permissions for the ldap user. And one more thing is that am not able to view home directory of ldap user. Kindly look into the this and do the needful ASAP. Thanks and Regards, Raghav-IT.

1 0

Re: (ITS#8505) LMDB vs. fork()
by lmb＠cloudflare.com 28 Sep '16

28 Sep '16

I think your proposal is going beyond what I was trying to do. >From the documentation, it sounds like a process using LMDB can not safely fork, under any circumstances. My idea was to specialise this to saying that fork() followed immediately by exec() is safe, but that a plain fork() is still not supported. If a child wants to use the same db, it goes through the regular mdb_env_open sequence, rather than somehow inheriting the environment across the fork. I think that should address your pthread_exit concern, as well? On your point that a user may not call mdb_env_close before the fork: the user might not want to, since s/he still wants to use the env in the original process.

1 0

(ITS#8504) LDMB: Return EPIPE from mdb_env_copyfd2 instead abort on SIGPIPE
by lmb＠cloudflare.com 27 Sep '16

27 Sep '16

On 26 September 2016 at 20:48, Hallvard Breien Furuseth <h.b.furuseth(a)usit.uio.no> wrote: > I think we can skip sigwait() too since the thread will exit when done. > Then any pending signals are presumably dropped. (I suppose some OS > could give them to another thread, but that goes against delivering > them to a particular thread like SIGPIPE is.) That's what my initial patch did. At least on OS X this leads to the process receiving SIGPIPE and dying. I can attach a test case as well if desired.

1 0

(ITS#8505) LMDB vs. fork()
by h.b.furuseth＠usit.uio.no 26 Sep '16

26 Sep '16

Full_Name: Hallvard B Furuseth Version: LMDB_0.9.18 OS: Linux URL: Submission from: (NULL) (81.191.45.31) Submitted by: hallvard This is from openldap-tecnical thread "Using LMDB safely with fork() and exec()" by Lorenz Bauer: <http://www.openldap.org/lists/openldap-technical/201608/msg00062.html> ==== The doc is too restrictive, it implicitly suggests an LMDB process can't fork another LDMB process (since the old LMDB would still be open). We should document the use of FD_CLOEXEC etc. ==== This breaks LMDB: if (fork() == 0) pthread_exit(NULL); because the child's me_txkey destructor releases the reader slot. This can happen if a single-threaded LMDB process forks a multi- threaded child, which is one way to deal with threads vs. fork(). Simplest fix: Check if (reader->mr_pid == getpid()) in reader_dest(). We could instead use pthread_atfork(), but there is not way to remove an atfork handler. It seems to be implementation-dependent what happens with an atfork handler if liblmdb is dynamically unloaded. ==== LMDB leaks file descriptors to a fork() child, since the user may not call mdb_env_close() and there is no way to get at some of the descriptors. Partial fix: Set FD_CLOEXEC for me_mfd and the mdb_env_copy() FD. Should I code up an mdb_env_child_cleanup() function which may be called in the child if it does not fork()? That makes sense either if the parent is single-threaded and forks a more complex child, or if the child is fairly simple. ==== Branch "mdb/fopen2" in my repo will soon include suggested fixes, except mdb_env_child_cleanup(). (Fixed version of older "mdb/fopen")

1 0

Re: (ITS#8504) LDMB: Return EPIPE from mdb_env_copyfd2 instead abort on SIGPIPE
by h.b.furuseth＠usit.uio.no 26 Sep '16

26 Sep '16

On 25/09/16 20:16, lmb(a)cloudflare.com wrote: > * Do not check for pending SIGPIPE: since this is a new thread there can't > be any I think we can skip sigwait() too since the thread will exit when done. Then any pending signals are presumably dropped. (I suppose some OS could give them to another thread, but that goes against delivering them to a particular thread like SIGPIPE is.)

1 0

Re: (ITS#8504) LDMB: Return EPIPE from mdb_env_copyfd2 instead abort on SIGPIPE
by lmb＠cloudflare.com 25 Sep '16

25 Sep '16

--94eb2c07d07472c185053d5903fe Content-Type: text/plain; charset=UTF-8 I've updated the patch to be much simpler: * Set pthread_sigmask in copythr and return an error in my->mc_error * Do not check for pending SIGPIPE: since this is a new thread there can't be any The URL stays the same. --94eb2c07d07472c185053d5903fe Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra">I've updated the patch to b= e much simpler:</div><div class=3D"gmail_extra"><br></div><div class=3D"gma= il_extra">* Set pthread_sigmask in copythr and return an error in my->mc= _error</div><div class=3D"gmail_extra">* Do not check for pending SIGPIPE: = since this is a new thread there can't be any</div><div class=3D"gmail_= extra"><br></div><div class=3D"gmail_extra">The URL stays the same.</div></= div> --94eb2c07d07472c185053d5903fe--

1 0

(ITS#8504) LDMB: Return EPIPE from mdb_env_copyfd2 instead abort on SIGPIPE
by lmb＠cloudflare.com 23 Sep '16

23 Sep '16

Full_Name: Lorenz Bauer Version: OS: URL: https://gist.github.com/lmb/63189f83c5f00dd59c86f3c9bc07694d Submission from: (NULL) (2606:4700:1000:8200:5530:fc35:2f95:4f31) As per the discussion in [1] I'm providing a patch to block and handle SIGPIPE in the copy thread used when performing a compacting copy. The attached file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by CloudFlare, Inc. CloudFlare, Inc. has not assigned rights and/or interest in this work to any party. I, Lorenz Bauer am authorized by CloudFlare, Inc., my employer, to release this work under the following terms. CloudFlare, Inc. hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice. 1: http://www.openldap.org/lists/openldap-technical/201609/msg00026.html

1 0

Re: (ITS#8475) Feature request: MDB low durability transactions
by h.b.furuseth＠usit.uio.no 16 Sep '16

16 Sep '16

I'll use this ITS to summarize details for "volatile commits". Hopefully I've managed to keep it all straight. "Volatile" vs. "durable" are the most accurate names I can think of. Not sure if that's more instructive than simply "soft" / "hard". Description: * Volatile commits omit fdatasync() without losing consistency. To survive, they must be checkpointed *before* all processes close the env. Un-checkpointed volatiles are lost when the env closes. Thus a separate checkpointing daemon can keep the env open to protect volatiles from application crash, at least if Robust locks are supported. (The lmdb.h doc seems a bit unclear about Robust.) * Checkpointing == committing a durable (non-volatile) write-txn. (If there is nothing to do, Commit writes nothing.) mdb_env_sync() will not checkpoint volatiles, since existing programs do not expect it to wait for the write mutex. It "checkpoints" MDB_NOMETASYNC/MDB_NOSYNC. Maybe mdb_checkpoint() will have a special case which obsoletes mdb_env_sync(). * Volatiles are unsupported with MDB_NOLOCK and pointless with MDB_NOSYNC. OTOH it makes sense to enable MDB_NOMETASYNC. * Volatiles need a bigger datafile, because it takes two durable commits to make a freed page reusable. (Plus awaiting old readers). Configuration. Too many options, ouch: LMDB can be configured to auto-checkpoint after X volatile commits and/or Y written kbytes(pages?). Programs can also checkpoint every Z minutes(seconds?) - configured in LMDB to mimic Berkeley DB's "checkpoint <kbytes> <minutes>", but regular LMDB ops ignore that. The lockfile gives the current config. An MDB_env could override for its particular process, e.g. with an MDB_NO_VOLATILE flag. Maybe resetting the lockfile should keep the previous config? OTOH I suppose MDB_meta can have default params the way it has mm_mapsize. That survives a backup/restore. Implementation - plain version first: * Bump MDB_LOCK_FORMAT, MDB_DATA_VERSION (or make MDB_DATA_FORMAT). * Keep 2 'MDB_meta's in the lockfile, for volatile commits. MDB_env.me_metas[] gets 4 elements: durable + volatile. * mdb_env_open() throws away volatiles if it re-inits the lockfile. * Add field MDB_meta.mm_oldest: 1 + (previous durable meta).mm_txnid in durable metas, and (previous meta).mm_oldest in volatile metas. Init 'oldest' in mdb_find_oldest() to new field MDB_env.me_oldest, which mdb_txn_renew0(write txn) sets to MDB_meta.mm_oldest. When are no volatiles, this ends up initing 'oldest' = same value as today. Usually we could just have used 1 + (oldest durable meta).mm_txnid, but a failed write_meta() may have clobbered that. * Replace MDB_txninfo.mti_txnid with mti_metaref = txnid*16 + P*4 + M: M = index to MDB_env.me_metas[], P = previous M during this session, initialized to the same as M, so we can get this info atomically. P may prove unnecessary, but it's simplest to just include it for now. For when meta M fails a checksum so we want an older meta, for mdb_mutex_failed(), maybe so we can see if there are volatiles yet. * Never use mdb_env_pick_meta() when the current metapage is known: Use it in mdb_env_open(), in mdb_mutex_failed(), and if MDB_NOLOCK. Or rather, I guess it gets a "force" param for those cases. * Add config in the lockfile. Maybe per-env config overriding it and defaults in the datafile. Txn flags "prefer volatile", "checkpoint". * Track the number of pages and volatiles since last durable commit. write_meta() compares with the config limits and makes the final decision of whether the new meta will be volatile. Add MDB_pgstate.mf_pgcount with #pages used so far. The rest goes in a lockfile array[4] indexed by mti_metaref % 4, or in MDB_meta. That way, switching to next snapshot stays atomic - just update mti_metaref. * txn_begin must verify metas, since we have no fdatasync barriers. Re-read and compare, or checksum. write_meta() and mutex_failed(): memory barrier between making a volatile meta and updating mti_metaref. Most modern compilers have that. Maybe a fallback implementation is lock;unlock an otherwise unused mutex. Should also include CACHEFLUSH(). It may make sense to have more than 2 volatile metas, so read-only txns will have more time to read a meta before it gets overwritten. MDB_WRITEMAP (and MDB_VL32?) has non-atomic issues we should deal with anyway. * We can have (durable metapage).mp_pgno == (txnid & 1) as before: mdb_txn_renew0() steps txnid by 2 instead of 1 if 'meta' is volatile. But note that the txnid doesn't say if the snapshot is durable. * "Try to checkpoint" feature, which does not await the write mutex: Trylock the write mutex in mdb_txn_begin(). If it fails, set a lockfile flag "Please checkpoint" and return. Hopefully someone will obey and clear the flag. mdb_env_commit(writer) does. Variants: * Put volatile MDB_metas in the datafile, behind the usual MDB_metas. That protects the metas from malicious/broken processes with read-only envs. Otherwise, using the lockfile (or non-file shared memory) extended read-only envs' ability to hack/break the DB. Be careful to not read volatile MDB_metas that are older than last lockfile-reset, since the reset did not clear them. That also means this variant does not enable volatiles with MDB_NOLOCK. * Stay with MDB_DATA_VERSION = 1, no change in datafile format: - Volatiles share next durable txn's txnid (for freeDB keys), but put last durable txn's txnid in mti_metaref (for mdb_find_oldest). - mdb_txn_id() / MDB_envinfo.me_last_txnid can no longer be used to distinguish txns. Apps doing that could set an env flag "ignore volatiles", or txn flag "fail if current snapshot is volatile". - Define V = 2-bit sequence number incremented by commit(writer). Include V in MDB_txninfo.mti_metaref. In volatile metas, include mm_metaref = copy of mti_metaref. This lets mutex_failed() figure out which MDB_meta is most recent: abs(V in mm_metaref - V in mti_metaref) is <= 1 whether mm_metaref or mti_metaref changes first in the thread. * Support full 32-bit txnids on 32-bit hosts. mti_metaref eats some txnid bits in order to stay atomic, but we can get them back: On 32-bit hosts, make mti_metaref a 64-bit value - something like ((txnid << 32) | R) where R = 32-bit metaref value described before. When reading mti_metaref, R is authoritative for the low txnid bits. If (txnid << 32) part does not match, adjust it so it does: That'll be + or - a small value. mti_metaref's high bits vary slowly, so this is normally "atomic". txn_renew0's loop re-reads it to verify. * We can squeeze away some bytes and mti_metaref bits, but don't get sucked into spending time on that before the rest is working. E.g. the array[4] counting pages/volatiles can be just [2] if Commit always toggles the (mti_metaref & 1) bit. And we can likely include some of the mti_metaref fields in the txnid. Roads to Hell: * Support checkpointing without waiting for the write-mutex. Programs which have used volatiles, may want this so they can exit quickly. But it requires a new shared mutex for write_meta(), or some clever tricks I've thought of which would not quite work. * Rescue volatiles in a dead lockfile when the user "knows" it's safe. Easy enough, just don't clear them when resetting the lockfile. But users screw up, and may then blame LMDB. Users who want to screw up, can run use MDB_NOSYNC instead of volatile commits. Or if volatile metas are in non-file shared memory which a system crash will kill, it's _almost_ safe to not reset them along with the lockfile. Unless someone unmounts/mounts the disk, or replaces the DB by overwriting it with another DB file, or who knows what else. -- Hallvard

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs