openldap-bugs September 2016

openldap-bugs@openldap.org

17 participants
49 discussions

Re: (ITS#8505) LMDB vs. fork()
by h.b.furuseth＠usit.uio.no 29 Sep '16

29 Sep '16

I wrote: >On 29. sep. 2016 18:27, lmb(a)cloudflare.com wrote: >> If I understand you correctly, LMDB would be safe to use in these >> scenarios in an ideal world: >> >> 1. mdb_env_open(), fork(), exec() (without fd leaks I'd argue) >> 2. mdb_env_open(), pthread_create(), fork() >> 3. mdb_env_open(), fork() >> >> As far as I understand you are after 2 and 3, while I want 1. > > I'm after all three. ...if the user calls the cleanup function. But: Whoops, correction: If you mean using LMDB after fork(), (2) can't be safe unless the user makes it safe with pthread_atfork(). But at least it can clean up some of the stuff (3) also does, and he'll have the pthread_atfork option if he really needs it.

1 0

Re: (ITS#8505) LMDB vs. fork()
by h.b.furuseth＠usit.uio.no 29 Sep '16

29 Sep '16

On 29. sep. 2016 18:27, lmb(a)cloudflare.com wrote: > <h.b.furuseth(a)usit.uio.no> wrote: > >> Yes. Your concern seems to reduce to a doc bug and an >> otherwise-harmless file descriptor leak which it is too late to fix >> completely. So I ended up thinking of resource leaks in general. > > Why is it harmless in your opinion? Because I wrote that documentation, and now I know what it gets wrong:-) Only the lockfile descriptor is dangerous. Therefore it already had FD_CLOEXEC. The other FD leaks are just leaks, they can't break LMDB. >> It'd be pretty intrusive of a _library_ to forbid the user to >> fork() at all without exec(). > > From my point of view, that is what the LMDB doc says. Yes. And that doc is half wrong, so I'll fix it. And it's half right with pthread_exit(), so I want to fix LMDB about that. > If I understand you correctly, LMDB would be safe to use in these > scenarios in an ideal world: > > 1. mdb_env_open(), fork(), exec() (without fd leaks I'd argue) > 2. mdb_env_open(), pthread_create(), fork() > 3. mdb_env_open(), fork() > > As far as I understand you are after 2 and 3, while I want 1. I'm after all three. (1) is already safe. The mdb_env_get_fd() descriptor will leak since otherwise programs using it could break. But you can close() it yourself in the child. Well, or mdb_env_open() could take an MDB_CLOEXEC flag which says to set FD_CLOEXEC on that FD too. And since the child may need to do something to avoid leaks anyway, I figure we might as well provide a cleanup function do do it properly. So... > Case 2 > seems unlikely, given that forking a multi-threaded program is so hard > that is rarely makes sense [1]. It's very limited what you can do in the child, yes. But sometimes if you have do then you have to. E.g. if the exec()ed program may not exist, and the child has do some simple task as a fallback. Or if you're not free to rewrite the main program, only some module. Case 2 will at least have memory leaks, since free() in the child is unsafe. But we can get rid of some OS resources (FDs, memory map, maybe semaphores, need to check about that.) > Case 3 is simply a matter of reordering the calls. Only if the program logic allows it. Cumbersome e.g. if the LMDB database contains the config controlling the child. -- Hallvard

1 0

(ITS#8507) adding the syncprov overlay does not load the associated schema elements
by elecharny＠apache.org 29 Sep '16

29 Sep '16

Full_Name: Emmanuel L.charny Version: 2.4.44 OS: Linux CentOS 6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (176.154.3.51) Start an OpenLDAP server using the dynamic configuration (slapd.d) from a slapd.conf file that does not contain a synprov module : $ /opt/symas/bin/slaptest -f /opt/symas/etc/openldap/slapd.conf -F /opt/symas/etc/openldap/slapd.d $ /etc/init.d/solserver start -d 256 $ grep -ri "olcSpNoPresent" slapd.d $ ... Then inject the syncprov module in the server : dn: cn=module{0},cn=config changetype: modify add: olcModuleLoad olcModuleLoad: syncprov.la - The olcSpNoPresent attribute is not present : $ grep -ri "olcSpNoPresent" slapd.d $ ... You can stop and restart the server, the attribute is still not present. Going back to the slapd.conf file, add the module : ... # Uncomment the mululeloads as needed to enable additional # functionalityi when configured. NOTE: We package many # more modules options than those found below. moduleload back_mdb.la moduleload back_monitor.la moduleload syncprov.la # Added module ... and regenerate the slapd.d content : $ /etc/init.d/solserver stop $ rm -rf /opt/symas/etc/openldap/slapd.d/* $ /opt/symas/bin/slaptest -f /opt/symas/etc/openldap/slapd.conf -F /opt/symas/etc/openldap/slapd.d $ grep -ri "olcSpNoPresent" /opt/symas/etc/openldap/slapd.d slapd.d/cn=config/cn=schema.ldif:olcAttributeTypes: ( OLcfgOvAt:1.3 NAME 'olcSpNoPresent' DESC 'Omit Present slapd.d/cn=config/cn=schema.ldif: $ olcSpSessionlog $ olcSpNoPresent $ olcSpReloadHint ) ) $... This is problematic...

1 0

Re: (ITS#8505) LMDB vs. fork()
by lmb＠cloudflare.com 29 Sep '16

29 Sep '16

On 29 September 2016 at 03:39, Hallvard Breien Furuseth <h.b.furuseth(a)usit.uio.no> wrote: > Yes. Your concern seems to reduce to a doc bug and an > otherwise- harmless file descriptor leak which it is too late to fix > completely. So I ended up thinking of resource leaks in general. Why is it harmless in your opinion? > It'd be pretty intrusive of a _library_ to forbid the user to > fork() at all without exec(). >From my point of view, that is what the LMDB doc says. I can't use LMDB and fork(). We're probably approaching this from different angles though: I want to take something away that is de-facto possible, but was not documented to be that way. > No, I mean a partial env_close to be called _after_ the fork, in the > child. A child which does not want FD-leaks must in any case do a > close() loop or close the mdb_env_get_fd() descriptor. So maybe we > just as well should give him a more thorough "free resources in child" > call which cleans up a bit for exec()-less programs as well. > (It'll need an argument which says just what is safe to clean up - > e.g. it can't do much much if the parent is already multi-threaded.) Sorry, I misunderstood you here. If I understand you correctly, LMDB would be safe to use in these scenarios in an ideal world: 1. mdb_env_open(), fork(), exec() (without fd leaks I'd argue) 2. mdb_env_open(), pthread_create(), fork() 3. mdb_env_open(), fork() As far as I understand you are after 2 and 3, while I want 1. Case 2 seems unlikely, given that forking a multi-threaded program is so hard that is rarely makes sense [1]. Case 3 is simply a matter of reordering the calls. 1: http://thorstenball.com/blog/2014/10/13/why-threads-cant-fork/

1 0

Re: (ITS#8505) LMDB vs. fork()
by h.b.furuseth＠usit.uio.no 29 Sep '16

29 Sep '16

On 28/09/16 21:29, lmb(a)cloudflare.com wrote: > I think your proposal is going beyond what I was trying to do. Yes. Your concern seems to reduce to a doc bug and an otherwise- harmless file descriptor leak which it is too late to fix completely. So I ended up thinking of resource leaks in general. > From the documentation, it sounds like a process using LMDB can not > safely fork, under any circumstances. My idea was to specialise this > to saying that fork() followed immediately by exec() is safe, Yes. That doc bug should be fixed. > but that > a plain fork() is still not supported. If a child wants to use the > same db, it goes through the regular mdb_env_open sequence, rather > than somehow inheriting the environment across the fork. I think that > should address your pthread_exit concern, as well? It'd be pretty intrusive of a _library_ to forbid the user to fork() at all without exec(). A _program_ could specify that. So I prefer to protect the parent from pthread_exit in the child. (Note, this has nothing to do with file descriptor leaks). > On your point that a user may not call mdb_env_close before the fork: > the user might not want to, since s/he still wants to use the env in > the original process. No, I mean a partial env_close to be called _after_ the fork, in the child. A child which does not want FD-leaks must in any case do a close() loop or close the mdb_env_get_fd() descriptor. So maybe we just as well should give him a more thorough "free resources in child" call which cleans up a bit for exec()-less programs as well. (It'll need an argument which says just what is safe to clean up - e.g. it can't do much much if the parent is already multi-threaded.)

1 0

(ITS#8506) Authentication issue in ldap client machine after successfully integrating ldap in freenas OS
by caraghavvarma1＠gmail.com 29 Sep '16

29 Sep '16

Full_Name: Raghavendra varma C A Version: 3 OS: ubuntu server and freenas OS URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (123.63.153.45) Hi Team, After successfully integrating Ldap server in freenas OS(FreeBSD base OS), am able to view ldap users but i am not able to login to freenas os using ldap user credentials. Can you please let me know should i configure any additional permissions for the ldap user. And one more thing is that am not able to view home directory of ldap user. Kindly look into the this and do the needful ASAP. Thanks and Regards, Raghav-IT.

1 0

Re: (ITS#8505) LMDB vs. fork()
by lmb＠cloudflare.com 28 Sep '16

28 Sep '16

I think your proposal is going beyond what I was trying to do. >From the documentation, it sounds like a process using LMDB can not safely fork, under any circumstances. My idea was to specialise this to saying that fork() followed immediately by exec() is safe, but that a plain fork() is still not supported. If a child wants to use the same db, it goes through the regular mdb_env_open sequence, rather than somehow inheriting the environment across the fork. I think that should address your pthread_exit concern, as well? On your point that a user may not call mdb_env_close before the fork: the user might not want to, since s/he still wants to use the env in the original process.

1 0

(ITS#8504) LDMB: Return EPIPE from mdb_env_copyfd2 instead abort on SIGPIPE
by lmb＠cloudflare.com 27 Sep '16

27 Sep '16

On 26 September 2016 at 20:48, Hallvard Breien Furuseth <h.b.furuseth(a)usit.uio.no> wrote: > I think we can skip sigwait() too since the thread will exit when done. > Then any pending signals are presumably dropped. (I suppose some OS > could give them to another thread, but that goes against delivering > them to a particular thread like SIGPIPE is.) That's what my initial patch did. At least on OS X this leads to the process receiving SIGPIPE and dying. I can attach a test case as well if desired.

1 0

(ITS#8505) LMDB vs. fork()
by h.b.furuseth＠usit.uio.no 26 Sep '16

26 Sep '16

Full_Name: Hallvard B Furuseth Version: LMDB_0.9.18 OS: Linux URL: Submission from: (NULL) (81.191.45.31) Submitted by: hallvard This is from openldap-tecnical thread "Using LMDB safely with fork() and exec()" by Lorenz Bauer: <http://www.openldap.org/lists/openldap-technical/201608/msg00062.html> ==== The doc is too restrictive, it implicitly suggests an LMDB process can't fork another LDMB process (since the old LMDB would still be open). We should document the use of FD_CLOEXEC etc. ==== This breaks LMDB: if (fork() == 0) pthread_exit(NULL); because the child's me_txkey destructor releases the reader slot. This can happen if a single-threaded LMDB process forks a multi- threaded child, which is one way to deal with threads vs. fork(). Simplest fix: Check if (reader->mr_pid == getpid()) in reader_dest(). We could instead use pthread_atfork(), but there is not way to remove an atfork handler. It seems to be implementation-dependent what happens with an atfork handler if liblmdb is dynamically unloaded. ==== LMDB leaks file descriptors to a fork() child, since the user may not call mdb_env_close() and there is no way to get at some of the descriptors. Partial fix: Set FD_CLOEXEC for me_mfd and the mdb_env_copy() FD. Should I code up an mdb_env_child_cleanup() function which may be called in the child if it does not fork()? That makes sense either if the parent is single-threaded and forks a more complex child, or if the child is fairly simple. ==== Branch "mdb/fopen2" in my repo will soon include suggested fixes, except mdb_env_child_cleanup(). (Fixed version of older "mdb/fopen")

1 0

Re: (ITS#8504) LDMB: Return EPIPE from mdb_env_copyfd2 instead abort on SIGPIPE
by h.b.furuseth＠usit.uio.no 26 Sep '16

26 Sep '16

On 25/09/16 20:16, lmb(a)cloudflare.com wrote: > * Do not check for pending SIGPIPE: since this is a new thread there can't > be any I think we can skip sigwait() too since the thread will exit when done. Then any pending signals are presumably dropped. (I suppose some OS could give them to another thread, but that goes against delivering them to a particular thread like SIGPIPE is.)

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs September 2016