4:31 a.m.
I already posted this to the IRC channel, but there was no
response, so I repost this here.
I'm trying out lmdb from master, including the robust mutex code.
We're experiencing lock ups after the process holding the lock dies, as if
the robust lock was not recovered. I tried to come up with an lmdb example
that shows it and I got it, just a few lines. It uses fork() just to
automate it; see that the environment is opened in both children. Here's
the code:
http://pastebin.com/Cbbri6az
If I run this, I see that one of the children waits for the write
lock and is not awakened when the other child dies without closing the txn
(but notice I close the env). This is on purpose, to simulate a crashing
process.The worst part is that I can't reproduce it using directly
libpthread and mmap. Here is the code I came up with:
http://pastebin.com/ybR5L4cP
It's a little bit more verbose because I based it on a glibc test
case.
Are we missing anything? It seems to us that the code follows does
not break any of LMDB's caveats (specially the one about creating the envs
before fork()'ing. Is it wrong to assume that the waiting process should
recover the lock from staleness?
--
Marcos Dione
Astek Sud-Est
R&D-SSP-DTA-TAE-TDS
for Amadeus SAS
T: +33 (4)4 9704 1727
marcos-david.dione(a)amadeus.com
6:11 a.m.
Marcos-David Dione wrote:
I already posted this to the IRC channel, but there was no
response, so I repost this here.
... already followed up in IRC.
I'm trying out lmdb from master, including the robust
mutex
code. We're experiencing lock ups after the process holding the lock
dies, as if the robust lock was not recovered. I tried to come up with
an lmdb example that shows it and I got it, just a few lines. It uses
fork() just to automate it; see that the environment is opened in both
children. Here's the code:
http://pastebin.com/Cbbri6az
The example is broken; it does not mimic the behavior of a crashed
process. In particular it does a clean call to mdb_env_close() but
doesn't call mdb_txn_abort() first. An actual crashing process would not
make the call to mdb_env_close(), and a cleanly exiting process would
close all outstanding transactions before calling env_close.
If I run this, I see that one of the children waits for the
write lock and is not awakened when the other child dies without closing
the txn (but notice I close the env). This is on purpose, to simulate a
crashing process.The worst part is that I can't reproduce it using
directly libpthread and mmap. Here is the code I came up with:
http://pastebin.com/ybR5L4cP
It's a little bit more verbose because I based it on a glibc
test case.
Are we missing anything? It seems to us that the code follows
does not break any of LMDB's caveats (specially the one about creating
the envs before fork()'ing. Is it wrong to assume that the waiting
process should recover the lock from staleness?
env_close does an munmap of the memory containing the mutex. According
to the manpages, a robust mutex is supposed to automatically unlock when
unmapped. Since this is not happening, it appears you've found a kernel
bug. Regardless, the example is invalid. If you modify the code to just
exit/abort/die without the bogus call to env_close, the other process
wakes up correctly. E.g.
http://pastebin.com/9jieDnUz
--
Marcos Dione
Astek Sud-Est
R&D-SSP-DTA-TAE-TDS
for Amadeus SAS
T: +33 (4)4 9704 1727
marcos-david.dione(a)amadeus.com
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
2:10 a.m.
Howard Chu <hyc(a)symas.com> wrote on 23/11/2014 15:11:29:
env_close does an munmap of the memory containing the mutex.
According
to the manpages, a robust mutex is supposed to automatically unlock when
unmapped. Since this is not happening, it appears you've found a
kernel
bug. Regardless, the example is invalid. If you modify the code to just
exit/abort/die without the bogus call to env_close, the other process
wakes up correctly. E.g.
http://pastebin.com/9jieDnUz
Ok, so now I managed to mimic the situation with pure pthreads
functions.
The gist of it is as following:
1. we open a file.
2. we mmap the file and use the mem space to store a mutex.
3. we lock the mutex.
4. we unmmap the file.
5. we close the file.
Seen like that I'm not sure if there's a defined behaviour for
that. I'll
ask in the glibc and/or kernel MLs and I'll come back with the answer.
Just for the record, here's the full program:
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <pthread.h>
#include <unistd.h>
#include <sys/time.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>
void *tf (int n, int f, pthread_mutex_t *m) {
int err = pthread_mutex_lock (m);
printf ("ml: %d\n", err);
if (err == EOWNERDEAD) {
err = pthread_mutex_consistent_np (m);
printf ("mc: %d\n", err);
if (err) {
puts ("pthread_mutex_consistent_np");
exit (1);
}
} else if (err) {
puts ("pthread_mutex_lock");
exit (1);
}
printf ("%ld got the lock.\n", n);
sleep (3);
/* exit without unlock */
munmap (m, sizeof (pthread_mutex_t));
close (f);
printf ("%ld out\n", n);
return NULL;
}
int main (void) {
int err, f;
pthread_mutex_t *m;
pthread_mutexattr_t ma;
pthread_mutexattr_init (&ma);
err = pthread_mutexattr_setrobust_np (&ma, PTHREAD_MUTEX_ROBUST_NP);
if (err) {
puts ("pthread_mutexattr_setrobust_np");
return 1;
}
err = pthread_mutexattr_setpshared (&ma, PTHREAD_PROCESS_SHARED);
if (err) {
puts ("pthread_mutexattr_setpshared");
return 1;
}
#ifdef ENABLE_PI
if (pthread_mutexattr_setprotocol (&ma, PTHREAD_PRIO_INHERIT) != 0) {
puts ("pthread_mutexattr_setprotocol failed");
return 1;
}
#endif
f= open ("mutex.mmap", O_CREAT|O_TRUNC|O_RDWR);
if (f<0) {
puts ("open");
return 1;
}
err= ftruncate (f, sizeof (pthread_mutex_t));
if (err) {
puts ("ftruncate");
return 1;
}
m= (pthread_mutex_t *) mmap (NULL, sizeof (pthread_mutex_t),
PROT_READ|PROT_WRITE, MAP_SHARED, f, 0);
err = pthread_mutex_init (m, &ma);
#ifdef ENABLE_PI
if (err == ENOTSUP) {
puts ("PI robust mutexes not supported");
return 0;
}
#endif
if (err) {
puts ("pthread_mutex_init");
return 1;
}
err= fork ();
if (err==0) {
tf (1, f, m);
} else if (err>0) {
int err2= fork ();
if (err2==0) {
tf (2, f, m);
} else if (err2>0) {
// sleep (1);
// kill (err);
// printf ("child killed\n");
// sleep (10);
puts ("main out");
} else {
puts ("fork2");
return 1;
}
} else {
puts ("fork1");
return 1;
}
return 0;
}
3293
days inactive
3296
days old
2 comments
2 participants
participants (2)
-
Howard Chu -
Marcos-David Dione