I already posted this to the IRC channel, but there was no response, so I repost this here.
I'm trying out lmdb from master, including the robust mutex code. We're experiencing lock ups after the process holding the lock dies, as if the robust lock was not recovered. I tried to come up with an lmdb example that shows it and I got it, just a few lines. It uses fork() just to automate it; see that the environment is opened in both children. Here's the code:
If I run this, I see that one of the children waits for the write lock and is not awakened when the other child dies without closing the txn (but notice I close the env). This is on purpose, to simulate a crashing process.The worst part is that I can't reproduce it using directly libpthread and mmap. Here is the code I came up with:
It's a little bit more verbose because I based it on a glibc test case.
Are we missing anything? It seems to us that the code follows does not break any of LMDB's caveats (specially the one about creating the envs before fork()'ing. Is it wrong to assume that the waiting process should recover the lock from staleness?
-- Marcos Dione Astek Sud-Est R&D-SSP-DTA-TAE-TDS for Amadeus SAS T: +33 (4)4 9704 1727 marcos-david.dione@amadeus.com
Marcos-David Dione wrote:
I already posted this to the IRC channel, but there was noresponse, so I repost this here.
... already followed up in IRC.
I'm trying out lmdb from master, including the robust mutexcode. We're experiencing lock ups after the process holding the lock dies, as if the robust lock was not recovered. I tried to come up with an lmdb example that shows it and I got it, just a few lines. It uses fork() just to automate it; see that the environment is opened in both children. Here's the code:
The example is broken; it does not mimic the behavior of a crashed process. In particular it does a clean call to mdb_env_close() but doesn't call mdb_txn_abort() first. An actual crashing process would not make the call to mdb_env_close(), and a cleanly exiting process would close all outstanding transactions before calling env_close.
If I run this, I see that one of the children waits for thewrite lock and is not awakened when the other child dies without closing the txn (but notice I close the env). This is on purpose, to simulate a crashing process.The worst part is that I can't reproduce it using directly libpthread and mmap. Here is the code I came up with:
It's a little bit more verbose because I based it on a glibctest case.
Are we missing anything? It seems to us that the code followsdoes not break any of LMDB's caveats (specially the one about creating the envs before fork()'ing. Is it wrong to assume that the waiting process should recover the lock from staleness?
env_close does an munmap of the memory containing the mutex. According to the manpages, a robust mutex is supposed to automatically unlock when unmapped. Since this is not happening, it appears you've found a kernel bug. Regardless, the example is invalid. If you modify the code to just exit/abort/die without the bogus call to env_close, the other process wakes up correctly. E.g.
-- Marcos Dione Astek Sud-Est R&D-SSP-DTA-TAE-TDS for Amadeus SAS T: +33 (4)4 9704 1727 marcos-david.dione@amadeus.com
Howard Chu hyc@symas.com wrote on 23/11/2014 15:11:29:
env_close does an munmap of the memory containing the mutex. According to the manpages, a robust mutex is supposed to automatically unlock when
unmapped. Since this is not happening, it appears you've found a kernel bug. Regardless, the example is invalid. If you modify the code to just exit/abort/die without the bogus call to env_close, the other process wakes up correctly. E.g.
Ok, so now I managed to mimic the situation with pure pthreads functions. The gist of it is as following:
1. we open a file. 2. we mmap the file and use the mem space to store a mutex. 3. we lock the mutex. 4. we unmmap the file. 5. we close the file.
Seen like that I'm not sure if there's a defined behaviour for that. I'll ask in the glibc and/or kernel MLs and I'll come back with the answer.
Just for the record, here's the full program:
#include <stdio.h> #include <stdlib.h> #include <errno.h> #include <pthread.h> #include <unistd.h> #include <sys/time.h> #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> #include <sys/mman.h>
void *tf (int n, int f, pthread_mutex_t *m) { int err = pthread_mutex_lock (m); printf ("ml: %d\n", err); if (err == EOWNERDEAD) { err = pthread_mutex_consistent_np (m); printf ("mc: %d\n", err); if (err) { puts ("pthread_mutex_consistent_np"); exit (1); } } else if (err) { puts ("pthread_mutex_lock"); exit (1); } printf ("%ld got the lock.\n", n); sleep (3); /* exit without unlock */ munmap (m, sizeof (pthread_mutex_t)); close (f); printf ("%ld out\n", n); return NULL; }
int main (void) { int err, f; pthread_mutex_t *m; pthread_mutexattr_t ma;
pthread_mutexattr_init (&ma); err = pthread_mutexattr_setrobust_np (&ma, PTHREAD_MUTEX_ROBUST_NP); if (err) { puts ("pthread_mutexattr_setrobust_np"); return 1; } err = pthread_mutexattr_setpshared (&ma, PTHREAD_PROCESS_SHARED); if (err) { puts ("pthread_mutexattr_setpshared"); return 1; } #ifdef ENABLE_PI if (pthread_mutexattr_setprotocol (&ma, PTHREAD_PRIO_INHERIT) != 0) { puts ("pthread_mutexattr_setprotocol failed"); return 1; } #endif
f= open ("mutex.mmap", O_CREAT|O_TRUNC|O_RDWR); if (f<0) { puts ("open"); return 1; }
err= ftruncate (f, sizeof (pthread_mutex_t)); if (err) { puts ("ftruncate"); return 1; }
m= (pthread_mutex_t *) mmap (NULL, sizeof (pthread_mutex_t), PROT_READ|PROT_WRITE, MAP_SHARED, f, 0);
err = pthread_mutex_init (m, &ma); #ifdef ENABLE_PI if (err == ENOTSUP) { puts ("PI robust mutexes not supported"); return 0; } #endif if (err) { puts ("pthread_mutex_init"); return 1; }
err= fork (); if (err==0) { tf (1, f, m); } else if (err>0) { int err2= fork (); if (err2==0) { tf (2, f, m); } else if (err2>0) { // sleep (1); // kill (err); // printf ("child killed\n"); // sleep (10); puts ("main out"); } else { puts ("fork2"); return 1; } } else { puts ("fork1"); return 1; } return 0; }
-
Howard Chu -
Marcos-David Dione