In LMDB, if you attempt to open more than 10 transactions (on different environments) on one process on MacOS, then mdb_txn_begin will fail. This can be reproduced by opening 11 different database environments in one process, and calling mdb_txn_begin (as write transactions) on each one. The 11th one will return EINVAL. I believe this is because (on this OS, MacOS 10.13.6) the System V semaphores have a limit of 10 SEM_UNDO locked semaphores on one process. So when mdb_txn_begin attempts to open an 11th transaction, the mdb_sem_wait/semop call fails, returning EINVAL. I was testing with latest LMDB from mdb.master.
Once I finally debugged this and figured out the issue, I have been able to work around it by compiling with MDB_USE_POSIX_SEM which seems to resolve the issue. But this still leaves a few questions:
Is there any issue with compiling with MDB_USE_POSIX_SEM on MacOS? Would it be better if LMDB defaulted to this shared lock implementation for this OS?
And/or would there be value in LMDB providing a more specific error message in this situation? The documentation doesn't indicate EINVAL as a possible return value for mdb_txn_begin, and this is a very generic error with little indication of the root problem, that was rather confusing to me, at least. Or is there an expectation that processes can only open a limited number of database environments (and have open transactions on them)? (Our server typically has about 30 environments open with about 2-16 dbs/env, with many concurrent transactions, without issue on other OSes.)
I would be happy to put together a patch for this, but I am not sure of the reasons for the selection of the different shared lock implementations on different OSes. Anyway, I'd be glad to help contribute a patch if there is a specific way this should work. Thank you!