Kris Zyp wrote:
In LMDB, if you attempt to open more than 10 transactions (on
different environments) on one process on MacOS, then mdb_txn_begin will fail. This can be
reproduced by opening 11 different database environments in one process, and calling
mdb_txn_begin (as write transactions) on each one. The 11th one will return EINVAL. I
believe this is because (on this OS, MacOS 10.13.6) the System V semaphores have a limit
of 10 SEM_UNDO locked semaphores on one process. So when mdb_txn_begin attempts to open an
11th transaction, the mdb_sem_wait/semop call fails, returning EINVAL. I was testing with
latest LMDB from mdb.master.
Once I finally debugged this and figured out the issue, I have been able to work around
it by compiling with MDB_USE_POSIX_SEM which seems to resolve the issue. But this still
leaves a few questions:
Is there any issue with compiling with MDB_USE_POSIX_SEM on MacOS? Would it be better if
LMDB defaulted to this shared lock implementation for this OS?
When we wrote the MacOS support there was no confirmation that POSIX semaphores worked on
that OS at the time.
This limit in number of semaphores should be a tunable kernel parameter or ulimit setting,
I'd look there first.
And/or would there be value in LMDB providing a more specific error message in this
situation? The documentation doesn't indicate EINVAL as a possible return value for
mdb_txn_begin, and this is a very generic error with little indication of the root
problem, that was rather confusing to me, at least. Or is there an expectation that
processes can only open a limited number of database environments (and have open
transactions on them)? (Our server typically has about 30 environments open with about
2-16 dbs/env, with many concurrent transactions, without issue on other OSes.)
I would be happy to put together a patch for this, but I am not sure of the reasons for
the selection of the different shared lock implementations on different OSes. Anyway,
I'd be glad to help contribute a patch if there is a specific way this should work.
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/