[Issue 10463] New: Windows write lock (me_wmutex) is recursive, allowing concurrent write txns and heap corruption

https://bugs.openldap.org/show_bug.cgi?id=10463

Issue ID: 10463 Summary: Windows write lock (me_wmutex) is recursive, allowing concurrent write txns and heap corruption Product: LMDB Version: 0.9.35 Hardware: x86_64 OS: Windows Status: UNCONFIRMED Keywords: needs_review Severity: normal Priority: --- Component: liblmdb Assignee: bugs@openldap.org Reporter: github@nicwatson.org Target Milestone: ---

On Windows, LMDB uses CreateMutexA for the write lock (me_wmutex) and reader lock (me_rmutex). Windows Mutex objects are inherently recursive: the owning thread can re-acquire the same mutex without blocking. This allows a single thread to open two write transactions concurrently, violating the single-writer invariant and corrupting the B-tree. The process crashes with `0xC0000374 STATUS_HEAP_CORRUPTION` on mdb_env_close or the next sync.

On POSIX, pthread_mutex_lock on a default (non-recursive) mutex from the owning thread is undefined behavior (typically deadlocks), so this is not observable there.

## Steps to reproduce

1. Build the following program with MSVC: `cl /I libraries/liblmdb repro.c libraries/liblmdb/mdb.c libraries/liblmdb/midl.c Advapi32.lib`

2. Run repro.exe.

```c #include <stdio.h> #include <stdlib.h> #include <direct.h> #include "lmdb.h"

int main(void) { MDB_env *env; MDB_txn *txn1, *txn2; MDB_dbi dbi1, dbi2; MDB_val key, val; int rc;

mdb_env_create(&env); mdb_env_set_mapsize(env, 1024ULL * 1024 * 1024); _mkdir("testdb"); mdb_env_open(env, "testdb", 0, 0664);

rc = mdb_txn_begin(env, NULL, 0, &txn1); printf("First write txn: rc=%d\n", rc);

rc = mdb_txn_begin(env, NULL, 0, &txn2); printf("Second write txn: rc=%d\n", rc);

mdb_dbi_open(txn1, NULL, 0, &dbi1); mdb_dbi_open(txn2, NULL, 0, &dbi2); key.mv_size = 4; val.mv_size = 4; key.mv_data = "key1"; val.mv_data = "val1"; mdb_put(txn1, dbi1, &key, &val, 0); key.mv_data = "key2"; val.mv_data = "val2"; mdb_put(txn2, dbi2, &key, &val, 0);

mdb_txn_commit(txn2); mdb_txn_abort(txn1); mdb_env_close(env);

return 0; } ```

## Actual results

Both mdb_txn_begin calls return MDB_SUCCESS. The process crashes with `0xC0000374 STATUS_HEAP_CORRUPTION` during mdb_env_close.

## Expected results

The second mdb_txn_begin should block (matching POSIX behavior) or return an error. Two write transactions must never coexist.

## Analysis

In mdb.c, the Windows `#ifdef _WIN32` block defines:

```c #define LOCK_MUTEX0(mutex) WaitForSingleObject(mutex, INFINITE) #define UNLOCK_MUTEX(mutex) ReleaseMutex(mutex) ```

The mutexes are created with CreateMutexA. Per Microsoft documentation: "After a thread obtains ownership of a mutex, it can specify the same mutex in repeated wait function calls without blocking its execution." This means LOCK_MUTEX0(env->me_wmutex) in mdb_txn_begin succeeds even when the calling thread already holds it.

## Proposed fix

Replace Windows Mutex objects with Semaphore objects (initial count = 1, max count = 1). Semaphores are not recursive: if the calling thread calls WaitForSingleObject when the count is 0, it blocks. This matches the POSIX pthread_mutex_lock behavior.

```c /* Unlock macros: */ -#define pthread_mutex_unlock(x) ReleaseMutex(*x) -#define UNLOCK_MUTEX(mutex) ReleaseMutex(mutex) +#define pthread_mutex_unlock(x) ReleaseSemaphore(*x, 1, NULL) +#define UNLOCK_MUTEX(mutex) ReleaseSemaphore(mutex, 1, NULL)

/* Lock creation (mdb_env_setup_locks): */ -env->me_rmutex = CreateMutexA(&mdb_all_sa, FALSE, name); +env->me_rmutex = CreateSemaphoreA(&mdb_all_sa, 1, 1, name); -env->me_wmutex = CreateMutexA(&mdb_all_sa, FALSE, name); +env->me_wmutex = CreateSemaphoreA(&mdb_all_sa, 1, 1, name);

/* Lock opening (existing env): */ -env->me_rmutex = OpenMutexA(SYNCHRONIZE, FALSE, name); +env->me_rmutex = OpenSemaphoreA(SYNCHRONIZE|SEMAPHORE_MODIFY_STATE, FALSE, name); -env->me_wmutex = OpenMutexA(SYNCHRONIZE, FALSE, name); +env->me_wmutex = OpenSemaphoreA(SYNCHRONIZE|SEMAPHORE_MODIFY_STATE, FALSE, name);

/* Copy thread mutex (mdb_env_copyfd1): */ -my.mc_mutex = CreateMutex(NULL, FALSE, NULL) +my.mc_mutex = CreateSemaphore(NULL, 1, 1, NULL) ```

No changes needed for LOCK_MUTEX0 (WaitForSingleObject works on both types), CloseHandle (works on both), or SignalObjectAndWait in pthread_cond_wait (signals a semaphore the same as a mutex). `SEMAPHORE_MODIFY_STATE` is required on OpenSemaphoreA so ReleaseSemaphore is permitted.

## Build information

- Windows 11, MSVC 14.44 (Visual Studio 2022 Build Tools), x86-64 - Python 3.13.2 (MSC v.1942 64-bit) — original report via py-lmdb binding - Confirmed not observable on Linux (deadlocks instead of corrupting)