Hi!

If it's like that (I had asked a similar question some time ago), we would need better MDB statistics. For example some extent-based filesystem did show a histogram of free extents, ordered by the power of two sizes. However MDB is not extent-based (AFAIK), but block based. Maybe in addition to "unused blocks" (olmMDBPagesMax - olmMDBPagesUsed) and "free blocks" (olmMDBPagesFree) (I was assuming those are "100% free") there should be a number "fragmented free space in blocks" (or so).

Kind regards, Ulrich Windl

-----Original Message----- From: hubouvie@microsoft.com hubouvie@microsoft.com Sent: Thursday, July 3, 2025 3:29 AM To: openldap-technical@openldap.org Subject: [EXT] LMDB and Disk fragmentation

Hi everyone,

We’ve been stress‑testing an LMDB database by continually inserting and deleting records ranging from 4 KB to 5 MB. To track utilization, we rely on two independent metrics: Application index – we track the sum of the payload bytes of keys currently present. mdb_stat output – pages in use /overlay etc.... The two figures agree closely.

Despite this, we repeatedly hit MDB_MAP_FULL when both metrics say the map is only ≈ 50 % occupied (on both metrics). Our tentative conclusion is heavy page fragmentation: LMDB can’t find a sufficiently large contiguous run of free pages even though half of the map is nominally free. Questions

• Is it normal to lose that much effective capacity with an insert/delete

workload like ours?

• How far can we realistically fill the map before hitting fragmentation limits?
• Are there recommended ways to mitigate or “defragment”?

Any advice would be greatly appreciated!

Thanks! Hugues

hubouvie@microsoft.com wrote:

Hi everyone,

We’ve been stress‑testing an LMDB database by continually inserting and deleting records ranging from 4 KB to 5 MB. To track utilization, we rely on two independent metrics: Application index – we track the sum of the payload bytes of keys currently present. mdb_stat output – pages in use /overlay etc.... The two figures agree closely.

Despite this, we repeatedly hit MDB_MAP_FULL when both metrics say the map is only ≈ 50 % occupied (on both metrics). Our tentative conclusion is heavy page fragmentation: LMDB can’t find a sufficiently large contiguous run of free pages even though half of the map is nominally free. Questions

• Is it normal to lose that much effective capacity with an insert/delete workload like ours?

Yes. LMDB is not meant for write intensive workloads, it's for read-heavy workloads.

• How far can we realistically fill the map before hitting fragmentation limits?
• Are there recommended ways to mitigate or “defragment”?

The only way is to make a new copy of the environment, e.g. using mdb_env_copy().

Any advice would be greatly appreciated!

Thanks! Hugues