Antw: [EXT] Re: MDB_MAP_FULL with plenty of free pages
>> Howard Chu <hyc(a)symas.com> schrieb am 24.03.2021 um
16:52 in Nachricht
<9d603719-4a02-1309-130f-901e63ffb209(a)symas.com>:
Michael Ströder wrote:
> On 3/24/21 4:35 PM, Howard Chu wrote:
>> Jean-Charles ROGEZ wrote:
>> This is a known limitation with regard to frequent updates of large
entries,
it causes
>> excessive fragmentation of the free page space.
>
> Does this also happen in case the LDAP client only changes a few
> attribute values of a large entry? If yes, the original poster could
> mitigate the bad effects by changing the sync process.
Yes it makes no difference since by default, back-mdb stores all entries as
single blobs. Thus the entire entry is always rewritten on any
modification.
LMDB stores all records as a single contiguous set of pages, so the
larger
the entry, the more contiguous pages it needs for storing them. Since the
free
page manager in LMDB is extremely simplistic, it doesn't do well with
random
allocations and deallocations of widely varying sizes, and so over
time the
free space fragmentation gets worse, resulting in spans of pages that are
too
small to service the requests for the larger entries.
Interestingly the BtrFS filesystem (on Linux) had similar problems in early
versions.
Today there are "reorganizing background jobs" that shuffle the blocks around
to make freespace available.
Still, if you filled your BtrFS filesystem to 100% there is no way to recover
(by removing files).
The only solution is to add more space, but I'm even unsure it works when the
filesystem is 100% full.
The traditional filesystems (like ext2) are more robust in such
circumstances.
Regards,
Ulrich
The multival feature in OpenLDAP 2.5 back-mdb lets you configure some
attributes to have their values stored as individual DB records, which
then means that modifications of individual values don't have to rewrite
the entire entry. And with smaller records, the fragmentation issue goes
away.
The tradeoff of course is that it takes multiple DB accesses to construct
an entry, so write performance improves at the cost of read performance.
There's no free lunch...
>
> (Yes, the add request with initial attribute value set is expected to be
> large. Also some subsequent modify requests in case of larger
> reorganisations.)
>
> Ciao, Michael.
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/