On Tue, Feb 28, 2023 at 16:12:25 +0100, Ondřej Kuzník wrote:
On Tue, Feb 28, 2023 at 01:42:20PM +0100, Geert Hendrickx wrote:
We've had (and still have) this issue with large attributes and large multi-valued attributes with Zimbra (see previous discussion with Quanah), where we applied sortvals and multival. But in this scenario it's not the case; all objects are of similar small size, with (mostly) single valued attributes. Yet our freelist reaches 200K+ free pages during periods with heavy updates (mostly deletes/adds), which has a measurable impact on write performance.
Hi Geert, are you sure it's the freelist and not the random access as pages become non-contiguous? The former would represent a constant decline in performance where the latter would eventually taper from high (best case) performance to regular performance you should be able to expect? Have you been able to rule that out?
mdb_copy -c fixes it, so I assume it's only the freelist size, not actual fragmentation (mdb_copy doesn't reorder any data, right?). Random access shouldn't matter much, as it's all on an SSD-based SAN.
Also, the decline isn't constant. In normal operations, the freelist stays fairly small (it is "consumed" all the time by regular updates). Only during batch updates (because of a currently ongoing migration) it explodes and doesn't get "consumed" in time for the next batch update, and causes performance degradation for subsequent batches.
After you kill accesslog, you disable deltasync. Since you're also restarting, the provider has no data on how to replay anything and needs to send the list of all entries (at least their UUIDs). This is expensive and slow. Replication seems to proceed in slow leaps that cost a *lot* of processing on the provider and a fair amount of bandwidth. Isn't that what you're seeing?
Yes, this is indeed the case and it keeps doing that as long as updates are coming in. Once there are no updates for a full refresh cycle (eg. during the night, or because we pause updates) it is able to revert to delta sync.
After you kill accesslog, you disable deltasync.
This is the essential part. I always assumed it could proceed with deltasync of the provider and replica have the same contextCSN, even with an empty accesslog.
This probably went un-noticed for a long time since dropping the accesslog on a non-active master causes no (visible) delays. Only on an active master.
Thanks for your insights, things are much clearer now, and we have adjusted our processes accordingly.
Geert