On 2/16/2022 3:08 AM, Ulrich Windl wrote:
Remember there are some classic tools like sar, vmstat, iostat, etc. to display or store some interesting information about what the OS is
Like I mentioned, the OS doesn't appear to be doing anything interesting when this has occurred. There's plenty of free memory, there is no swap paging, and CPU utilization isn't particularly high.
iotop is another nice utility.
Yes, I was using that; the only abnormal behavior when the problem occurred was extremely high read IO by the slapd process. Normally, read IO is virtually nil, jumping up to a MB/s or two occasionally. When the memberOf query is taking excessive time, there is like continuous 200-400 MB/s read IO being performed by slapd. Despite the high read IO, all queries that don't involve memberOf continue to operate with usual speed. Just the queries that return memberOf are degraded.
That's why it feels to me like an issue somehow with slapd, not the OS.
You didn't tell what hypervisor you are using, bu tdid you know that Xen PVMs support "memory hotplugging"?
We are using VMware ESXi, which also supports memory hot plugging. Although they do something weird where if you have less than 3G of RAM allocated they only let you hot plug up to 3G. If you have more than 3G allocated, you can hot plug up to the max. There is some knowledge base article about it, ah, here it is:
https://kb.vmware.com/s/article/2008405
It claims that linux "freezes" if you increase from less than 3G to more than 3G? I can believe that maybe that occurred once upon a time due to a bug, but find it hard to believe that it still does. It seems like VMware implemented a kludge workaround and has never removed it...
Yup. I just bumped the memory from 2GB to 8GB on an instance running under kvm, no problem. Stupid vmware. Hmm, nifty, kvm even lets you dynamically remove memory, another thing VMware doesn't let you do.
Thanks…