On Sun, Feb 13, 2022 at 08:00:29PM -0800, Paul B. Henson wrote:
I'm still trying to figure out why my servers sometimes get into a state where queries requesting the memberOf attribute take an exceedingly long
So one of my servers got into this state again:
Total DISK READ : 89.60 M/s | Total DISK WRITE : 241.97 K/s [0/9406] Actual DISK READ: 91.61 M/s | Actual DISK WRITE: 140.50 K/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 430373 be/4 ldap 34.88 M/s 0.00 B/s 0.00 % 96.44 % slapd -d 0 -h ldap:/// ~dapi:/// -u ldap -g ldap 430064 be/4 ldap 45.04 M/s 0.00 B/s 0.00 % 94.93 % slapd -d 0 -h ldap:/// ~dapi:/// -u ldap -g ldap 430069 be/4 ldap 6.34 M/s 0.00 B/s 0.00 % 8.22 % slapd -d 0 -h ldap:/// ~dapi:/// -u ldap -g ldap
There's plenty of free memory:
ldap-02 ~ # free -m total used free shared buff/cache available Mem: 3901 418 113 0 3368 3255 Swap: 2047 763 1284
Just for giggles, I removed all swap:
total used free shared buff/cache available Mem: 3901 730 102 1 3068 2949 Swap: 0 0 0
The problem immediately went away. Didn't restart slapd, didn't do anything, other than remove all swap and force it to use the plethora of memory it had available. Disk read went back to virtually 0, response time went back to subsecond.
I updated vm.swappiness to 1 (it defaulted to 30) and added swap back, I'm going to see what happens.
Have no idea what's causing it, but it seems somehow the system and slapd get into a state where doing memberof queries make it read lmdb pages from disk rather than keeping them in memory?