I've run into another problem with the memberOf implementation on my 2.5 servers. After I sorted out the proper configuration, queries requesting memberOf were very performant:
Feb 4 13:26:44 ldap-01 slapd[1207]: conn=23393 op=1 SRCH base="ou=user,dc=cpp,dc=edu" scope=2 deref=3 filter="(&(objectClass=person)(calstateEduPersonEmplID=013522522))" Feb 4 13:26:44 ldap-01 slapd[1207]: conn=23393 op=1 SRCH attr=memberOf Feb 4 13:26:44 ldap-01 slapd[1207]: conn=23393 op=1 SEARCH RESULT tag=101 err=0 qtime=0.000015 etime=0.191860 nentries=1 text=
However, intermittently the server gets into a state where the exact same query takes over 30 seconds:
Feb 4 08:05:11 ldap-01 slapd[1425456]: conn=40797 op=1 SRCH base="ou=user,dc=cpp,dc=edu" scope=2 deref=3 filter="(&(objectClass=person)(calstateEduPersonEmplID=015559557))" Feb 4 08:05:11 ldap-01 slapd[1425456]: conn=40797 op=1 SRCH attr=memberOf Feb 4 08:05:50 ldap-01 slapd[1425456]: conn=40797 op=1 SEARCH RESULT tag=101 err=0 qtime=0.000019 etime=39.435523 nentries=1 text=
When this occurs, the only way to resolve the issue that I have found is to reboot the server. Simply restarting slapd results in the same degraded performance on these queries.
Normally there is very low read I/O load on the servers during operation, probably averaging less than 1M/s, peaking up to maybe 20-30M/s for just an instant occasionally. When the memberOf query performance is degraded, there is a very high read I/O load on the server, continuously about 200-300M/s.
Any thoughts on this? It seems like for some reason the server gets into a state where it is not using the cache or memory map for doing the search required to construct the memberOf results? But instead is doing a full disk read of the entire database?
It's also weird that restarting the service is not resolve this, but rebooting the server does. I'm not intimately familiar with the internals of lmdb, is there some state that persists with the environment or memory map in between service runs that is only cleared by a reboot?
I initially thought I might have had a theory on it, relating to an unrelated bug in RHEL 8.5 that broke the "needs-rebooting" command resulting in servers not properly rebooting after kernel/library updates. The most recent occurrence of this issue started up after such an update without the required reboot, but upon reviewing historic occurrences it has occurred at times that don't meet that criteria, so I find myself clueless again as to what's going on.
Any advice on how to fix or do further debugging on this issue much appreciated, thanks…