So I upgraded one of my test systems to 2.5, from 2.4. Doing a quick basic functionality check after the upgrade, I noticed that the performance on the upgraded system was significantly slower.
When the system was running 2.4, searching for my entry after a cold start took a little more than a second:
# time ldapsearch -x -b dc=cpp,dc=edu -H ldap://localhost/ uid=henson [...] dn: uid=henson,ou=user,dc=cpp,dc=edu [...] real 0m1.274s
and then subsequent searches generally less than 1/10th of a second:
real 0m0.010s real 0m0.009s real 0m0.007s real 0m0.006s real 0m0.010s real 0m0.008s
After upgrading to 2.5, searches, both cold start and subsequent, all took between 1.5-2 seconds:
real 0m1.513s real 0m1.523s real 0m1.308s real 0m1.874s real 0m2.150s real 0m1.406s
These are test systems, so really have no load. I noticed on the upgraded box that the CPU kept spiking to 100% every few seconds, and tracked it down to the load balancer health checks, which basically run:
# time /opt/symas/bin/ldapsearch -x -b dc=cpp,dc=edu '(objectClass=dcObject)'
On the upgraded system, this again takes about 1.5 seconds on every search:
real 0m1.483s
whereas on the 2.4 system, less than 1/10th of a second:
real 0m0.006s
It feels like something weird is going on with indexing. I turned on debugging, and this is what the index activity looked like on the 2.4 system:
Oct 21 15:27:06 ldap-dev-02 slapd[73033]: <= mdb_index_read: failed (-30798) Oct 21 15:27:06 ldap-dev-02 slapd[73033]: <= mdb_index_read: failed (-30798) Oct 21 15:27:12 ldap-dev-02 slapd[73033]: <= mdb_index_read: failed (-30798) Oct 21 15:27:12 ldap-dev-02 slapd[73033]: <= mdb_index_read 1 candidates
Which makes sense, there's only one of me :). On the other hand, on the upgraded system:
Oct 21 15:25:55 ldap-dev-03 slapd[42259]: <= mdb_index_read: failed (-30798) Oct 21 15:25:55 ldap-dev-03 slapd[42259]: <= mdb_index_read: failed (-30798) Oct 21 15:26:19 ldap-dev-03 slapd[42259]: <= mdb_index_read 132354 candidates
It looks like it's doing a full directory search for my entry? My configuration on the two systems is exactly the same, other than the changed paths for the 2.5 directory layout, removing the password policy schema include, and removing a pres index on automountInformation for the upgrade advice.
I have tested this both with using the existing binary mdb database files from the 2.4 system as well as dumping them out via slapcat, deleting the old ones, and creating new ones with the 2.5 version of slapadd, with the exact same behavior
Any thoughts on what might be going on or how I can debug it to track down exactly what is causing it? There was obviously a lot more debug info in the logs that I didn't include, but none of it jumped out to me as "smoking gun".
Thanks much…