OK. I can reproduce in test, but I'm starting to not care due to the
circumstances.
My January 29 production database produces err=80s when hit. The important
open question is what caused the *initial* corruption: the hardware (which
I note has been reliable since downgrading to 2.3.39), 2.3.37 (or possibly
even some earlier version), or 2.3.40. I can think of no way to answer
this question at this time, short of additional experimentation/data
collection, and it may be near-impossible to find out. (Unfortunate,
because it is the important question.)
A perverse side effect is that the same corrupted production database,
loaded against 2.3.39, happily accepts all changes. Now, .39 and .40 would
be expected to produce identical results. Which of the two is wrong in
this case? I find myself saying, again, that expectations during
"impossible situations" are a difficult subject.
Now, why don't I care about figuring out which one of them is right?
Because I can only instigate failure when something is in an "impossible
situation" at t=0. In the test environment, I gained the luxury of a
debugging procedure I couldn't afford in production: a full rm/slapadd of
the entire database. If I rm/slapadd using entirely 2.3.39, things work.
And if I rm/slapadd using entirely 2.3.40, things work: so far I cannot
make an initial corruption with 2.3.40, although corruption at t=0 can be
worsened by it.
Given the fact that I observed database issues like #5262 in my own test
environment, I find it plausible that 2.3.37 corrupted my database on the
way out the door. I've also observed, in test, that I can work around this
with slapcat/rm/slapadd.
While all this has been going on, I've been stressing 2.3.40 with test008,
and it seems to be fine. I will likely try 2.3.40 again next week, but
plan on a slightly abnormal upgrade procedure that includes a
slapcat/rm/slapadd. This way, at least the sins of the past will be fully
purged prior to the first 2.3.40 start, so if any future issues arise
we'll have end-to-end 2.3.40 accountability.