Update:
After more experimenting we realized the the ldif we ran was doing a few things.
1. updating cn=config rootdn and password (was not changing anything just re-running modify to the same values) 2. updating dc=example,dc=com rootdn and password (only changing password) 3. updating the db suffix for dc=example,dc=com (was not changing the value just running a modify to the same value)
We've narrowed it down to updating the suffix seemed to be giving the providers issues. The scope of our changes really only needs to modify olcRootPW on the dc=example,dc=com database which seems to be working just fine and not giving any databases a hard time when we only run that change.
After investigation the logs closer we noticed that on provider instances when we updated all 3 in the same LDIF the providers seemed to crash and restart. When they restarted the syncrepl aspect of the providers determined the most recent CSN correctly, however the sncprov aspect of the providers was reset to a CSN hours old, therefore thinking the consumers upon reconnect were 'ahead'. After issuing a write to the crashed provider syncprov determined it's CSN was now 'newer' than the consumer and would oblige the delta sync repl requests.
Anyway we have changed our approach here to rotate the password and think it will be okay.
As Chris mentions we should be in a model where we can simply operate without a root password.