You also need to walk through the database and delete any attributes object classes defined by the overlay. That'll be at least any operational attributes since they must be hardcoded into the C source.
Only the overlay can know how to do this, so the overlay needs undo_me() code. Then that code must figure out how to interact with backends and other overlays - like translucent. Or how to figure out whether it is able to figure it out, and return LDAP_UNWILLING_TO_PERFORM if not.
I don't quite remember how ppolicy works but I assume it can "deactivate" an existing userPassword attribute. If so it also needs a policy about what to do with deactivated userPasswords which will be reactivated when ppolicy is deleted. I.e. should these userPasswords be deleted? And if an overlay does that - deletes data which might not be managed by itself - then things could get _really_ hairy to get right...