If the clients and servers are on the same machine, the servers could listen to 'ldapi://<URL-escaped unique socket path>' instead of to 'ldap://host:unique port/'. E.g. the users could use 'ldapi://<escape $HOME/ldapi>' or '%2Ftmp%2Fldapi%2F<escape $USER>'.

However you'd need to upgrade OpenLDAP first, due to a security issue in "ldapi://" implementations before 2.3.35. (In particular, ensure your users won't use ldapi: with pre-2.3.35 clients against someone else's server.)