Posix group with /etc/ldap.conf read priv

I don't like having the /etc/ldap.conf world readable because then anyone who has shell access can see our general ldap login credentials (without which you cannot see anything in the ldap tree). So I have added a posixgroup in ldap, added our shell users to it and did:

chown root:usergroup /etc/ldap.conf && chmod 640 /etc/ldap.conf

But when logging in to the shell, users still get the "I have no name!" problem because they cannot read the /etc/ldap.conf and cannot map their uid / guid numbers to names from the ldap tree.

Advice?