Ok, I resolved the issue. It was little forgotten line at the end of the /etc/openldap/test/access.conf. The line says:
access to * by * none
I have learned that "cn=Manager,dc=test,dc=com" is never affected by any ACL, which is the reason it was authenticating properly, but not any other user. Commented out the line, restarted ldap, and everything is great.
Thank you