ldapi:/// without TLS + ldap:// with TLS?

I'm running OpenLDAP 2.4 on CentOS 6.5. I'm trying to set it up so clients can use the ldapi:/// socket without TLS, but any clients using ldap:// must use TLS.

I believe that the relevant olc variables are olcLocalSSF and olcSecurity. I can't get it to work - either TLS is required no matter which URI I use, or clients can connect without TLS at all.

According to the docs, if I set olcLocalSSF to 128, and olcSecurity to ssf=128, it should work, but it's not. I can only connect without TLS if I delete the olcSecurity attribute, which allows anyone to connect without TLS. What am I dong wrong?