From our testing it appears that slapd's usage of the crypt function, to check
a user's password on a bind request, is single threaded, rather than being distributed across all of slapds thread. We encountered this problem when bumping the number of hashing rounds for our password hashes from 5,000 to 500,000 as was suggested by our security team.
Is it expected that the hashing of a users password would be bound to one thread?
We ran our tests on a default install of of slapd 2.4.44 on Debian Jessie box with 8 cores.
# Running script with butter user and 10,000,000 rounds of hashing:
$ pidstat -t -p $(pgrep slapd) 5 3 <snip> Average: UID TGID TID %usr %system %guest %CPU CPU Command Average: 108 28458 - 100.00 0.00 0.00 100.00 - slapd Average: 108 - 28458 0.00 0.00 0.00 0.00 - |__slapd Average: 108 - 28459 0.00 0.00 0.00 0.00 - |__slapd Average: 108 - 28460 0.00 0.00 0.00 0.00 - |__slapd Average: 108 - 10679 0.00 0.00 0.00 0.00 - |__slapd Average: 108 - 10680 0.00 0.00 0.00 0.00 - |__slapd Average: 108 - 17988 0.00 0.00 0.00 0.00 - |__slapd Average: 108 - 17993 0.00 0.00 0.00 0.00 - |__slapd Average: 108 - 17998 0.00 0.00 0.00 0.00 - |__slapd Average: 108 - 18007 22.53 0.00 0.00 22.53 - |__slapd Average: 108 - 19109 16.80 0.00 0.00 16.80 - |__slapd Average: 108 - 19110 0.07 0.00 0.00 0.07 - |__slapd Average: 108 - 19111 0.00 0.00 0.00 0.00 - |__slapd Average: 108 - 19112 60.73 0.00 0.00 60.73 - |__slapd Average: 108 - 19113 0.00 0.00 0.00 0.00 - |__slapd Average: 108 - 27438 0.00 0.00 0.00 0.00 - |__slapd Average: 108 - 27439 0.00 0.00 0.00 0.00 - |__slapd Average: 108 - 27440 0.00 0.00 0.00 0.00 - |__slapd Average: 108 - 27441 0.00 0.00 0.00 0.00 - |__slapd
# Running script with bubbles user and 5,000 rounds of hashing:
$ pidstat -t -p $(pgrep slapd) 5 3 <snip> Average: UID TGID TID %usr %system %guest %CPU CPU Command Average: 108 28458 - 109.59 0.87 0.00 110.46 - slapd Average: 108 - 28458 0.00 0.00 0.00 0.00 - |__slapd Average: 108 - 28459 0.80 2.80 0.00 3.60 - |__slapd Average: 108 - 28460 8.79 0.07 0.00 8.86 - |__slapd Average: 108 - 10679 7.00 0.07 0.00 7.06 - |__slapd Average: 108 - 10680 8.19 0.07 0.00 8.26 - |__slapd Average: 108 - 17988 3.80 0.07 0.00 3.86 - |__slapd Average: 108 - 17993 3.73 0.00 0.00 3.73 - |__slapd Average: 108 - 17998 7.46 0.00 0.00 7.46 - |__slapd Average: 108 - 18007 7.66 0.00 0.00 7.66 - |__slapd Average: 108 - 19109 8.93 0.07 0.00 8.99 - |__slapd Average: 108 - 19110 4.73 0.07 0.00 4.80 - |__slapd Average: 108 - 19111 9.33 0.00 0.00 9.33 - |__slapd Average: 108 - 19112 9.26 0.13 0.00 9.39 - |__slapd Average: 108 - 19113 2.40 0.00 0.00 2.40 - |__slapd Average: 108 - 27438 8.13 0.07 0.00 8.19 - |__slapd Average: 108 - 27439 1.87 0.07 0.00 1.93 - |__slapd Average: 108 - 27440 7.79 0.00 0.00 7.79 - |__slapd Average: 108 - 27441 7.00 0.00 0.00 7.00 - |__slapd
# Test ldif:
$ cat example.ldif dn: o=example o: example objectclass: organization
dn: ou=people, o=example ou: people objectclass: organizationalunit
dn: ou=groups, o=example ou: groups objectclass: organizationalunit
dn: cn=butter, ou=people, o=example objectclass: inetOrgPerson cn: butter sn: butter # {CRYPT}$6$rounds=10000000$ykk4zGD3ODNR$iMP/zYeisoWTYgxLtPv1qzoo/dVrYQLAb9sKlRMBgPTfFrr9lTzEEkJ9NcFdGI/MiRxHSx/1x3rnw3RkNRMer/ # 'everyone loves butter' userPassword:: e0NSWVBUfSQ2JHJvdW5kcz0xMDAwMDAwMCR5a2s0ekdEM09ETlIkaU1QL3pZZWlzb1dUWWd4THRQdjFxem9vL2RWcllRTEFiOXNLbFJNQmdQVGZGcnI5bFR6RUVrSjlOY0ZkR0kvTWlSeEhTeC8xeDNybnczUmtOUk1lci8= uid: butter
dn: cn=bubbles, ou=people, o=example objectclass: inetOrgPerson cn: bubbles sn: bubbles # {CRYPT}$6$rounds=5000$vY9dl4.rTyasVXCH$nlSXEZFnMn31NThCBWmfDy8EhaaZz6amUI8iukO8dpAnxMUaKK/78FqMVSi5iRbMvwkS6RNYhh4WhNzxDe.8r0 # 'everyone loves bubbles' userPassword:: e0NSWVBUfSQ2JHJvdW5kcz01MDAwJHZZOWRsNC5yVHlhc1ZYQ0gkbmxTWEVaRm5NbjMxTlRoQ0JXbWZEeThFaGFhWno2YW1VSThpdWtPOGRwQW54TVVhS0svNzhGcU1WU2k1aVJiTXZ3a1M2Uk5ZaGg0V2hOenhEZS44cjA= uid: bubbles
dn: cn=admin,o=example objectClass: simpleSecurityObject objectClass: organizationalRole cn: admin description: LDAP administrator userPassword:: e1NTSEF9SmFDaTRvZkRqQ2F3RHR2VVJQZWdleFoxZHFWNWRENUo= structuralObjectClass: organizationalRole
# Test script:
$ cat whoamis #!/bin/bash
USER=$1
trap cleanup SIGINT
cleanup() { kill -9 $(jobs -p) exit 1 }
while true; do if [[ $(jobs | wc -l) -lt 8 ]]; then ldapwhoami -x -D cn=${USER},ou=people,o=example -H ldap://localhost -w "everyone loves ${USER}" >/dev/null & else wait -n fi done
Jesse Hathaway wrote:
From our testing it appears that slapd's usage of the crypt function, to check
a user's password on a bind request, is single threaded, rather than being distributed across all of slapds thread. We encountered this problem when bumping the number of hashing rounds for our password hashes from 5,000 to 500,000 as was suggested by our security team.
Is it expected that the hashing of a users password would be bound to one thread?
Depends entirely on whether or not your libc supports crypt_r() (reentrant crypt). If not then yes, it has to be single-threaded because crypt() is not reentrant, it returns a pointer to static storage.
And of course, even if you use crypt_r() it's always possible that the underlying cipher is itself single-threaded. We have no way to know and no control over that.
We ran our tests on a default install of of slapd 2.4.44 on Debian Jessie box with 8 cores.
On Fri, Feb 16, 2018 at 12:54 PM, Howard Chu hyc@symas.com wrote:
Depends entirely on whether or not your libc supports crypt_r() (reentrant crypt). If not then yes, it has to be single-threaded because crypt() is not reentrant, it returns a pointer to static storage.
And of course, even if you use crypt_r() it's always possible that the underlying cipher is itself single-threaded. We have no way to know and no control over that.
Thanks Chu for the followup, I made the mistake of looking at the master source code which includes crypt_r support rather than the source code for 2.4.44 which we are presently running. It appears you added support for crypt_r on Sep 6, 2017 with commit afa861bf22, however that commit does not appear in any tagged version, `git tag --contains afa861bf22`
Do you have any idea when a version with this commit might be released?
I compiled master with crypt_r support and the results are much better
$ pidstat -t -p $(pgrep slapd) 5 3 <snip> Average: UID TGID TID %usr %system %guest %CPU CPU Command Average: 108 15720 - 788.67 0.07 0.00 788.73 - slapd Average: 108 - 15720 0.00 0.00 0.00 0.00 - |__slapd Average: 108 - 15721 0.00 0.00 0.00 0.00 - |__slapd Average: 108 - 15722 98.40 0.00 0.00 98.40 - |__slapd Average: 108 - 19581 38.80 0.00 0.00 38.80 - |__slapd Average: 108 - 19585 94.40 0.00 0.00 94.40 - |__slapd Average: 108 - 19591 94.00 0.00 0.00 94.00 - |__slapd Average: 108 - 19592 65.27 0.00 0.00 65.27 - |__slapd Average: 108 - 19650 98.80 0.00 0.00 98.80 - |__slapd Average: 108 - 19754 97.93 0.00 0.00 97.93 - |__slapd Average: 108 - 2526 39.00 0.00 0.00 39.00 - |__slapd Average: 108 - 3293 98.67 0.00 0.00 98.67 - |__slapd Average: 108 - 4694 63.60 0.00 0.00 63.60 - |__slapd
On Fri, Feb 16, 2018 at 12:01:37PM -0600, Jesse Hathaway wrote:
# {CRYPT}$6$rounds=10000000$ykk4zGD3ODNR$iMP/zYeisoWTYgxLtPv1qzoo/dVrYQLAb9sKlRMBgPTfFrr9lTzEEkJ9NcFdGI/MiRxHSx/1x3rnw3RkNRMer/ # 'everyone loves butter'
Have you tested this using the native SHA-2 support (slapd-sha2 contrib module and {SSHA512}) instead of libc crypt?
-
Howard Chu -
Jesse Hathaway -
Ryan Tandy