10:01 a.m.
From our testing it appears that slapd's usage of the crypt
function, to check
a user's password on a bind request, is single threaded,
rather than being
distributed across all of slapds thread. We encountered this problem when
bumping the number of hashing rounds for our password hashes from 5,000 to
500,000 as was suggested by our security team.
Is it expected that the hashing of a users password would be bound to one
thread?
We ran our tests on a default install of of slapd 2.4.44 on Debian Jessie box
with 8 cores.
# Running script with butter user and 10,000,000 rounds of hashing:
$ pidstat -t -p $(pgrep slapd) 5 3
<snip>
Average: UID TGID TID %usr %system %guest
%CPU CPU Command
Average: 108 28458 - 100.00 0.00 0.00
100.00 - slapd
Average: 108 - 28458 0.00 0.00 0.00
0.00 - |__slapd
Average: 108 - 28459 0.00 0.00 0.00
0.00 - |__slapd
Average: 108 - 28460 0.00 0.00 0.00
0.00 - |__slapd
Average: 108 - 10679 0.00 0.00 0.00
0.00 - |__slapd
Average: 108 - 10680 0.00 0.00 0.00
0.00 - |__slapd
Average: 108 - 17988 0.00 0.00 0.00
0.00 - |__slapd
Average: 108 - 17993 0.00 0.00 0.00
0.00 - |__slapd
Average: 108 - 17998 0.00 0.00 0.00
0.00 - |__slapd
Average: 108 - 18007 22.53 0.00 0.00
22.53 - |__slapd
Average: 108 - 19109 16.80 0.00 0.00
16.80 - |__slapd
Average: 108 - 19110 0.07 0.00 0.00
0.07 - |__slapd
Average: 108 - 19111 0.00 0.00 0.00
0.00 - |__slapd
Average: 108 - 19112 60.73 0.00 0.00
60.73 - |__slapd
Average: 108 - 19113 0.00 0.00 0.00
0.00 - |__slapd
Average: 108 - 27438 0.00 0.00 0.00
0.00 - |__slapd
Average: 108 - 27439 0.00 0.00 0.00
0.00 - |__slapd
Average: 108 - 27440 0.00 0.00 0.00
0.00 - |__slapd
Average: 108 - 27441 0.00 0.00 0.00
0.00 - |__slapd
# Running script with bubbles user and 5,000 rounds of hashing:
$ pidstat -t -p $(pgrep slapd) 5 3
<snip>
Average: UID TGID TID %usr %system %guest
%CPU CPU Command
Average: 108 28458 - 109.59 0.87 0.00
110.46 - slapd
Average: 108 - 28458 0.00 0.00 0.00
0.00 - |__slapd
Average: 108 - 28459 0.80 2.80 0.00
3.60 - |__slapd
Average: 108 - 28460 8.79 0.07 0.00
8.86 - |__slapd
Average: 108 - 10679 7.00 0.07 0.00
7.06 - |__slapd
Average: 108 - 10680 8.19 0.07 0.00
8.26 - |__slapd
Average: 108 - 17988 3.80 0.07 0.00
3.86 - |__slapd
Average: 108 - 17993 3.73 0.00 0.00
3.73 - |__slapd
Average: 108 - 17998 7.46 0.00 0.00
7.46 - |__slapd
Average: 108 - 18007 7.66 0.00 0.00
7.66 - |__slapd
Average: 108 - 19109 8.93 0.07 0.00
8.99 - |__slapd
Average: 108 - 19110 4.73 0.07 0.00
4.80 - |__slapd
Average: 108 - 19111 9.33 0.00 0.00
9.33 - |__slapd
Average: 108 - 19112 9.26 0.13 0.00
9.39 - |__slapd
Average: 108 - 19113 2.40 0.00 0.00
2.40 - |__slapd
Average: 108 - 27438 8.13 0.07 0.00
8.19 - |__slapd
Average: 108 - 27439 1.87 0.07 0.00
1.93 - |__slapd
Average: 108 - 27440 7.79 0.00 0.00
7.79 - |__slapd
Average: 108 - 27441 7.00 0.00 0.00
7.00 - |__slapd
# Test ldif:
$ cat example.ldif
dn: o=example
o: example
objectclass: organization
dn: ou=people, o=example
ou: people
objectclass: organizationalunit
dn: ou=groups, o=example
ou: groups
objectclass: organizationalunit
dn: cn=butter, ou=people, o=example
objectclass: inetOrgPerson
cn: butter
sn: butter
#
{CRYPT}$6$rounds=10000000$ykk4zGD3ODNR$iMP/zYeisoWTYgxLtPv1qzoo/dVrYQLAb9sKlRMBgPTfFrr9lTzEEkJ9NcFdGI/MiRxHSx/1x3rnw3RkNRMer/
# 'everyone loves butter'
userPassword::
e0NSWVBUfSQ2JHJvdW5kcz0xMDAwMDAwMCR5a2s0ekdEM09ETlIkaU1QL3pZZWlzb1dUWWd4THRQdjFxem9vL2RWcllRTEFiOXNLbFJNQmdQVGZGcnI5bFR6RUVrSjlOY0ZkR0kvTWlSeEhTeC8xeDNybnczUmtOUk1lci8=
uid: butter
dn: cn=bubbles, ou=people, o=example
objectclass: inetOrgPerson
cn: bubbles
sn: bubbles
#
{CRYPT}$6$rounds=5000$vY9dl4.rTyasVXCH$nlSXEZFnMn31NThCBWmfDy8EhaaZz6amUI8iukO8dpAnxMUaKK/78FqMVSi5iRbMvwkS6RNYhh4WhNzxDe.8r0
# 'everyone loves bubbles'
userPassword::
e0NSWVBUfSQ2JHJvdW5kcz01MDAwJHZZOWRsNC5yVHlhc1ZYQ0gkbmxTWEVaRm5NbjMxTlRoQ0JXbWZEeThFaGFhWno2YW1VSThpdWtPOGRwQW54TVVhS0svNzhGcU1WU2k1aVJiTXZ3a1M2Uk5ZaGg0V2hOenhEZS44cjA=
uid: bubbles
dn: cn=admin,o=example
objectClass: simpleSecurityObject
objectClass: organizationalRole
cn: admin
description: LDAP administrator
userPassword:: e1NTSEF9SmFDaTRvZkRqQ2F3RHR2VVJQZWdleFoxZHFWNWRENUo=
structuralObjectClass: organizationalRole
# Test script:
$ cat whoamis
#!/bin/bash
USER=$1
trap cleanup SIGINT
cleanup() {
kill -9 $(jobs -p)
exit 1
}
while true; do
if [[ $(jobs | wc -l) -lt 8 ]]; then
ldapwhoami -x -D cn=${USER},ou=people,o=example -H
ldap://localhost -w "everyone loves ${USER}" >/dev/null &
else
wait -n
fi
done
10:54 a.m.
Jesse Hathaway wrote:
>From our testing it appears that slapd's usage of the crypt
function, to check
a user's password on a bind request, is single threaded, rather than being
distributed across all of slapds thread. We encountered this problem when
bumping the number of hashing rounds for our password hashes from 5,000 to
500,000 as was suggested by our security team.
Is it expected that the hashing of a users password would be bound to one
thread?
Depends entirely on whether or not your libc supports crypt_r() (reentrant
crypt). If not then yes, it has to be single-threaded because crypt() is not
reentrant, it returns a pointer to static storage.
And of course, even if you use crypt_r() it's always possible that the
underlying cipher is itself single-threaded. We have no way to know and no
control over that.
We ran our tests on a default install of of slapd 2.4.44 on Debian Jessie box
with 8 cores.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
1:14 p.m.
On Fri, Feb 16, 2018 at 12:54 PM, Howard Chu <hyc(a)symas.com> wrote:
Depends entirely on whether or not your libc supports crypt_r()
(reentrant
crypt). If not then yes, it has to be single-threaded because crypt() is not
reentrant, it returns a pointer to static storage.
And of course, even if you use crypt_r() it's always possible that the
underlying cipher is itself single-threaded. We have no way to know and no
control over that.
Thanks Chu for the followup, I made the mistake of looking at the master
source code which includes crypt_r support rather than the source code for
2.4.44 which we are presently running. It appears you added support for
crypt_r on Sep 6, 2017 with commit afa861bf22, however that commit does not
appear in any tagged version, `git tag --contains afa861bf22`
Do you have any idea when a version with this commit might be released?
I compiled master with crypt_r support and the results are much better
$ pidstat -t -p $(pgrep slapd) 5 3
<snip>
Average: UID TGID TID %usr %system %guest
%CPU CPU Command
Average: 108 15720 - 788.67 0.07 0.00
788.73 - slapd
Average: 108 - 15720 0.00 0.00 0.00
0.00 - |__slapd
Average: 108 - 15721 0.00 0.00 0.00
0.00 - |__slapd
Average: 108 - 15722 98.40 0.00 0.00
98.40 - |__slapd
Average: 108 - 19581 38.80 0.00 0.00
38.80 - |__slapd
Average: 108 - 19585 94.40 0.00 0.00
94.40 - |__slapd
Average: 108 - 19591 94.00 0.00 0.00
94.00 - |__slapd
Average: 108 - 19592 65.27 0.00 0.00
65.27 - |__slapd
Average: 108 - 19650 98.80 0.00 0.00
98.80 - |__slapd
Average: 108 - 19754 97.93 0.00 0.00
97.93 - |__slapd
Average: 108 - 2526 39.00 0.00 0.00
39.00 - |__slapd
Average: 108 - 3293 98.67 0.00 0.00
98.67 - |__slapd
Average: 108 - 4694 63.60 0.00 0.00
63.60 - |__slapd
11:09 a.m.
On Fri, Feb 16, 2018 at 12:01:37PM -0600, Jesse Hathaway wrote:
#
{CRYPT}$6$rounds=10000000$ykk4zGD3ODNR$iMP/zYeisoWTYgxLtPv1qzoo/dVrYQLAb9sKlRMBgPTfFrr9lTzEEkJ9NcFdGI/MiRxHSx/1x3rnw3RkNRMer/
# 'everyone loves butter'
Have you tested this using the native SHA-2 support (slapd-sha2 contrib
module and {SSHA512}) instead of libc crypt?
1860
days inactive
1860
days old
3 comments
3 participants
participants (3)
-
Howard Chu -
Jesse Hathaway -
Ryan Tandy