Aaron Richton wrote:
> Actually, this could be it exactly. To my reading, the 0.9.8d tarball
> still defaults to (an extremely dangerous) getpid(). 2.3.30 never uses
> CRYPTO_set_id_callback. And the most recent thread I see on the matter
> ended
> (http://www.mail-archive.com/openssl-dev@openssl.org/msg21037.html) with
> an attitude of "Yeah, if anything, we should make things break more
> frequently when there's no callback set." Perhaps we should be adding
> …
[View More]one, with a bit of platform awareness through lutil?
In the current OpenSSL, the address of errno is tested as well. Since
this is always unique per thread, there's really no need to set the id
callback any more. The problem with just using CRYPTO_set_id_callback is
that it doesn't work on platforms where a thread ID is not an integer
(e.g. OS/390). I don't think CRYPTO_set_idptr_callback was available in
earlier OpenSSL releases.
Too bad they didn't define CRYPTO_set_id_callback correctly, to return
the actual type of a thread ID instead of unsigned long.
>
> On Wed, 29 Nov 2006, Howard Chu wrote:
>
>> Aaron Richton wrote:
>>> I'm on latest 0.9.7 release. I can try and put together a slapd with
>>> 0.9.8d, and I guess if we're going to (potentially?) be pointing
>>> fingers toward OpenSSL that's a good idea anyway...
>>
>> Yes, definitely a good idea. The prior releases always used getpid()
>> to determine the threadID of the current thread, to decide if locking
>> was needed. This is obviously only correct on old systems running
>> LinuxThreads, where each thread was actually a separate process. It's
>> surprising that it wasn't until recently that we've started seeing
>> crashes caused by this bug.
>
> .
>
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/
[View Less]
Actually, this could be it exactly. To my reading, the 0.9.8d tarball
still defaults to (an extremely dangerous) getpid(). 2.3.30 never uses
CRYPTO_set_id_callback. And the most recent thread I see on the matter
ended (http://www.mail-archive.com/openssl-dev@openssl.org/msg21037.html)
with an attitude of "Yeah, if anything, we should make things break more
frequently when there's no callback set." Perhaps we should be adding one,
with a bit of platform awareness through lutil?
On Wed, 29 …
[View More]Nov 2006, Howard Chu wrote:
> Aaron Richton wrote:
>> I'm on latest 0.9.7 release. I can try and put together a slapd with
>> 0.9.8d, and I guess if we're going to (potentially?) be pointing fingers
>> toward OpenSSL that's a good idea anyway...
>
> Yes, definitely a good idea. The prior releases always used getpid() to
> determine the threadID of the current thread, to decide if locking was
> needed. This is obviously only correct on old systems running LinuxThreads,
> where each thread was actually a separate process. It's surprising that it
> wasn't until recently that we've started seeing crashes caused by this bug.
[View Less]
Aaron Richton wrote:
> I'm on latest 0.9.7 release. I can try and put together a slapd with
> 0.9.8d, and I guess if we're going to (potentially?) be pointing fingers
> toward OpenSSL that's a good idea anyway...
Yes, definitely a good idea. The prior releases always used getpid() to
determine the threadID of the current thread, to decide if locking was
needed. This is obviously only correct on old systems running
LinuxThreads, where each thread was actually a separate process. …
[View More]It's
surprising that it wasn't until recently that we've started seeing
crashes caused by this bug.
>
> On Tue, 28 Nov 2006, Howard Chu wrote:
>
>> richton(a)nbcs.rutgers.edu wrote:
>>> Here's another, although without a malloc debugger:
>>
>>> What would be good ways to go about figuring out if this is an issue
>>> with the SSL library or slapd's usage thereof?
>>
>> What version of OpenSSL are you using? I've definitely encountered
>> locking issues in 0.9.7 and early 0.9.8 releases. Some of those are
>> supposed to be fixed in the latest 0.9.8.
>>
>> --
>> -- Howard Chu
>> Chief Architect, Symas Corp. http://www.symas.com
>> Director, Highland Sun http://highlandsun.com/hyc
>> OpenLDAP Core Team http://www.openldap.org/project/
>>
>
> .
>
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/
[View Less]
Here's another, although without a malloc debugger:
current thread: t@11
[1] t_delete(0x9e4fa18, 0x80, 0xd7d0bc0, 0xfec3c000, 0x0, 0xedbff3f1), at 0xfebc783c
[2] _malloc_unlocked(0x80, 0x0, 0xedbff3e8, 0xfec3c000, 0x4, 0x5), at 0xfebc6ecc
[3] malloc(0x80, 0xff000000, 0x0, 0x0, 0x0, 0x0), at 0xfebc6d00
=>[4] default_malloc_ex(num = 128U, file = 0xfefe30b0 "bn_rand.c", line = 134), line 79 in "mem.c"
[5] CRYPTO_malloc(num = 128, file = 0xfefe30b0 "bn_rand.c", line = 134), line 304 …
[View More]in "mem.c"
[6] bnrand(pseudorand = 0, rnd = 0xa6b753c, bits = 1024, top = -1, bottom = 0), line 134 in "bn_rand.c"
[7] BN_rand(rnd = 0xa6b753c, bits = 1024, top = -1, bottom = 0), line 212 in "bn_rand.c"
[8] bn_rand_range(pseudo = 0, r = 0xa6b753c, range = 0x368558), line 274 in "bn_rand.c"
[9] BN_rand_range(r = 0xa6b753c, range = 0x368558), line 285 in "bn_rand.c"
[10] setup_blinding(rsa = 0x368298, ctx = 0xa6b7538), line 308 in "rsa_eay.c"
[11] RSA_eay_private_decrypt(flen = 128, from = 0x10edc0e6 "j\xb46\x8bb^Y\x8d\xcc\x86\xb2\xb2\x9c\n&\xdaYu^R^T\xc9Z\xca!\x9c\xaa\xd9i\x8d7\xf9x\xa1\xb0;\xa0 T", to = 0x10edc0e6 "j\xb46\x8bb^Y\x8d\xcc\x86\xb2\xb2\x9c\n&\xdaYu^R^T\xc9Z\xca!\x9c\xaa\xd9i\x8d7\xf9x\xa1\xb0;\xa0 T", rsa = 0x368298, padding = 1), line 536 in "rsa_eay.c"
[12] RSA_private_decrypt(flen = 128, from = 0x10edc0e6 "j\xb46\x8bb^Y\x8d\xcc\x86\xb2\xb2\x9c\n&\xdaYu^R^T\xc9Z\xca!\x9c\xaa\xd9i\x8d7\xf9x\xa1\xb0;\xa0 T", to = 0x10edc0e6 "j\xb46\x8bb^Y\x8d\xcc\x86\xb2\xb2\x9c\n&\xdaYu^R^T\xc9Z\xca!\x9c\xaa\xd9i\x8d7\xf9x\xa1\xb0;\xa0 T", rsa = 0x368298, padding = 1), line 292 in "rsa_lib.c"
[13] ssl3_get_client_key_exchange(s = 0xd25c868), line 1454 in "s3_srvr.c"
[14] ssl3_accept(s = 0xd25c868), line 448 in "s3_srvr.c"
[15] SSL_accept(s = 0xd25c868), line 816 in "ssl_lib.c"
[16] ldap_pvt_tls_accept(sb = 0xcc316f8, ctx_arg = 0x31f4e8), line 863 in "tls.c"
[17] connection_read(s = 63), line 1337 in "connection.c"
[18] slapd_daemon_task(ptr = (nil)), line 2352 in "daemon.c"
What would be good ways to go about figuring out if this is an issue with
the SSL library or slapd's usage thereof?
[View Less]
------=_Part_3513_4952625.1164743928641
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Sorry folks, it does look like a libdb mismatch. A little bit of poking
around found conflicting versions on the system and slapd opening both, and
setting up another testbed with 2.3.30 finds the server immune to the NASL
test.
Sorry to bother you guys, and thanks for the help.
Brian
------=_Part_3513_4952625.1164743928641
Content-…
[View More]Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Sorry folks, it does look like a libdb mismatch. A little bit of poking around found conflicting versions on the system and slapd opening both, and setting up another testbed with 2.3.30 finds the server immune to the NASL test.
<br><br>Sorry to bother you guys, and thanks for the help.<br><br>Brian<br>
------=_Part_3513_4952625.1164743928641--
[View Less]
------=_Part_3380_11468608.1164742144339
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
So after playing with this a bit more, I think there may be 'something else'
going on, although what that something else is I'm not entirely sure. This
may be another facet of the bug, a new bug entirely, or nothing to do with
OpenLDAP code.
Specifically, I played around with what the NASL script was sending to the
server. I tried …
[View More]sending different space counts, followed by padding of some
other character ('A'), to see what it would do. I'm afraid the results
aren't as conclusive as I'd hoped.
1) If I send a SINGLE SPACE to start the data portion, like so:
mkbyte(4) + mkbyte(8) + "CRAM-MD5" +
mkbyte(4) + mkbyte(0x82) + mkword(0x0400) +
crap(data:" ", length:1) +
crap(data:"A", length:1023);
the server NEVER crashes, no matter how many times I try it.
2) If I send more than a single space, even just 2 (followed by 1022 A's,
for example), the server will USUALLY, but NOT ALWAYS crash. (Approx 80% of
the time it chokes)
3) If I send more than 64 spaces, the server ALWAYS crashes. I do not know
if 64 is a magic number; I originally started out trying to send only 254
spaces + padding, which reliably caused crashes, and kept ratching down the
number until I suddenly stopped getting crashes reliably at 64 spaces. I
started to send a eureka email of sorts, but a couple more tests yielded
crashes at 64, and now it's crashing relatively reliably (90% of the time,
I'd say) from spaces=2 on up.
Finally, and I'm not sure how significant this is, every time it crashes the
[len=XXX] portion of the slapd_sasl_getdn debug line is ONE CHARACTER LESS
than the number of spaces I send. So, for example, if I send 2 spaces
followed by 1022 A's, then I see:
slap_sasl_getdn: conn 10 id= [len=1]
If I send 64 spaces followed by 960 A's, then len=63, and so on. If the
server does not crash, getdn is never called, with SASL complaining
"Failure: need authentication name".
I'm not sure if this helps or just muddies the issue further.
------=_Part_3380_11468608.1164742144339
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
So after playing with this a bit more, I think there may be 'something else' going on, although what that something else is I'm not entirely sure. This may be another facet of the bug, a new bug entirely, or nothing to do with OpenLDAP code.
<br><br>Specifically, I played around with what the NASL script was sending to the server. I tried sending different space counts, followed by padding of some other character ('A'), to see what it would do. I'm afraid the results aren't as conclusive as I'd hoped.
<br><br>1) If I send a SINGLE SPACE to start the data portion, like so:<br><br> mkbyte(4) + mkbyte(8) + "CRAM-MD5" +<br> mkbyte(4) + mkbyte(0x82) + mkword(0x0400) +<br> crap(data:" ", length:1) +
<br> crap(data:"A", length:1023);<br><br>the server NEVER crashes, no matter how many times I try it.<br><br>2) If I send more than a single space, even just 2 (followed by 1022 A's, for example), the server will USUALLY, but NOT ALWAYS crash. (Approx 80% of the time it chokes)
<br><br>3) If I send more than 64 spaces, the server ALWAYS crashes. I do not know if 64 is a magic number; I originally started out trying to send only 254 spaces + padding, which reliably caused crashes, and kept ratching down the number until I suddenly stopped getting crashes reliably at 64 spaces. I started to send a eureka email of sorts, but a couple more tests yielded crashes at 64, and now it's crashing relatively reliably (90% of the time, I'd say) from spaces=2 on up.
<br><br>Finally, and I'm not sure how significant this is, every time it crashes the [len=XXX] portion of the slapd_sasl_getdn debug line is ONE CHARACTER LESS than the number of spaces I send. So, for example, if I send 2 spaces followed by 1022 A's, then I see:
<br><br>slap_sasl_getdn: conn 10 id= [len=1]<br><br> If I send 64 spaces followed by 960 A's, then len=63, and so on. If the server does not crash, getdn is never called, with SASL complaining "Failure: need authentication name".
<br><br>I'm not sure if this helps or just muddies the issue further.<br>
------=_Part_3380_11468608.1164742144339--
[View Less]
------=_Part_2868_7949219.1164737844232
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Entirely possible (conflicting libdb versions). However, this crash is 100%
reproduceable on my system using the Nessus test specifically for the bug at
issue here. Nessus script ID is 20939, entitled "OpenLDAP SASL Bind Denial
of Service Vulnerability".
I actually was going to try to reproduce the problem with a perl script but
it …
[View More]seemed more time than it was worth since I had a reliable failure case
with the NASL script.
I've searched to see if there appears to be a libdb conflict of some sort,
and can't find anything, but that may just mean I haven't looked hard
enough!
I spent some more time poring of the code last night, and nothing jumped out
at me. However, as a further data point, I played with the NASL script some
to send different data, and I can ONLY get the server to crash if I send
spaces. Specifically, the relevant (and hopefully self-documenting) line of
NASL script is as follows:
mkbyte(4) + mkbyte(0x82) + mkword(0x0400) + crap(data:" ", length:1024);
If 'data:" "' is changed to be 'data:"A"', for example, the server does not
crash, and the message in the debug output is what I expect (snipped for
brevity).
Last line of hex dump follows:
0400: 41 41 41 41 AAAA
ber_scanf fmt (}}) ber:
ber_dump: buf=0x09ec9330 ptr=0x09ec974f end=0x09ec974f len=0
>>> dnPrettyNormal: <>
<<< dnPrettyNormal: <>, <>
do_sasl_bind: dn () mech CRAM-MD5
==> sasl_bind: dn="" mech=<continuing> datalen=1024
SASL [conn=3] Failure: need authentication name
send_ldap_result: conn=3 op=1 p=3
send_ldap_result: err=80 matched="" text="SASL(-5): bad protocol / cancel:
need authentication name"
send_ldap_response: msgid=509 tag=97 err=80
ber_flush: 72 bytes to sd 13
Does this help?
Brian
On 11/27/06, Howard Chu <hyc(a)symas.com> wrote:
>
> Kurt D. Zeilenga wrote:
> > At 08:06 PM 11/27/2006, hyc(a)symas.com wrote:
> >> Kurt(a)OpenLDAP.org wrote:
> >>> At 07:51 PM 11/27/2006, Kurt D. Zeilenga wrote:
> >>>> Spoke too soon.
> >>>> You code appears to be sending the same requests as
> >>>> Nessus, at least as described here:
> >>>> http://www.nessus.org/plugins/index.php?view=viewsrc&id=23625
> >>>>
> >>>> Suspect a mismatch between what you and Brian are
> >>>> testing...
> >>> Howard, is the normalized authcDN in your testing correct?
> >> It has a single escaped space.
> >
> > And that's correct (I was wrong before). A directory string of
> > N spaces normalizes to a single space, which must be escaped in
> > the DN.
> >
> > So it does seem like you and Brian are simply not running the
> > same code.
>
> The only difference between my current RE23 tree and 2.3.30 is in
> syncprov.c which is obviously not involved here. I would guess Brian's
> issue may be libsasl2 related, and no longer something resident in the
> OpenLDAP code. (E.g., conflicting libdb versions.)
>
> --
> -- Howard Chu
> Chief Architect, Symas Corp. http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc
> OpenLDAP Core Team http://www.openldap.org/project/
>
------=_Part_2868_7949219.1164737844232
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Entirely possible (conflicting libdb versions). However, this crash is 100% reproduceable on my system using the Nessus test specifically for the bug at issue here. Nessus script ID is 20939, entitled "OpenLDAP SASL Bind Denial of Service Vulnerability".
<br><br>I actually was going to try to reproduce the problem with a perl script but it seemed more time than it was worth since I had a reliable failure case with the NASL script. <br><br>I've searched to see if there appears to be a libdb conflict of some sort, and can't find anything, but that may just mean I haven't looked hard enough!
<br><br>I spent some more time poring of the code last night, and nothing jumped out at me. However, as a further data point, I played with the NASL script some to send different data, and I can ONLY get the server to crash if I send spaces. Specifically, the relevant (and hopefully self-documenting) line of NASL script is as follows:
<br><br>mkbyte(4) + mkbyte(0x82) + mkword(0x0400) + crap(data:" ", length:1024);<br><br>If 'data:" "' is changed to be 'data:"A"', for example, the server does not crash, and the message in the debug output is what I expect (snipped for brevity).
<br><br>Last line of hex dump follows:<br> 0400: 41 41 41 41 AAAA<br>ber_scanf fmt (}}) ber:<br>ber_dump: buf=0x09ec9330 ptr=0x09ec974f end=0x09ec974f len=0<br><br>>>> dnPrettyNormal: <>
<br><<< dnPrettyNormal: <>, <><br>do_sasl_bind: dn () mech CRAM-MD5<br>==> sasl_bind: dn="" mech=<continuing> datalen=1024<br>SASL [conn=3] Failure: need authentication name<br>send_ldap_result: conn=3 op=1 p=3
<br>send_ldap_result: err=80 matched="" text="SASL(-5): bad protocol / cancel: need authentication name"<br>send_ldap_response: msgid=509 tag=97 err=80<br>ber_flush: 72 bytes to sd 13<br><br>Does this help?
<br><br>Brian<br><br><div><span class="gmail_quote">On 11/27/06, <b class="gmail_sendername">Howard Chu</b> <<a href="mailto:hyc@symas.com">hyc(a)symas.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Kurt D. Zeilenga wrote:<br>> At 08:06 PM 11/27/2006, <a href="mailto:hyc@symas.com">hyc(a)symas.com</a> wrote:<br>>> <a href="mailto:Kurt@OpenLDAP.org">Kurt(a)OpenLDAP.org</a> wrote:<br>>>> At 07:51 PM 11/27/2006, Kurt D. Zeilenga wrote:
<br>>>>> Spoke too soon.<br>>>>> You code appears to be sending the same requests as<br>>>>> Nessus, at least as described here:<br>>>>> <a href="http://www.nessus.org/plugins/index.php?view=viewsrc&id=23625">
http://www.nessus.org/plugins/index.php?view=viewsrc&id=23625</a><br>>>>><br>>>>> Suspect a mismatch between what you and Brian are<br>>>>> testing...<br>>>> Howard, is the normalized authcDN in your testing correct?
<br>>> It has a single escaped space.<br>><br>> And that's correct (I was wrong before). A directory string of<br>> N spaces normalizes to a single space, which must be escaped in<br>> the DN.<br>><br>
> So it does seem like you and Brian are simply not running the<br>> same code.<br><br>The only difference between my current RE23 tree and 2.3.30 is in<br>syncprov.c which is obviously not involved here. I would guess Brian's
<br>issue may be libsasl2 related, and no longer something resident in the<br>OpenLDAP code. (E.g., conflicting libdb versions.)<br><br>--<br> -- Howard Chu<br> Chief Architect, Symas Corp. <a href="http://www.symas.com">
http://www.symas.com</a><br> Director, Highland Sun <a href="http://highlandsun.com/hyc">http://highlandsun.com/hyc</a><br> OpenLDAP Core Team <a href="http://www.openldap.org/project/">http://www.openldap.org/project/
</a><br></blockquote></div><br>
------=_Part_2868_7949219.1164737844232--
[View Less]
Kurt D. Zeilenga wrote:
> At 08:06 PM 11/27/2006, hyc(a)symas.com wrote:
>> Kurt(a)OpenLDAP.org wrote:
>>> At 07:51 PM 11/27/2006, Kurt D. Zeilenga wrote:
>>>> Spoke too soon.
>>>> You code appears to be sending the same requests as
>>>> Nessus, at least as described here:
>>>> http://www.nessus.org/plugins/index.php?view=viewsrc&id=23625
>>>>
>>>> Suspect a mismatch between what you and Brian are
&…
[View More]gt;>>> testing...
>>> Howard, is the normalized authcDN in your testing correct?
>> It has a single escaped space.
>
> And that's correct (I was wrong before). A directory string of
> N spaces normalizes to a single space, which must be escaped in
> the DN.
>
> So it does seem like you and Brian are simply not running the
> same code.
The only difference between my current RE23 tree and 2.3.30 is in
syncprov.c which is obviously not involved here. I would guess Brian's
issue may be libsasl2 related, and no longer something resident in the
OpenLDAP code. (E.g., conflicting libdb versions.)
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/
[View Less]