sgallagh(a)redhat.com wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 01/19/2011 11:39 AM, masarati(a)aero.polimi.it wrote:
>>> Full_Name: Stephen Gallagher
>>> Version: 2.4.21-0ubuntu5.2
>>
>> I think this issue report is obsoleted by ITS#6510 (released with OL
>> 2.4.22), which prevents binds from returning referrals. Please upgrade
>> and re-check.
>
> Upgraded to 2.4.23 and the same behavior occurs. I note in issue 6510
> that it was suggested that nested mutexes could be used to resolve this.
> Perhaps we should revisit that.
No. Nested mutexes are non-portable, and as already discussed in #6510, it is
incorrect to process referrals returned in response to Bind requests.
> As you can see from our backtrace, this is actually occurring during the
> processing of ldap_result(). We're handling a rebind as ldap_result
> travels through a series of referrals.
Most likely your server is not using an actual Bind Response tag in its
response message. It would be good if you could run this using a debug build
(no optimization, full debug symbols present) and examine the tag that was
parsed from the referral result.
Looks like libldap needs to be changed to actually record the tag of the
outgoing requests. (It ought to do so anyway, and probably should return a
ProtocolError result if it receives a response message whose tag doesn't match
its request type.)
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
I wrote:
> In addition to the issues listed in my last message, this ITS depends
> on these: [ITS#6763,#6776#6760].
Sorry, that was quite wrong. The changes done so far can stand alone,
I've tried hard for that. Taking this ITS _further_ depends on those
ITSes. But those ITSes are safest to tackle after the changes I've done
so far.
I've done this ITS backwards after some IRC miscommunication and by
sitting too close to the code: First commit a huge patch to clean up
SlapReply, then explain why. Sorry about that. I intended to get it
done for 2.4.24, but when Howard balked at this approach I expect I
instead ended up complicating HEAD->RE24 merges.
So to explain better: This is ITS#5340 with a better name. See
http://www.openldap.org/its/?findid=5340#followup16 - 18
which actually explains most of the rationale for these changes:
"I'm a bit nervous about doing more than minimal fixes to this
until we do something major, because it's so broken. (...)
[Howard suggested] code which puts something into a SlapReply
should ensure it's taken out again. (...) I think that's not
the right fix because it's the most fragile solution:
Practically every module which depends on it can break if some
other module gets it wrong.
OTOH, modules can't just reset SlapReply since some of it is
input values to op_*() (...)"
SlapReply is so broken and confusing that for 3 years I've not dared to
do the obvious fixes to ITS#5340. Early on I kept prompting others who
understood slapd better than me to take on #5430, but nothing happened.
So now I've "done something major": Cleaned up SlapReply enough that
it's no longer an act of bravery to take on the original issues. Most
of the changes are a preliminary cleanup for going for the original
ITS#5340 fixes (obeying the flags).
elsewhere:
By the original design, it's not a bug for an op_*() function to expect
an incoming SlapReply to be clean, nor to output a "dirty" SlapReply, so
it' has got to be wrong to reuse a SlapReply without cleaning it up. So
I've taken a simpler and more robust approach this time, adding bigger
but more trivial changes: In addition to cleaning up some flags, I've
killed most cases of SlapReply reuse, which also isolates what damage
one module can do to another through SlapReply. Even if they are
third-party overlays where the first is broken because it accurately
copied what OpenLDAP's overlays were doing:(
I've also mostly fixed rs.sr_type. That one was almost as broken,
somewhat in the spirit of this awful and confusing idea in result.c:
/* FIXME: I think rs->sr_type should be explicitly set to
* REP_SEARCH here. That's what it was when we entered this
* function. send_ldap_error may have changed it, but we
* should set it back so that the cleanup functions know
* what they're doing.
*/
No. If something set rs.sr_type = REP_INTERMEDIATE, result.c should
not tell its caller that OID rs.sr_rspoid is an entry, rs.sr_entry.
As for user-visible issues the cleanup caught: Half of my December and
January ITSes came from this effort. Some minor, some bigger. Generally
issues somewhat out of scope of what I was doing in this ITS.
Howard asked if all these changes fixed any crashes. Thinking it over,
I've had few crashes because of this mess, and I have had a hard time
trying to remember which ones. I've had many crashes in the truckload
of asserts I added in my private code, which pointed out these ITSes and
how to clean up. I could not have found the ITSes before the cleanup,
since the asserts then brought down slapd in an instant.
Also I've broken slapd a number of times by attempting "obvious" fixes
before cleaning up, because slapd is more creative than me about reusing
SlapReply strangely. Like pcache.c which reused a Search SlapReply with
an entry in it for be_modify() and expected the entry to still be there
after the modify:-( Looks like a good candidate for trying to construct
a pre-cleanup crash, if I ever find time to try: Find a way to feed that
SlapReply to something which reuses it for a Search or an Intermediate
Response, thus overwriting the entry sitting in the Modify SlapReply.
That kind if thing is why I'm much more eager to get the initial cleanup
into RE24 than to get the final entry leak fixes etc into RE24. After
the cleanup, there should only be an entry in there if sr_type is
REP_SEARCH/REP_SEARCHREF, not also REP_RESULT/REP_GLUE_RESULT as before.
ITSes:
ITS#6760 slapo-rwm broken entry handling
ITS#6752 slapo-dynlist issues [+ related non-SlapReply issues]
ITS#6746, #6763 entry/control leaks [+ related non-SlapReply issues]
My commits using rs_flush_entry() & co have fixed some other entry leaks
or potential leaks. Some modules - in particular BDB - catch potential
leaks of their entries. Others do not, giving us memory leaks.
ITS#6731, #6748 used scrambled sr_text on an exited function's stack.
ITS#6774, ITS#6785, some of my back-config commits: sr_text/sr_matched
sent with sr_err=LDAP_SUCCESS, or related to another error than sr_err.
Browsing 2.4.24 patches, an earlier cleanup would have caught #6633
(slapd-bdb error propagation) - my asserts with test047 catch sr_text
!= NULL when sr_err == 0 - and likely #6629 (freeing controls).
--
Regards,
Hallvard
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 01/19/2011 11:39 AM, masarati(a)aero.polimi.it wrote:
>> Full_Name: Stephen Gallagher
>> Version: 2.4.21-0ubuntu5.2
>
> I think this issue report is obsoleted by ITS#6510 (released with OL
> 2.4.22), which prevents binds from returning referrals. Please upgrade
> and re-check.
Upgraded to 2.4.23 and the same behavior occurs. I note in issue 6510
that it was suggested that nested mutexes could be used to resolve this.
Perhaps we should revisit that.
As you can see from our backtrace, this is actually occurring during the
processing of ldap_result(). We're handling a rebind as ldap_result
travels through a series of referrals.
- --
Stephen Gallagher
RHCE 804006346421761
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAk03WlYACgkQeiVVYja6o6PiXACgmykXZhhKZkixweJTb/qIVPAc
DMAAn2GHK7wu3nC5mkcYq7jI2f13Ql+N
=rj7f
-----END PGP SIGNATURE-----
Full_Name: Pierangelo Masarati
Version: HEAD/re24
OS: irrelevant
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (188.153.31.111)
Submitted by: ando
back-meta's man page lists pseudoroot{dn,pw} as deprecated configuration
statements; move them to a separate block.
p.
> Full_Name: Stephen Gallagher
> Version: 2.4.21-0ubuntu5.2
I think this issue report is obsoleted by ITS#6510 (released with OL
2.4.22), which prevents binds from returning referrals. Please upgrade
and re-check.
p.
Full_Name: Pierangelo Masarati
Version: HEAD/re24
OS: irrelevant
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (188.153.31.111)
Submitted by: ando
meta_back_is_candidate() marks as candidate any target whose suffix is
subordinate to the search request when the request has "subtree" scope. This in
principle can give false positives and could also give false negatives.
However, that piece of code is always exploited with a "subtree" scope,
regardless of the actual scope of the search. As a consequence, only false
positives can occur (which is harmless but may result in unnecessary activity).
The function meta_back_is_candidate() has been fixed to correctly manage all
conditions; when called during a search request, the actual scope is used, to
make sure only actual candidates are considered by search operations.
p.
Full_Name: Stephen Gallagher
Version: 2.4.21-0ubuntu5.2
OS: Ubuntu 10.04
URL:
Submission from: (NULL) (98.110.239.235)
This was discovered by a user of SSSD. When referrals are enabled with
LDAP_OPT_REFERRALS on, SSSD has a rebind procedure set up to handle
authenticating to the new server. This seems to work fine when we're dealing
with a simple bind, but when we attempt to use SASL bind (for Kerberos-based
GSSAPI authentication), we discovered a problem.
Our rebind procedure calls ldap_sasl_interactive_bind_s() with LDAP_SASL_QUIET.
This arrangement works fine for a single referral, however if the server has
nested referrals (say, entry1 refers to entry2 which refers to entry3 on another
server) then we hit a deadlock condition.
Attaching gdb, we see the backtrace included at the bottom of this message. What
appears to be happening is that for the first ldap_sasl_interactive_bind_s(),
openldap is locking a mutex, and when it is called a second time it's attempting
to lock that same mutex that has not yet been released.
Backtrace:
#0 __lll_lock_wait () at
../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
No locals.
#1 0x00007f48285975d9 in _L_lock_953 () from /lib/libpthread.so.0
No symbol table info available.
#2 0x00007f48285973fb in __pthread_mutex_lock (mutex=0x7f4829c05980) at
pthread_mutex_lock.c:61
ignore1 = <value optimized out>
ignore2 = 700471680
ignore3 = -512
__PRETTY_FUNCTION__ = "__pthread_mutex_lock"
type = <value optimized out>
#3 0x00007f48299d3e4e in ldap_sasl_interactive_bind_s () from
/usr/lib/libldap_r-2.4.so.2
No symbol table info available.
#4 0x00007f4825ae00fd in sdap_rebind_proc (ldap=0x2241b80,
url=0x2250220 "ldap://DomainDnsZones.org.example.com/DC=DomainDnsZones,DC=org,DC=example,D…",
request=<value optimized out>, msgid=<value optimized out>, params=<value
optimized out>)
at src/providers/ldap/sdap_async_connection.c:1624
p = <value optimized out>
sasl_mech = <value optimized out>
user_dn = <value optimized out>
password = {bv_len = 0, bv_val = 0x0}
ctrls = {0x0, 0x0}
tmp_ctx = <value optimized out>
ret = <value optimized out>
__FUNCTION__ = "sdap_rebind_proc"
#5 0x00007f48299df6d1 in ldap_new_connection () from
/usr/lib/libldap_r-2.4.so.2
No symbol table info available.
#6 0x00007f48299e0523 in ldap_send_server_request () from
/usr/lib/libldap_r-2.4.so.2
No symbol table info available.
#7 0x00007f48299e11cd in ldap_chase_v3referrals () from
/usr/lib/libldap_r-2.4.so.2
No symbol table info available.
#8 0x00007f48299cbf95 in ?? () from /usr/lib/libldap_r-2.4.so.2
No symbol table info available.
#9 0x00007f48299ccc2d in ldap_result () from /usr/lib/libldap_r-2.4.so.2
No symbol table info available.
#10 0x00007f48299d4788 in ldap_sasl_bind_s () from /usr/lib/libldap_r-2.4.so.2
No symbol table info available.
#11 0x00007f48299d1751 in ldap_int_sasl_bind () from
/usr/lib/libldap_r-2.4.so.2
No symbol table info available.
#12 0x00007f48299d3ea8 in ldap_sasl_interactive_bind_s () from
/usr/lib/libldap_r-2.4.so.2
No symbol table info available.
#13 0x00007f4825ae00fd in sdap_rebind_proc (ldap=0x2241b80,
url=0x225bc20 "ldap://ForestDnsZones.org.example.com/DC=ForestDnsZones,DC=org,DC=example,D…",
request=<value optimized out>, msgid=<value optimized out>, params=<value
optimized out>)
at src/providers/ldap/sdap_async_connection.c:1624
p = <value optimized out>
sasl_mech = <value optimized out>
user_dn = <value optimized out>
password = {bv_len = 0, bv_val = 0x0}
ctrls = {0x0, 0x0}
tmp_ctx = <value optimized out>
ret = <value optimized out>
__FUNCTION__ = "sdap_rebind_proc"
#14 0x00007f48299df6d1 in ldap_new_connection () from
/usr/lib/libldap_r-2.4.so.2
No symbol table info available.
#15 0x00007f48299e0523 in ldap_send_server_request () from
/usr/lib/libldap_r-2.4.so.2
No symbol table info available.
#16 0x00007f48299e11cd in ldap_chase_v3referrals () from
/usr/lib/libldap_r-2.4.so.2
No symbol table info available.
#17 0x00007f48299cbf95 in ?? () from /usr/lib/libldap_r-2.4.so.2
No symbol table info available.
#18 0x00007f48299ccc2d in ldap_result () from /usr/lib/libldap_r-2.4.so.2
No symbol table info available.
#19 0x00007f4825acfcb4 in sdap_process_result (ev=0x221b1f0, pvt=<value
optimized out>)
at src/providers/ldap/sdap_async.c:178
sh = 0x2241650
no_timeout = {tv_sec = 0, tv_usec = 0}
te = <value optimized out>
msg = <value optimized out>
ret = <value optimized out>
__FUNCTION__ = "sdap_process_result"
#20 0x00007f482b4f1825 in ?? () from /usr/lib/libtevent.so.0
No symbol table info available.
#21 0x00007f482b4f363b in ?? () from /usr/lib/libtevent.so.0
No symbol table info available.
#22 0x00007f482b4f09f0 in _tevent_loop_once () from /usr/lib/libtevent.so.0
No symbol table info available.
#23 0x00007f482b4f0a5b in ?? () from /usr/lib/libtevent.so.0
No symbol table info available.
#24 0x0000000000431e51 in server_loop (main_ctx=0x221c360) at
src/util/server.c:526
No locals.
#25 0x000000000040f8bb in main (argc=5, argv=<value optimized out>)
at src/providers/data_provider_be.c:1333
opt = <value optimized out>
pc = <value optimized out>
be_domain = 0x2212490 "AALTO"
srv_name = <value optimized out>
conf_entry = <value optimized out>
main_ctx = 0x221c360
ret = 0
long_options = {{longName = 0x0, shortName = 0 '\000', argInfo = 4, arg
= 0x644d00,
val = 0, descrip = 0x436e64 "Help options:", argDescrip = 0x0}, {
longName = 0x436e72 "debug-level", shortName = 100 'd', argInfo = 2,
arg = 0x644de0, val = 0, descrip = 0x436e43 "Debug level",
argDescrip = 0x0}, {
longName = 0x436e7e "debug-to-files", shortName = 102 'f', argInfo =
0,
arg = 0x644de4, val = 0,
descrip = 0x437ad8 "Send the debug output to files instead of
stderr",
argDescrip = 0x0}, {longName = 0x436e8d "debug-timestamps",
shortName = 0 '\000',
argInfo = 2, arg = 0x644cc0, val = 0, descrip = 0x436e4f "Add debug
timestamps",
argDescrip = 0x0}, {longName = 0x438458 "domain", shortName = 0
'\000',
argInfo = 1, arg = 0x7fffe703f208, val = 0,
descrip = 0x437b10 "Domain of the information provider (mandatory)",
argDescrip = 0x0}, {longName = 0x0, shortName = 0 '\000', argInfo =
0, arg = 0x0,
val = 0, descrip = 0x0, argDescrip = 0x0}}
__FUNCTION__ = "main"
Apparently it works.
p.
÷ÁÒÎÁËÏ× ëÉÒÉÌÌ wrote:
> It works!!!
>
> 18.01.2011, 22:47, masarati(a)aero.polimi.it:
>>> Full_Name: Varnakov Kiril
>>> Version: openldap-sasl-server-2.4.23
>>> OS: FreeBSD 8.0-RELEASE-p3
>>> URL: ftp://ftp.openldap.org/incoming/
>>> Submission from: (NULL) (89.235.170.154)
>>>
>>> Hi. When i configure ldap server with overlay proxycache and set
>>> pcachePersist=TRUE, only positive answer stay in cache after restart proxy
>>> ldap.
>>> If not restart proxy ldap, all good, negative answer in cache stay. Why?
>>>
>>> proxyTemplate (target=) 0 18400 18400
>> Good catch; a fix is in HEAD. Please test it (pcache.c 1.218 -> 1.219)
>> and report. Please also note that while tracking this issue I also
>> spotted a couple of bugs which have been fixed as ITS#6797 (pcache.c 1.217
>> -> 1.218).
>>
>> p.
>
It works!!!
18.01.2011, 22:47, masarati(a)aero.polimi.it:
>> Full_Name: Varnakov Kiril
>> Version: openldap-sasl-server-2.4.23
>> OS: FreeBSD 8.0-RELEASE-p3
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (89.235.170.154)
>>
>> Hi. When i configure ldap server with overlay proxycache and set
>> pcachePersist=TRUE, only positive answer stay in cache after restart proxy
>> ldap.
>> If not restart proxy ldap, all good, negative answer in cache stay. Why?
>>
>> proxyTemplate (target=) 0 18400 18400
>
> Good catch; a fix is in HEAD. Please test it (pcache.c 1.218 -> 1.219)
> and report. Please also note that while tracking this issue I also
> spotted a couple of bugs which have been fixed as ITS#6797 (pcache.c 1.217
> -> 1.218).
>
> p.
--
Best regards