Re: (ITS#5092) page size value of 1.2.840.113556.1.4.319 control
by randolf.werner＠sap.com
This is a multi-part message in MIME format.
------_=_NextPart_001_01C7DFD5.8B035F66
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Hi,
=20
thanks for your replies. Anyway i did not opened this message to discuss
who implented RFC 2696 correctly and who not. The point is something
else:
=20
1) I tried to share my experience (and solution) with other LDAP
application programmers, since i am pretty sure there are/will be more
clients running into this issue (e.g. ldp.exe from Windows support tools
seems to have the same problem when enabling 1.2.840.113556.1.4.319 /
pages search type again slapd).
2) Try to point out how it can easily happen to write a client working
with Active Directory but not with slapd and suggested an easy but
perhaps not 100% RFC 2696 compliant slapd change.
=20
Perhaps i should explain 2) a little bit more from my application
develeopers point of view. Here is my rough application history:
=20
a) In ~1999 we started building a LDAP V3 client for SAP that had to
work on a very wide variety of OS platforms and as much as possible LDAP
V3 compliant directories.
b) It was first shipped in 2000/2001 and customers were happy.
c) Data volume at customer side increazed and it stopped working for
Active Directory customers.
d) We learned about size limits, page size and the fact you simply have
to use 1.2.840.113556.1.4.319 to get more than 1000 entries from Active
Directroy. We implemented it and everybody was happy again. Implenting
ldap_create_page_control for all OS platforms was not that trivial at
that time.
e) In ~2004 the first customer liked to use slapd. We noticed some minor
issues (e.g. setting V3 protocol version explicitly) fixed it and once
more everybody was happy.
f) Yesterday a slapd 2.3.35 slapd customer reported a problem similar to
e) ("Protocol Error"). So we analyzed it and fixed it again by reducing
default page size to 2**31-1.
=20
Here is why i believe others might run into the same issue. If you need
to create a ldap client dealing with big search results for various LDAP
directories you had to use 1.2.840.113556.1.4.319 even if you are only
interested in collecting the entire result otherwise you would fail with
Active Directory. You look at the ldap_create_page_control API spec
(developers oftern use API specs and not protocol RFCs) and notice page
size to be an usinged integer. Since you are not really interested in
pages you specify the largest possible value 2**32-1. This way you end
up with client working with Active Directory, all directories not
supporting 1.2.840.113556.1.4.319 but not with slapd supporting
1.2.840.113556.1.4.319.
=20
I am not interested in discussing who is right and who is wrong, i am
just interested in making as much as possible working together in the
real world with as little as possible customer pain. Therefore i am
almost 100% sure Active Directory will not break existing clients by
using signed instead of unsigned integer page size just to be more RFC
2696 compliant.On the other hand slapd has a chance to detect negavtive
page sizes and assume some reasonable page size in that situation
instead of "Protocol Error" without breaking any existing client. For my
customers it will not make much difference since the patch will be
rolled out soon, but perhaps others can benefit.
=20
Best regards
Randolf
=20
=20
------_=_NextPart_001_01C7DFD5.8B035F66
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<META content=3D"MSHTML 6.00.3790.2920" name=3DGENERATOR></HEAD>
<BODY>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20
size=3D2>Hi,</FONT></SPAN></DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>thanks =
for your=20
replies. Anyway i did not opened this message to discuss who implented =
RFC 2696=20
correctly and who not. The point is something=20
else:</FONT></SPAN></DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>1) I =
tried to share=20
my experience (and solution) with other LDAP application programmers, =
since i am=20
pretty sure there are/will be more clients running into this issue (e.g. =
ldp.exe=20
from Windows support tools seems to have the same problem when enabling=20
1.2.840.113556.1.4.319 / pages search type again =
slapd).</FONT></SPAN></DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>2) Try =
to point out=20
how it can easily happen to write a client working with Active Directory =
but not=20
with slapd and suggested an easy but perhaps not 100% RFC 2696 compliant =
slapd=20
change.</FONT></SPAN></DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial =
size=3D2>Perhaps i should=20
explain 2) a little bit more from my application develeopers point of =
view. Here=20
is my rough application history:</FONT></SPAN></DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>a) In =
~1999 we=20
started building a LDAP V3 client for SAP that had to work on a very =
wide=20
variety of OS platforms and as much as possible LDAP V3 compliant=20
directories.</FONT></SPAN></DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>b) It =
was first=20
shipped in 2000/2001 and customers were happy.</FONT></SPAN></DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>c) =
Data volume at=20
customer side increazed and it stopped working for Active Directory=20
customers.</FONT></SPAN></DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>d) We =
learned about=20
size limits, page size and the fact you simply have to use=20
1.2.840.113556.1.4.319 to get more than 1000 entries from Active=20
Directroy. We implemented it and everybody was happy again. Implenting=20
ldap_create_page_control for all OS platforms was not that trivial at =
that=20
time.</FONT></SPAN></DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>e) In =
~2004 the=20
first customer liked to use slapd. We noticed some minor issues (e.g. =
setting V3=20
protocol version explicitly) fixed it and once more everybody was=20
happy.</FONT></SPAN></DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>f) =
Yesterday a slapd=20
2.3.35 slapd customer reported a problem similar to e) ("Protocol =
Error"). So we=20
analyzed it and fixed it again by reducing default page size to=20
2**31-1.</FONT></SPAN></DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>Here =
is why i=20
believe others might run into the same issue. If y</FONT></SPAN><SPAN=20
class=3D021122006-16082007><FONT face=3DArial size=3D2>ou need to create =
a ldap client=20
dealing with big search results for various LDAP directories you had to =
use=20
1.2.840.113556.1.4.319 even if you are only interested in collecting the =
entire=20
result otherwise you would fail with Active Directory. You look at the=20
ldap_create_page_control API spec (developers oftern use API specs and =
not=20
protocol RFCs) and notice page size to be an usinged integer. Since you =
are not=20
really interested in pages you specify the largest possible value =
2**32-1.=20
This way you end up with client working with Active Directory, all=20
directories not supporting 1.2.840.113556.1.4.319 but not with =
slapd=20
supporting 1.2.840.113556.1.4.319.</FONT></SPAN></DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>I am =
not interested=20
in discussing who is right and who is wrong, i am just interested in =
making as=20
much as possible working together in the real world with as little as =
possible=20
customer pain. Therefore i am almost 100% sure Active Directory will not =
break=20
existing clients by using signed instead of unsigned integer page size =
just to=20
be more RFC 2696 compliant.On the other hand slapd has a chance to =
detect=20
negavtive page sizes and assume some reasonable page size in that =
situation=20
instead of "Protocol Error" without breaking any existing client. For my =
customers it will not make much difference since the patch will be =
rolled=20
out soon, but perhaps others can benefit.</FONT></SPAN></DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>Best=20
regards</FONT></SPAN></DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial =
size=3D2> =20
Randolf</FONT></SPAN></DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV></BODY></HTML>
------_=_NextPart_001_01C7DFD5.8B035F66--
15 years, 5 months
Re: (ITS#4627) back-ldif issues
by ando＠sys-net.it
> Yes, we require POSIX semantics here. rename() unlinks the target
> atomically;
In fact, that was my understanding of rename(2).
>>>> .inum should be signed, or should be read with strtoul instead of
>>>> strtol.
>>> strtoul(3) used
>
> I think strtoul() here is a mistake, and we've fixed/refixed this point a
> couple times already. Remember that frontendDB has an index of {-1} in the
> config tree.
Right. Then we need to use a signed integer.
p.
Ing. Pierangelo Masarati
OpenLDAP Core Team
SysNet s.r.l.
via Dossi, 8 - 27100 Pavia - ITALIA
http://www.sys-net.it
---------------------------------------
Office: +39 02 23998309
Mobile: +39 333 4963172
Email: pierangelo.masarati(a)sys-net.it
---------------------------------------
15 years, 5 months
Re: (ITS#5092) page size value of 1.2.840.113556.1.4.319 control
by ando＠sys-net.it
> randolf.werner(a)sap.com wrote:
>> Full_Name: Randolf Werner
>> Version: 2.3.35
>> OS: Windows XP
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (155.56.68.221)
>>
>>
>> Hi,
>> i just have found the following compatibilty issue when using the
>> 1.2.840.113556.1.4.319 control (http://www.ietf.org/rfc/rfc2696.txt)
>> supported
>> by slapd. According to Microsoft and IBM LDAP API
>> ldap_create_page_control
>>
>> http://msdn2.microsoft.com/en-us/library/aa366547.aspx
>> http://publib.boulder.ibm.com/infocenter/iseries/v5r3/index.jsp?topic=/ap...
>>
>> the page size parameter should be an 32 Bit unsigned integer, however
>> the RFC
>> specifies it as "(INTEGER (0..maxInt)". Therefore our LDAP client uses
>> 2**32-1
>> as default page size value. This works fine with Microsoft Active
>> Directroy for
>> many years but fails with "Protocol Error" (Error 2) when using slapd.
>> Our
>> workaround is to use 2**31-1 as default value in our client. I am not
>> sure which
>> implementation is correct, anyway it is a compatibility issue for ldap
>> clients
>> and might be "fixed" to improve compatibility.
>
> I suggest you file bug reports with Microsoft and IBM to fix their broken
> implementations then. RFC4511 clearly defines maxInt to be 2^31-1. That
> value
> has not changed since RFC2251 either.
Or, fix your clients so that they don't use pagedResults when not
necessary. 2**31-1 is a pretty large number.
p.
Ing. Pierangelo Masarati
OpenLDAP Core Team
SysNet s.r.l.
via Dossi, 8 - 27100 Pavia - ITALIA
http://www.sys-net.it
---------------------------------------
Office: +39 02 23998309
Mobile: +39 333 4963172
Email: pierangelo.masarati(a)sys-net.it
---------------------------------------
15 years, 5 months
Re: (ITS#4627) back-ldif issues
by h.b.furuseth＠usit.uio.no
Howard Chu writes:
>h.b.furuseth(a)usit.uio.no wrote:
>> Not there yet... you create it in the current directory instead
>> of in the ldif directory.
>>
>> Possibly it's best to create it in the top directory of the ldif
>> database. That way, if a temp file is somehow created and not
>> cleaned away, it's (a) in the easiest place to see and (b) won't
>> prevent rmdir() if one tries to delete that database or whatever.
>
> I would use "rm -rf" to delete the DB, in which case (b) is a moot point.
Sorry, I meant if one uses LDAP operations to delete a non-leaf entry.
(I was thinking of a "cn=config/olcDatabase=.../" subtree.)
>>> (...)
>>>> .inum should be signed, or should be read with strtoul instead of
>>>> strtol.
>>> strtoul(3) used
>
> I think strtoul() here is a mistake, and we've fixed/refixed this point a
> couple times already. Remember that frontendDB has an index of {-1} in the
> config tree.
Time to re-fix again and add a comment about that this time...
Hm. It is read with strto(u)l but is int and not long. Maybe that's
what I was talking about with what its type should be.
--
Regards,
Hallvard
15 years, 5 months
Re: (ITS#4627) back-ldif issues
by hyc＠symas.com
h.b.furuseth(a)usit.uio.no wrote:
> Possibly it's best to create it in the top directory of the ldif
> database. That way, if a temp file is somehow created and not
> cleaned away, it's (a) in the easiest place to see and (b) won't
> prevent rmdir() if one tries to delete that database or whatever.
>
> I don't know if there are non-Unix OSes which need special system calls
> to move file between directories though. Or for that matter, if they
> allow rename() to replace an existing file, maybe unlink is needed
> first. The code looks very Unixy already, so I assume a Unix emulation
> is needed to run it anyway. I don't know how those work.
For that matter, I think the temp file must be created in the same directory as
the existing file. It's always possible that someone will mount a new
filesystem somewhere in the hierarchy, and you cannot rename() across mountpoints.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
15 years, 5 months
Re: (ITS#4627) back-ldif issues
by hyc＠symas.com
h.b.furuseth(a)usit.uio.no wrote:
> Pierangelo Masarati writes:
>>> Error handling:
>>>
>>> ldif_tool_entry_first() cannot return NOID.
>> Dunno (untouched)
>
> Howard fixed. One remaining issue: ldif_tool_entry_next() increments
> li->li_tool_current even when it returns NOID. Should it do that?
Harmless, but stupid. Fixed now.
>
>> (...)
>>> back-ldif overwrites files instead writing to a temp file and
>>> renaming when complete. The current way corrupts entries if a
>>> write fails halfway through, e.g. due to a full disk. Also the
>>> file can be left truncated if internal slapd operations fail.
>>> If the OS supports it, back-ldif should make a new file in the
>>> same directory and rename to the old filename when complete.
>> Fixed; now mkstemp(3) is used, and rename(2) only in case of success.
>
> Not there yet... you create it in the current directory instead
> of in the ldif directory.
>
> Possibly it's best to create it in the top directory of the ldif
> database. That way, if a temp file is somehow created and not
> cleaned away, it's (a) in the easiest place to see and (b) won't
> prevent rmdir() if one tries to delete that database or whatever.
I would use "rm -rf" to delete the DB, in which case (b) is a moot point.
> I don't know if there are non-Unix OSes which need special system calls
> to move file between directories though. Or for that matter, if they
> allow rename() to replace an existing file, maybe unlink is needed
> first. The code looks very Unixy already, so I assume a Unix emulation
> is needed to run it anyway. I don't know how those work.
Yes, we require POSIX semantics here. rename() unlinks the target atomically;
it would be a mistake to code this in a different manner.
> Also there were some spew_entry() coding errors, rev 1.70 fixes them.
>
>> (...)
>>> .inum should be signed, or should be read with strtoul instead of
>>> strtol.
>> strtoul(3) used
I think strtoul() here is a mistake, and we've fixed/refixed this point a
couple times already. Remember that frontendDB has an index of {-1} in the
config tree.
>>> I suppose it and cmp in the function should be ber_int_t,
>>> since LDAP integers are typically 32-bit.
>> Not sure what you mean.
>
> I don't remember. Maybe the number can end up being sent over the
> protocol, and it's possible to create a filename with a long > max
> ber_int_t so the wrong number will be sent?
>
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
15 years, 5 months
Re: (ITS#4627) back-ldif issues
by h.b.furuseth＠usit.uio.no
Pierangelo Masarati writes:
>> Error handling:
>>
>> ldif_tool_entry_first() cannot return NOID.
>
> Dunno (untouched)
Howard fixed. One remaining issue: ldif_tool_entry_next() increments
li->li_tool_current even when it returns NOID. Should it do that?
> (...)
>> back-ldif overwrites files instead writing to a temp file and
>> renaming when complete. The current way corrupts entries if a
>> write fails halfway through, e.g. due to a full disk. Also the
>> file can be left truncated if internal slapd operations fail.
>> If the OS supports it, back-ldif should make a new file in the
>> same directory and rename to the old filename when complete.
>
> Fixed; now mkstemp(3) is used, and rename(2) only in case of success.
Not there yet... you create it in the current directory instead
of in the ldif directory.
Possibly it's best to create it in the top directory of the ldif
database. That way, if a temp file is somehow created and not
cleaned away, it's (a) in the easiest place to see and (b) won't
prevent rmdir() if one tries to delete that database or whatever.
I don't know if there are non-Unix OSes which need special system calls
to move file between directories though. Or for that matter, if they
allow rename() to replace an existing file, maybe unlink is needed
first. The code looks very Unixy already, so I assume a Unix emulation
is needed to run it anyway. I don't know how those work.
Also there were some spew_entry() coding errors, rev 1.70 fixes them.
> (...)
>> .inum should be signed, or should be read with strtoul instead of
>> strtol.
>
> strtoul(3) used
>
>> I suppose it and cmp in the function should be ber_int_t,
>> since LDAP integers are typically 32-bit.
>
> Not sure what you mean.
I don't remember. Maybe the number can end up being sent over the
protocol, and it's possible to create a filename with a long > max
ber_int_t so the wrong number will be sent?
--
Hallvard
15 years, 5 months
Re: (ITS#5092) page size value of 1.2.840.113556.1.4.319 control
by kurt＠OpenLDAP.org
On Aug 15, 2007, at 2:21 PM, randolf.werner(a)sap.com wrote:
> Full_Name: Randolf Werner
> Version: 2.3.35
> OS: Windows XP
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (155.56.68.221)
>
>
> Hi,
> i just have found the following compatibilty issue when using the
> 1.2.840.113556.1.4.319 control (http://www.ietf.org/rfc/
> rfc2696.txt) supported
> by slapd. According to Microsoft and IBM LDAP API
> ldap_create_page_control
>
> http://msdn2.microsoft.com/en-us/library/aa366547.aspx
> http://publib.boulder.ibm.com/infocenter/iseries/v5r3/index.jsp?
> topic=/apis/ldap_create_page_control.htm
>
> the page size parameter should be an 32 Bit unsigned integer,
> however the RFC
> specifies it as "(INTEGER (0..maxInt)". Therefore our LDAP client
> uses 2**32-1
> as default page size value. This works fine with Microsoft Active
> Directroy for
> many years but fails with "Protocol Error" (Error 2) when using
> slapd. Our
> workaround is to use 2**31-1 as default value in our client. I am
> not sure which
> implementation is correct, anyway it is a compatibility issue for
> ldap clients
> and might be "fixed" to improve compatibility.
Given maxInt is defined in RFC 4511 as:
maxInt INTEGER ::= 2147483647 -- (2^^31 - 1) --
this doesn't appear to be a bug in OpenLDAP Software.
-- Kurt
15 years, 5 months