openldap-bugs August 2007

openldap-bugs@openldap.org

42 participants
211 discussions

Re: (ITS#5092) page size value of 1.2.840.113556.1.4.319 control
by randolf.werner＠sap.com 16 Aug '07

16 Aug '07

This is a multi-part message in MIME format. ------_=_NextPart_001_01C7DFD5.8B035F66 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, =20 thanks for your replies. Anyway i did not opened this message to discuss who implented RFC 2696 correctly and who not. The point is something else: =20 1) I tried to share my experience (and solution) with other LDAP application programmers, since i am pretty sure there are/will be more clients running into this issue (e.g. ldp.exe from Windows support tools seems to have the same problem when enabling 1.2.840.113556.1.4.319 / pages search type again slapd). 2) Try to point out how it can easily happen to write a client working with Active Directory but not with slapd and suggested an easy but perhaps not 100% RFC 2696 compliant slapd change. =20 Perhaps i should explain 2) a little bit more from my application develeopers point of view. Here is my rough application history: =20 a) In ~1999 we started building a LDAP V3 client for SAP that had to work on a very wide variety of OS platforms and as much as possible LDAP V3 compliant directories. b) It was first shipped in 2000/2001 and customers were happy. c) Data volume at customer side increazed and it stopped working for Active Directory customers. d) We learned about size limits, page size and the fact you simply have to use 1.2.840.113556.1.4.319 to get more than 1000 entries from Active Directroy. We implemented it and everybody was happy again. Implenting ldap_create_page_control for all OS platforms was not that trivial at that time. e) In ~2004 the first customer liked to use slapd. We noticed some minor issues (e.g. setting V3 protocol version explicitly) fixed it and once more everybody was happy. f) Yesterday a slapd 2.3.35 slapd customer reported a problem similar to e) ("Protocol Error"). So we analyzed it and fixed it again by reducing default page size to 2**31-1. =20 Here is why i believe others might run into the same issue. If you need to create a ldap client dealing with big search results for various LDAP directories you had to use 1.2.840.113556.1.4.319 even if you are only interested in collecting the entire result otherwise you would fail with Active Directory. You look at the ldap_create_page_control API spec (developers oftern use API specs and not protocol RFCs) and notice page size to be an usinged integer. Since you are not really interested in pages you specify the largest possible value 2**32-1. This way you end up with client working with Active Directory, all directories not supporting 1.2.840.113556.1.4.319 but not with slapd supporting 1.2.840.113556.1.4.319. =20 I am not interested in discussing who is right and who is wrong, i am just interested in making as much as possible working together in the real world with as little as possible customer pain. Therefore i am almost 100% sure Active Directory will not break existing clients by using signed instead of unsigned integer page size just to be more RFC 2696 compliant.On the other hand slapd has a chance to detect negavtive page sizes and assume some reasonable page size in that situation instead of "Protocol Error" without breaking any existing client. For my customers it will not make much difference since the patch will be rolled out soon, but perhaps others can benefit. =20 Best regards Randolf =20 =20 ------_=_NextPart_001_01C7DFD5.8B035F66 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML><HEAD> <META http-equiv=3DContent-Type content=3D"text/html; = charset=3Dus-ascii"> <META content=3D"MSHTML 6.00.3790.2920" name=3DGENERATOR></HEAD> <BODY> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20 size=3D2>Hi,</FONT></SPAN></DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20 size=3D2></FONT></SPAN> </DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>thanks = for your=20 replies. Anyway i did not opened this message to discuss who implented = RFC 2696=20 correctly and who not.  The point is something=20 else:</FONT></SPAN></DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20 size=3D2></FONT></SPAN> </DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>1) I = tried to share=20 my experience (and solution) with other LDAP application programmers, = since i am=20 pretty sure there are/will be more clients running into this issue (e.g. = ldp.exe=20 from Windows support tools seems to have the same problem when enabling=20 1.2.840.113556.1.4.319 / pages search type again = slapd).</FONT></SPAN></DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>2) Try = to point out=20 how it can easily happen to write a client working with Active Directory = but not=20 with slapd and suggested an easy but perhaps not 100% RFC 2696 compliant = slapd=20 change.</FONT></SPAN></DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20 size=3D2></FONT></SPAN> </DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial = size=3D2>Perhaps i should=20 explain 2) a little bit more from my application develeopers point of = view. Here=20 is my rough application history:</FONT></SPAN></DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20 size=3D2></FONT></SPAN> </DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>a) In = ~1999 we=20 started building a LDAP V3 client for SAP that had to work on a very = wide=20 variety of OS platforms and as much as possible LDAP V3 compliant=20 directories.</FONT></SPAN></DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>b) It = was first=20 shipped in 2000/2001 and customers were happy.</FONT></SPAN></DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>c) = Data volume at=20 customer side increazed and it stopped working for Active Directory=20 customers.</FONT></SPAN></DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>d) We = learned about=20 size limits, page size and the fact you simply have to use=20 1.2.840.113556.1.4.319  to get more than 1000 entries from Active=20 Directroy. We implemented it and everybody was happy again. Implenting=20 ldap_create_page_control for all OS platforms was not that trivial at = that=20 time.</FONT></SPAN></DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>e) In = ~2004 the=20 first customer liked to use slapd. We noticed some minor issues (e.g. = setting V3=20 protocol version explicitly) fixed it and once more everybody was=20 happy.</FONT></SPAN></DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>f) = Yesterday a slapd=20 2.3.35 slapd customer reported a problem similar to e) ("Protocol = Error"). So we=20 analyzed it and fixed it again by reducing default page size to=20 2**31-1.</FONT></SPAN></DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20 size=3D2></FONT></SPAN> </DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>Here = is why i=20 believe others might run into the same issue. If y</FONT></SPAN><SPAN=20 class=3D021122006-16082007><FONT face=3DArial size=3D2>ou need to create = a ldap client=20 dealing with big search results for various LDAP directories you had to = use=20 1.2.840.113556.1.4.319 even if you are only interested in collecting the = entire=20 result otherwise you would fail with Active Directory. You look at the=20 ldap_create_page_control API spec (developers oftern use API specs and = not=20 protocol RFCs) and notice page size to be an usinged integer. Since you = are not=20 really interested in pages you specify the largest possible value = 2**32-1.=20 This way you end up with client working with Active Directory, all=20 directories not supporting 1.2.840.113556.1.4.319 but not with = slapd=20 supporting 1.2.840.113556.1.4.319.</FONT></SPAN></DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20 size=3D2></FONT></SPAN> </DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>I am = not interested=20 in discussing who is right and who is wrong, i am just interested in = making as=20 much as possible working together in the real world with as little as = possible=20 customer pain. Therefore i am almost 100% sure Active Directory will not = break=20 existing clients by using signed instead of unsigned integer page size = just to=20 be more RFC 2696 compliant.On the other hand slapd has a chance to = detect=20 negavtive page sizes and assume some reasonable page size in that = situation=20 instead of "Protocol Error" without breaking any existing client. For my = customers it will not make much difference since the patch will be = rolled=20 out soon, but perhaps others can benefit.</FONT></SPAN></DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20 size=3D2></FONT></SPAN> </DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial size=3D2>Best=20 regards</FONT></SPAN></DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial = size=3D2>   =20 Randolf</FONT></SPAN></DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20 size=3D2></FONT></SPAN> </DIV> <DIV><SPAN class=3D021122006-16082007><FONT face=3DArial=20 size=3D2></FONT></SPAN> </DIV></BODY></HTML> ------_=_NextPart_001_01C7DFD5.8B035F66--

1 0

Re: (ITS#4627) back-ldif issues
by ando＠sys-net.it 15 Aug '07

15 Aug '07

> Yes, we require POSIX semantics here. rename() unlinks the target > atomically; In fact, that was my understanding of rename(2). >>>> .inum should be signed, or should be read with strtoul instead of >>>> strtol. >>> strtoul(3) used > > I think strtoul() here is a mistake, and we've fixed/refixed this point a > couple times already. Remember that frontendDB has an index of {-1} in the > config tree. Right. Then we need to use a signed integer. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#5092) page size value of 1.2.840.113556.1.4.319 control
by ando＠sys-net.it 15 Aug '07

15 Aug '07

> randolf.werner(a)sap.com wrote: >> Full_Name: Randolf Werner >> Version: 2.3.35 >> OS: Windows XP >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (155.56.68.221) >> >> >> Hi, >> i just have found the following compatibilty issue when using the >> 1.2.840.113556.1.4.319 control (http://www.ietf.org/rfc/rfc2696.txt) >> supported >> by slapd. According to Microsoft and IBM LDAP API >> ldap_create_page_control >> >> http://msdn2.microsoft.com/en-us/library/aa366547.aspx >> http://publib.boulder.ibm.com/infocenter/iseries/v5r3/index.jsp?topic=/apis… >> >> the page size parameter should be an 32 Bit unsigned integer, however >> the RFC >> specifies it as "(INTEGER (0..maxInt)". Therefore our LDAP client uses >> 2**32-1 >> as default page size value. This works fine with Microsoft Active >> Directroy for >> many years but fails with "Protocol Error" (Error 2) when using slapd. >> Our >> workaround is to use 2**31-1 as default value in our client. I am not >> sure which >> implementation is correct, anyway it is a compatibility issue for ldap >> clients >> and might be "fixed" to improve compatibility. > > I suggest you file bug reports with Microsoft and IBM to fix their broken > implementations then. RFC4511 clearly defines maxInt to be 2^31-1. That > value > has not changed since RFC2251 either. Or, fix your clients so that they don't use pagedResults when not necessary. 2**31-1 is a pretty large number. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati(a)sys-net.it ---------------------------------------

1 0

Re: (ITS#4627) back-ldif issues
by h.b.furuseth＠usit.uio.no 15 Aug '07

15 Aug '07

Howard Chu writes: >h.b.furuseth(a)usit.uio.no wrote: >> Not there yet... you create it in the current directory instead >> of in the ldif directory. >> >> Possibly it's best to create it in the top directory of the ldif >> database. That way, if a temp file is somehow created and not >> cleaned away, it's (a) in the easiest place to see and (b) won't >> prevent rmdir() if one tries to delete that database or whatever. > > I would use "rm -rf" to delete the DB, in which case (b) is a moot point. Sorry, I meant if one uses LDAP operations to delete a non-leaf entry. (I was thinking of a "cn=config/olcDatabase=.../" subtree.) >>> (...) >>>> .inum should be signed, or should be read with strtoul instead of >>>> strtol. >>> strtoul(3) used > > I think strtoul() here is a mistake, and we've fixed/refixed this point a > couple times already. Remember that frontendDB has an index of {-1} in the > config tree. Time to re-fix again and add a comment about that this time... Hm. It is read with strto(u)l but is int and not long. Maybe that's what I was talking about with what its type should be. -- Regards, Hallvard

1 0

Re: (ITS#4627) back-ldif issues
by hyc＠symas.com 15 Aug '07

15 Aug '07

h.b.furuseth(a)usit.uio.no wrote: > Possibly it's best to create it in the top directory of the ldif > database. That way, if a temp file is somehow created and not > cleaned away, it's (a) in the easiest place to see and (b) won't > prevent rmdir() if one tries to delete that database or whatever. > > I don't know if there are non-Unix OSes which need special system calls > to move file between directories though. Or for that matter, if they > allow rename() to replace an existing file, maybe unlink is needed > first. The code looks very Unixy already, so I assume a Unix emulation > is needed to run it anyway. I don't know how those work. For that matter, I think the temp file must be created in the same directory as the existing file. It's always possible that someone will mount a new filesystem somewhere in the hierarchy, and you cannot rename() across mountpoints. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#4627) back-ldif issues
by hyc＠symas.com 15 Aug '07

15 Aug '07

h.b.furuseth(a)usit.uio.no wrote: > Pierangelo Masarati writes: >>> Error handling: >>> >>> ldif_tool_entry_first() cannot return NOID. >> Dunno (untouched) > > Howard fixed. One remaining issue: ldif_tool_entry_next() increments > li->li_tool_current even when it returns NOID. Should it do that? Harmless, but stupid. Fixed now. > >> (...) >>> back-ldif overwrites files instead writing to a temp file and >>> renaming when complete. The current way corrupts entries if a >>> write fails halfway through, e.g. due to a full disk. Also the >>> file can be left truncated if internal slapd operations fail. >>> If the OS supports it, back-ldif should make a new file in the >>> same directory and rename to the old filename when complete. >> Fixed; now mkstemp(3) is used, and rename(2) only in case of success. > > Not there yet... you create it in the current directory instead > of in the ldif directory. > > Possibly it's best to create it in the top directory of the ldif > database. That way, if a temp file is somehow created and not > cleaned away, it's (a) in the easiest place to see and (b) won't > prevent rmdir() if one tries to delete that database or whatever. I would use "rm -rf" to delete the DB, in which case (b) is a moot point. > I don't know if there are non-Unix OSes which need special system calls > to move file between directories though. Or for that matter, if they > allow rename() to replace an existing file, maybe unlink is needed > first. The code looks very Unixy already, so I assume a Unix emulation > is needed to run it anyway. I don't know how those work. Yes, we require POSIX semantics here. rename() unlinks the target atomically; it would be a mistake to code this in a different manner. > Also there were some spew_entry() coding errors, rev 1.70 fixes them. > >> (...) >>> .inum should be signed, or should be read with strtoul instead of >>> strtol. >> strtoul(3) used I think strtoul() here is a mistake, and we've fixed/refixed this point a couple times already. Remember that frontendDB has an index of {-1} in the config tree. >>> I suppose it and cmp in the function should be ber_int_t, >>> since LDAP integers are typically 32-bit. >> Not sure what you mean. > > I don't remember. Maybe the number can end up being sent over the > protocol, and it's possible to create a filename with a long > max > ber_int_t so the wrong number will be sent? > -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#4627) back-ldif issues
by h.b.furuseth＠usit.uio.no 15 Aug '07

15 Aug '07

Pierangelo Masarati writes: >> Error handling: >> >> ldif_tool_entry_first() cannot return NOID. > > Dunno (untouched) Howard fixed. One remaining issue: ldif_tool_entry_next() increments li->li_tool_current even when it returns NOID. Should it do that? > (...) >> back-ldif overwrites files instead writing to a temp file and >> renaming when complete. The current way corrupts entries if a >> write fails halfway through, e.g. due to a full disk. Also the >> file can be left truncated if internal slapd operations fail. >> If the OS supports it, back-ldif should make a new file in the >> same directory and rename to the old filename when complete. > > Fixed; now mkstemp(3) is used, and rename(2) only in case of success. Not there yet... you create it in the current directory instead of in the ldif directory. Possibly it's best to create it in the top directory of the ldif database. That way, if a temp file is somehow created and not cleaned away, it's (a) in the easiest place to see and (b) won't prevent rmdir() if one tries to delete that database or whatever. I don't know if there are non-Unix OSes which need special system calls to move file between directories though. Or for that matter, if they allow rename() to replace an existing file, maybe unlink is needed first. The code looks very Unixy already, so I assume a Unix emulation is needed to run it anyway. I don't know how those work. Also there were some spew_entry() coding errors, rev 1.70 fixes them. > (...) >> .inum should be signed, or should be read with strtoul instead of >> strtol. > > strtoul(3) used > >> I suppose it and cmp in the function should be ber_int_t, >> since LDAP integers are typically 32-bit. > > Not sure what you mean. I don't remember. Maybe the number can end up being sent over the protocol, and it's possible to create a filename with a long > max ber_int_t so the wrong number will be sent? -- Hallvard

1 0

Re: (ITS#5092) page size value of 1.2.840.113556.1.4.319 control
by hyc＠symas.com 15 Aug '07

15 Aug '07

randolf.werner(a)sap.com wrote: > Full_Name: Randolf Werner > Version: 2.3.35 > OS: Windows XP > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (155.56.68.221) > > > Hi, > i just have found the following compatibilty issue when using the > 1.2.840.113556.1.4.319 control (http://www.ietf.org/rfc/rfc2696.txt) supported > by slapd. According to Microsoft and IBM LDAP API ldap_create_page_control > > http://msdn2.microsoft.com/en-us/library/aa366547.aspx > http://publib.boulder.ibm.com/infocenter/iseries/v5r3/index.jsp?topic=/apis… > > the page size parameter should be an 32 Bit unsigned integer, however the RFC > specifies it as "(INTEGER (0..maxInt)". Therefore our LDAP client uses 2**32-1 > as default page size value. This works fine with Microsoft Active Directroy for > many years but fails with "Protocol Error" (Error 2) when using slapd. Our > workaround is to use 2**31-1 as default value in our client. I am not sure which > implementation is correct, anyway it is a compatibility issue for ldap clients > and might be "fixed" to improve compatibility. I suggest you file bug reports with Microsoft and IBM to fix their broken implementations then. RFC4511 clearly defines maxInt to be 2^31-1. That value has not changed since RFC2251 either. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5092) page size value of 1.2.840.113556.1.4.319 control
by kurt＠OpenLDAP.org 15 Aug '07

15 Aug '07

On Aug 15, 2007, at 2:21 PM, randolf.werner(a)sap.com wrote: > Full_Name: Randolf Werner > Version: 2.3.35 > OS: Windows XP > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (155.56.68.221) > > > Hi, > i just have found the following compatibilty issue when using the > 1.2.840.113556.1.4.319 control (http://www.ietf.org/rfc/ > rfc2696.txt) supported > by slapd. According to Microsoft and IBM LDAP API > ldap_create_page_control > > http://msdn2.microsoft.com/en-us/library/aa366547.aspx > http://publib.boulder.ibm.com/infocenter/iseries/v5r3/index.jsp? > topic=/apis/ldap_create_page_control.htm > > the page size parameter should be an 32 Bit unsigned integer, > however the RFC > specifies it as "(INTEGER (0..maxInt)". Therefore our LDAP client > uses 2**32-1 > as default page size value. This works fine with Microsoft Active > Directroy for > many years but fails with "Protocol Error" (Error 2) when using > slapd. Our > workaround is to use 2**31-1 as default value in our client. I am > not sure which > implementation is correct, anyway it is a compatibility issue for > ldap clients > and might be "fixed" to improve compatibility. Given maxInt is defined in RFC 4511 as: maxInt INTEGER ::= 2147483647 -- (2^^31 - 1) -- this doesn't appear to be a bug in OpenLDAP Software. -- Kurt

1 0

(ITS#5092) page size value of 1.2.840.113556.1.4.319 control
by randolf.werner＠sap.com 15 Aug '07

15 Aug '07

Full_Name: Randolf Werner Version: 2.3.35 OS: Windows XP URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (155.56.68.221) Hi, i just have found the following compatibilty issue when using the 1.2.840.113556.1.4.319 control (http://www.ietf.org/rfc/rfc2696.txt) supported by slapd. According to Microsoft and IBM LDAP API ldap_create_page_control http://msdn2.microsoft.com/en-us/library/aa366547.aspx http://publib.boulder.ibm.com/infocenter/iseries/v5r3/index.jsp?topic=/apis… the page size parameter should be an 32 Bit unsigned integer, however the RFC specifies it as "(INTEGER (0..maxInt)". Therefore our LDAP client uses 2**32-1 as default page size value. This works fine with Microsoft Active Directroy for many years but fails with "Protocol Error" (Error 2) when using slapd. Our workaround is to use 2**31-1 as default value in our client. I am not sure which implementation is correct, anyway it is a compatibility issue for ldap clients and might be "fixed" to improve compatibility. Best regards Randolf

1 0

← Newer
1
...
9
10
11
12
13
14
15
...
22
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs August 2007