openldap-bugs September 2010

openldap-bugs@openldap.org

26 participants
109 discussions

Start a nNew thread

Re: (ITS#6645) Yet Another LDIF Wrap Patch
by masarati＠aero.polimi.it 10 Sep '10

10 Sep '10

1 0

Re: (ITS#6645) Yet Another LDIF Wrap Patch
by masarati＠aero.polimi.it 10 Sep '10

10 Sep '10

> hyc(a)symas.com writes: >> Not thrilled with the magic constant 2^31-1. Seems like you should have >> just >> used "if ( wrap && len > wrap )" instead of a word-size-dependent magic. > > The code just needs to be a little less clever:-) Pass -1 to a > 'ber_len_t wrap'. The constant (ber_len_t)-1 will prevent wrapping: > "if (len > (ber_len_t)-1)" always fails. Thanks. > But if we are going to have smarter LDIF output, I'm more interested in > an option to avoid base64 encoding of valid UTF-8 strings. Also one > to force base64-encoding for caller-specified attributes. Or just > a bitflag to ldif_sput() after the caller checks the attribute type. See ldif_must_b64_encode_register() :) We already use it to force encoding of olcRootPW and other password-like attrs in proxy backends. p.

1 0

Re: (ITS#6645) Yet Another LDIF Wrap Patch
by h.b.furuseth＠usit.uio.no 10 Sep '10

10 Sep '10

hyc(a)symas.com writes: > Not thrilled with the magic constant 2^31-1. Seems like you should have just > used "if ( wrap && len > wrap )" instead of a word-size-dependent magic. The code just needs to be a little less clever:-) Pass -1 to a 'ber_len_t wrap'. The constant (ber_len_t)-1 will prevent wrapping: "if (len > (ber_len_t)-1)" always fails. But if we are going to have smarter LDIF output, I'm more interested in an option to avoid base64 encoding of valid UTF-8 strings. Also one to force base64-encoding for caller-specified attributes. Or just a bitflag to ldif_sput() after the caller checks the attribute type. -- Hallvard

1 0

Re: (ITS#6646) back-hdb dncache disregarded in some searches
by ondrej.kuznik＠acision.com 10 Sep '10

10 Sep '10

On 09/10/2010 11:19 AM, hyc(a)symas.com wrote: > This fact is already documented in the manpage. This ITS will be closed. I disagree. The manpage states: "It should be noted that the DN cache is allowed to temporarily grow beyond the configured size. It does this if many entries are locked when it tries to do a purge, because that means they're legitimately in use. Also, the DN cache never purges entries that have cached children, so depending on the shape of the DIT, it could have lots of cached DNs over the defined limit." However, in the last case with the sizelimit, the DN cache grows over its configured size permanently. They do not seem to be legitimately in use and if they are, it is because the code that used them told noone it stopped doing so as the sizelimit has been reached or connection severed. All those entries are leafs, so no children either. From all this I deduced that either the code or documentation are missing something. And I will repeat myself, but two searches that differ only in their search bases behave _very_ differently. If one search iterates though the database and uses and frees the entries it processes emitting and the other one gets all entries first and parses and releases them afterwards, and if something interrupts it, does not release some. But yet again, I am only slowly gaining familiarity with OpenLDAP's immerse codebase and thus have no authority to speak about code I am nowhere near familiar with. Still, I am eager to learn as much as I can, so I'll be glad for some pointers. Thanks, Ondra This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.

1 0

Re: (ITS#6639) syncrepl failing using SASL/GSSAPI
by hyc＠symas.com 10 Sep '10

10 Sep '10

whm(a)stanford.edu wrote: > --On Thursday, September 09, 2010 10:43:06 PM -0700 Howard Chu<hyc(a)symas.com> wrote: > >> whm(a)stanford.edu wrote: >>> --On Friday, September 03, 2010 01:23:17 AM -0700 Bill MacAllister<whm(a)stanford.edu> wrote: >>> >>> The problem with the database was only coincidental. Restoring the database >>> got the failing replica past the problem replication event. >>> >>> In the replica pool of 6 servers we have seen the problem on there of the >>> servers. In thinking about this more it is unlikely that it is a slave >>> problem since the slaves have been in use for about 6 weeks and we did >>> not see the problem. Only when we changed the master to 2.4.23 did we >>> see the problem. I have captured a master debug log of the problem >>> event. It is at http://www.stanford.edu/~whm/files/master-debug.txt. >>> >>> Bill >>> >> Please try with this patch: >> >> Index: sasl.c >> =================================================================== >> RCS file: /repo/OpenLDAP/pkg/ldap/libraries/libldap/sasl.c,v >> retrieving revision 1.79 >> diff -u -r1.79 sasl.c >> --- sasl.c 13 Apr 2010 20:17:56 -0000 1.79 >> +++ sasl.c 10 Sep 2010 05:42:22 -0000 >> @@ -733,8 +733,9 @@ >> return ret; >> } else if ( p->buf_out.buf_ptr != p->buf_out.buf_end ) { >> /* partial write? pretend nothing got written */ >> - len2 = 0; >> p->flags |= LDAP_PVT_SASL_PARTIAL_WRITE; >> + sock_errset(EAGAIN); >> + len2 = -1; >> } >> >> /* return number of bytes encoded, not written, to ensure > > I have applied the patch and the debian packages built fine. I have > installed the new packages on the master servers in our dev and test > environments. Initial tests show that basic replication is working. > Of course, this problem did not exhibit itself in our test > environments. I will install it in our production environment Friday > evening and let you know what happens. > > Am I correct in assuming that the root of the problem is on the providers? Yes. Really, the problem will occur when either a client or server writes a large buffer thru a SASL encrypted connection, but this is more likely to happen when a server is sending responses. > I am asking because I was not planning installing the new version on the > consumers right away since 'emergency' changes to production software > require special approval. I would like to upgrade the providers in our > next maintenance window which is next Thursday. If you think I should > do it with the update of the master let me know. It's probably not urgent for the consumers, just bear in mind that any clients querying them thru SASL could also run into similar issues. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6639) syncrepl failing using SASL/GSSAPI
by whm＠stanford.edu 10 Sep '10

10 Sep '10

--On Thursday, September 09, 2010 10:43:06 PM -0700 Howard Chu <hyc(a)symas.com> wrote: > whm(a)stanford.edu wrote: >> --On Friday, September 03, 2010 01:23:17 AM -0700 Bill MacAllister<whm(a)stanford.edu> wrote: >> >> The problem with the database was only coincidental. Restoring the database >> got the failing replica past the problem replication event. >> >> In the replica pool of 6 servers we have seen the problem on there of the >> servers. In thinking about this more it is unlikely that it is a slave >> problem since the slaves have been in use for about 6 weeks and we did >> not see the problem. Only when we changed the master to 2.4.23 did we >> see the problem. I have captured a master debug log of the problem >> event. It is at http://www.stanford.edu/~whm/files/master-debug.txt. >> >> Bill >> > Please try with this patch: > > Index: sasl.c > =================================================================== > RCS file: /repo/OpenLDAP/pkg/ldap/libraries/libldap/sasl.c,v > retrieving revision 1.79 > diff -u -r1.79 sasl.c > --- sasl.c 13 Apr 2010 20:17:56 -0000 1.79 > +++ sasl.c 10 Sep 2010 05:42:22 -0000 > @@ -733,8 +733,9 @@ > return ret; > } else if ( p->buf_out.buf_ptr != p->buf_out.buf_end ) { > /* partial write? pretend nothing got written */ > - len2 = 0; > p->flags |= LDAP_PVT_SASL_PARTIAL_WRITE; > + sock_errset(EAGAIN); > + len2 = -1; > } > > /* return number of bytes encoded, not written, to ensure I have applied the patch and the debian packages built fine. I have installed the new packages on the master servers in our dev and test environments. Initial tests show that basic replication is working. Of course, this problem did not exhibit itself in our test environments. I will install it in our production environment Friday evening and let you know what happens. Am I correct in assuming that the root of the problem is on the providers? I am asking because I was not planning installing the new version on the consumers right away since 'emergency' changes to production software require special approval. I would like to upgrade the providers in our next maintenance window which is next Thursday. If you think I should do it with the update of the master let me know. Bill -- Bill MacAllister Infrastructure Delivery Group, Stanford University

1 0

Re: (ITS#6646) back-hdb dncache disregarded in some searches
by hyc＠symas.com 10 Sep '10

10 Sep '10

ondrej.kuznik(a)acision.com wrote: > Full_Name: Ondrej Kuznik > Version: 2.4.23 > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (62.168.56.1) This fact is already documented in the manpage. This ITS will be closed. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#6646) back-hdb dncache disregarded in some searches
by ondrej.kuznik＠acision.com 10 Sep '10

10 Sep '10

Full_Name: Ondrej Kuznik Version: 2.4.23 OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (62.168.56.1) Example: database hdb suffix o=test dncache 1000 is a database that has most of its 1M entries as uid=%d,o=one,o=test. perform an unindexed search: ldapsearch -b o=test uid=1 the entry is first in the db, so is spit out immediately. Watching dn cn=Database 2,cn=Databases,cn=Monitor shows: olmBDBDNCache: 1002 This is OK, but: perform an unindexed search with base o=one,o=test: ldapsearch -b o=one,o=test uid=1 watching dn cn=Database 2,cn=Databases,cn=Monitor shows that all 1M dns are loaded into the cache and only after that the entry uid=1,o=one,o=test is spit out. Then the dncache starts decreasing as the database filters out the remaining entries: olmBDBDNCache: 1000002 Another twist: perform an unindexed search with base o=one,o=test but either kill ldapsearch while hdb is still crunching or add a sizelimit greater than the number of entries that should be returned: ldapsearch -z2 -b o=one,o=test "uid=1*" olmBDBDNCache contains most of the database and never decreases. olmBDBDNCache: 1000002 A dn is later freed from the cache only in case it is referenced, should there be another similar arc e.g. o=another,o=test, its dns are added inside the cache too if we use the same technique. This makes hdb unusable for a large database (whose set of dns alone can eat the entire memory if loaded into dncache) unless one can guarantee that every search is either indexed or with search base == suffix of the db. In the example scenario, the value of cachefree affects almost nothing.

1 0

Re: (ITS#6639) syncrepl failing using SASL/GSSAPI
by hyc＠symas.com 09 Sep '10

09 Sep '10

whm(a)stanford.edu wrote: > --On Friday, September 03, 2010 01:23:17 AM -0700 Bill MacAllister<whm(a)stanford.edu> wrote: > > The problem with the database was only coincidental. Restoring the database > got the failing replica past the problem replication event. > > In the replica pool of 6 servers we have seen the problem on there of the > servers. In thinking about this more it is unlikely that it is a slave > problem since the slaves have been in use for about 6 weeks and we did > not see the problem. Only when we changed the master to 2.4.23 did we > see the problem. I have captured a master debug log of the problem > event. It is at http://www.stanford.edu/~whm/files/master-debug.txt. > > Bill > Please try with this patch: Index: sasl.c =================================================================== RCS file: /repo/OpenLDAP/pkg/ldap/libraries/libldap/sasl.c,v retrieving revision 1.79 diff -u -r1.79 sasl.c --- sasl.c 13 Apr 2010 20:17:56 -0000 1.79 +++ sasl.c 10 Sep 2010 05:42:22 -0000 @@ -733,8 +733,9 @@ return ret; } else if ( p->buf_out.buf_ptr != p->buf_out.buf_end ) { /* partial write? pretend nothing got written */ - len2 = 0; p->flags |= LDAP_PVT_SASL_PARTIAL_WRITE; + sock_errset(EAGAIN); + len2 = -1; } /* return number of bytes encoded, not written, to ensure -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6645) Yet Another LDIF Wrap Patch
by masarati＠aero.polimi.it 09 Sep '10

09 Sep '10

<ftp://ftp.openldap.org/incoming/pierangelo-masarati-2010-09-09-ldif-wrap.2.…> :)

1 0

← Newer
1
...
4
5
6
7
8
9
10
11
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs September 2010