openldap-bugs April 2008

openldap-bugs@openldap.org

45 participants
192 discussions

Re: (ITS#5408) back-ldif bugs
by h.b.furuseth＠usit.uio.no 09 Apr '08

09 Apr '08

hyc(a)symas.com writes: > h.b.furuseth(a)usit.uio.no wrote: >> - back-ldif escapes the directory separator as \<hex value>. But >> Windows uses "\" as directory separator, so that doesn't help. It >> needs another escape character. >> >> To avoid double hex-escaping of DNs that contain "\", we can pick >> another escape char, e.g. "^", escape that too, and translate "\" to >> it. Thus DN "cn=x^y\2Cz" gets filename "cn=x^5Ey^2Cz.ldif". > > That sounds fine. Is there any reason not to use %, which is already > used for URL encoding? Good point. The filenames I suggested will not be valid URLs but will resemble them, but that can be fixed by using proper URL-encoding of RDNs. Filenames for RDNs with \ in them get a bit longer though. >> - More characters should likely be hex-escaped: >> + ":" (as in "C:\") and "/" on Windows? I've seen programs use the >> latter as directory separator even on Windows. Others? >> + 8-bit chars in case the OS gets clever about charset handling. >> + Control chars. >> >> I don't use Windows myself though. Nor Mac... > > Again, the rules for URL encoding probably already cover any cases we > should worry about. I don't think so. At least "/" is not "unsafe", if I read it right. >> - RDNs ending with ".ldif" should be escaped. Currently "cn=foo.ldif" >> is the name of both RDN "cn=foo"'s entry file and RDN "cn=foo.ldif"'s >> non-leaf directory. > > Perhaps we should escape '.' Hm, true. "." is (was?) a special character on some filesystems. E.g. a file can only have one of them, or the 2nd is followed by a version number. >> - Sorted-value RDNs: No reply? :-) Ok, sort order can be adjusted separately anyway. One error in my previous attempt: bconfig.c mentions "pseudo-indexed" entries (cn=Include{xx}, cn=Module{xx}) which I presume must be sorted by {xx} too, so back-ldif can't treat only "attr={number}val" as indexed. I guess I'll either check both variants or allow {number} anywhere, depending on convenience. -- Hallvard

1 0

Re: (ITS#5408) back-ldif bugs
by hyc＠symas.com 09 Apr '08

09 Apr '08

h.b.furuseth(a)usit.uio.no wrote: > Full_Name: Hallvard B Furuseth > Version: HEAD, RE23, RE24 > OS: Linux > URL: http://folk.uio.no/hbf/OpenLDAP/back-ldif.c > Submission from: (NULL) (129.240.6.233) > Submitted by: hallvard > > > Here are a bunch of back-ldif bugs some questions. I have code for most > of it, but need advice/discussion on functionality changes. > > I've been sitting on it too long and would keep sitting if I waited > until I'd cleaned up everything, so, posting now instead. I enclose an > URL for a *draft* back-ldif/ldif.c. > > > Functionality changes. > > * Some LDIF files/directories with special chars need to be renamed: > > RE24 or RE25 change? If anyone uses back-ldif as a regular database, > they may need to slapcat/slapadd to upgrade past these changes. > > - back-ldif escapes the directory separator as \<hex value>. But > Windows uses "\" as directory separator, so that doesn't help. It > needs another escape character. > > To avoid double hex-escaping of DNs that contain "\", we can pick > another escape char, e.g. "^", escape that too, and translate "\" to > it. Thus DN "cn=x^y\2Cz" gets filename "cn=x^5Ey^2Cz.ldif". That sounds fine. Is there any reason not to use %, which is already used for URL encoding? > - More characters should likely be hex-escaped: > + ":" (as in "C:\") and "/" on Windows? I've seen programs use the > latter as directory separator even on Windows. Others? > + 8-bit chars in case the OS gets clever about charset handling. > + Control chars. > > I don't use Windows myself though. Nor Mac... Again, the rules for URL encoding probably already cover any cases we should worry about. > > - When back-ldif uses OS-specific escaping, we can't move a directory > tree between Windows and Unix hosts if some RDN in the tree contains > characters with OS-specific escaping. > > Should we special-case both Unix' and Windows' directory separators > on both OSes? Then instead there will be more directory trees which > can't be move between OpenLDAP versions, before and after the > change. Assuming anyone uses back-ldif in the first place:-) Might as well break it all at once. > > - The database suffix must be hex-escaped like the rest of the > filenames. > > - RDNs ending with ".ldif" should be escaped. Currently "cn=foo.ldif" > is the name of both RDN "cn=foo"'s entry file and RDN "cn=foo.ldif"'s > non-leaf directory. Perhaps we should escape '.' > > - Sorted-value RDNs: > > + In dn2path() when IX_FSL != IX_DNL (filename '{' != RDN '{'): > IX_FS[LR] must be hex-escaped. They are treated as equivalent to > '{' - '}', thus different RDNs can map to the same filename. > > The "{1}" in path "/foo/cn=x{,cn={1}y" is not recognized. > The IX_DN[LR] (i.e. '{', '}') to IX_FS[LR] translation is a bit > strange: It translates the first '{' it sees, then the next > '}', then the next '{', etc. Any reason not to just translate > all '{' and '}' chars? If so, we should at least reset at > each filename and '+'. > > + Sorting (in ldif_r_enum_tree()): > > Should "a={1}x" sort before or after "a=x"? Currently it sorts > like "a={". "a=" (normally before) or "a=<CHAR_MAX>" (normally > after) would be better. > > RDN "attr=foo{bar}baz" is treated as a sorted value. Can easily > check for '={'<successful strtol parse'> '}' instead. > "attr={0<octal>}val" and "attr={0x<hex>}val" are recognized as > sorted values, should they be? > > Some of this is from ITS#4627 (back-ldif issues). -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5457) bdb_add() does not always release entries
by h.b.furuseth＠usit.uio.no 09 Apr '08

09 Apr '08

hyc(a)symas.com writes: >> bdb_unlocked_cache_return_entry_r(&bdb->bi_cache, p ); >> (...) > Releasing entries is somewhat irrelevant, since all locks are released when > the transaction commits. Note that this function is a no-op in proto-bdb.h. > All of those statements are just relics from the first entry cache design, > before we switched to using BDB locks for everything. OK. Does that apply to bdb_cache_return_entry_r( bdb, oe, &lock ); after bdb_cache_add( bdb, ei, op->ora_e, &nrdn, rlocker, &lock ); failed TXN_COMMIT( ltid, 0 ); too? bdb_cache_return_entry_r() is not a no-op. -- Hallvard

1 0

Re: (ITS#5451) syncprov_checkpoint deadlock bugfix?
by rein＠tollevik.no 09 Apr '08

09 Apr '08

On Wed, 9 Apr 2008, h.b.furuseth(a)usit.uio.no wrote: > Does this help? From my fiddling with ITS#5340 (REP_ENTRY_MODIFIABLE). > I do not understand syncprov's handling of REP_ENTRY_MUSTRELEASE though. > (For one thing it seems to assume that REP_ENTRY_MUSTRELEASE is set if > and only if rs.sr_entry->e_private != NULL. Which is possibly true with > back-bdb but seems a shaky assumption in general.) I have found out how to reliably reproduce this bug in our master server, and it definitely involves syncprov. I'm now working on reproducing it in a minimalistic test configuration, I'll try your patch there and report back. Rein

1 0

Re: (ITS#5452) seg fault in syncprov_op abandon on master
by hyc＠symas.com 09 Apr '08

09 Apr '08

quanah(a)zimbra.com wrote: > --On Friday, April 04, 2008 1:23 PM +0200 Hallvard B Furuseth > <h.b.furuseth(a)usit.uio.no> wrote: > >> Since it crashes on >> if ( so->s_op->o_connid == op->o_connid&& >> so->s_op->o_msgid == op->orn_msgid ) >> please try >> frame 0 >> print *op >> print *so->s_op >> and maybe >> backtrace full >> while you are at it. So far there don't appear to be any invalid pointers anywhere, so there's no clue why you hit a SEGV here. Can you reproduce this situation using valgrind? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5451) syncprov_checkpoint deadlock bugfix?
by h.b.furuseth＠usit.uio.no 09 Apr '08

09 Apr '08

Does this help? From my fiddling with ITS#5340 (REP_ENTRY_MODIFIABLE). I do not understand syncprov's handling of REP_ENTRY_MUSTRELEASE though. (For one thing it seems to assume that REP_ENTRY_MUSTRELEASE is set if and only if rs.sr_entry->e_private != NULL. Which is possibly true with back-bdb but seems a shaky assumption in general.) Index: syncprov.c =================================================================== RCS file: /repo/OpenLDAP/pkg/ldap/servers/slapd/overlays/syncprov.c,v retrieving revision 1.227 diff -u -2 -r1.227 syncprov.c --- syncprov.c 3 Apr 2008 06:43:48 -0000 1.227 +++ syncprov.c 9 Apr 2008 10:29:40 -0000 @@ -2390,7 +2390,13 @@ if ( !ap ) { if ( !(rs->sr_flags & REP_ENTRY_MODIFIABLE) ) { - rs->sr_entry = entry_dup( rs->sr_entry ); - rs->sr_flags |= - REP_ENTRY_MODIFIABLE|REP_ENTRY_MUSTBEFREED; + Entry *new = entry_dup( rs->sr_entry ); + if ( rs->sr_flags & REP_ENTRY_MUSTRELEASE ) { + be_entry_release_r( op, rs->sr_entry ); + } else if ( rs->sr_flags & REP_ENTRY_MUSTBEFREED ) { + entry_free( rs->sr_entry ); + } + rs->sr_entry = new; + rs->sr_flags = (rs->sr_flags & ~REP_ENTRY_MASK) | + REP_ENTRY_MODIFIABLE | REP_ENTRY_MUSTBEFREED; a = attr_find( rs->sr_entry->e_attrs, slap_schema.si_ad_contextCSN ); -- Hallvard

1 0

Re: (ITS#5457) bdb_add() does not always release entries
by hyc＠symas.com 09 Apr '08

09 Apr '08

h.b.furuseth(a)usit.uio.no wrote: > Full_Name: Hallvard B Furuseth > Version: HEAD, 2.3, 2.4 > OS: > URL: > Submission from: (NULL) (129.240.6.233) > Submitted by: hallvard > > > back-bdb/add.c has several code paths where the parent entry > "p" is not released - it does "goto return_results;" without > first doing > > if ( p&& p !=&slap_entry_root ) > bdb_unlocked_cache_return_entry_r(&bdb->bi_cache, p ); > p = NULL; Releasing entries is somewhat irrelevant, since all locks are released when the transaction commits. Note that this function is a no-op in proto-bdb.h. All of those statements are just relics from the first entry cache design, before we switched to using BDB locks for everything. Probably could stand to do some cleanup. The diffs would be quite drastic though. Again, this may be something that's best deferred until 2.3 is dropped. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#5457) bdb_add() does not always release entries
by h.b.furuseth＠usit.uio.no 09 Apr '08

09 Apr '08

Full_Name: Hallvard B Furuseth Version: HEAD, 2.3, 2.4 OS: URL: Submission from: (NULL) (129.240.6.233) Submitted by: hallvard back-bdb/add.c has several code paths where the parent entry "p" is not released - it does "goto return_results;" without first doing if ( p && p != &slap_entry_root ) bdb_unlocked_cache_return_entry_r( &bdb->bi_cache, p ); p = NULL; The code looks like it expects that to be done below the return_results lablel, since it sets p to NULL before the goto. I don't know how the bdb cache code works though. Also I imagine the 'if ( !bvmatch( &pdn, &p->e_nname ) )' branch should check that p != &slap_entry_root before releasing p. Not sure when the parent needs to be freed. By ITS#3671 entries should be released before doing network I/O - but maybe the parent must be held until the slap_graduate_commit_csn()? I don't know what that does. Is it bdb_cache_add(,,op->ora_e,,,) or the following TXN_COMMIT which makes it necessary to bdb_cache_return_entry_r the entry? HEAD and RE24 only do bdb_cache_return_entry_r( bdb, oe, &lock ); if rs->sr_err == LDAP_SUCCESS at the return_results label, but rs->sr_err can change after the bdb_cache_add(). (A bit hard to read since the entry is referred to as both 'oe', op->ora_e and op->oq_add.rs_e. Not sure what the 'oe' variable is used for anyway.)

1 0

ITS#5378 delta-sync issues
by hyc＠symas.com 09 Apr '08

09 Apr '08

A fix is in HEAD. delta-syncrepl will now fallback to regular refresh in additional cases - LDAP_ALREADY_EXISTS, which may occur for old Adds, LDAP_NO_SUCH_OBJECT, for cases already covered in ITS#5376, LDAP_NO_SUCH_ATTRIBUTE, for old Modifies, and LDAP_TYPE_OR_VALUE_EXISTS also for old Modifies. Pretty sure that covers all of the possible out-of-sync cases. Please test. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5409) paged results invalid sizelimit error
by hyc＠symas.com 08 Apr '08

08 Apr '08

Thanks for the test program. I was able to compile it with JDK 1.5 and reproduce the error in both 2.3.41 and in HEAD. This is now fixed in RE23 and HEAD. Phoebe Shao wrote: > when connection pooling is off: > > if page size = 1000 > test 1 failed at the second iteration > test 2 failed at the third iteration > > if page size = 999 > test 1 always works > test 2 failed at the second iteration > > > I have not tried other tests where expected result is other number. > > > phoebe > > > > ----- "Howard Chu"<hyc(a)symas.com> wrote: > >> pshao(a)zimbra.com wrote: >>> ------=_Part_3887_1760018525.1205731478084 Content-Type: >> text/plain; >>> charset=utf-8 Content-Transfer-Encoding: 7bit >>> >>> >>> I did more testing and found there is more to the bug, page size is >> not the >>> only factor. The number of results in the next search also >> matters. >>> However, in all cases, if JNDI connection pooling is disabled, then >> it >>> always works. >>> >>> Test description: >>> >>> create 10000 account: user-1 to user-10000 search size limit = 5000 >>> >>> test 1: step 1. search with filter (uid=*user*): expect >>> SizeLimitExceededException step 2. search with filter >> (uid=*user-9999*): >>> expect 1 entry >>> >>> test 2: step 1. search with filter (uid=*user*): expect >>> SizeLimitExceededException step 2. search with filter >> (uid=*user-10*): >>> expect 112 entries >> Please be more descriptive, re: intermittent failure - how many >> iterations of >> a test a required (on average) before the problem shows up? >> >> -- >> -- Howard Chu >> Chief Architect, Symas Corp. http://www.symas.com >> Director, Highland Sun http://highlandsun.com/hyc/ >> Chief Architect, OpenLDAP http://www.openldap.org/project/ > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

← Newer
1
...
12
13
14
15
16
17
18
19
20
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs April 2008