I'm currently trying to run "make test" with back-ndb and most of the comparisons fail even though the backend is behaving correctly. 1) back-ndb always fills in all objectclasses in the objectclass chain, while the test.ldif files omit most of the superior classes. 2) back-ndb always returns attributes in the order they were defined in their corresponding objectclass definition, instead of preserving whatever order they were submitted in.
Any thoughts on how to address this?
--On Wednesday, June 04, 2008 7:35 PM -0700 Howard Chu hyc@symas.com wrote:
I'm currently trying to run "make test" with back-ndb and most of the comparisons fail even though the backend is behaving correctly.
- back-ndb always fills in all objectclasses in the objectclass
chain, while the test.ldif files omit most of the superior classes. 2) back-ndb always returns attributes in the order they were defined in their corresponding objectclass definition, instead of preserving whatever order they were submitted in.
Any thoughts on how to address this?
Hm, for #2, some sort of intelligent sort, since LDAP doesn't support the concept of ordered results, any test relying on order is fundamentally flawed. ;) That'd deal with all backends correctly, I would think.
#1 has always somewhat annoyed me, since I'd generally think you would see all relevant objectClasses included in an objects output. Is there an RFC covering it either way?
--Quanah
--
Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount writes:
Hm, for #2, some sort of intelligent sort, since LDAP doesn't support the concept of ordered results, any test relying on order is fundamentally flawed. ;) That'd deal with all backends correctly, I would think.
ldapsearch -S "" sorts by DN. But such a change cascades into large changes to the testdata files, and IIRC there are a few tests where a specific order is expected. Don't quite remember.
So when testing back-ldif I modify scripts/acfilter.sh to sort some LDIF output files before comparing them. If Perl is present I sort by DN (I think), and it could sort the lines in each entry too. Probably loses some possible errors, but not many. E.g. where an LDIF to be compared consists of the concatenated output of several commands. Without Perl it just sorts line by line. Better than nothing. If anyone cares we could write an LDIF-sort program in C.
(I'm about ready to resume my back-ldif changes now that things seem to pass make test again. Will wait for 2.4.10 to be released to avoid interfering.)
#1 has always somewhat annoyed me, since I'd generally think you would see all relevant objectClasses included in an objects output. Is there an RFC covering it either way?
Ah, ITS#5517 (add superclass of existing class fails)...
Hallvard B Furuseth wrote:
Quanah Gibson-Mount writes:
Hm, for #2, some sort of intelligent sort, since LDAP doesn't support the concept of ordered results, any test relying on order is fundamentally flawed. ;) That'd deal with all backends correctly, I would think.
ldapsearch -S "" sorts by DN. But such a change cascades into large changes to the testdata files, and IIRC there are a few tests where a specific order is expected. Don't quite remember.
Right, the glue tests expect a particular order. Don't recall if any others do.
So when testing back-ldif I modify scripts/acfilter.sh to sort some LDIF output files before comparing them. If Perl is present I sort by DN (I think), and it could sort the lines in each entry too. Probably loses some possible errors, but not many. E.g. where an LDIF to be compared consists of the concatenated output of several commands. Without Perl it just sorts line by line. Better than nothing. If anyone cares we could write an LDIF-sort program in C.
I think awk would be a better choice, since we already use it. I'll look into that.
(I'm about ready to resume my back-ldif changes now that things seem to pass make test again. Will wait for 2.4.10 to be released to avoid interfering.)
Cool, would be nice to get the back-ldif ITS's closed.
#1 has always somewhat annoyed me, since I'd generally think you would see all relevant objectClasses included in an objects output. Is there an RFC covering it either way?
Ah, ITS#5517 (add superclass of existing class fails)...
Yeah, looks like another mess. In back-ndb we have to explicitly store all values, because that's pretty much the only way to be able to find them again if searching on superclasses etc. (I guess the alternative would be to rewrite any filters to OR in all superclasses, but that would be more work on the search side of things, which we obviously want to avoid.)
Howard Chu wrote:
Hallvard B Furuseth wrote:
Quanah Gibson-Mount writes:
Hm, for #2, some sort of intelligent sort, since LDAP doesn't support the concept of ordered results, any test relying on order is fundamentally flawed. ;) That'd deal with all backends correctly, I would think.
ldapsearch -S "" sorts by DN. But such a change cascades into large changes to the testdata files, and IIRC there are a few tests where a specific order is expected. Don't quite remember.
Right, the glue tests expect a particular order. Don't recall if any others do.
So when testing back-ldif I modify scripts/acfilter.sh to sort some LDIF output files before comparing them. If Perl is present I sort by DN (I think), and it could sort the lines in each entry too. Probably loses some possible errors, but not many. E.g. where an LDIF to be compared consists of the concatenated output of several commands. Without Perl it just sorts line by line. Better than nothing. If anyone cares we could write an LDIF-sort program in C.
I think awk would be a better choice, since we already use it. I'll look into that.
This works with gawk: #! /bin/sh # $OpenLDAP: pkg/ldap/tests/scripts/acfilter.sh,v 1.13 2008/01/07 23:20:16 kurt Exp $ grep -v '^#' | awk 'BEGIN{FS="\n";RS=""} {j=0; for (i=1; i<=NF; i++){ if ($i ~ /^ /){ x[j] = x[j] "\n" $i; } else { j++; x[j] = $i } } print x[1]; delete x[1]; j=asort(x); for (i=1; i<=j; i++){ print x[i]; } delete x; print "" }'
(all as a single line of text)
Unfortunately the asort() function is a GNU extension, so I think we have to pipe to the sort command instead:
grep -v '^#' | awk 'BEGIN{FS="\n";RS=""}{j=0; for (i=1; i<=NF; i++){ if ($i ~ /^ /) { x[j] = x[j] "\n" $i; } else { j++; x[j] = $i }} print x[1]; for (i=2; i<=j; i++){print x[i] | "sort +0";} delete x; close("sort +0"); print ""}'
Howard Chu wrote:
This works with gawk: #! /bin/sh # $OpenLDAP: pkg/ldap/tests/scripts/acfilter.sh,v 1.13 2008/01/07 23:20:16 kurt Exp $ grep -v '^#' | awk 'BEGIN{FS="\n";RS=""} {j=0; for (i=1; i<=NF; i++){ if ($i ~ /^ /){ x[j] = x[j] "\n" $i; } else { j++; x[j] = $i } } print x[1]; delete x[1]; j=asort(x); for (i=1; i<=j; i++){ print x[i]; } delete x; print "" }'
(all as a single line of text)
Unfortunately the asort() function is a GNU extension, so I think we have to pipe to the sort command instead:
grep -v '^#' | awk 'BEGIN{FS="\n";RS=""}{j=0; for (i=1; i<=NF; i++){ if ($i ~ /^ /) { x[j] = x[j] "\n" $i; } else { j++; x[j] = $i }} print x[1]; for (i=2; i<=j; i++){print x[i] | "sort +0";} delete x; close("sort +0"); print ""}'
Oops. Except piping to sort loses the field boundaries, so continuation lines are messed up.
Howard Chu wrote:
Howard Chu wrote:
(all as a single line of text)
Unfortunately the asort() function is a GNU extension, so I think we have to pipe to the sort command instead:
grep -v '^#' | awk 'BEGIN{FS="\n";RS=""}{j=0; for (i=1; i<=NF; i++){ if ($i ~ /^ /) { x[j] = x[j] "\n" $i; } else { j++; x[j] = $i }} print x[1]; for (i=2; i<=j; i++){print x[i] | "sort +0";} delete x; close("sort +0"); print ""}'
Oops. Except piping to sort loses the field boundaries, so continuation lines are messed up.
Frankly I ask myself why bothering with such a mess? I know it's hard to introduce another binary dependency but I'd recommend to use a decent (readable) scripting language. Because having to maintain code like the example above will hold people back from writing detailed tests.
Ciao, Michael.
Howard Chu writes:
Oops. Except piping to sort loses the field boundaries, so continuation lines are messed up.
Also the POSIX folks in their wisdom normalized the arguments to sort. Thus GNU sort no longer accepts sort +0, you must say sort -k 1. And of course comparison obeys the current locale, which may do "interesting" things to the ordering. But then, so does awk. And several perl versions have had broken locale support. Maybe defines.sh should hunt down $LC_*, $LANG etc and unset them:-(
Anyway, I don't expect we lose much by dropping proper compares if perl is absent. So I've been doing something like this. Though that was for back-ldif where only some test outputs need to be sorted.
# With arguments -S backend,backend,...backend, also sort the input for # those backends: By entries+attrs if possible, just by lines otherwise. # That's a poor man's testdata verification for backends that return # entries in a different order than the test expects. It allows us # to at least get some verification. # if test "$BACKEND" = null ; then # Do not output anything to compare for null backend : else case $1,$2, in -S*,$BACKEND,*) grep -v '^#' | if (perl -e 0) >/dev/null 2>&1 ; then # There is a Perl in $PATH perl -ne ' if (/^ /) { $entry[-1] .= $_; } elsif (/^\r?$/) { &entry; } else { push @entry, $_; } sub entry { $dn = shift(@entry); push @out, join "", ($dn || ""), sort @entry; @entry = (); } END { &entry; print join "\n", sort @out; }' else sort fi ;; *) grep -v '^#' ;; esac fi
Howard Chu wrote:
I'm currently trying to run "make test" with back-ndb and most of the comparisons fail even though the backend is behaving correctly.
- back-ndb always fills in all objectclasses in the objectclass
chain, while the test.ldif files omit most of the superior classes. 2) back-ndb always returns attributes in the order they were defined in their corresponding objectclass definition, instead of preserving whatever order they were submitted in.
Any thoughts on how to address this?
Well, my first reaction is that a test suite based on shell-scripts makes it hard to really be LDAPv3 compliant.
The complete solution is to have schema-aware LDAP clients for testing. Obviously you need a decent programming language for the tests.
Fine-grained error checking/handling is another issue with the shell-based approach.
And I wonder whether it's worth to think about speeding up the test suite by clustering tests so that not every trivial test script has to bother with slapd configuration and loading test data. I'm aware that isolating tests from each other has some advantages too though.
Ciao, Michael.
-
Hallvard B Furuseth -
Howard Chu -
Michael Ströder -
Quanah Gibson-Mount