anomy-list

Test case misunderstandings

From: Bjarni R. Einarsson (46762@xyz.molar.is)
Date: Fri 31 May 2002 - 15:12:53 UTC


On 2002-05-31, 10:15:12 (-0300), Ret wrote:
> I think I bumped into the same problem. The only test that didn't run
> correctly was default, and the cause was something about deleting a file
> (tmp).
> What I did is delete all the files in the results directory. I run the tests
> twice and everything went smoothly after that.

THIS IS WRONG. Totally, completely wrong. :-)

The test cases work by comparing the results of your current run
with those of the last run - the results supplied with the
distribution contain the output exepected if the sanitizer works
properly.

If the result files are missing, then the test case script assumes
you are creating a new "baseline" which all latter runs are to be
compared to. So deleting everything and running the test cases
twice will ALWAYS report "success", even if the sanitizer doesn't
work at all on your machine.

The original poster's problems were if I recall correclty due to the
fact that he is running a variant of BSD, not GNU (Linux). His
shell and the various unix tools used by the test cases therefore
have slightly different output (different amounts of white space
etc.) or in some cases different paths. One of the errors was
probably due to the fact that his machine doesn't have a /bin/false
binary, which is used by the filenames test script.

All of this means his result files will always differ from those
supplied by me, whether the sanitizer works or not - the test cases
just don't work on non-GNU systems.

This is obviously a deficiency of the test-case system, but I don't
have time to set up *BSD, Solaris, HP-UX etc. in order to fix it. I
guess I should add a warning to the scripts so this stops confusing
people.

I would also be quite happy if people sent me patches for the test
case system which make it more portable accross different platforms.
:-)

-- 
Bjarni R. Einarsson                           PGP: 02764305, B7A3AB89
 46762@xyz.molar.is                -><-              http://bre.klaki.net/

Check out my open-source email sanitizer: http://mailtools.anomy.net/ Spammers, please send plenty of email to: 46845@xyz.molar.is



hosted by molar.is