Subject: Re: Regression testing [was: A suggestion for the release notes?]
From: Sam TH (sam@uchicago.edu)
Date: Fri Dec 01 2000 - 04:52:19 CST
On Fri, Dec 01, 2000 at 11:14:46AM +0100, Mike Nordell wrote:
> WJCarpenter wrote:
> > 2. Be more systematic about regression testing.
>
> I have been thinking about this also. Is wv enough free-standing to create a
> test program for it that simply tries to read (and let wv parse) a word
> document? If so, I'd like to input a wv-test program to do just this, or
> possibly more appropriately an "AW importer test" that can be given any type
> of document AW can import.
>
> I don't know how much of AW we actually can test by small console-mode test
> programs, but at least importers should be possible.
>
> I also suggest that we create an ABW document named ReleaseProcedures that
> outlines exactly what, and especially in what order, tests we are to run
> before a new release. I also think these results should be saved for future
> reference (at least 4-6 months).
>
> Am I insane? :-)
Well, at least not with regard to this. ;)
I very much would like to have an automated regression tester. So
far, the only thing that can be done usefully on the command line is
conversion from one file format to another. So, I suggest creating a
set of documents, in various formats, and representatives of what they
should convert to. Then, we can run the tests in a shell script, and
use diff to compare the generated files with the benchmark files. If
they differ, we have a regression.
If this was set up nicely, I could very easily run this as part of the
nightly build script, which would allow us nightly regression testing.
Better yet, you can add test runs to Tinderbox, so we could see
regressions much more quickly.
For this to happen, the following needs to take place
1) Bug 1070 (which I just filed) need to be fixed, so that I can run
AbiWord on parsons. Basically, we shouldn't be loading fonts if we
aren't opening the app.
2) Someone needs to creat the set of documents and baseline documents
to use for this test suite. Most of the test document can be taken
from the test/wp directory.
3) Someone needs to write the infrastructure. We might want to do
this just in shell, which would be pretty easy, most likely. We also
might want to use something like DejaGNU [1], which would be more work,
and neccessitate someone (probably me) learning TCL.
I am willing to do step 3 on my own. I might do step 1 myself too,
just because it annoys me. And I'll help people with step 2.
And I just tested it, mass conversion is *really* quick. So this
wouldn't take to much time, even on tinderbox.
At some point, of course, we will add scripting, and get our selves a
real test suite. But that's a bit out there currently.
But what we can do now would be very good. Consider it a version 1.0
release goal.
sam th
sam@uchicago.edu
http://www.abisource.com/~sam/
GnuPG Key:
http://www.abisource.com/~sam/key
This archive was generated by hypermail 2b25 : Fri Dec 01 2000 - 04:52:21 CST