Nightly test suite cron job on lots of targets.

[Denys Vlasenko]

On Sunday 09 May 2010 00:49, Rob Landley wrote:
> Impact Linux has an 8-way server to run bilds on, and it wouldn't be that much 
> work for me to use my Aboriginal Linux project* to set up a nightly test suite 
> cron job that built current busybox-git on arm and mips and powerpc and such 
> (under QEM), run the busybox test suite, and output a log of the results to a 
> web page or something.
> 
> What kind of format do you want this in?  Is just doing "make defconfig, make, 
> make test" sufficient, or is there a better option?

Please see testsuite/README.

I think the good sequence is

make defconfig
make -j<NUM>
cd testsuite
# Run verbose test
if SKIP_KNOWN_BUGS=1 SKIP_INTERNET_TESTS=1 ./runtest -v >RUNTEST.ERR 2>&1; then
	report failure
fi

Just tested: with static uclibc all pass, with glibc some fails
(at least mdev are false positives).

Also, there is scripts/randomtest[.loop] scripts which I use for
randomconfig build tests. They are not usable as-is, but can be tweaked.
I will look into it. I think I will tweak it so that it takes
source directory as $1 instead of hard-coding it as "tree",
and uses $LIBC and $CROSS_COMPILER_PREFIX from environment.

I do use randomconfig build test fairly regularly, so it should not
error out - that would be a regression.

Unfortunately, I did not run the testsuite against the resulting build,
thus there are sure tons of cases when it will fail.