Breaking testing.sh

[Rob Landley]

On Monday 31 October 2005 20:48, Mike Frysinger wrote:
> > I'd been planning on submitting the new busybox testing to the Linux Test
> > Project people,

Sorry if I'm barking here, I'm guessing I didn't communicate my intentions 
clear.

If the Linux Test Project managed to make use of any of our tests, then A) 
they would not be running in the busybox build directory, so there's 
no .config file for them to look at, B) they would be running these tests 
against whatever is on their system (which is probably not busybox).

> > since they have so few command line tests.  It was also 
> > designed so that I could trivially run the same set of tests against some
> > other version of the command in question (like the gnu one) to check
> > compatability.

This is another case in which there may not even _be_ a busybox binary yet.  I 
wanted these tests to be portable, and flexible.

The changes have made it so that they won't run outside of the busybox 
directory, and they'll only run if you've already configured and built 
busybox.  This is breakage.  A flag to override breakage is not an excuse for 
breakage.

> > Now it's hardwired to be busybox-specific by default, for absolutely no
> > reason as far as I can see.
>
> how ?  i dont really see anything wrong with code as it is ... --force only
> controls whether an applet will be tested even if it wasnt detected in the
> busybox binary

You're assuming we're run in the busybox testing directory, against a 
configured version of busybox.  The original code went out of its way _not_ 
to have this assumption (hence the COMMAND= environment variable).

> if _BB_CONFIG_DEP is not set in the env, then --force has no meaning at all
>
> we can change config_is_set() to be like 'if .config doesnt exist, assume
> we want to test the applet'

Or we could do it the _sane_ way and assume that when you call the script you 
mean it, and have the makefile pass in a --skip flag to skip ones that aren't 
configured in.

I hate --actually-do-what-I-told-you-to flags.  It's the rough equivalent of a 
confirmation pop-up box.  When what you're about to do either does no damage 
or is easily undoable, and the alternative is to do _nothing_, the extra 
confirmation hoop to jump through is bad UI design.

By the way, adding _BB_CONFIG_DEP to each of the tests is very ugly.  Is there 
a reason this can't be determined from the binary name by default, and we 
only feed it the exceptions?

_BB_CONFIG_DEP = CONFIG_`echo $name | tr "[:lower:]" "[:upper:]"`

And in any case, the busybox-specific stuff should be in runtest, not in the 
individual tests.  I want to test standards compliance.  I _want_ to 
regularly run these tests against other implementations to make sure they're 
good tests.  Heck, if somebody wanted to run these suckers on slowaris, lsd, 
or MacOS X, that would be good....

Rob