[Buildroot] Buildroot runtime test infrastructure prototype

Tue Jun 30 16:06:05 UTC 2015

I am replying to an earlier mail of Thomas because there were 
unanswered questions in there. see below...

----- Mail original -----
> Jeremy,
> 
> On Fri, 26 Jun 2015 18:20:33 +0200 (CEST), Jeremy Rosen wrote:
> 
> > more answers in the mail below, but we have a trainee working on an
> > automated test framework based on "Robot Framework" this summer and
> > we planned to upstream it when we have something to show...
> 
> In fact, when I started working on this runtime test infrastructure,
> I
> also posted on Google Plus about the available frameworks, and one of
> the answers was Robot Framework. So I had a quick look, and
> definitely
> could not understand how it can work: you are apparently not writing
> the tests in some real programming language, but in some sort of
> weird
> "tabular" format. I really don't understand how you can express
> complicated test scenarios with such a limited language.
> 

There is the possibility to do loops in RFW (see the "continue for loop"
and related keywords in the builtin library) but complicated things are
usually implemented as python modules (or other languages, but mainly
python).

Note that the philosophy of RFW is to implement building blocks as 
libraries and assamble them into actual tests using the tabular language.

Tests suites tend to be lots of very similar tests with slight variation
and the tabular language is designed to make that kind of tests easy
to read an write

The existing libraries are pretty handy to implement all the common tasks
of system testing, like starting/stoping/checking processes, opening network 
connections and crafting packets, using a SSH collection, dealing with
XML to talk to an application etc... Python can do that too, of course
(RFW is coded in python after all) but RFW does all the heavy lifting
for us. Using raw python or a lower level python framework would
probably mean reinventing the wheel in that regard...

The point is that the high level language is very readable, esp for non-
coders, which allows autogenerated documentation of test. 

> The Python unittest stuff just runs Python code, so you can express
> whatever complicated test logic you want.
> 
> > Our approch was mainly to have each package (on the buildroot side
> > have its own RFW scriptlet and have buildroot assemble these
> > scriptlets at make time to provide a test for the resulting system
> > that would only contain the tests for whatever package was actually
> > compiled in.
> 
> At some point I indeed though of having some per-package test cases
> directly in package/<foo>/, for each package. But you also anyway
> need
> to describe a complete Buildroot configuration for each test, in
> order
> to have a complete system that actually boots and make sense.
> 

yes and no... your approch is targeting the autobuild system, I was more
thinking of providing tests for a user's own configuration or a 
semi-randomly generated config. Easily testing on real hardware was
also part of the idea. 

If we want to combine the two, there would be a need for a base, bootable
config (for example based on qemu) with a randomized overlay.

The per-package part would help mainly with the randomly added packages.
That would not prevent to also have a package independant base of test
for the most basic functionalities. Per-Package also has the advantage
that the tests will be maintained with the packages, thus by the people
with the knowhow of what is being tested.

overall I don't really see a reason not to have both approch. they are
not really exclusive...

now, one of the big difference is that you have host tests and target
tests intermixed. That is a bit problematic if one wants to use your
framework on a real board since the part responsible for starting 
the card and login is intermixed with the test. Again, my use-case is
slightly different from yours. Having host-side tests is interesting 
and something we hadn't thought of. It should not be very hard to add
with a RFW approch if we go in that direction

separating the target and host test (whatever approch is taken) seems
like a good idea to me, a way to override/disable the boot/login part
would allow users to run your tests on real hardware. That seems like
a good idea independantly of the framework used.

> And in fact, testing packages is not the only target of this "runtime
> test infrastructure". I also want to validate core features like
> rootfs
> overlay, post-build script, users/permission/device table, the myriad
> possibilities of Linux kernel building, global patch directories,
> etc.
> 

Ok, so you want to test not just the image generated by buildroot but
also buildroot itself. That is different than what we have in mind,
but I can see the need for that. A complete framework would have to 
deal with both.

Again, I think it is important to separate target testing from 
host/buildroot testing. The former can be qemu but might also be
real hardware. In the case of real hardware buildroot can't guess
how to upload and boot the image, but there is still things to 
do. Either by sending the tests to the target or by running them
through a remote connection (serial or ssh) 

> 
> > At this point it would be possible for us to switch back to any
> > test
> > framework if RFW is not the one the buildroot project wants to use
> > but I personally think that RFW is really a good framework for high
> > level system testing (opposed to single-software unit testing) it
> > is very simple to learn by looking at examples, the keyword approch
> > is very readable and adding our own modules in python is fairly
> > easy
> 
> Do you have examples on what the RFW test cases for Buildroot look
> like?
> 
> I've released my code with the intent that others can have a look and
> get a clear view of what the prototype looks like. Without seeing how
> it looks with RFW, I can't really make up my mind on whether it is a
> good alternate solution or not.
> 

Denis reimplemented an infrastructure to redo your tests. The complete
infrastructure can be found here :

https://github.com/etkaDT/Rfw-buildroot-tests

Note that this is still a quick and dirty job that is not integrated
into buildroot, but it can start the custom qemu to test things on
the target. A better integration could lead to easier keywords etc.

Denis also included the output of running the python package test in
the output/ subdirectory. The RFW specific XML is quite interesting
because it contains the complete decomposition of keywords and how
they failed (there is a failed test in the example)

(Denis is the Open Wide trainee working on buildroot this summer,
he is the one pushing the scanpypy patch and the related 
robotframework package)

We hope this helps with the overall debate. 

> Best regards,
> 
> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux, Kernel and Android engineering
> http://free-electrons.com
> 