[Buildroot] Buildroot runtime test infrastructure prototype
Andreas Naumann
dev at andin.de
Tue Jun 30 20:26:28 UTC 2015
Hi Thomas, Jeremy,
here is my first shot:
https://bitbucket.org/anaum/br_robot
For now I have implemented building a config and running the ext2/3/4
filesystem checks. Didnt get around doing the qemu part yet, but it
should be enough to get the idea. Just follow the README.
Install the RIDE editor if you can, even though it crashes sometimes it
makes the structure very clear and provides additional debug output
while executing the tests.
Also I have not invested in test documentation, but look at the
report/log.html and see if that's suitable. Comments on other
interesting features below...
>> syntax you have to learn, while Python is known by a large number of
>> people already.
True to some extend, and I must admit that the variable syntax takes a
little getting used to.
But besides that the API documentation for the libraries is very good
and what's more, if you use RIDE editor to write the testcases (which I
do almost exclusively), you hardly have to worry about the syntax. It
works a bit Excel-like, suggests keywords and shows their documentation
in a tooltip when pressing CTRL). Helps in navigation from testcases to
keyword implementation, does global rename...
> pretty good, RFW can report in an xunit-compatible xml which can be
> easily parsed by whatever tool you prefer. I have been autogenerating
> reports with it for quite some time
I'd be interested, how is that different from the report.html that you
mention below? What tools are you using?
> RFW also generates some HTML pages ready to be pushed on a server,
> but that's less usefull for the buildroot use-case.
On feature that I really like is the html linkage of the test report to
the log, so often it's possible to find the problematic spot without
digging through ascii-logfiles. Have a look at that and maybe make a
testcase fail.
>
>> Can it run tests in parallel?
>
> no, RFW core has no parallel testing capabilities by itself. There
> are plugins to do that, though...
Well I havnt looked at any parallel plugin, but you can start the pybot
execution as often as you like. It's just a matter of defining different
tests for different runs. I think the tagging feature helps a lot with
that. E.g. each package specific tests would be tagged with their
package name and then you would have a run for each tag starting with
a*, b*, ... so about 26 threads running side by side.
Another solution could be running each suite separatly.
There's a tool called rebot which can be used to combine the results
(output.xml) of the different runs.
And there's an option in pybot to run just tests that either failed or
have not been executed in a previous run (using the previous output.xml).
>> Can we
>> easily integrate the tests with Jenkins to have them run everyday?
The jenkins plugin works great, we use it in combination with jenkins
matrix jobs to get the parallelization. At the moment we run about 10
different targets in our testbed, but they mostly do the same tests with
different configuration data. Anyway there's a job at the end that
accumulates all the results into one report.
And finally we are interested in getting more coverage in that report as
well as we would be able to provide results from real HW instead of qemu
only. So of course I'd be happy to see a way of integrating buildroots
test with our system...
regards,
Andreas
More information about the buildroot
mailing list