[Buildroot] [RFC] Continuous integration

Matthew Weber mlweber1 at rockwellcollins.com
Fri Nov 8 19:35:46 UTC 2013

Jérôme Pouiller <j.pouiller at expemb.com> wrote on 11/07/2013 07:37:21 AM:

> From: Jérôme Pouiller <jezz at sysmic.org>
> To: Matt Weber <mlweber1 at rockwellcollins.com>
> Cc: buildroot at busybox.net
> Date: 11/07/2013 07:37 AM
> Subject: Re: [Buildroot] [RFC] Continuous integration
> Sent by: Jérôme Pouiller <j.pouiller at expemb.com>

> > I fixed a couple items that may have been bugs (or possibly just how I
> > used the script).  I current have been running into an issue where
> > after git updates are detected, previously built pkgs try to rebuild
> > without doing a clean. I see where the conditional is that should be
> > executing the clean but haven't tracked down why I'm not hitting that
> > path yet.
> By the way, it's difficult to find the right place to do a clean. 
>  - Between each package, it take to much time.
>  - After any changes on git. But, for packages with many dependencies 
> (like qt5webkit), it means we will have to wait at least 3h per 
> configuration to have results. However, this solution is simple and 
> guarantee reproducibility of builds.
>  - Currently, I check if a clean happens since last build (in reality, 
> not any kind of build) of package. If it not the case, I launch make 
> clean. 
We might have a fourth semi-continuous integration scenario that prevents 
server thrashing at each check-in. 
- We run the script every night processing the current GIT tip at that 
- We clean each of the full cfg builds and let the script do a complete 
re-build.  (We do this because the individual pkg  cleans aren't 
to cleanup the pkg installs)
- Once all jobs complete we terminate the script and eventually will send
the report with status.
(I'll clean-up what I changed and get it committed.  Would you be up for 
optional runtime config options?  One might be to enable continuous 
or a one-stop script execution.)


> > I noticed there isn't a good way to restart the script after you stop
> > it, is my understanding correct?  I (temporarily) added a cleanup
> > sequence before the mainloop executes to remove history and each cfgs
> > build before proceeding.
> Normally, current implementation should work. It should populate 
> database with previous results. It is just long to launch (20min on my 
> server !). It would be better to change format of results to improve 
> launch time.
I had a chance to look at this a bit more and I removed some of the 
and made it so that the script could be used to do a single "one-shot" 
of the current GIT.  (I'll push this once I get it cleaned up)

> > I think there might be a bug with detecting when a package fails to
> > build, I'll see if I can resolve this and let you know.  I haven't had
> > a package actually error yet, but have had some that never built
> > because of host dependency error or download error
> I did not noticed that.
I figured out what was going on.  It was a failure of a Dependent package. 

So in my case the busybox was failing, however the busybox package never 
failed itself.  I think this was because it was built as a Dep for other 
packages and never having it's own build attempted.  I was going to look
at this a bit more, as I think it's doing this because you flag a build 
failure for the current package build, not for a failure of a Dep.

> > I was wondering what your thoughts were for the report?  I'm guessing
> > what's currently committed was the start of something based on this
> > thread of discussion.  I could see gathering a list of package err,
> > git id, and list to build output (similarly to the autobuilder
> > results).
> List of all package errors would be too long. I think report should only 

> include package status changes (OK -> KO, N/A -> KO, but also KO -> OK). 

Yeah, I agree, something with a simple status is preferred.  Maybe with 
a cfg summary at the top and a drill down with hyper links to the package 
status (similarly to the webpage).

> Report should also report configuration changes after "yes | make 
> oldconfig".
Ok, so you're thinking additions/removals of buildroot configuration 

> Currently my main problem is execution of "yes | make oldconfig". In 
> case "y" is not a valid answer, this command will loop infinitely (and 
> "yes '' | make oldconfig" is not a good solution since i want to build 
> new packages).
Yeah, already had some fun with that :-)

Matt Weber
mlweber1 at rockwellcollins.com

More information about the buildroot mailing list