[Buildroot] Buildroot hackathon day 3 highlights

Thomas Petazzoni thomas.petazzoni at bootlin.com
Tue Apr 3 12:09:37 UTC 2018


The Buildroot hackathon continued and finished on its third day, which
was shorter than the others as most of the participants headed home
towards the end of the afternoon/beginning of the evening.

Nonetheless, some useful progress has been made:

 - By far and large, the most significant achievement of the day, and
   perhaps of the entire hackathon, is the merge of the patch series
   switching to per-package folders for the download directory, and
   the introduction of caching for Git downloads. This is the result
   of work done by Maxime Hadjinlian and Yann E. Morin, started during
   the Buildroot Summer Camp 2017, and which had seen several
   iterations since then, including one from Peter Seiderer. Yann and
   Maxime worked hard on this last day of the hackathon to iron out
   the remaining issues, sending several iterations.

   I'll try to send a separate e-mail to describe in more details what
   it is and what is the visible impact for the user.

 - The iqvlinux bump was sitting in patchwork for a while, due to the
   fact that upstream names the tarball iqvlinux.tar.gz (with no
   version included), which doesn't work well with our download/hash
   infrastructure. Carlos proposal was to make a direct call to the
   download wrapper in a pre-download hook to be able to rename the
   file during the download, but the fact that it was using internals
   of the download infrastructure was not very nice, and indeed such
   internals were being changed at the same time by Yann and Maxime
   for the Git caching. Thefore, after discussion with Peter, we opted
   for hosting an iqvlinux-<version>.tar.gz tarball on
   sources.buildroot.net, and let the package use that.

 - Arnout continued his work on the Qt5 patches, and committed the
   bump to Qt 5.10, together with some preparation patches:

   qt5script: correct license information
   package: add libnss host package
   package: add libnpsr host package
   qt5: bump latest version to 5.10.1

 - Arnout and Thomas had a look at the patch series from Carlos Santos
   adding support for nmap in ncat, netcf and libvirt. As a result of
   this, Thomas reworked the nmap patch up to the point where it could
   be comitted, and sent a number of review comments for the netcf and
   libvirt patches.

 - Peter investigated two separate hash mismatch issues, which we
   discussed quite at length.

   The first one happened on tarballs that we generate after doing a
   Git clone, and was exhibited by the recently merged gst1-interpipe
   package. This package has long file paths, which are encoded
   slightly differently in tar versions < 1.27 and tar >= 1.27. To fix
   this, we raised the minimum required tar version to 1.27, which
   means that machines having a tar older than 1.27 will now build
   their own host-tar. Ideally, this should only be done when
   Buildroot does git/bzr/hg/cvs/svn downloads, but that could be
   optimized later.

   The second one happened with Github-generated tarballs, which have
   changed subtly over time (it was noticed on the "runc"
   package). Unfortunately, since the tarball is generated by Github,
   there is not much we can do. sources.buildroot.net has a backup of
   the original tarball with the correct hash, so Buildroot properly
   falls back on sources.buildroot.net and the build succeeds, but it
   isn't really nice. One solution that was proposed by Thomas is that
   for Github downloads we extract the tarball, re-create it ourself,
   and store the hash of this new tarball. This way, we control how
   the tarball is created.

 - In a related discussion, we looked at
   https://bugs.busybox.net/show_bug.cgi?id=10511, which is a problem
   with wget 1.19.2 automatically uncompressing tarballs served by
   some HTTP servers. Since the problem has been fixed in wget 1.19.3,
   and therefore only affects 1.19.2 precisely, and that anyway the
   build succeeds thanks to the fallback to sources.buildroot.net, we
   decided to not change anything in Buildroot to cover what really is
   a wget bug.

 - Romain worked on the LLVM patch series, taking into account (some
   of) the comments made by Thomas the day before. Thomas and Romain
   then did a live review of the new version, and things are going in
   the right direction.

 - We also had some discussion about the fact that Linux 4.16 now
   requires bison and flex on the host. Bernd initially proposed new
   options "Needs bison" and "Needs flex", very much like we already
   have the "Needs OpenSSL" and "Needs libelf" options in the Linux
   kernel menu. However, while OpenSSL and libelf on the host are only
   needed for very specific kernel configurations, host-bison and
   host-flex will now be needed to build any kernel >= 4.16, so we
   suggested that those dependencies should be made mandatory.

   However, building host-flex and host-bison takes time, and since
   they are needed for the kernel kconfig code, building those
   dependencies will be needed even to run "make
   linux-menuconfig". This is made even worse with the fact that
   host-flex needs to be autoreconf'ed and gettextize'd, which
   requires building a whole bunch of dependencies, including the
   infamous host-gettext. Arnout suggested that perhaps we should
   handle host-bison and host-flex like host-cmake: only build them if
   the host doesn't provide them. This is obviously an interesting
   idea, but it reduces the "reproducibility" in that users would now
   be using different versions of flex and bison.

 - Arnout and Thomas also had a lengthy discussion about
   http://patchwork.ozlabs.org/patch/863021/, an apparently simple
   patch, but that is papering over a more serious problem: our
   handling of compiler flags is messy. Some of the flags are encoded
   in the wrapper, some are passed in the CFLAGS environment variable,
   and there is real consistency. Arnout and Thomas went through a
   detailed review of what we are doing today, and sketched out a plan
   to clean this up. Thomas has some (relatively messy) notes about
   this, that he'll try to clean up and post to the mailing list.

 - Thomas continued to go through the patchwork backlog, applying a
   number of patches, old and new, on various random topics.

In total, we applied 222 patches during this 3-day hackathon, and we
reduced the patchwork backlog from ~350 patches to ~175, roughly
cutting it in half. We merged some significant patch series: rework of
the filesystem image generation, support for Git caching, Golang
package infrastructure, Qt5 bump to 5.10.

We would like to thank once again Online and Scaleway for hosting our
event at La Maison, in Paris, where we had really excellent hosting
conditions. We look forward to organizing a similar hackathon in the

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering

More information about the buildroot mailing list