[Buildroot] Buildroot hackathon day 3 highlights
Thomas Petazzoni
thomas.petazzoni at bootlin.com
Tue Apr 3 12:09:37 UTC 2018
Hello,
The Buildroot hackathon continued and finished on its third day, which
was shorter than the others as most of the participants headed home
towards the end of the afternoon/beginning of the evening.
Nonetheless, some useful progress has been made:
- By far and large, the most significant achievement of the day, and
perhaps of the entire hackathon, is the merge of the patch series
switching to per-package folders for the download directory, and
the introduction of caching for Git downloads. This is the result
of work done by Maxime Hadjinlian and Yann E. Morin, started during
the Buildroot Summer Camp 2017, and which had seen several
iterations since then, including one from Peter Seiderer. Yann and
Maxime worked hard on this last day of the hackathon to iron out
the remaining issues, sending several iterations.
I'll try to send a separate e-mail to describe in more details what
it is and what is the visible impact for the user.
- The iqvlinux bump was sitting in patchwork for a while, due to the
fact that upstream names the tarball iqvlinux.tar.gz (with no
version included), which doesn't work well with our download/hash
infrastructure. Carlos proposal was to make a direct call to the
download wrapper in a pre-download hook to be able to rename the
file during the download, but the fact that it was using internals
of the download infrastructure was not very nice, and indeed such
internals were being changed at the same time by Yann and Maxime
for the Git caching. Thefore, after discussion with Peter, we opted
for hosting an iqvlinux-<version>.tar.gz tarball on
sources.buildroot.net, and let the package use that.
- Arnout continued his work on the Qt5 patches, and committed the
bump to Qt 5.10, together with some preparation patches:
qt5script: correct license information
package: add libnss host package
package: add libnpsr host package
qt5: bump latest version to 5.10.1
- Arnout and Thomas had a look at the patch series from Carlos Santos
adding support for nmap in ncat, netcf and libvirt. As a result of
this, Thomas reworked the nmap patch up to the point where it could
be comitted, and sent a number of review comments for the netcf and
libvirt patches.
- Peter investigated two separate hash mismatch issues, which we
discussed quite at length.
The first one happened on tarballs that we generate after doing a
Git clone, and was exhibited by the recently merged gst1-interpipe
package. This package has long file paths, which are encoded
slightly differently in tar versions < 1.27 and tar >= 1.27. To fix
this, we raised the minimum required tar version to 1.27, which
means that machines having a tar older than 1.27 will now build
their own host-tar. Ideally, this should only be done when
Buildroot does git/bzr/hg/cvs/svn downloads, but that could be
optimized later.
The second one happened with Github-generated tarballs, which have
changed subtly over time (it was noticed on the "runc"
package). Unfortunately, since the tarball is generated by Github,
there is not much we can do. sources.buildroot.net has a backup of
the original tarball with the correct hash, so Buildroot properly
falls back on sources.buildroot.net and the build succeeds, but it
isn't really nice. One solution that was proposed by Thomas is that
for Github downloads we extract the tarball, re-create it ourself,
and store the hash of this new tarball. This way, we control how
the tarball is created.
- In a related discussion, we looked at
https://bugs.busybox.net/show_bug.cgi?id=10511, which is a problem
with wget 1.19.2 automatically uncompressing tarballs served by
some HTTP servers. Since the problem has been fixed in wget 1.19.3,
and therefore only affects 1.19.2 precisely, and that anyway the
build succeeds thanks to the fallback to sources.buildroot.net, we
decided to not change anything in Buildroot to cover what really is
a wget bug.
- Romain worked on the LLVM patch series, taking into account (some
of) the comments made by Thomas the day before. Thomas and Romain
then did a live review of the new version, and things are going in
the right direction.
- We also had some discussion about the fact that Linux 4.16 now
requires bison and flex on the host. Bernd initially proposed new
options "Needs bison" and "Needs flex", very much like we already
have the "Needs OpenSSL" and "Needs libelf" options in the Linux
kernel menu. However, while OpenSSL and libelf on the host are only
needed for very specific kernel configurations, host-bison and
host-flex will now be needed to build any kernel >= 4.16, so we
suggested that those dependencies should be made mandatory.
However, building host-flex and host-bison takes time, and since
they are needed for the kernel kconfig code, building those
dependencies will be needed even to run "make
linux-menuconfig". This is made even worse with the fact that
host-flex needs to be autoreconf'ed and gettextize'd, which
requires building a whole bunch of dependencies, including the
infamous host-gettext. Arnout suggested that perhaps we should
handle host-bison and host-flex like host-cmake: only build them if
the host doesn't provide them. This is obviously an interesting
idea, but it reduces the "reproducibility" in that users would now
be using different versions of flex and bison.
- Arnout and Thomas also had a lengthy discussion about
http://patchwork.ozlabs.org/patch/863021/, an apparently simple
patch, but that is papering over a more serious problem: our
handling of compiler flags is messy. Some of the flags are encoded
in the wrapper, some are passed in the CFLAGS environment variable,
and there is real consistency. Arnout and Thomas went through a
detailed review of what we are doing today, and sketched out a plan
to clean this up. Thomas has some (relatively messy) notes about
this, that he'll try to clean up and post to the mailing list.
- Thomas continued to go through the patchwork backlog, applying a
number of patches, old and new, on various random topics.
In total, we applied 222 patches during this 3-day hackathon, and we
reduced the patchwork backlog from ~350 patches to ~175, roughly
cutting it in half. We merged some significant patch series: rework of
the filesystem image generation, support for Git caching, Golang
package infrastructure, Qt5 bump to 5.10.
We would like to thank once again Online and Scaleway for hosting our
event at La Maison, in Paris, where we had really excellent hosting
conditions. We look forward to organizing a similar hackathon in the
future.
Thomas
--
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com
More information about the buildroot
mailing list