[Buildroot] Report from the Buildroot Meeting in Berlin

[Thomas Petazzoni]

Hello,

From Friday to Sunday last week, a Buildroot Developers meeting was
organized in Berlin, right after the Embedded Linux Conference Europe.

A number of people participated to this meeting:

 - Peter Korsgaard
 - Samuel Martin
 - Romain Naour
 - Maxime Hadjinlian
 - Thomas Petazzoni
 - Arnout Vandecappelle
 - Julien Rosener (only on Friday)
 - Waldemar Brodkorb (except Sunday)
 - Vicente Olivert Riera
 - Luca Ceresoli (except Sunday)
 - Yann E. Morin
 - Martin Thomas (only on Friday morning)

The meeting was kindly hosted by IN-Berlin, a local non-profit
Internet provider. Special thanks to Sven Guckes from IN-Berlin, who
really helped organizing the event.

Thanks also to our sponsors:

 - Essenssium/Mind, who sponsored the Friday dinner for all the
   participants!

 - Free Electrons, who sponsored the event by making a donation to the
   IN-Berlin organization, who hosted the event.

In total, 161 patches have been merged during the meeting. The
interesting changes merged have been:

 - Finally got rid of the texinfo dependency for gdb and binutils.

 - Added new options for the MIPS architecture to select specific
   cores. Patches from Vicente.

 - Merged the packages for vexpress-firmware and arm-trusted-firmware,
   useful for ARM64 platforms. Patches from Joao Pinto.

 - Merged many patches adding $(TARGET_MAKE_ENV) when calling
   $(MAKE). Patches from Gustavo Zacarias.

 - Removed entirely the BR2_DEPRECATED mechanism. Now, when a feature
   needs to be removed, it is removed immediately, and the
   Config.in.legacy mechanism warns the user. Patches from Arnout
   Vandecappelle.

 - Merged the multi-BR2_EXTERNAL patch series, at last. This allows to
   have multiple BR2_EXTERNAL directories listed in the BR2_EXTERNAL
   variable. Patches from Yann E. Morin.

On Friday, we had a long discussion session, on various topics,
prepared by a few participants. The discussion is summarized in the
Wiki at http://elinux.org/Buildroot:DeveloperDaysELCE2016#Report. The
topics covered were:

 - Optimizing the Git download method to avoid re-fetching the same
   data over and over again. This topic was raised by Julien Rosener,
   and discussed during a long time. A general design has been
   discussed, and should now be implemented.

 - Improving the developer experience when using _OVERRIDE_SRCDIR or
   "local" packages. Currently, since the source code is copied in the
   build directory, the build errors and warnings refers to the build
   directory. That's not convenient for users running an IDE, since
   clicking on the warning or error does not bring to the correct
   file. After much discusion, it appears that the only reasonable
   solution is to implement per-package out-of-tree build.

 - Multi BR2_EXTERNAL. Yann presented the motivation for this feature,
   and then the current state of the patches.

 - Runtime testing infrastructure. Thomas Petazzoni presented the
   prototype he has so far, showed a few examples of test cases.

 - Security. We had a long discussion around providing LTS releases
   for Buildroot, the effort behind it, and tooling to track for CVEs
   that apply to our packages. The conclusion was that we should first
   implement the tools to track for CVEs, and once that's done, we'll
   see what we can do for LTS releases.

 - Google Summer of Code. Whether we should candidate or not in
   2017. General opinion is that we should candidate, despite the
   failure the last two years.

Other topics have been discussed during the meeting, while reviewing
patch series:

 - Toolchain wrapper and CFLAGS. A patch series from Jan Heylen that
   changes the toolchain wrapper to also wrap 'ld' triggered a fairly
   lengthy discussion on how we handle CFLAGS. Some of them are
   encoded in the wrapper, some are not, with no clear
   definition. After this long discussion, we came to the conclusion
   that wrapping 'ld' was not needed (packages should not use 'ld'
   directly), and that the wrapper/CFLAGS handling should be reworked
   as follows:

   * Combine BR2_TARGET_OPTIMIZATION and BR2_TARGET_LDFLAGS into a
     single BR2_TARGET_EXTRA_FLAGS

   * Have a make variable called TARGET_WRAPPER_FLAGS that includes
     all flags we want to include in the wrapper:

      * $(TARGET_ABI)
      * $(TARGET_CPPFLAGS)
      * $(BR2_TARGET_EXTRA_FLAGS)
      * Custom flags added by package/Makefile.in

     $(TARGET_OPTIMIZATION) and $(TARGET_DEBUGGING) are *NOT* in the
     wrapper.

   * TARGET_CFLAGS = $(TARGET_WRAPPER_CFLAGS) $(TARGET_OPTIMIZATION) \
                     $(TARGET_DEBUGGING)
     TARGET_CXXFLAGS = $(TARGET_CFLAGS)
     TARGET_LDFLAGS = $(TARGET_CFLAGS)
     TARGET_FCFLAGS = $(TARGET_CFLAGS)

   * The wrapper, instead of having special cases for different flags,
     just receives (at build time), the list of hardcoded flag. A
     special logic needs to be put in place to handle conflicting
     flags (mcpu, march, mtune, mfloat-abi, etc.), for which the
     command-line passed flag must be preserved if passed.

   Arnout has volunteered to do this work.

 - The patch series from Samuel doing RPATH cleanup and other related
   RPATH activities was discussed. This patch series does multiple
   things at once, which makes it a bit lengthy and scary. It was
   generally agreed that handling each topic separately would be
   better. The different topics are:

    - Misc cleanups and improvements in the main Makefile to ensure
      the build is done from a canonical path. These patches were
      almost ready, so we suggested to Samuel to only resend those 3-4
      patches, and get them merged. Samuel did so by the end of the
      meeting.

    - Tweak the RPATH of host binaries and libraries so that they are
      relative instead of absolute. This currently requires more than
      700 lines of complicated shell script, which nobody really
      likes. Instead, it was suggested to add such a feature directly
      to the "patchelf" utility, and contribute it upstream. Then this
      step could just be turned into calling patchelf on all binaries.

    - Tweak the RPATH of target binaries and libraries so that they
      don't contain RPATH referencing paths of the build machine. This
      also uses the same complicated shell script logic. It turns out
      that patchelf already has a feature to filter rpaths based on a
      list of "allowed prefixes". This can be extended with a list of
      "forbidden prefixes".

    - Detect other "leaks" of build machine paths into the target,
      simply by grepping in all files. This part looked generally
      good, and we probably want more checks at the end of the build
      (that libraries needed by programs and other libraries are
      present, that all files are built for the right architecture,
      etc.).

 - Discussed the patch from Eric Le Bihan adding the meson build
   system. This build system is not used by any package in Buildroot,
   and we do not foresee any package using it in the near
   future. Therefore, we decide to reject the patch, as we don't want
   to support build system that are not even used/tested within
   Buildroot itself.

 - We also discussed the patches from Eric Le Bihan adding support for
   the Rust language. Arnout did a detailed review of the patches, but
   mainly, we decided that such tools (compilers like rust or go)
   should have a user-visible Config.in.host entry.

 - Following a patch from Gustavo Zacarias adding a python-libxml2
   package that uses the same source as the libxml2 package, but only
   to build the Python libxml2 module, we discussed whether we should
   start adding Config.in options for host packages.

   Generally, the opinion is that we should start doing this. Of
   course, most of those options would remain blind options, so there
   is no user-visible change. But there are many reasons for which
   this is desirable. People interested in this topic are welcome to
   help.

Besides those things that have been merged, a number of topics have
been worked on:

 - The testing infrastructure proposed by Thomas has been analyzed by
   Luca Ceresoli, and then Maxime Hadjinlian took over to improve the
   Python code written by Thomas. The conclusion of this is that Luca
   and Maxime believe the testing infrastructure is generally good,
   and we should go with it. At the end of the meeting, Maxime was
   finalizing his Python cleanup pass.

   In parallel, Peter Korsgaard looked at replacing the telnet usage
   to interact with Qemu by just the standard input/output, using the
   pexpect module. This is needed to allow running tests in parallel
   (the test infrastructure currently runs Qemu with the telnet port
   on TCP 1234, which doesn't allow multiple instances of Qemu to run
   at the same time).

 - autobuild.buildroot.org SQL database. The SQL database at
   autobuild.buildroot.org not only contains the build results, but
   also the configuration of each build result. This is supposed to
   allow queries like "which sucessful builds in the last month had
   BR2_PACKAGE_FOO=y". Unfortunately, the table storing the per-build
   result configuration is enormous (the entire database is more than
   50 GB) and the queries take ages. Thomas discussed the problem with
   Arnout and Maxime, and they came up with a possible new database
   schema to solve this issue. Maxime is working on this topic.

 - Splitting the toolchain-external package. This is a topic that
   Thomas started working on a while ago, and Romain Naour has now
   taken over the work. He rebased Thomas work (which required a huge
   effort), and then started fixing the remaining problems.

   Thanks to this, we hope to simplify a bit the external toolchain
   logic, and also to allow external toolchains to be provided by
   BR2_EXTERNAL.

 - Adding a Qemu defconfig for the NIOS2 architecture has been started
   by Romain Naour and Waldemar Brodkorb. Indeed, support for this
   architecture was contributed in Qemu upstream, and we can therefore
   now use it in Buildroot.

 - The CMake handling had a few issues, which Samuel and Maxime worked
   on. Especially around setting the BUILD_TYPE, and how flags are
   passed to CMake packages. A number of iterations have been posted
   on the mailing list, with lots of review from Arnout. Hopefully,
   the latest set of patches posted on the mailing list can be merged.

I'd like to thank all the participants, but also our sponsors who made
this event possible!

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com