[Buildroot] [PATCH v2 00/12] Support for Cargo and Go vendoring

Thomas Petazzoni thomas.petazzoni at bootlin.com
Sat Dec 19 15:35:13 UTC 2020


Here is a patch series that implements support for vendoring of Cargo
and Go packages.

As you're aware, many Cargo and Go packages describe their
dependencies in a very language-specific way, and expect such
dependencies to be downloaded by their language-specific tools.

This series builds on top of the work from Patrick Havelange
<patrick.havelange at essensium.com> who worked on the Cargo integration,
but extends it to also cover the case of Go, showing that we have a
solution solving both of these language-specific package managers. For
the Go vendoring, I've also used the indication provided by Christian
Stewart <christian at paral.in>.

Overall, the solution consists in a concept of "download post-process
helper" is introduced in the download infrastructure. These are shell
scripts that can be invoked right after the download is done, but
before the tarball is hash-checked and stored in DL_DIR. The idea is
that such "download post-process helpers" can run the
language-specific vendoring logic to fetch the package
dependencies. Thanks to this, the tarball in DL_DIR for a given
package not only contains the package source code itself, but also the
source code of its Cargo or Go dependencies. The tarball hash covers
the entire source code, the complete tarball is cached in DL_DIR, in
the primary site and backup site, and the build can fully be done

Such "download post-process helpers" are registered by means of the
<pkg>_DOWNLOAD_POST_PROCESS variable by the appropriate package

This series is also available at:


More specifically, the series goes like this:

 - PATCH 1: add the concept of download post-processing in the
   download helper support/download/dl-wrapper

 - PATCH 2: add the <pkg>_DOWNLOAD_POST_PROCESS variable to the
   package infrastructure, which gets passed down to

 - PATCH 3: add the <pkg>_DL_ENV variable to the package
   infrastructure, which allows to pass extra variables in the
   download wrapper environment. This will be used to pass extra
   variables specific to Cargo/Go vendoring.

 - PATCH 4: add a set of two helper shell functions that will be used
   by the post processing scripts

 - PATCH 5: implements the Go vendoring itself

 - PATCH 6: adds package/tinifier, which is one package that requires
   Go vendoring. I don't particularly need this package, it was just
   added because the other Go packages we have in Buildroot today
   don't need vendoring.

 - PATCH 7: introduces a cargo-package infrastructure, for both target
   and host packages

 - PATCH 8: documents the added cargo-package infrastructure

 - PATCH 9 and 10: converts ripgrep and sentry-cli to the
   cargo-package and host-cargo-package infrastructures respecitively

 - PATCH 11: implements the Cargo vendoring itself. Note that we need
   to bump the ripgrep and sentry-cli package at the same time, as
   with the vendoring their tarball contents change and therefore
   their hash changes

 - PATCH 12: rewrite the cargo-package documentation about dependency

Changes since v1:

 - Add -modcacherw when calling Go to get files read/write instead of
   read-only, to make sure they can be deleted by a "make clean". Was
   reported by Ryan Barnett.

 - Change the Cargo infrastructure to "cd" into the build directory
   before calling Cargo, instead of calling Cargo with the full path
   to Cargo.toml as this was causing issues. Was reported by Ryan
   Barnett, investigated and fixed by Patrick Havelange.

 - Call "go mod init" in the go-post-process script if go.mod doesn't
   exist. Was reported by Christian Stewart, who also provided the

 - Add a license warning about Go/Cargo packages, as they might have
   vendored dependencies that are not taken into account by Buildroot
   licensing details. Was reported by Ryan Barnett, and Yann E. Morin
   suggested this simple solution.

Best regards,


Patrick Havelange (3):
  package/pkg-cargo.mk: introduce the cargo package infrastructure
  docs/manual/cargo: document the cargo-package infrastructure
  package/ripgrep: convert to cargo infrastructure

Thomas Petazzoni (9):
  support/download/dl-wrapper: add concept of download post-processing
  package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable
  package/pkg-download.mk: add <pkg>_DL_ENV variable
  support/download/post-process-helpers: add helper function for post
    process scripts
  support/download/go-post-process: implement Go vendoring support
  package/tinifier: new package
  package/sentry-cli: convert to host-cargo-package infrastructure
  support/download/cargo-post-process, package/pkg-cargo.mk: enable
    vendoring for Cargo packages
  docs/manual/adding-packages-cargo.txt: rewrite explanation about
    dependency management

 DEVELOPERS                            |   1 +
 docs/manual/adding-packages-cargo.txt | 106 ++++++++----------
 package/Config.in                     |   1 +
 package/Makefile.in                   |   1 +
 package/pkg-cargo.mk                  | 149 ++++++++++++++++++++++++++
 package/pkg-download.mk               |   5 +-
 package/pkg-golang.mk                 |  20 ++--
 package/ripgrep/ripgrep.hash          |   2 +-
 package/ripgrep/ripgrep.mk            |  30 +-----
 package/sentry-cli/sentry-cli.hash    |   2 +-
 package/sentry-cli/sentry-cli.mk      |  24 +----
 package/tinifier/Config.in            |  10 ++
 package/tinifier/tinifier.hash        |   3 +
 package/tinifier/tinifier.mk          |  13 +++
 support/download/cargo-post-process   |  38 +++++++
 support/download/dl-wrapper           |   9 +-
 support/download/go-post-process      |  35 ++++++
 support/download/post-process-helpers |  30 ++++++
 18 files changed, 358 insertions(+), 121 deletions(-)
 create mode 100644 package/pkg-cargo.mk
 create mode 100644 package/tinifier/Config.in
 create mode 100644 package/tinifier/tinifier.hash
 create mode 100644 package/tinifier/tinifier.mk
 create mode 100755 support/download/cargo-post-process
 create mode 100755 support/download/go-post-process
 create mode 100644 support/download/post-process-helpers


More information about the buildroot mailing list