[Buildroot] [RFC PATCH v1 1/1] package/pkg-golang: download deps to vendor tree if not present

Sam Voss sam.voss at rockwellcollins.com
Tue Sep 15 20:08:02 UTC 2020


Hey Arnout,

On Tue, Sep 15, 2020 at 2:23 PM Arnout Vandecappelle <arnout at mind.be> wrote:
>
>
>
> On 11/09/2020 00:33, Christian Stewart wrote:
> > Yann,
> >
> > On Fri, Sep 4, 2020 at 1:06 AM Yann E. MORIN <yann.morin.1998 at free.fr> wrote:
> >> Ah, because you store all the go vendored dependencies "away" of the
> >> package.
> >>
> >> I am not sure I like that, because it breaks the semantics of the dl/
> >> directory: all the csource needed by one package is in dl/foo/ and the
> >> vendored dependencies *are* part of the package.
> >>
> >> So I am absolutely not in favour of storing the go modules on the side.
> >
> > I've thought about this quite a bit and I think you're probably right,
> > for a v1 implementation in Buildroot we most likely should do the
> > approach of "go mod vendor" just before compressing the source file
> > for "make source." It should work fine for Go (can't speak for node or
> > others) because the vendor/ tree is always consistent.
>
>  Now that apparently everyone is on the same page, I think it's good to keep
> track of the conclusions, so I've created a wiki page [1].
>
>  If there is anything that needs further discussion, please continue that here
> in the list. However, if there is something I just didn't write down correctly
> or clearly enough, simply update the wiki.

Thanks for writing all of this down and creating a wiki for this. I
think this helps clarify our options and will give us room for
discussion.

>
>  For your convenience, I've copied the wiki page below.
>
>
> == Problem statement and requirements
>
> Language package managers make it possible to express dependencies and make sure
> the correct version of those dependencies are used.
>
> For Buildroot, the problem is to make sure those dependencies are downloaded.
> Sometimes the upstream package simply bundles all dependencies (this is called
> "vendored"), but sometimes they only have them by reference.
>
> Buildroot needs to make sure that:
>
> * the dependencies are downloaded;
> * the dependencies are extracted in a way that they're accessible by the
> language's build system;
> * everything is properly hashed so we are sure the correct tarballs were downloaded;
> * it is possible to do "make source", unplug the network, and do a build;
> * it is possible to use a BR2_PRIMARY_SITE and BR2_PRIMARY_SITE_ONLY;
> * it is possible to use BR2_BACKUP_SITE;
> * the information produced by "make legal-info" is correct.
>
> An additional nice-to-have is to be able to split off the open source
> dependencies from the non-redistributable dependencies.
>
>
> == Possible solutions
>
> 1. Add a download helper that downloads the base source, runs the package
> manager to vendor all dependencies, and creates a tarball of this bundle.
> 2. Add a download helper that downloads the base source, runs the package
> manager to vendor all dependencies, and creates a separate tarball containing
> only the dependencies.
> 3. Add a download helper that downloads the base source, runs the package
> manager to vendor all dependencies, splits off the non-redistributable
> dependencies, and creates a separate tarball for the open-source and the
> non-redistributable dependencies.
> 4. Add a download helper that downloads the base source, runs the package
> manager to get all dependencies, and creates a separate tarball for each
> individual dependency.
> 5. Solution 1 + extra legal-info infrastructure that makes it possible to strip
> off non-redistributable parts when generating the legal-info tarball.
>
> Solution 1 is the simplest to implement. It is the approach taken by
> http://patchwork.ozlabs.org/project/buildroot/list/?series=159771 for cargo.
>
> Solution 2 is also easy to implement. However:
>
> - the download infra has to "know" how to download the additional tarball (it
> should not be downloaded when using the normal method, but it should be
> downloaded for PRIMARY and BACKUP);
> - the additional tarball has to be added to the hash file;
> - the additional tarball has to be extracted as well.

I sent an RFC last night, which created a very crude initial
implementation of this, as I needed/wanted to leverage the atomic
saves in support/download/dl-wrapper.

After briefly chatting with Christian, we believe this could very
easily be leveraged by using `go mod vendor` instead of `cargo
vendor`.

http://lists.busybox.net/pipermail/buildroot/2020-September/292211.html

Thanks,
-- 
Sam Voss | Sr. Software Engineer | Commercial Avionics
COLLINS AEROSPACE
400 Collins Road NE, Cedar Rapids, Iowa 52498, USA
Tel: +1 319 263 4039
sam.voss at collins.com | collinsaerospace.com

CONFIDENTIALITY WARNING: This message may contain proprietary and/or
privileged information of Collins Aerospace and its affiliated
companies. If you are not the intended recipient, please 1) Do not
disclose, copy, distribute or use this message or its contents. 2)
Advise the sender by return email. 3) Delete all copies (including all
attachments) from your computer. Your cooperation is greatly
appreciated.


More information about the buildroot mailing list