[Buildroot] [RFC PATCH v1 1/1] package/pkg-golang: download deps to vendor tree if not present

Sam Voss sam.voss at rockwellcollins.com
Thu Sep 3 13:01:27 UTC 2020


Hey Thomas,

On Thu, Sep 3, 2020 at 7:00 AM Thomas Petazzoni
<thomas.petazzoni at bootlin.com> wrote:
>
> Hello,
>
> On Thu, 3 Sep 2020 05:52:56 -0500
> Sam Voss <sam.voss at gmail.com> wrote:
>
> > > My proposal was that we introduce per-package managers download
> > > backends, which would typically do something like the following
> > > (pseudo-code):
> > >
> > >     #!/usr/bin/env bash
> > >     # This file is support/fownload/go
> > >
> > >     # Parse options:
> > >     actual_site_method="$(get_option '--actual-site-method')"
> > >
> > >     # Call actual download helper:
> > >     "${0%/*}/${actual_site_method}" "${@}" -o "${temp_tarball}"
> > >
> > >     # Populate the vendor:
> > >     tar xf "${temp_tarball}" -C "${temp_directory}"
> > >     cd "${temp_directory}/${package_name_version}"
> > >     go mod vendor
> > >     cd ..
> > >     tar czf "${final_tarball}" "${package_name_version}"
> > >
> > > (of course, the details would be a bit more complex, and would require
> > > that we pass the actual site method vi the download infra, but the idea
> > > is there)
> > >
> > > What's your opinion on this?
> >
> > As we spoke about plenty on the IRC, I'm sure you know my opinion on
> > this but I figure I mention it anyway: I believe we should split these
> > recursive-requirements from the base tar. This should allow
> > Proprietary applications and TPIP requirements to be captured while
> > maintaining separation between them.
>
> For those tuning in: TPIP stands for (I believe) "Third Party
> Intellectual Property".

Sorry, that is correct, I should have clarified. Thanks for mentioning it.

>
> I think I agree with the idea of splitting into two tarballs the
> package source code itself from the source code of all its dependencies.
>
> > Now, to your point about "well, what if the repository has proprietary
> > dependencies?".
>
> That was indeed my question as well.
>
> > I think this is still a valid point, especially when
> > looking at the case you mentioned of "proprietary app with commingled
> > TPIP+proprietary requirements", but I believe we should be able to
> > handle this at a buildroot level fairly reasonably.
> >
> > I took a look at `go mod`, it seems to share a similar mechanism with
> > cargo which allows us to pass local paths for dependencies. My
> > proposition is to put the responsibility of whomever is adding a
> > proprietary application, which has mixed dependencies, to instead
> > split any proprietary modules into selectable options in buildroot,
> > and use the standard depends mechanism. To enforce this, we could
> > investigate using license-coalescing options of the package managers
> > to find any proprietary dependencies and fail if they're found to be
> > pointing to upstream URLs (and would be captured) with an error
> > message clearly describing our intentions.
>
> This feels like a reasonable approach to me.
>
> Question: for the dependencies, instead of having a single tarball for
> all of them, would there be a way of having separate tarballs for each
> dependency that gets pulled by "go mod" or "cargo" ? If so, then we
> could perhaps be able to teach the package which parts of the package
> are proprietary (including the proprietary dependencies) and which
> parts are open-source and under which license.

I don't exactly know, but to your point I know it pulls them into
correctly-named folders, however no version string attached:

VENDOR/ws2_32-sys/Cargo.toml

>
> Note that as a first step, I am personally perfectly fine with what you
> are proposing. The above question is really just a question, not a
> request to implement something like that.

Understood, I figured since I had a vendor tar I may as well take a
look at what it produced.

>
> > In my (so far unshared) patchset, my solution to do this agrees with
> > your suggestion above, by adding download-backend support for these
> > package managers. I leveraged the implementation suggested previously
> > by patrick[1] to use the cargo-dl step to then create two tarballs:
> >
> > <package>-<ver>.tar.gz   <- we're all familiar with
> > <package>-<ver>-vendor.tar.gz <- the TPIP portion
> >
> > The reasons for splitting are shared above. I believe that patchset is
> > a good initial direction, and I think those interested in this
> > patchset who are unfamiliar with that one should give it a review.
>
> Could you submit your patch series ? Did you fix the issues that we
> pointed out in the review of Patrick's series ?

I did not change Patrick's series at all, built a new change-set on
top of it. I will have to go back and integrate them together, and I
can take a look at the issues with him in the meantime.

Thanks,

Sam


-- 
Sam Voss | Sr. Software Engineer | Commercial Avionics
COLLINS AEROSPACE
400 Collins Road NE, Cedar Rapids, Iowa 52498, USA
Tel: +1 319 263 4039
sam.voss at collins.com | collinsaerospace.com

CONFIDENTIALITY WARNING: This message may contain proprietary and/or
privileged information of Collins Aerospace and its affiliated
companies. If you are not the intended recipient, please 1) Do not
disclose, copy, distribute or use this message or its contents. 2)
Advise the sender by return email. 3) Delete all copies (including all
attachments) from your computer. Your cooperation is greatly
appreciated.


More information about the buildroot mailing list