[Buildroot] github helper is broken
Arnout Vandecappelle
arnout at mind.be
Thu Nov 13 19:43:23 UTC 2014
On 13/11/14 19:23, Yann E. MORIN wrote:
> Hello All,
>
> The github helper is now broken, because GitHub changed their download
> scheme, again.
>
> The new scheme is:
> https://github.com/USER/REPO/archive/VERSION.tar.gz
>
> What is not-so-obvious in this new scheme, is that it omits the package
> name from the final component of the URU, so we can no longer rely on
> the default value of the FOO_SOURCE variable to construct the URL.
>
> So we now have to set FOO_SOURCE explicitly, which was not needed so far,
> and set it to $(FOO_VERSION).tar.gz
>
> But then, it means we would store tarballs named FOO_VERSION.tar.gz,
> which is not so nice in the end.
>
> We've discussed this on IRC, and came to three main proposals:
>
> 1) rely on the server to pass an appropriate content-disposition header
> with the filename.
>
> Unfortunately, that's not doable in Buildroot:
> - we need to know the filename prior to doing the download. We could
> use a workaround by having wget do a HEAD fetch to just get the
> filename;
> - but then this would not work for off-line builds.
>
> So this solution is a no-no.
>
> 2) differentiate upstream tarball name from local tarball name
>
> This introduces a new variable, like FOO_UPSTREAM_SOURCE, which besides
> not being nice, would require quite some work in the pkg-download infra,
> which is a bit risky that close to the release.
I think long-term this is the way to go, but indeed it can't be done this close
to the release.
>
> 3) no longer do tarball downloads from github, but do a git clone
>
> This would protect us from any future change in the GitHub tarballs
> download naming scheme. And after all, GitHub is a git hosting forge,
> so let's use it for what it is.
>
> The problems with that solution are two fold:
> - downloads might take more time than a tarball download, since a
> complete repository would probably be bigger than a single tarball;
> - we must use the http:// scheme for the URL (because, proxies), so
> all packages must now specify FOO_SITE_METHOD = git
>
> Although the download time is not much of an issue, the way we are using
> the github helper for now does not allow for setting more than one
> variable.
>
>
> In the end, solution 3 seems the most appropriate, and would require
> just a bit of easy modifications:
>
> - tweaking the github helper to emit the repository URL instead of the
> tarball URL;
>
> - add FOO_SITE_METHOD to all packages.
>
> The first one is trivial, and the second one is relatively easy with a
> bit of 'find' and some clever 'sed' experession.
>
> I have all setup and ready to implement solution 3, but before I do, I'd
> like some feedback on those proposals, and if possible new proposals
> that are easy to implement for the release.
>
>
> Notes: I thought of a few other proposals, like:
>
> - define the github helper so that it emits the necessary variables:
> $(eval $(call github,user,repo))
> would expand to:
> FOO_SITE = https://github.com/user/repo
> FOO_SITE_METHOD = git
>
> That would allow us to introduce new types of forges, like gitorious
> of bitbucket.
>
> But that's not so nice, because so far we always relied on only
> setting variables, not generating Makefile code.
I actually prefer this option. It makes it really easy to convert to option 2
later. Also I don't have a problem with generating more Makefile code.
>
> Also, it implies changing all packages as well.
This we have to do anyway. But doing the eval approach now avoids that we have
to do it again in the future.
>
> - introduce new vriables to define a new semantic for the downloads:
> FOO_FORGE = github
> FOO_FORGE_PATH = /user/repo
> which would be interpreted by the pkg-download infra to generate the
> correct URL
That's indeed another option if you want to avoid generating more Makefile
code. But I prefer generating Makefile code, because it reduces the complexity
of the generic infrastructure.
> That would also allow us to add more forge-related handlers, like
> gitorious or bitbucket (others?), but is really diverging from the
> current semantics of the variables we already have.
>
> Also, it implies changing all packages as well.
>
> So, we could not find a solution that would limit the changes to just the
> github helper; all solutions we have for now imply touching all packages
> hosted on GitHub.
That's why it's important to now make sure we never have to do something like
that - i.e., we have to use one of the options you don't want to take :-)
Regards,
Arnout
>
> Comments and suggestions most welcome! ;-)
>
> Regards,
> Yann E. MORIN.
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
More information about the buildroot
mailing list