[Buildroot] [RFC] Add download helper for PyPi

Yegor Yefremov yegorslists at googlemail.com
Mon Mar 6 13:32:27 UTC 2017


Hi Arnout,

On Sat, Mar 4, 2017 at 12:38 AM, Arnout Vandecappelle <arnout at mind.be> wrote:
>  Hi Yegor,
>
> On 23-12-16 08:18, yegorslists at googlemail.com wrote:
>> From: Yegor Yefremov <yegorslists at googlemail.com>
>>
>> PyPI has changed package download location. The old scheme below is
>> working only for the old package versions:
>>
>> https://pypi.python.org/packages/source/{first pkg name char}/{pkg name}
>>
>> All new packages have following scheme:
>>
>> https://pypi.python.org/packages/{hash[:2]}/{hash[2:4]}/{hash[4:]}/{filename}
>>
>> This means every time package's version is bumped, one have to change download
>> URL as well.
>
>  When bumping a package, you also have to edit the hash file as well. I don't
> see it as a big issue to change the _SITE variable.
>
>
>>  So PyPI helper takes care of handling the URL and in the future
>> version bumping will only touch package's version variable.
>
>  It's nicer if things are explicit - that's one of the coding style aspects of
> Buildroot: explicit is better than magic behaviour.
>
>  In addition, since this script is a Python script, we would really depend on
> Python on the host, even just for a download. That's something that I'd prefer
> to avoid. If we do that, we could just as well use bitbake instead of make :-P

Can you really imagine life without Python? :-)

>  Note BTW that I also don't like the github helper much.
>
>
>  I think it would be more useful to update the scanpypi script so that it
> assists in bumping an existing package. That still relieves the burden for the
> package bumper, but the _SITE variable still contains the explicit URL. This
> bumper helper could then also update the hash file automatically.

That would also do the job. I have it on my TODO list, but I still
couldn't my hands on it. Patches are welcome :-)

In this case the names for the packages should be different for those
you would like to create a new package from. For update one will have
to specify the exact BR package name, i.e. python-tornado and not
tornado. This is needed because not all package have the same name as
on PyPI. Especially such packages, that are not modules like circus or
supervisor.

Yegor

>> pypi-dl-url.py reads package's JSON file and extracts download URL according
>> to the given version.
>>
>> Usage example:
>>
>> PYTHON_PYTZ_SITE = $(call pypi,pytz,$(PYTHON_PYTZ_VERSION),tar.bz2)
>>
>> Signed-off-by: Yegor Yefremov <yegorslists at googlemail.com>
>> ---
>>  package/pkg-download.mk         |  3 ++
>>  support/download/pypi-dl-url.py | 69 +++++++++++++++++++++++++++++++++++++++++
>>  2 files changed, 72 insertions(+)
>>  create mode 100755 support/download/pypi-dl-url.py
>>
>> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
>> index cfc550e..676f8f6 100644
>> --- a/package/pkg-download.mk
>> +++ b/package/pkg-download.mk
>> @@ -55,6 +55,9 @@ domainseparator = $(if $(1),$(1),/)
>>  # github(user,package,version): returns site of GitHub repository
>>  github = https://github.com/$(1)/$(2)/archive/$(3)
>>
>> +# pypi(package, version, packer extention): returns site of PyPi download location
>
>  extension
>
>> +pypi = $(shell support/download/pypi-dl-url.py $(1) $(2) $(3))
>> +
>>  # Expressly do not check hashes for those files
>>  # Exported variables default to immediately expanded in some versions of
>>  # make, but we need it to be recursively-epxanded, so explicitly assign it.
>> diff --git a/support/download/pypi-dl-url.py b/support/download/pypi-dl-url.py
>> new file mode 100755
>> index 0000000..e9d36bd
>> --- /dev/null
>> +++ b/support/download/pypi-dl-url.py
>> @@ -0,0 +1,69 @@
>> +#!/usr/bin/python
>
>  Typically /usr/bin/env python (to support non-default but in-PATH python
> installations).
>
>> +
>> +# vim: tabstop=8 expandtab shiftwidth=4 softtabstop=4
>> +
>> +import sys
>> +import json
>> +try:
>> +    from urllib.request import urlopen
>> +    from urllib.error import HTTPError, URLError
>> +except ImportError:
>> +    from urllib2 import urlopen, HTTPError, URLError
>> +
>> +
>> +def pypi_get_metadata(pkg_name, version):
>> +    '''Get package release metadata from PyPI'''
>> +
>> +    metadata_url = 'https://pypi.python.org/pypi/{pkg}/json'.format(pkg=pkg_name)
>> +    try:
>> +        pkg_json = urlopen(metadata_url).read().decode()
>> +    except HTTPError as error:
>> +        print('ERROR:', error.getcode(), error.msg)
>> +        print('ERROR: Could not find package {pkg}.\n'
>> +              'Check syntax inside the python package index:\n'
>> +              'https://pypi.python.org/pypi/ '
>> +              .format(pkg=pkg_name))
>> +        raise
>> +    except URLError:
>> +        print('ERROR: Could not find package {pkg}.\n'
>> +              'Check syntax inside the python package index:\n'
>> +              'https://pypi.python.org/pypi/ '
>> +              .format(pkg=pkg_name))
>> +        raise
>
>  This functionality already exists in scanpypi so should be refactored with it.
>
>
>  Regards,
>  Arnout
>
>> +
>> +    ver_metadata = None
>> +    try:
>> +        ver_metadata = json.loads(pkg_json)['releases'][version]
>> +    except KeyError:
>> +        print('ERROR: Could not find release {ver}.\n'
>> +              .format(ver=version))
>> +        raise
>> +
>> +    return ver_metadata
>> +
>> +
>> +if __name__ == '__main__':
>> +    if len(sys.argv) != 4:
>> +        print('Wrong command line arguments number.\n')
>> +        print('Please supply package name, version and file extention.\n')
>> +        sys.exit(1)
>> +
>> +    metadata = None
>> +    try:
>> +        metadata = pypi_get_metadata(sys.argv[1], sys.argv[2])
>> +    except:
>> +        sys.exit(1)
>> +
>> +    br_pypi_url = ''
>> +    full_pkg_name = '{name}-{ver}.{ext}'.format(name=sys.argv[1],
>> +                                                ver=sys.argv[2],
>> +                                                ext=sys.argv[3])
>> +    for download_url in metadata:
>> +        if 'bdist' in download_url['packagetype']:
>> +            continue
>> +
>> +        if download_url['url'].endswith(full_pkg_name):
>> +            br_pypi_url = download_url['url'][:-(len(full_pkg_name) + 1)]
>> +            break
>> +
>> +    print(br_pypi_url)
>>
>
> --
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF


More information about the buildroot mailing list