[Buildroot] [PATCH 01/14] support/scripts/pkg-stats: URL checking support
Matthew Weber
matthew.weber at rockwellcollins.com
Fri Sep 21 12:56:46 UTC 2018
On Thu, Sep 20, 2018 at 3:58 PM Thomas Petazzoni
<thomas.petazzoni at bootlin.com> wrote:
>
> Hello Matt,
>
> On Thu, 20 Sep 2018 08:24:46 -0500, Matt Weber wrote:
>
> > diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
> > index b7b00e8..c83545b 100755
> > --- a/support/scripts/pkg-stats
> > +++ b/support/scripts/pkg-stats
> > @@ -24,6 +24,8 @@ from collections import defaultdict
> > import re
> > import subprocess
> > import sys
> > +import time
> > +import requests # URL checking
> >
> > INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)")
> >
> > @@ -43,10 +45,30 @@ class Package:
> > self.patch_count = 0
> > self.warnings = 0
> > self.current_version = None
> > + self.url = None
> >
> > def pkgvar(self):
> > return self.name.upper().replace("-", "_")
> >
> > + def set_url(self):
> > + """
> > + Fills in the .url field
> > + """
> > + in_help_section = False
> > + self.url = "No Config"
> > + for filename in os.listdir(os.path.dirname(self.path)):
> > + if fnmatch.fnmatch(filename, 'Config.*'):
> > + fp = open(os.path.join(os.path.dirname(self.path), filename), "r")
> > + for config_line in fp:
> > + if config_line.strip() == "help":
> > + in_help_section = True
> > + if in_help_section and re.match("(.*)https?://", config_line):
> > + self.url = ''.join(config_line.split())
> > + fp.close()
> > + return
> > + self.url = "Missing Entry"
> > + fp.close()
> > +
> > def set_infra(self):
> > """
> > Fills in the .infras field
> > @@ -356,7 +378,19 @@ def boolean_str(b):
> >
> > def dump_html_pkg(f, pkg):
> > f.write(" <tr>\n")
> > - f.write(" <td>%s</td>\n" % pkg.path[2:])
> > + url_status = "Ok"
> > + if str(pkg.url) == "Missing Entry":
> > + f.write(" <td>%s<br> (URL: Missing URL)</td>\n" % pkg.path[2:])
> > + elif str(pkg.url) == "No Config":
> > + f.write(" <td>%s<br> (URL: No Config File)</td>\n" % pkg.path[2:])
> > + else:
> > + try:
> > + url_status_code = requests.head(pkg.url, verify=False).status_code
> > + if url_status_code > 308:
> > + url_status = "Error(" + str(url_status_code) + ")"
> > + except requests.exceptions.RequestException as e:
> > + url_status = e
> > + f.write(" <td>%s<br> (<a href=%s>URL: %s</a>)</td>\n" % (pkg.path[2:], str(pkg.url), url_status))
>
> Could you instead add a new column, like we already have for the
> version, package type, number of patches, etc. and use the existing
> green/orange/red colors to indicate URL present and working (green),
> URL not present (orange) and URL present but not working (red) ?
>
How does this look? (Still some bugs with some of the invalid sites,
the webservers are reporting incorrect codes....)
https://pste.eu/p/JpsH.html
> How long does it take to run the script before/after your addition, on
> all packages ? I'm sure you remember I was working on adding support
> for using release-monitoring.org to this script to keep track of
> upstream versions of packages, but doing it sequentially for the 2000+
> packages we have was way too slow and I had to use several threads to
> speed things up and make it reasonable.
>
Before adding URL checking, it takes ~2mins. I was able to keep the
runtime at 2mins 20sec by pulling in your code for threading and using
64 threads.
Matt
More information about the buildroot
mailing list