[Buildroot] [PATCH v3 1/2] support/scripts/pkg-stats: add support for CVE reporting
Thomas De Schampheleire
patrickdepinguin+buildroot at gmail.com
Thu Feb 20 06:31:37 UTC 2020
On Wed, Feb 19, 2020, 22:34 Peter Korsgaard <peter at korsgaard.com> wrote:
> >>>>> "Thomas" == Thomas De Schampheleire <
> patrickdepinguin+buildroot at gmail.com> writes:
> > In the memory usage, RES first increased to ~4G, and then dropped back
> > to 2.2G. I think this is the Python garbage collector kicking in,
> > _after_ the new file is loaded.
> > So the worst case memory usage is roughly the size of two largest
> > subsequent years.
> > Doing some quick google search, I stumbled upon the 'pandas' python
> > package, which has a read_json function too. During a quick test, it
> > seemed to be more memory efficient, and the total memory size on
> > subsequent reads stayed in the 2.x GB range.
> > content = pandas.read_json('/tmp/nvd/nvdcve-1.0-2019.json.gz')
> > content = pandas.read_json('/tmp/nvd/nvdcve-1.0-2018.json.gz')
> > In the full test of pkg-stats, I still saw a peak memory usage near
> > the end, but it 'seemed' better :-)
> Or perhaps something that doesn't load the entire json structure into
> memory? Jsonslicer seems like a possible solution:
Interesting, I hadn't seen it.
Note that we'll need to make the code py3-ready because jsonslicer is
python 3.4+ only.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the buildroot