[Buildroot] [PATCH v3 4/5] support/scripts/pkg-stats-new: add latest upstream version information
Peter Korsgaard
peter at korsgaard.com
Thu Apr 5 08:56:43 UTC 2018
>>>>> "Ricardo" == Ricardo Martincoski <ricardo.martincoski at gmail.com> writes:
Hi,
> There is a lot of tutorials and articles in the wild saying this is the way to
> go. After some digging online I think most of these articles are incomplete.
> This seems to be a more complete article about these modules:
> https://christopherdavis.me/blog/threading-basics.html
> But then I tested the module multiprocessing.
> IMO it is the way to go for this case.
> See below the comparison.
> 1) serialized requests:
> - really simple code
> - would take 2 hours to run in my machine
> 2) threading + Queue:
> - lots of boilerplate code to work properly
> - 20 minutes in my machine
> 3) multiprocessing:
> - simpler code than threading + Queue
> - 16 minutes in my machine
> - 9 minutes in the Gitlab CI elastic runner:
> https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/60290644
> The demo code is here (a commit on the top of this series 1 to 4):
> https://gitlab.com/RicardoMartincoski/buildroot/commit/dc5f447c30157499cd925c9e79c7bc9c29252219
> Of course, as any solution, there are some downsides.
> - Pool.apply_async can't call object methods. There are solutions to this using
> other modules, but I think the simpler code wins. We just need to offload the
> code that runs asynchronously to helper functions. Yes, like you did in a
> previous iteration of the series.
> - more RAM is consumed per worker. I did a very simple measurement and htop
> shows 60MB per worker. I don't think it is too much in this case. I did not
> measured the other solutions.
> Can we switch to use multiprocessing?
I'm far from a Python expert, but it certainly sounds sensible to me! Thomas?
--
Bye, Peter Korsgaard
More information about the buildroot
mailing list