[Buildroot] [PATCH v3 4/5] support/scripts/pkg-stats-new: add latest upstream version information

Peter Korsgaard peter at korsgaard.com
Thu Apr 5 08:56:43 UTC 2018


>>>>> "Ricardo" == Ricardo Martincoski <ricardo.martincoski at gmail.com> writes:

Hi,

 > There is a lot of tutorials and articles in the wild saying this is the way to
 > go. After some digging online I think most of these articles are incomplete.
 > This seems to be a more complete article about these modules:
 > https://christopherdavis.me/blog/threading-basics.html

 > But then I tested the module multiprocessing.
 > IMO it is the way to go for this case.
 > See below the comparison.

 > 1) serialized requests:
 >  - really simple code
 >  - would take 2 hours to run in my machine

 > 2) threading + Queue:
 >  - lots of boilerplate code to work properly
 >  - 20 minutes in my machine

 > 3) multiprocessing:
 >  - simpler code than threading + Queue
 >  - 16 minutes in my machine
 >  - 9 minutes in the Gitlab CI elastic runner:
 > https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/60290644

 > The demo code is here (a commit on the top of this series 1 to 4):
 > https://gitlab.com/RicardoMartincoski/buildroot/commit/dc5f447c30157499cd925c9e79c7bc9c29252219

 > Of course, as any solution, there are some downsides.
 >  - Pool.apply_async can't call object methods. There are solutions to this using
 >    other modules, but I think the simpler code wins. We just need to offload the
 >    code that runs asynchronously to helper functions. Yes, like you did in a
 >    previous iteration of the series.
 >  - more RAM is consumed per worker. I did a very simple measurement and htop
 >    shows 60MB per worker. I don't think it is too much in this case. I did not
 >    measured the other solutions.

 > Can we switch to use multiprocessing?

I'm far from a Python expert, but it certainly sounds sensible to me! Thomas?

-- 
Bye, Peter Korsgaard


More information about the buildroot mailing list