[Buildroot] [PATCH v3 0/3] Use aiohttp in pkg-stats

Peter Korsgaard peter at korsgaard.com
Fri Aug 28 15:52:40 UTC 2020


>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni at bootlin.com> writes:

 > Hello,
 > I started investigating why pkg-stats was sometimes stuck on the
 > server running it on a daily basis to populate
 > autobuild.buildroot.org/stats/ and send the autobuilder e-mails. The
 > subprocesses started by "multiprocessing" to retrieve the latest
 > upstream version from release-monitoring.org were stuck holding a
 > lock. Without providing a definitive conclusion, some preliminary
 > research showed that multiprocessing can be tricky and cause some
 > issues with locks.

 > Discussing this with Titouan, he suggested to use aiohttp instead of
 > multiprocessing. And indeed, it makes a lot of sense to use this
 > popular asynchronous HTTP library.

 > This patch series switches the latest version retrieval and the
 > upstream URL checking to aiohttp, and as a bonus adds some logging to
 > show the progress of the retrieval, as it can be quite long.

 > Changes since v2:

 >  - Use python3 in the shebang

 >  - Use asyncio.TimeoutError in the exception handling

 >  - Slightly rework how packages with "no valid infra" are handled in
 >    the "latest version" check, but we keep a loop to handle such
 >    packages before the main loop, as we want the real count of
 >    packages that the main loop will handle.

 >  - Use asyncio.get_event_loop() + loop.run_until_complete() instead of
 >    asyncio.run().

Committed to 2020.02.x and 2020.05.x, thanks.

-- 
Bye, Peter Korsgaard


More information about the buildroot mailing list