Parallel make bug in busybox

Jacob Kroon jacob.kroon at gmail.com
Fri Aug 19 13:47:21 UTC 2016


As someone who has had similar problems before I'll give 2 cents..

On Fri, Aug 19, 2016 at 1:56 PM, Richard Purdie <rpurdie at rpsys.net> wrote:
> I think the recent "fix" actually made things worse. Whenever I run "rm
> include/applet_tables.h include/NUM_APPLETS.h; make clean; make -j 48",
> I see the applet_tables command running twice in parallel.
>
> There are two issues:
>
> a) There is nothing stopping make running the appliet_tables command
> twice in parallel, once to generate include/applet_tables.h and once to
> generate include/NUM_APPLETS.h.
>

The way I learned to think about this is that make thinks of

A B : C
<tab>rule

as

A : C
<tab>rule

B : C
<tab>rule

which isn't good for parallell builds.
On the other hand, if files share a common pattern then one can write
a pattern rule

%-A %-B : %-C
<tab>rule

which _do_ work as intended in parallell builds IIRC (run only once).
https://www.gnu.org/software/automake/manual/html_node/Multiple-Outputs.html
has some more info on how to solve these situations.

/Jacob

> b) applets/usage_pod depends upon include/applet_tables.h *and*
> include/NUM_APPLETS.h. After include/usage_compressed.h has been
> generated, the system may want to regenerate include/applet_tables.h
> and include/NUM_APPLETS.h so it gets the timestamp ordering it wants. I
> get the feeling from the make output there is some circular logic in
> here which causes problems.
>
> A change which definitely improved things and stopped the direct double
> command execution is:
>
> Index: busybox-1.24.1/applets/Kbuild.src
> ===================================================================
> --- busybox-1.24.1.orig/applets/Kbuild.src
> +++ busybox-1.24.1/applets/Kbuild.src
> @@ -29,7 +29,7 @@ applets/applets.o: include/usage_compres
>
>  applets/applet_tables: .config include/applets.h
>  applets/usage:         .config include/applets.h
> -applets/usage_pod:     .config include/applets.h include/applet_tables.h include/NUM_APPLETS.h
> +applets/usage_pod:     .config include/applets.h include/NUM_APPLETS.h
>
>  quiet_cmd_gen_usage_compressed = GEN     include/usage_compressed.h
>        cmd_gen_usage_compressed = $(srctree_slash)applets/usage_compressed include/usage_compressed.h applets
> @@ -42,3 +42,5 @@ quiet_cmd_gen_applet_tables = GEN     in
>
>  include/applet_tables.h include/NUM_APPLETS.h: applets/applet_tables
>         $(call cmd,gen_applet_tables)
> +
> +include/NUM_APPLETS.h: include/applet_tables.h
>
> Since this makes make wait until include/applet_tables.h is generated
> before generating NUM_APPLETS.h which it then doesn't need to generate
> as it already exists. Now we have this dependency, we can tweak
> applets/usage_pod to rely on that ordering, hence removing the circular
> command calls to get the right timestamp ordering.
>
> I suspect this still doesn't solve the original parallel make bug
> though and we've seen yet more build failures from that :(
>
> (e.g. https://autobuilder.yoctoproject.org/main/builders/nightly-ppc-lsb/builds/882/steps/BuildImages_1/logs/stdio)
>
> Cheers,
>
> Richard
>
>
> _______________________________________________
> busybox mailing list
> busybox at busybox.net
> http://lists.busybox.net/mailman/listinfo/busybox


More information about the busybox mailing list