[Buildroot] [RFC 0/3] Speeding up the Makefile parsing
Arnout Vandecappelle
arnout at mind.be
Tue Mar 25 21:03:51 UTC 2014
Hi Thomas,
On 23/03/14 21:57, Thomas Petazzoni wrote:
> Dear Peter Korsgaard,
>
> On Sun, 23 Mar 2014 21:38:50 +0100, Peter Korsgaard wrote:
>>>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni at free-electrons.com> writes:
>>
>> > Hello,
>> > The current strategy of Buildroot with regard to .mk files is to
>> > always include all of them. However, with an ever-increasing number of
>> > packages, the number of .mk files to parse is growing significantly,
>> > and this makes the execution of any trivial make target take at least
>> > 12 seconds on a relatively fast SSD equipped laptop. For example, a
>> > simple thing like "make help" or "make external-deps" takes 12 seconds
>> > solely because of the parsing time of more than a thousand of package
>> > makefiles.
>>
>> .. and how fast is it after this series?
>
> Before the series:
>
> $ time make help
> [...]
> real 0m15.760s
> user 0m14.818s
> sys 0m0.969s
>
> After the series:
>
> $ time make help
> [...]
> real 0m1.209s
> user 0m0.807s
> sys 0m0.416s
I was at first a bit wary of your proposed approach, as I would have
preferred to still include all .mk files and instead optimise the
implementation of generic-package. However, I did some further
experiments with
make -qp > /dev/null
on a reasonable-sized .config (make -qp is what is used by bash
completion so is very relevant IMHO).
Before the series:
$ time make -qp > /dev/null
make: *** [_all] Error 1
real 0m3.452s
user 0m2.980s
sys 0m0.092s
After the series:
$ time make -qp > /dev/null
make: *** [_all] Error 1
real 0m0.389s
user 0m0.280s
sys 0m0.004s
Upper bound of generic-package optimisation (just define *-package macros
as empty):
$ time make -qp > /dev/null
make: *** [_all] Error 1
real 0m1.234s
user 0m0.872s
sys 0m0.064s
And on my allpackageyesconfig, there is even a tiny speedup (due to the
twenty-ish packages that are not included):
Before:
$ time make -qp > /dev/null
make: *** [_all] Error 1
real 0m3.325s
user 0m3.044s
sys 0m0.116s
After:
$ time make -qp > /dev/null
make: *** [_all] Error 1
real 0m3.040s
user 0m2.792s
sys 0m0.096s
With those kind of results, the approach you propose really seems the
way to go.
>> > The proposed RFC solution is to look at the BR2_PACKAGE_<foo>
>> > variables, and from them derive the name of the package .mk files to
>> > be included. It is a simple change in the main Makefile, but it has
>> > two major consequences that we need to discuss on whether they are
>> > acceptable or not:
>>
>> > * We need to have BR2_PACKAGE_HOST_<foo> options for *all* host
>> > packages. Of course, the vast majority of them can be blind
>> > options, but it means that we have to ensure all the host packages
>> > are properly selected at the Config.in level. That's a fairly
>> > significant change.
>>
>> I don't think this is too bad. Maybe we could look into generating the
>> _DEPENDENCIES line from depends/select statements in the Config.in
>> automatically, then we only need to explicitly handle the optional
>> dependencies in the .mk file (or decide to also handle them in Kconfig
>> if possible).
Actually, we could generate the Config.in from the .mk file (like
OpenWRT does). Probably easier than the reverse.
>
> Why not, but doing this would require a pre-processing step to process
> the Config.in file and generates the makefile dependencies from it.
> Also, having the mandatory dependencies handled automatically but not
> the optional dependencies seems a bit weird, no?
I think that Peter meant that the _DEPENDENCIES variable would be
defined by some make function magic, not that a .mk file is generated.
Something like:
_DEPENDENCIES = $(call LOWERCASE,$(shell \
sed -n '/.*select BR2_PACKAGE_\([A-Z0-9_]*\).*/s//\1/p' $(pkgdir)))
>
> That being said, doing for host packages what we do today for target
> packages doesn't that horrible, especially since we don't have all the
> crazy dependencies on toolchain options that are complicated to handle
> for target packages.
Right.
There are anyway a number of other reasons why it would be nice to have
blind options for the host packages.
* Facilitate package check script.
* Remove the hacks to generate *-source target.
* Probably others.
>
>> > * Since the logic assume that BR2_PACKAGE_FOO is always
>> > package/foo/foo.mk, it means we have to 1/ guarantee that the
>> > package option name matches the package file name, and 2/ guarantee
>> > that we don't use any sub-directories to store packages.
>>
>> The 2nd part is more serious as we use subdirs today to group related
>> packages and have supported them "forever". We also used to tell people
>> (before BR2_EXTERNAL) to put their own packages under package/<company>.
>>
>> I guess playing tricks with $(wildcard) to find the .mk file is going to
>> add quite some performance overhead.
>
> Yes. This part is more problematic, and I don't have a good solution
> for it.
We can require that packages in subdirs have a corresponding option for
the directory, cfr. BR2_PACKAGE_XORG7 (which would mean that the
directory has to be renamed). In some cases this could be a blind option
that is enabled any time that one of its packages is enabled (e.g. for
the freescale-imx stuff), in some cases there already happens to be a
package with that name (e.g. gstreamer). gstreamer doesn't even have to
be modified for it to work. It does mean that a few more .mk files will
be included than strictly necessary, but I don't think that's a worry.
For xorg7, we could still apply the same pattern in package/xorg7/xorg7.mk
Bottom line: I'd like to see this series!
Regards,
Arnout
>
> Thomas
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
More information about the buildroot
mailing list