[Buildroot] [PATCH 2/2] core/instrumentation: shave minutes off the build time
Peter Korsgaard
peter at korsgaard.com
Sun Mar 18 14:14:40 UTC 2018
>>>>> "Yann" == Yann E MORIN <yann.morin.1998 at free.fr> writes:
Hi,
> As part of the build, we run some instrumentation hooks to gather
> statistics about the usage of the target/, staging/ and host/
> directories, so that we can generate reports for the user, that
> shows:
> - for each file, what package installed it,
> - for each package,the size that it installed.
> In so doing, we run a double md5 pass on all files of the affected
> directories. These passes were mostly invisible when we were only
> scanning target/, but has greatly increased in time now that we also
> scan staging/ and host/ (but only in the corresponding _CMDS, of
> course).
> This md5 wsa mostly aimed at catching packages that would "cheat" with
> mtime/atime/ctime somehow. They can't really cheat on md5, though [0].
> Timings however speak for themselves, with this defconfig (slightly
> biggish-but-still-manageable build) [1].
> host/ 20965 files 1.2GiB
> staging/ 4715 files 333MiB
> target/ 1801 files 44MiB
> All instrumentation steps, using md5: 19min 27s
> All instrumentation steps, using mtime: 14min 45s
> No instrumentation step at all: 14min 31s
> So, using mtime is an almost-5min improvement, i.e. about 25% faster,
> while removing all instrumentation steps does not gain that much more...
> So, we switch to using mtime, because in the end that's still good-enough
> for our use-case: generating some graphs. It is not mission-critical, and
> if a graph is slightly off, that's not biggy. It can anyway be attributed
> to a broken package's buildsystem, which should get fixed.
> However, we lose the ability to track directories. Non-empty directories
> can be tracked back by a bit of scripting, but empty directories are
> simply not caught. If we were to also look for directories using mtime,
> we would catch parents of installed files:
> - /foo/bar/ exists
> - a package installs /foo/bar/buz
> - mtime of /foo/bar/ is changed to account for the nex file in it.
Playing around with this, I noticed two other issues:
- It doesn't work for packages using rsync to install,
E.G. skeleton-init-common as rsync also sets the mtime to match the
source files
- It breaks for <pkg>-reinstall
I don't think either of those are really big issues compared to the huge
slowdown, but it is worth noticing.
> +define step_pkg_size_inner
> + cd $(2); \
> + find . \( -type f -o -type L \) \
> + -newer $($(PKG)_DIR)/.stamp_built \
> + -exec printf '$(1),%s\n' {} + \
> + >> $(BUILD_DIR)/packages-file-list$(3).txt
What find version are you using? My fileutils find (and the busybox
applet) use 'l' for symlinks, so I've changed it to that.
Committed with that fixed (and a few tweaks to the commit message),
thanks.
--
Bye, Peter Korsgaard
More information about the buildroot
mailing list