[Buildroot] [PATCH v2 17/18] ccache: use mtime for external toolchain, CONF_OPTS for internal toolchain
Romain Naour
romain.naour at openwide.fr
Sun Oct 4 14:55:03 UTC 2015
Arnout,
Le 04/10/2015 14:28, Arnout Vandecappelle (Essensium/Mind) a écrit :
> Our current ccache disables hashing of the compiler executable itself,
> because using the default 'mtime' doesn't work in buildroot: we always
> rebuild the compiler, so the mtime is always different, so the cache
> always misses.
>
> However, in the current situation, if a user changes the compiler
> configuration (which would result in the compiler generating different
> object files than before) and does 'make clean all', ccache may in fact
> reuse object files from the previous run. This rarely gives problems,
> because
> (1) the cache expires quite quickly (it's only 1GB by default),
> (2) radically changing compiler options will cause cache misses because
> different header files are used,
> (3) many compiler changes (e.g. changing -mtune) have little practical
> effect because the resulting code is usually still compatible,
> (4) we currently don't use CCACHE_BASEDIR, and almost all object files
> will contain an absolute path (e.g. in debug info), so when
> building in a different directory, most of it will miss,
> (5) we do mostly build test, and many of the potential problems only
> appear at runtime.
> Still, when ccache _does_ use the wrong cached object files, the
> effects are really weird and hard to debug. Also, we want reproducible
> builds and obviously the above makes builds non-reproducible. So we
> have a FAQ entry that warns against using ccache and tells the user to
> clear the cache in case of problems.
>
> Now that ccache is called from the toolchain wrapper, it is in fact
> possible to at least use the 'mtime' compiler hash for the external
> toolchain and for the host-gcc. Indeed, in this case, the compiler
> executable comes from a tarball so the mtime will be a good reference
> for it state. Therefore, the patch (sed script) that changes the
> default from 'mtime' to 'none' is removed.
>
> For the internal toolchain, we can do better by providing a hash of
> the relevant toolchain options. We are only interested in things that
> affect the compiler itself, because ccache also processes the header
> files and it doesn't look at libraries because it doesn't cache the
> link step, just compilation. Everything that affects the compiler
> itself can nicely be summarised in $(HOST_GCC_FINAL_CONF_OPTS). Of
> course, also the compiler source itself is relevant, so the source
> tarball and all the patches are included in the hash. For this purpose,
> a new HOST_GCC_XTENSA_OVERLAY_TAR is introduced.
>
> The following procedure tests the ccache behaviour:
>
> Use this defconfig:
> BR2_arm=y
> BR2_CCACHE=y
>
> make
> readelf -A output/build/uclibc-1.0.6/libc/signal/signal.os
> -> Tag_CPU_name: "ARM926EJ-S"
>
> Now make menuconfig, change variant into BR2_cortex_a9
>
> make clean; make
> readelf -A output/build/uclibc-1.0.6/libc/signal/signal.os
> -> Tag_CPU_name: "ARM926EJ-S"
> should be "Cortex-A9"
>
> After this commit, it is "Cortex-A9".
>
> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout at mind.be>
> Cc: Danomi Manchego <danomimanchego123 at gmail.com>
> Cc: Károly Kasza <kaszak at gmail.com>
> Cc: Samuel Martin <s.martin49 at gmail.com>
> Cc: Romain Naour <romain.naour at openwide.fr>
> ---
> v2: - Add BR2_GLOBAL_PATCH_DIR to the patches used to generate the hash.
> [Samuel]
> - Use $(wildcard ...) for the patches, so missing patch dir (e.g. for
> OVERRIDE_SRCDIR) doesn't lead to an error. [Arnout]
> - Remove redundant newline [Romain]
> ---
> package/ccache/ccache.mk | 5 -----
> package/gcc/gcc.mk | 29 +++++++++++++++++++++++++++--
> toolchain/toolchain-wrapper.c | 12 ++++++++++++
> 3 files changed, 39 insertions(+), 7 deletions(-)
>
> diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
> index 3bd2803..52cf309 100644
> --- a/package/ccache/ccache.mk
> +++ b/package/ccache/ccache.mk
> @@ -26,15 +26,10 @@ HOST_CCACHE_CONF_OPTS += --with-bundled-zlib
> # is already used by autotargets for the ccache package.
> # BR_CACHE_DIR is exported by Makefile based on config option
> # BR2_CCACHE_DIR.
> -# - ccache shouldn't use the compiler binary mtime to detect a change in
> -# the compiler, because in the context of Buildroot, that completely
> -# defeats the purpose of ccache. Of course, that leaves the user
> -# responsible for purging its cache when the compiler changes.
> # - Change hard-coded last-ditch default to match path in .config, to avoid
> # the need to specify BR_CACHE_DIR when invoking ccache directly.
> define HOST_CCACHE_PATCH_CONFIGURATION
> sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/ccache.c
> - sed -i 's,conf->compiler_check = x_strdup("mtime"),conf->compiler_check = x_strdup("none"),' $(@D)/conf.c
> sed -i 's,"%s/.ccache","$(BR_CACHE_DIR)",' $(@D)/conf.c
> endef
>
> diff --git a/package/gcc/gcc.mk b/package/gcc/gcc.mk
> index b9da398..103cbe1 100644
> --- a/package/gcc/gcc.mk
> +++ b/package/gcc/gcc.mk
> @@ -23,9 +23,10 @@ GCC_SOURCE ?= gcc-$(GCC_VERSION).tar.bz2
> # Xtensa special hook
> #
>
> +HOST_GCC_XTENSA_OVERLAY_TAR = $(BR2_XTENSA_OVERLAY_DIR)/xtensa_$(call qstrip,$(BR2_XTENSA_CORE_NAME)).tar
> +
> define HOST_GCC_XTENSA_OVERLAY_EXTRACT
> - tar xf $(BR2_XTENSA_OVERLAY_DIR)/xtensa_$(call qstrip,\
> - $(BR2_XTENSA_CORE_NAME)).tar -C $(@D) --strip-components=1 gcc
> + tar xf $(HOST_GCC_XTENSA_OVERLAY_TAR) -C $(@D) --strip-components=1 gcc
> endef
>
> #
> @@ -237,6 +238,30 @@ endif
>
> HOST_GCC_COMMON_TOOLCHAIN_WRAPPER_ARGS += -DBR_CROSS_PATH_SUFFIX='".real"'
>
> +ifeq ($(BR2_CCACHE),y)
> +HOST_GCC_COMMON_CCACHE_HASH_FILES += $(DL_DIR)/$(GCC_SOURCE)
> +# Cfr. PATCH_BASE_DIRS in .stamp_patched, but we catch both versioned and
> +# unversioned patches unconditionally
> +HOST_GCC_COMMON_CCACHE_HASH_FILES += \
> + $(wildcard \
> + package/gcc/$(GCC_VERSION)/*.patch \
> + $(addsuffix $((PKG)_RAWNAME)/$(GCC_VERSION)/*.patch,$(call qstrip,$(BR2_GLOBAL_PATCH_DIR))) \
> + $(addsuffix $((PKG)_RAWNAME)/*.patch,$(call qstrip,$(BR2_GLOBAL_PATCH_DIR))))
> +ifeq ($(BR2_xtensa),y)
> +HOST_GCC_COMMON_CCACHE_HASH_FILES += $(HOST_GCC_XTENSA_OVERLAY_TAR)
> +endif
> +ifeq ($(ARCH),powerpc)
> +ifneq ($(BR2_SOFT_FLOAT),)
> +HOST_GCC_COMMON_CCACHE_HASH_FILES += package/gcc/$(GCC_VERSION)/1000-powerpc-link-with-math-lib.patch.conditional
> +endif
> +endif
> +
> +HOST_GCC_COMMON_TOOLCHAIN_WRAPPER_ARGS += -DBR_CCACHE_HASH=\"`\
> + printf '%s' $($(PKG)_CONF_OPTS) $(GCC_VERSION) \
> + | sha256sum - $(HOST_GCC_COMMON_CCACHE_HASH_FILES) \
> + | cut -c -64 | tr -d '\n'`\"
> +endif # BR2_CCACHE
> +
> # The LTO support in gcc creates wrappers for ar, ranlib and nm which load
> # the lto plugin. These wrappers are called *-gcc-ar, *-gcc-ranlib, and
> # *-gcc-nm and should be used instead of the real programs when -flto is
> diff --git a/toolchain/toolchain-wrapper.c b/toolchain/toolchain-wrapper.c
> index cf66fac..7ee3028 100644
> --- a/toolchain/toolchain-wrapper.c
> +++ b/toolchain/toolchain-wrapper.c
> @@ -254,6 +254,10 @@ int main(int argc, char **argv)
> debug = atoi(env_debug);
> if (debug > 0) {
> fprintf(stderr, "Toolchain wrapper executing:");
> +#ifdef BR_CCACHE_HASH
> + fprintf(stderr, "CCACHE_COMPILERCHECK=string:" BR_CCACHE_HASH "%s",
> + (debug == 2) ? "\n " : " ");
> +#endif
There is probably a formatting issue here when debug==2:
Toolchain wrapper executing:CCACHE_COMPILERCHECK=string:[hash]
CCACHE_BASEDIR=/home/naourr/git/buildroot/test/ccache
'/home/naourr/git/buildroot/test/ccache/host/usr/bin/ccache'
'/home/naourr/git/buildroot/test/ccache/host/usr/bin/i686-buildroot-linux-uclibc-gcc.real'
'--sysroot'
The fprintf line should look-like:
fprintf(stderr, "%sCCACHE_COMPILERCHECK=string:'" BR_CCACHE_HASH "'",
(debug == 2) ? "\n " : " ");
Toolchain wrapper executing:
CCACHE_COMPILERCHECK='string:[hash]'
CCACHE_BASEDIR='/home/naourr/git/buildroot/test/ccache'
'/home/naourr/git/buildroot/test/ccache/host/usr/bin/i686-buildroot-linux-uclibc-gcc.real'
'--sysroot'
'/home/naourr/git/buildroot/test/ccache/host/usr/i686-buildroot-linux-uclibc/sysroot'
'-E'
'-xc'
'-'
> for (i = 0; exec_args[i]; i++)
> fprintf(stderr, "%s'%s'",
> (debug == 2) ? "\n " : " ", exec_args[i]);
> @@ -261,6 +265,14 @@ int main(int argc, char **argv)
Same here.
Best regards,
Romain
> }
> }
>
> +#ifdef BR_CCACHE_HASH
> + /* Allow compilercheck to be overridden through the environment */
> + if (setenv("CCACHE_COMPILERCHECK", "string:" BR_CCACHE_HASH, 0)) {
> + perror(__FILE__ ": Failed to set CCACHE_COMPILERCHECK");
> + return 3;
> + }
> +#endif
> +
> if (execv(exec_args[0], exec_args))
> perror(path);
>
>
More information about the buildroot
mailing list