[Buildroot] [PATCH 0/1] [RFC] add support for icecc
Charlie Turner
cturner at igalia.com
Wed Apr 29 14:07:28 UTC 2020
I would like to use Icecream for my builds, primarily because I rely
on WebKit at work which is a horror to on small-iron. For now I focus
only on distributing compilation of target packages.
In theory this should be easy,
1) Create a tarball containing the cross-toolchain buildroot pulled in
for the cluster machines to compile with.
2) Point at this tarball from `ICECC_VERSION'
3) Arrange for buildroot's toolchain wrappers to call `icecc'.
4) Enjoy faster builds?
1) can be done using the `icecc --build-native' command. As an example
of what it produces, here's the output on my x86-64 host. The logic for
what files is here[1],
┌────
│ $ icecc --build-native
│ [...]
│ # Outputs c54a871ca4de453cdba3ddda4e92a802.tar.gz
│ $ tar -tf c54a871ca4de453cdba3ddda4e92a802.tar.gz
│ bin/true
│ etc/ld.so.cache
│ etc/ld.so.conf
│ lib64/ld-linux-x86-64.so.2
│ lib/x86_64-linux-gnu/libbfd-2.31.1-system.so
│ lib/x86_64-linux-gnu/libc.so.6
│ lib/x86_64-linux-gnu/libdl.so.2
│ lib/x86_64-linux-gnu/libgmp.so.10
│ lib/x86_64-linux-gnu/libisl.so.19
│ lib/x86_64-linux-gnu/libmpc.so.3
│ lib/x86_64-linux-gnu/libmpfr.so.6
│ lib/x86_64-linux-gnu/libm.so.6
│ lib/x86_64-linux-gnu/libopcodes-2.31.1-system.so
│ lib/x86_64-linux-gnu/libz.so.1
│ usr/bin/as
│ usr/bin/cc1
│ usr/bin/cc1plus
│ usr/bin/g++
│ usr/bin/gcc
│ usr/bin/objcopy
│ usr/lib/gcc/x86_64-linux-gnu/8/liblto_plugin.so
└────
This seems fairly unsurprising, bin/true is there for sanity checking,
the other files are basically resolving the `ldd' chain from a
compiler binary. There will be details here I am missing surely as a
toolchain ignoramus.
First attempt in buildroot, `make qemu_arm_versatile_defconfig' and,
┌────
│ $ icecc --build-native ./output/host/bin/arm-buildroot-linux-uclibcgnueabi-gcc
│ '/home/cht/igalia/buildroot/output/host/bin/arm-buildroot-linux-uclibcgnueabi-g++.br_real' is no executable.
└────
The path
`/home/cht/igalia/buildroot/output/host/bin/arm-buildroot-linux-uclibcgnueabi-g++.br_real'
does not actually exist. Turns out this defconfig does not come with C++
support, strange that icecc's toolchain crawler somehow felt it needed
this. Maybe a hardcode in icecc.
It works when I enable C++ support, (need to investigate the icecc
source about why this is necessary)
┌────
│ $ icecc --build-native ./output/host/bin/arm-buildroot-linux-uclibcgnueabi-gcc
│ adding file /bin/true=/usr/bin/true
│ adding file /lib/x86_64-linux-gnu/libc.so.6=/usr/lib/x86_64-linux-gnu/libc-2.28.so
│ adding file /lib64/ld-linux-x86-64.so.2=/usr/lib/x86_64-linux-gnu/ld-2.28.so
│ adding file /usr/bin/gcc=/home/cht/igalia/buildroot/output/host/bin/arm-buildroot-linux-uclibcgnueabi-gcc.br_real
│ adding file /lib/x86_64-linux-gnu/libm.so.6=/usr/lib/x86_64-linux-gnu/libm-2.28.so
│ adding file /usr/bin/g++=/home/cht/igalia/buildroot/output/host/bin/arm-buildroot-linux-uclibcgnueabi-g++.br_real
│ adding file /usr/bin/cc1=/home/cht/igalia/buildroot/output/host/libexec/gcc/arm-buildroot-linux-uclibcgnueabi/8.4.0/cc1
│ adding file /usr/lib/libmpc.so.3=/home/cht/igalia/buildroot/output/host/lib/libmpc.so.3.1.0
│ adding file /usr/lib/libmpfr.so.6=/home/cht/igalia/buildroot/output/host/lib/libmpfr.so.6.0.2
│ adding file /usr/lib/libgmp.so.10=/home/cht/igalia/buildroot/output/host/lib/libgmp.so.10.3.2
│ adding file /lib/x86_64-linux-gnu/libdl.so.2=/usr/lib/x86_64-linux-gnu/libdl-2.28.so
│ adding file /usr/lib/libz.so.1=/home/cht/igalia/buildroot/output/host/lib/libz.so.1.2.11
│ adding file /usr/bin/cc1plus=/home/cht/igalia/buildroot/output/host/libexec/gcc/arm-buildroot-linux-uclibcgnueabi/8.4.0/cc1plus
│ adding file /usr/bin/as=/home/cht/igalia/buildroot/output/host/arm-buildroot-linux-uclibcgnueabi/bin/as
│ adding file /usr/libexec/gcc/arm-buildroot-linux-uclibcgnueabi/8.4.0/liblto_plugin.so=/home/cht/igalia/buildroot/output/host/libexec/gcc/arm-buildroot-linux-uclibcgnueabi/8.4.0/liblto_plugin.so.0.0.0
│ adding file /usr/bin/objcopy=/home/cht/igalia/buildroot/output/host/arm-buildroot-linux-uclibcgnueabi/bin/objcopy
│ adding file /etc/ld.so.conf=/tmp/icecc_ld_so_confSllt0p
│ cp: -r not specified; omitting directory '/lib'
│ cp: -r not specified; omitting directory '/lib64'
│ creating bdd71538f0cc87b1653982f1c1b709e5.tar.gz
└────
There's some obvious pollution from my host environment, don't know if
it will cause issues yet. This tarball is theoretically under our
complete control, any errors in what `icecc' picks we can resolve
by creating it ourselves in the Makefiles.
Now, to hack this into target package builds,
┌────
│ diff --git a/toolchain/toolchain-wrapper.c b/toolchain/toolchain-wrapper.c
│ index 8cfc963ccd..b91f12a52d 100644
│ --- a/toolchain/toolchain-wrapper.c
│ +++ b/toolchain/toolchain-wrapper.c
│ @@ -60,6 +60,7 @@ static char *predef_args[] = {
│ #ifdef BR_CCACHE
│ ccache_path,
│ #endif
│ + "/usr/bin/icecc",
│ path,
│ "--sysroot", sysroot,
│ #ifdef BR_ABI
└────
After `export
ICECC_VERSION=/path/to/bdd71538f0cc87b1653982f1c1b709e5.tar.gz' and
running `make', this does "work" in the sense I get something built
without error (not tried running it yet). The problem is all builds
are done locally, just via the `icecc' daemon!
┌────
│ ICECC[1729] 2020-04-27 19:15:29: invoked as: /usr/bin/icecc /home/cht/igalia/buildroot/output/host/bin/arm-buildroot-linux-uclibcgnueabi-gcc.br_real --sysroot /home/cht/igalia/buildroot/output/host/arm-buildroot-linux-uclibcgnueabi/sysroot --version
│ ICECC[1729] 2020-04-27 19:15:29: connected to /var/run/icecc/iceccd.socket
│ ICECC[1729] 2020-04-27 19:15:29: custom command, running locally.
└────
To see such debugging, set the following variables for client-side
logging,
┌────
│ export ICECC_LOGFILE=/tmp/icecc.log
│ export ICECC_DEBUG=10 # overkill
└────
This is a [previously discussed bug]. The issue is that icecc gets
confused by `arm-buildroot-linux-uclibcgnueabi-gcc.br_real' as the
name of a `gcc'. Upgrading to a master version of icecc has fixed this
by relaxing the check, but worth knowing that the version Debian
Buster ships has this bug.
I installed `icecc' from git
(`ff7f4bddfb9212cb401ab64a9d08227058287aa1') and this has been fixed
to not be so fussy. With that and this trivial patch to
`toolchain-wrapper', I can at least build
`qemu_arm_versatile_defconfig' across a cluster.
RFCs
• Host packages are still compiled locally, I would like this to
extend to host builds as well. Could just prepend icecc in the
$PATH on the host. Need to test, potential complication is
switching icecc's toolchain mid-build. Might not be an issue, just
not gone there yet.
• Generating the `ICECC_VERSION' tarball from the Makefiles. Ideally
it would all be transparent to the user if, say,
`BR2_ICECC=y'. It's inconvenient for the user to have to call
`--build-native' and arrange for `ICECC_VERSION' to be in the
`make' environment. You need to first build the cross-toolchain,
setup the tarball, and then so long as your toolchain doesn't
change, subsequent rebuilds are distributed. This all needs
automation.
• Number of jobs to simultaneously run has be manually selected, I
don't know of a method to find out how many remote CPUs you have
available with Icecream automatically.
[1] https://github.com/icecc/icecream/blob/master/client/icecc-create-env.in#L438
[previously discussed bug] https://github.com/icecc/icecream/issues/164
Charlie Turner (1):
[RFC] add support for icecc
toolchain/toolchain-wrapper.c | 22 ++++++++++++++++++++++
1 file changed, 22 insertions(+)
--
2.20.1
More information about the buildroot
mailing list