[Buildroot] [PATCH 2/2] package/zstd: link programs dynamically with libzstd to save space
Yann E. MORIN
yann.morin.1998 at free.fr
Mon Sep 28 19:40:37 UTC 2020
Thomas, All,
On 2020-09-28 13:42 +0200, Thomas De Schampheleire spake thusly:
> From: Thomas De Schampheleire <thomas.de_schampheleire at nokia.com>
>
> Even when a shared libzstd is built, the zstd command-line programs are
> statically linked with libzstd, causing a large rootfs footprint.
>
> While the cmake backend in zstd already supported a flag
> ZSTD_PROGRAMS_LINK_SHARED, the make backend did not.
Any reason why we do not switch over to using cmake or meson? Both seem
to be officially supported, see README.md:
## Build instructions
### Makefile
[...]
### cmake
[...]
### meson
[...]
So, I'd rather we expend the effort to switch to cmake, as it already
has ZSTD_PROGRAMS_LINK_SHARED (or to meson if it also fits the bill,
whatever floats your boat), rather than patching the Makefile-based
buildsystem.
Regards,
Yann E. MORIN.
> This commit adds support for ZSTD_PROGRAMS_LINK_SHARED in the make system
> and applies it for the target compilation, unless only static libs are
> supported.
>
> Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire at nokia.com>
> ---
> ...D_PROGRAMS_LINK_SHARED-to-link-zstd-.patch | 80 +++++++++++++++++++
> package/zstd/zstd.mk | 10 +++
> 2 files changed, 90 insertions(+)
> create mode 100644 package/zstd/0002-make-support-ZSTD_PROGRAMS_LINK_SHARED-to-link-zstd-.patch
>
> diff --git a/package/zstd/0002-make-support-ZSTD_PROGRAMS_LINK_SHARED-to-link-zstd-.patch b/package/zstd/0002-make-support-ZSTD_PROGRAMS_LINK_SHARED-to-link-zstd-.patch
> new file mode 100644
> index 0000000000..100cfaba95
> --- /dev/null
> +++ b/package/zstd/0002-make-support-ZSTD_PROGRAMS_LINK_SHARED-to-link-zstd-.patch
> @@ -0,0 +1,80 @@
> +From afc368e7247abfe89250def5b7673a9ccb79e0b1 Mon Sep 17 00:00:00 2001
> +From: Thomas De Schampheleire <thomas.de_schampheleire at nokia.com>
> +Date: Wed, 5 Aug 2020 15:35:01 +0200
> +Subject: [PATCH] make: support ZSTD_PROGRAMS_LINK_SHARED to link zstd with
> + libzstd
> +
> +zstd allows building via make, cmake, meson, and others, but unfortunately
> +the features and flags used for these build systems are inconsistent.
> +
> +The cmake version respects an option 'ZSTD_PROGRAMS_LINK_SHARED' to let the
> +zstd binary link dynamically with its own libzstd. Without it, the zstd
> +program uses static linking and thus has a big size.
> +
> +This option is not provided in the 'make' system. There does exist a target
> +'zstd-dll' which intends to do exactly this, but it does not work out of the
> +box due to linking issues. These in turn are caused because the lib makefile
> +passes '-fvisibility=hidden' to the linker, which hides certain symbols that
> +the zstd program needs. The cmake-based compilation does not pass this
> +visibility flag, due to which all symbols are visible.
> +
> +Unfortunately, there is no 'install' rule that will install zstd-dll and
> +create the derived programs like zstdless etc.
> +
> +With the above in mind, when ZSTD_PROGRAMS_LINK_SHARED is set in make
> +context:
> +- remove '-fvisibility=hidden' from the lib compilation
> +- alter the 'zstd' target to not include the lib objects directly but link
> + dynamically with libzstd.
> +
> +See also https://github.com/facebook/zstd/issues/2261 which reported these
> +inconsistencies between make and cmake compilation.
> +
> +Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire at nokia.com>
> +
> +---
> + lib/Makefile | 5 ++++-
> + programs/Makefile | 8 +++++++-
> + 2 files changed, 11 insertions(+), 2 deletions(-)
> +
> +diff --git a/lib/Makefile b/lib/Makefile
> +index 7c6dff02..8f9f36a6 100644
> +--- a/lib/Makefile
> ++++ b/lib/Makefile
> +@@ -204,7 +204,10 @@ $(LIBZSTD): $(ZSTD_FILES)
> + else
> +
> + LIBZSTD = libzstd.$(SHARED_EXT_VER)
> +-$(LIBZSTD): LDFLAGS += -shared -fPIC -fvisibility=hidden
> ++$(LIBZSTD): LDFLAGS += -shared -fPIC
> ++ifndef ZSTD_PROGRAMS_LINK_SHARED
> ++$(LIBZSTD): LDFLAGS += -fvisibility=hidden
> ++endif
> + $(LIBZSTD): $(ZSTD_FILES)
> + @echo compiling dynamic library $(LIBVER)
> + $(Q)$(CC) $(FLAGS) $^ $(LDFLAGS) $(SONAME_FLAGS) -o $@
> +diff --git a/programs/Makefile b/programs/Makefile
> +index 418ad4e6..8897dcf9 100644
> +--- a/programs/Makefile
> ++++ b/programs/Makefile
> +@@ -169,10 +169,16 @@ $(ZSTDDECOMP_O): CFLAGS += $(ALIGN_LOOP)
> + zstd : CPPFLAGS += $(THREAD_CPP) $(ZLIBCPP) $(LZMACPP) $(LZ4CPP)
> + zstd : LDFLAGS += $(THREAD_LD) $(ZLIBLD) $(LZMALD) $(LZ4LD) $(DEBUGFLAGS_LD)
> + zstd : CPPFLAGS += -DZSTD_LEGACY_SUPPORT=$(ZSTD_LEGACY_SUPPORT)
> ++ifdef ZSTD_PROGRAMS_LINK_SHARED
> ++# link dynamically against own library
> ++zstd : LDFLAGS += -L$(ZSTDDIR) -lzstd
> ++else
> ++zstd : $(ZSTDLIB_FILES)
> ++endif
> + ifneq (,$(filter Windows%,$(OS)))
> + zstd : $(RES_FILE)
> + endif
> +-zstd : $(ZSTDLIB_FILES) $(ZSTD_CLI_OBJ)
> ++zstd : $(ZSTD_CLI_OBJ)
> + @echo "$(THREAD_MSG)"
> + @echo "$(ZLIB_MSG)"
> + @echo "$(LZMA_MSG)"
> +--
> +2.26.2
> +
> diff --git a/package/zstd/zstd.mk b/package/zstd/zstd.mk
> index 35002da332..c7b224b002 100644
> --- a/package/zstd/zstd.mk
> +++ b/package/zstd/zstd.mk
> @@ -43,9 +43,19 @@ ZSTD_INSTALL_LIBS = install-static
> else ifeq ($(BR2_SHARED_LIBS),y)
> ZSTD_BUILD_LIBS = libzstd
> ZSTD_INSTALL_LIBS = install-shared
> +ZSTD_OPTS += ZSTD_PROGRAMS_LINK_SHARED=1
> else
> ZSTD_BUILD_LIBS = libzstd.a libzstd
> ZSTD_INSTALL_LIBS = install-static install-shared
> +ZSTD_OPTS += ZSTD_PROGRAMS_LINK_SHARED=1
> +endif
> +
> +# The HAVE_THREAD flag is read by the 'programs' makefile but not by the 'lib'
> +# one. Building a multi-threaded binary with a library (which defaults to
> +# single-threaded) gives a runtime error when compressing files.
> +# The 'lib' makefile provides specific '%-mt' targets for this purpose.
> +ifeq ($(BR2_TOOLCHAIN_HAS_THREADS),y)
> +ZSTD_BUILD_LIBS := $(addsuffix -mt,$(ZSTD_BUILD_LIBS))
> endif
>
> define ZSTD_BUILD_CMDS
> --
> 2.26.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
More information about the buildroot
mailing list