[Buildroot] [PATCH] support/scripts/check-host-libs: add new check on host binaries/libs
yann.morin at orange.com
yann.morin at orange.com
Tue Sep 20 07:46:24 UTC 2022
Thomas, All,
On 2022-09-20 08:45 +0200, Thomas Petazzoni spake thusly:
> One frequent issue in Buildroot is that when building host libraries
> or applications, the build system of the package detects some
> libraries provided by the system, and happily links to them, without
> Buildroot knowing. Sometimes this doesn't cause any problem, but
> sometimes this causes issues, and we're regularly eliminating such
> mis-detection by forcing those packages to not detect the system
> libraries that have not been built by Buildroot.
>
> The new script check-host-libs added in this commit, which is executed
> during the host-finalize step at the end of the build is an attempt at
> detecting at least some of these situations.
>
> What it does is that at the end of the build, it verifies that all
> binaries and libraries in $(HOST_DIR) only have shared library
> dependencies on libraries that are in Buildroot $(HOST_DIR), to the
> exception of the C library, for which we of course use the system C
> library.
>
> For example, if the binary output/host/bin/plop is linked against
> libpng, but libpng was not built and installed by Buildroot, the build
> will now fail with:
>
> ERROR: in /home/thomas/projets/buildroot/output/host/bin/plop, libpng16.so.16 unknown
> make: *** [Makefile:715: host-finalize] Error 1
I'm afraid this is going to be too big a hammer, at least if that's a
hard error.
Indeed, when the build environment is reproducible (like, it is a
container image), and there are no package in Buildroot, and the host
directory will not be distributed (i.e. one does not care about doing an
SDK), then it is usually good-enough to use the system-installed
libraries, rather than add a Buildroot package.
> The script includes an allowlist of libraries provided by the C
> library. It is potentially possible that this list might need to be
> extended to cover all systems/distributions/C libraries, but only
> wider testing of this script will help detect such cases.
There are at least missing entries already:
librt.so.1
libutil.so.1
libresolv.so.2
Also, there are error messages for some go stuff:
==> HOST_DIR/lib/go/src/debug/elf/testdata/go-relocation-test-gcc930-ranges-no-rela-x86-64
readelf: Error: no .dynamic section in the dynamic segment
==> HOST_DIR/lib/go/src/debug/elf/testdata/go-relocation-test-gcc930-ranges-with-rela-x86-64
readelf: Error: no .dynamic section in the dynamic segment
Finally, this script takes more than a minute to run on our build, about
a 10% increase from ~12min. This is not nice at all.
$ find host/*bin host/lib* -type f |wc -l
16186
$ find host/*bin host/lib* -type f |(keep just libs + execs) |wc -l
339
That last command, though, is what takes time: there is a huge ton of go
junk installed in HOST_DIR, and this takes ages to filter-out.
So, although I understand the rationale, this should probably be opt-in.
Or it should only be ran when doing an SDK?
Regards,
Yann E. MORIN.
> It is worth mentioning that for now this script is executed only once
> at the end of the build. This means that if a package A gets built,
> detects and uses a system library libfoo and uses it, and then by
> chance later Buildroot package B builds and installs libfoo into
> HOST_DIR/lib, this script will believe that package A is correct, as
> it finds libfoo in HOST_DIR/lib, even though while package A was being
> built, the libfoo being detected was the system one. Detecting this
> would require running check-host-libs at the end of each package
> build, but that would imply re-checking over and over again all host
> binaries/libraries, which could have a noticeable impact on the build
> time. So for now, we simply check at the end of the build, which
> should already help to detect a lot of interesting bogus situations.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni at bootlin.com>
> ---
> It would be very useful if a few people could apply this patch to
> their local tree, run their usual build, and see how it behaves. This
> way, I can get some feedback to address the most obvious issues before
> it gets merged and starts causing build failures in the autobuilders.
> ---
> Makefile | 1 +
> support/scripts/check-host-libs | 36 +++++++++++++++++++++++++++++++++
> 2 files changed, 37 insertions(+)
> create mode 100755 support/scripts/check-host-libs
>
> diff --git a/Makefile b/Makefile
> index ec7c034ac1..7ba8ccd535 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -712,6 +712,7 @@ STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.t
> host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK)
> @$(call MESSAGE,"Finalizing host directory")
> $(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR))
> + ./support/scripts/check-host-libs $(HOST_DIR)
>
> .PHONY: staging-finalize
> staging-finalize: $(STAGING_DIR_SYMLINK)
> diff --git a/support/scripts/check-host-libs b/support/scripts/check-host-libs
> new file mode 100755
> index 0000000000..ef307bb6dd
> --- /dev/null
> +++ b/support/scripts/check-host-libs
> @@ -0,0 +1,36 @@
> +#!/bin/bash
> +
> +HOST_DIR=$1
> +
> +if test -z "${HOST_DIR}" ; then
> + echo "usage: check-host-libs HOST_DIR"
> + exit 1
> +fi
> +
> +bailout="no"
> +
> +for f in $(find ${HOST_DIR}/*bin ${HOST_DIR}/lib* -type f); do
> + mime=$(file -b --mime-type ${f})
> + if test "${mime}" != "application/x-sharedlib" -a \
> + "${mime}" != "application/x-executable" ; then
> + continue
> + fi
> + for lib in $(LC_ALL=C readelf -d ${f} | grep NEEDED | sed 's,.*Shared library: \[\(.*\)\].*,\1,'); do
> + case ${lib} in
> + libc.so*|libm.so*|libstdc++.so*|libpthread.so*|libgcc_s.so*|libdl.so*|ld-*|libgomp.so*|libcrypt.so*|libcrypto.so*|libatomic.so*)
> + continue
> + ;;
> + *)
> + if test -e ${HOST_DIR}/lib/${lib} ; then
> + continue
> + fi
> + echo "ERROR: in ${f}, ${lib} unknown"
> + bailout="yes"
> + ;;
> + esac
> + done
> +done
> +
> +if test "${bailout}" = "yes" ; then
> + exit 1
> +fi
> --
> 2.37.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
--
____________
.-----------------.--------------------: _ :------------------.
| Yann E. MORIN | Real-Time Embedded | __/ ) | /"\ ASCII RIBBON |
| | Software Designer | _/ - /' | \ / CAMPAIGN |
| +33 638.411.245 '--------------------: (_ `--, | X AGAINST |
| yann.morin (at) orange.com |_=" ,--' | / \ HTML MAIL |
'--------------------------------------:______/_____:------------------'
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
More information about the buildroot
mailing list