[Buildroot] [RFC v4 00/16] Add per-package staging feature
Fabio Porcedda
fabio.porcedda at gmail.com
Sun Jun 28 19:42:34 UTC 2015
Hi all,
this patch set aims to improve build reproducibility by using a
per-package staging directory.
Also this feature aims to enable the top-level parallel make.
This patch set is available at:
https://github.com/fabio-porcedda/buildroot/tree/pps-v4
Every package instead of installing the files in the shared staging
directory (STAGING_DIR) install all files in a subdirectory of the
STAGINGPKG_DIR.
At the beginning of the configure stage, all the dependencies staging
directory are copied in the package staging directory, e.g.:
zlib-install-staging: install files in /output/stagingpkg/zlib/
cramsfs-configure: copy all files from /output/stagingpkg/zlib/* to
/output/stagingpkg/cramfs/
The STAGING_DIR variable is redefined for every package in order to
point to the per-package directory, so a package cannot access
automatically to staging files of packages that aren't in the
(PKG)_DENDENDENCIES variable.
To improve performance and reduce space utilization hard links are used.
To evaluate performance and space utilization I've done some tests.
HW-MED:
CPU: Intel i7-4790K (4x2 @4GHz) = 8 threads
RAM: 16GB
SSD: 512GB
HW-HIGH:
CPU: 2 x Xeon 2620v3 (6x2 @2.40Ghz) = 24 threads
RAM: 32GB
SATA: RAID0 2x2TB 3.5" 7200rpm
defconfig-small, long chain dependency, unfavorable scenario for parallelization:
BR2_arm=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_PACKAGE_XORG7=y
BR2_PACKAGE_XSERVER_XORG_SERVER=y
BR2_PACKAGE_LIBGTK3=y
defconfig-full:
BR2_arm=y
BR2_TOOLCHAIN_EXTERNAL=y
make allyespackageconfig
Also i removed few packages that do not built on my system.
make show-targets | wc -w
1366
Test about output using hard links or note using the "defconfig-full"
on HW-MED:
| 35GB | 99m | master branch
| 37GB | 100m | patch set
| 153GB | 105m | patch set using hard links only for the toolchain sysroot
| 225GB | 106m | patch set not using hardlinks at all
Test about performances of this patch set vs master branch:
| 199m | HW-MED | defconfig-full | master branch | no top-level make |
| 99m | HW-MED | defconfig-full | master branch | top-level make |
| 100m | HW-MED | defconfig-full | patch set | top-level make |
| 350m | HW-HIGH | defconfig-full | master branch | no top-level make |
| 73m | HW-HIGH | defconfig-full | master branch | top-level make |
| 77m | HW-HIGH | defconfig-full | patch set | top-level make |
| 10m | HW-MED | defconfig-small | master branch | no top-level make |
| 5m | HW-MED | defconfig-small | master branch | top-level make |
| 5m | HW-MED | defconfig-samll | patch set | top-level make |
| 21m18s | HW-HIGH | defconfig-small | master branch | no top-level make |
| 7m53s | HW-HIGH | defconfig-small | master branch | top-level make |
| 7m54s | HW-HIGH | defconfig-samll | patch set | top-level make |
TODO:
- use the toolchaing wrapper even for the internal toolchain
- rebase over master (this patch set is based on 2015.05-161-gca5f11e).
- more testing (tested only for arm target).
Best regads
Fabio Porcedda (16):
packages: use TARGET_MAKE_ENV to add pps support
packages: for staging stuff use backtick instead of the shell function
packages/scons: make avaiable the GCC_SYSROOT environment variable
packages/qt: read spec files from the per-package staging directory
pkg-perl: use TARGET_MAKE_ENV to add pps support
tooclahin-external: add support for GCC_SYSROOT
toolchain-external: move some code in a new function
toolchain-external: use the wrapper even for the linker "ld"
infra: add GCC_SYSROOT environment variable to support pps feature
pkg-cmake: add GCC_SYSROOT environment variable to support pps feature
pkg-python: add GCC_SYSROOT variable to add pps support
pacakge/luarocks: add GCC_SYSROOT environment variable to support pps
package/pkgconf: use GCC_SYSROOT to support pps feature
Makefile: add STAGINGNOPKG_DIR variable
pkg-generic: ADD_TOOLCHAIN_DEPENDENCY is true only for target packages
infra: add per-package staging feature
Makefile | 16 ++--
package/Makefile.in | 5 +-
package/aiccu/aiccu.mk | 2 +-
package/blktrace/blktrace.mk | 2 +-
package/boost/boost.mk | 2 +-
package/botan/botan.mk | 2 +-
package/bsdiff/bsdiff.mk | 4 +-
package/dbus-python/dbus-python.mk | 8 +-
package/dhcpdump/dhcpdump.mk | 3 +-
package/dtc/dtc.mk | 3 +-
package/exim/exim.mk | 5 +-
package/fbv/fbv.mk | 2 +-
package/feh/feh.mk | 4 +-
package/gadgetfs-test/gadgetfs-test.mk | 2 +-
package/gdk-pixbuf/gdk-pixbuf.mk | 2 +-
package/iodine/iodine.mk | 2 +-
package/iw/iw.mk | 2 +-
package/jack2/jack2.mk | 3 +-
package/libmicrohttpd/libmicrohttpd.mk | 2 +-
package/libnss/libnss.mk | 4 +-
package/libupnpp/libupnpp.mk | 2 +-
package/lmbench/lmbench.mk | 8 +-
package/logrotate/logrotate.mk | 2 +-
package/lua-msgpack-native/lua-msgpack-native.mk | 3 +-
...ctions-to-the-global-environment-of-the-c.patch | 54 ++++++++++++
package/luarocks/0002-fixed-typo.patch | 27 ++++++
...e-environment-table-for-the-system-config.patch | 76 +++++++++++++++++
...is-not-being-integrated-bug-introduced-in.patch | 62 ++++++++++++++
package/luarocks/luarocks.mk | 11 ++-
package/madplay/madplay.mk | 2 +-
package/moarvm/moarvm.mk | 2 +-
package/mongoose/mongoose.mk | 8 +-
package/mpg123/mpg123.mk | 2 +-
package/mplayer/mplayer.mk | 2 +-
package/ncurses/ncurses.mk | 4 +-
package/neardal/neardal.mk | 2 +-
package/nginx/nginx.mk | 1 +
package/nmap/nmap.mk | 2 +-
package/open2300/open2300.mk | 2 +-
package/openldap/openldap.mk | 2 +-
package/openssl/openssl.mk | 2 +-
package/opentyrian/opentyrian.mk | 2 +-
package/pax-utils/pax-utils.mk | 3 +-
package/php/php.mk | 4 +-
package/pifmrds/pifmrds.mk | 3 +-
package/pkg-cmake.mk | 1 +
package/pkg-generic.mk | 43 +++++++++-
package/pkg-perl.mk | 2 +-
package/pkg-python.mk | 2 +
package/pkgconf/pkg-config.in | 4 +-
package/pkgconf/pkgconf.mk | 5 +-
package/poco/poco.mk | 3 +-
package/poppler/poppler.mk | 2 +-
package/pppd/pppd.mk | 2 +-
package/python-gobject/python-gobject.mk | 4 +-
package/python-pygame/python-pygame.mk | 6 +-
package/python-pyqt/python-pyqt.mk | 1 +
package/python-sip/python-sip.mk | 2 +-
package/qt/qt.mk | 18 ++--
package/qtuio/qtuio.mk | 3 +-
package/rtmpdump/rtmpdump.mk | 2 +-
package/scons/scons.mk | 3 +-
package/scons/site_init.py | 19 +++++
package/speex/speex.mk | 3 +-
package/statserial/statserial.mk | 3 +-
package/tcpreplay/tcpreplay.mk | 6 +-
package/ti-utils/ti-utils.mk | 6 +-
package/tvheadend/tvheadend.mk | 4 +-
package/upmpdcli/upmpdcli.mk | 2 +-
package/ushare/ushare.mk | 3 +-
package/vsftpd/vsftpd.mk | 4 +-
support/misc/toolchainfile.cmake.in | 9 +-
.../toolchain-external/ext-toolchain-wrapper.c | 96 ++++++++++++++--------
toolchain/toolchain-external/toolchain-external.mk | 2 +-
74 files changed, 476 insertions(+), 147 deletions(-)
create mode 100644 package/luarocks/0001-added-2-functions-to-the-global-environment-of-the-c.patch
create mode 100644 package/luarocks/0002-fixed-typo.patch
create mode 100644 package/luarocks/0003-fix-use-same-environment-table-for-the-system-config.patch
create mode 100644 package/luarocks/0004-config.lua-is-not-being-integrated-bug-introduced-in.patch
create mode 100644 package/scons/site_init.py
--
2.4.3
More information about the buildroot
mailing list