[Buildroot] [PATCH] infra: add the transient download mechanism
Thomas Petazzoni
thomas.petazzoni at bootlin.com
Wed Apr 8 06:39:03 UTC 2020
Hello,
On Wed, 15 Jan 2020 21:37:53 +0100
"Yann E. MORIN" <yann.morin.1998 at free.fr> wrote:
> NOTE: NOT TO BE COMMITTED!
>
> This patch is a proof of concept for the transient download mechanism,
> that would help using branches as versions, and keep using the head of
> the branch each time a build started.
So what is the status of this patch? I see Yann you marked it as to be
ignored by patchwork, so we don't have it in patchwork.
I think there's generally been some positive feedback, with Arnout
saying:
""
Bottom line: even though repo is the better tool to manage the trees both for
development and release, the transient mechanism is a good quick win.
""
> A package declares its download as transient with:
> FOO_DOWNLOAD_TRANSIENT = YES
>
> this means that the download infra will not use any already downloaded
> archive, and will instead always download it as if missing.
>
> Since the check is done in the download wrapper, we have no TOCTOU race
> in case two bulds would attempt the same transient download: the archive
> is only replaced ato,mically as usual.
Typo: atomically
> So, if the package uses a branch as version, the branch's HEAD at that
> very moment will be redownloaded.
>
> This stil has the drawback that two builds in parallel may get slightly
> different content for the same branch, and the first build could end up
> using the download of the second build:
>
> build-1 build-2
>
> download
> | download
> | |
> save to dl-dir |
> [...] save to dl-dir
> extract
That is not a drawback, that is exactly what you expect with this
feature.
> Furthermore, even with a single build, it might get what it expects:
>
> developer-1 developer-2
>
> git push branch
> trigger CI git push branch
> [...]
> download
>
> In that case the build of delopper-1 would get the code of developper-2
> who pushed on the same branch.
Ditto here: this is exactly what we expect for a transient download
feature. If developer-1 wants to test only his own feature, then he has
to trigger CI on a branch that only he has control over.
There is however one drawback:
* You run a build (1), which downloads package A from a Git repo, and
places the tarball in global DL_DIR.
* You run a different build (2), which downloads the same package A
from the Git repo. Due to the transient download mechanism being
enabled for package A, it re-downloads it, and places an updated
tarball in the global DL_DIR.
* Back to build (1), you run "make legal-info", which picks up the
updated tarball, which was generated by build (2).
So, in build (1), your image uses the "previous" version of package A,
while your legal-info contains the "new" version of package A.
To solve this, one solution would be to use a per-build download
directory for the transient packages, rather than the global DL_DIR.
But I think this could be a separate improvement. Indeed, the transient
download mechanism is clearly oriented towards development/CI workflow,
where consistency of legal-info may not be a primary concern. For a
final release build, you would not use TRANSIENT_DOWNLOAD = YES.
For example, custom packages that make use of TRANSIENT_DOWNLOAD could
be implemented like this:
ifeq ($(BR2_DEVELOPER_MODE),y)
<pkg>_TRANSIENT_DOWNLOAD = YES
<pkg>_VERSION = some-branch
else
<pkg>_VERSION = some-tag
endif
So, while in development mode, users and CI use BR2_DEVELOPER_MODE=y,
but for final releases, BR2_DEVELOPER_MODE is disabled, which ensures
we have a fully reproducible build, and consistent legal-info output.
Yann, do you think you could resubmit your patch, for actual merging ?
Of course, we don't need the busybox.mk change.
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
More information about the buildroot
mailing list