[Buildroot] [PATCH v2 2/3] package/ripgrep: bump to version 14.0.3

Sun Dec 31 14:25:18 UTC 2023

All,

On 2023-12-31 12:01 +0100, Yann E. MORIN spake thusly:
> On 2023-12-31 10:25 +0100, Yann E. MORIN spake thusly:
> > On 2023-12-30 20:44 +0100, Yann E. MORIN spake thusly:
> > > On 2023-12-30 19:00 +0100, Thomas Petazzoni via buildroot spake thusly:
> > > > Note that I also encountered some hash mismatch on some Rust/Cargo
> > > > package recently, and BR was falling back to sources.buildroot.net. I
> > > > did not have the time to investigate at the time (I was looking into
> > > > another issue, and didn't want to enter an infinite recursion of
> > > > problem solving quest). And now, I don't remember with which package I
> > > > encountered this. But yes, it seems like we have a reproducibility
> > > > issue.
[--SNIP--]
> So, we're now left with just the cargo version issue... Which is in fact
> rather good news: we do not have an unknown dependency on the build
> environment.

So, I could not resist starting a git-bisect.

I assumed that the issue was due to some rust version bump. So, now that
I had established that the conversion of ripgrep over to the cargo infra
was indeed correct, and that the bump to rust 1.74.1 (i.e. master as it
is today) produces an incorrectly vendored tarball, I started a git
bisect looking for the rust version bump that introduced the failure:

    $ cat defconfig
    BR2_arm=y
    BR2_cortex_a7=y
    BR2_TOOLCHAIN_EXTERNAL=y
    BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
    BR2_BACKUP_SITE=""
    BR2_PACKAGE_RIPGREP=y

    $ export BR2_DL_DIR=$(pwd)/trash-dl

    $ git bisect 05392a5eae6 e27a700f3e140 -- package/rust-bin/
    $ rm -rf "${BR2_DL_DIR}"; make clean; make ripgrep-source

Rince and repeat until the first bad commit is found, and... tada...
It's the bump to rust 1.74.1 that casues the build failure!

Wait, wait, no, that can't be correct: I remember that 05392a5eae6^ was
bad. And indeed, a manual test on 05392a5eae6^ yields a broken
vendoring. Damn.

OK, so e27a700f3e140 *is* defnitely correct, while both 05392a5eae6^ and
05392a5eae6 *are* definitely bad, and there was no rust version bump
in-between. So something else causes the bad vendoring.

Let's start a full git bisect on the whole tree, not limited to jsut
rust version bumps:

    $ export BR2_DL_DIR=$(pwd)/trash-dl

    $ git bisect 05392a5eae6 e27a700f3e140
    $ rm -rf "${BR2_DL_DIR}"; make clean; make ripgrep-source

Rnce and repeat, and after quite a few more tests compared to the
previos bisect, we eventually find the first bad commit.

Sorry, that's all my fault:

    768f9f80f62c1da6e298c680f0f4bfa887f38c78 is the first bad commit
    commit 768f9f80f62c1da6e298c680f0f4bfa887f38c78
    Author: Yann E. MORIN <yann.morin.1998 at free.fr>
    Date:   Wed Sep 13 00:15:49 2023 +0200

        support/download: generate even more reproducible tarballs

        When we generate the taballs off a local working copy of a VCS tree,
        the umask is the one that we enforce in out top-level Makefile.

        However, it is possible that a user manually tinkers in said working
        copy (e.g. to check an upstream bug fix, or regression). If the user
        umask is different from the one Buildroot enfirces, such tinkering
        can impact the mode bits of the files, even if their content is not
        modified.

        When we eventually need to create a tarball from said working copy,
        the VCS (e.g. git) will only be interested in checking whether the
        content of the files have changed before chcking them out, and will
        not look at, and restore/fix the mode bits.

        As a consequence, we may create non-reproducible archives.

        We fix that by enforcing the mode bits on the files before we create
        the tarball: we disable the write and execute bits, and only set the
        execute bit if the user execute bit is set.

        Signed-off-by: Yann E. MORIN <yann.morin.1998 at free.fr>
        Cc: Vincent Fazio <vfazio at xes-inc.com>
        Signed-off-by: Thomas Petazzoni <thomas.petazzoni at bootlin.com>

     support/download/helpers | 3 +++
     1 file changed, 3 insertions(+)

So, I am not 100% sure how to fix that...

For master, it's relatively trivial: we can bump the version and compute
a new hash, and be done with that.

But for the 2023.11 maintenance branch, which contains the commit above,
we do not want to bump the versions. For ripgrep, we have a way out: we
can chop one digit off the end of the commit hash, and recompute the new
hash, and that's going to be OK. For sentry-cli, since we use a version
string, not a hash, we don't have much choice but to switch over to
using a commit hash from github...

And more generally, we do not have a way to express the fact that our
download infra changes. For git and svn, we do have a per-backend
version, but not for the generic infra, nor for the vendoring
post-processs...

I'll think a bit more on that, and send patches as soon as I can...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'