[Buildroot] Github download helper possibly not working

[Yann E. MORIN]

Thomas, Woody, All,

On 2023-08-28 22:45 +0200, Thomas Petazzoni via buildroot spake thusly:
> On Mon, 28 Aug 2023 20:02:34 +0000
> Woody Douglass via buildroot <buildroot at buildroot.org> wrote:
> > I've noticed that packages that use the `github` download helper are 
> > falling over to buildroot mirrors. I've tried with packages `yaml-cpp`, 
> > `zlog`, `swupdate`, and `pcm-tools` -- all are redirected before 
> > eventually getting a 403 from codeload.github.com and falling back to 
> > sources.buildroot.net. Is anyone else seeing this behavior? I'm trying 
> > to find a workaround now, but I'd appreciate any help!
> It works fine here:

Works fine here too.

[--SNIP--]
> However, for pcm-tools, we do have a problem (though not the one you
> mentioned):
> $ make pcm-tools-source
[--SNIP--]
> ERROR: pcm-tools-202110.tar.gz has wrong sha256 hash:
> ERROR: expected: aa48ab1473720aeb7837b67bfc612100f484748720a8b8034daff00419709057
> ERROR: got     : 90a5931cea24f1b0da76e22c712e55375df157e87f26edaa70b9660405852725
> ERROR: Incomplete download, or man-in-the-middle (MITM) attack

I also have this issue, and indeed the cntent changed; here's the diff:

    diff -durN pcm-202110.old/version.h pcm-202110.new/version.h
    --- pcm-202110.old/version.h    2021-10-25 16:07:54.000000000 +0200
    +++ pcm-202110.new/version.h    2021-10-25 16:07:54.000000000 +0200
    @@ -1 +1 @@
    -#define PCM_VERSION " (2021-10-25 16:07:54 +0200 ID=93fc9193)"
    +#define PCM_VERSION " (2021-10-25 16:07:54 +0200 ID=93fc919)"

That's all: a delta in the length of the short hash.

So, what does version.h looks like in git (at the tag):

    $ cat version.h
    #define PCM_VERSION " ($Format:%ci ID=%h$)"

OK, does that ring a bell? Yes, that's the same thing that we solved for
subversion in c92be85e3a29 (support/download: make the svn backend more
reproducible):

    $ man 5 gitattributes
       Creating an archive
         export-subst
           If the attribute export-subst is set for a file then Git
           will expand several placeholders when adding this file to
           an archive. The expansion depends on the availability of
           a commit ID, i.e., if git-archive(1) has been given a tree
           instead of a commit or a tag then no replacement will be
           done. The placeholders are the same as those for the option
           --pretty=format: of git-log(1), except that they need to be
           wrapped like this: $Format:PLACEHOLDERS$ in the file. E.g.
           the string $Format:%H$ will be replaced by the commit hash.

This is something that is then done when the archive is generated, i.e.
on the github side. So, Github again changed the way they generate their
archives, except this is a very sneaky change.

So, for pcm-tools, the only solution we have is to drop use of the
github helper and switch over to a git download...

Long term, I am still of the opinion that we should no longer, ever,
rely on the remote to generate the archive, and we should always do
that localy, even at the cost of download bandwidth, because we too
often have similar issues, and this one is indeed very, very sneaky, and
there is nothing that prevents a remote to change their archive
generation on a whim (Github never guaranteed stability for those
autogenerated archives to begin with...)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'