[Buildroot] Flutter-engine package help wanted.

[Arnout Vandecappelle]

On 15/07/2023 05:02, Adam Duskett wrote:
> Hello! I have a task to port Flutter to Buildroot and to make the packages
> suitable for the master branch.
> 
> I have talked with Yann in IRC about the problems facing the first package,
> flutter-engine [https://github.com/flutter/engine 
> <https://github.com/flutter/engine>]. However, after another full
> day of hacking away at the package, it's time to talk about a significant
> problem I must overcome.
> 
> Here is a summary of the issue I am facing and the main reason for the email:
> 
> Flutter is developed and maintained by Google.
> Of course, it stands to reason that Flutter does not use a conventional
> configure/build system. Oh no, it couldn't be that easy. Instead, Flutter uses
> gclient and gn from the chromium depot_tools repository, which contains all
> sorts of exciting utilities. One of which is gclient.

  My first reaction was: oh, we'll need specific download infrastructure like 
for go and cargo...

> What is gclient? From: https://www.chromium.org/developers/how-tos/depottools 
> <https://www.chromium.org/developers/how-tos/depottools>
> 
> gclient is a Python script to manage a workspace of modular dependencies that
> are each checked out independently from different subversion or git
> repositories. Features include:
> 
>    - Dependencies can be specified on a per-OS basis.
>    - Dependencies can be specified relative to their parent dependency.
>    - Variables can be used to abstract concepts.
>    - Hooks can be specified to be run after a checkout.
>    - .gclient and DEPS are Python scripts. You can hack in easily or add
>       additional configuration data.

  I haven't looked at the details, but it sounds like even with separate 
download infrastructure it may be tricky. It sounds like it might be difficult 
to decouple the download from the configuration and/or build...

  (And also bad: this stuff is running arbitrary code before checking the hash. 
But we have the same problem with go and cargo already as well.)

> .gclient file: It's the primary file. It is, in fact, a Python script. It
> specifies the following variables:
> 
>    - solutions: an array of dictionaries specifying the projects that will be
>                 fetched.
>    - hooks: additional hooks to be run when this meta checkout is synced.
>    - target_os: an optional array of (target) operating systems to fetch
>                 OS-specific dependencies for.
>    - cache_dir: Primarily for bots, multiple working sets use a single git
>                 cache.
> 
> 
> Why is this a significant issue?
> 
> The release tarballs from https://github.com/flutter/engine 
> <https://github.com/flutter/engine> are in no state to
> compile! No, they are only for the use of gclient to download and build a
> source directory structure suitable to build the Flutter engine! If you
> download, extract and attempt to run
> ./tools/gn --no-goma --no-prebuilt-dart-sdk,
> you receive the error message:
> No such file or directory: 'flutter/flutter/third_party/gn/gn.'
> 
> But wait! Wasn't the gn binary just called? No, that's a wrapper in the Flutter
> source tree that formats arguments to call the real gn binary. The real gn is
> not provided in the tarball but is downloaded via gclient
> (among many other repositories.)
> 
> The above leads to the question: What do we do about this? We can't use the
> existing download infrastructure to download the tarball from Git Hub, as that
> isn't meant for us. The gclient download works if called out directly via a
> pre_download hook; however, this leads to other issues in recreating a
> consistent tarball because gclient writes the date to various files all around
> the directory, thus changing the shasum every time I attempt to generate a
> tarball. I can sed the dates in a flutter-engine-gen-tarball script I've made,

  You could also use fakedate to pre-set the date to a fixed value.

  Or could you patch gclient to not do that rewriting?

> which solves the problem; however, there is one last hurdle to overcome: When I
> compress the directory created by gclient, any .git folder in a subdirectory
> also changes when running mk_tar_gz from the script.

  Huh? That shouldn't happen... mk_tar_gz doesn't do anything that modifies the 
filesystem, other than changing access time filestamps. And it certainly doesn't 
do anything specific for .git directories.

> The .git directories, of
> course, can be removed, and that DOES create a reproducible tarball, but the
> build system relies on the .git directories to fill in a bunch of information
> before configuring, causing failures.
> 
> I won't ask the best course of action because anything we decide will be pretty
> ugly, thanks to Google and its decisions. Instead, I will ask, what is the
> least-bad solution that can be upstreamed to the master branch?

  If we're going for ugly solutions: one option would be to download everything 
that gclient downloads in a pre-download hook, create an extra tarball out of 
it, and in a pre-extract hook extract that tarball. That way, what we present to 
gn in the configure and build steps looks like an already-expanded gclient tree 
and it won't download again. It's of course horrible to find out what all needs 
to be downloaded and where to put this stuff, but you can e.g. write a script 
that monkey-patches gclient to get URLs instead of actually downloading. But in 
the end, I guess there's not much improvement compared to just doing the 
download by calling gclient/gn.

  Regards,
  Arnout


> 
> If you want to play around with any of this, you can clone
> git at gitlab.com:aduskett/buildroot-flutter.git and build the flutter_defconfig.
> 
> Thanks!
> Adam