[Buildroot] Flutter-engine package help wanted.

[Adam Duskett]

Hey Arnout!

On Sun, Jul 16, 2023 at 8:32 AM Arnout Vandecappelle <arnout at mind.be> wrote:

>
>
> On 15/07/2023 05:02, Adam Duskett wrote:
> > Hello! I have a task to port Flutter to Buildroot and to make the
> packages
> > suitable for the master branch.
> >
> > I have talked with Yann in IRC about the problems facing the first
> package,
> > flutter-engine [https://github.com/flutter/engine
> > <https://github.com/flutter/engine>]. However, after another full
> > day of hacking away at the package, it's time to talk about a significant
> > problem I must overcome.
> >
> > Here is a summary of the issue I am facing and the main reason for the
> email:
> >
> > Flutter is developed and maintained by Google.
> > Of course, it stands to reason that Flutter does not use a conventional
> > configure/build system. Oh no, it couldn't be that easy. Instead,
> Flutter uses
> > gclient and gn from the chromium depot_tools repository, which contains
> all
> > sorts of exciting utilities. One of which is gclient.
>
>   My first reaction was: oh, we'll need specific download infrastructure
> like
> for go and cargo...
>
> > What is gclient? From:
> https://www.chromium.org/developers/how-tos/depottools
> > <https://www.chromium.org/developers/how-tos/depottools>
> >
> > gclient is a Python script to manage a workspace of modular dependencies
> that
> > are each checked out independently from different subversion or git
> > repositories. Features include:
> >
> >    - Dependencies can be specified on a per-OS basis.
> >    - Dependencies can be specified relative to their parent dependency.
> >    - Variables can be used to abstract concepts.
> >    - Hooks can be specified to be run after a checkout.
> >    - .gclient and DEPS are Python scripts. You can hack in easily or add
> >       additional configuration data.
>
>   I haven't looked at the details, but it sounds like even with separate
> download infrastructure it may be tricky. It sounds like it might be
> difficult
> to decouple the download from the configuration and/or build..
>


>   (And also bad: this stuff is running arbitrary code before checking the
> hash.
> But we have the same problem with go and cargo already as well.)
>
> Extremely difficult unfortunately. :(


> > .gclient file: It's the primary file. It is, in fact, a Python script. It
> > specifies the following variables:
> >
> >    - solutions: an array of dictionaries specifying the projects that
> will be
> >                 fetched.
> >    - hooks: additional hooks to be run when this meta checkout is synced.
> >    - target_os: an optional array of (target) operating systems to fetch
> >                 OS-specific dependencies for.
> >    - cache_dir: Primarily for bots, multiple working sets use a single
> git
> >                 cache.
> >
> >
> > Why is this a significant issue?
> >
> > The release tarballs from https://github.com/flutter/engine
> > <https://github.com/flutter/engine> are in no state to
> > compile! No, they are only for the use of gclient to download and build a
> > source directory structure suitable to build the Flutter engine! If you
> > download, extract and attempt to run
> > ./tools/gn --no-goma --no-prebuilt-dart-sdk,
> > you receive the error message:
> > No such file or directory: 'flutter/flutter/third_party/gn/gn.'
> >
> > But wait! Wasn't the gn binary just called? No, that's a wrapper in the
> Flutter
> > source tree that formats arguments to call the real gn binary. The real
> gn is
> > not provided in the tarball but is downloaded via gclient
> > (among many other repositories.)
> >
> > The above leads to the question: What do we do about this? We can't use
> the
> > existing download infrastructure to download the tarball from Git Hub,
> as that
> > isn't meant for us. The gclient download works if called out directly
> via a
> > pre_download hook; however, this leads to other issues in recreating a
> > consistent tarball because gclient writes the date to various files all
> around
> > the directory, thus changing the shasum every time I attempt to generate
> a
> > tarball. I can sed the dates in a flutter-engine-gen-tarball script I've
> made,
>
>   You could also use fakedate to pre-set the date to a fixed value.
>
>   Or could you patch gclient to not do that rewriting?
>
> I didn't think about fakedate, I'll play around with it!


> > which solves the problem; however, there is one last hurdle to overcome:
> When I
> > compress the directory created by gclient, any .git folder in a
> subdirectory
> > also changes when running mk_tar_gz from the script.
>
>   Huh? That shouldn't happen... mk_tar_gz doesn't do anything that
> modifies the
> filesystem, other than changing access time filestamps. And it certainly
> doesn't
> do anything specific for .git directories.
>
> Yeah, I misremembered. It's actually gclient (again) creating new
checkouts on a clone with the current date.


> > The .git directories, of
> > course, can be removed, and that DOES create a reproducible tarball, but
> the
> > build system relies on the .git directories to fill in a bunch of
> information
> > before configuring, causing failures.
> >
> > I won't ask the best course of action because anything we decide will be
> pretty
> > ugly, thanks to Google and its decisions. Instead, I will ask, what is
> the
> > least-bad solution that can be upstreamed to the master branch?
>
>   If we're going for ugly solutions: one option would be to download
> everything
> that gclient downloads in a pre-download hook, create an extra tarball out
> of
> it, and in a pre-extract hook extract that tarball. That way, what we
> present to
> gn in the configure and build steps looks like an already-expanded gclient
> tree
> and it won't download again. It's of course horrible to find out what all
> needs
> to be downloaded and where to put this stuff, but you can e.g. write a
> script
> that monkey-patches gclient to get URLs instead of actually downloading.
> But in
> the end, I guess there's not much improvement compared to just doing the
> download by calling gclient/gn.
>

A pre-download hook is exactly what I used at first, but creating a
reproducible
tarball was.. painful. I wasn't aware of fakedate, however, so I'll give
that a try!

  Regards,
>   Arnout
>
>
> >
> > If you want to play around with any of this, you can clone
> > git at gitlab.com:aduskett/buildroot-flutter.git and build the
> flutter_defconfig.
> >
> > Thanks!
> > Adam
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildroot.org/pipermail/buildroot/attachments/20230717/53a870e3/attachment-0001.html>