[Buildroot] [PATCH v3 1/1] utils/scanpypi: add flit package support

[James Hilliard]

On Mon, Nov 28, 2022 at 2:40 AM Yegor Yefremov
<yegorslists at googlemail.com> wrote:
>
> Hi James,
>
> On Thu, Aug 18, 2022 at 10:26 AM James Hilliard
> <james.hilliard1 at gmail.com> wrote:
> >
> > These packages don't have a setup.py so we instead need to parse their
> > pyproject.toml file.
> >
> > Note that this currently doesn't handle flit package dependency
> > resolution.
> >
> > Signed-off-by: James Hilliard <james.hilliard1 at gmail.com>
> > ---
> > Changes v2 -> v3:
> >   - minor cleanup
> >   - rebase
> >   - more detailed  commit log
> > Changes v1 -> v2:
> >   - remove homepage format fixes(sent as separate patch)
> >   - remove load_setup fixes
> > ---
> >  utils/scanpypi | 78 ++++++++++++++++++++++++++++++++++++++++++++++++--
> >  1 file changed, 76 insertions(+), 2 deletions(-)
> >
> > diff --git a/utils/scanpypi b/utils/scanpypi
> > index 452b4a3fc3..bce7c01494 100755
> > --- a/utils/scanpypi
> > +++ b/utils/scanpypi
> > @@ -42,6 +42,48 @@ except ImportError:
> >            'pip install spdx_lookup')
> >      liclookup = None
> >
> > +def toml_load(f):
> > +    with open(f, 'rb') as fh:
> > +        ex = None
> > +
> > +        # Try regular tomli first
> > +        try:
> > +            from tomli import load
> > +            return load(fh)
> > +        except ImportError as e:
> > +            ex = e
> > +
> > +        # Try pip's vendored tomli
> > +        try:
> > +            from pip._vendor.tomli import load
> > +            try:
> > +                return load(fh)
> > +            except TypeError:
> > +                # Fallback to handle older version
> > +                try:
> > +                    fh.seek(0)
> > +                    w = io.TextIOWrapper(fh, encoding="utf8", newline="")
> > +                    return load(w)
> > +                finally:
> > +                    w.detach()
> > +        except ImportError as e:
> > +            pass
> > +
> > +        # Try regular toml last
> > +        try:
> > +            from toml import load
> > +            fh.seek(0)
> > +            w = io.TextIOWrapper(fh, encoding="utf8", newline="")
> > +            try:
> > +                return load(w)
> > +            finally:
> > +                w.detach()
> > +        except ImportError:
> > +            pass
> > +
> > +        print('This package needs tomli')
> > +        raise ex
> > +
> >
> >  def setup_decorator(func, method):
> >      """
> > @@ -316,6 +358,35 @@ class BuildrootPackage():
> >              os.chdir(current_dir)
> >              sys.path.remove(self.tmp_extract)
> >
> > +    def load_pyproject(self):
> > +        """
> > +        Loads the corresponding pyproject.toml and store its metadata
> > +        """
> > +        current_dir = os.getcwd()
> > +        os.chdir(self.tmp_extract)
> > +        sys.path.insert(0, self.tmp_extract)
> > +        try:
> > +            pyproject_data = toml_load('pyproject.toml')
> > +            try:
> > +                self.setup_metadata = pyproject_data.get('project', {})
> > +                self.metadata_name = self.setup_metadata.get('name', self.real_name)
> > +                build_system = pyproject_data.get('build-system', {})
> > +                build_backend = build_system.get('build-backend', None)
> > +                if build_backend is not None and build_backend == 'flit_core.buildapi':
>
> What about using the bool interpretation of None?
>                      if build_backend and build_backend == 'flit_core.buildapi':

Sure, changed to that style in v4:
https://patchwork.ozlabs.org/project/buildroot/patch/20221128182304.1915237-1-james.hilliard1@gmail.com/

>
> > +                    self.setup_metadata['method'] = 'flit'
> > +                elif build_system.get('backend-path', None) is not None:
> > +                    self.setup_metadata['method'] = 'pep517'
>
> What about detecting "setuptools"?
>                     elif 'setuptools.build_meta' in build_backend:
>                          self.setup_metadata['method'] = 'setuptools'
>
> See [1] for the reference.
>
> [1] https://github.com/pythongssapi/python-gssapi/blob/main/pyproject.toml#L6

We already try to use setuptools before pep517/pyproject.toml, there's some
issues there still but I was waiting for this to get merged before looking into
that in more detail, I don't think searching for setuptools from pyproject.toml
is likely to be all that helpful since setuptools is the implicit fallback when
pyproject.toml is not present in a pypi package anyways.

>
> Regards,
> Yegor
>
> > +                else:
> > +                    self.setup_metadata['method'] = 'unknown'
> > +            except KeyError:
> > +                print('ERROR: Could not determine package metadata for {pkg}.\n'
> > +                      .format(pkg=self.real_name))
> > +                raise
> > +        except FileNotFoundError:
> > +            raise
> > +        os.chdir(current_dir)
> > +        sys.path.remove(self.tmp_extract)
> > +
> >      def get_requirements(self, pkg_folder):
> >          """
> >          Retrieve dependencies from the metadata found in the setup.py script of
> > @@ -699,9 +770,12 @@ def main():
> >              except ImportError as err:
> >                  if 'buildutils' in str(err):
> >                      print('This package needs buildutils')
> > +                    continue
> >                  else:
> > -                    raise
> > -                continue
> > +                    try:
> > +                        package.load_pyproject()
> > +                    except Exception as e:
> > +                        raise
> >              except (AttributeError, KeyError) as error:
> >                  print('Error: Could not install package {pkg}: {error}'.format(
> >                      pkg=package.real_name, error=error))
> > --
> > 2.34.1
> >