[Buildroot] [PATCH 1/1] package/bearssl: disable parallel build

[James Hilliard]

On Mon, Oct 17, 2022 at 8:21 AM Thomas Petazzoni
<thomas.petazzoni at bootlin.com> wrote:
>
> On Mon, 17 Oct 2022 07:29:52 -0400
> James Hilliard <james.hilliard1 at gmail.com> wrote:
>
> > The shuffle=random changes don't require modifying autobuild-run, I'm
> > just tweaking the path to make in my autobuild service environment and
> > then passing an additional config flag to autobuild-run like this:
> > --make-opts="--shuffle=random"
>
> Right, but my point stands: it's custom usage/configuration of
> autobuild-run that we would prefer to discuss on the mailing list first
> before deploying/enabling.

I've generally been deploying autobuild-run tweaks after initial local testing.

Typically I send them to the mailing list after running them deployed for
a little to clean them up and make sure they work as expected and verify
the builds that are failing are not due to bugs in the autobuild-run tweaks
themselves.

They don't usually get looked at for a while in patchwork so I haven't really
been bringing them up on the mailing list until after I've tested/deployed
them myself for a little, I have a couple pending in patchwork at the moment
for autobuild-run for example:
https://patchwork.ozlabs.org/project/buildroot/patch/20220607231211.533086-1-james.hilliard1@gmail.com/
https://patchwork.ozlabs.org/project/buildroot/patch/20220718043901.873466-1-james.hilliard1@gmail.com/

Sometimes I override genrandconfig as well to test patches such as:
https://patchwork.ozlabs.org/project/buildroot/patch/20220514214612.3221647-1-james.hilliard1@gmail.com/

I have a local patch to autobuild-run that redirects to an out of tree
genrandconfig so that I can test tweaks there. I also have it validating
that the in tree version of genrandconfig hasn't changed by comparing
against a known sha256 hash before using the out of tree tweaked version
to ensure that autobuild-run automatically falls back to using the in-tree
genrandconfig if genrandconfig gets changed/fixes in upstream buildroot.

For autobuild-run changes that increase error rates significantly due to
higher rates of bugs which need fixes in buildroot I sometimes have held off
on sending them as patches until the rates of errors triggered get low enough
that it would make sense for others to use those changes as well.

In this case I was also holding off on a patch automatically enabling shuffle
mode to autobuild-run as using that feature currently requires running a yet
to be released version of make and was triggering a few issues in buildroot
that needed fixing on the buildroot side.

I also figured that by testing master make using the autobuilders we can
get make issues fixed upstream before the next make release, such as
this shuffle mode bug so far:
https://savannah.gnu.org/bugs/index.php?63215

What sort of autobuilder changes would you like me to discuss on the mailing
list before deploying? I tend to experiment a bit but am usually pretty careful
to make sure that my autobuilder tweaks only cause build failures due to real
bugs that need fixing in buildroot/buildroot packages.

>
> > Well builds were mostly failing in the same place due to deterministic
> > build ordering, I was trying to make it so that we would see failures
> > in different places as that's more useful compared with always seeing
> > the same errors.
> >
> > To me seeing the same errors most of the time indicates there's some
> > major coverage issues with the autobuilders which need fixing.
>
> I think you're missing a point here: many of the builds today are
> failing due to the glibc 2.36 bump. So pretty much every single build
> that is glibc based will indeed fail with some error related to this
> glibc bump.

Probably a lot were failing for a couple days due to this glibc bug:
https://github.com/buildroot/buildroot/commit/b749d6dd7b5fc6cc8b85a316878721d42654759c

There's probably some failures still due to this issue but most of the others
seem to be dependency bugs in package makefiles:
https://patchwork.ozlabs.org/project/buildroot/patch/20221016193014.3384022-1-james.hilliard1@gmail.com/

However I'm actually referring to a separate issue with test coverage
that has always been present.

>
> Your additional randomization will not change anything to this. It's
> just that we now have the glibc bump issues + the newly discovered
> issues related to the extra randomization + the usual amount of build
> issues.

When I brought patch up this on irc:
https://patchwork.ozlabs.org/project/buildroot/patch/20220514214612.3221647-1-james.hilliard1@gmail.com/

arnout mentioned:
2022-07-25.log-[03:28:02] <arnout> Lightsword: about the build time: I
think the issue is that if you increase the number of selected
packages, you also reduce the number of successful builds (because it
fails as soon as one failing package is selected) - and that way,
packages that get built late (i.e. that come late in the alphabet and
have no reverse dependencies) don't get built.

This is indeed the case when I tested increasing the probability further,
however by randomizing the dependency ordering using make 4.4's new
shuffle mode we should be able to mitigate the alphabetical ordering bias
issue and thus get better test coverage of packages with larger dependency
trees by increasing the number of selected packages while using shuffle mode.

Looking at the current autobuilder failure logs I'm seeing that most
autobuilders
other than my own have much lower build failure rates and not much variety
in terms of failure reasons due to poor coverage of packages with larger
dependency trees.

For my own autobuilders running with shuffle mode I'm seeing much higher
failure rates for different reasons so these changes do appear to be
increasing out test coverage overall, there's a lot of issues with makefile
dependency ordering that are new and I suspect once those are fixed
we'll be able to get much better overall coverage of more common issues,
especially issues hit when running parallel builds.

>
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com