[Buildroot] Some analysis of the major build failure reasons

Tue Aug 3 06:29:12 UTC 2021

Hi James,

> Il giorno 3 ago 2021, alle ore 04:44, James Hilliard <james.hilliard1 at gmail.com> ha scritto:
> 
> On Mon, Aug 2, 2021 at 4:56 PM Giulio Benetti
> <giulio.benetti at benettiengineering.com> wrote:
>> 
>> Hello Thomas, James, All,
>> 
>>> On 8/2/21 11:46 PM, Thomas Petazzoni wrote:
>>> Hello,
>>> 
>>> Giulio, Bernd, Adam, there are questions for you below. Thanks for your
>>> support!
>>> 
>>> On Mon, 02 Aug 2021 06:09:39 -0000
>>> Thomas Petazzoni <thomas.petazzoni at bootlin.com> wrote:
>>> 
>>>>     master   | 1903 | 802 | 53  | 2758 |
>>> 
>>> So it's getting a bit better, but we still have lots of build failures.
>>> 
>>>> Classification of failures by reason for master
>>>> -----------------------------------------------
>>>>                  pixman-0.40.0 | 31
>>> 
>>> I have investigated this. It fails only on sh4, due an internal
>>> compiler error. It only occurs at -Os, at -O0 and -O2 it builds fine. I
>>> have reported gcc bug
>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101737 for this. Since I
>>> tested only with gcc 9.3.0 for now, I've started a build with gcc 11.x,
>>> to see how it goes.
>>> 
>>> Based on the result, I'll send a patch adding a new
>>> BR2_TOOLCHAIN_GCC_HAS_BUG_101737 and disable -Os on pixman on SuperH
>>> based on this.
>> 
>> I can do that since I've treated a lot of gcc bugs. Is it ok?
>> 
>>>>                        unknown | 31
>>> 
>>> I did not look into these for now.
>> 
>> I've taken a look into the last ones of today(the 31 ones)
>> @James: lot of your builds simply stuck after Make has finished(not on
>> linking but exactly on 'make: Leaving directory').
>> 
>> I've noticed it time ago some time, and now it became very more
>> frequent. This is 9/10 your autobuilder giving that problem.
>> And this happens at different Build Duration:
>> http://autobuild.buildroot.net/?reason=unknown
>> Also I see you use /tmp/ folder but I don't see anyone else doing that.
>> Isn't it maybe that your distro automatically cleans /tmp folder up? Or
>> it is mapped somewhere where disk gets full randomly?
>> I would move it to a specific user folder(i.e. buildroot user) and that
>> should fix the problem. If you're trying to do such thing to save disk
>> space I tell you that I've already done it time ago with this patch that
>> now needs to be rebased:
>> https://patchwork.ozlabs.org/project/buildroot/patch/20180919114541.17670-1-giulio.benetti@micronovasrl.com/
>> But in my case NOK was clear. I'm pretty sure /tmp/ is the problem.
> 
> The autobuild service is run with a private temporary tmp directory,
> I use a similar setup for my x399 autobuilder:
> [Unit]
> Description=Buildroot autobuild
> After=network.target
> ConditionPathExists=/etc/default/buildroot-autobuild
> StartLimitIntervalSec=0
> 
> [Service]
> User=autobuild
> Group=autobuild
> Type=simple
> PrivateTmp=yes
> WorkingDirectory=/tmp
> ExecStart=/opt/buildroot-test/scripts/autobuild-run --config
> /etc/default/buildroot-autobuild
> 
> Restart=always
> 
> [Install]
> WantedBy=multi-user.target

Ah, I sounds everything in order. Can you please try to issue it manually the exact script on a shell and see for 1 day long what happens?

Giulio

> 
>> 
>> On @Thomas autobuilder I see failure for:
>> - optee-client(already found NOK):
>> http://autobuild.buildroot.net/results/5e9/5e91bc53c3fbcd2ed232db79dc5c947394d66a1e/
>> - failure on fetching as mentioned above
>> 
>>>>                   zeromq-4.3.4 | 30
>>> 
>>> Giulio: this is happening only on or1k, with a binutils assert. Do you
>>> think this is solved by your or1k fixes?
>> 
>> It is. Those build failures are due to the use of binutils-2.33.1 that
>> have not patches for or1k. While all the other binutils versions have
>> local or upstreamed or1k patches.
>> 
>> Of course this can still happen with actual external Bootlin or1k
>> toolchain that use exactly binutils-2.33.1 not patched. So the problem
>> will be solved once you recompile and bump them with patches provided. I
>> still see that we're on 2020.08-1:
>> https://git.buildroot.net/buildroot/tree/toolchain/toolchain-external/toolchain-external-bootlin/toolchain-external-bootlin.mk#n586
>> 
>>>>                libfuse3-3.10.4 | 23
>>> 
>>> This only happens on Microblaze, with:
>>> 
>>> ../lib/fuse_loop_mt.c:361:1: error: symver is only supported on ELF platforms
>>> 
>>> Giulio, does it ring a bell to you ? It's weird because Microblaze is
>>> using ELF binaries.
>> 
>> I'm going to look into this.
>> 
>>>>                nfs-utils-2.5.4 | 19
>>> 
>>> I assume would be fixed by
>>> https://patchwork.ozlabs.org/project/buildroot/patch/20210802172116.10073-1-petr.vorel@gmail.com/
>> 
>> Yes, it does. This bug confused me because of lacking of -werror.
>> 
>>>>                      gpsd-3.21 | 17
>>> 
>>> Giulio, the workaround to pass -O0 no longer works for some reason. -O0
>>> is still passed, but -O2 is passed *after*, causing the build issue to
>>> pop up again. Could you have a look ?
>> 
>> Sure, some upstream change that makes my work-around useless.
>> 
>> NOTE:
>> I also add libnss-3.68 new bug on Aarch64_BE to be fixed. I'm already
>> working on it on spare time.
>> 
>> PS: I've found my autobuilders stopped, I think I've forgotten to
>> restart the daemon after updating the Distro. Now they're up and running.
>> 
>> Best regards
>> --
>> Giulio Benetti
>> Benetti Engineering sas