GNU bug report logs - #26936
grub_cmd_set_date test sometimes fail

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 26936 in the body.
You can then email your comments to 26936 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: bug-guix <bug-guix <at> gnu.org>
Subject: grub_cmd_set_date test sometimes fail
Date: Sun, 14 May 2017 23:31:06 -0700

[Message part 1 (text/plain, inline)]

Hello,

I've been encountering this test failure a couple times (3 out of 4 last
build I've tried) recently on my GuixSD x86_64 system when building grub
version 2.02.

This test failure is not deterministic. The Guix automated build farm
has yet to stumble on that failure.

Maxim

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 26936 <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 26936 <at> debbugs.gnu.org
Subject: Re: bug#26936: grub_cmd_set_date test sometimes fail
Date: Mon, 15 May 2017 22:14:52 -0400

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

> I've been encountering this test failure a couple times (3 out of 4 last
> build I've tried) recently on my GuixSD x86_64 system when building grub
> version 2.02.

I've encountered the same problem on my x86_64 system.

      Mark

Message #11 received at 26936 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Mark H Weaver <mhw <at> netris.org>
Cc: 26936 <at> debbugs.gnu.org
Subject: Re: bug#26936: grub_cmd_set_date test sometimes fail
Date: Mon, 15 May 2017 20:57:19 -0700

Mark H Weaver <mhw <at> netris.org> writes:

> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:
>
>> I've been encountering this test failure a couple times (3 out of 4 last
>> build I've tried) recently on my GuixSD x86_64 system when building grub
>> version 2.02.
>
> I've encountered the same problem on my x86_64 system.
>
>       Mark

Thanks for raising the flag as well. I was starting to wonder if there
was something about my system which kept creating race conditions that
only it would suffer about, since many failures I've been finding in the
passed weeks were never triggered on hydra.

Maxim

Message #18 received at 26936-done <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 26936-done <at> debbugs.gnu.org
Subject: Re: bug#26936: grub_cmd_set_date test sometimes fail
Date: Sat, 03 Jun 2017 00:54:46 +0200

Hello,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:

> I've been encountering this test failure a couple times (3 out of 4 last
> build I've tried) recently on my GuixSD x86_64 system when building grub
> version 2.02.
>
> This test failure is not deterministic. The Guix automated build farm
> has yet to stumble on that failure.

I can reproduce it after a few tries.  Upon failure, the
‘test-suite.log’ file reads:

--8<---------------cut here---------------start------------->8---
FAIL: grub_cmd_set_date
=======================

qemu-system-i386: Trying to execute code outside RAM or ROM at 0xff62f000
This usually means one of the following happened:

(1) You told QEMU to execute a kernel for the wrong machine type, and it crashed on startup (eg trying to run a raspberry pi kernel on a versatilepb QEMU machine)
(2) You didn't give QEMU a kernel or BIOS filename at all, and QEMU executed a ROM full of no-op instructions until it fell off the end
(3) Your guest kernel has a bug and crashed by jumping off into nowhere

This is almost always one of the first two, so check your command line and that you are using the right type of kernel for this machine.
If you think option (3) is likely then you can try debugging your guest with the -d debug options; in particular -d guest_errors will cause the log to include a dump of the guest register state at this point.

Execution cannot continue; stopping here.

Test failed: 2017-06-02 19:03:48 Friday
--8<---------------cut here---------------end--------------->8---

If we add “set -x” in the ‘grub-shell’ script, which is used by the
test, we see that it invokes QEMU like this:

--8<---------------cut here---------------start------------->8---
+ ./grub-mkrescue --output=/tmp/guix-build-grub-2.02.drv-0/tmp.cwW8BQx8t8 --override-directory=./grub-core --rom-directory=/tmp/guix-build-grub-2.02.drv-0/tmp.d2FE5o8cWd --locale-directory=./po --themes-directory=./themes /boot/grub/grub.cfg=/tmp/guix-build-grub-2.02.drv-0/tmp.IxQUpfc7pT /boot/grub/testcase.cfg=/tmp/guix-build-grub-2.02.drv-0/tmp.v4GOg503yw
+ '[' xcd = xhd ']'
+ '[' xcd = xcd ']'
+ '[' i386-pc = arm64-efi ']'
+ '[' i386-pc = arm-efi ']'
+ '[' i386-pc = powerpc-ieee1275 ']'
+ device='cdrom '
+ bootdev='-boot d'
+ '[' xcd = xfd ']'
+ '[' xcd = xqemu ']'
+ '[' xcd = xmipsel_qemu ']'
+ '[' xcd = xmipsel_fulong2e ']'
+ '[' xcd = xmips_qemu ']'
+ '[' xcd = xcoreboot ']'
+ '[' i386-pc = mipsel-arc ']'
+ '[' xcd = xnet ']'
+ '[' xcd = xemu ']'
+ timeout -s KILL 60 qemu-system-i386 -nographic -monitor file:/dev/null -serial file:/dev/stdout -cdrom /tmp/guix-build-grub-2.02.drv-0/tmp.cwW8BQx8t8 -boot d
--8<---------------cut here---------------end--------------->8---

It would seem that the test uses unique files, which would rule out the
possibility of a race condition when running tests with -jN.

However, running “make check -j20” led to another failure, in
‘grub_cmd_sleep’:

--8<---------------cut here---------------start------------->8---
Interval not in range 1496434911-1496434898 != 10
--8<---------------cut here---------------end--------------->8---

All in all, I’m tempted to think this has to do with running the tests
in parallel.

As a stop-gap measure, I’ve committed a change to run tests
sequentially.  With this “guix build grub --rounds=5” passes for on my
x86_64 laptop.  Let’s reopen the bug if we find that’s not enough.

Ludo’.

Message #21 received at 26936 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: 26936 <at> debbugs.gnu.org
Subject: Re: bug#26936: grub_cmd_set_date test sometimes fail
Date: Fri, 23 Jun 2017 15:30:10 -0400

[Message part 1 (text/plain, inline)]

On Sat, Jun 03, 2017 at 12:54:46AM +0200, Ludovic Courtès wrote:
> FAIL: grub_cmd_set_date
 
[...]

> All in all, I’m tempted to think this has to do with running the tests
> in parallel.
> 
> As a stop-gap measure, I’ve committed a change to run tests
> sequentially.  With this “guix build grub --rounds=5” passes for on my
> x86_64 laptop.  Let’s reopen the bug if we find that’s not enough.

This test just failed for me on x86_64, even with #:parallel-tests? #f.

So, perhaps it's not an inter-test race, or perhaps the tests are still
running in parallel, or some other thing.

[signature.asc (application/pgp-signature, inline)]

Message #24 received at 26936 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: 26936 <at> debbugs.gnu.org
Subject: Re: bug#26936: grub_cmd_set_date test sometimes fail
Date: Sat, 24 Jun 2017 13:19:25 -0400

[Message part 1 (text/plain, inline)]

On Fri, Jun 23, 2017 at 03:30:10PM -0400, Leo Famulari wrote:
> On Sat, Jun 03, 2017 at 12:54:46AM +0200, Ludovic Courtès wrote:
> > FAIL: grub_cmd_set_date
> 
> > All in all, I’m tempted to think this has to do with running the tests
> > in parallel.
> > 
> > As a stop-gap measure, I’ve committed a change to run tests
> > sequentially.  With this “guix build grub --rounds=5” passes for on my
> > x86_64 laptop.  Let’s reopen the bug if we find that’s not enough.
> 
> This test just failed for me on x86_64, even with #:parallel-tests? #f.
> 
> So, perhaps it's not an inter-test race, or perhaps the tests are still
> running in parallel, or some other thing.

Spurred by another report of this test failing, here's a patch to
disable it completely.

[0001-gnu-grub-Disable-a-flaky-test.patch (text/plain, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #27 received at 26936 <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Leo Famulari <leo <at> famulari.name>, 26936 <at> debbugs.gnu.org
Subject: Re: bug#26936: grub_cmd_set_date test sometimes fail
Date: Sun, 25 Jun 2017 12:55:51 +0200

[Message part 1 (text/plain, inline)]

Leo Famulari <leo <at> famulari.name> writes:

> On Fri, Jun 23, 2017 at 03:30:10PM -0400, Leo Famulari wrote:
>> On Sat, Jun 03, 2017 at 12:54:46AM +0200, Ludovic Courtès wrote:
>> > FAIL: grub_cmd_set_date
>> 
>> > All in all, I’m tempted to think this has to do with running the tests
>> > in parallel.
>> > 
>> > As a stop-gap measure, I’ve committed a change to run tests
>> > sequentially.  With this “guix build grub --rounds=5” passes for on my
>> > x86_64 laptop.  Let’s reopen the bug if we find that’s not enough.
>> 
>> This test just failed for me on x86_64, even with #:parallel-tests? #f.
>> 
>> So, perhaps it's not an inter-test race, or perhaps the tests are still
>> running in parallel, or some other thing.
>
> Spurred by another report of this test failing, here's a patch to
> disable it completely.
> From 71299a035f793c3e2ac723b326cd881e97df6274 Mon Sep 17 00:00:00 2001
> From: Leo Famulari <leo <at> famulari.name>
> Date: Sat, 24 Jun 2017 13:17:37 -0400
> Subject: [PATCH] gnu: grub: Disable a flaky test.
>
> * gnu/packages/bootloaders.scm (grub)[arguments]: Add 'disable-flaky-test' phase
> and run the tests in parallel.

LGTM.

[signature.asc (application/pgp-signature, inline)]

Message #30 received at 26936 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Marius Bakke <mbakke <at> fastmail.com>
Cc: 26936 <at> debbugs.gnu.org
Subject: Re: bug#26936: grub_cmd_set_date test sometimes fail
Date: Sun, 25 Jun 2017 09:48:21 -0400

[Message part 1 (text/plain, inline)]

On Sun, Jun 25, 2017 at 12:55:51PM +0200, Marius Bakke wrote:
> Leo Famulari <leo <at> famulari.name> writes:
> > Subject: [PATCH] gnu: grub: Disable a flaky test.
> >
> > * gnu/packages/bootloaders.scm (grub)[arguments]: Add 'disable-flaky-test' phase
> > and run the tests in parallel.
> 
> LGTM.

Thanks for the review! Pushed as 23a53a95f1f5f44df401c120276eebf196ba15d2.

Also, I reported it upstream:

https://lists.gnu.org/archive/html/bug-grub/2017-06/msg00012.html

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.