GNU bug report logs - #30299
shepherd fails tests on all systems except x86_64

Previous Next

Package: guix;

Reported by: Mark H Weaver <mhw <at> netris.org>

Date: Wed, 31 Jan 2018 03:09:02 UTC

Severity: serious

Done: Marius Bakke <mbakke <at> fastmail.com>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 30299 in the body.
You can then email your comments to 30299 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-guix <at> gnu.org:
bug#30299; Package guix. (Wed, 31 Jan 2018 03:09:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Mark H Weaver <mhw <at> netris.org>:
New bug report received and forwarded. Copy sent to bug-guix <at> gnu.org. (Wed, 31 Jan 2018 03:09:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: bug-guix <at> gnu.org
Subject: [core-updates] shepherd fails tests on all systems except x86_64
Date: Tue, 30 Jan 2018 22:07:35 -0500
On core-updates, Hydra has been unable to successfully build 'shepherd'
on any system except x86_64-linux.  I can also report that on my
mips64el-linux GuixSD system, which is running something close to
'core-updates', I had to disable tests on shepherd in order to build it.
I don't know about aarch64-linux.

These are the tests that commonly fail:

  FAIL: tests/respawn.sh
  FAIL: tests/respawn-throttling.sh
  FAIL: tests/basic.sh

      Mark




Information forwarded to bug-guix <at> gnu.org:
bug#30299; Package guix. (Sat, 03 Feb 2018 19:54:02 GMT) Full text and rfc822 format available.

Message #8 received at 30299 <at> debbugs.gnu.org (full text, mbox):

From: Efraim Flashner <efraim <at> flashner.co.il>
To: Mark H Weaver <mhw <at> netris.org>
Cc: 30299 <at> debbugs.gnu.org
Subject: Re: bug#30299: [core-updates] shepherd fails tests on all systems
 except x86_64
Date: Sat, 3 Feb 2018 21:53:35 +0200
[Message part 1 (text/plain, inline)]
On Tue, Jan 30, 2018 at 10:07:35PM -0500, Mark H Weaver wrote:
> On core-updates, Hydra has been unable to successfully build 'shepherd'
> on any system except x86_64-linux.  I can also report that on my
> mips64el-linux GuixSD system, which is running something close to
> 'core-updates', I had to disable tests on shepherd in order to build it.
> I don't know about aarch64-linux.
> 
> These are the tests that commonly fail:
> 
>   FAIL: tests/respawn.sh
>   FAIL: tests/respawn-throttling.sh
>   FAIL: tests/basic.sh
> 
>       Mark

Shepherd built successfully on aarch64-linux, and again with '--check'.

-- 
Efraim Flashner   <efraim <at> flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[signature.asc (application/pgp-signature, inline)]

Information forwarded to bug-guix <at> gnu.org:
bug#30299; Package guix. (Sun, 04 Feb 2018 22:49:01 GMT) Full text and rfc822 format available.

Message #11 received at 30299 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Mark H Weaver <mhw <at> netris.org>
Cc: 30299 <at> debbugs.gnu.org
Subject: Re: bug#30299: [core-updates] shepherd fails tests on all systems
 except x86_64
Date: Sun, 04 Feb 2018 23:48:29 +0100
Hello,

Mark H Weaver <mhw <at> netris.org> skribis:

> On core-updates, Hydra has been unable to successfully build 'shepherd'
> on any system except x86_64-linux.  I can also report that on my
> mips64el-linux GuixSD system, which is running something close to
> 'core-updates', I had to disable tests on shepherd in order to build it.
> I don't know about aarch64-linux.
>
> These are the tests that commonly fail:
>
>   FAIL: tests/respawn.sh
>   FAIL: tests/respawn-throttling.sh
>   FAIL: tests/basic.sh

This is a non-deterministic failure.  I could reproduce the
tests/basic.sh one and it is fixed by this:

  https://git.savannah.gnu.org/cgit/shepherd.git/commit/?id=cc9564586729a5bb90dd5d2722b543fdde9ab821

I’ll roll a new Shepherd release soon and update the package.

The rest may be a duplicate of 23811, though I couldn’t reproduce it.

Thanks,
Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#30299; Package guix. (Wed, 14 Feb 2018 08:54:01 GMT) Full text and rfc822 format available.

Message #14 received at 30299 <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30299 <at> debbugs.gnu.org
Subject: Re: bug#30299: [core-updates] shepherd fails tests on all systems
 except x86_64
Date: Wed, 14 Feb 2018 03:52:59 -0500
ludo <at> gnu.org (Ludovic Courtès) writes:

> Mark H Weaver <mhw <at> netris.org> skribis:
>
>> On core-updates, Hydra has been unable to successfully build 'shepherd'
>> on any system except x86_64-linux.  I can also report that on my
>> mips64el-linux GuixSD system, which is running something close to
>> 'core-updates', I had to disable tests on shepherd in order to build it.
>> I don't know about aarch64-linux.
>>
>> These are the tests that commonly fail:
>>
>>   FAIL: tests/respawn.sh
>>   FAIL: tests/respawn-throttling.sh
>>   FAIL: tests/basic.sh
>
> This is a non-deterministic failure.  I could reproduce the
> tests/basic.sh one and it is fixed by this:
>
>   https://git.savannah.gnu.org/cgit/shepherd.git/commit/?id=cc9564586729a5bb90dd5d2722b543fdde9ab821

Thank you!

> I’ll roll a new Shepherd release soon and update the package.

For now, I added your patch to the shepherd package in core-updates, in
commit f2d2ee42f168909f27c0c3b6532ef16febfd3b86.

      Mark




Information forwarded to bug-guix <at> gnu.org:
bug#30299; Package guix. (Wed, 14 Feb 2018 09:18:01 GMT) Full text and rfc822 format available.

Message #17 received at 30299 <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: 30299 <at> debbugs.gnu.org
Subject: Re: bug#30299: [core-updates] shepherd fails tests on all systems
 except x86_64
Date: Wed, 14 Feb 2018 04:16:52 -0500
Mark H Weaver <mhw <at> netris.org> writes:

> On core-updates, Hydra has been unable to successfully build 'shepherd'
> on any system except x86_64-linux.  I can also report that on my
> mips64el-linux GuixSD system, which is running something close to
> 'core-updates', I had to disable tests on shepherd in order to build it.
> I don't know about aarch64-linux.
>
> These are the tests that commonly fail:
>
>   FAIL: tests/respawn.sh
>   FAIL: tests/respawn-throttling.sh
>   FAIL: tests/basic.sh

FYI, after 18 failed attempts, Hydra finally built shepherd on i686
successfully on the 19th try:

  https://hydra.gnu.org/build/2425454#tabs-buildsteps

Now that I've added Ludovic's patch, hopefully it will require fewer
attempts this time :)

       Mark




Information forwarded to bug-guix <at> gnu.org:
bug#30299; Package guix. (Wed, 14 Feb 2018 13:01:02 GMT) Full text and rfc822 format available.

Message #20 received at 30299 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Mark H Weaver <mhw <at> netris.org>
Cc: 30299 <at> debbugs.gnu.org
Subject: Re: bug#30299: [core-updates] shepherd fails tests on all systems
 except x86_64
Date: Wed, 14 Feb 2018 14:00:18 +0100
Mark H Weaver <mhw <at> netris.org> skribis:

> ludo <at> gnu.org (Ludovic Courtès) writes:
>
>> Mark H Weaver <mhw <at> netris.org> skribis:
>>
>>> On core-updates, Hydra has been unable to successfully build 'shepherd'
>>> on any system except x86_64-linux.  I can also report that on my
>>> mips64el-linux GuixSD system, which is running something close to
>>> 'core-updates', I had to disable tests on shepherd in order to build it.
>>> I don't know about aarch64-linux.
>>>
>>> These are the tests that commonly fail:
>>>
>>>   FAIL: tests/respawn.sh
>>>   FAIL: tests/respawn-throttling.sh
>>>   FAIL: tests/basic.sh
>>
>> This is a non-deterministic failure.  I could reproduce the
>> tests/basic.sh one and it is fixed by this:
>>
>>   https://git.savannah.gnu.org/cgit/shepherd.git/commit/?id=cc9564586729a5bb90dd5d2722b543fdde9ab821
>
> Thank you!
>
>> I’ll roll a new Shepherd release soon and update the package.
>
> For now, I added your patch to the shepherd package in core-updates, in
> commit f2d2ee42f168909f27c0c3b6532ef16febfd3b86.

Thanks, and sorry for not acting earlier!  (I was waiting for a reply
from the Translation Project to make the new release, but that hasn’t
happened yet.)

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#30299; Package guix. (Thu, 15 Feb 2018 19:23:02 GMT) Full text and rfc822 format available.

Message #23 received at 30299 <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30299 <at> debbugs.gnu.org
Subject: Re: bug#30299: [core-updates] shepherd fails tests on all systems
 except x86_64
Date: Thu, 15 Feb 2018 14:21:25 -0500
Hi Ludovic,

Mark H Weaver <mhw <at> netris.org> writes:

> ludo <at> gnu.org (Ludovic Courtès) writes:
>
>> Mark H Weaver <mhw <at> netris.org> skribis:
>>
>>> On core-updates, Hydra has been unable to successfully build 'shepherd'
>>> on any system except x86_64-linux.  I can also report that on my
>>> mips64el-linux GuixSD system, which is running something close to
>>> 'core-updates', I had to disable tests on shepherd in order to build it.
>>> I don't know about aarch64-linux.
>>>
>>> These are the tests that commonly fail:
>>>
>>>   FAIL: tests/respawn.sh
>>>   FAIL: tests/respawn-throttling.sh
>>>   FAIL: tests/basic.sh
>>
>> This is a non-deterministic failure.  I could reproduce the
>> tests/basic.sh one and it is fixed by this:
>>
>>   https://git.savannah.gnu.org/cgit/shepherd.git/commit/?id=cc9564586729a5bb90dd5d2722b543fdde9ab821
>
> For now, I added your patch to the shepherd package in core-updates, in
> commit f2d2ee42f168909f27c0c3b6532ef16febfd3b86.

With your patch, Hydra built shepherd successfully on i686-linux on the
first try, which is much better than the 19th try :)

However, on armhf-linux, three tests failed: respawn.sh,
respawn-throttling.sh, and pid-file.sh.

  https://hydra.gnu.org/build/2499835

We should probably arrange for test-suite.log to be printed when "make
check" fails.  We could add this functionality to selected packages like
shepherd and guix the next time we update them, and maybe consider
adding something generic to gnu-build-system's check phase in the next
core-updates cycle.  What do you think?

      Mark

--8<---------------cut here---------------start------------->8---
starting phase `check'
make  check-am
make[1]: Entering directory '/tmp/guix-build-shepherd-0.3.2.drv-0/shepherd-0.3.2'
make  check-TESTS
make[2]: Entering directory '/tmp/guix-build-shepherd-0.3.2.drv-0/shepherd-0.3.2'
make[3]: Entering directory '/tmp/guix-build-shepherd-0.3.2.drv-0/shepherd-0.3.2'
PASS: tests/misbehaved-client.sh
FAIL: tests/respawn.sh
PASS: tests/no-home.sh
FAIL: tests/pid-file.sh
PASS: tests/status-sexp.sh
PASS: tests/sigint.sh
PASS: tests/basic.sh
FAIL: tests/respawn-throttling.sh
============================================================================
Testsuite summary for GNU Shepherd 0.3.2
============================================================================
# TOTAL: 8
# PASS:  5
# SKIP:  0
# XFAIL: 0
# FAIL:  3
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
Please report to bug-guix <at> gnu.org
============================================================================
make[3]: *** [Makefile:1220: test-suite.log] Error 1
make[3]: Leaving directory '/tmp/guix-build-shepherd-0.3.2.drv-0/shepherd-0.3.2'
make[2]: *** [Makefile:1328: check-TESTS] Error 2
make[2]: Leaving directory '/tmp/guix-build-shepherd-0.3.2.drv-0/shepherd-0.3.2'
make[1]: *** [Makefile:1527: check-am] Error 2
make[1]: Leaving directory '/tmp/guix-build-shepherd-0.3.2.drv-0/shepherd-0.3.2'
make: *** [Makefile:1529: check] Error 2
phase `check' failed after 29.6 seconds
builder for `/gnu/store/sk0l3ll4x1ddn9zwxmfqjipr578hiqx1-shepherd-0.3.2.drv' failed with exit code 1
@ build-failed /gnu/store/sk0l3ll4x1ddn9zwxmfqjipr578hiqx1-shepherd-0.3.2.drv - 1 builder for `/gnu/store/sk0l3ll4x1ddn9zwxmfqjipr578hiqx1-shepherd-0.3.2.drv' failed with exit code 1
--8<---------------cut here---------------end--------------->8---




Information forwarded to bug-guix <at> gnu.org:
bug#30299; Package guix. (Thu, 15 Feb 2018 21:27:01 GMT) Full text and rfc822 format available.

Message #26 received at 30299 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Mark H Weaver <mhw <at> netris.org>
Cc: 30299 <at> debbugs.gnu.org
Subject: Re: bug#30299: [core-updates] shepherd fails tests on all systems
 except x86_64
Date: Thu, 15 Feb 2018 22:26:02 +0100
Hi Mark,

Mark H Weaver <mhw <at> netris.org> skribis:

> Mark H Weaver <mhw <at> netris.org> writes:

[...]

>>> This is a non-deterministic failure.  I could reproduce the
>>> tests/basic.sh one and it is fixed by this:
>>>
>>>   https://git.savannah.gnu.org/cgit/shepherd.git/commit/?id=cc9564586729a5bb90dd5d2722b543fdde9ab821
>>
>> For now, I added your patch to the shepherd package in core-updates, in
>> commit f2d2ee42f168909f27c0c3b6532ef16febfd3b86.
>
> With your patch, Hydra built shepherd successfully on i686-linux on the
> first try, which is much better than the 19th try :)

We’re making progress.  :-)

> However, on armhf-linux, three tests failed: respawn.sh,
> respawn-throttling.sh, and pid-file.sh.
>
>   https://hydra.gnu.org/build/2499835

I’ll try to reproduce the failure here.

> We should probably arrange for test-suite.log to be printed when "make
> check" fails.  We could add this functionality to selected packages like
> shepherd and guix the next time we update them, and maybe consider
> adding something generic to gnu-build-system's check phase in the next
> core-updates cycle.  What do you think?

Definitely.  We discussed it before and I think it’s a good idea.  I
wanted to add it to ‘guix’ as a starting point but never got around to
doing it.

Thanks,
Ludo’.




Severity set to 'important' from 'normal' Request was from ludo <at> gnu.org (Ludovic Courtès) to control <at> debbugs.gnu.org. (Fri, 16 Feb 2018 10:29:01 GMT) Full text and rfc822 format available.

Information forwarded to bug-guix <at> gnu.org:
bug#30299; Package guix. (Sat, 17 Feb 2018 00:05:02 GMT) Full text and rfc822 format available.

Message #31 received at 30299 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Mark H Weaver <mhw <at> netris.org>
Cc: 30299 <at> debbugs.gnu.org
Subject: Re: bug#30299: [core-updates] shepherd fails tests on all systems
 except x86_64
Date: Sat, 17 Feb 2018 01:04:00 +0100
Hello,

Mark H Weaver <mhw <at> netris.org> skribis:

> However, on armhf-linux, three tests failed: respawn.sh,
> respawn-throttling.sh, and pid-file.sh.
>
>   https://hydra.gnu.org/build/2499835

(Similar issue on aarch64:
<https://berlin.guixsd.org/build/2419870/log/raw>.  Though of course it
passed on the 2nd and 3rd attempts…)

I was able to reproduce a tests/respawn.sh failure on hardware (ARMv7).
The issue is that a service is not respawned, and the log shows:

--8<---------------cut here---------------start------------->8---
+ assert_killed_service_is_respawned t-service2-pid-695
++ cat t-service2-pid-695
+ old_pid=789
+ rm t-service2-pid-695
+ kill 789
+ wait_for_file t-service2-pid-695
+ i=0
+ test -f t-service2-pid-695
+ test 0 -lt 20
+ sleep 0.3
++ expr 0 + 1

[...]

2018-02-16 11:13:31 Service root has been started.
2018-02-16 11:13:32 Service test1 has been started.
2018-02-16 11:13:34 Service test2 has been started.
2018-02-16 11:13:35 Respawning test1.
2018-02-16 11:13:35 Service test1 has been started.
2018-02-16 11:13:36 Respawning test2.
2018-02-16 11:13:37 Service test2 has been started.
2018-02-16 11:13:37 Respawning test1.
2018-02-16 11:13:37 Service test1 has been started.
2018-02-16 11:13:38 Respawning test2.
2018-02-16 11:13:43 Service test2 could not be started.
--8<---------------cut here---------------end--------------->8---

So SIGCHLD was correctly delivered, but somehow restarting that service
didn’t work (its PID file didn’t show up again; the 5 seconds between
“Respawning” and “could not be started” correspond to the delay in
‘read-pid-file’ in (shepherd service)).  

These test failures seem to be more frequent when the machine is loaded.

Ludo’.




Changed bug title to 'shepherd fails tests on all systems except x86_64' from '[core-updates] shepherd fails tests on all systems except x86_64' Request was from Andreas Enge <andreas <at> enge.fr> to control <at> debbugs.gnu.org. (Sat, 24 Feb 2018 22:58:02 GMT) Full text and rfc822 format available.

Severity set to 'serious' from 'important' Request was from Andreas Enge <andreas <at> enge.fr> to control <at> debbugs.gnu.org. (Sat, 24 Feb 2018 22:58:02 GMT) Full text and rfc822 format available.

Information forwarded to bug-guix <at> gnu.org:
bug#30299; Package guix. (Sat, 24 Feb 2018 23:01:02 GMT) Full text and rfc822 format available.

Message #38 received at 30299 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Enge <andreas <at> enge.fr>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 30299 <at> debbugs.gnu.org, Mark H Weaver <mhw <at> netris.org>
Subject: Re: bug#30299: [core-updates] shepherd fails tests on all systems
 except x86_64
Date: Sun, 25 Feb 2018 00:00:33 +0100
Hello,

I changed the severity to "serious", since this bug prevents installing
GuixSD on arm, or creating an installation image. Also, it is now present
on master instead of core-updates.

Andreas





Information forwarded to bug-guix <at> gnu.org:
bug#30299; Package guix. (Thu, 19 Mar 2020 08:29:02 GMT) Full text and rfc822 format available.

Message #41 received at 30299 <at> debbugs.gnu.org (full text, mbox):

From: Brice Waegeneire <brice <at> waegenei.re>
To: 30299 <at> debbugs.gnu.org
Subject: Re: bug#30299: [core-updates] shepherd fails tests on all systems
 except x86_64
Date: Thu, 19 Mar 2020 08:28:44 +0000
Hello,

Does this bug is still relevant?
It was reported on core-updates 2 years ago with shepherd 0.3.2.
I can't see any CI failures[0] related to a test failing since cuirass 
was setup, the only time the build of shepherd failed[1] it wasn't due 
to a test.
I wasn't able to reproduce the failing test with some building rounds on 
armhf and i686.

> I changed the severity to "serious", since this bug prevents installing
> GuixSD on arm, or creating an installation image. Also, it is now 
> present
> on master instead of core-updates.

As far as I know this isn't the case anymore, so at least the priority 
should be lowered.

[0]: http://ci.guix.gnu.org/search?query=shepherd
[1]: http://ci.guix.gnu.org/build/1312246/details

Brice.




Reply sent to Marius Bakke <mbakke <at> fastmail.com>:
You have taken responsibility. (Thu, 19 Mar 2020 09:17:02 GMT) Full text and rfc822 format available.

Notification sent to Mark H Weaver <mhw <at> netris.org>:
bug acknowledged by developer. (Thu, 19 Mar 2020 09:17:02 GMT) Full text and rfc822 format available.

Message #46 received at 30299-done <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Brice Waegeneire <brice <at> waegenei.re>, 30299-done <at> debbugs.gnu.org
Subject: Re: bug#30299: [core-updates] shepherd fails tests on all systems
 except x86_64
Date: Thu, 19 Mar 2020 10:16:06 +0100
[Message part 1 (text/plain, inline)]
Brice Waegeneire <brice <at> waegenei.re> writes:

> Hello,
>
> Does this bug is still relevant?
> It was reported on core-updates 2 years ago with shepherd 0.3.2.
> I can't see any CI failures[0] related to a test failing since cuirass 
> was setup, the only time the build of shepherd failed[1] it wasn't due 
> to a test.
> I wasn't able to reproduce the failing test with some building rounds on 
> armhf and i686.

Thank you for doing bug triage Brice.  I'm closing the issue as we
haven't had problems building the shepherd in a while.
[signature.asc (application/pgp-signature, inline)]

bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Thu, 16 Apr 2020 11:24:07 GMT) Full text and rfc822 format available.

This bug report was last modified 3 years and 347 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.