GNU bug report logs - #41116
Guix deploy fails with new version of Herd

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 41116 in the body.
You can then email your comments to 41116 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Alex Sassmannshausen <alex <at> komputilo.eu>
To: <bug-guix <at> gnu.org>
Subject: Guix deploy fails with new version of Herd
Date: Thu, 07 May 2020 00:21:36 +0200

Hello,

I maintain a number of servers using Guix deploy.  It seems that the
recent upgrade to Herd in Guix, and specifically commit
4c0cc7bed3de2c0e2d3a6e95b88693941e839eec might have introduced a bug.

From my testing, guix deploy currently consistently fails with:
-----------------8<----------------------------->8-------------------
ice-9/boot-9.scm:1667:16: In procedure raise-exception:
ERROR:
  1. &inferior-exception:
      arguments: (srfi-34 #<inferior-object #<condition &action-exception-error [service: root action: eval key: keyword-argument-error args: ("#<procedure 7fe24816e240 at shepherd/service.scm:903:4 (command #:key user group directory environment-variables pid-file pid-file-timeout log-file) | (program . program-args)>" "Unrecognized keyword" () (#:file-creation-mask))] 7eff2bd7be00>>)
      inferior: #f
      stack: ()
-----------------8<----------------------------->8-------------------

A workaround is to build the system configuration locally on the target
server, then to reconfigure.  It will still error at the same place, but
at this point, after restarting the server, the new version of Herd will
be running and both deploy and reconfigure will work.

I don't know what a good solution to this could be, but it may be
something we need to consider in future development of Herd.

Best wishes,

Alex

Message #8 received at 41116 <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Alex Sassmannshausen <alex <at> komputilo.eu>, 41116 <at> debbugs.gnu.org
Subject: Re: bug#41116: Guix deploy fails with new version of Herd
Date: Thu, 07 May 2020 00:45:33 +0200

[Message part 1 (text/plain, inline)]

Hi Alex,

Alex Sassmannshausen via Bug reports for GNU Guix <bug-guix <at> gnu.org>
writes:

> Hello,
>
> I maintain a number of servers using Guix deploy.  It seems that the
> recent upgrade to Herd in Guix, and specifically commit
> 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec might have introduced a bug.
>
> From my testing, guix deploy currently consistently fails with:
> -----------------8<----------------------------->8-------------------
> ice-9/boot-9.scm:1667:16: In procedure raise-exception:
> ERROR:
>   1. &inferior-exception:
>       arguments: (srfi-34 #<inferior-object #<condition &action-exception-error [service: root action: eval key: keyword-argument-error args: ("#<procedure 7fe24816e240 at shepherd/service.scm:903:4 (command #:key user group directory environment-variables pid-file pid-file-timeout log-file) | (program . program-args)>" "Unrecognized keyword" () (#:file-creation-mask))] 7eff2bd7be00>>)
>       inferior: #f
>       stack: ()
> -----------------8<----------------------------->8-------------------
>
> A workaround is to build the system configuration locally on the target
> server, then to reconfigure.  It will still error at the same place, but
> at this point, after restarting the server, the new version of Herd will
> be running and both deploy and reconfigure will work.
>
> I don't know what a good solution to this could be, but it may be
> something we need to consider in future development of Herd.

This issue has been reported by a number of users on IRC.  I think the
problem is that the the #:file-creation-mask keyword requires support
from the running Shepherd, which may not have it yet.  I think we should
revert commit 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec until we find a
smooth upgrade path.  Can you try it and push if that fixes guix deploy?

[signature.asc (application/pgp-signature, inline)]

Message #11 received at 41116 <at> debbugs.gnu.org (full text, mbox):

From: Alex Sassmannshausen <alex <at> komputilo.eu>
To: 41116 <at> debbugs.gnu.org
Subject: A naive proposal for a solution
Date: Thu, 07 May 2020 00:38:44 +0200

Upon thinking further about this it seems to me the problem is caused by
guix deploy attempting to restart services as well as it can during
deployment.  When this fails deployment fails.

guix system reconfigure on the other hand does not do this (afaik).  As
a result it can complete.

Once reconfigure is completed a reboot switches to the new system
version and is then thus able to restart the services.

If all this is correct, then the long-discussed guix deploy feature of
service restart policies would resolve this issue elegantly:
When a similar herd upgrade in future looms, a switch away from "restart
running services" to "no restart services" or "reboot after deployment"
would avoid this currently hard-coded failure mode.

Food for thought perhaps, if my understanding is anywhere close to
right, that is.

Alex

Message #18 received at 41116-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Marius Bakke <mbakke <at> fastmail.com>
Cc: 41116-done <at> debbugs.gnu.org, Alex Sassmannshausen <alex <at> komputilo.eu>,
 Diego Nicola Barbato <dnbarbato <at> posteo.de>
Subject: Re: bug#41116: Guix deploy fails with new version of Herd
Date: Thu, 07 May 2020 14:27:46 +0200

Hello Alex & Marius,

Marius Bakke <mbakke <at> fastmail.com> skribis:

> Alex Sassmannshausen via Bug reports for GNU Guix <bug-guix <at> gnu.org>
> writes:
>
>> Hello,
>>
>> I maintain a number of servers using Guix deploy.  It seems that the
>> recent upgrade to Herd in Guix, and specifically commit
>> 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec might have introduced a bug.
>>
>> From my testing, guix deploy currently consistently fails with:
>> -----------------8<----------------------------->8-------------------
>> ice-9/boot-9.scm:1667:16: In procedure raise-exception:
>> ERROR:
>>   1. &inferior-exception:
>>       arguments: (srfi-34 #<inferior-object #<condition &action-exception-error [service: root action: eval key: keyword-argument-error args: ("#<procedure 7fe24816e240 at shepherd/service.scm:903:4 (command #:key user group directory environment-variables pid-file pid-file-timeout log-file) | (program . program-args)>" "Unrecognized keyword" () (#:file-creation-mask))] 7eff2bd7be00>>)
>>       inferior: #f
>>       stack: ()
>> -----------------8<----------------------------->8-------------------
>>
>> A workaround is to build the system configuration locally on the target
>> server, then to reconfigure.  It will still error at the same place, but
>> at this point, after restarting the server, the new version of Herd will
>> be running and both deploy and reconfigure will work.
>>
>> I don't know what a good solution to this could be, but it may be
>> something we need to consider in future development of Herd.
>
> This issue has been reported by a number of users on IRC.  I think the
> problem is that the the #:file-creation-mask keyword requires support
> from the running Shepherd, which may not have it yet.  I think we should
> revert commit 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec until we find a
> smooth upgrade path.  Can you try it and push if that fixes guix deploy?

I’ve reverted the patch in 5aa4d2dcf2f4f8786358feb45338893ed08a4cd9.

Diego: I guess we can reinstate the patch “later”, once Shepherd 0.8 can
be considered widespread.

More importantly, we should handle service reload failures more
gracefully, as proposed in <https://issues.guix.gnu.org/issue/30706>,
for both ‘reconfigure’ and ‘deploy’.

Thanks,
Ludo’.

Message #21 received at 41116-done <at> debbugs.gnu.org (full text, mbox):

From: Diego Nicola Barbato <dnbarbato <at> posteo.de>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 41116-done <at> debbugs.gnu.org, Marius Bakke <mbakke <at> fastmail.com>,
 Alex Sassmannshausen <alex <at> komputilo.eu>
Subject: Re: bug#41116: Guix deploy fails with new version of Herd
Date: Thu, 07 May 2020 15:29:29 +0200

Hey,

Ludovic Courtès <ludo <at> gnu.org> writes:

> Hello Alex & Marius,
>
> Marius Bakke <mbakke <at> fastmail.com> skribis:
>
>> Alex Sassmannshausen via Bug reports for GNU Guix <bug-guix <at> gnu.org>
>> writes:
>>
>>> Hello,
>>>
>>> I maintain a number of servers using Guix deploy.  It seems that the
>>> recent upgrade to Herd in Guix, and specifically commit
>>> 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec might have introduced a bug.
>>>
>>> From my testing, guix deploy currently consistently fails with:
>>> -----------------8<----------------------------->8-------------------
>>> ice-9/boot-9.scm:1667:16: In procedure raise-exception:
>>> ERROR:
>>>   1. &inferior-exception:
>>>       arguments: (srfi-34 #<inferior-object #<condition &action-exception-error [service: root action: eval key: keyword-argument-error args: ("#<procedure 7fe24816e240 at shepherd/service.scm:903:4 (command #:key user group directory environment-variables pid-file pid-file-timeout log-file) | (program . program-args)>" "Unrecognized keyword" () (#:file-creation-mask))] 7eff2bd7be00>>)
>>>       inferior: #f
>>>       stack: ()
>>> -----------------8<----------------------------->8-------------------
>>>
>>> A workaround is to build the system configuration locally on the target
>>> server, then to reconfigure.  It will still error at the same place, but
>>> at this point, after restarting the server, the new version of Herd will
>>> be running and both deploy and reconfigure will work.
>>>
>>> I don't know what a good solution to this could be, but it may be
>>> something we need to consider in future development of Herd.
>>
>> This issue has been reported by a number of users on IRC.  I think the
>> problem is that the the #:file-creation-mask keyword requires support
>> from the running Shepherd, which may not have it yet.  I think we should
>> revert commit 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec until we find a
>> smooth upgrade path.  Can you try it and push if that fixes guix deploy?
>
> I’ve reverted the patch in 5aa4d2dcf2f4f8786358feb45338893ed08a4cd9.
>
> Diego: I guess we can reinstate the patch “later”, once Shepherd 0.8 can
> be considered widespread.

I'm sorry I broke reconfigure and deploy.  I didn't consider testing
upgrading from before Shepherd 0.8 to after my change and I didn't even
think of deploy.  Going forth I'll leave messing with core functionality
to the pros.

> More importantly, we should handle service reload failures more
> gracefully, as proposed in <https://issues.guix.gnu.org/issue/30706>,
> for both ‘reconfigure’ and ‘deploy’.

Regards,

Diego

Message #24 received at 41116 <at> debbugs.gnu.org (full text, mbox):

From: Alex Sassmannshausen <alex <at> komputilo.eu>
To: Marius Bakke <mbakke <at> fastmail.com>
Cc: 41116 <at> debbugs.gnu.org
Subject: Re: bug#41116: Guix deploy fails with new version of Herd
Date: Thu, 07 May 2020 13:37:37 +0200

Hi Marius,

Marius Bakke <mbakke <at> fastmail.com> writes:

> Hi Alex,
>
> [...]
>
> This issue has been reported by a number of users on IRC.  I think the
> problem is that the the #:file-creation-mask keyword requires support
> from the running Shepherd, which may not have it yet.  I think we should
> revert commit 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec until we find a
> smooth upgrade path.  Can you try it and push if that fixes guix deploy?

I believe Ludovic has now done this.  I will test and close this bug
if it is now working.

Cheers,

Alex

Message #27 received at 41116-done <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Diego Nicola Barbato <dnbarbato <at> posteo.de>, Ludovic Courtès <ludo <at> gnu.org>
Cc: 41116-done <at> debbugs.gnu.org, Alex Sassmannshausen <alex <at> komputilo.eu>
Subject: Re: bug#41116: Guix deploy fails with new version of Herd
Date: Fri, 08 May 2020 15:44:18 +0200

[Message part 1 (text/plain, inline)]

Diego Nicola Barbato <dnbarbato <at> posteo.de> writes:

> Hey,
>
> Ludovic Courtès <ludo <at> gnu.org> writes:
>
>> Hello Alex & Marius,
>>
>> Marius Bakke <mbakke <at> fastmail.com> skribis:
>>
>>> Alex Sassmannshausen via Bug reports for GNU Guix <bug-guix <at> gnu.org>
>>> writes:
>>>
>>>> Hello,
>>>>
>>>> I maintain a number of servers using Guix deploy.  It seems that the
>>>> recent upgrade to Herd in Guix, and specifically commit
>>>> 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec might have introduced a bug.
>>>>
>>>> From my testing, guix deploy currently consistently fails with:
>>>> -----------------8<----------------------------->8-------------------
>>>> ice-9/boot-9.scm:1667:16: In procedure raise-exception:
>>>> ERROR:
>>>>   1. &inferior-exception:
>>>>       arguments: (srfi-34 #<inferior-object #<condition &action-exception-error [service: root action: eval key: keyword-argument-error args: ("#<procedure 7fe24816e240 at shepherd/service.scm:903:4 (command #:key user group directory environment-variables pid-file pid-file-timeout log-file) | (program . program-args)>" "Unrecognized keyword" () (#:file-creation-mask))] 7eff2bd7be00>>)
>>>>       inferior: #f
>>>>       stack: ()
>>>> -----------------8<----------------------------->8-------------------
>>>>
>>>> A workaround is to build the system configuration locally on the target
>>>> server, then to reconfigure.  It will still error at the same place, but
>>>> at this point, after restarting the server, the new version of Herd will
>>>> be running and both deploy and reconfigure will work.
>>>>
>>>> I don't know what a good solution to this could be, but it may be
>>>> something we need to consider in future development of Herd.
>>>
>>> This issue has been reported by a number of users on IRC.  I think the
>>> problem is that the the #:file-creation-mask keyword requires support
>>> from the running Shepherd, which may not have it yet.  I think we should
>>> revert commit 4c0cc7bed3de2c0e2d3a6e95b88693941e839eec until we find a
>>> smooth upgrade path.  Can you try it and push if that fixes guix deploy?
>>
>> I’ve reverted the patch in 5aa4d2dcf2f4f8786358feb45338893ed08a4cd9.
>>
>> Diego: I guess we can reinstate the patch “later”, once Shepherd 0.8 can
>> be considered widespread.
>
> I'm sorry I broke reconfigure and deploy.  I didn't consider testing
> upgrading from before Shepherd 0.8 to after my change and I didn't even
> think of deploy.  Going forth I'll leave messing with core functionality
> to the pros.

Mistakes happen, don't worry about it.

One thing that would be really useful and can prevent such situations in
the future is to have a "system test" that tries to run reconfigure from
the latest released version of Guix (currently 1.1.0).

There are already a few Shepherd tests in gnu/tests/base.scm and
gnu/tests/reconfigure.scm that can be used as inspiration.

Food for thought, patches welcome, etc.  :-)

[signature.asc (application/pgp-signature, inline)]

Message #30 received at 41116-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Marius Bakke <mbakke <at> fastmail.com>
Cc: 41116-done <at> debbugs.gnu.org, Alex Sassmannshausen <alex <at> komputilo.eu>,
 Diego Nicola Barbato <dnbarbato <at> posteo.de>
Subject: Re: bug#41116: Guix deploy fails with new version of Herd
Date: Sun, 10 May 2020 01:23:15 +0200

Hi,

Marius Bakke <mbakke <at> fastmail.com> skribis:

> Diego Nicola Barbato <dnbarbato <at> posteo.de> writes:

[...]

>>> I’ve reverted the patch in 5aa4d2dcf2f4f8786358feb45338893ed08a4cd9.
>>>
>>> Diego: I guess we can reinstate the patch “later”, once Shepherd 0.8 can
>>> be considered widespread.
>>
>> I'm sorry I broke reconfigure and deploy.  I didn't consider testing
>> upgrading from before Shepherd 0.8 to after my change and I didn't even
>> think of deploy.  Going forth I'll leave messing with core functionality
>> to the pros.
>
> Mistakes happen, don't worry about it.

Yup!  Plus, the person who reviewed the patch, undoubtedly an equally
nice person, didn’t notice the issue either.  :-)

> One thing that would be really useful and can prevent such situations in
> the future is to have a "system test" that tries to run reconfigure from
> the latest released version of Guix (currently 1.1.0).
>
> There are already a few Shepherd tests in gnu/tests/base.scm and
> gnu/tests/reconfigure.scm that can be used as inspiration.

Yes.

I was also wondering if it would make sense for services to somehow
state the major+minor version they’re targeting.

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.