GNU bug report logs - #70051
guix system hangs on boot with LUKS /home partition

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 70051 in the body.
You can then email your comments to 70051 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Fulbert <fulbert <at> bluewin.ch>
To: bug-guix <at> gnu.org
Subject: guix system hangs on boot with LUKS /home partition
Date: Thu, 28 Mar 2024 12:24:57 +0100

Hello,

Up to guix 9b84b36, my system was properly booting with a LUKS2 partition
mounted on /home. Starting with guix d5f857a (22 mar 2024), the boot hangs on
the same system using the same configuration.scm file. The only way out I found
when it hangs is hardware shutdown. There are no avaible console nor ssh server
started to help troubleshoot and there is nothing written to /var/log/messages
when it hangs.

I have tried to transfer my /home data to a brand new LUKS1 partition, (as well
as removing pointers to the old LUKS2 partition in my config.scm, of course) and
the problem remains exactly the same, including those error messages (obtained with
a video capture of the screen at boot, after removing 'quiet' from the kernel
command line in grub) :

#+begin_src boot
shepherd[1]: Starting service device-mapping-luks-homes...
shepherd[1]: Service device-mapping-luks-homes failed to start.
shepherd[1]: Exception caught while while starting device-mapping-luks-homes: (unbound-variable #f "Unbound variable: "S" (bytevector?) #f)
#+end_src

Maybe it's worth mentionning that I have then tried one configuration of the
'mapped-device' with 'luks-device-mapping' and another one with
'luks-device-mapping-with-options #:keyfile "/…"'. I also tried one
configuration with the 'source' declared in plain "/dev/..." and another one
declared with the luks '(uuid "…")', but this didnt change anything to the
"symptoms".

So, although I have learned in the process that LUKS2 is not yet fully
supported in guix, this problem also prevents booting using a LUKS1 /home
partition in my case.

Transfering the /home data to a clear (unencrypted) partition is my current
workaround to this problem.

Below is the configuration that has worked for several weeks, if not months, using my LUKS2 /home :

  (mapped-devices
    (list
      (mapped-device
        (source (uuid "<the uuid>"))
        (target "luks-homes")
        (type luks-device-mapping))))

  (file-systems
    (append
      (list
        […]
        (file-system (mount-point "/home")
                     (device (file-system-label "luks-homes"))
                     (type "ext4")
                     (dependencies mapped-devices))
        […]

Best regards and thanks for guix !

Message #8 received at 70051 <at> debbugs.gnu.org (full text, mbox):

From: Fulbert <fulbert <at> bluewin.ch>
To: 70051 <at> debbugs.gnu.org
Subject: bug#70051
Date: Thu, 28 Mar 2024 12:49:02 +0100

… And I forgot to mention that, when the boot hangs, shepherd still responds to ctrl-alt-del by closing some services and then the system hangs with hardware button shutdown as last resort.

Message #11 received at 70051 <at> debbugs.gnu.org (full text, mbox):

From: Fulbert <fulbert <at> bluewin.ch>
To: 70051 <at> debbugs.gnu.org
Subject: bug#70051
Date: Thu, 28 Mar 2024 12:57:21 +0100

And I also forgot to mention *sigh* that I am not the only one affected
by this problem. See : https://lists.gnu.org/archive/html/help-guix/2024-03/msg00152.html

Message #14 received at 70051 <at> debbugs.gnu.org (full text, mbox):

From: Remco van 't Veer <remco <at> remworks.net>
To: Fulbert <fulbert <at> bluewin.ch>
Cc: Ludovic Courtès <ludo <at> gnu.org>, help-guix <at> gnu.org,
 Adrien 'neox' Bourmault <neox <at> gnu.org>, 70051 <at> debbugs.gnu.org
Subject: Re: system hangs at boot - LUKS /home/ problem(?)
Date: Sat, 30 Mar 2024 16:25:07 +0100

Hi,

Confirmed on a couple of my installs.  I too have an unencrypted root
and encrypted home filesystems.  The passphrase prompt never appears and
the system seems to be waiting for something or is halted.

I've git bisected it down to:

  commit 6f9d844d2ece7b369d17bbe678978462425f869c (HEAD)
  Author: Ludovic Courtès <ludo <at> gnu.org>
  Date:   Wed Mar 20 18:48:38 2024 +0100

    services: shepherd: Load each service file in a fresh module.

    Fixes <https://issues.guix.gnu.org/67649>.

    * gnu/home/services/shepherd.scm (home-shepherd-configuration-file)[config]:
    Define ‘make-user-module’.  Call ‘load’ in ‘save-module-excursion’.
    * gnu/services/shepherd.scm (shepherd-configuration-file): Likewise.

Commit 2b052fe3c0fa85e9faa8873a581568ad4c78e151 still works.

Cheers,
Remco

Message #17 received at 70051 <at> debbugs.gnu.org (full text, mbox):

From: Lilah Tascheter <lilah <at> lunabee.space>
To: 70051 <at> debbugs.gnu.org
Cc: Fulbert <fulbert <at> bluewin.ch>, Remco van 't Veer <remco <at> remworks.net>
Subject: same
Date: Sat, 30 Mar 2024 19:56:03 -0500

yep I got the same issue too. but, in my case, I have an encrypted root
with three other encrypted partitions, none of them my home. initrd
decryption succeeds, but shepherd device-mapper services fail as usual

Message #20 received at 70051 <at> debbugs.gnu.org (full text, mbox):

From: aurtzy <aurtzy <at> gmail.com>
To: 70051 <at> debbugs.gnu.org
Cc: aurtzy <aurtzy <at> gmail.com>
Subject: [PATCH] gnu: open-luks-device: Fix unbound variables.
Date: Tue,  2 Apr 2024 02:23:29 -0400

It seems like `use-modules' never actually worked due to the way it is eval'd
by the Shepherd, and was only apparent after a change that prevented other
module imports from leaking into the namespace.  This is fixed by using direct
references instead.

* gnu/system/mapped-devices.scm (open-luks-device): Use direct references for
variables from other modules.

Fixes: https://issues.guix.gnu.org/70051

Change-Id: I993798e161c4b4fca6e8a4f14eea5042b184ebc9
---

Hi!

I encountered this issue as well, and think I've figured out what was
happening: `use-modules' appears to not work due to the way g-expressions are
evaluated by Shepherd, so after services became properly isolated in their own
modules, the variable references became no longer available.  There's a
comment further down in the file that seems to confirm this ("XXX: We're not
at the top level here...").

Can anyone confirm this patch works for them too?

Cheers,

aurtzy

 gnu/system/mapped-devices.scm | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/gnu/system/mapped-devices.scm b/gnu/system/mapped-devices.scm
index c19a818453..e46e2c7954 100644
--- a/gnu/system/mapped-devices.scm
+++ b/gnu/system/mapped-devices.scm
@@ -201,14 +201,12 @@ (define* (open-luks-device source targets #:key key-file)
        #~(let ((source #$(if (uuid? source)
                              (uuid-bytevector source)
                              source))
-               (keyfile #$key-file))
-           ;; XXX: 'use-modules' should be at the top level.
-           (use-modules (rnrs bytevectors) ;bytevector?
-                        ((gnu build file-systems)
-                         #:select (find-partition-by-luks-uuid
-                                   system*/tty))
-                        ((guix build utils) #:select (mkdir-p)))
-
+               (keyfile #$key-file)
+               (bytevector? (@ (rnrs bytevectors) bytevector?))
+               (find-partition-by-luks-uuid (@ (gnu build file-systems)
+                                               find-partition-by-luks-uuid))
+               (system*/tty (@ (gnu build file-systems) system*/tty))
+               (mkdir-p (@ (guix build utils) mkdir-p)))
            ;; Create '/run/cryptsetup/' if it does not exist, as device locking
            ;; is mandatory for LUKS2.
            (mkdir-p "/run/cryptsetup/")

base-commit: 6e2db85ca83528199a46b002d2592bd4bef017c8
-- 
2.41.0

Message #23 received at 70051 <at> debbugs.gnu.org (full text, mbox):

From: Remco van 't Veer <remco <at> remworks.net>
To: aurtzy <aurtzy <at> gmail.com>, 70051 <at> debbugs.gnu.org
Subject: Re: [PATCH] gnu: open-luks-device: Fix unbound variables.
Date: Tue, 02 Apr 2024 14:14:39 +0200

2024/04/02, aurtzy:

> Can anyone confirm this patch works for them too?

Yes, it does.

Cheers,
Remco

Message #26 received at 70051 <at> debbugs.gnu.org (full text, mbox):

From: Remco van 't Veer <remco <at> remworks.net>
To: Benjamin Slade <beoram <at> gmail.com>
Cc: Ludovic Courtès <ludo <at> gnu.org>,
 Fulbert <fulbert <at> bluewin.ch>, help-guix <at> gnu.org,
 Adrien 'neox' Bourmault <neox <at> gnu.org>, 70051 <at> debbugs.gnu.org
Subject: Re: system hangs at boot - LUKS /home/ problem(?)
Date: Tue, 02 Apr 2024 22:19:09 +0200

2024/04/02, Benjamin Slade:

> I can't roll back to the earlier commit mentioned by Remco because
> other things/channels depend on me being roughly up-to-date on the
> main guix channel.

Reverting the commit on a local checkout of guix worked for me but isn't
workable of course.  I tested the patch provided by aurtzy
(https://issues.guix.gnu.org/70051#5) and that worked worked too.

For now I won't reconfigure my system until this issue is fixed or try
out "guix pull --switch-generation" to go back to some earlier situation
when I really need to deploy some configuration change.

Remco

Message #29 received at 70051 <at> debbugs.gnu.org (full text, mbox):

From: Benjamin Slade <beoram <at> gmail.com>
To: Remco van 't Veer <remco <at> remworks.net>, Fulbert <fulbert <at> bluewin.ch>,
 help-guix <at> gnu.org,  70051 <at> debbugs.gnu.org,  Adrien 'neox' Bourmault
 <neox <at> gnu.org>,  Ludovic Courtès <ludo <at> gnu.org>
Subject: Re: system hangs at boot - LUKS /home/ problem(?)
Date: Tue, 02 Apr 2024 15:00:02 -0500

[Message part 1 (text/plain, inline)]

I can't roll back to the earlier commit mentioned by Remco because other things/channels depend on me being roughly up-to-date on the main guix channel.

However, I can confirm the issue, as changing my configuration *not* to mount an encrypted /home resolves the boot issue.

I note two things:

a. when I try to configure with an encrypted /home, I get error/warning messages at the end: (earlier I also got a message about the "find-crypthome-by-uuid" process failing; I changed to specify a /dev/sXN device instead)

        guix system: warning: exception caught while executing 'start' on service 'device-mapping-crypthome':
      error: system*/tty: unbound variable
      guix system: warning: some services could not be upgraded
      hint: to allow changes to all the systems to take effect, you will need to reboot.

b. no `crypttab' is created (I don't remember how Guix handles encrypted /home's to know whether or not this is expected).


--B.

On Sat, 30 Mar 2024 16:25:07 +0100 (3 days, 4 hours, 30 minutes ago), Remco van 't Veer <remco <at> remworks.net> wrote:

> Hi,

> Confirmed on a couple of my installs.  I too have an unencrypted root
> and encrypted home filesystems.  The passphrase prompt never appears and
> the system seems to be waiting for something or is halted.

> I've git bisected it down to:

>   commit 6f9d844d2ece7b369d17bbe678978462425f869c (HEAD)
>   Author: Ludovic Courtès <ludo <at> gnu.org>
>   Date:   Wed Mar 20 18:48:38 2024 +0100

>     services: shepherd: Load each service file in a fresh module.

>     Fixes <https://issues.guix.gnu.org/67649>.

>     * gnu/home/services/shepherd.scm (home-shepherd-configuration-file)[config]:
>     Define ‘make-user-module’.  Call ‘load’ in ‘save-module-excursion’.
>     * gnu/services/shepherd.scm (shepherd-configuration-file): Likewise.

> Commit 2b052fe3c0fa85e9faa8873a581568ad4c78e151 still works.

> Cheers,
> Remco

Message #32 received at 70051 <at> debbugs.gnu.org (full text, mbox):

From: Adrien 'neox' Bourmault <neox <at> gnu.org>
To: 70051 <at> debbugs.gnu.org
Subject: guix system hangs on boot with LUKS /home partition
Date: Wed, 03 Apr 2024 20:01:00 +0200

I can confirm aurtzy's patch works (just tested on top of
7af70efd7633b0d70091762cf43ce01a86176e8e)

Message #37 received at 70051 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: aurtzy <aurtzy <at> gmail.com>
Cc: 70266 <at> debbugs.gnu.org, 70051 <at> debbugs.gnu.org
Subject: Re: bug#70266: Failure to open LUKS devices from a Shepherd service
Date: Mon, 08 Apr 2024 01:43:35 +0200

Hi aurtzy,

aurtzy <aurtzy <at> gmail.com> skribis:

> This bug has also been reported here: https://issues.guix.gnu.org/70051
>
> I sent a patch that a few others have confirmed fixes the issue:
> https://issues.guix.gnu.org/70051#5

Oops, sorry for not noticing it earlier!  (That was a hard-to-debug one
so kudos for the work you and others put in it.)

I pushed these two commits to address the problem:

  49f82fca41 mapped-devices: luks: Specify modules needed at the top-level.
  6062339156 mapped-devices: <mapped-device-type> can specify modules to import.

It works well in my tests but please let me know if something’s amiss.

Thanks,
Ludo’.

Message #40 received at 70051 <at> debbugs.gnu.org (full text, mbox):

From: aurtzy <aurtzy <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 70266 <at> debbugs.gnu.org, 70051 <at> debbugs.gnu.org
Subject: Re: bug#70266: Failure to open LUKS devices from a Shepherd service
Date: Sun, 7 Apr 2024 21:05:13 -0400

Hi Ludo',

On 4/7/24 19:43, Ludovic Courtès wrote:
> Oops, sorry for not noticing it earlier!  (That was a hard-to-debug one
> so kudos for the work you and others put in it.)
>
> I pushed these two commits to address the problem:
>
>    49f82fca41 mapped-devices: luks: Specify modules needed at the top-level.
>    6062339156 mapped-devices: <mapped-device-type> can specify modules to import.
>
> It works well in my tests but please let me know if something’s amiss.

Just pulled+reconfigured, and my machine boots just fine with the 
problem LUKS device being decrypted as expected again. Thanks!

Cheers,

aurtzy

Message #47 received at 70051-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: aurtzy <aurtzy <at> gmail.com>
Cc: 70266-done <at> debbugs.gnu.org, 70051-done <at> debbugs.gnu.org
Subject: Re: bug#70266: Failure to open LUKS devices from a Shepherd service
Date: Mon, 08 Apr 2024 14:19:04 +0200

Hi,

aurtzy <aurtzy <at> gmail.com> skribis:

> On 4/7/24 19:43, Ludovic Courtès wrote:
>> Oops, sorry for not noticing it earlier!  (That was a hard-to-debug one
>> so kudos for the work you and others put in it.)
>>
>> I pushed these two commits to address the problem:
>>
>>    49f82fca41 mapped-devices: luks: Specify modules needed at the top-level.
>>    6062339156 mapped-devices: <mapped-device-type> can specify modules to import.
>>
>> It works well in my tests but please let me know if something’s amiss.
>
> Just pulled+reconfigured, and my machine boots just fine with the
> problem LUKS device being decrypted as expected again. Thanks!

Awesome, thanks for confirming, and apologies for introducing this
regression in the first place!

Ludo’.

Message #55 received at 70051 <at> debbugs.gnu.org (full text, mbox):

From: Fulbert <fulbert <at> bluewin.ch>
To: 70051 <at> debbugs.gnu.org
Date: Mon, 8 Apr 2024 15:20:06 +0200

guix pull + reconfigure worked for me as well.

Thank you verry much.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.