GNU bug report logs - #30604
[PATCH 0/4] Load Linux module only when supported hardware is present.

Previous Next

Package: guix-patches;

Reported by: Danny Milosavljevic <dannym <at> scratchpost.org>

Date: Sun, 25 Feb 2018 11:47:02 UTC

Severity: important

Tags: patch

To reply to this bug, email your comments to 30604 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 25 Feb 2018 11:47:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Danny Milosavljevic <dannym <at> scratchpost.org>:
New bug report received and forwarded. Copy sent to guix-patches <at> gnu.org. (Sun, 25 Feb 2018 11:47:03 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: guix-patches <at> gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH 0/4] Load Linux module only when supported hardware is present.
Date: Sun, 25 Feb 2018 12:45:57 +0100
Danny Milosavljevic (4):
  gnu: kmod: Split off kmod-minimal.
  gnu: Add kmod-minimal-static.
  linux-initrd: Add kmod.
  linux-boot: Load kernel modules only when the hardware is present.

 gnu/build/linux-boot.scm    | 24 ++++++++++--
 gnu/build/linux-initrd.scm  | 13 ++++++-
 gnu/packages/linux.scm      | 91 +++++++++++++++++++++++++++++++++++++--------
 gnu/system/linux-initrd.scm | 52 ++++++++++++++++++++------
 4 files changed, 148 insertions(+), 32 deletions(-)





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 25 Feb 2018 11:49:02 GMT) Full text and rfc822 format available.

Message #8 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH 1/4] gnu: kmod: Split off kmod-minimal.
Date: Sun, 25 Feb 2018 12:48:13 +0100
* gnu/packages/linux.scm (kmod-minimal): New variable.
(kmod): Modify.
---
 gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++----------------
 1 file changed, 33 insertions(+), 16 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 238398e84..1f8bf3050 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1933,8 +1933,35 @@ for systems using the Linux kernel.  This includes commands such as
 to use Linux' inotify mechanism, which allows file accesses to be monitored.")
     (license license:gpl2+)))
 
-(define-public kmod
+(define kmod-minimal
   (package
+    (name "kmod-minimal")
+    (version "13")
+    (source (origin
+              (method url-fetch)
+              (uri
+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+                              "kmod-" version ".tar.xz"))
+              (sha256
+               (base32
+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+              (patches (search-patches "kmod-13-module-directory.patch"))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:tests? #f)) ; FIXME: Investigate test failures
+    (home-page "https://www.kernel.org/")
+    (synopsis "Kernel module tools")
+    (description "Kmod is a set of tools to handle common tasks with Linux
+kernel modules like insert, remove, list, check properties, resolve
+dependencies and aliases.
+
+These tools are designed on top of libkmod, a library that is shipped with
+kmod.  The aim is to be compatible with tools, configurations and indices
+from the module-init-tools project.")
+    (license license:gpl2+))) ; library under lgpl2.1+
+
+(define-public kmod
+  (package (inherit kmod-minimal)
     (name "kmod")
     (version "24")
     (source (origin
@@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
                (base32
                 "15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))
               (patches (search-patches "kmod-module-directory.patch"))))
-    (build-system gnu-build-system)
     (native-inputs
      `(("pkg-config" ,pkg-config)))
     (inputs
      `(("xz" ,xz)
        ("zlib" ,zlib)))
     (arguments
-     `(#:tests? #f ; FIXME: Investigate test failures
-       #:configure-flags '("--with-xz" "--with-zlib")
+     `(#:configure-flags '("--with-xz" "--with-zlib")
+       #:tests? #f ; FIXME: Investigate test failures
        #:phases (alist-cons-after
                  'install 'install-modprobe&co
                  (lambda* (#:key outputs #:allow-other-keys)
@@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
                                  (symlink "kmod"
                                           (string-append bin "/" tool)))
                                '("insmod" "rmmod" "lsmod" "modprobe"
-                                 "modinfo" "depmod"))))
-                 %standard-phases)))
-    (home-page "https://www.kernel.org/")
-    (synopsis "Kernel module tools")
-    (description "Kmod is a set of tools to handle common tasks with Linux
-kernel modules like insert, remove, list, check properties, resolve
-dependencies and aliases.
-
-These tools are designed on top of libkmod, a library that is shipped with
-kmod.  The aim is to be compatible with tools, configurations and indices
-from the module-init-tools project.")
-    (license license:gpl2+))) ; library under lgpl2.1+
+                                 "modinfo" "depmod"))
+                     #t))
+                 %standard-phases)))))
 
 (define-public eudev
   ;; The post-systemd fork, maintained by Gentoo.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 25 Feb 2018 11:49:02 GMT) Full text and rfc822 format available.

Message #11 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH 2/4] gnu: Add kmod-minimal-static.
Date: Sun, 25 Feb 2018 12:48:14 +0100
* gnu/packages/linux.scm (kmod-minimal/static): New variable.
---
 gnu/packages/linux.scm | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 1f8bf3050..b2e47f79a 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1994,6 +1994,50 @@ from the module-init-tools project.")
                      #t))
                  %standard-phases)))))
 
+(define-public kmod-minimal/static
+  (static-package
+   (package (inherit kmod-minimal)
+    (name "kmod-minimal-static")
+    (version "13")
+    (source (origin
+              (method url-fetch)
+              (uri
+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+                              "kmod-" version ".tar.xz"))
+              (sha256
+               (base32
+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+              (patches (search-patches "kmod-13-module-directory.patch"))))
+    (arguments
+     (substitute-keyword-arguments
+       (package-arguments (static-package kmod-minimal))
+       ((#:configure-flags flags ''())
+        `(cons* "--disable-manpages" "--disable-static" "--disable-shared" ,flags))
+       ((#:make-flags flags ''())
+        `(cons* "LDFLAGS=-all-static" ,flags))
+       ((#:phases phases '%standard-phases)
+        `(modify-phases ,phases
+          (delete 'install-license-files)
+          (add-after 'unpack 'patch-kmod
+           (lambda _
+             ;; Reduce size by 200 kiB.
+             (substitute* "tools/kmod.c"
+              (("[&]kmod_cmd_compat_lsmod,") "")
+              (("[&]kmod_cmd_compat_rmmod,") "")
+              (("[&]kmod_cmd_compat_insmod,") "")
+              (("[&]kmod_cmd_compat_modinfo,") ""))
+             #t))
+          (replace 'install
+            (lambda* (#:key outputs #:allow-other-keys)
+              (let* ((out (assoc-ref outputs "out"))
+                     (bin (string-append out "/bin")))
+                (install-file "tools/kmod" bin)
+                (for-each
+                 (lambda (tool)
+                   (symlink "kmod" (string-append bin "/" tool)))
+                 '("modprobe" "depmod"))
+                #t))))))))))
+
 (define-public eudev
   ;; The post-systemd fork, maintained by Gentoo.
   (package




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 25 Feb 2018 11:49:03 GMT) Full text and rfc822 format available.

Message #14 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH 3/4] linux-initrd: Add kmod.
Date: Sun, 25 Feb 2018 12:48:15 +0100
* gnu/system/linux-initrd.scm (raw-initrd): Add kmod.
(base-initrd): Add kmod.
(expression->initrd): Add kmod, linux-module-directory.
(flat-linux-module-directory): Add kmod; invoke depmod.
* gnu/build/linux-initrd.scm (build-initrd): Add kmod, linux-module-directory.
---
 gnu/build/linux-initrd.scm  | 13 +++++++++++-
 gnu/system/linux-initrd.scm | 48 ++++++++++++++++++++++++++++++++++-----------
 2 files changed, 49 insertions(+), 12 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..52cc180b4 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."
 
 (define* (build-initrd output
                        #:key
-                       guile init
+                       guile init kmod linux-module-directory
                        (references-graphs '())
                        (gzip "gzip"))
   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script
@@ -131,6 +131,17 @@ REFERENCES-GRAPHS."
     (symlink (string-append guile "/bin/guile") "proc/self/exe")
     (readlink "proc/self/exe")
 
+    ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+    (if kmod
+        (begin
+          (mkdir-p "sbin")
+          (symlink (string-append kmod "/bin/modprobe") "sbin/modprobe")))
+
+    ;; Make modules available as /lib/modules so modprobe finds them.
+    (mkdir-p "lib")
+    (symlink (string-append linux-module-directory "/lib/modules")
+             "lib/modules")
+
     ;; Reset the timestamps of all the files that will make it in the initrd.
     (for-each (lambda (file)
                 (unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 7170d1c0e..93089a869 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -59,6 +59,8 @@
                              #:key
                              (guile %guile-static-stripped)
                              (gzip gzip)
+                             kmod
+                             linux-module-directory
                              (name "guile-initrd")
                              (system (%current-system)))
   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -94,6 +96,8 @@ the derivations referenced by EXP are automatically copied to the initrd."
           (build-initrd (string-append #$output "/initrd")
                         #:guile #$guile
                         #:init #$init
+                        #:kmod #$kmod
+                        #:linux-module-directory #$linux-module-directory
                         ;; Copy everything INIT refers to into the initrd.
                         #:references-graphs '("closure")
                         #:gzip (string-append #$gzip "/bin/gzip")))))
@@ -101,7 +105,7 @@ the derivations referenced by EXP are automatically copied to the initrd."
   (gexp->derivation name builder
                     #:references-graphs `(("closure" ,init))))
 
-(define (flat-linux-module-directory linux modules)
+(define (flat-linux-module-directory linux modules kmod)
   "Return a flat directory containing the Linux kernel modules listed in
 MODULES and taken from LINUX."
   (define build-exp
@@ -109,7 +113,7 @@ MODULES and taken from LINUX."
                             '((guix build utils)
                               (gnu build linux-modules)))
       #~(begin
-          (use-modules (ice-9 match) (ice-9 regex)
+          (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)
                        (srfi srfi-1)
                        (guix build utils)
                        (gnu build linux-modules))
@@ -138,13 +142,30 @@ MODULES and taken from LINUX."
                       (recursive-module-dependencies modules
                                                      #:lookup-module lookup))))
 
-          (mkdir #$output)
-          (for-each (lambda (module)
-                      (format #t "copying '~a'...~%" module)
-                      (copy-file module
-                                 (string-append #$output "/"
-                                                (basename module))))
-                    (delete-duplicates modules)))))
+          (define version
+            (car
+             (filter
+              (lambda (name)
+                (not (string-prefix? "." name)))
+              (scandir module-dir))))
+
+          (display "VERSION")
+          (display version)
+          (newline)
+
+          (let ((output (string-append #$output "/lib/modules/" version)))
+            (mkdir-p output)
+            (for-each (lambda (module)
+                        (format #t "copying '~a'...~%" module)
+                        (copy-file module
+                                   (string-append output "/"
+                                                  (basename module))))
+                      (delete-duplicates modules)))
+          (invoke (string-append #$kmod "/bin/depmod") "-a" "-b" #$output
+                  ; -E
+                  "-F" (string-append #$linux "/System.map")
+                  version)
+          #t)))
 
   (computed-file "linux-modules" build-exp))
 
@@ -152,6 +173,7 @@ MODULES and taken from LINUX."
                       #:key
                       (linux linux-libre)
                       (linux-modules '())
+                      (kmod kmod-minimal/static)
                       (mapped-devices '())
                       (helper-packages '())
                       qemu-networking?
@@ -185,7 +207,7 @@ upon error."
          mapped-devices))
 
   (define kodir
-    (flat-linux-module-directory linux linux-modules))
+    (flat-linux-module-directory linux linux-modules kmod))
 
   (expression->initrd
    (with-imported-modules (source-module-closure
@@ -223,6 +245,8 @@ upon error."
                       #:qemu-guest-networking? #$qemu-networking?
                       #:volatile-root? '#$volatile-root?
                       #:on-error '#$on-error)))
+   #:kmod kmod
+   #:linux-module-directory kodir
    #:name "raw-initrd"))
 
 (define* (file-system-packages file-systems #:key (volatile-root? #f))
@@ -245,6 +269,7 @@ FILE-SYSTEMS."
 (define* (base-initrd file-systems
                       #:key
                       (linux linux-libre)
+                      (kmod kmod-minimal/static)
                       (mapped-devices '())
                       qemu-networking?
                       volatile-root?
@@ -322,8 +347,9 @@ loaded at boot time in the order in which they appear."
   (raw-initrd file-systems
               #:linux linux
               #:linux-modules linux-modules
+              #:kmod kmod
               #:mapped-devices mapped-devices
-              #:helper-packages helper-packages
+              #:helper-packages (cons kmod helper-packages)
               #:qemu-networking? qemu-networking?
               #:volatile-root? volatile-root?
               #:on-error on-error))




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 25 Feb 2018 11:49:03 GMT) Full text and rfc822 format available.

Message #17 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH 4/4] linux-boot: Load kernel modules only when the hardware is
 present.
Date: Sun, 25 Feb 2018 12:48:16 +0100
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
 gnu/build/linux-boot.scm    | 24 ++++++++++++++++++++----
 gnu/system/linux-initrd.scm |  4 +++-
 2 files changed, 23 insertions(+), 5 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..65c91c50f 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -473,6 +473,25 @@ upon error."
     (string-append linux-module-directory "/"
                    (ensure-dot-ko name)))
 
+  (define (load-kernel-modules)
+    (define enter?
+      (const #t))
+    (file-system-fold
+     enter?
+     ;; This is the Leaf handler.  It tries to modprobe all the modaliases.
+     (lambda (path stat result) ; leaf
+       (let ((modalias-name (string-append path "/modalias")))
+         (if (file-exists? modalias-name)
+             (let ((modalias (call-with-input-file modalias-name read-string)))
+               (if (not (string=? modalias ""))
+                   (system* "/sbin/modprobe" "-q" "--" modalias))))))
+     (const #t) ; down
+     (const #t) ; up
+     (const #f) ; skip
+     (const #f) ; error
+     #f ; init
+     "/sys/devices"))
+
   (display "Welcome, this is GNU's early boot Guile.\n")
   (display "Use '--repl' for an initrd REPL.\n\n")
 
@@ -486,10 +505,7 @@ upon error."
        (when (member "--repl" args)
          (start-repl))
 
-       (display "loading kernel modules...\n")
-       (for-each (cut load-linux-module* <>
-                      #:lookup-module lookup-module)
-                 (map lookup-module linux-modules))
+       (load-kernel-modules)
 
        (when qemu-guest-networking?
          (unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 93089a869..573f4a324 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -228,7 +228,9 @@ upon error."
                       ;; this info via gexps.
                       ((gnu build file-systems)
                        #:select (find-partition-by-luks-uuid))
-                      (rnrs bytevectors))
+                      (rnrs bytevectors)
+                      (ice-9 ftw)
+                      (ice-9 rdelim))
 
          (with-output-to-port (%make-void-port "w")
            (lambda ()




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 25 Feb 2018 14:06:02 GMT) Full text and rfc822 format available.

Message #20 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <m.othacehe <at> gmail.com>
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH 3/4] linux-initrd: Add kmod.
Date: Sun, 25 Feb 2018 15:05:24 +0100
Hi Danny,

Here are some small remarks,

> +    ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
> +    (if kmod
> +        (begin

You can use "when" to avoid "begin".

> +          (define version
> +            (car

Prefer match to car/cdr use.

> +             (filter
> +              (lambda (name)
> +                (not (string-prefix? "." name)))
> +              (scandir module-dir))))
> +
> +          (display "VERSION")
> +          (display version)
> +          (newline)

(format #t "VERSION" ~a~%" version) would be shorter.

> +                  ; -E

Why is this commented ?

> +                  "-F" (string-append #$linux "/System.map")
> +                  version)
> +          #t)))
>  

Thanks,

Mathieu




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 25 Feb 2018 15:08:02 GMT) Full text and rfc822 format available.

Message #23 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Mathieu Othacehe <m.othacehe <at> gmail.com>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH 3/4] linux-initrd: Add kmod.
Date: Sun, 25 Feb 2018 16:07:50 +0100
Hi Mathieu,

thanks for the review.

On Sun, 25 Feb 2018 15:05:24 +0100
Mathieu Othacehe <m.othacehe <at> gmail.com> wrote:

> > +                  ; -E  
> 
> Why is this commented ?

Because it's an option for specifying the location of "Module.symvers" - and
I don't know whether guix uses it (probably not).  If one doesn't specify an
option, depmod will default to the running kernel - which is not what we want.

I should elaborate in the comment that, if we start using Module.symvers, we
MUST pass "-E" there.  Maybe better to even just check for the file existence
and add it right now, otherwise we might forget later.  What do you think?




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 26 Feb 2018 01:11:01 GMT) Full text and rfc822 format available.

Message #26 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Danny Milosavljevic <dannym <at> scratchpost.org>, 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH 1/4] gnu: kmod: Split off kmod-minimal.
Date: Mon, 26 Feb 2018 02:10:14 +0100
[Message part 1 (text/plain, inline)]
Danny Milosavljevic <dannym <at> scratchpost.org> writes:

> * gnu/packages/linux.scm (kmod-minimal): New variable.
> (kmod): Modify.
> ---
>  gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++----------------
>  1 file changed, 33 insertions(+), 16 deletions(-)
>
> diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
> index 238398e84..1f8bf3050 100644
> --- a/gnu/packages/linux.scm
> +++ b/gnu/packages/linux.scm
> @@ -1933,8 +1933,35 @@ for systems using the Linux kernel.  This includes commands such as
>  to use Linux' inotify mechanism, which allows file accesses to be monitored.")
>      (license license:gpl2+)))
>  
> -(define-public kmod
> +(define kmod-minimal
>    (package
> +    (name "kmod-minimal")
> +    (version "13")

Why this old version?

> +    (source (origin
> +              (method url-fetch)
> +              (uri
> +               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
> +                              "kmod-" version ".tar.xz"))
> +              (sha256
> +               (base32
> +                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
> +              (patches (search-patches "kmod-13-module-directory.patch"))))

This patch seems to be missing.

> +    (build-system gnu-build-system)
> +    (arguments
> +     `(#:tests? #f)) ; FIXME: Investigate test failures
> +    (home-page "https://www.kernel.org/")
> +    (synopsis "Kernel module tools")
> +    (description "Kmod is a set of tools to handle common tasks with Linux
> +kernel modules like insert, remove, list, check properties, resolve
> +dependencies and aliases.
> +
> +These tools are designed on top of libkmod, a library that is shipped with
> +kmod.  The aim is to be compatible with tools, configurations and indices
> +from the module-init-tools project.")
> +    (license license:gpl2+))) ; library under lgpl2.1+
> +
> +(define-public kmod
> +  (package (inherit kmod-minimal)
>      (name "kmod")
>      (version "24")
>      (source (origin
> @@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
>                 (base32
>                  "15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))
>                (patches (search-patches "kmod-module-directory.patch"))))
> -    (build-system gnu-build-system)
>      (native-inputs
>       `(("pkg-config" ,pkg-config)))
>      (inputs
>       `(("xz" ,xz)
>         ("zlib" ,zlib)))
>      (arguments
> -     `(#:tests? #f ; FIXME: Investigate test failures
> -       #:configure-flags '("--with-xz" "--with-zlib")
> +     `(#:configure-flags '("--with-xz" "--with-zlib")
> +       #:tests? #f ; FIXME: Investigate test failures
>         #:phases (alist-cons-after
>                   'install 'install-modprobe&co
>                   (lambda* (#:key outputs #:allow-other-keys)
> @@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
>                                   (symlink "kmod"
>                                            (string-append bin "/" tool)))
>                                 '("insmod" "rmmod" "lsmod" "modprobe"
> -                                 "modinfo" "depmod"))))
> -                 %standard-phases)))
> -    (home-page "https://www.kernel.org/")
> -    (synopsis "Kernel module tools")
> -    (description "Kmod is a set of tools to handle common tasks with Linux
> -kernel modules like insert, remove, list, check properties, resolve
> -dependencies and aliases.
> -
> -These tools are designed on top of libkmod, a library that is shipped with
> -kmod.  The aim is to be compatible with tools, configurations and indices
> -from the module-init-tools project.")
> -    (license license:gpl2+))) ; library under lgpl2.1+
> +                                 "modinfo" "depmod"))
> +                     #t))
> +                 %standard-phases)))))
>  
>  (define-public eudev
>    ;; The post-systemd fork, maintained by Gentoo.
[signature.asc (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 26 Feb 2018 03:51:02 GMT) Full text and rfc822 format available.

Message #29 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v2 1/6] gnu: kmod: Split off kmod-minimal.
Date: Mon, 26 Feb 2018 04:50:20 +0100
* gnu/packages/linux.scm (kmod-minimal): New variable.
(kmod): Modify.
---
 gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++----------------
 1 file changed, 33 insertions(+), 16 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 238398e84..1f8bf3050 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1933,8 +1933,35 @@ for systems using the Linux kernel.  This includes commands such as
 to use Linux' inotify mechanism, which allows file accesses to be monitored.")
     (license license:gpl2+)))
 
-(define-public kmod
+(define kmod-minimal
   (package
+    (name "kmod-minimal")
+    (version "13")
+    (source (origin
+              (method url-fetch)
+              (uri
+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+                              "kmod-" version ".tar.xz"))
+              (sha256
+               (base32
+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+              (patches (search-patches "kmod-13-module-directory.patch"))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:tests? #f)) ; FIXME: Investigate test failures
+    (home-page "https://www.kernel.org/")
+    (synopsis "Kernel module tools")
+    (description "Kmod is a set of tools to handle common tasks with Linux
+kernel modules like insert, remove, list, check properties, resolve
+dependencies and aliases.
+
+These tools are designed on top of libkmod, a library that is shipped with
+kmod.  The aim is to be compatible with tools, configurations and indices
+from the module-init-tools project.")
+    (license license:gpl2+))) ; library under lgpl2.1+
+
+(define-public kmod
+  (package (inherit kmod-minimal)
     (name "kmod")
     (version "24")
     (source (origin
@@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
                (base32
                 "15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))
               (patches (search-patches "kmod-module-directory.patch"))))
-    (build-system gnu-build-system)
     (native-inputs
      `(("pkg-config" ,pkg-config)))
     (inputs
      `(("xz" ,xz)
        ("zlib" ,zlib)))
     (arguments
-     `(#:tests? #f ; FIXME: Investigate test failures
-       #:configure-flags '("--with-xz" "--with-zlib")
+     `(#:configure-flags '("--with-xz" "--with-zlib")
+       #:tests? #f ; FIXME: Investigate test failures
        #:phases (alist-cons-after
                  'install 'install-modprobe&co
                  (lambda* (#:key outputs #:allow-other-keys)
@@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
                                  (symlink "kmod"
                                           (string-append bin "/" tool)))
                                '("insmod" "rmmod" "lsmod" "modprobe"
-                                 "modinfo" "depmod"))))
-                 %standard-phases)))
-    (home-page "https://www.kernel.org/")
-    (synopsis "Kernel module tools")
-    (description "Kmod is a set of tools to handle common tasks with Linux
-kernel modules like insert, remove, list, check properties, resolve
-dependencies and aliases.
-
-These tools are designed on top of libkmod, a library that is shipped with
-kmod.  The aim is to be compatible with tools, configurations and indices
-from the module-init-tools project.")
-    (license license:gpl2+))) ; library under lgpl2.1+
+                                 "modinfo" "depmod"))
+                     #t))
+                 %standard-phases)))))
 
 (define-public eudev
   ;; The post-systemd fork, maintained by Gentoo.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 26 Feb 2018 03:51:02 GMT) Full text and rfc822 format available.

Message #32 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v2 2/6] gnu: Add kmod-minimal-static.
Date: Mon, 26 Feb 2018 04:50:21 +0100
* gnu/packages/linux.scm (kmod-minimal/static): New variable.
* gnu/packages/patches/kmod-13-module-directory.patch: New file.
* gnu/local.mk: Add it.
---
 gnu/local.mk                                       |  1 +
 gnu/packages/linux.scm                             | 44 ++++++++++++++++++++++
 .../patches/kmod-13-module-directory.patch         | 33 ++++++++++++++++
 3 files changed, 78 insertions(+)
 create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 21195f8c1..b1e3c878d 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -795,6 +795,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/kiki-makefile.patch			\
   %D%/packages/patches/kiki-missing-includes.patch		\
   %D%/packages/patches/kiki-portability-64bit.patch		\
+  %D%/packages/patches/kmod-13-module-directory.patch		\
   %D%/packages/patches/kmod-module-directory.patch		\
   %D%/packages/patches/kobodeluxe-paths.patch			\
   %D%/packages/patches/kobodeluxe-enemies-pipe-decl.patch	\
diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 1f8bf3050..b2e47f79a 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1994,6 +1994,50 @@ from the module-init-tools project.")
                      #t))
                  %standard-phases)))))
 
+(define-public kmod-minimal/static
+  (static-package
+   (package (inherit kmod-minimal)
+    (name "kmod-minimal-static")
+    (version "13")
+    (source (origin
+              (method url-fetch)
+              (uri
+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+                              "kmod-" version ".tar.xz"))
+              (sha256
+               (base32
+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+              (patches (search-patches "kmod-13-module-directory.patch"))))
+    (arguments
+     (substitute-keyword-arguments
+       (package-arguments (static-package kmod-minimal))
+       ((#:configure-flags flags ''())
+        `(cons* "--disable-manpages" "--disable-static" "--disable-shared" ,flags))
+       ((#:make-flags flags ''())
+        `(cons* "LDFLAGS=-all-static" ,flags))
+       ((#:phases phases '%standard-phases)
+        `(modify-phases ,phases
+          (delete 'install-license-files)
+          (add-after 'unpack 'patch-kmod
+           (lambda _
+             ;; Reduce size by 200 kiB.
+             (substitute* "tools/kmod.c"
+              (("[&]kmod_cmd_compat_lsmod,") "")
+              (("[&]kmod_cmd_compat_rmmod,") "")
+              (("[&]kmod_cmd_compat_insmod,") "")
+              (("[&]kmod_cmd_compat_modinfo,") ""))
+             #t))
+          (replace 'install
+            (lambda* (#:key outputs #:allow-other-keys)
+              (let* ((out (assoc-ref outputs "out"))
+                     (bin (string-append out "/bin")))
+                (install-file "tools/kmod" bin)
+                (for-each
+                 (lambda (tool)
+                   (symlink "kmod" (string-append bin "/" tool)))
+                 '("modprobe" "depmod"))
+                #t))))))))))
+
 (define-public eudev
   ;; The post-systemd fork, maintained by Gentoo.
   (package
diff --git a/gnu/packages/patches/kmod-13-module-directory.patch b/gnu/packages/patches/kmod-13-module-directory.patch
new file mode 100644
index 000000000..5ff2f8a60
--- /dev/null
+++ b/gnu/packages/patches/kmod-13-module-directory.patch
@@ -0,0 +1,33 @@
+This patch changes libkmod so it honors the 'LINUX_MODULE_DIRECTORY'
+environment variable, rather than looking for modules exclusively in
+/lib/modules.
+
+Patch by Shea Levy and Eelco Dolstra, from Nixpkgs; adjusted to
+use 'LINUX_MODULE_DIRECTORY' rather than 'MODULE_DIR' as the variable
+name.
+
+
+--- kmod-7/libkmod/libkmod.c	2012-03-15 08:19:16.750010226 -0400
++++ kmod-7/libkmod/libkmod.c	2012-04-04 15:21:29.532074313 -0400
+@@ -200,7 +200,7 @@
+ static char *get_kernel_release(const char *dirname)
+ {
+ 	struct utsname u;
+-	char *p;
++	char *p, *dirname_prefix;
+ 
+ 	if (dirname != NULL)
+ 		return path_make_absolute_cwd(dirname);
+@@ -208,7 +208,10 @@
+ 	if (uname(&u) < 0)
+ 		return NULL;
+ 
+-	if (asprintf(&p, "%s/%s", dirname_default_prefix, u.release) < 0)
++	if ((dirname_prefix = getenv("LINUX_MODULE_DIRECTORY")) == NULL)
++		dirname_prefix = dirname_default_prefix;
++
++	if (asprintf(&p, "%s/%s", dirname_prefix, u.release) < 0)
+ 		return NULL;
+ 
+ 	return p;
+




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 26 Feb 2018 03:51:03 GMT) Full text and rfc822 format available.

Message #35 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v2 0/6] Load Linux module only when supported hardware is
 present.
Date: Mon, 26 Feb 2018 04:50:19 +0100
Danny Milosavljevic (6):
  gnu: kmod: Split off kmod-minimal.
  gnu: Add kmod-minimal-static.
  linux-initrd: Add kmod.
  linux-boot: Load kernel modules only when the hardware is present.
  vm: Allow qemu-image builder to load Linux kernel modules.
  vm: Make the virtio-blk is uniquely identifyable in /sys.

 gnu/build/linux-boot.scm                           | 31 +++++++-
 gnu/build/linux-initrd.scm                         | 12 ++-
 gnu/build/vm.scm                                   |  2 +-
 gnu/local.mk                                       |  1 +
 gnu/packages/linux.scm                             | 91 ++++++++++++++++++----
 .../patches/kmod-13-module-directory.patch         | 33 ++++++++
 gnu/system/linux-initrd.scm                        | 48 +++++++++---
 gnu/system/vm.scm                                  | 28 ++++++-
 8 files changed, 210 insertions(+), 36 deletions(-)
 create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 26 Feb 2018 03:51:03 GMT) Full text and rfc822 format available.

Message #38 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v2 3/6] linux-initrd: Add kmod.
Date: Mon, 26 Feb 2018 04:50:22 +0100
* gnu/system/linux-initrd.scm (raw-initrd): Add kmod.
(base-initrd): Add kmod.
(expression->initrd): Add kmod, linux-module-directory.
(flat-linux-module-directory): Add kmod; invoke depmod.
* gnu/build/linux-initrd.scm (build-initrd): Add kmod, linux-module-directory.
---
 gnu/build/linux-initrd.scm  | 12 +++++++++++-
 gnu/system/linux-initrd.scm | 44 +++++++++++++++++++++++++++++++++-----------
 2 files changed, 44 insertions(+), 12 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..6356007df 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."
 
 (define* (build-initrd output
                        #:key
-                       guile init
+                       guile init kmod linux-module-directory
                        (references-graphs '())
                        (gzip "gzip"))
   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script
@@ -131,6 +131,16 @@ REFERENCES-GRAPHS."
     (symlink (string-append guile "/bin/guile") "proc/self/exe")
     (readlink "proc/self/exe")
 
+    ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+    (when kmod
+      (mkdir-p "sbin")
+      (symlink (string-append kmod "/bin/modprobe") "sbin/modprobe"))
+
+    ;; Make modules available as /lib/modules so modprobe finds them.
+    (mkdir-p "lib")
+    (symlink (string-append linux-module-directory "/lib/modules")
+             "lib/modules")
+
     ;; Reset the timestamps of all the files that will make it in the initrd.
     (for-each (lambda (file)
                 (unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 7170d1c0e..91e498787 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -59,6 +59,8 @@
                              #:key
                              (guile %guile-static-stripped)
                              (gzip gzip)
+                             kmod
+                             linux-module-directory
                              (name "guile-initrd")
                              (system (%current-system)))
   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -94,6 +96,8 @@ the derivations referenced by EXP are automatically copied to the initrd."
           (build-initrd (string-append #$output "/initrd")
                         #:guile #$guile
                         #:init #$init
+                        #:kmod #$kmod
+                        #:linux-module-directory #$linux-module-directory
                         ;; Copy everything INIT refers to into the initrd.
                         #:references-graphs '("closure")
                         #:gzip (string-append #$gzip "/bin/gzip")))))
@@ -101,7 +105,7 @@ the derivations referenced by EXP are automatically copied to the initrd."
   (gexp->derivation name builder
                     #:references-graphs `(("closure" ,init))))
 
-(define (flat-linux-module-directory linux modules)
+(define (flat-linux-module-directory linux modules kmod)
   "Return a flat directory containing the Linux kernel modules listed in
 MODULES and taken from LINUX."
   (define build-exp
@@ -109,7 +113,7 @@ MODULES and taken from LINUX."
                             '((guix build utils)
                               (gnu build linux-modules)))
       #~(begin
-          (use-modules (ice-9 match) (ice-9 regex)
+          (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)
                        (srfi srfi-1)
                        (guix build utils)
                        (gnu build linux-modules))
@@ -138,13 +142,26 @@ MODULES and taken from LINUX."
                       (recursive-module-dependencies modules
                                                      #:lookup-module lookup))))
 
-          (mkdir #$output)
-          (for-each (lambda (module)
-                      (format #t "copying '~a'...~%" module)
-                      (copy-file module
-                                 (string-append #$output "/"
-                                                (basename module))))
-                    (delete-duplicates modules)))))
+          (define version
+            (car
+             (filter
+              (lambda (name)
+                (not (string-prefix? "." name)))
+              (scandir module-dir))))
+
+          (let ((output (string-append #$output "/lib/modules/" version)))
+            (mkdir-p output)
+            (for-each (lambda (module)
+                        (format #t "copying '~a'...~%" module)
+                        (copy-file module
+                                   (string-append output "/"
+                                                  (basename module))))
+                      (delete-duplicates modules)))
+          (invoke (string-append #$kmod "/bin/depmod") "-a" "-b" #$output
+                  ; -E
+                  "-F" (string-append #$linux "/System.map")
+                  version)
+          #t)))
 
   (computed-file "linux-modules" build-exp))
 
@@ -152,6 +169,7 @@ MODULES and taken from LINUX."
                       #:key
                       (linux linux-libre)
                       (linux-modules '())
+                      (kmod kmod-minimal/static)
                       (mapped-devices '())
                       (helper-packages '())
                       qemu-networking?
@@ -185,7 +203,7 @@ upon error."
          mapped-devices))
 
   (define kodir
-    (flat-linux-module-directory linux linux-modules))
+    (flat-linux-module-directory linux linux-modules kmod))
 
   (expression->initrd
    (with-imported-modules (source-module-closure
@@ -223,6 +241,8 @@ upon error."
                       #:qemu-guest-networking? #$qemu-networking?
                       #:volatile-root? '#$volatile-root?
                       #:on-error '#$on-error)))
+   #:kmod kmod
+   #:linux-module-directory kodir
    #:name "raw-initrd"))
 
 (define* (file-system-packages file-systems #:key (volatile-root? #f))
@@ -245,6 +265,7 @@ FILE-SYSTEMS."
 (define* (base-initrd file-systems
                       #:key
                       (linux linux-libre)
+                      (kmod kmod-minimal/static)
                       (mapped-devices '())
                       qemu-networking?
                       volatile-root?
@@ -322,8 +343,9 @@ loaded at boot time in the order in which they appear."
   (raw-initrd file-systems
               #:linux linux
               #:linux-modules linux-modules
+              #:kmod kmod
               #:mapped-devices mapped-devices
-              #:helper-packages helper-packages
+              #:helper-packages (cons kmod helper-packages)
               #:qemu-networking? qemu-networking?
               #:volatile-root? volatile-root?
               #:on-error on-error))




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 26 Feb 2018 03:51:03 GMT) Full text and rfc822 format available.

Message #41 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v2 4/6] linux-boot: Load kernel modules only when the hardware
 is present.
Date: Mon, 26 Feb 2018 04:50:23 +0100
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
 gnu/build/linux-boot.scm    | 31 +++++++++++++++++++++++++++----
 gnu/system/linux-initrd.scm |  4 +++-
 2 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..6d00ea9be 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -473,6 +473,32 @@ upon error."
     (string-append linux-module-directory "/"
                    (ensure-dot-ko name)))
 
+  (define (load-kernel-modules)
+    "Examine /sys/devices to find out which modules to load and load them."
+    (define enter?
+      (const #t))
+    (define (down! path stat result)
+     ;; Note: modprobe mutates the tree starting with path.
+     (let ((modalias-name (string-append path "/modalias")))
+       (if (file-exists? modalias-name)
+           (let ((modalias
+                 (string-trim-right (call-with-input-file modalias-name
+                                                          read-string)
+                                    #\newline)))
+             (system* "/sbin/modprobe" "-q" "--" modalias))))
+       #t)
+    (define up
+      (const #t))
+    (define skip
+      (const #t))
+    (define leaf
+      (const #t))
+    (define (error name stat errno result)
+      (format (current-error-port) "warning: ~a: ~a~%"
+              name (strerror errno))
+      result)
+    (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))
+
   (display "Welcome, this is GNU's early boot Guile.\n")
   (display "Use '--repl' for an initrd REPL.\n\n")
 
@@ -486,10 +512,7 @@ upon error."
        (when (member "--repl" args)
          (start-repl))
 
-       (display "loading kernel modules...\n")
-       (for-each (cut load-linux-module* <>
-                      #:lookup-module lookup-module)
-                 (map lookup-module linux-modules))
+       (load-kernel-modules)
 
        (when qemu-guest-networking?
          (unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 91e498787..6d130ccac 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -224,7 +224,9 @@ upon error."
                       ;; this info via gexps.
                       ((gnu build file-systems)
                        #:select (find-partition-by-luks-uuid))
-                      (rnrs bytevectors))
+                      (rnrs bytevectors)
+                      (ice-9 ftw)
+                      (ice-9 rdelim))
 
          (with-output-to-port (%make-void-port "w")
            (lambda ()




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 26 Feb 2018 03:51:04 GMT) Full text and rfc822 format available.

Message #44 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v2 5/6] vm: Allow qemu-image builder to load Linux kernel
 modules.
Date: Mon, 26 Feb 2018 04:50:24 +0100
* gnu/system/vm.scm (%modprobe-wrapper): New variable.
(qemu-image): Modify.
---
 gnu/system/vm.scm | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 345cecedd..c64c9678f 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."
    #:single-file-output? #t
    #:references-graphs inputs))
 
+(define (%modprobe-wrapper modprobe linux-module-directory)
+  ;; Wrapper for the 'modprobe' command that knows where modules live.
+  ;;
+  ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',
+  ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'
+  ;; environment variable is not set---hence the need for this wrapper.
+  (program-file "modprobe"
+    #~(begin
+        (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)
+        (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))
+
 (define* (qemu-image #:key
                      (name "qemu-image")
                      (system (%current-system))
@@ -275,6 +286,8 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy
 all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true,
 register INPUTS in the store database of the image so that Guix can be used in
 the image."
+  (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))
+        (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))
   (expression->derivation-in-linux-vm
    name
    (with-imported-modules (source-module-closure '((gnu build bootloader)
@@ -288,7 +301,7 @@ the image."
                       (ice-9 binary-ports))
 
          (let ((inputs
-                '#$(append (list qemu parted e2fsprogs dosfstools)
+                '#$(append (list qemu parted e2fsprogs dosfstools kmod)
                            (map canonical-package
                                 (list sed grep coreutils findutils gawk))
                            (if register-closures? (list guix) '())))
@@ -302,6 +315,14 @@ the image."
                         inputs)))
 
            (set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+           ;; It's possible that we need to load nls modules in order to
+           ;; mount the new partition.
+           (if (file-exists? #$modprobe-name)
+               (activate-modprobe #$(%modprobe-wrapper modprobe-name
+                                     linux-module-directory))
+               (format (current-error-port)
+                "WARNING: No modprobe found in ~s.  \
+Loading kernel modules will be impossible.\n" #$modprobe-name))
 
            (let* ((graphs     '#$(match inputs
                                    (((names . _) ...)
@@ -364,7 +385,7 @@ the image."
    #:make-disk-image? #t
    #:disk-image-size disk-image-size
    #:disk-image-format disk-image-format
-   #:references-graphs inputs))
+   #:references-graphs inputs)))
 
 
 ;;;




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 26 Feb 2018 03:51:04 GMT) Full text and rfc822 format available.

Message #47 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v2 6/6] vm: Make the virtio-blk is uniquely identifyable in
 /sys.
Date: Mon, 26 Feb 2018 04:50:25 +0100
* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.
---
 gnu/build/vm.scm  | 2 +-
 gnu/system/vm.scm | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..ebf9e9f6e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."
                                            builder)
                   (append
                    (if make-disk-image?
-                       `("-device" "virtio-blk,drive=myhd"
+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
                          "-drive" ,(string-append "if=none,file=" output
                                                   ",format=" disk-image-format
                                                   ",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index c64c9678f..ad48999ee 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -704,7 +704,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."
 
      #$@(map virtfs-option shared-fs)
      "-vga std"
-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
              #$image)))
 
 (define* (system-qemu-image/shared-store-script os




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 26 Feb 2018 04:07:01 GMT) Full text and rfc822 format available.

Message #50 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v3 1/6] gnu: kmod: Split off kmod-minimal.
Date: Mon, 26 Feb 2018 05:06:04 +0100
* gnu/packages/linux.scm (kmod-minimal): New variable.
(kmod): Modify.
---
 gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++----------------
 1 file changed, 33 insertions(+), 16 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 238398e84..1f8bf3050 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1933,8 +1933,35 @@ for systems using the Linux kernel.  This includes commands such as
 to use Linux' inotify mechanism, which allows file accesses to be monitored.")
     (license license:gpl2+)))
 
-(define-public kmod
+(define kmod-minimal
   (package
+    (name "kmod-minimal")
+    (version "13")
+    (source (origin
+              (method url-fetch)
+              (uri
+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+                              "kmod-" version ".tar.xz"))
+              (sha256
+               (base32
+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+              (patches (search-patches "kmod-13-module-directory.patch"))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:tests? #f)) ; FIXME: Investigate test failures
+    (home-page "https://www.kernel.org/")
+    (synopsis "Kernel module tools")
+    (description "Kmod is a set of tools to handle common tasks with Linux
+kernel modules like insert, remove, list, check properties, resolve
+dependencies and aliases.
+
+These tools are designed on top of libkmod, a library that is shipped with
+kmod.  The aim is to be compatible with tools, configurations and indices
+from the module-init-tools project.")
+    (license license:gpl2+))) ; library under lgpl2.1+
+
+(define-public kmod
+  (package (inherit kmod-minimal)
     (name "kmod")
     (version "24")
     (source (origin
@@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
                (base32
                 "15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))
               (patches (search-patches "kmod-module-directory.patch"))))
-    (build-system gnu-build-system)
     (native-inputs
      `(("pkg-config" ,pkg-config)))
     (inputs
      `(("xz" ,xz)
        ("zlib" ,zlib)))
     (arguments
-     `(#:tests? #f ; FIXME: Investigate test failures
-       #:configure-flags '("--with-xz" "--with-zlib")
+     `(#:configure-flags '("--with-xz" "--with-zlib")
+       #:tests? #f ; FIXME: Investigate test failures
        #:phases (alist-cons-after
                  'install 'install-modprobe&co
                  (lambda* (#:key outputs #:allow-other-keys)
@@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
                                  (symlink "kmod"
                                           (string-append bin "/" tool)))
                                '("insmod" "rmmod" "lsmod" "modprobe"
-                                 "modinfo" "depmod"))))
-                 %standard-phases)))
-    (home-page "https://www.kernel.org/")
-    (synopsis "Kernel module tools")
-    (description "Kmod is a set of tools to handle common tasks with Linux
-kernel modules like insert, remove, list, check properties, resolve
-dependencies and aliases.
-
-These tools are designed on top of libkmod, a library that is shipped with
-kmod.  The aim is to be compatible with tools, configurations and indices
-from the module-init-tools project.")
-    (license license:gpl2+))) ; library under lgpl2.1+
+                                 "modinfo" "depmod"))
+                     #t))
+                 %standard-phases)))))
 
 (define-public eudev
   ;; The post-systemd fork, maintained by Gentoo.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 26 Feb 2018 04:07:02 GMT) Full text and rfc822 format available.

Message #53 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v3 0/6] Load Linux module only when supported hardware is
 present.
Date: Mon, 26 Feb 2018 05:06:03 +0100
Danny Milosavljevic (6):
  gnu: kmod: Split off kmod-minimal.
  gnu: Add kmod-minimal-static.
  linux-initrd: Add kmod.
  linux-boot: Load kernel modules only when the hardware is present.
  vm: Allow qemu-image builder to load Linux kernel modules.
  vm: Make the virtio-blk is uniquely identifyable in /sys.

 gnu/build/linux-boot.scm                           | 31 +++++++-
 gnu/build/linux-initrd.scm                         | 12 ++-
 gnu/build/vm.scm                                   |  2 +-
 gnu/local.mk                                       |  1 +
 gnu/packages/linux.scm                             | 91 ++++++++++++++++++----
 .../patches/kmod-13-module-directory.patch         | 33 ++++++++
 gnu/system/linux-initrd.scm                        | 49 +++++++++---
 gnu/system/vm.scm                                  | 34 ++++++--
 8 files changed, 215 insertions(+), 38 deletions(-)
 create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 26 Feb 2018 04:07:02 GMT) Full text and rfc822 format available.

Message #56 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v3 2/6] gnu: Add kmod-minimal-static.
Date: Mon, 26 Feb 2018 05:06:05 +0100
* gnu/packages/linux.scm (kmod-minimal/static): New variable.
* gnu/packages/patches/kmod-13-module-directory.patch: New file.
* gnu/local.mk: Add it.
---
 gnu/local.mk                                       |  1 +
 gnu/packages/linux.scm                             | 44 ++++++++++++++++++++++
 .../patches/kmod-13-module-directory.patch         | 33 ++++++++++++++++
 3 files changed, 78 insertions(+)
 create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 21195f8c1..b1e3c878d 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -795,6 +795,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/kiki-makefile.patch			\
   %D%/packages/patches/kiki-missing-includes.patch		\
   %D%/packages/patches/kiki-portability-64bit.patch		\
+  %D%/packages/patches/kmod-13-module-directory.patch		\
   %D%/packages/patches/kmod-module-directory.patch		\
   %D%/packages/patches/kobodeluxe-paths.patch			\
   %D%/packages/patches/kobodeluxe-enemies-pipe-decl.patch	\
diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 1f8bf3050..b2e47f79a 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1994,6 +1994,50 @@ from the module-init-tools project.")
                      #t))
                  %standard-phases)))))
 
+(define-public kmod-minimal/static
+  (static-package
+   (package (inherit kmod-minimal)
+    (name "kmod-minimal-static")
+    (version "13")
+    (source (origin
+              (method url-fetch)
+              (uri
+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+                              "kmod-" version ".tar.xz"))
+              (sha256
+               (base32
+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+              (patches (search-patches "kmod-13-module-directory.patch"))))
+    (arguments
+     (substitute-keyword-arguments
+       (package-arguments (static-package kmod-minimal))
+       ((#:configure-flags flags ''())
+        `(cons* "--disable-manpages" "--disable-static" "--disable-shared" ,flags))
+       ((#:make-flags flags ''())
+        `(cons* "LDFLAGS=-all-static" ,flags))
+       ((#:phases phases '%standard-phases)
+        `(modify-phases ,phases
+          (delete 'install-license-files)
+          (add-after 'unpack 'patch-kmod
+           (lambda _
+             ;; Reduce size by 200 kiB.
+             (substitute* "tools/kmod.c"
+              (("[&]kmod_cmd_compat_lsmod,") "")
+              (("[&]kmod_cmd_compat_rmmod,") "")
+              (("[&]kmod_cmd_compat_insmod,") "")
+              (("[&]kmod_cmd_compat_modinfo,") ""))
+             #t))
+          (replace 'install
+            (lambda* (#:key outputs #:allow-other-keys)
+              (let* ((out (assoc-ref outputs "out"))
+                     (bin (string-append out "/bin")))
+                (install-file "tools/kmod" bin)
+                (for-each
+                 (lambda (tool)
+                   (symlink "kmod" (string-append bin "/" tool)))
+                 '("modprobe" "depmod"))
+                #t))))))))))
+
 (define-public eudev
   ;; The post-systemd fork, maintained by Gentoo.
   (package
diff --git a/gnu/packages/patches/kmod-13-module-directory.patch b/gnu/packages/patches/kmod-13-module-directory.patch
new file mode 100644
index 000000000..5ff2f8a60
--- /dev/null
+++ b/gnu/packages/patches/kmod-13-module-directory.patch
@@ -0,0 +1,33 @@
+This patch changes libkmod so it honors the 'LINUX_MODULE_DIRECTORY'
+environment variable, rather than looking for modules exclusively in
+/lib/modules.
+
+Patch by Shea Levy and Eelco Dolstra, from Nixpkgs; adjusted to
+use 'LINUX_MODULE_DIRECTORY' rather than 'MODULE_DIR' as the variable
+name.
+
+
+--- kmod-7/libkmod/libkmod.c	2012-03-15 08:19:16.750010226 -0400
++++ kmod-7/libkmod/libkmod.c	2012-04-04 15:21:29.532074313 -0400
+@@ -200,7 +200,7 @@
+ static char *get_kernel_release(const char *dirname)
+ {
+ 	struct utsname u;
+-	char *p;
++	char *p, *dirname_prefix;
+ 
+ 	if (dirname != NULL)
+ 		return path_make_absolute_cwd(dirname);
+@@ -208,7 +208,10 @@
+ 	if (uname(&u) < 0)
+ 		return NULL;
+ 
+-	if (asprintf(&p, "%s/%s", dirname_default_prefix, u.release) < 0)
++	if ((dirname_prefix = getenv("LINUX_MODULE_DIRECTORY")) == NULL)
++		dirname_prefix = dirname_default_prefix;
++
++	if (asprintf(&p, "%s/%s", dirname_prefix, u.release) < 0)
+ 		return NULL;
+ 
+ 	return p;
+




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 26 Feb 2018 04:07:03 GMT) Full text and rfc822 format available.

Message #59 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v3 3/6] linux-initrd: Add kmod.
Date: Mon, 26 Feb 2018 05:06:06 +0100
* gnu/system/linux-initrd.scm (raw-initrd): Add kmod.
(base-initrd): Add kmod.
(expression->initrd): Add kmod, linux-module-directory.
(flat-linux-module-directory): Add kmod; invoke depmod.
* gnu/build/linux-initrd.scm (build-initrd): Add kmod, linux-module-directory.
---
 gnu/build/linux-initrd.scm  | 12 +++++++++++-
 gnu/system/linux-initrd.scm | 45 ++++++++++++++++++++++++++++++++++-----------
 2 files changed, 45 insertions(+), 12 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..6356007df 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."
 
 (define* (build-initrd output
                        #:key
-                       guile init
+                       guile init kmod linux-module-directory
                        (references-graphs '())
                        (gzip "gzip"))
   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script
@@ -131,6 +131,16 @@ REFERENCES-GRAPHS."
     (symlink (string-append guile "/bin/guile") "proc/self/exe")
     (readlink "proc/self/exe")
 
+    ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+    (when kmod
+      (mkdir-p "sbin")
+      (symlink (string-append kmod "/bin/modprobe") "sbin/modprobe"))
+
+    ;; Make modules available as /lib/modules so modprobe finds them.
+    (mkdir-p "lib")
+    (symlink (string-append linux-module-directory "/lib/modules")
+             "lib/modules")
+
     ;; Reset the timestamps of all the files that will make it in the initrd.
     (for-each (lambda (file)
                 (unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 7170d1c0e..46ef055f0 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -59,6 +59,8 @@
                              #:key
                              (guile %guile-static-stripped)
                              (gzip gzip)
+                             kmod
+                             linux-module-directory
                              (name "guile-initrd")
                              (system (%current-system)))
   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -94,6 +96,8 @@ the derivations referenced by EXP are automatically copied to the initrd."
           (build-initrd (string-append #$output "/initrd")
                         #:guile #$guile
                         #:init #$init
+                        #:kmod #$kmod
+                        #:linux-module-directory #$linux-module-directory
                         ;; Copy everything INIT refers to into the initrd.
                         #:references-graphs '("closure")
                         #:gzip (string-append #$gzip "/bin/gzip")))))
@@ -101,7 +105,7 @@ the derivations referenced by EXP are automatically copied to the initrd."
   (gexp->derivation name builder
                     #:references-graphs `(("closure" ,init))))
 
-(define (flat-linux-module-directory linux modules)
+(define (flat-linux-module-directory linux modules kmod)
   "Return a flat directory containing the Linux kernel modules listed in
 MODULES and taken from LINUX."
   (define build-exp
@@ -109,7 +113,7 @@ MODULES and taken from LINUX."
                             '((guix build utils)
                               (gnu build linux-modules)))
       #~(begin
-          (use-modules (ice-9 match) (ice-9 regex)
+          (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)
                        (srfi srfi-1)
                        (guix build utils)
                        (gnu build linux-modules))
@@ -138,13 +142,27 @@ MODULES and taken from LINUX."
                       (recursive-module-dependencies modules
                                                      #:lookup-module lookup))))
 
-          (mkdir #$output)
-          (for-each (lambda (module)
-                      (format #t "copying '~a'...~%" module)
-                      (copy-file module
-                                 (string-append #$output "/"
-                                                (basename module))))
-                    (delete-duplicates modules)))))
+          (define version
+            (match
+             (filter
+              (lambda (name)
+                (not (string-prefix? "." name)))
+              (scandir module-dir))
+             ((item) item)))
+
+          (let ((output (string-append #$output "/lib/modules/" version)))
+            (mkdir-p output)
+            (for-each (lambda (module)
+                        (format #t "copying '~a'...~%" module)
+                        (copy-file module
+                                   (string-append output "/"
+                                                  (basename module))))
+                      (delete-duplicates modules)))
+          (invoke (string-append #$kmod "/bin/depmod") "-a" "-b" #$output
+                  ; -E
+                  "-F" (string-append #$linux "/System.map")
+                  version)
+          #t)))
 
   (computed-file "linux-modules" build-exp))
 
@@ -152,6 +170,7 @@ MODULES and taken from LINUX."
                       #:key
                       (linux linux-libre)
                       (linux-modules '())
+                      (kmod kmod-minimal/static)
                       (mapped-devices '())
                       (helper-packages '())
                       qemu-networking?
@@ -185,7 +204,7 @@ upon error."
          mapped-devices))
 
   (define kodir
-    (flat-linux-module-directory linux linux-modules))
+    (flat-linux-module-directory linux linux-modules kmod))
 
   (expression->initrd
    (with-imported-modules (source-module-closure
@@ -223,6 +242,8 @@ upon error."
                       #:qemu-guest-networking? #$qemu-networking?
                       #:volatile-root? '#$volatile-root?
                       #:on-error '#$on-error)))
+   #:kmod kmod
+   #:linux-module-directory kodir
    #:name "raw-initrd"))
 
 (define* (file-system-packages file-systems #:key (volatile-root? #f))
@@ -245,6 +266,7 @@ FILE-SYSTEMS."
 (define* (base-initrd file-systems
                       #:key
                       (linux linux-libre)
+                      (kmod kmod-minimal/static)
                       (mapped-devices '())
                       qemu-networking?
                       volatile-root?
@@ -322,8 +344,9 @@ loaded at boot time in the order in which they appear."
   (raw-initrd file-systems
               #:linux linux
               #:linux-modules linux-modules
+              #:kmod kmod
               #:mapped-devices mapped-devices
-              #:helper-packages helper-packages
+              #:helper-packages (cons kmod helper-packages)
               #:qemu-networking? qemu-networking?
               #:volatile-root? volatile-root?
               #:on-error on-error))




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 26 Feb 2018 04:07:03 GMT) Full text and rfc822 format available.

Message #62 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v3 4/6] linux-boot: Load kernel modules only when the hardware
 is present.
Date: Mon, 26 Feb 2018 05:06:07 +0100
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
 gnu/build/linux-boot.scm    | 31 +++++++++++++++++++++++++++----
 gnu/system/linux-initrd.scm |  4 +++-
 2 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..6d00ea9be 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -473,6 +473,32 @@ upon error."
     (string-append linux-module-directory "/"
                    (ensure-dot-ko name)))
 
+  (define (load-kernel-modules)
+    "Examine /sys/devices to find out which modules to load and load them."
+    (define enter?
+      (const #t))
+    (define (down! path stat result)
+     ;; Note: modprobe mutates the tree starting with path.
+     (let ((modalias-name (string-append path "/modalias")))
+       (if (file-exists? modalias-name)
+           (let ((modalias
+                 (string-trim-right (call-with-input-file modalias-name
+                                                          read-string)
+                                    #\newline)))
+             (system* "/sbin/modprobe" "-q" "--" modalias))))
+       #t)
+    (define up
+      (const #t))
+    (define skip
+      (const #t))
+    (define leaf
+      (const #t))
+    (define (error name stat errno result)
+      (format (current-error-port) "warning: ~a: ~a~%"
+              name (strerror errno))
+      result)
+    (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))
+
   (display "Welcome, this is GNU's early boot Guile.\n")
   (display "Use '--repl' for an initrd REPL.\n\n")
 
@@ -486,10 +512,7 @@ upon error."
        (when (member "--repl" args)
          (start-repl))
 
-       (display "loading kernel modules...\n")
-       (for-each (cut load-linux-module* <>
-                      #:lookup-module lookup-module)
-                 (map lookup-module linux-modules))
+       (load-kernel-modules)
 
        (when qemu-guest-networking?
          (unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 46ef055f0..c8a9e4950 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -225,7 +225,9 @@ upon error."
                       ;; this info via gexps.
                       ((gnu build file-systems)
                        #:select (find-partition-by-luks-uuid))
-                      (rnrs bytevectors))
+                      (rnrs bytevectors)
+                      (ice-9 ftw)
+                      (ice-9 rdelim))
 
          (with-output-to-port (%make-void-port "w")
            (lambda ()




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 26 Feb 2018 04:07:03 GMT) Full text and rfc822 format available.

Message #65 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v3 5/6] vm: Allow qemu-image builder to load Linux kernel
 modules.
Date: Mon, 26 Feb 2018 05:06:08 +0100
* gnu/system/vm.scm (%modprobe-wrapper): New variable.
(qemu-image): Modify.
---
 gnu/system/vm.scm | 31 +++++++++++++++++++++++++++----
 1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 345cecedd..b5a559012 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."
    #:single-file-output? #t
    #:references-graphs inputs))
 
+(define (%modprobe-wrapper modprobe linux-module-directory)
+  ;; Wrapper for the 'modprobe' command that knows where modules live.
+  ;;
+  ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',
+  ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'
+  ;; environment variable is not set---hence the need for this wrapper.
+  (program-file "modprobe"
+    #~(begin
+        (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)
+        (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))
+
 (define* (qemu-image #:key
                      (name "qemu-image")
                      (system (%current-system))
@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy
 all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true,
 register INPUTS in the store database of the image so that Guix can be used in
 the image."
+  (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))
+        (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))
   (expression->derivation-in-linux-vm
    name
-   (with-imported-modules (source-module-closure '((gnu build bootloader)
+   (with-imported-modules (source-module-closure '((gnu build activation)
+                                                   (gnu build bootloader)
                                                    (gnu build vm)
                                                    (guix build utils)))
      #~(begin
-         (use-modules (gnu build bootloader)
+         (use-modules (gnu build activation)
+                      (gnu build bootloader)
                       (gnu build vm)
                       (guix build utils)
                       (srfi srfi-26)
                       (ice-9 binary-ports))
 
          (let ((inputs
-                '#$(append (list qemu parted e2fsprogs dosfstools)
+                '#$(append (list qemu parted e2fsprogs dosfstools kmod)
                            (map canonical-package
                                 (list sed grep coreutils findutils gawk))
                            (if register-closures? (list guix) '())))
@@ -302,6 +317,14 @@ the image."
                         inputs)))
 
            (set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+           ;; It's possible that we need to load nls modules in order to
+           ;; mount the new partition.
+           (if (file-exists? #$modprobe-name)
+               (activate-modprobe #$(%modprobe-wrapper modprobe-name
+                                     linux-module-directory))
+               (format (current-error-port)
+                "WARNING: No modprobe found in ~s.  \
+Loading kernel modules will be impossible.\n" #$modprobe-name))
 
            (let* ((graphs     '#$(match inputs
                                    (((names . _) ...)
@@ -364,7 +387,7 @@ the image."
    #:make-disk-image? #t
    #:disk-image-size disk-image-size
    #:disk-image-format disk-image-format
-   #:references-graphs inputs))
+   #:references-graphs inputs)))
 
 
 ;;;




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 26 Feb 2018 04:07:04 GMT) Full text and rfc822 format available.

Message #68 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v3 6/6] vm: Make the virtio-blk is uniquely identifyable in
 /sys.
Date: Mon, 26 Feb 2018 05:06:09 +0100
* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.
---
 gnu/build/vm.scm  | 2 +-
 gnu/system/vm.scm | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..ebf9e9f6e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."
                                            builder)
                   (append
                    (if make-disk-image?
-                       `("-device" "virtio-blk,drive=myhd"
+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
                          "-drive" ,(string-append "if=none,file=" output
                                                   ",format=" disk-image-format
                                                   ",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index b5a559012..fdff64ed9 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."
 
      #$@(map virtfs-option shared-fs)
      "-vga std"
-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
              #$image)))
 
 (define* (system-qemu-image/shared-store-script os




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 26 Feb 2018 11:52:02 GMT) Full text and rfc822 format available.

Message #71 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <m.othacehe <at> gmail.com>
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH 3/4] linux-initrd: Add kmod.
Date: Mon, 26 Feb 2018 12:51:51 +0100
Hey Danny,

> Because it's an option for specifying the location of "Module.symvers" - and
> I don't know whether guix uses it (probably not).  If one doesn't specify an
> option, depmod will default to the running kernel - which is not what we want.
>
> I should elaborate in the comment that, if we start using Module.symvers, we
> MUST pass "-E" there.  Maybe better to even just check for the file existence
> and add it right now, otherwise we might forget later.  What do you think?

Reading 'depmod' manpage, I understand that -E and -F are mutually
exclusive. Both options seem to have an interest only if -e is supplied
to "reports any symbols which a module needs which are not supplied by
other modules or the kernel".

So maybe something like "-F (string-append #$linux "/System.map") -e"
would make more sense ?

Mathieu




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 27 Feb 2018 10:27:01 GMT) Full text and rfc822 format available.

Message #74 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v4 1/7] gnu: kmod: Split off kmod-minimal.
Date: Tue, 27 Feb 2018 12:26:13 +0100
* gnu/packages/linux.scm (kmod-minimal): New variable.
(kmod): Modify.
---
 gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++----------------
 1 file changed, 33 insertions(+), 16 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 238398e84..1f8bf3050 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1933,8 +1933,35 @@ for systems using the Linux kernel.  This includes commands such as
 to use Linux' inotify mechanism, which allows file accesses to be monitored.")
     (license license:gpl2+)))
 
-(define-public kmod
+(define kmod-minimal
   (package
+    (name "kmod-minimal")
+    (version "13")
+    (source (origin
+              (method url-fetch)
+              (uri
+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+                              "kmod-" version ".tar.xz"))
+              (sha256
+               (base32
+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+              (patches (search-patches "kmod-13-module-directory.patch"))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:tests? #f)) ; FIXME: Investigate test failures
+    (home-page "https://www.kernel.org/")
+    (synopsis "Kernel module tools")
+    (description "Kmod is a set of tools to handle common tasks with Linux
+kernel modules like insert, remove, list, check properties, resolve
+dependencies and aliases.
+
+These tools are designed on top of libkmod, a library that is shipped with
+kmod.  The aim is to be compatible with tools, configurations and indices
+from the module-init-tools project.")
+    (license license:gpl2+))) ; library under lgpl2.1+
+
+(define-public kmod
+  (package (inherit kmod-minimal)
     (name "kmod")
     (version "24")
     (source (origin
@@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
                (base32
                 "15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))
               (patches (search-patches "kmod-module-directory.patch"))))
-    (build-system gnu-build-system)
     (native-inputs
      `(("pkg-config" ,pkg-config)))
     (inputs
      `(("xz" ,xz)
        ("zlib" ,zlib)))
     (arguments
-     `(#:tests? #f ; FIXME: Investigate test failures
-       #:configure-flags '("--with-xz" "--with-zlib")
+     `(#:configure-flags '("--with-xz" "--with-zlib")
+       #:tests? #f ; FIXME: Investigate test failures
        #:phases (alist-cons-after
                  'install 'install-modprobe&co
                  (lambda* (#:key outputs #:allow-other-keys)
@@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
                                  (symlink "kmod"
                                           (string-append bin "/" tool)))
                                '("insmod" "rmmod" "lsmod" "modprobe"
-                                 "modinfo" "depmod"))))
-                 %standard-phases)))
-    (home-page "https://www.kernel.org/")
-    (synopsis "Kernel module tools")
-    (description "Kmod is a set of tools to handle common tasks with Linux
-kernel modules like insert, remove, list, check properties, resolve
-dependencies and aliases.
-
-These tools are designed on top of libkmod, a library that is shipped with
-kmod.  The aim is to be compatible with tools, configurations and indices
-from the module-init-tools project.")
-    (license license:gpl2+))) ; library under lgpl2.1+
+                                 "modinfo" "depmod"))
+                     #t))
+                 %standard-phases)))))
 
 (define-public eudev
   ;; The post-systemd fork, maintained by Gentoo.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 27 Feb 2018 10:27:02 GMT) Full text and rfc822 format available.

Message #77 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v4 2/7] gnu: Add kmod-minimal-static.
Date: Tue, 27 Feb 2018 12:26:14 +0100
* gnu/packages/linux.scm (kmod-minimal/static): New variable.
* gnu/packages/patches/kmod-13-module-directory.patch: New file.
* gnu/local.mk: Add it.
---
 gnu/local.mk                                       |  1 +
 gnu/packages/linux.scm                             | 44 ++++++++++++++++++++++
 .../patches/kmod-13-module-directory.patch         | 33 ++++++++++++++++
 3 files changed, 78 insertions(+)
 create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 21195f8c1..b1e3c878d 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -795,6 +795,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/kiki-makefile.patch			\
   %D%/packages/patches/kiki-missing-includes.patch		\
   %D%/packages/patches/kiki-portability-64bit.patch		\
+  %D%/packages/patches/kmod-13-module-directory.patch		\
   %D%/packages/patches/kmod-module-directory.patch		\
   %D%/packages/patches/kobodeluxe-paths.patch			\
   %D%/packages/patches/kobodeluxe-enemies-pipe-decl.patch	\
diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 1f8bf3050..b2e47f79a 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1994,6 +1994,50 @@ from the module-init-tools project.")
                      #t))
                  %standard-phases)))))
 
+(define-public kmod-minimal/static
+  (static-package
+   (package (inherit kmod-minimal)
+    (name "kmod-minimal-static")
+    (version "13")
+    (source (origin
+              (method url-fetch)
+              (uri
+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+                              "kmod-" version ".tar.xz"))
+              (sha256
+               (base32
+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+              (patches (search-patches "kmod-13-module-directory.patch"))))
+    (arguments
+     (substitute-keyword-arguments
+       (package-arguments (static-package kmod-minimal))
+       ((#:configure-flags flags ''())
+        `(cons* "--disable-manpages" "--disable-static" "--disable-shared" ,flags))
+       ((#:make-flags flags ''())
+        `(cons* "LDFLAGS=-all-static" ,flags))
+       ((#:phases phases '%standard-phases)
+        `(modify-phases ,phases
+          (delete 'install-license-files)
+          (add-after 'unpack 'patch-kmod
+           (lambda _
+             ;; Reduce size by 200 kiB.
+             (substitute* "tools/kmod.c"
+              (("[&]kmod_cmd_compat_lsmod,") "")
+              (("[&]kmod_cmd_compat_rmmod,") "")
+              (("[&]kmod_cmd_compat_insmod,") "")
+              (("[&]kmod_cmd_compat_modinfo,") ""))
+             #t))
+          (replace 'install
+            (lambda* (#:key outputs #:allow-other-keys)
+              (let* ((out (assoc-ref outputs "out"))
+                     (bin (string-append out "/bin")))
+                (install-file "tools/kmod" bin)
+                (for-each
+                 (lambda (tool)
+                   (symlink "kmod" (string-append bin "/" tool)))
+                 '("modprobe" "depmod"))
+                #t))))))))))
+
 (define-public eudev
   ;; The post-systemd fork, maintained by Gentoo.
   (package
diff --git a/gnu/packages/patches/kmod-13-module-directory.patch b/gnu/packages/patches/kmod-13-module-directory.patch
new file mode 100644
index 000000000..5ff2f8a60
--- /dev/null
+++ b/gnu/packages/patches/kmod-13-module-directory.patch
@@ -0,0 +1,33 @@
+This patch changes libkmod so it honors the 'LINUX_MODULE_DIRECTORY'
+environment variable, rather than looking for modules exclusively in
+/lib/modules.
+
+Patch by Shea Levy and Eelco Dolstra, from Nixpkgs; adjusted to
+use 'LINUX_MODULE_DIRECTORY' rather than 'MODULE_DIR' as the variable
+name.
+
+
+--- kmod-7/libkmod/libkmod.c	2012-03-15 08:19:16.750010226 -0400
++++ kmod-7/libkmod/libkmod.c	2012-04-04 15:21:29.532074313 -0400
+@@ -200,7 +200,7 @@
+ static char *get_kernel_release(const char *dirname)
+ {
+ 	struct utsname u;
+-	char *p;
++	char *p, *dirname_prefix;
+ 
+ 	if (dirname != NULL)
+ 		return path_make_absolute_cwd(dirname);
+@@ -208,7 +208,10 @@
+ 	if (uname(&u) < 0)
+ 		return NULL;
+ 
+-	if (asprintf(&p, "%s/%s", dirname_default_prefix, u.release) < 0)
++	if ((dirname_prefix = getenv("LINUX_MODULE_DIRECTORY")) == NULL)
++		dirname_prefix = dirname_default_prefix;
++
++	if (asprintf(&p, "%s/%s", dirname_prefix, u.release) < 0)
+ 		return NULL;
+ 
+ 	return p;
+




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 27 Feb 2018 10:27:02 GMT) Full text and rfc822 format available.

Message #80 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v4 0/7] Load Linux module only when supported hardware is
 present.
Date: Tue, 27 Feb 2018 12:26:12 +0100
Danny Milosavljevic (7):
  gnu: kmod: Split off kmod-minimal.
  gnu: Add kmod-minimal-static.
  linux-initrd: Add kmod.
  linux-boot: Load kernel modules only when the hardware is present.
  vm: Allow qemu-image builder to load Linux kernel modules.
  vm: Make the virtio-blk uniquely identifyable in /sys.
  linux-boot: Call make-static-device-nodes much earlier.

 gnu/build/linux-boot.scm                           | 43 ++++++++--
 gnu/build/linux-initrd.scm                         | 12 ++-
 gnu/build/vm.scm                                   |  2 +-
 gnu/local.mk                                       |  1 +
 gnu/packages/linux.scm                             | 91 ++++++++++++++++++----
 .../patches/kmod-13-module-directory.patch         | 33 ++++++++
 gnu/services/base.scm                              | 11 ---
 gnu/system/linux-initrd.scm                        | 50 +++++++++---
 gnu/system/vm.scm                                  | 34 ++++++--
 9 files changed, 224 insertions(+), 53 deletions(-)
 create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 27 Feb 2018 10:27:02 GMT) Full text and rfc822 format available.

Message #83 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v4 3/7] linux-initrd: Add kmod.
Date: Tue, 27 Feb 2018 12:26:15 +0100
* gnu/system/linux-initrd.scm (raw-initrd): Add kmod.
(base-initrd): Add kmod.
(expression->initrd): Add kmod, linux-module-directory.
(flat-linux-module-directory): Add kmod; invoke depmod.
* gnu/build/linux-initrd.scm (build-initrd): Add kmod, linux-module-directory.
---
 gnu/build/linux-initrd.scm  | 12 +++++++++++-
 gnu/system/linux-initrd.scm | 45 ++++++++++++++++++++++++++++++++++-----------
 2 files changed, 45 insertions(+), 12 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..6356007df 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."
 
 (define* (build-initrd output
                        #:key
-                       guile init
+                       guile init kmod linux-module-directory
                        (references-graphs '())
                        (gzip "gzip"))
   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script
@@ -131,6 +131,16 @@ REFERENCES-GRAPHS."
     (symlink (string-append guile "/bin/guile") "proc/self/exe")
     (readlink "proc/self/exe")
 
+    ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+    (when kmod
+      (mkdir-p "sbin")
+      (symlink (string-append kmod "/bin/modprobe") "sbin/modprobe"))
+
+    ;; Make modules available as /lib/modules so modprobe finds them.
+    (mkdir-p "lib")
+    (symlink (string-append linux-module-directory "/lib/modules")
+             "lib/modules")
+
     ;; Reset the timestamps of all the files that will make it in the initrd.
     (for-each (lambda (file)
                 (unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 7170d1c0e..46ef055f0 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -59,6 +59,8 @@
                              #:key
                              (guile %guile-static-stripped)
                              (gzip gzip)
+                             kmod
+                             linux-module-directory
                              (name "guile-initrd")
                              (system (%current-system)))
   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -94,6 +96,8 @@ the derivations referenced by EXP are automatically copied to the initrd."
           (build-initrd (string-append #$output "/initrd")
                         #:guile #$guile
                         #:init #$init
+                        #:kmod #$kmod
+                        #:linux-module-directory #$linux-module-directory
                         ;; Copy everything INIT refers to into the initrd.
                         #:references-graphs '("closure")
                         #:gzip (string-append #$gzip "/bin/gzip")))))
@@ -101,7 +105,7 @@ the derivations referenced by EXP are automatically copied to the initrd."
   (gexp->derivation name builder
                     #:references-graphs `(("closure" ,init))))
 
-(define (flat-linux-module-directory linux modules)
+(define (flat-linux-module-directory linux modules kmod)
   "Return a flat directory containing the Linux kernel modules listed in
 MODULES and taken from LINUX."
   (define build-exp
@@ -109,7 +113,7 @@ MODULES and taken from LINUX."
                             '((guix build utils)
                               (gnu build linux-modules)))
       #~(begin
-          (use-modules (ice-9 match) (ice-9 regex)
+          (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)
                        (srfi srfi-1)
                        (guix build utils)
                        (gnu build linux-modules))
@@ -138,13 +142,27 @@ MODULES and taken from LINUX."
                       (recursive-module-dependencies modules
                                                      #:lookup-module lookup))))
 
-          (mkdir #$output)
-          (for-each (lambda (module)
-                      (format #t "copying '~a'...~%" module)
-                      (copy-file module
-                                 (string-append #$output "/"
-                                                (basename module))))
-                    (delete-duplicates modules)))))
+          (define version
+            (match
+             (filter
+              (lambda (name)
+                (not (string-prefix? "." name)))
+              (scandir module-dir))
+             ((item) item)))
+
+          (let ((output (string-append #$output "/lib/modules/" version)))
+            (mkdir-p output)
+            (for-each (lambda (module)
+                        (format #t "copying '~a'...~%" module)
+                        (copy-file module
+                                   (string-append output "/"
+                                                  (basename module))))
+                      (delete-duplicates modules)))
+          (invoke (string-append #$kmod "/bin/depmod") "-a" "-b" #$output
+                  "-e"
+                  "-F" (string-append #$linux "/System.map")
+                  version)
+          #t)))
 
   (computed-file "linux-modules" build-exp))
 
@@ -152,6 +170,7 @@ MODULES and taken from LINUX."
                       #:key
                       (linux linux-libre)
                       (linux-modules '())
+                      (kmod kmod-minimal/static)
                       (mapped-devices '())
                       (helper-packages '())
                       qemu-networking?
@@ -185,7 +204,7 @@ upon error."
          mapped-devices))
 
   (define kodir
-    (flat-linux-module-directory linux linux-modules))
+    (flat-linux-module-directory linux linux-modules kmod))
 
   (expression->initrd
    (with-imported-modules (source-module-closure
@@ -223,6 +242,8 @@ upon error."
                       #:qemu-guest-networking? #$qemu-networking?
                       #:volatile-root? '#$volatile-root?
                       #:on-error '#$on-error)))
+   #:kmod kmod
+   #:linux-module-directory kodir
    #:name "raw-initrd"))
 
 (define* (file-system-packages file-systems #:key (volatile-root? #f))
@@ -245,6 +266,7 @@ FILE-SYSTEMS."
 (define* (base-initrd file-systems
                       #:key
                       (linux linux-libre)
+                      (kmod kmod-minimal/static)
                       (mapped-devices '())
                       qemu-networking?
                       volatile-root?
@@ -322,8 +344,9 @@ loaded at boot time in the order in which they appear."
   (raw-initrd file-systems
               #:linux linux
               #:linux-modules linux-modules
+              #:kmod kmod
               #:mapped-devices mapped-devices
-              #:helper-packages helper-packages
+              #:helper-packages (cons kmod helper-packages)
               #:qemu-networking? qemu-networking?
               #:volatile-root? volatile-root?
               #:on-error on-error))




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 27 Feb 2018 10:27:03 GMT) Full text and rfc822 format available.

Message #86 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v4 4/7] linux-boot: Load kernel modules only when the hardware
 is present.
Date: Tue, 27 Feb 2018 12:26:16 +0100
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
 gnu/build/linux-boot.scm    | 31 +++++++++++++++++++++++++++----
 gnu/system/linux-initrd.scm |  4 +++-
 2 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..6d00ea9be 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -473,6 +473,32 @@ upon error."
     (string-append linux-module-directory "/"
                    (ensure-dot-ko name)))
 
+  (define (load-kernel-modules)
+    "Examine /sys/devices to find out which modules to load and load them."
+    (define enter?
+      (const #t))
+    (define (down! path stat result)
+     ;; Note: modprobe mutates the tree starting with path.
+     (let ((modalias-name (string-append path "/modalias")))
+       (if (file-exists? modalias-name)
+           (let ((modalias
+                 (string-trim-right (call-with-input-file modalias-name
+                                                          read-string)
+                                    #\newline)))
+             (system* "/sbin/modprobe" "-q" "--" modalias))))
+       #t)
+    (define up
+      (const #t))
+    (define skip
+      (const #t))
+    (define leaf
+      (const #t))
+    (define (error name stat errno result)
+      (format (current-error-port) "warning: ~a: ~a~%"
+              name (strerror errno))
+      result)
+    (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))
+
   (display "Welcome, this is GNU's early boot Guile.\n")
   (display "Use '--repl' for an initrd REPL.\n\n")
 
@@ -486,10 +512,7 @@ upon error."
        (when (member "--repl" args)
          (start-repl))
 
-       (display "loading kernel modules...\n")
-       (for-each (cut load-linux-module* <>
-                      #:lookup-module lookup-module)
-                 (map lookup-module linux-modules))
+       (load-kernel-modules)
 
        (when qemu-guest-networking?
          (unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 46ef055f0..c8a9e4950 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -225,7 +225,9 @@ upon error."
                       ;; this info via gexps.
                       ((gnu build file-systems)
                        #:select (find-partition-by-luks-uuid))
-                      (rnrs bytevectors))
+                      (rnrs bytevectors)
+                      (ice-9 ftw)
+                      (ice-9 rdelim))
 
          (with-output-to-port (%make-void-port "w")
            (lambda ()




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 27 Feb 2018 10:27:03 GMT) Full text and rfc822 format available.

Message #89 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v4 5/7] vm: Allow qemu-image builder to load Linux kernel
 modules.
Date: Tue, 27 Feb 2018 12:26:17 +0100
* gnu/system/vm.scm (%modprobe-wrapper): New variable.
(qemu-image): Modify.
---
 gnu/system/vm.scm | 31 +++++++++++++++++++++++++++----
 1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 345cecedd..b5a559012 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."
    #:single-file-output? #t
    #:references-graphs inputs))
 
+(define (%modprobe-wrapper modprobe linux-module-directory)
+  ;; Wrapper for the 'modprobe' command that knows where modules live.
+  ;;
+  ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',
+  ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'
+  ;; environment variable is not set---hence the need for this wrapper.
+  (program-file "modprobe"
+    #~(begin
+        (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)
+        (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))
+
 (define* (qemu-image #:key
                      (name "qemu-image")
                      (system (%current-system))
@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy
 all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true,
 register INPUTS in the store database of the image so that Guix can be used in
 the image."
+  (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))
+        (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))
   (expression->derivation-in-linux-vm
    name
-   (with-imported-modules (source-module-closure '((gnu build bootloader)
+   (with-imported-modules (source-module-closure '((gnu build activation)
+                                                   (gnu build bootloader)
                                                    (gnu build vm)
                                                    (guix build utils)))
      #~(begin
-         (use-modules (gnu build bootloader)
+         (use-modules (gnu build activation)
+                      (gnu build bootloader)
                       (gnu build vm)
                       (guix build utils)
                       (srfi srfi-26)
                       (ice-9 binary-ports))
 
          (let ((inputs
-                '#$(append (list qemu parted e2fsprogs dosfstools)
+                '#$(append (list qemu parted e2fsprogs dosfstools kmod)
                            (map canonical-package
                                 (list sed grep coreutils findutils gawk))
                            (if register-closures? (list guix) '())))
@@ -302,6 +317,14 @@ the image."
                         inputs)))
 
            (set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+           ;; It's possible that we need to load nls modules in order to
+           ;; mount the new partition.
+           (if (file-exists? #$modprobe-name)
+               (activate-modprobe #$(%modprobe-wrapper modprobe-name
+                                     linux-module-directory))
+               (format (current-error-port)
+                "WARNING: No modprobe found in ~s.  \
+Loading kernel modules will be impossible.\n" #$modprobe-name))
 
            (let* ((graphs     '#$(match inputs
                                    (((names . _) ...)
@@ -364,7 +387,7 @@ the image."
    #:make-disk-image? #t
    #:disk-image-size disk-image-size
    #:disk-image-format disk-image-format
-   #:references-graphs inputs))
+   #:references-graphs inputs)))
 
 
 ;;;




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 27 Feb 2018 10:27:04 GMT) Full text and rfc822 format available.

Message #92 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v4 6/7] vm: Make the virtio-blk uniquely identifyable in /sys.
Date: Tue, 27 Feb 2018 12:26:18 +0100
* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.
---
 gnu/build/vm.scm  | 2 +-
 gnu/system/vm.scm | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..ebf9e9f6e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."
                                            builder)
                   (append
                    (if make-disk-image?
-                       `("-device" "virtio-blk,drive=myhd"
+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
                          "-drive" ,(string-append "if=none,file=" output
                                                   ",format=" disk-image-format
                                                   ",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index b5a559012..fdff64ed9 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."
 
      #$@(map virtfs-option shared-fs)
      "-vga std"
-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
              #$image)))
 
 (define* (system-qemu-image/shared-store-script os




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 27 Feb 2018 10:27:04 GMT) Full text and rfc822 format available.

Message #95 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v4 7/7] linux-boot: Call make-static-device-nodes much earlier.
Date: Tue, 27 Feb 2018 12:26:19 +0100
* gnu/system/linux-initrd.scm (expression->initrd): Store data files for
make-static-device-nodes.
* gnu/build/linux-boot.scm (make-static-device-nodes): Unexport.
(boot-system): Call make-static-device-nodes.  Delete lookup-module.
* gnu/services/base.scm (udev-shepherd-service): Delete
make-static-device-nodes call.
---
 gnu/build/linux-boot.scm    | 14 +++++++++-----
 gnu/services/base.scm       | 11 -----------
 gnu/system/linux-initrd.scm |  3 ++-
 3 files changed, 11 insertions(+), 17 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 6d00ea9be..f0ac755f8 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -39,7 +39,6 @@
             find-long-option
             find-long-options
             make-essential-device-nodes
-            make-static-device-nodes
             configure-qemu-networking
 
             device-number
@@ -469,10 +468,6 @@ upon error."
              mounts)
         "ext4"))
 
-  (define (lookup-module name)
-    (string-append linux-module-directory "/"
-                   (ensure-dot-ko name)))
-
   (define (load-kernel-modules)
     "Examine /sys/devices to find out which modules to load and load them."
     (define enter?
@@ -512,6 +507,15 @@ upon error."
        (when (member "--repl" args)
          (start-repl))
 
+       (let* ((kernel-release
+               (utsname:release (uname)))
+              (directory
+               (string-append linux-module-directory "/lib/modules/"
+                              kernel-release))
+              (old-umask (umask #o022)))
+         (make-static-device-nodes directory)
+         (umask old-umask))
+
        (load-kernel-modules)
 
        (when qemu-guest-networking?
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index 69e211ffa..0cba1c66f 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -1857,17 +1857,6 @@ item of @var{packages}."
                     (setenv "EUDEV_RULES_DIRECTORY"
                             #$(file-append rules "/lib/udev/rules.d"))
 
-                    (let* ((kernel-release
-                            (utsname:release (uname)))
-                           (linux-module-directory
-                            (getenv "LINUX_MODULE_DIRECTORY"))
-                           (directory
-                            (string-append linux-module-directory "/"
-                                           kernel-release))
-                           (old-umask (umask #o022)))
-                      (make-static-device-nodes directory)
-                      (umask old-umask))
-
                     (let ((pid (primitive-fork)))
                       (case pid
                         ((0)




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 27 Feb 2018 14:27:02 GMT) Full text and rfc822 format available.

Message #98 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v3 4/6] linux-boot: Load kernel modules only
 when the hardware is present.
Date: Tue, 27 Feb 2018 15:26:06 +0100
Hello,

Nice patch series!

Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

> * gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
> the hardware is present.
> * gnu/system/linux-initrd.scm (raw-initrd): Add imports.

[...]

> +  (define (load-kernel-modules)
> +    "Examine /sys/devices to find out which modules to load and load them."
> +    (define enter?
> +      (const #t))
> +    (define (down! path stat result)
> +     ;; Note: modprobe mutates the tree starting with path.
> +     (let ((modalias-name (string-append path "/modalias")))
> +       (if (file-exists? modalias-name)
> +           (let ((modalias
> +                 (string-trim-right (call-with-input-file modalias-name
> +                                                          read-string)
> +                                    #\newline)))
> +             (system* "/sbin/modprobe" "-q" "--" modalias))))

Can we build upon (gnu build linux-modules) to achieve this?

Hopefully the tools at
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30629#14> can help a bit.

Thanks,
Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 27 Feb 2018 14:51:01 GMT) Full text and rfc822 format available.

Message #101 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v5 1/7] gnu: kmod: Split off kmod-minimal.
Date: Tue, 27 Feb 2018 16:50:45 +0100
* gnu/packages/linux.scm (kmod-minimal): New variable.
(kmod): Modify.
---
 gnu/packages/linux.scm | 49 +++++++++++++++++++++++++++++++++----------------
 1 file changed, 33 insertions(+), 16 deletions(-)

diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 238398e84..1f8bf3050 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1933,8 +1933,35 @@ for systems using the Linux kernel.  This includes commands such as
 to use Linux' inotify mechanism, which allows file accesses to be monitored.")
     (license license:gpl2+)))
 
-(define-public kmod
+(define kmod-minimal
   (package
+    (name "kmod-minimal")
+    (version "13")
+    (source (origin
+              (method url-fetch)
+              (uri
+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+                              "kmod-" version ".tar.xz"))
+              (sha256
+               (base32
+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+              (patches (search-patches "kmod-13-module-directory.patch"))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:tests? #f)) ; FIXME: Investigate test failures
+    (home-page "https://www.kernel.org/")
+    (synopsis "Kernel module tools")
+    (description "Kmod is a set of tools to handle common tasks with Linux
+kernel modules like insert, remove, list, check properties, resolve
+dependencies and aliases.
+
+These tools are designed on top of libkmod, a library that is shipped with
+kmod.  The aim is to be compatible with tools, configurations and indices
+from the module-init-tools project.")
+    (license license:gpl2+))) ; library under lgpl2.1+
+
+(define-public kmod
+  (package (inherit kmod-minimal)
     (name "kmod")
     (version "24")
     (source (origin
@@ -1946,15 +1973,14 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
                (base32
                 "15xkkkzvca9flvkm48gkh8y8f13vlm3sl7nz9ydc7b3jy4fqs2v1"))
               (patches (search-patches "kmod-module-directory.patch"))))
-    (build-system gnu-build-system)
     (native-inputs
      `(("pkg-config" ,pkg-config)))
     (inputs
      `(("xz" ,xz)
        ("zlib" ,zlib)))
     (arguments
-     `(#:tests? #f ; FIXME: Investigate test failures
-       #:configure-flags '("--with-xz" "--with-zlib")
+     `(#:configure-flags '("--with-xz" "--with-zlib")
+       #:tests? #f ; FIXME: Investigate test failures
        #:phases (alist-cons-after
                  'install 'install-modprobe&co
                  (lambda* (#:key outputs #:allow-other-keys)
@@ -1964,18 +1990,9 @@ to use Linux' inotify mechanism, which allows file accesses to be monitored.")
                                  (symlink "kmod"
                                           (string-append bin "/" tool)))
                                '("insmod" "rmmod" "lsmod" "modprobe"
-                                 "modinfo" "depmod"))))
-                 %standard-phases)))
-    (home-page "https://www.kernel.org/")
-    (synopsis "Kernel module tools")
-    (description "Kmod is a set of tools to handle common tasks with Linux
-kernel modules like insert, remove, list, check properties, resolve
-dependencies and aliases.
-
-These tools are designed on top of libkmod, a library that is shipped with
-kmod.  The aim is to be compatible with tools, configurations and indices
-from the module-init-tools project.")
-    (license license:gpl2+))) ; library under lgpl2.1+
+                                 "modinfo" "depmod"))
+                     #t))
+                 %standard-phases)))))
 
 (define-public eudev
   ;; The post-systemd fork, maintained by Gentoo.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 27 Feb 2018 14:51:02 GMT) Full text and rfc822 format available.

Message #104 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v5 0/7] Load Linux module only when supported hardware is
 present.
Date: Tue, 27 Feb 2018 16:50:44 +0100
Danny Milosavljevic (7):
  gnu: kmod: Split off kmod-minimal.
  gnu: Add kmod-minimal-static.
  linux-initrd: Add kmod.
  linux-boot: Load kernel modules only when the hardware is present.
  vm: Allow qemu-image builder to load Linux kernel modules.
  vm: Make the virtio-blk is uniquely identifyable in /sys.
  linux-boot: Call make-static-device-nodes much earlier.

 gnu/build/linux-boot.scm                           | 42 ++++++++--
 gnu/build/linux-initrd.scm                         | 12 ++-
 gnu/build/vm.scm                                   |  2 +-
 gnu/local.mk                                       |  1 +
 gnu/packages/linux.scm                             | 91 ++++++++++++++++++----
 .../patches/kmod-13-module-directory.patch         | 33 ++++++++
 gnu/system/linux-initrd.scm                        | 50 +++++++++---
 gnu/system/vm.scm                                  | 34 ++++++--
 8 files changed, 224 insertions(+), 41 deletions(-)
 create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 27 Feb 2018 14:51:02 GMT) Full text and rfc822 format available.

Message #107 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v5 2/7] gnu: Add kmod-minimal-static.
Date: Tue, 27 Feb 2018 16:50:46 +0100
* gnu/packages/linux.scm (kmod-minimal/static): New variable.
* gnu/packages/patches/kmod-13-module-directory.patch: New file.
* gnu/local.mk: Add it.
---
 gnu/local.mk                                       |  1 +
 gnu/packages/linux.scm                             | 44 ++++++++++++++++++++++
 .../patches/kmod-13-module-directory.patch         | 33 ++++++++++++++++
 3 files changed, 78 insertions(+)
 create mode 100644 gnu/packages/patches/kmod-13-module-directory.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 21195f8c1..b1e3c878d 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -795,6 +795,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/kiki-makefile.patch			\
   %D%/packages/patches/kiki-missing-includes.patch		\
   %D%/packages/patches/kiki-portability-64bit.patch		\
+  %D%/packages/patches/kmod-13-module-directory.patch		\
   %D%/packages/patches/kmod-module-directory.patch		\
   %D%/packages/patches/kobodeluxe-paths.patch			\
   %D%/packages/patches/kobodeluxe-enemies-pipe-decl.patch	\
diff --git a/gnu/packages/linux.scm b/gnu/packages/linux.scm
index 1f8bf3050..b2e47f79a 100644
--- a/gnu/packages/linux.scm
+++ b/gnu/packages/linux.scm
@@ -1994,6 +1994,50 @@ from the module-init-tools project.")
                      #t))
                  %standard-phases)))))
 
+(define-public kmod-minimal/static
+  (static-package
+   (package (inherit kmod-minimal)
+    (name "kmod-minimal-static")
+    (version "13")
+    (source (origin
+              (method url-fetch)
+              (uri
+               (string-append "mirror://kernel.org/linux/utils/kernel/kmod/"
+                              "kmod-" version ".tar.xz"))
+              (sha256
+               (base32
+                "0mkrklih0f33c3zc4mkk9qqbzy36r18mj9xffd4wi61gpamx6dkc"))
+              (patches (search-patches "kmod-13-module-directory.patch"))))
+    (arguments
+     (substitute-keyword-arguments
+       (package-arguments (static-package kmod-minimal))
+       ((#:configure-flags flags ''())
+        `(cons* "--disable-manpages" "--disable-static" "--disable-shared" ,flags))
+       ((#:make-flags flags ''())
+        `(cons* "LDFLAGS=-all-static" ,flags))
+       ((#:phases phases '%standard-phases)
+        `(modify-phases ,phases
+          (delete 'install-license-files)
+          (add-after 'unpack 'patch-kmod
+           (lambda _
+             ;; Reduce size by 200 kiB.
+             (substitute* "tools/kmod.c"
+              (("[&]kmod_cmd_compat_lsmod,") "")
+              (("[&]kmod_cmd_compat_rmmod,") "")
+              (("[&]kmod_cmd_compat_insmod,") "")
+              (("[&]kmod_cmd_compat_modinfo,") ""))
+             #t))
+          (replace 'install
+            (lambda* (#:key outputs #:allow-other-keys)
+              (let* ((out (assoc-ref outputs "out"))
+                     (bin (string-append out "/bin")))
+                (install-file "tools/kmod" bin)
+                (for-each
+                 (lambda (tool)
+                   (symlink "kmod" (string-append bin "/" tool)))
+                 '("modprobe" "depmod"))
+                #t))))))))))
+
 (define-public eudev
   ;; The post-systemd fork, maintained by Gentoo.
   (package
diff --git a/gnu/packages/patches/kmod-13-module-directory.patch b/gnu/packages/patches/kmod-13-module-directory.patch
new file mode 100644
index 000000000..5ff2f8a60
--- /dev/null
+++ b/gnu/packages/patches/kmod-13-module-directory.patch
@@ -0,0 +1,33 @@
+This patch changes libkmod so it honors the 'LINUX_MODULE_DIRECTORY'
+environment variable, rather than looking for modules exclusively in
+/lib/modules.
+
+Patch by Shea Levy and Eelco Dolstra, from Nixpkgs; adjusted to
+use 'LINUX_MODULE_DIRECTORY' rather than 'MODULE_DIR' as the variable
+name.
+
+
+--- kmod-7/libkmod/libkmod.c	2012-03-15 08:19:16.750010226 -0400
++++ kmod-7/libkmod/libkmod.c	2012-04-04 15:21:29.532074313 -0400
+@@ -200,7 +200,7 @@
+ static char *get_kernel_release(const char *dirname)
+ {
+ 	struct utsname u;
+-	char *p;
++	char *p, *dirname_prefix;
+ 
+ 	if (dirname != NULL)
+ 		return path_make_absolute_cwd(dirname);
+@@ -208,7 +208,10 @@
+ 	if (uname(&u) < 0)
+ 		return NULL;
+ 
+-	if (asprintf(&p, "%s/%s", dirname_default_prefix, u.release) < 0)
++	if ((dirname_prefix = getenv("LINUX_MODULE_DIRECTORY")) == NULL)
++		dirname_prefix = dirname_default_prefix;
++
++	if (asprintf(&p, "%s/%s", dirname_prefix, u.release) < 0)
+ 		return NULL;
+ 
+ 	return p;
+




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 27 Feb 2018 14:51:03 GMT) Full text and rfc822 format available.

Message #110 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v5 3/7] linux-initrd: Add kmod.
Date: Tue, 27 Feb 2018 16:50:47 +0100
* gnu/system/linux-initrd.scm (raw-initrd): Add kmod.
(base-initrd): Add kmod.
(expression->initrd): Add kmod, linux-module-directory.
(flat-linux-module-directory): Add kmod; invoke depmod.
* gnu/build/linux-initrd.scm (build-initrd): Add kmod, linux-module-directory.
---
 gnu/build/linux-initrd.scm  | 12 +++++++++++-
 gnu/system/linux-initrd.scm | 45 ++++++++++++++++++++++++++++++++++-----------
 2 files changed, 45 insertions(+), 12 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..6356007df 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."
 
 (define* (build-initrd output
                        #:key
-                       guile init
+                       guile init kmod linux-module-directory
                        (references-graphs '())
                        (gzip "gzip"))
   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script
@@ -131,6 +131,16 @@ REFERENCES-GRAPHS."
     (symlink (string-append guile "/bin/guile") "proc/self/exe")
     (readlink "proc/self/exe")
 
+    ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+    (when kmod
+      (mkdir-p "sbin")
+      (symlink (string-append kmod "/bin/modprobe") "sbin/modprobe"))
+
+    ;; Make modules available as /lib/modules so modprobe finds them.
+    (mkdir-p "lib")
+    (symlink (string-append linux-module-directory "/lib/modules")
+             "lib/modules")
+
     ;; Reset the timestamps of all the files that will make it in the initrd.
     (for-each (lambda (file)
                 (unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 7170d1c0e..46ef055f0 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -59,6 +59,8 @@
                              #:key
                              (guile %guile-static-stripped)
                              (gzip gzip)
+                             kmod
+                             linux-module-directory
                              (name "guile-initrd")
                              (system (%current-system)))
   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -94,6 +96,8 @@ the derivations referenced by EXP are automatically copied to the initrd."
           (build-initrd (string-append #$output "/initrd")
                         #:guile #$guile
                         #:init #$init
+                        #:kmod #$kmod
+                        #:linux-module-directory #$linux-module-directory
                         ;; Copy everything INIT refers to into the initrd.
                         #:references-graphs '("closure")
                         #:gzip (string-append #$gzip "/bin/gzip")))))
@@ -101,7 +105,7 @@ the derivations referenced by EXP are automatically copied to the initrd."
   (gexp->derivation name builder
                     #:references-graphs `(("closure" ,init))))
 
-(define (flat-linux-module-directory linux modules)
+(define (flat-linux-module-directory linux modules kmod)
   "Return a flat directory containing the Linux kernel modules listed in
 MODULES and taken from LINUX."
   (define build-exp
@@ -109,7 +113,7 @@ MODULES and taken from LINUX."
                             '((guix build utils)
                               (gnu build linux-modules)))
       #~(begin
-          (use-modules (ice-9 match) (ice-9 regex)
+          (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)
                        (srfi srfi-1)
                        (guix build utils)
                        (gnu build linux-modules))
@@ -138,13 +142,27 @@ MODULES and taken from LINUX."
                       (recursive-module-dependencies modules
                                                      #:lookup-module lookup))))
 
-          (mkdir #$output)
-          (for-each (lambda (module)
-                      (format #t "copying '~a'...~%" module)
-                      (copy-file module
-                                 (string-append #$output "/"
-                                                (basename module))))
-                    (delete-duplicates modules)))))
+          (define version
+            (match
+             (filter
+              (lambda (name)
+                (not (string-prefix? "." name)))
+              (scandir module-dir))
+             ((item) item)))
+
+          (let ((output (string-append #$output "/lib/modules/" version)))
+            (mkdir-p output)
+            (for-each (lambda (module)
+                        (format #t "copying '~a'...~%" module)
+                        (copy-file module
+                                   (string-append output "/"
+                                                  (basename module))))
+                      (delete-duplicates modules)))
+          (invoke (string-append #$kmod "/bin/depmod") "-a" "-b" #$output
+                  "-e"
+                  "-F" (string-append #$linux "/System.map")
+                  version)
+          #t)))
 
   (computed-file "linux-modules" build-exp))
 
@@ -152,6 +170,7 @@ MODULES and taken from LINUX."
                       #:key
                       (linux linux-libre)
                       (linux-modules '())
+                      (kmod kmod-minimal/static)
                       (mapped-devices '())
                       (helper-packages '())
                       qemu-networking?
@@ -185,7 +204,7 @@ upon error."
          mapped-devices))
 
   (define kodir
-    (flat-linux-module-directory linux linux-modules))
+    (flat-linux-module-directory linux linux-modules kmod))
 
   (expression->initrd
    (with-imported-modules (source-module-closure
@@ -223,6 +242,8 @@ upon error."
                       #:qemu-guest-networking? #$qemu-networking?
                       #:volatile-root? '#$volatile-root?
                       #:on-error '#$on-error)))
+   #:kmod kmod
+   #:linux-module-directory kodir
    #:name "raw-initrd"))
 
 (define* (file-system-packages file-systems #:key (volatile-root? #f))
@@ -245,6 +266,7 @@ FILE-SYSTEMS."
 (define* (base-initrd file-systems
                       #:key
                       (linux linux-libre)
+                      (kmod kmod-minimal/static)
                       (mapped-devices '())
                       qemu-networking?
                       volatile-root?
@@ -322,8 +344,9 @@ loaded at boot time in the order in which they appear."
   (raw-initrd file-systems
               #:linux linux
               #:linux-modules linux-modules
+              #:kmod kmod
               #:mapped-devices mapped-devices
-              #:helper-packages helper-packages
+              #:helper-packages (cons kmod helper-packages)
               #:qemu-networking? qemu-networking?
               #:volatile-root? volatile-root?
               #:on-error on-error))




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 27 Feb 2018 14:51:03 GMT) Full text and rfc822 format available.

Message #113 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v5 4/7] linux-boot: Load kernel modules only when the hardware
 is present.
Date: Tue, 27 Feb 2018 16:50:48 +0100
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
 gnu/build/linux-boot.scm    | 31 +++++++++++++++++++++++++++----
 gnu/system/linux-initrd.scm |  4 +++-
 2 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..6d00ea9be 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -473,6 +473,32 @@ upon error."
     (string-append linux-module-directory "/"
                    (ensure-dot-ko name)))
 
+  (define (load-kernel-modules)
+    "Examine /sys/devices to find out which modules to load and load them."
+    (define enter?
+      (const #t))
+    (define (down! path stat result)
+     ;; Note: modprobe mutates the tree starting with path.
+     (let ((modalias-name (string-append path "/modalias")))
+       (if (file-exists? modalias-name)
+           (let ((modalias
+                 (string-trim-right (call-with-input-file modalias-name
+                                                          read-string)
+                                    #\newline)))
+             (system* "/sbin/modprobe" "-q" "--" modalias))))
+       #t)
+    (define up
+      (const #t))
+    (define skip
+      (const #t))
+    (define leaf
+      (const #t))
+    (define (error name stat errno result)
+      (format (current-error-port) "warning: ~a: ~a~%"
+              name (strerror errno))
+      result)
+    (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))
+
   (display "Welcome, this is GNU's early boot Guile.\n")
   (display "Use '--repl' for an initrd REPL.\n\n")
 
@@ -486,10 +512,7 @@ upon error."
        (when (member "--repl" args)
          (start-repl))
 
-       (display "loading kernel modules...\n")
-       (for-each (cut load-linux-module* <>
-                      #:lookup-module lookup-module)
-                 (map lookup-module linux-modules))
+       (load-kernel-modules)
 
        (when qemu-guest-networking?
          (unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 46ef055f0..c8a9e4950 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -225,7 +225,9 @@ upon error."
                       ;; this info via gexps.
                       ((gnu build file-systems)
                        #:select (find-partition-by-luks-uuid))
-                      (rnrs bytevectors))
+                      (rnrs bytevectors)
+                      (ice-9 ftw)
+                      (ice-9 rdelim))
 
          (with-output-to-port (%make-void-port "w")
            (lambda ()




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 27 Feb 2018 14:52:01 GMT) Full text and rfc822 format available.

Message #116 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v5 5/7] vm: Allow qemu-image builder to load Linux kernel
 modules.
Date: Tue, 27 Feb 2018 16:50:49 +0100
* gnu/system/vm.scm (%modprobe-wrapper): New variable.
(qemu-image): Modify.
---
 gnu/system/vm.scm | 31 +++++++++++++++++++++++++++----
 1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 345cecedd..b5a559012 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."
    #:single-file-output? #t
    #:references-graphs inputs))
 
+(define (%modprobe-wrapper modprobe linux-module-directory)
+  ;; Wrapper for the 'modprobe' command that knows where modules live.
+  ;;
+  ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',
+  ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'
+  ;; environment variable is not set---hence the need for this wrapper.
+  (program-file "modprobe"
+    #~(begin
+        (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)
+        (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))
+
 (define* (qemu-image #:key
                      (name "qemu-image")
                      (system (%current-system))
@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy
 all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true,
 register INPUTS in the store database of the image so that Guix can be used in
 the image."
+  (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))
+        (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))
   (expression->derivation-in-linux-vm
    name
-   (with-imported-modules (source-module-closure '((gnu build bootloader)
+   (with-imported-modules (source-module-closure '((gnu build activation)
+                                                   (gnu build bootloader)
                                                    (gnu build vm)
                                                    (guix build utils)))
      #~(begin
-         (use-modules (gnu build bootloader)
+         (use-modules (gnu build activation)
+                      (gnu build bootloader)
                       (gnu build vm)
                       (guix build utils)
                       (srfi srfi-26)
                       (ice-9 binary-ports))
 
          (let ((inputs
-                '#$(append (list qemu parted e2fsprogs dosfstools)
+                '#$(append (list qemu parted e2fsprogs dosfstools kmod)
                            (map canonical-package
                                 (list sed grep coreutils findutils gawk))
                            (if register-closures? (list guix) '())))
@@ -302,6 +317,14 @@ the image."
                         inputs)))
 
            (set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+           ;; It's possible that we need to load nls modules in order to
+           ;; mount the new partition.
+           (if (file-exists? #$modprobe-name)
+               (activate-modprobe #$(%modprobe-wrapper modprobe-name
+                                     linux-module-directory))
+               (format (current-error-port)
+                "WARNING: No modprobe found in ~s.  \
+Loading kernel modules will be impossible.\n" #$modprobe-name))
 
            (let* ((graphs     '#$(match inputs
                                    (((names . _) ...)
@@ -364,7 +387,7 @@ the image."
    #:make-disk-image? #t
    #:disk-image-size disk-image-size
    #:disk-image-format disk-image-format
-   #:references-graphs inputs))
+   #:references-graphs inputs)))
 
 
 ;;;




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 27 Feb 2018 14:52:01 GMT) Full text and rfc822 format available.

Message #119 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v5 6/7] vm: Make the virtio-blk is uniquely identifyable in
 /sys.
Date: Tue, 27 Feb 2018 16:50:50 +0100
* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.
---
 gnu/build/vm.scm  | 2 +-
 gnu/system/vm.scm | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..ebf9e9f6e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."
                                            builder)
                   (append
                    (if make-disk-image?
-                       `("-device" "virtio-blk,drive=myhd"
+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
                          "-drive" ,(string-append "if=none,file=" output
                                                   ",format=" disk-image-format
                                                   ",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index b5a559012..fdff64ed9 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."
 
      #$@(map virtfs-option shared-fs)
      "-vga std"
-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
              #$image)))
 
 (define* (system-qemu-image/shared-store-script os




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 27 Feb 2018 14:52:02 GMT) Full text and rfc822 format available.

Message #122 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v5 7/7] linux-boot: Call make-static-device-nodes much earlier.
Date: Tue, 27 Feb 2018 16:50:51 +0100
* gnu/system/linux-initrd.scm (expression->initrd): Store data files for
make-static-device-nodes.
* gnu/build/linux-boot.scm (boot-system): Call make-static-device-nodes.
Delete lookup-module.
---
 gnu/build/linux-boot.scm    | 13 +++++++++----
 gnu/system/linux-initrd.scm |  3 ++-
 2 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 6d00ea9be..1b16f267a 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -469,10 +469,6 @@ upon error."
              mounts)
         "ext4"))
 
-  (define (lookup-module name)
-    (string-append linux-module-directory "/"
-                   (ensure-dot-ko name)))
-
   (define (load-kernel-modules)
     "Examine /sys/devices to find out which modules to load and load them."
     (define enter?
@@ -512,6 +508,15 @@ upon error."
        (when (member "--repl" args)
          (start-repl))
 
+       (let* ((kernel-release
+               (utsname:release (uname)))
+              (directory
+               (string-append linux-module-directory "/lib/modules/"
+                              kernel-release))
+              (old-umask (umask #o022)))
+         (make-static-device-nodes directory)
+         (umask old-umask))
+
        (load-kernel-modules)
 
        (when qemu-guest-networking?




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 02 Mar 2018 14:17:02 GMT) Full text and rfc822 format available.

Message #125 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v6 0/6] Load Linux module only when supported hardware is
 present.
Date: Fri,  2 Mar 2018 15:16:06 +0100
Danny Milosavljevic (6):
  linux-modules: Add module-aliases.
  linux-modules: Add install-modules.
  linux-boot: Load kernel modules only when the hardware is present.
  vm: Allow qemu-image builder to load Linux kernel modules.
  vm: Make the virtio-blk uniquely identifiable in /sys.
  linux-initrd: Provide modprobe to the initrd.

 gnu/build/linux-boot.scm    | 42 ++++++++++++++++++----
 gnu/build/linux-initrd.scm  | 13 ++++++-
 gnu/build/linux-modules.scm | 61 +++++++++++++++++++++++++++++++
 gnu/build/vm.scm            |  2 +-
 gnu/system/linux-initrd.scm | 88 +++++++++++++++++++++++++++++++++++++++------
 gnu/system/vm.scm           | 34 +++++++++++++++---
 6 files changed, 216 insertions(+), 24 deletions(-)





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 02 Mar 2018 14:18:01 GMT) Full text and rfc822 format available.

Message #128 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v6 1/6] linux-modules: Add module-aliases.
Date: Fri,  2 Mar 2018 15:17:15 +0100
* gnu/build/linux-modules.scm (module-aliases): New variable.
---
 gnu/build/linux-modules.scm | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 4a6d4ff08..364339df9 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -30,6 +30,7 @@
   #:use-module (ice-9 rdelim)
   #:export (dot-ko
             ensure-dot-ko
+            module-aliases
             module-dependencies
             recursive-module-dependencies
             modules-loaded
@@ -95,6 +96,14 @@ contains module names, not actual file names."
       (('depends . what)
        (string-tokenize what %not-comma)))))
 
+(define (module-aliases file)
+  "Return the list of aliases for FILE."
+  (let ((info (modinfo-section-contents file)))
+    (filter-map (match-lambda
+                 (('alias . value)
+                  value)
+                 (_ #f)) (modinfo-section-contents file))))
+
 (define dot-ko
   (cut string-append <> ".ko"))
 




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 02 Mar 2018 14:18:02 GMT) Full text and rfc822 format available.

Message #131 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v6 2/6] linux-modules: Add install-modules.
Date: Fri,  2 Mar 2018 15:17:16 +0100
* gnu/build/linux-modules.scm (install-modules): New procedure.
(%not-dash): New variable.
---
 gnu/build/linux-modules.scm | 52 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 364339df9..af217c974 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -36,6 +36,7 @@
             modules-loaded
             module-loaded?
             load-linux-module*
+            install-module-files
 
             current-module-debugging-port
 
@@ -379,4 +380,55 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
                       module)))
               known-aliases))
 
+(define %not-dash
+  (char-set-complement (char-set #\-)))
+
+(define (install-module-files module-files output)
+  "Install MODULE-FILES to OUTPUT.
+Precondition: OUTPUT is an empty directory."
+  (let ((aliases
+         (map (lambda (module-file-name)
+                (format #t "copying '~a'...~%" module-file-name)
+                (copy-file module-file-name
+                           (string-append output "/"
+                                          (basename module-file-name)))
+                `(,(file-name->module-name module-file-name) .
+                  ,(module-aliases module-file-name)))
+              (sort module-files string<))))
+    (call-with-output-file (string-append output "/modules.alias")
+      (lambda (port)
+        (format port
+                "# Aliases extracted from modules themselves.\n")
+        (for-each (match-lambda ((module . aliases)
+                                 (for-each (lambda (alias)
+                                             (format port "alias ~a ~a\n" alias
+                                                     module))
+                                           aliases)))
+                  aliases)))
+    (call-with-output-file (string-append output "/modules.devname")
+      (lambda (port)
+        (format port
+                "# Device nodes to trigger on-demand module loading.\n")
+        (let* ((aliases (append-map (match-lambda
+                                     ((module . aliases) aliases))
+                                    aliases))
+               (devname #f))
+          ;; Note: there's only one devname and then only one (char-major|block-major).
+          (for-each
+           (match-lambda
+            (((? (cut string-prefix? "devname:" <>) alias) . value)
+             (set! devname (string-drop value (string-length "devname:"))))
+            (((? (cut string-prefix? "char-major-" <>) alias) . value)
+             (let ((parts (string-tokenize %not-dash)))
+               (match parts
+                      ((a b major minor)
+                       (format port "~a ~a ~a:~a\n" devname "c" major minor)))))
+            (((? (cut string-prefix? "block-major-" <>) alias) . value)
+             (let ((parts (string-tokenize %not-dash)))
+               (match parts
+                      ((a b major minor)
+                       (format port "~a ~a ~a:~a\n" devname "b" major minor)))))
+            (_ #f))
+           aliases))))))
+
 ;;; linux-modules.scm ends here




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 02 Mar 2018 14:18:02 GMT) Full text and rfc822 format available.

Message #134 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v6 3/6] linux-boot: Load kernel modules only when the hardware
 is present.
Date: Fri,  2 Mar 2018 15:17:17 +0100
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
(lookup-module): Delete procedure.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
 gnu/build/linux-boot.scm    | 42 +++++++++++++++++++++++++++++++++++-------
 gnu/system/linux-initrd.scm |  4 +++-
 2 files changed, 38 insertions(+), 8 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..1b16f267a 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -469,9 +469,31 @@ upon error."
              mounts)
         "ext4"))
 
-  (define (lookup-module name)
-    (string-append linux-module-directory "/"
-                   (ensure-dot-ko name)))
+  (define (load-kernel-modules)
+    "Examine /sys/devices to find out which modules to load and load them."
+    (define enter?
+      (const #t))
+    (define (down! path stat result)
+     ;; Note: modprobe mutates the tree starting with path.
+     (let ((modalias-name (string-append path "/modalias")))
+       (if (file-exists? modalias-name)
+           (let ((modalias
+                 (string-trim-right (call-with-input-file modalias-name
+                                                          read-string)
+                                    #\newline)))
+             (system* "/sbin/modprobe" "-q" "--" modalias))))
+       #t)
+    (define up
+      (const #t))
+    (define skip
+      (const #t))
+    (define leaf
+      (const #t))
+    (define (error name stat errno result)
+      (format (current-error-port) "warning: ~a: ~a~%"
+              name (strerror errno))
+      result)
+    (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))
 
   (display "Welcome, this is GNU's early boot Guile.\n")
   (display "Use '--repl' for an initrd REPL.\n\n")
@@ -486,10 +508,16 @@ upon error."
        (when (member "--repl" args)
          (start-repl))
 
-       (display "loading kernel modules...\n")
-       (for-each (cut load-linux-module* <>
-                      #:lookup-module lookup-module)
-                 (map lookup-module linux-modules))
+       (let* ((kernel-release
+               (utsname:release (uname)))
+              (directory
+               (string-append linux-module-directory "/lib/modules/"
+                              kernel-release))
+              (old-umask (umask #o022)))
+         (make-static-device-nodes directory)
+         (umask old-umask))
+
+       (load-kernel-modules)
 
        (when qemu-guest-networking?
          (unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index e7f97bb88..b50d3ff80 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -208,7 +208,9 @@ upon error."
                       ;; this info via gexps.
                       ((gnu build file-systems)
                        #:select (find-partition-by-luks-uuid))
-                      (rnrs bytevectors))
+                      (rnrs bytevectors)
+                      (ice-9 ftw)
+                      (ice-9 rdelim))
 
          (with-output-to-port (%make-void-port "w")
            (lambda ()




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 02 Mar 2018 14:18:03 GMT) Full text and rfc822 format available.

Message #137 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v6 4/6] vm: Allow qemu-image builder to load Linux kernel
 modules.
Date: Fri,  2 Mar 2018 15:17:18 +0100
* gnu/system/vm.scm (%modprobe-wrapper): New variable.
(qemu-image): Modify.
---
 gnu/system/vm.scm | 31 +++++++++++++++++++++++++++----
 1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 345cecedd..b5a559012 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."
    #:single-file-output? #t
    #:references-graphs inputs))
 
+(define (%modprobe-wrapper modprobe linux-module-directory)
+  ;; Wrapper for the 'modprobe' command that knows where modules live.
+  ;;
+  ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',
+  ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'
+  ;; environment variable is not set---hence the need for this wrapper.
+  (program-file "modprobe"
+    #~(begin
+        (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)
+        (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))
+
 (define* (qemu-image #:key
                      (name "qemu-image")
                      (system (%current-system))
@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy
 all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true,
 register INPUTS in the store database of the image so that Guix can be used in
 the image."
+  (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))
+        (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))
   (expression->derivation-in-linux-vm
    name
-   (with-imported-modules (source-module-closure '((gnu build bootloader)
+   (with-imported-modules (source-module-closure '((gnu build activation)
+                                                   (gnu build bootloader)
                                                    (gnu build vm)
                                                    (guix build utils)))
      #~(begin
-         (use-modules (gnu build bootloader)
+         (use-modules (gnu build activation)
+                      (gnu build bootloader)
                       (gnu build vm)
                       (guix build utils)
                       (srfi srfi-26)
                       (ice-9 binary-ports))
 
          (let ((inputs
-                '#$(append (list qemu parted e2fsprogs dosfstools)
+                '#$(append (list qemu parted e2fsprogs dosfstools kmod)
                            (map canonical-package
                                 (list sed grep coreutils findutils gawk))
                            (if register-closures? (list guix) '())))
@@ -302,6 +317,14 @@ the image."
                         inputs)))
 
            (set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+           ;; It's possible that we need to load nls modules in order to
+           ;; mount the new partition.
+           (if (file-exists? #$modprobe-name)
+               (activate-modprobe #$(%modprobe-wrapper modprobe-name
+                                     linux-module-directory))
+               (format (current-error-port)
+                "WARNING: No modprobe found in ~s.  \
+Loading kernel modules will be impossible.\n" #$modprobe-name))
 
            (let* ((graphs     '#$(match inputs
                                    (((names . _) ...)
@@ -364,7 +387,7 @@ the image."
    #:make-disk-image? #t
    #:disk-image-size disk-image-size
    #:disk-image-format disk-image-format
-   #:references-graphs inputs))
+   #:references-graphs inputs)))
 
 
 ;;;




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 02 Mar 2018 14:18:03 GMT) Full text and rfc822 format available.

Message #140 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v6 5/6] vm: Make the virtio-blk uniquely identifiable in /sys.
Date: Fri,  2 Mar 2018 15:17:19 +0100
* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.
---
 gnu/build/vm.scm  | 2 +-
 gnu/system/vm.scm | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..ebf9e9f6e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."
                                            builder)
                   (append
                    (if make-disk-image?
-                       `("-device" "virtio-blk,drive=myhd"
+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
                          "-drive" ,(string-append "if=none,file=" output
                                                   ",format=" disk-image-format
                                                   ",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index b5a559012..fdff64ed9 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."
 
      #$@(map virtfs-option shared-fs)
      "-vga std"
-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
              #$image)))
 
 (define* (system-qemu-image/shared-store-script os




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 02 Mar 2018 14:18:03 GMT) Full text and rfc822 format available.

Message #143 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v6 6/6] linux-initrd: Provide modprobe to the initrd.
Date: Fri,  2 Mar 2018 15:17:20 +0100
* gnu/build/linux-initrd.scm (build-initrd): Provide modprobe and the
linux modules to the initrd.
* gnu/system/linux-initrd.scm (%modprobe): New procedure.
(expression->initrd): Use it.  Add linux-module-directory.
(raw-initrd): Pass linux-module-directory.
---
 gnu/build/linux-initrd.scm  | 13 ++++++-
 gnu/system/linux-initrd.scm | 84 ++++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 87 insertions(+), 10 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..d4cb5e2d8 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."
 
 (define* (build-initrd output
                        #:key
-                       guile init
+                       guile init modprobe linux-module-directory
                        (references-graphs '())
                        (gzip "gzip"))
   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script
@@ -131,6 +131,17 @@ REFERENCES-GRAPHS."
     (symlink (string-append guile "/bin/guile") "proc/self/exe")
     (readlink "proc/self/exe")
 
+     ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+    (when modprobe
+      (mkdir-p "sbin")
+      (symlink modprobe "sbin/modprobe")
+      (compile-to-cache "sbin/modprobe"))
+
+    ;; Make modules available as /lib/modules so modprobe finds them.
+    (mkdir-p "lib")
+    (symlink (string-append linux-module-directory "/lib/modules")
+             "lib/modules")
+
     ;; Reset the timestamps of all the files that will make it in the initrd.
     (for-each (lambda (file)
                 (unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index b50d3ff80..a69497ff8 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -56,11 +56,73 @@
 ;;;
 ;;; Code:
 
+(define* (%modprobe linux-module-directory #:key
+                    (guile %guile-static-stripped))
+  (program-file "modprobe"
+    (with-imported-modules (source-module-closure
+                            '((gnu build linux-modules)))
+      #~(begin
+          (use-modules (gnu build linux-modules) (ice-9 getopt-long)
+                       (ice-9 match) (srfi srfi-1) (ice-9 ftw))
+          (define (find-only-entry directory)
+            (match (scandir directory)
+             (("." ".." basename)
+              (string-append directory "/" basename))))
+          (define (resolve-alias alias)
+            (let* ((linux-release-module-directory
+                    (find-only-entry (string-append "/lib/modules"))))
+              (match (delete-duplicates (matching-modules alias
+                      (known-module-aliases
+                        (string-append linux-release-module-directory
+                                       "/modules.alias"))))
+               (()
+                (error "no alias by that name" alias))
+               (items
+                items))))
+          (define (lookup-module module)
+            (let* ((linux-release-module-directory
+                    (find-only-entry (string-append "/lib/modules")))
+                   (file-name (string-append linux-release-module-directory
+                                             "/" (ensure-dot-ko module))))
+              (if (file-exists? file-name)
+                  file-name
+                  (error "no module file found for module" module))))
+          (define option-spec
+           '((quiet    (single-char #\q) (value #f))))
+          (define options
+            (getopt-long (command-line) option-spec))
+          (when (option-ref options 'quiet #f)
+            (current-error-port (%make-void-port "w"))
+            (current-output-port (%make-void-port "w")))
+          (let ((exit-status 0))
+            (for-each (match-lambda
+                        (('quiet . #t)
+                         #f)
+                        ((() modules ...)
+                         (for-each (lambda (alias)
+                                     (catch #t
+                                       (lambda ()
+                                         (let ((modules (resolve-alias alias)))                                           (for-each (lambda (module)
+                                                       (load-linux-module*
+                                                        (lookup-module module)
+                                                        #:lookup-module
+                                                        lookup-module))
+                                                     modules)))
+                                       (lambda (key . args)
+                                         (display (cons* key args)
+                                                  (current-error-port))
+                                         (newline (current-error-port))
+                                         (set! exit-status 1))))
+                                   modules)))
+                      options)
+            (exit exit-status))))
+  #:guile guile))
 
 (define* (expression->initrd exp
                              #:key
                              (guile %guile-static-stripped)
                              (gzip gzip)
+                             linux-module-directory
                              (name "guile-initrd")
                              (system (%current-system)))
   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -73,6 +135,9 @@ the derivations referenced by EXP are automatically copied to the initrd."
   (define init
     (program-file "init" exp #:guile guile))
 
+  (define modprobe
+    (%modprobe linux-module-directory #:guile guile))
+
   (define builder
     (with-imported-modules (source-module-closure
                             '((gnu build linux-initrd)))
@@ -96,12 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd."
           (build-initrd (string-append #$output "/initrd")
                         #:guile #$guile
                         #:init #$init
-                        ;; Copy everything INIT refers to into the initrd.
-                        #:references-graphs '("closure")
+                        #:modprobe #$modprobe
+                        #:linux-module-directory #$linux-module-directory
+                        ;; Copy everything INIT and MODPROBE refer to into the
+                        ;; initrd.
+                        #:references-graphs '("init-closure"
+                                              "modprobe-closure")
                         #:gzip (string-append #$gzip "/bin/gzip")))))
 
   (gexp->derivation name builder
-                    #:references-graphs `(("closure" ,init))))
+                    #:references-graphs `(("init-closure" ,init)
+                                          ("modprobe-closure" ,modprobe))))
 
 (define (flat-linux-module-directory linux modules)
   "Return a flat directory containing the Linux kernel modules listed in
@@ -141,12 +211,7 @@ MODULES and taken from LINUX."
                                                      #:lookup-module lookup))))
 
           (mkdir #$output)
-          (for-each (lambda (module)
-                      (format #t "copying '~a'...~%" module)
-                      (copy-file module
-                                 (string-append #$output "/"
-                                                (basename module))))
-                    (delete-duplicates modules)))))
+          (install-module-files (delete-duplicates modules) #$output))))
 
   (computed-file "linux-modules" build-exp))
 
@@ -227,6 +292,7 @@ upon error."
                       #:qemu-guest-networking? #$qemu-networking?
                       #:volatile-root? '#$volatile-root?
                       #:on-error '#$on-error)))
+   #:linux-module-directory kodir
    #:name "raw-initrd"))
 
 (define* (file-system-packages file-systems #:key (volatile-root? #f))




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 02 Mar 2018 15:35:02 GMT) Full text and rfc822 format available.

Message #146 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v7 0/6] Load Linux module only when supported hardware is
 present.
Date: Fri,  2 Mar 2018 16:34:02 +0100
Danny Milosavljevic (6):
  linux-modules: Add module-aliases.
  linux-modules: Add install-modules.
  linux-boot: Load kernel modules only when the hardware is present.
  vm: Allow qemu-image builder to load Linux kernel modules.
  vm: Make the virtio-blk uniquely identifiable in /sys.
  linux-initrd: Provide modprobe to the initrd.

 gnu/build/linux-boot.scm    |  42 +++++++++++++++---
 gnu/build/linux-initrd.scm  |  13 +++++-
 gnu/build/linux-modules.scm |  61 ++++++++++++++++++++++++++
 gnu/build/vm.scm            |   2 +-
 gnu/system/linux-initrd.scm | 103 ++++++++++++++++++++++++++++++++++++++------
 gnu/system/vm.scm           |  34 ++++++++++++---
 6 files changed, 228 insertions(+), 27 deletions(-)





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 02 Mar 2018 15:35:02 GMT) Full text and rfc822 format available.

Message #149 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v7 1/6] linux-modules: Add module-aliases.
Date: Fri,  2 Mar 2018 16:34:03 +0100
* gnu/build/linux-modules.scm (module-aliases): New variable.
---
 gnu/build/linux-modules.scm | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 4a6d4ff08..364339df9 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -30,6 +30,7 @@
   #:use-module (ice-9 rdelim)
   #:export (dot-ko
             ensure-dot-ko
+            module-aliases
             module-dependencies
             recursive-module-dependencies
             modules-loaded
@@ -95,6 +96,14 @@ contains module names, not actual file names."
       (('depends . what)
        (string-tokenize what %not-comma)))))
 
+(define (module-aliases file)
+  "Return the list of aliases for FILE."
+  (let ((info (modinfo-section-contents file)))
+    (filter-map (match-lambda
+                 (('alias . value)
+                  value)
+                 (_ #f)) (modinfo-section-contents file))))
+
 (define dot-ko
   (cut string-append <> ".ko"))
 




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 02 Mar 2018 15:35:03 GMT) Full text and rfc822 format available.

Message #152 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v7 2/6] linux-modules: Add install-modules.
Date: Fri,  2 Mar 2018 16:34:04 +0100
* gnu/build/linux-modules.scm (install-modules): New procedure.
(%not-dash): New variable.
---
 gnu/build/linux-modules.scm | 52 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 364339df9..af217c974 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -36,6 +36,7 @@
             modules-loaded
             module-loaded?
             load-linux-module*
+            install-module-files
 
             current-module-debugging-port
 
@@ -379,4 +380,55 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
                       module)))
               known-aliases))
 
+(define %not-dash
+  (char-set-complement (char-set #\-)))
+
+(define (install-module-files module-files output)
+  "Install MODULE-FILES to OUTPUT.
+Precondition: OUTPUT is an empty directory."
+  (let ((aliases
+         (map (lambda (module-file-name)
+                (format #t "copying '~a'...~%" module-file-name)
+                (copy-file module-file-name
+                           (string-append output "/"
+                                          (basename module-file-name)))
+                `(,(file-name->module-name module-file-name) .
+                  ,(module-aliases module-file-name)))
+              (sort module-files string<))))
+    (call-with-output-file (string-append output "/modules.alias")
+      (lambda (port)
+        (format port
+                "# Aliases extracted from modules themselves.\n")
+        (for-each (match-lambda ((module . aliases)
+                                 (for-each (lambda (alias)
+                                             (format port "alias ~a ~a\n" alias
+                                                     module))
+                                           aliases)))
+                  aliases)))
+    (call-with-output-file (string-append output "/modules.devname")
+      (lambda (port)
+        (format port
+                "# Device nodes to trigger on-demand module loading.\n")
+        (let* ((aliases (append-map (match-lambda
+                                     ((module . aliases) aliases))
+                                    aliases))
+               (devname #f))
+          ;; Note: there's only one devname and then only one (char-major|block-major).
+          (for-each
+           (match-lambda
+            (((? (cut string-prefix? "devname:" <>) alias) . value)
+             (set! devname (string-drop value (string-length "devname:"))))
+            (((? (cut string-prefix? "char-major-" <>) alias) . value)
+             (let ((parts (string-tokenize %not-dash)))
+               (match parts
+                      ((a b major minor)
+                       (format port "~a ~a ~a:~a\n" devname "c" major minor)))))
+            (((? (cut string-prefix? "block-major-" <>) alias) . value)
+             (let ((parts (string-tokenize %not-dash)))
+               (match parts
+                      ((a b major minor)
+                       (format port "~a ~a ~a:~a\n" devname "b" major minor)))))
+            (_ #f))
+           aliases))))))
+
 ;;; linux-modules.scm ends here




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 02 Mar 2018 15:35:05 GMT) Full text and rfc822 format available.

Message #155 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v7 3/6] linux-boot: Load kernel modules only when the hardware
 is present.
Date: Fri,  2 Mar 2018 16:34:05 +0100
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
(lookup-module): Delete procedure.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
 gnu/build/linux-boot.scm    | 42 +++++++++++++++++++++++++++++++++++-------
 gnu/system/linux-initrd.scm |  4 +++-
 2 files changed, 38 insertions(+), 8 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..1b16f267a 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -469,9 +469,31 @@ upon error."
              mounts)
         "ext4"))
 
-  (define (lookup-module name)
-    (string-append linux-module-directory "/"
-                   (ensure-dot-ko name)))
+  (define (load-kernel-modules)
+    "Examine /sys/devices to find out which modules to load and load them."
+    (define enter?
+      (const #t))
+    (define (down! path stat result)
+     ;; Note: modprobe mutates the tree starting with path.
+     (let ((modalias-name (string-append path "/modalias")))
+       (if (file-exists? modalias-name)
+           (let ((modalias
+                 (string-trim-right (call-with-input-file modalias-name
+                                                          read-string)
+                                    #\newline)))
+             (system* "/sbin/modprobe" "-q" "--" modalias))))
+       #t)
+    (define up
+      (const #t))
+    (define skip
+      (const #t))
+    (define leaf
+      (const #t))
+    (define (error name stat errno result)
+      (format (current-error-port) "warning: ~a: ~a~%"
+              name (strerror errno))
+      result)
+    (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))
 
   (display "Welcome, this is GNU's early boot Guile.\n")
   (display "Use '--repl' for an initrd REPL.\n\n")
@@ -486,10 +508,16 @@ upon error."
        (when (member "--repl" args)
          (start-repl))
 
-       (display "loading kernel modules...\n")
-       (for-each (cut load-linux-module* <>
-                      #:lookup-module lookup-module)
-                 (map lookup-module linux-modules))
+       (let* ((kernel-release
+               (utsname:release (uname)))
+              (directory
+               (string-append linux-module-directory "/lib/modules/"
+                              kernel-release))
+              (old-umask (umask #o022)))
+         (make-static-device-nodes directory)
+         (umask old-umask))
+
+       (load-kernel-modules)
 
        (when qemu-guest-networking?
          (unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index e7f97bb88..b50d3ff80 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -208,7 +208,9 @@ upon error."
                       ;; this info via gexps.
                       ((gnu build file-systems)
                        #:select (find-partition-by-luks-uuid))
-                      (rnrs bytevectors))
+                      (rnrs bytevectors)
+                      (ice-9 ftw)
+                      (ice-9 rdelim))
 
          (with-output-to-port (%make-void-port "w")
            (lambda ()




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 02 Mar 2018 15:35:05 GMT) Full text and rfc822 format available.

Message #158 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v7 4/6] vm: Allow qemu-image builder to load Linux kernel
 modules.
Date: Fri,  2 Mar 2018 16:34:06 +0100
* gnu/system/vm.scm (%modprobe-wrapper): New variable.
(qemu-image): Modify.
---
 gnu/system/vm.scm | 31 +++++++++++++++++++++++++++----
 1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 345cecedd..b5a559012 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."
    #:single-file-output? #t
    #:references-graphs inputs))
 
+(define (%modprobe-wrapper modprobe linux-module-directory)
+  ;; Wrapper for the 'modprobe' command that knows where modules live.
+  ;;
+  ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',
+  ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'
+  ;; environment variable is not set---hence the need for this wrapper.
+  (program-file "modprobe"
+    #~(begin
+        (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)
+        (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))
+
 (define* (qemu-image #:key
                      (name "qemu-image")
                      (system (%current-system))
@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy
 all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true,
 register INPUTS in the store database of the image so that Guix can be used in
 the image."
+  (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))
+        (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))
   (expression->derivation-in-linux-vm
    name
-   (with-imported-modules (source-module-closure '((gnu build bootloader)
+   (with-imported-modules (source-module-closure '((gnu build activation)
+                                                   (gnu build bootloader)
                                                    (gnu build vm)
                                                    (guix build utils)))
      #~(begin
-         (use-modules (gnu build bootloader)
+         (use-modules (gnu build activation)
+                      (gnu build bootloader)
                       (gnu build vm)
                       (guix build utils)
                       (srfi srfi-26)
                       (ice-9 binary-ports))
 
          (let ((inputs
-                '#$(append (list qemu parted e2fsprogs dosfstools)
+                '#$(append (list qemu parted e2fsprogs dosfstools kmod)
                            (map canonical-package
                                 (list sed grep coreutils findutils gawk))
                            (if register-closures? (list guix) '())))
@@ -302,6 +317,14 @@ the image."
                         inputs)))
 
            (set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+           ;; It's possible that we need to load nls modules in order to
+           ;; mount the new partition.
+           (if (file-exists? #$modprobe-name)
+               (activate-modprobe #$(%modprobe-wrapper modprobe-name
+                                     linux-module-directory))
+               (format (current-error-port)
+                "WARNING: No modprobe found in ~s.  \
+Loading kernel modules will be impossible.\n" #$modprobe-name))
 
            (let* ((graphs     '#$(match inputs
                                    (((names . _) ...)
@@ -364,7 +387,7 @@ the image."
    #:make-disk-image? #t
    #:disk-image-size disk-image-size
    #:disk-image-format disk-image-format
-   #:references-graphs inputs))
+   #:references-graphs inputs)))
 
 
 ;;;




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 02 Mar 2018 15:35:06 GMT) Full text and rfc822 format available.

Message #161 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v7 5/6] vm: Make the virtio-blk uniquely identifiable in /sys.
Date: Fri,  2 Mar 2018 16:34:07 +0100
* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.
---
 gnu/build/vm.scm  | 2 +-
 gnu/system/vm.scm | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..ebf9e9f6e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."
                                            builder)
                   (append
                    (if make-disk-image?
-                       `("-device" "virtio-blk,drive=myhd"
+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
                          "-drive" ,(string-append "if=none,file=" output
                                                   ",format=" disk-image-format
                                                   ",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index b5a559012..fdff64ed9 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."
 
      #$@(map virtfs-option shared-fs)
      "-vga std"
-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
              #$image)))
 
 (define* (system-qemu-image/shared-store-script os




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 02 Mar 2018 15:35:06 GMT) Full text and rfc822 format available.

Message #164 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v7 6/6] linux-initrd: Provide modprobe to the initrd.
Date: Fri,  2 Mar 2018 16:34:08 +0100
* gnu/build/linux-initrd.scm (build-initrd): Provide modprobe and the
linux modules to the initrd.
* gnu/system/linux-initrd.scm (%modprobe): New procedure.
(expression->initrd): Use it.  Add linux-module-directory.
(raw-initrd): Pass linux-module-directory.
---
 gnu/build/linux-initrd.scm  | 13 +++++-
 gnu/system/linux-initrd.scm | 99 +++++++++++++++++++++++++++++++++++++++------
 2 files changed, 99 insertions(+), 13 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..d4cb5e2d8 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."
 
 (define* (build-initrd output
                        #:key
-                       guile init
+                       guile init modprobe linux-module-directory
                        (references-graphs '())
                        (gzip "gzip"))
   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script
@@ -131,6 +131,17 @@ REFERENCES-GRAPHS."
     (symlink (string-append guile "/bin/guile") "proc/self/exe")
     (readlink "proc/self/exe")
 
+     ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+    (when modprobe
+      (mkdir-p "sbin")
+      (symlink modprobe "sbin/modprobe")
+      (compile-to-cache "sbin/modprobe"))
+
+    ;; Make modules available as /lib/modules so modprobe finds them.
+    (mkdir-p "lib")
+    (symlink (string-append linux-module-directory "/lib/modules")
+             "lib/modules")
+
     ;; Reset the timestamps of all the files that will make it in the initrd.
     (for-each (lambda (file)
                 (unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index b50d3ff80..8050ac47e 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -56,11 +56,73 @@
 ;;;
 ;;; Code:
 
+(define* (%modprobe linux-module-directory #:key
+                    (guile %guile-static-stripped))
+  (program-file "modprobe"
+    (with-imported-modules (source-module-closure
+                            '((gnu build linux-modules)))
+      #~(begin
+          (use-modules (gnu build linux-modules) (ice-9 getopt-long)
+                       (ice-9 match) (srfi srfi-1) (ice-9 ftw))
+          (define (find-only-entry directory)
+            (match (scandir directory)
+             (("." ".." basename)
+              (string-append directory "/" basename))))
+          (define (resolve-alias alias)
+            (let* ((linux-release-module-directory
+                    (find-only-entry (string-append "/lib/modules"))))
+              (match (delete-duplicates (matching-modules alias
+                      (known-module-aliases
+                        (string-append linux-release-module-directory
+                                       "/modules.alias"))))
+               (()
+                (error "no alias by that name" alias))
+               (items
+                items))))
+          (define (lookup-module module)
+            (let* ((linux-release-module-directory
+                    (find-only-entry (string-append "/lib/modules")))
+                   (file-name (string-append linux-release-module-directory
+                                             "/" (ensure-dot-ko module))))
+              (if (file-exists? file-name)
+                  file-name
+                  (error "no module file found for module" module))))
+          (define option-spec
+           '((quiet    (single-char #\q) (value #f))))
+          (define options
+            (getopt-long (command-line) option-spec))
+          (when (option-ref options 'quiet #f)
+            (current-error-port (%make-void-port "w"))
+            (current-output-port (%make-void-port "w")))
+          (let ((exit-status 0))
+            (for-each (match-lambda
+                        (('quiet . #t)
+                         #f)
+                        ((() modules ...)
+                         (for-each (lambda (alias)
+                                     (catch #t
+                                       (lambda ()
+                                         (let ((modules (resolve-alias alias)))                                           (for-each (lambda (module)
+                                                       (load-linux-module*
+                                                        (lookup-module module)
+                                                        #:lookup-module
+                                                        lookup-module))
+                                                     modules)))
+                                       (lambda (key . args)
+                                         (display (cons* key args)
+                                                  (current-error-port))
+                                         (newline (current-error-port))
+                                         (set! exit-status 1))))
+                                   modules)))
+                      options)
+            (exit exit-status))))
+  #:guile guile))
 
 (define* (expression->initrd exp
                              #:key
                              (guile %guile-static-stripped)
                              (gzip gzip)
+                             linux-module-directory
                              (name "guile-initrd")
                              (system (%current-system)))
   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -73,6 +135,9 @@ the derivations referenced by EXP are automatically copied to the initrd."
   (define init
     (program-file "init" exp #:guile guile))
 
+  (define modprobe
+    (%modprobe linux-module-directory #:guile guile))
+
   (define builder
     (with-imported-modules (source-module-closure
                             '((gnu build linux-initrd)))
@@ -96,12 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd."
           (build-initrd (string-append #$output "/initrd")
                         #:guile #$guile
                         #:init #$init
-                        ;; Copy everything INIT refers to into the initrd.
-                        #:references-graphs '("closure")
+                        #:modprobe #$modprobe
+                        #:linux-module-directory #$linux-module-directory
+                        ;; Copy everything INIT and MODPROBE refer to into the
+                        ;; initrd.
+                        #:references-graphs '("init-closure"
+                                              "modprobe-closure")
                         #:gzip (string-append #$gzip "/bin/gzip")))))
 
   (gexp->derivation name builder
-                    #:references-graphs `(("closure" ,init))))
+                    #:references-graphs `(("init-closure" ,init)
+                                          ("modprobe-closure" ,modprobe))))
 
 (define (flat-linux-module-directory linux modules)
   "Return a flat directory containing the Linux kernel modules listed in
@@ -111,7 +181,7 @@ MODULES and taken from LINUX."
                             '((guix build utils)
                               (gnu build linux-modules)))
       #~(begin
-          (use-modules (ice-9 match) (ice-9 regex)
+          (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)
                        (srfi srfi-1)
                        (guix build utils)
                        (gnu build linux-modules))
@@ -140,14 +210,18 @@ MODULES and taken from LINUX."
                       (recursive-module-dependencies modules
                                                      #:lookup-module lookup))))
 
-          (mkdir #$output)
-          (for-each (lambda (module)
-                      (format #t "copying '~a'...~%" module)
-                      (copy-file module
-                                 (string-append #$output "/"
-                                                (basename module))))
-                    (delete-duplicates modules)))))
-
+          (define version
+            (match
+             (filter
+              (lambda (name)
+                (not (string-prefix? "." name)))
+              (scandir module-dir))
+             ((item) item)))
+
+          (let ((output (string-append #$output "/lib/modules/" version)))
+            (mkdir-p output)
+            (install-module-files (delete-duplicates modules) output))
+          #t)))
   (computed-file "linux-modules" build-exp))
 
 (define* (raw-initrd file-systems
@@ -227,6 +301,7 @@ upon error."
                       #:qemu-guest-networking? #$qemu-networking?
                       #:volatile-root? '#$volatile-root?
                       #:on-error '#$on-error)))
+   #:linux-module-directory kodir
    #:name "raw-initrd"))
 
 (define* (file-system-packages file-systems #:key (volatile-root? #f))




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 02 Mar 2018 16:48:01 GMT) Full text and rfc822 format available.

Message #167 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Subject: Re: [PATCH v7 3/6] linux-boot: Load kernel modules only when the
 hardware is present.
Date: Fri, 2 Mar 2018 17:47:11 +0100
> +  (define (load-kernel-modules)
> +    "Examine /sys/devices to find out which modules to load and load them."
> +    (define enter?
> +      (const #t))
> +    (define (down! path stat result)
> +     ;; Note: modprobe mutates the tree starting with path.
> +     (let ((modalias-name (string-append path "/modalias")))

I should rename "path" to "directory".





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 02 Mar 2018 16:48:02 GMT) Full text and rfc822 format available.

Message #170 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Subject: Re: [PATCH v7 1/6] linux-modules: Add module-aliases.
Date: Fri, 2 Mar 2018 17:47:51 +0100
> +  "Return the list of aliases for FILE."
                                 ^^^ replace by "in"




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sat, 03 Mar 2018 13:56:01 GMT) Full text and rfc822 format available.

Message #173 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v8 0/7] Load Linux module only when supported hardware is
 present.
Date: Sat,  3 Mar 2018 14:55:26 +0100
Danny Milosavljevic (7):
  linux-modules: Add module-aliases.
  linux-modules: Add install-modules.
  linux-boot: Load kernel modules only when the hardware is present.
  vm: Allow qemu-image builder to load Linux kernel modules.
  vm: Make the virtio-blk uniquely identifiable in /sys.
  linux-initrd: Provide modprobe to the initrd.
  linux-initrd: Factorize %modprobe and flat-linux-module-directory.

 gnu/build/linux-boot.scm    |  42 ++++++++++++---
 gnu/build/linux-initrd.scm  |  13 ++++-
 gnu/build/linux-modules.scm | 115 +++++++++++++++++++++++++++++++++++++++++
 gnu/build/vm.scm            |   2 +-
 gnu/system/linux-initrd.scm | 121 ++++++++++++++++++++++++++++++++------------
 gnu/system/vm.scm           |  34 +++++++++++--
 6 files changed, 281 insertions(+), 46 deletions(-)





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sat, 03 Mar 2018 13:56:02 GMT) Full text and rfc822 format available.

Message #176 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v8 1/7] linux-modules: Add module-aliases.
Date: Sat,  3 Mar 2018 14:55:27 +0100
* gnu/build/linux-modules.scm (module-aliases): New variable.
---
 gnu/build/linux-modules.scm | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 4a6d4ff08..364339df9 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -30,6 +30,7 @@
   #:use-module (ice-9 rdelim)
   #:export (dot-ko
             ensure-dot-ko
+            module-aliases
             module-dependencies
             recursive-module-dependencies
             modules-loaded
@@ -95,6 +96,14 @@ contains module names, not actual file names."
       (('depends . what)
        (string-tokenize what %not-comma)))))
 
+(define (module-aliases file)
+  "Return the list of aliases of module FILE."
+  (let ((info (modinfo-section-contents file)))
+    (filter-map (match-lambda
+                 (('alias . value)
+                  value)
+                 (_ #f)) (modinfo-section-contents file))))
+
 (define dot-ko
   (cut string-append <> ".ko"))
 




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sat, 03 Mar 2018 13:56:02 GMT) Full text and rfc822 format available.

Message #179 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v8 2/7] linux-modules: Add install-modules.
Date: Sat,  3 Mar 2018 14:55:28 +0100
* gnu/build/linux-modules.scm (install-modules): New procedure.
(%not-dash): New variable.
---
 gnu/build/linux-modules.scm | 52 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 364339df9..af217c974 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -36,6 +36,7 @@
             modules-loaded
             module-loaded?
             load-linux-module*
+            install-module-files
 
             current-module-debugging-port
 
@@ -379,4 +380,55 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
                       module)))
               known-aliases))
 
+(define %not-dash
+  (char-set-complement (char-set #\-)))
+
+(define (install-module-files module-files output)
+  "Install MODULE-FILES to OUTPUT.
+Precondition: OUTPUT is an empty directory."
+  (let ((aliases
+         (map (lambda (module-file-name)
+                (format #t "copying '~a'...~%" module-file-name)
+                (copy-file module-file-name
+                           (string-append output "/"
+                                          (basename module-file-name)))
+                `(,(file-name->module-name module-file-name) .
+                  ,(module-aliases module-file-name)))
+              (sort module-files string<))))
+    (call-with-output-file (string-append output "/modules.alias")
+      (lambda (port)
+        (format port
+                "# Aliases extracted from modules themselves.\n")
+        (for-each (match-lambda ((module . aliases)
+                                 (for-each (lambda (alias)
+                                             (format port "alias ~a ~a\n" alias
+                                                     module))
+                                           aliases)))
+                  aliases)))
+    (call-with-output-file (string-append output "/modules.devname")
+      (lambda (port)
+        (format port
+                "# Device nodes to trigger on-demand module loading.\n")
+        (let* ((aliases (append-map (match-lambda
+                                     ((module . aliases) aliases))
+                                    aliases))
+               (devname #f))
+          ;; Note: there's only one devname and then only one (char-major|block-major).
+          (for-each
+           (match-lambda
+            (((? (cut string-prefix? "devname:" <>) alias) . value)
+             (set! devname (string-drop value (string-length "devname:"))))
+            (((? (cut string-prefix? "char-major-" <>) alias) . value)
+             (let ((parts (string-tokenize %not-dash)))
+               (match parts
+                      ((a b major minor)
+                       (format port "~a ~a ~a:~a\n" devname "c" major minor)))))
+            (((? (cut string-prefix? "block-major-" <>) alias) . value)
+             (let ((parts (string-tokenize %not-dash)))
+               (match parts
+                      ((a b major minor)
+                       (format port "~a ~a ~a:~a\n" devname "b" major minor)))))
+            (_ #f))
+           aliases))))))
+
 ;;; linux-modules.scm ends here




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sat, 03 Mar 2018 13:56:03 GMT) Full text and rfc822 format available.

Message #182 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v8 3/7] linux-boot: Load kernel modules only when the hardware
 is present.
Date: Sat,  3 Mar 2018 14:55:29 +0100
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
(lookup-module): Delete procedure.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
 gnu/build/linux-boot.scm    | 42 +++++++++++++++++++++++++++++++++++-------
 gnu/system/linux-initrd.scm |  4 +++-
 2 files changed, 38 insertions(+), 8 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..1b16f267a 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -469,9 +469,31 @@ upon error."
              mounts)
         "ext4"))
 
-  (define (lookup-module name)
-    (string-append linux-module-directory "/"
-                   (ensure-dot-ko name)))
+  (define (load-kernel-modules)
+    "Examine /sys/devices to find out which modules to load and load them."
+    (define enter?
+      (const #t))
+    (define (down! directory stat result)
+     ;; Note: modprobe mutates the tree starting with DIRECTORY.
+     (let ((modalias-name (string-append directory "/modalias")))
+       (if (file-exists? modalias-name)
+           (let ((modalias
+                 (string-trim-right (call-with-input-file modalias-name
+                                                          read-string)
+                                    #\newline)))
+             (system* "/sbin/modprobe" "-q" "--" modalias))))
+       #t)
+    (define up
+      (const #t))
+    (define skip
+      (const #t))
+    (define leaf
+      (const #t))
+    (define (error name stat errno result)
+      (format (current-error-port) "warning: ~a: ~a~%"
+              name (strerror errno))
+      result)
+    (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))
 
   (display "Welcome, this is GNU's early boot Guile.\n")
   (display "Use '--repl' for an initrd REPL.\n\n")
@@ -486,10 +508,16 @@ upon error."
        (when (member "--repl" args)
          (start-repl))
 
-       (display "loading kernel modules...\n")
-       (for-each (cut load-linux-module* <>
-                      #:lookup-module lookup-module)
-                 (map lookup-module linux-modules))
+       (let* ((kernel-release
+               (utsname:release (uname)))
+              (directory
+               (string-append linux-module-directory "/lib/modules/"
+                              kernel-release))
+              (old-umask (umask #o022)))
+         (make-static-device-nodes directory)
+         (umask old-umask))
+
+       (load-kernel-modules)
 
        (when qemu-guest-networking?
          (unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index e7f97bb88..b50d3ff80 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -208,7 +208,9 @@ upon error."
                       ;; this info via gexps.
                       ((gnu build file-systems)
                        #:select (find-partition-by-luks-uuid))
-                      (rnrs bytevectors))
+                      (rnrs bytevectors)
+                      (ice-9 ftw)
+                      (ice-9 rdelim))
 
          (with-output-to-port (%make-void-port "w")
            (lambda ()




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sat, 03 Mar 2018 13:56:03 GMT) Full text and rfc822 format available.

Message #185 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v8 4/7] vm: Allow qemu-image builder to load Linux kernel
 modules.
Date: Sat,  3 Mar 2018 14:55:30 +0100
* gnu/system/vm.scm (%modprobe-wrapper): New variable.
(qemu-image): Modify.
---
 gnu/system/vm.scm | 31 +++++++++++++++++++++++++++----
 1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 345cecedd..b5a559012 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."
    #:single-file-output? #t
    #:references-graphs inputs))
 
+(define (%modprobe-wrapper modprobe linux-module-directory)
+  ;; Wrapper for the 'modprobe' command that knows where modules live.
+  ;;
+  ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',
+  ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'
+  ;; environment variable is not set---hence the need for this wrapper.
+  (program-file "modprobe"
+    #~(begin
+        (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)
+        (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))
+
 (define* (qemu-image #:key
                      (name "qemu-image")
                      (system (%current-system))
@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy
 all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true,
 register INPUTS in the store database of the image so that Guix can be used in
 the image."
+  (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))
+        (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))
   (expression->derivation-in-linux-vm
    name
-   (with-imported-modules (source-module-closure '((gnu build bootloader)
+   (with-imported-modules (source-module-closure '((gnu build activation)
+                                                   (gnu build bootloader)
                                                    (gnu build vm)
                                                    (guix build utils)))
      #~(begin
-         (use-modules (gnu build bootloader)
+         (use-modules (gnu build activation)
+                      (gnu build bootloader)
                       (gnu build vm)
                       (guix build utils)
                       (srfi srfi-26)
                       (ice-9 binary-ports))
 
          (let ((inputs
-                '#$(append (list qemu parted e2fsprogs dosfstools)
+                '#$(append (list qemu parted e2fsprogs dosfstools kmod)
                            (map canonical-package
                                 (list sed grep coreutils findutils gawk))
                            (if register-closures? (list guix) '())))
@@ -302,6 +317,14 @@ the image."
                         inputs)))
 
            (set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+           ;; It's possible that we need to load nls modules in order to
+           ;; mount the new partition.
+           (if (file-exists? #$modprobe-name)
+               (activate-modprobe #$(%modprobe-wrapper modprobe-name
+                                     linux-module-directory))
+               (format (current-error-port)
+                "WARNING: No modprobe found in ~s.  \
+Loading kernel modules will be impossible.\n" #$modprobe-name))
 
            (let* ((graphs     '#$(match inputs
                                    (((names . _) ...)
@@ -364,7 +387,7 @@ the image."
    #:make-disk-image? #t
    #:disk-image-size disk-image-size
    #:disk-image-format disk-image-format
-   #:references-graphs inputs))
+   #:references-graphs inputs)))
 
 
 ;;;




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sat, 03 Mar 2018 13:56:03 GMT) Full text and rfc822 format available.

Message #188 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v8 5/7] vm: Make the virtio-blk uniquely identifiable in /sys.
Date: Sat,  3 Mar 2018 14:55:31 +0100
* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.
---
 gnu/build/vm.scm  | 2 +-
 gnu/system/vm.scm | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..ebf9e9f6e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."
                                            builder)
                   (append
                    (if make-disk-image?
-                       `("-device" "virtio-blk,drive=myhd"
+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
                          "-drive" ,(string-append "if=none,file=" output
                                                   ",format=" disk-image-format
                                                   ",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index b5a559012..fdff64ed9 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -706,7 +706,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."
 
      #$@(map virtfs-option shared-fs)
      "-vga std"
-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
              #$image)))
 
 (define* (system-qemu-image/shared-store-script os




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sat, 03 Mar 2018 13:56:04 GMT) Full text and rfc822 format available.

Message #191 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v8 6/7] linux-initrd: Provide modprobe to the initrd.
Date: Sat,  3 Mar 2018 14:55:32 +0100
* gnu/build/linux-initrd.scm (build-initrd): Provide modprobe and the
linux modules to the initrd.
* gnu/system/linux-initrd.scm (%modprobe): New procedure.
(expression->initrd): Use it.  Add linux-module-directory.
(raw-initrd): Pass linux-module-directory.
---
 gnu/build/linux-initrd.scm  | 13 +++++-
 gnu/system/linux-initrd.scm | 99 +++++++++++++++++++++++++++++++++++++++------
 2 files changed, 99 insertions(+), 13 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..d4cb5e2d8 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."
 
 (define* (build-initrd output
                        #:key
-                       guile init
+                       guile init modprobe linux-module-directory
                        (references-graphs '())
                        (gzip "gzip"))
   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script
@@ -131,6 +131,17 @@ REFERENCES-GRAPHS."
     (symlink (string-append guile "/bin/guile") "proc/self/exe")
     (readlink "proc/self/exe")
 
+     ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+    (when modprobe
+      (mkdir-p "sbin")
+      (symlink modprobe "sbin/modprobe")
+      (compile-to-cache "sbin/modprobe"))
+
+    ;; Make modules available as /lib/modules so modprobe finds them.
+    (mkdir-p "lib")
+    (symlink (string-append linux-module-directory "/lib/modules")
+             "lib/modules")
+
     ;; Reset the timestamps of all the files that will make it in the initrd.
     (for-each (lambda (file)
                 (unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index b50d3ff80..8050ac47e 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -56,11 +56,73 @@
 ;;;
 ;;; Code:
 
+(define* (%modprobe linux-module-directory #:key
+                    (guile %guile-static-stripped))
+  (program-file "modprobe"
+    (with-imported-modules (source-module-closure
+                            '((gnu build linux-modules)))
+      #~(begin
+          (use-modules (gnu build linux-modules) (ice-9 getopt-long)
+                       (ice-9 match) (srfi srfi-1) (ice-9 ftw))
+          (define (find-only-entry directory)
+            (match (scandir directory)
+             (("." ".." basename)
+              (string-append directory "/" basename))))
+          (define (resolve-alias alias)
+            (let* ((linux-release-module-directory
+                    (find-only-entry (string-append "/lib/modules"))))
+              (match (delete-duplicates (matching-modules alias
+                      (known-module-aliases
+                        (string-append linux-release-module-directory
+                                       "/modules.alias"))))
+               (()
+                (error "no alias by that name" alias))
+               (items
+                items))))
+          (define (lookup-module module)
+            (let* ((linux-release-module-directory
+                    (find-only-entry (string-append "/lib/modules")))
+                   (file-name (string-append linux-release-module-directory
+                                             "/" (ensure-dot-ko module))))
+              (if (file-exists? file-name)
+                  file-name
+                  (error "no module file found for module" module))))
+          (define option-spec
+           '((quiet    (single-char #\q) (value #f))))
+          (define options
+            (getopt-long (command-line) option-spec))
+          (when (option-ref options 'quiet #f)
+            (current-error-port (%make-void-port "w"))
+            (current-output-port (%make-void-port "w")))
+          (let ((exit-status 0))
+            (for-each (match-lambda
+                        (('quiet . #t)
+                         #f)
+                        ((() modules ...)
+                         (for-each (lambda (alias)
+                                     (catch #t
+                                       (lambda ()
+                                         (let ((modules (resolve-alias alias)))                                           (for-each (lambda (module)
+                                                       (load-linux-module*
+                                                        (lookup-module module)
+                                                        #:lookup-module
+                                                        lookup-module))
+                                                     modules)))
+                                       (lambda (key . args)
+                                         (display (cons* key args)
+                                                  (current-error-port))
+                                         (newline (current-error-port))
+                                         (set! exit-status 1))))
+                                   modules)))
+                      options)
+            (exit exit-status))))
+  #:guile guile))
 
 (define* (expression->initrd exp
                              #:key
                              (guile %guile-static-stripped)
                              (gzip gzip)
+                             linux-module-directory
                              (name "guile-initrd")
                              (system (%current-system)))
   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -73,6 +135,9 @@ the derivations referenced by EXP are automatically copied to the initrd."
   (define init
     (program-file "init" exp #:guile guile))
 
+  (define modprobe
+    (%modprobe linux-module-directory #:guile guile))
+
   (define builder
     (with-imported-modules (source-module-closure
                             '((gnu build linux-initrd)))
@@ -96,12 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd."
           (build-initrd (string-append #$output "/initrd")
                         #:guile #$guile
                         #:init #$init
-                        ;; Copy everything INIT refers to into the initrd.
-                        #:references-graphs '("closure")
+                        #:modprobe #$modprobe
+                        #:linux-module-directory #$linux-module-directory
+                        ;; Copy everything INIT and MODPROBE refer to into the
+                        ;; initrd.
+                        #:references-graphs '("init-closure"
+                                              "modprobe-closure")
                         #:gzip (string-append #$gzip "/bin/gzip")))))
 
   (gexp->derivation name builder
-                    #:references-graphs `(("closure" ,init))))
+                    #:references-graphs `(("init-closure" ,init)
+                                          ("modprobe-closure" ,modprobe))))
 
 (define (flat-linux-module-directory linux modules)
   "Return a flat directory containing the Linux kernel modules listed in
@@ -111,7 +181,7 @@ MODULES and taken from LINUX."
                             '((guix build utils)
                               (gnu build linux-modules)))
       #~(begin
-          (use-modules (ice-9 match) (ice-9 regex)
+          (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)
                        (srfi srfi-1)
                        (guix build utils)
                        (gnu build linux-modules))
@@ -140,14 +210,18 @@ MODULES and taken from LINUX."
                       (recursive-module-dependencies modules
                                                      #:lookup-module lookup))))
 
-          (mkdir #$output)
-          (for-each (lambda (module)
-                      (format #t "copying '~a'...~%" module)
-                      (copy-file module
-                                 (string-append #$output "/"
-                                                (basename module))))
-                    (delete-duplicates modules)))))
-
+          (define version
+            (match
+             (filter
+              (lambda (name)
+                (not (string-prefix? "." name)))
+              (scandir module-dir))
+             ((item) item)))
+
+          (let ((output (string-append #$output "/lib/modules/" version)))
+            (mkdir-p output)
+            (install-module-files (delete-duplicates modules) output))
+          #t)))
   (computed-file "linux-modules" build-exp))
 
 (define* (raw-initrd file-systems
@@ -227,6 +301,7 @@ upon error."
                       #:qemu-guest-networking? #$qemu-networking?
                       #:volatile-root? '#$volatile-root?
                       #:on-error '#$on-error)))
+   #:linux-module-directory kodir
    #:name "raw-initrd"))
 
 (define* (file-system-packages file-systems #:key (volatile-root? #f))




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sat, 03 Mar 2018 13:56:04 GMT) Full text and rfc822 format available.

Message #194 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v8 7/7] linux-initrd: Factorize %modprobe and
 flat-linux-module-directory.
Date: Sat,  3 Mar 2018 14:55:33 +0100
* gnu/build/linux-modules.scm (module-aliases->module-file-names): New
procedure.
* gnu/system/linux-initrd.scm (%modprobe): Use
module-aliases->module-file-names.
(flat-linux-module-directory): Use module-aliases->module-file-names.
---
 gnu/build/linux-modules.scm |  56 +++++++++++++++++++++-
 gnu/system/linux-initrd.scm | 110 ++++++++++++++++++--------------------------
 2 files changed, 100 insertions(+), 66 deletions(-)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index af217c974..44059ad93 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -21,6 +21,7 @@
   #:use-module (guix elf)
   #:use-module (guix glob)
   #:use-module (guix build syscalls)
+  #:use-module (guix build utils) ; find-files
   #:use-module (rnrs io ports)
   #:use-module (rnrs bytevectors)
   #:use-module (srfi srfi-1)
@@ -28,9 +29,12 @@
   #:use-module (ice-9 vlist)
   #:use-module (ice-9 match)
   #:use-module (ice-9 rdelim)
+  #:use-module (ice-9 regex)
+  #:use-module (ice-9 ftw)
   #:export (dot-ko
             ensure-dot-ko
             module-aliases
+            module-aliases->module-file-names
             module-dependencies
             recursive-module-dependencies
             modules-loaded
@@ -385,7 +389,7 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
 
 (define (install-module-files module-files output)
   "Install MODULE-FILES to OUTPUT.
-Precondition: OUTPUT is an empty directory."
+Precondition: OUTPUT is an empty directory except for \"modules.builtin\"."
   (let ((aliases
          (map (lambda (module-file-name)
                 (format #t "copying '~a'...~%" module-file-name)
@@ -431,4 +435,54 @@ Precondition: OUTPUT is an empty directory."
             (_ #f))
            aliases))))))
 
+(define (module-aliases->module-file-names linux aliases)
+  "Resolve ALIASES to module file names, including their dependencies (which will appear
+first).  Each alias will map to a list of module file names.
+LINUX is the directory containing \"lib\"."
+  (define (string->regexp str)
+    ;; Return a regexp that matches STR exactly.
+    (string-append "^" (regexp-quote str) "$"))
+
+  (define module-dir
+    (string-append linux "/lib/modules"))
+
+  (define (find-only-entry directory)
+    (match (scandir directory)
+     (("." ".." basename)
+      (string-append directory "/" basename))))
+
+  (define linux-release-module-directory
+    (find-only-entry module-dir))
+
+  (define known-module-aliases*
+    (known-module-aliases
+     (string-append linux-release-module-directory
+                    "/modules.alias")))
+  (define (resolve-alias alias)
+    "If possible, resolve ALIAS to a list of module names.
+Otherwise return just ALIAS as possible module names."
+    (match (delete-duplicates (matching-modules alias
+                                                known-module-aliases*))
+           (()
+            (list alias))
+           (items
+            items)))
+
+  (define (lookup module)
+    (let ((name (ensure-dot-ko module)))
+      (match (find-files module-dir (string->regexp name))
+             ((file)
+              file)
+             (()
+              (error "module not found" name module-dir))
+             ((_ ...)
+              (error "several modules by that name"
+                     name module-dir)))))
+  (append-map (lambda (alias)
+                (let ((modules (map lookup (resolve-alias alias))))
+                  (append (recursive-module-dependencies modules
+                                                         #:lookup-module
+                                                         lookup) modules)))
+              aliases))
+
 ;;; linux-modules.scm ends here
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 8050ac47e..dc826c63e 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -58,35 +58,14 @@
 
 (define* (%modprobe linux-module-directory #:key
                     (guile %guile-static-stripped))
+  "Minimal implementation of modprobe for our initrd.
+LINUX-MODULE-DIRECTORY is the directory that contains \"lib\"."
   (program-file "modprobe"
     (with-imported-modules (source-module-closure
                             '((gnu build linux-modules)))
       #~(begin
           (use-modules (gnu build linux-modules) (ice-9 getopt-long)
-                       (ice-9 match) (srfi srfi-1) (ice-9 ftw))
-          (define (find-only-entry directory)
-            (match (scandir directory)
-             (("." ".." basename)
-              (string-append directory "/" basename))))
-          (define (resolve-alias alias)
-            (let* ((linux-release-module-directory
-                    (find-only-entry (string-append "/lib/modules"))))
-              (match (delete-duplicates (matching-modules alias
-                      (known-module-aliases
-                        (string-append linux-release-module-directory
-                                       "/modules.alias"))))
-               (()
-                (error "no alias by that name" alias))
-               (items
-                items))))
-          (define (lookup-module module)
-            (let* ((linux-release-module-directory
-                    (find-only-entry (string-append "/lib/modules")))
-                   (file-name (string-append linux-release-module-directory
-                                             "/" (ensure-dot-ko module))))
-              (if (file-exists? file-name)
-                  file-name
-                  (error "no module file found for module" module))))
+                       (ice-9 match) (srfi srfi-1))
           (define option-spec
            '((quiet    (single-char #\q) (value #f))))
           (define options
@@ -98,22 +77,31 @@
             (for-each (match-lambda
                         (('quiet . #t)
                          #f)
-                        ((() modules ...)
-                         (for-each (lambda (alias)
-                                     (catch #t
-                                       (lambda ()
-                                         (let ((modules (resolve-alias alias)))                                           (for-each (lambda (module)
-                                                       (load-linux-module*
-                                                        (lookup-module module)
-                                                        #:lookup-module
-                                                        lookup-module))
-                                                     modules)))
-                                       (lambda (key . args)
-                                         (display (cons* key args)
-                                                  (current-error-port))
-                                         (newline (current-error-port))
-                                         (set! exit-status 1))))
-                                   modules)))
+                        ((() aliases ...)
+                         (catch #t
+                           (lambda ()
+                             (let ((module-file-names
+                                    (module-aliases->module-file-names
+                                     #$linux-module-directory aliases)))
+                               (for-each (lambda (name)
+                                           (catch 'system-error
+                                             (lambda ()
+                                               (when (not (load-linux-module* name
+                                                                              #:recursive?
+                                                                              #f))
+                                                 (set! exit-status 1)))
+                                             (lambda (key . args)
+                                               (when (not (= EEXIST
+                                                             (system-error-errno
+                                                              (cons key args))))
+                                                 (print-exception (current-error-port)
+                                                                  #f key args)
+                                                 (set! exit-status 1)))))
+                                         module-file-names)))
+                           (lambda (key . args)
+                             (print-exception (current-error-port)
+                                              #f key args)
+                             (set! exit-status 1)))))
                       options)
             (exit exit-status))))
   #:guile guile))
@@ -173,17 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd."
                     #:references-graphs `(("init-closure" ,init)
                                           ("modprobe-closure" ,modprobe))))
 
-(define (flat-linux-module-directory linux modules)
-  "Return a flat directory containing the Linux kernel modules listed in
-MODULES and taken from LINUX."
+(define (flat-linux-module-directory linux aliases)
+  "Return a flat directory containing the Linux kernel modules resolved by
+ALIASES and taken from LINUX."
   (define build-exp
     (with-imported-modules (source-module-closure
                             '((guix build utils)
                               (gnu build linux-modules)))
       #~(begin
-          (use-modules (ice-9 match) (ice-9 regex) (ice-9 ftw)
+          (use-modules (ice-9 match) (ice-9 ftw)
                        (srfi srfi-1)
-                       (guix build utils)
+                       (guix build utils) ; TODO: Remove
                        (gnu build linux-modules))
 
           (define (string->regexp str)
@@ -193,33 +181,25 @@ MODULES and taken from LINUX."
           (define module-dir
             (string-append #$linux "/lib/modules"))
 
-          (define (lookup module)
-            (let ((name (ensure-dot-ko module)))
-              (match (find-files module-dir (string->regexp name))
-                ((file)
-                 file)
-                (()
-                 (error "module not found" name module-dir))
-                ((_ ...)
-                 (error "several modules by that name"
-                        name module-dir)))))
+          (define (find-only-entry directory)
+            (match (scandir directory)
+             (("." ".." basename)
+              (string-append directory "/" basename))))
+
+          (define linux-release-module-directory
+            (find-only-entry module-dir))
 
           (define modules
-            (let ((modules (map lookup '#$modules)))
-              (append modules
-                      (recursive-module-dependencies modules
-                                                     #:lookup-module lookup))))
+            (module-aliases->module-file-names #$linux '#$aliases))
 
           (define version
-            (match
-             (filter
-              (lambda (name)
-                (not (string-prefix? "." name)))
-              (scandir module-dir))
-             ((item) item)))
+            (basename linux-release-module-directory))
 
           (let ((output (string-append #$output "/lib/modules/" version)))
             (mkdir-p output)
+            (install-file
+             (string-append linux-release-module-directory "/modules.builtin")
+             output)
             (install-module-files (delete-duplicates modules) output))
           #t)))
   (computed-file "linux-modules" build-exp))




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sat, 03 Mar 2018 15:33:02 GMT) Full text and rfc822 format available.

Message #197 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Subject: Re: [PATCH v8 2/7] linux-modules: Add install-modules.
Date: Sat, 3 Mar 2018 16:32:12 +0100
[Message part 1 (text/plain, inline)]
> +          ;; Note: there's only one devname and then only one (char-major|block-major).
> +          (for-each
> +           (match-lambda
> +            (((? (cut string-prefix? "devname:" <>) alias) . value)
> +             (set! devname (string-drop value (string-length "devname:"))))
> +            (((? (cut string-prefix? "char-major-" <>) alias) . value)
> +             (let ((parts (string-tokenize %not-dash)))
> +               (match parts
> +                      ((a b major minor)
> +                       (format port "~a ~a ~a:~a\n" devname "c" major minor)))))
> +            (((? (cut string-prefix? "block-major-" <>) alias) . value)
> +             (let ((parts (string-tokenize %not-dash)))
> +               (match parts
> +                      ((a b major minor)
> +                       (format port "~a ~a ~a:~a\n" devname "b" major minor)))))
> +            (_ #f))
> +           aliases))))))

Probably better to be more careful that devname is set early enough.
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sat, 03 Mar 2018 18:02:01 GMT) Full text and rfc822 format available.

Message #200 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Subject: Re: [PATCH v8 7/7] linux-initrd: Factorize %modprobe and
 flat-linux-module-directory.
Date: Sat, 3 Mar 2018 19:01:31 +0100
[Message part 1 (text/plain, inline)]
> +                       (guix build utils) ; TODO: Remove

Required for mkdir-p
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sat, 03 Mar 2018 21:59:01 GMT) Full text and rfc822 format available.

Message #203 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v8 1/7] linux-modules: Add module-aliases.
Date: Sat, 03 Mar 2018 22:58:36 +0100
Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

> * gnu/build/linux-modules.scm (module-aliases): New variable.

LGTM!

> +(define (module-aliases file)
> +  "Return the list of aliases of module FILE."
> +  (let ((info (modinfo-section-contents file)))
> +    (filter-map (match-lambda
> +                 (('alias . value)
> +                  value)
> +                 (_ #f)) (modinfo-section-contents file))))

Nitpick: align like this:

  (filter-map first
              second)

when the first arg spans several lines.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sat, 03 Mar 2018 22:09:01 GMT) Full text and rfc822 format available.

Message #206 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v8 2/7] linux-modules: Add install-modules.
Date: Sat, 03 Mar 2018 23:07:57 +0100
Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

> * gnu/build/linux-modules.scm (install-modules): New procedure.
> (%not-dash): New variable.

We could reuse modules.alias and modules.devname from the ‘linux-libre’
package (right?), but I guess it doesn’t hurt to generate custom ones.

> +(define (install-module-files module-files output)
> +  "Install MODULE-FILES to OUTPUT.
> +Precondition: OUTPUT is an empty directory."
> +  (let ((aliases
> +         (map (lambda (module-file-name)
> +                (format #t "copying '~a'...~%" module-file-name)
> +                (copy-file module-file-name
> +                           (string-append output "/"
> +                                          (basename module-file-name)))
> +                `(,(file-name->module-name module-file-name) .
> +                  ,(module-aliases module-file-name)))
> +              (sort module-files string<))))
> +    (call-with-output-file (string-append output "/modules.alias")
> +      (lambda (port)
> +        (format port
> +                "# Aliases extracted from modules themselves.\n")
> +        (for-each (match-lambda ((module . aliases)
> +                                 (for-each (lambda (alias)
> +                                             (format port "alias ~a ~a\n" alias
> +                                                     module))
> +                                           aliases)))
> +                  aliases)))
> +    (call-with-output-file (string-append output "/modules.devname")
> +      (lambda (port)
> +        (format port
> +                "# Device nodes to trigger on-demand module loading.\n")
> +        (let* ((aliases (append-map (match-lambda
> +                                     ((module . aliases) aliases))
> +                                    aliases))
> +               (devname #f))
> +          ;; Note: there's only one devname and then only one (char-major|block-major).
> +          (for-each
> +           (match-lambda
> +            (((? (cut string-prefix? "devname:" <>) alias) . value)
> +             (set! devname (string-drop value (string-length "devname:"))))
> +            (((? (cut string-prefix? "char-major-" <>) alias) . value)
> +             (let ((parts (string-tokenize %not-dash)))
> +               (match parts
> +                      ((a b major minor)
> +                       (format port "~a ~a ~a:~a\n" devname "c" major minor)))))
> +            (((? (cut string-prefix? "block-major-" <>) alias) . value)
> +             (let ((parts (string-tokenize %not-dash)))
> +               (match parts
> +                      ((a b major minor)
> +                       (format port "~a ~a ~a:~a\n" devname "b" major minor)))))
> +            (_ #f))
> +           aliases))))))

I think we need different procedures here:

  (write-module-alias-database modules port)  ;for “modules.alias”
  (write-module-device-database modules port) ;for “modules.devname”

with docstrings.

I’m not sure we need ‘install-module-files’ itself.  Perhaps we can
inline it at the call site?

For the devname code, please avoid ‘set!’.  Instead you can thread the
current devname as the state of a loop:

  (let loop ((devname #f)
             (aliases aliases))
    (match aliases
      (() …)
      (((? devname-alias? devname) . rest)
       (loop devname rest))
      …))
       
The indentation of ‘match’ forms is wrong.  Would it be OK for you to
pass it through ./etc/indent-code.el?  (It’s non interactive, you don’t
need to actually use Emacs.)

Thanks,
Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sat, 03 Mar 2018 22:49:02 GMT) Full text and rfc822 format available.

Message #209 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only
 when the hardware is present.
Date: Sat, 03 Mar 2018 23:48:38 +0100
[Message part 1 (text/plain, inline)]
Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

> * gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
> the hardware is present.
> (lookup-module): Delete procedure.
> * gnu/system/linux-initrd.scm (raw-initrd): Add imports.

[...]

> +  (define (load-kernel-modules)
> +    "Examine /sys/devices to find out which modules to load and load them."
> +    (define enter?
> +      (const #t))
> +    (define (down! directory stat result)
> +     ;; Note: modprobe mutates the tree starting with DIRECTORY.
> +     (let ((modalias-name (string-append directory "/modalias")))
> +       (if (file-exists? modalias-name)
> +           (let ((modalias
> +                 (string-trim-right (call-with-input-file modalias-name
> +                                                          read-string)
> +                                    #\newline)))
> +             (system* "/sbin/modprobe" "-q" "--" modalias))))

If we change ‘flat-linux-module-directory’ to produce a ‘modules.alias’
file, here we could read ‘modules.aliases’ directly and load the right
thing.

With the patch below, we get ‘needed-modules’, and we could simply do:

  (for-each (catch-ENOENT load-linux-module*)
            (needed-modules
             (known-module-aliases (string-append linux-module-directory
                                                  "/modules.alias"))))

and we can do away with kmod’s modprobe.

Thoughts?

Ludo’.

[Message part 2 (text/x-patch, inline)]
diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 4a6d4ff08..251095072 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -20,6 +20,7 @@
 (define-module (gnu build linux-modules)
   #:use-module (guix elf)
   #:use-module (guix glob)
+  #:use-module (guix build utils)
   #:use-module (guix build syscalls)
   #:use-module (rnrs io ports)
   #:use-module (rnrs bytevectors)
@@ -40,7 +41,8 @@
 
             device-module-aliases
             known-module-aliases
-            matching-modules))
+            matching-modules
+            needed-modules))
 
 ;;; Commentary:
 ;;;
@@ -370,4 +372,25 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
                       module)))
               known-aliases))
 
+(define (system-device-aliases)
+  "Browse /sys/devices in search of \"modalias\" files and return the list of
+device aliases for the current system."
+  (let ((files (find-files "/sys/devices"
+                           (lambda (file stat)
+                             (and (eq? 'regular (stat:type stat))
+                                  (string=? "modalias" (basename file)))))))
+    (filter-map (lambda (file)
+                  (match (string-trim-right
+                          (call-with-input-file file get-string-all))
+                    ("" #f)
+                    (alias alias)))
+                files)))
+
+(define* (needed-modules #:optional (known-aliases (known-module-aliases)))
+  "Return the list of modules needed by devices on the current system.  This
+is achieved by browsing /sys/devices and returning the maching modules from
+KNOWN-ALIASES."
+  (append-map (cut matching-modules <> known-aliases)
+              (system-device-aliases)))
+
 ;;; linux-modules.scm ends here

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 04 Mar 2018 01:08:02 GMT) Full text and rfc822 format available.

Message #212 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only
 when the hardware is present.
Date: Sun, 4 Mar 2018 02:06:43 +0100
[Message part 1 (text/plain, inline)]
Hi Ludo,

On Sat, 03 Mar 2018 23:48:38 +0100
ludo <at> gnu.org (Ludovic Courtès) wrote:

> > +             (system* "/sbin/modprobe" "-q" "--" modalias))))  
> 
> If we change ‘flat-linux-module-directory’ to produce a ‘modules.alias’
> file, here we could read ‘modules.aliases’ directly and load the right
> thing.
> 
> With the patch below, we get ‘needed-modules’, and we could simply do:
> 
>   (for-each (catch-ENOENT load-linux-module*)
>             (needed-modules
>              (known-module-aliases (string-append linux-module-directory
>                                                   "/modules.alias"))))
> 
> and we can do away with kmod’s modprobe.

It's not kmod's modprobe anymore.  It's our pure-Guile implementation.

Linux lazy-invokes modprobe (for example when mounting stuff), so
/sbin/modprobe is never going away - but it can be our implementation.

I doubt it will take the modules from the correct directory with your patch.
(Of course otherwise it looks much nicer - but I think it won't pass the tests)

My newest version (v9) will use the same procedure for both computing the list
of modules for flat-linux-module-directory and the list of modules that are to
be modprobed - I think it's nice to be able to keep those in sync so we don't
get nasty surprises.  Let's see how that goes...
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 04 Mar 2018 01:10:02 GMT) Full text and rfc822 format available.

Message #215 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v9 0/7] Load Linux module only when supported hardware is
 present.
Date: Sun,  4 Mar 2018 02:09:07 +0100
Danny Milosavljevic (7):
  linux-modules: Add "modules.devname" and "modules.alias" writer.
  linux-modules: Add module-aliases->module-file-names.
  linux-initrd: Provide pure-Guile modprobe.
  linux-boot: Load kernel modules only when the hardware is present.
  vm: Allow qemu-image builder to load Linux kernel modules.
  vm: Make the virtio-blk uniquely identifiable in /sys.
  linux-initrd: Use module-aliases->module-file-names, too.

 gnu/build/linux-boot.scm    |  42 +++++++++++---
 gnu/build/linux-initrd.scm  |  13 ++++-
 gnu/build/linux-modules.scm | 114 +++++++++++++++++++++++++++++++++++++
 gnu/build/vm.scm            |   2 +-
 gnu/system/linux-initrd.scm | 136 +++++++++++++++++++++++++++++++++-----------
 gnu/system/vm.scm           |  34 +++++++++--
 6 files changed, 295 insertions(+), 46 deletions(-)





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 04 Mar 2018 01:10:03 GMT) Full text and rfc822 format available.

Message #218 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v9 1/7] linux-modules: Add "modules.devname" and
 "modules.alias" writer.
Date: Sun,  4 Mar 2018 02:09:08 +0100
* gnu/build/linux-modules.scm (write-module-alias-database): New procedure.
(write-module-device-database): New procedure.
(%not-dash): New variable.
---
 gnu/build/linux-modules.scm | 58 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 4fe673cca..0aaf2ff6f 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -36,6 +36,8 @@
             modules-loaded
             module-loaded?
             load-linux-module*
+            write-module-alias-database
+            write-module-device-database
 
             current-module-debugging-port
 
@@ -380,4 +382,60 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
                       module)))
               known-aliases))
 
+(define %not-dash
+  (char-set-complement (char-set #\-)))
+
+(define (write-module-alias-database aliases output)
+  "Install \"modules.alias\" for ALIASES to directory OUTPUT."
+  (call-with-output-file (string-append output "/modules.alias")
+    (lambda (port)
+      (format port
+              "# Aliases extracted from modules themselves.\n")
+      (for-each (match-lambda ((module . aliases)
+                               (for-each (lambda (alias)
+                                           (format port "alias ~a ~a\n" alias
+                                                   module))
+                                         aliases)))
+                aliases))))
+
+(define (write-module-device-database aliases output)
+  "Install \"modules.devname\" for ALIASES to directory OUTPUT."
+  (call-with-output-file (string-append output "/modules.devname")
+    (lambda (port)
+      (format port
+              "# Device nodes to trigger on-demand module loading.\n")
+      (for-each (match-lambda
+                 ((module . aliases)
+                  (let* ((interesting-aliases
+                          ;; Note: there's only one devname and then only one
+                          ;; (char-major|block-major).
+                          (filter-map
+                           (match-lambda
+                            ((? (cut string-prefix? "devname:" <>) alias)
+                             `(devname . ,(string-drop alias (string-length "devname:"))))
+                            ((? (cut string-prefix? "char-major-" <>) alias)
+                             `(char-major . ,(string-drop alias (string-length "char-major-"))))
+                            ((? (cut string-prefix? "block-major-" <>) alias)
+                             `(block-major . ,(string-drop alias (string-length "block-major-"))))
+                            (_ #f))
+                           aliases))
+                         (devname (assq-ref interesting-aliases
+                                            'devname))
+                         (char-major (assq-ref interesting-aliases
+                                               'char-major))
+                         (block-major (assq-ref interesting-aliases
+                                               'block-major)))
+                    (when (and devname char-major)
+                      (let ((parts (string-tokenize char-major %not-dash)))
+                        (match parts
+                         ((major minor)
+                          (format port "~a ~a ~a~a:~a\n" module devname
+                                       "c" major minor)))))
+                    (when (and devname block-major)
+                      (let ((parts (string-tokenize block-major %not-dash)))
+                        (match parts
+                         ((major minor)
+                          (format port "~a ~a ~a~a:~a\n" module devname
+                                       "b" major minor)))))))) aliases))))
+
 ;;; linux-modules.scm ends here




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 04 Mar 2018 01:10:03 GMT) Full text and rfc822 format available.

Message #221 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v9 2/7] linux-modules: Add module-aliases->module-file-names.
Date: Sun,  4 Mar 2018 02:09:09 +0100
* gnu/build/linux-modules.scm (module-aliases->module-file-names): New
procedure.
---
 gnu/build/linux-modules.scm | 55 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 0aaf2ff6f..f6bb0512b 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -21,6 +21,7 @@
   #:use-module (guix elf)
   #:use-module (guix glob)
   #:use-module (guix build syscalls)
+  #:use-module (guix build utils) ; find-files
   #:use-module (rnrs io ports)
   #:use-module (rnrs bytevectors)
   #:use-module (srfi srfi-1)
@@ -28,9 +29,12 @@
   #:use-module (ice-9 vlist)
   #:use-module (ice-9 match)
   #:use-module (ice-9 rdelim)
+  #:use-module (ice-9 regex)
+  #:use-module (ice-9 ftw)
   #:export (dot-ko
             ensure-dot-ko
             module-aliases
+            module-aliases->module-file-names
             module-dependencies
             recursive-module-dependencies
             modules-loaded
@@ -438,4 +442,55 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
                           (format port "~a ~a ~a~a:~a\n" module devname
                                        "b" major minor)))))))) aliases))))
 
+(define (module-aliases->module-file-names linux aliases)
+  "Resolve ALIASES to module file names, including their dependencies (which will appear
+first).  Each alias will map to a list of module file names.
+LINUX is the directory containing \"lib\"."
+  (define (string->regexp str)
+    ;; Return a regexp that matches STR exactly.
+    (string-append "^" (regexp-quote str) "$"))
+
+  (define module-dir
+    (string-append linux "/lib/modules"))
+
+  (define (find-only-entry directory)
+    (match (scandir directory)
+     (("." ".." basename)
+      (string-append directory "/" basename))))
+
+  (define linux-release-module-directory
+    (find-only-entry module-dir))
+
+  (define known-module-aliases*
+    (known-module-aliases
+     (string-append linux-release-module-directory
+                    "/modules.alias")))
+  (define (resolve-alias alias)
+    "If possible, resolve ALIAS to a list of module names.
+Otherwise return just ALIAS as possible module names."
+    (match (delete-duplicates (matching-modules alias
+                                                known-module-aliases*))
+           (()
+            (list alias))
+           (items
+            items)))
+
+  (define (lookup module)
+    (let ((name (ensure-dot-ko module)))
+      (match (find-files module-dir (string->regexp name))
+             ((file)
+              file)
+             (()
+              (error "module not found" name module-dir))
+             ((_ ...)
+              (error "several modules by that name"
+                     name module-dir)))))
+  (append-map (lambda (alias)
+                (let ((modules (map lookup (resolve-alias alias))))
+                  (append (recursive-module-dependencies modules
+                                                         #:lookup-module
+                                                         lookup)
+                          modules)))
+              aliases))
+
 ;;; linux-modules.scm ends here




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 04 Mar 2018 01:10:04 GMT) Full text and rfc822 format available.

Message #224 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v9 3/7] linux-initrd: Provide pure-Guile modprobe.
Date: Sun,  4 Mar 2018 02:09:10 +0100
* gnu/system/linux-initrd.scm (%modprobe): New variable.
(expression->initrd): Add modprobe, LINUX-MODULE-DIRECTORY.
(raw-initrd): Pass KODIR as LINUX-MODULE-DIRECTORY.
* gnu/build/linux-initrd.scm (build-initrd): Add modprobe.
---
 gnu/build/linux-initrd.scm  | 13 ++++++++-
 gnu/system/linux-initrd.scm | 65 ++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 74 insertions(+), 4 deletions(-)

diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..d4cb5e2d8 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -107,7 +107,7 @@ This is similar to what 'compiled-file-name' in (system base compile) does."
 
 (define* (build-initrd output
                        #:key
-                       guile init
+                       guile init modprobe linux-module-directory
                        (references-graphs '())
                        (gzip "gzip"))
   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script
@@ -131,6 +131,17 @@ REFERENCES-GRAPHS."
     (symlink (string-append guile "/bin/guile") "proc/self/exe")
     (readlink "proc/self/exe")
 
+     ;; Make modprobe available as /sbin/modprobe so the kernel finds it.
+    (when modprobe
+      (mkdir-p "sbin")
+      (symlink modprobe "sbin/modprobe")
+      (compile-to-cache "sbin/modprobe"))
+
+    ;; Make modules available as /lib/modules so modprobe finds them.
+    (mkdir-p "lib")
+    (symlink (string-append linux-module-directory "/lib/modules")
+             "lib/modules")
+
     ;; Reset the timestamps of all the files that will make it in the initrd.
     (for-each (lambda (file)
                 (unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index e0cb59c00..6ad6d75f7 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -46,7 +46,8 @@
             %base-initrd-modules
             raw-initrd
             file-system-packages
-            base-initrd))
+            base-initrd
+            %modprobe))
 
 
 ;;; Commentary:
@@ -56,11 +57,61 @@
 ;;;
 ;;; Code:
 
+(define* (%modprobe linux-module-directory #:key
+                    (guile %guile-static-stripped))
+  "Minimal implementation of modprobe for our initrd.
+LINUX-MODULE-DIRECTORY is the directory that contains \"lib\"."
+  (program-file "modprobe"
+    (with-imported-modules (source-module-closure
+                            '((gnu build linux-modules)))
+      #~(begin
+          (use-modules (gnu build linux-modules) (ice-9 getopt-long)
+                       (ice-9 match) (srfi srfi-1))
+          (define option-spec
+           '((quiet    (single-char #\q) (value #f))))
+          (define options
+            (getopt-long (command-line) option-spec))
+          (when (option-ref options 'quiet #f)
+            (current-error-port (%make-void-port "w"))
+            (current-output-port (%make-void-port "w")))
+          (let ((exit-status 0))
+            (for-each (match-lambda
+                        (('quiet . #t)
+                         #f)
+                        ((() aliases ...)
+                         (catch #t
+                           (lambda ()
+                             (let ((module-file-names
+                                    (module-aliases->module-file-names
+                                     #$linux-module-directory aliases)))
+                               (for-each (lambda (name)
+                                           (catch 'system-error
+                                             (lambda ()
+                                               (when (not (load-linux-module* name
+                                                                              #:recursive?
+                                                                              #f))
+                                                 (set! exit-status 1)))
+                                             (lambda (key . args)
+                                               (when (not (= EEXIST
+                                                             (system-error-errno
+                                                              (cons key args))))
+                                                 (print-exception (current-error-port)
+                                                                  #f key args)
+                                                 (set! exit-status 1)))))
+                                         module-file-names)))
+                           (lambda (key . args)
+                             (print-exception (current-error-port)
+                                              #f key args)
+                             (set! exit-status 1)))))
+                      options)
+            (exit exit-status))))
+  #:guile guile))
 
 (define* (expression->initrd exp
                              #:key
                              (guile %guile-static-stripped)
                              (gzip gzip)
+                             linux-module-directory
                              (name "guile-initrd")
                              (system (%current-system)))
   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)
@@ -73,6 +124,9 @@ the derivations referenced by EXP are automatically copied to the initrd."
   (define init
     (program-file "init" exp #:guile guile))
 
+  (define modprobe
+    (%modprobe linux-module-directory #:guile guile))
+
   (define builder
     (with-imported-modules (source-module-closure
                             '((gnu build linux-initrd)))
@@ -96,12 +150,16 @@ the derivations referenced by EXP are automatically copied to the initrd."
           (build-initrd (string-append #$output "/initrd")
                         #:guile #$guile
                         #:init #$init
+                        #:modprobe #$modprobe
+                        #:linux-module-directory #$linux-module-directory
                         ;; Copy everything INIT refers to into the initrd.
-                        #:references-graphs '("closure")
+                        #:references-graphs '("init-closure"
+                                              "modprobe-closure")
                         #:gzip (string-append #$gzip "/bin/gzip")))))
 
   (gexp->derivation name builder
-                    #:references-graphs `(("closure" ,init))))
+                    #:references-graphs `(("init-closure" ,init)
+                                          ("modprobe-closure" ,modprobe))))
 
 (define (flat-linux-module-directory linux modules)
   "Return a flat directory containing the Linux kernel modules listed in
@@ -225,6 +283,7 @@ upon error."
                       #:qemu-guest-networking? #$qemu-networking?
                       #:volatile-root? '#$volatile-root?
                       #:on-error '#$on-error)))
+   #:linux-module-directory kodir
    #:name "raw-initrd"))
 
 (define* (file-system-packages file-systems #:key (volatile-root? #f))




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 04 Mar 2018 01:10:04 GMT) Full text and rfc822 format available.

Message #227 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v9 4/7] linux-boot: Load kernel modules only when the hardware
 is present.
Date: Sun,  4 Mar 2018 02:09:11 +0100
* gnu/build/linux-boot.scm (boot-system): Load kernel modules only when
the hardware is present.
* gnu/system/linux-initrd.scm (raw-initrd): Add imports.
---
 gnu/build/linux-boot.scm    | 42 +++++++++++++++++++++++++++++++++++-------
 gnu/system/linux-initrd.scm |  4 +++-
 2 files changed, 38 insertions(+), 8 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..2236d8971 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -469,9 +469,31 @@ upon error."
              mounts)
         "ext4"))
 
-  (define (lookup-module name)
-    (string-append linux-module-directory "/"
-                   (ensure-dot-ko name)))
+  (define (load-kernel-modules)
+    "Examine /sys/devices to find out which modules to load and load them."
+    (define enter?
+      (const #t))
+    (define (down! directory stat result)
+     ;; Note: modprobe mutates the tree starting with DIRECTORY.
+     (let ((modalias-name (string-append directory "/modalias")))
+       (if (file-exists? modalias-name)
+           (let ((modalias
+                 (string-trim-right (call-with-input-file modalias-name
+                                                          read-string)
+                                    #\newline)))
+             (system* "/sbin/modprobe" "-q" "--" modalias))))
+       #t)
+    (define up
+      (const #t))
+    (define skip
+      (const #t))
+    (define leaf
+      (const #t))
+    (define (error name stat errno result)
+      (format (current-error-port) "warning: ~a: ~a~%"
+              name (strerror errno))
+      result)
+    (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))
 
   (display "Welcome, this is GNU's early boot Guile.\n")
   (display "Use '--repl' for an initrd REPL.\n\n")
@@ -486,10 +508,16 @@ upon error."
        (when (member "--repl" args)
          (start-repl))
 
-       (display "loading kernel modules...\n")
-       (for-each (cut load-linux-module* <>
-                      #:lookup-module lookup-module)
-                 (map lookup-module linux-modules))
+       (let* ((kernel-release
+               (utsname:release (uname)))
+              (directory
+               (string-append linux-module-directory "/lib/modules/"
+                              kernel-release))
+              (old-umask (umask #o022)))
+         (make-static-device-nodes directory)
+         (umask old-umask))
+
+       (load-kernel-modules)
 
        (when qemu-guest-networking?
          (unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 6ad6d75f7..339ecf754 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -266,7 +266,9 @@ upon error."
                       ;; this info via gexps.
                       ((gnu build file-systems)
                        #:select (find-partition-by-luks-uuid))
-                      (rnrs bytevectors))
+                      (rnrs bytevectors)
+                      (ice-9 ftw)
+                      (ice-9 rdelim))
 
          (with-output-to-port (%make-void-port "w")
            (lambda ()




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 04 Mar 2018 01:10:05 GMT) Full text and rfc822 format available.

Message #230 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v9 5/7] vm: Allow qemu-image builder to load Linux kernel
 modules.
Date: Sun,  4 Mar 2018 02:09:12 +0100
* gnu/system/vm.scm (%modprobe-wrapper): New variable.
(qemu-image): Modify.
---
 gnu/system/vm.scm | 31 +++++++++++++++++++++++++++----
 1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 91ff32ce9..cf1ec651a 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -246,6 +246,17 @@ INPUTS is a list of inputs (as for packages)."
    #:single-file-output? #t
    #:references-graphs inputs))
 
+(define (%modprobe-wrapper modprobe linux-module-directory)
+  ;; Wrapper for the 'modprobe' command that knows where modules live.
+  ;;
+  ;; This wrapper is typically invoked by the Linux kernel ('call_modprobe',
+  ;; in kernel/kmod.c), a situation where the 'LINUX_MODULE_DIRECTORY'
+  ;; environment variable is not set---hence the need for this wrapper.
+  (program-file "modprobe"
+    #~(begin
+        (setenv "LINUX_MODULE_DIRECTORY" #$linux-module-directory)
+        (apply execl #$modprobe (cons #$modprobe (cdr (command-line)))))))
+
 (define* (qemu-image #:key
                      (name "qemu-image")
                      (system (%current-system))
@@ -275,20 +286,24 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy
 all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true,
 register INPUTS in the store database of the image so that Guix can be used in
 the image."
+  (let ((modprobe-name (file-append os-drv "/profile/bin/modprobe"))
+        (linux-module-directory (file-append (file-append os-drv "/kernel/lib/modules"))))
   (expression->derivation-in-linux-vm
    name
-   (with-imported-modules (source-module-closure '((gnu build bootloader)
+   (with-imported-modules (source-module-closure '((gnu build activation)
+                                                   (gnu build bootloader)
                                                    (gnu build vm)
                                                    (guix build utils)))
      #~(begin
-         (use-modules (gnu build bootloader)
+         (use-modules (gnu build activation)
+                      (gnu build bootloader)
                       (gnu build vm)
                       (guix build utils)
                       (srfi srfi-26)
                       (ice-9 binary-ports))
 
          (let ((inputs
-                '#$(append (list qemu parted e2fsprogs dosfstools)
+                '#$(append (list qemu parted e2fsprogs dosfstools kmod)
                            (map canonical-package
                                 (list sed grep coreutils findutils gawk))
                            (if register-closures? (list guix) '())))
@@ -302,6 +317,14 @@ the image."
                         inputs)))
 
            (set-path-environment-variable "PATH" '("bin" "sbin") inputs)
+           ;; It's possible that we need to load nls modules in order to
+           ;; mount the new partition.
+           (if (file-exists? #$modprobe-name)
+               (activate-modprobe #$(%modprobe-wrapper modprobe-name
+                                     linux-module-directory))
+               (format (current-error-port)
+                "WARNING: No modprobe found in ~s.  \
+Loading kernel modules will be impossible.\n" #$modprobe-name))
 
            (let* ((graphs     '#$(match inputs
                                    (((names . _) ...)
@@ -364,7 +387,7 @@ the image."
    #:make-disk-image? #t
    #:disk-image-size disk-image-size
    #:disk-image-format disk-image-format
-   #:references-graphs inputs))
+   #:references-graphs inputs)))
 
 
 ;;;




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 04 Mar 2018 01:10:05 GMT) Full text and rfc822 format available.

Message #233 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v9 6/7] vm: Make the virtio-blk uniquely identifiable in /sys.
Date: Sun,  4 Mar 2018 02:09:13 +0100
* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.
---
 gnu/build/vm.scm  | 2 +-
 gnu/system/vm.scm | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..ebf9e9f6e 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,7 @@ the #:references-graphs parameter of 'derivation'."
                                            builder)
                   (append
                    (if make-disk-image?
-                       `("-device" "virtio-blk,drive=myhd"
+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
                          "-drive" ,(string-append "if=none,file=" output
                                                   ",format=" disk-image-format
                                                   ",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index cf1ec651a..78cc8cad1 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -700,7 +700,8 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."
 
      #$@(map virtfs-option shared-fs)
      "-vga std"
-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
              #$image)))
 
 (define* (system-qemu-image/shared-store-script os




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 04 Mar 2018 01:10:06 GMT) Full text and rfc822 format available.

Message #236 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v9 7/7] linux-initrd: Use module-aliases->module-file-names,
 too.
Date: Sun,  4 Mar 2018 02:09:14 +0100
* gnu/system/linux-initrd.scm (flat-linux-module-directory): Use
module-aliases->module-file-names.
* gnu/build/linux-modules.scm (file-name->module-name): Export.
---
 gnu/build/linux-modules.scm |  1 +
 gnu/system/linux-initrd.scm | 67 ++++++++++++++++++++++++++-------------------
 2 files changed, 40 insertions(+), 28 deletions(-)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index f6bb0512b..81a4b15b1 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -35,6 +35,7 @@
             ensure-dot-ko
             module-aliases
             module-aliases->module-file-names
+            file-name->module-name
             module-dependencies
             recursive-module-dependencies
             modules-loaded
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 339ecf754..0b976afad 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -161,17 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd."
                     #:references-graphs `(("init-closure" ,init)
                                           ("modprobe-closure" ,modprobe))))
 
-(define (flat-linux-module-directory linux modules)
-  "Return a flat directory containing the Linux kernel modules listed in
-MODULES and taken from LINUX."
+(define (flat-linux-module-directory linux aliases)
+  "Return a flat directory containing the Linux kernel modules resolved by
+ALIASES and taken from LINUX."
   (define build-exp
     (with-imported-modules (source-module-closure
                             '((guix build utils)
                               (gnu build linux-modules)))
       #~(begin
-          (use-modules (ice-9 match) (ice-9 regex)
+          (use-modules (ice-9 match) (ice-9 ftw)
                        (srfi srfi-1)
-                       (guix build utils)
+                       (guix build utils) ; mkdir-p
                        (gnu build linux-modules))
 
           (define (string->regexp str)
@@ -181,31 +181,42 @@ MODULES and taken from LINUX."
           (define module-dir
             (string-append #$linux "/lib/modules"))
 
-          (define (lookup module)
-            (let ((name (ensure-dot-ko module)))
-              (match (find-files module-dir (string->regexp name))
-                ((file)
-                 file)
-                (()
-                 (error "module not found" name module-dir))
-                ((_ ...)
-                 (error "several modules by that name"
-                        name module-dir)))))
+          (define (find-only-entry directory)
+            (match (scandir directory)
+             (("." ".." basename)
+              (string-append directory "/" basename))))
 
-          (define modules
-            (let ((modules (map lookup '#$modules)))
-              (append modules
-                      (recursive-module-dependencies modules
-                                                     #:lookup-module lookup))))
-
-          (mkdir #$output)
-          (for-each (lambda (module)
-                      (format #t "copying '~a'...~%" module)
-                      (copy-file module
-                                 (string-append #$output "/"
-                                                (basename module))))
-                    (delete-duplicates modules)))))
+          (define linux-release-module-directory
+            (find-only-entry module-dir))
 
+          (define modules
+            (module-aliases->module-file-names #$linux '#$aliases))
+
+          (define version
+            (basename linux-release-module-directory))
+
+          (define (install-module-files module-files output)
+            "Install MODULE-FILES to OUTPUT.
+Precondition: OUTPUT is an empty directory except for \"modules.builtin\"."
+            (let ((aliases
+                   (map (lambda (module-file-name)
+                          (format #t "copying '~a'...~%" module-file-name)
+                          (copy-file module-file-name
+                           (string-append output "/"
+                                          (basename module-file-name)))
+                         `(,(file-name->module-name module-file-name) .
+                            ,(module-aliases module-file-name)))
+                     (sort module-files string<))))
+              (install-file (string-append linux-release-module-directory
+                                           "/modules.builtin")
+                            output)
+              (write-module-alias-database aliases output)
+              (write-module-device-database aliases output)))
+
+          (let ((output (string-append #$output "/lib/modules/" version)))
+            (mkdir-p output)
+            (install-module-files (delete-duplicates modules) output)
+            #t))))
   (computed-file "linux-modules" build-exp))
 
 (define* (raw-initrd file-systems




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 04 Mar 2018 01:56:01 GMT) Full text and rfc822 format available.

Message #239 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only
 when the hardware is present.
Date: Sun, 4 Mar 2018 02:54:54 +0100
[Message part 1 (text/plain, inline)]
Ah, your attached patch is fine, just the proposed usage 

  (for-each (catch-ENOENT load-linux-module*)
                          ^^^^^^^^^^^^^^^^^^
            (needed-modules
             (known-module-aliases (string-append linux-module-directory
                                                  "/modules.alias"))))

is probably not working.
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 04 Mar 2018 12:35:01 GMT) Full text and rfc822 format available.

Message #242 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only
 when the hardware is present.
Date: Sun, 4 Mar 2018 13:34:44 +0100
[Message part 1 (text/plain, inline)]
Hi Ludo,

I've been thinking about it some more - I think it can be made to work "your" way.

I'm trying to integrate it right now.

But!  The module-aliases->module-file-names way I've been pursuing (which also
returns the recursive dependencies) has the advantage of using exactly the same
procedure for both flat-linux-module-directory (which copies files) and for
our modprobe (which loads them later).

This means that it's very likely these two will match.

But if load-linux-module* recursively loads modules on its own it's easily
possible that these will diverge - what flat-linux-module-directory is doing
could (accidentially) differ in the future.

Maybe that's overly defensive programming.  What do you think?

My flat-linux-module-directory basically just returns a subset of the directory
tree it gets as parameter (and flattens it for no reason).

Also, I've been trying to use find-files for /sys before, like you do.
It doesn't work correctly - probably because of the mutation that modprobe does
(find-files sorts - not sure whether that's the whole story).

Integration tests just finished - even the basic system test fails because of it
now.

I tried only this:

   (define (load-kernel-modules)
     "Examine /sys/devices to find out which modules to load and load them."
-    (define enter?
-      (const #t))
-    (define (down! directory stat result)
-     ;; Note: modprobe mutates the tree starting with DIRECTORY.
-     (let ((modalias-name (string-append directory "/modalias")))
-       (if (file-exists? modalias-name)
-           (let ((modalias
-                 (string-trim-right (call-with-input-file modalias-name
-                                                          read-string)
-                                    #\newline)))
-             (system* "/sbin/modprobe" "-q" "--" modalias))))
-       #t)
-    (define up
-      (const #t))
-    (define skip
-      (const #t))
-    (define leaf
-      (const #t))
-    (define (error name stat errno result)
-      (format (current-error-port) "warning: ~a: ~a~%"
-              name (strerror errno))
-      result)
-    (file-system-fold enter? leaf down! up skip error #t "/sys/devices"))
+    (for-each (lambda (modalias)
+                (system* "/sbin/modprobe" "-q" "--" modalias))
+              (system-device-aliases)))

Doesn't work anymore...

ERROR: In procedure network-interface-flags:
In procedure network-interface-flags: No such device

(I've exported system-device-aliases)
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 04 Mar 2018 12:46:01 GMT) Full text and rfc822 format available.

Message #245 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v8 2/7] linux-modules: Add install-modules.
Date: Sun, 4 Mar 2018 13:45:04 +0100
[Message part 1 (text/plain, inline)]
Hi Ludo,

>   (write-module-alias-database modules port)  ;for “modules.alias”
>   (write-module-device-database modules port) ;for “modules.devname”

I'd prefer "install-" because their interface is somewhat like install-file
(caller provides output directory and callee adds "modules."* automatically).

But I used the names "write-module-"* in the patchset v9.

> I’m not sure we need ‘install-module-files’ itself.  Perhaps we can
> inline it at the call site?

Sure.  Did so now.

> For the devname code, please avoid ‘set!’.

I found an even better way to avoid both.

> The indentation of ‘match’ forms is wrong.  Would it be OK for you to
> pass it through ./etc/indent-code.el?  (It’s non interactive, you don’t
> need to actually use Emacs.)

I'm actually trying to switch to emacs but when I press Tab it messes up
the stuff more.  Is there some kind of configuration for guile indentation?

Also, every time I open gnu/build/linux-modules.scm I get

"The local variables list .. may not be safe

* bug-reference-bug-repexp
* eval: (modify-syntax-entry 126 "'")
* eval: (modify-syntax-entry 36 "'")
* eval: (modify-syntax-entry 43 "'")"

???
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 04 Mar 2018 12:54:02 GMT) Full text and rfc822 format available.

Message #248 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v8 2/7] linux-modules: Add install-modules.
Date: Sun, 4 Mar 2018 13:53:27 +0100
[Message part 1 (text/plain, inline)]
Hi Ludo,

On Sat, 03 Mar 2018 23:07:57 +0100
ludo <at> gnu.org (Ludovic Courtès) wrote:

> We could reuse modules.alias and modules.devname from the ‘linux-libre’
> package (right?), but I guess it doesn’t hurt to generate custom ones.

Yes, but it would be somewhat dirty.  Here's why:

Linux-libre contains a superset of modules and also their modules.alias
and modules.devname contain a superset of modules compared to the initrd.

That means

(1) modules.alias would contain aliases which map to modules which
don't exist.  Since we don't for-each it, who cares.

(2) modules.devname would contain devnames for modules which don't
exist.  That means the boot code would create files in /dev which,
when someone accesses them, would try to load modules which don't
exist - and then the access would fail.

Technically both are not bad, but (2) is bad "UX" practise nowadays - and
not done in the Linux world any more since udev exists.

I can still remember the days of 8540 entries static /dev filesystem (as in,
really on the hard disk), 7000 of which didn't work (or even valid for this
system type :P).

If something goes wrong, good luck finding the needle in the haystack
(which /dev file you can use to boot).
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 09 Mar 2018 22:07:02 GMT) Full text and rfc822 format available.

Message #251 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only
 when the hardware is present.
Date: Fri, 09 Mar 2018 23:06:50 +0100
Hello!

I’ve reworked this patch series a bit, but now I’m stuck on this:

Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

> ERROR: In procedure network-interface-flags:
> In procedure network-interface-flags: No such device

When than happens, our modprobe gets called with the “netdev-eth0”
alias, which it cannot handle, leading to this error.

By explicitly loading “virtio_net” and sleeping a bit, we get past that
point, but fail later on:

--8<---------------cut here---------------start------------->8---
loading '/gnu/store/n2zvdxp25kvg7wy5xr1y5n9r4502fw80-linux-vm-loader'...
environment variable `PATH' set to `/gnu/store/vxl918zb9brnrgaipzsykpy3mzcafvcr-qemu-minimal-2.11.1/bin:/gnu/store/lvkaf3xlvy57bvap51xlzb209ilbkgcv-parted-3.2/sbin:/gnu/store/nm305rpb2mvridkyj4l3636nc9ql4lf9-e2fsprogs-1.43.6/bin:/gnu/store/nm305rpb2mvridkyj4l3636nc9ql4lf9-e2fsprogs-1.43.6/sbin:/gnu/store/0sq2nflm42x0znkv44add0gk82khkcb6-dosfstools-4.1/sbin:/gnu/store/0hl513mnpkhszm2hjai2w9cxmpxs0vgq-sed-4.4/bin:/gnu/store/1h44pkgdd7n6s3i2vjh54awsvfmc219j-grep-3.1/bin:/gnu/store/kgzvfby2ggi1xawsh5vjh4s93qk2dp9k-coreutils-8.28/bin:/gnu/store/k7r2m2wgj8x8jjhccwjsiimp0dlzxb7i-findutils-4.6.0/bin:/gnu/store/ahxc89r6npzf2bbl8yg5vdjicskzzjf3-gawk-4.1.4/bin'
creating partition table with 2 partitions (20.0 MiB, 40.0 MiB)...
Warning: The resulting partition is not properly aligned for best performance.
creating ext4 partition...
mke2fs 1.43.6 (29-Aug-2017)
ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/vda1 is mounted.
Creating filesystem with 20480 1k blocks and 5136 inodes
Filesystem UUID: 81a538e9-3efe-406b-bf67-fa153fa3c6a6
Superblock backups stored on blocks: 
	8193

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done

[    3.190801] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
populating...
clearing file timestamps...
creating FAT partition...
mkfs.fat 4.1 (2017-01-24)
[    3.235493] FAT-fs (vda2): IO charset iso8859-1 not found
ERROR: In procedure mount:
In procedure mount: Invalid argument
--8<---------------cut here---------------end--------------->8---

Here the nls_iso8859-1 module doesn’t get loaded and modprobe isn’t even
invoked, although my understanding is that it should (per ‘load_nls’ in
nls_base.c in the kernel, called from vfat/inode.c).

Similarly, it seems that virtio_blk has to be loaded explicitly: is not
reported by ‘needed-modules’ (i.e., not listed in /sys/…/modalias), nor
do we get a modprobe query for it.

All in all on-demand loading seems more complicated that it seemed.

What are we missing here?

Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 09 Mar 2018 22:14:02 GMT) Full text and rfc822 format available.

Message #254 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only
 when the hardware is present.
Date: Fri, 9 Mar 2018 23:13:20 +0100
[Message part 1 (text/plain, inline)]
Hi Ludo,

On Fri, 09 Mar 2018 23:06:50 +0100
ludo <at> gnu.org (Ludovic Courtès) wrote:

> When than happens, our modprobe gets called with the “netdev-eth0”
> alias, which it cannot handle, leading to this error.
> 
> By explicitly loading “virtio_net” and


> sleeping a bit, 

Uhhhhh no please don't.

> point, but fail later on:
...
> --8<---------------cut here---------------end--------------->8---
> 
> Here the nls_iso8859-1 module doesn’t get loaded and modprobe isn’t even
> invoked, although my understanding is that it should (per ‘load_nls’ in
> nls_base.c in the kernel, called from vfat/inode.c).
> 
> Similarly, it seems that virtio_blk has to be loaded explicitly: is not
> reported by ‘needed-modules’ (i.e., not listed in /sys/…/modalias), nor
> do we get a modprobe query for it.
> 
> All in all on-demand loading seems more complicated that it seemed.
> 
> What are we missing here?

As I said, modprobe mutates /sys - you cannot use find-files.  I haven't used
ftw just to be contrarian.  There's even a warning comment :-)

In order to find that out, try to print how /sys looked before modprobe - then
in the early guile recovery REPL print how /sys looks again - the alias it
was juuust complaining about will be there just fine.

My patch series works fine lazy-loading the IO charset, virtio_blk, virtio_net.

It took about 30 h to work out the correct minimal combination - just saying :-)
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 09 Mar 2018 22:20:01 GMT) Full text and rfc822 format available.

Message #257 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only
 when the hardware is present.
Date: Fri, 9 Mar 2018 23:19:42 +0100
[Message part 1 (text/plain, inline)]
oops,

On Fri, 9 Mar 2018 23:13:20 +0100
Danny Milosavljevic <dannym <at> scratchpost.org> wrote:

> As I said, modprobe mutates /sys - you cannot use find-files.  

I guess technically it's the Linux kernel module that mutates it.

>I haven't used
> ftw just to be contrarian.  There's even a warning comment :-)
> 
> In order to find that out, try to print how /sys looked before modprobe - then
> in the early guile recovery REPL print how /sys looks again - the alias it
> was juuust complaining about will be there just fine.
> 
> My patch series works fine lazy-loading the IO charset, virtio_blk, virtio_net.

(v9)
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 09 Mar 2018 22:27:01 GMT) Full text and rfc822 format available.

Message #260 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Subject: Re: [PATCH v9 7/7] linux-initrd: Use
 module-aliases->module-file-names, too.
Date: Fri, 9 Mar 2018 23:26:50 +0100
[Message part 1 (text/plain, inline)]
On Sun,  4 Mar 2018 02:09:14 +0100
Danny Milosavljevic <dannym <at> scratchpost.org> wrote:

> --- a/gnu/system/linux-initrd.scm
> +++ b/gnu/system/linux-initrd.scm
> @@ -161,17 +161,17 @@ the derivations referenced by EXP are automatically copied to the initrd."
>                      #:references-graphs `(("init-closure" ,init)
>                                            ("modprobe-closure" ,modprobe))))
>  
> -(define (flat-linux-module-directory linux modules)
> -  "Return a flat directory containing the Linux kernel modules listed in
> -MODULES and taken from LINUX."
> +(define (flat-linux-module-directory linux aliases)
> +  "Return a flat directory containing the Linux kernel modules resolved by
> +ALIASES and taken from LINUX."
...
> +          (define (find-only-entry directory)
> +            (match (scandir directory)
> +             (("." ".." basename)
> +              (string-append directory "/" basename))))

Probably unsafe since the order is not guaranteed.
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 09 Mar 2018 22:37:01 GMT) Full text and rfc822 format available.

Message #263 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only
 when the hardware is present.
Date: Fri, 9 Mar 2018 23:36:44 +0100
[Message part 1 (text/plain, inline)]
> Similarly, it seems that virtio_blk has to be loaded explicitly: is not
> reported by ‘needed-modules’ (i.e., not listed in /sys/…/modalias), nor
> do we get a modprobe query for it.

qemu seems to have a bug that it doesn't assign pci ids properly.

Patch v9 6/7 works around it by fixing the virtio-blk-pci address.

Nicer fixes welcome.
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 09 Mar 2018 22:45:02 GMT) Full text and rfc822 format available.

Message #266 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only
 when the hardware is present.
Date: Fri, 9 Mar 2018 23:44:34 +0100
[Message part 1 (text/plain, inline)]
Hi Ludo,

sorry for the many e-mails - I'm recollecting all the workarounds :)

>--8<---------------cut here---------------start------------->8---
>loading '/gnu/store/n2zvdxp25kvg7wy5xr1y5n9r4502fw80-linux-vm-loader'...
>environment variable `PATH' set to `/gnu/store/vxl918zb9brnrgaipzsykpy3mzcafvcr-qemu-minimal-2.11.1/bin:/gnu/store/lvkaf3xlvy57bvap51xlzb209ilbkgcv-parted-3.2/sbin:/>gnu/store/nm305rpb2mvridkyj4l3636nc9ql4lf9-e2fsprogs-1.43.6/bin:/gnu/store/nm305rpb2mvridkyj4l3636nc9ql4lf9-e2fsprogs-1.43.6/sbin:/gnu/>store/0sq2nflm42x0znkv44add0gk82khkcb6-dosfstools-4.1/sbin:/gnu/store/0hl513mnpkhszm2hjai2w9cxmpxs0vgq-sed-4.4/bin:/gnu/store/1h44pkgdd7n6s3i2vjh54awsvfmc219j->grep-3.1/bin:/gnu/store/kgzvfby2ggi1xawsh5vjh4s93qk2dp9k-coreutils-8.28/bin:/gnu/store/k7r2m2wgj8x8jjhccwjsiimp0dlzxb7i-findutils-4.6.0/bin:/gnu/store/>ahxc89r6npzf2bbl8yg5vdjicskzzjf3-gawk-4.1.4/bin'
>creating partition table with 2 partitions (20.0 MiB, 40.0 MiB)...
>Warning: The resulting partition is not properly aligned for best performance.
>creating ext4 partition...
...
[    3.235493] FAT-fs (vda2): IO charset iso8859-1 not found
ERROR: In procedure mount:
In procedure mount: Invalid argument
--8<---------------cut here---------------end--------------->8---

>Here the nls_iso8859-1 module doesn’t get loaded and modprobe isn’t even
>invoked, although my understanding is that it should (per ‘load_nls’ in
>nls_base.c in the kernel, called from vfat/inode.c).

Yeah, that was an evil problem.  There are two different ways linux-boot
can boot:
* Either it loads a full system in which case /run/booted-system will
contain kernel module files which can and will be loaded
* Or it loads a "--load" file only - in which case there's no system
modprobe, but switch-root just switched our pure-Guile modprobe away,
so now we are out of modprobes :)

[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 11 Mar 2018 20:25:01 GMT) Full text and rfc822 format available.

Message #269 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: 30604 <at> debbugs.gnu.org
Subject: Re: [PATCH v9 3/7] linux-initrd: Provide pure-Guile modprobe.
Date: Sun, 11 Mar 2018 21:24:30 +0100
[Message part 1 (text/plain, inline)]
> +          (define option-spec
> +           '((quiet    (single-char #\q) (value #f))))

Would make sense to also support "-a" (used when invoking modprobe to load multiple
kernel modules at once).
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 12:39:01 GMT) Full text and rfc822 format available.

Message #272 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v8 3/7] linux-boot: Load kernel modules only
 when the hardware is present.
Date: Mon, 12 Mar 2018 13:38:51 +0100
Heya Danny,

Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

> As I said, modprobe mutates /sys - you cannot use find-files.  I haven't used
> ftw just to be contrarian.  There's even a warning comment :-)

I did see this comment, and I don’t doubt you’re acting in good faith!

> In order to find that out, try to print how /sys looked before modprobe - then
> in the early guile recovery REPL print how /sys looks again - the alias it
> was juuust complaining about will be there just fine.

It wasn’t clear to me what “modprobe mutates /sys” meant, and I guess I
didn’t think it could lead to not seeing relevant “modalias” files.  Now
I’ve experienced first-hand that it does matter.  :-)

> It took about 30 h to work out the correct minimal combination - just saying :-)

Yeah I can imagine (and it would have taken me much longer I guess.)

While reviewing this series I realized I didn’t understand everything,
as in the example above, some things seemed unnecessary, and I thought
we could improve the style of a few things.

The patches that follow build upon your work with a few changes:

  • The “modules.alias” writer style is made more idiomatic and
    hopefully simpler.

  • The “modules.devname” writer is actually unnecessary.  I changed it
    to a more functional style and with smaller functions.  It’s the
    last patch of this series; it’s completely optional since we don’t
    use it currently, but we might want to keep it around for later.

  • The pure-Scheme modprobe uses SRFI-37 instead of (ice-9
    getopt-long), as in the rest of the code base.

  • /proc/sys/kernel/modprobe is set in ‘boot-system’ directly, so that
    we don’t have to special-case the “modprobe” closure in
    ‘build-initrd’.

  • ‘module-aliases->module-file-names’ is gone.  Instead there’s
    ‘load-needed-linux-modules’, similar to the ‘load-kernel-modules’
    you had, but a bit simpler.

  • I added comments here and there, especially about the
    virtio-blk-pci, which was far from obvious.  :-)

I didn’t add the ‘needed-modules’ and ‘system-device-aliases’ procedures
I posted earlier because we don’t need them currently.  If you think
they could be useful, we can always add them.

The following command succeeds:

  make check-system TESTS="basic installed-os iso-image-installer"

WDYT?


I realize that makes for a not-so-efficient review process, though I
think the result is worth it.  I’m open to suggestions though.

Overall, these changes are mostly about (1) making self-contained
commits, (2) commenting code, and (3) coding style.  About #3, I think
the guidelines I applied were:

  1. Avoid imperative style.  When we do need side effects, try hard to
     move them separately.

  2. Keep functions short.

  3. Decompose functions in a way that allows them to be combined
     “nicely”.

Some of that is a bit subjective, but overall we should be able to
converge.

Thanks for the great work, and sorry for the back-and-forth and delays!

Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 12:40:01 GMT) Full text and rfc822 format available.

Message #275 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v10 1/6] linux-modules: Add "modules.alias" writer.
Date: Mon, 12 Mar 2018 13:39:13 +0100
From: Danny Milosavljevic <dannym <at> scratchpost.org>

* gnu/build/linux-modules.scm (write-module-alias-database): New
procedure.
(%not-dash): New variable.

Co-authored-by: Ludovic Courtès <ludo <at> gnu.org>
---
 gnu/build/linux-modules.scm | 26 +++++++++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 4fe673cca..6101b8dc3 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2014, 2016, 2018 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe <at> gmail.com>
+;;; Copyright © 2018 Danny Milosavljevic <dannym <at> scratchpost.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -21,6 +22,7 @@
   #:use-module (guix elf)
   #:use-module (guix glob)
   #:use-module (guix build syscalls)
+  #:use-module ((guix build utils) #:select (find-files))
   #:use-module (rnrs io ports)
   #:use-module (rnrs bytevectors)
   #:use-module (srfi srfi-1)
@@ -41,7 +43,8 @@
 
             device-module-aliases
             known-module-aliases
-            matching-modules))
+            matching-modules
+            write-module-alias-database))
 
 ;;; Commentary:
 ;;;
@@ -380,4 +383,25 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
                       module)))
               known-aliases))
 
+(define %not-dash
+  (char-set-complement (char-set #\-)))
+
+(define (write-module-alias-database directory)
+  "Traverse the '.ko' files in DIRECTORY and create the corresponding
+'modules.alias' file."
+  (define aliases
+    (map (lambda (file)
+           (cons (file-name->module-name file) (module-aliases file)))
+         (find-files directory "\\.ko$")))
+
+  (call-with-output-file (string-append directory "/modules.alias")
+    (lambda (port)
+      (display "# Aliases extracted from modules themselves.\n" port)
+      (for-each (match-lambda
+                  ((module . aliases)
+                   (for-each (lambda (alias)
+                               (format port "alias ~a ~a\n" alias module))
+                             aliases)))
+                aliases))))
+
 ;;; linux-modules.scm ends here
-- 
2.16.2





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 12:40:02 GMT) Full text and rfc822 format available.

Message #278 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 30604 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH v10 2/6] linux-modules: Add
 'load-linux-modules-from-directory'.
Date: Mon, 12 Mar 2018 13:39:14 +0100
* gnu/build/linux-modules.scm (load-linux-modules-from-directory): New
procedure.
* gnu/build/linux-boot.scm (boot-system)[lookup-module]: Remove.
Use 'load-linux-modules-from-directory' instead.
---
 gnu/build/linux-boot.scm    |  9 ++-------
 gnu/build/linux-modules.scm | 12 ++++++++++++
 2 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..df0b2b2d1 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -469,10 +469,6 @@ upon error."
              mounts)
         "ext4"))
 
-  (define (lookup-module name)
-    (string-append linux-module-directory "/"
-                   (ensure-dot-ko name)))
-
   (display "Welcome, this is GNU's early boot Guile.\n")
   (display "Use '--repl' for an initrd REPL.\n\n")
 
@@ -487,9 +483,8 @@ upon error."
          (start-repl))
 
        (display "loading kernel modules...\n")
-       (for-each (cut load-linux-module* <>
-                      #:lookup-module lookup-module)
-                 (map lookup-module linux-modules))
+       (load-linux-modules-from-directory linux-modules
+                                          linux-module-directory)
 
        (when qemu-guest-networking?
          (unless (configure-qemu-networking)
diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 6101b8dc3..1337b5e88 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -38,6 +38,7 @@
             modules-loaded
             module-loaded?
             load-linux-module*
+            load-linux-modules-from-directory
 
             current-module-debugging-port
 
@@ -232,6 +233,17 @@ appears in BLACK-LIST are not loaded."
              (or (and recursive? (= EEXIST (system-error-errno args)))
                  (apply throw args)))))))
 
+(define (load-linux-modules-from-directory modules directory)
+  "Load MODULES and their dependencies from DIRECTORY, a directory containing
+the '.ko' files.  The '.ko' suffix is automatically added to MODULES if
+needed."
+  (define (lookup-module name)
+    (string-append directory "/" (ensure-dot-ko name)))
+
+  (for-each (cut load-linux-module* <>
+                 #:lookup-module lookup-module)
+            (map lookup-module modules)))
+
 
 ;;;
 ;;; Device modules.
-- 
2.16.2





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 12:40:02 GMT) Full text and rfc822 format available.

Message #281 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 30604 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH v10 3/6] linux-modules: Add 'load-needed-linux-modules'.
Date: Mon, 12 Mar 2018 13:39:15 +0100
Based on a patch by Danny Milosavljevic <dannym <at> scratchpost.org>.

* gnu/build/linux-modules.scm (load-needed-linux-modules): New
procedure.
---
 gnu/build/linux-modules.scm | 34 +++++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 1337b5e88..9aabe6d37 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -30,6 +30,7 @@
   #:use-module (ice-9 vlist)
   #:use-module (ice-9 match)
   #:use-module (ice-9 rdelim)
+  #:use-module (ice-9 ftw)
   #:export (dot-ko
             ensure-dot-ko
             module-aliases
@@ -45,7 +46,8 @@
             device-module-aliases
             known-module-aliases
             matching-modules
-            write-module-alias-database))
+            write-module-alias-database
+            load-needed-linux-modules))
 
 ;;; Commentary:
 ;;;
@@ -416,4 +418,34 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
                              aliases)))
                 aliases))))
 
+(define (load-needed-linux-modules module-directory)
+  "Examine /sys/devices to find out which modules are needed and load those we
+have in MODULE-DIRECTORY.  Return the list of modules loaded, not including
+dependencies.
+
+Note: loading modules leads to the creation of new entries in /sys/devices,
+which is why we need traversal and loading to be interleaved.  If we walked
+/sys/devices and *then* loaded modules, we'd miss the entries added as a
+side-effect and would thus need to traverse /sys/devices again."
+  (define aliases
+    (known-module-aliases
+     (string-append module-directory "/modules.alias")))
+
+  (define (enter? director stat result) result)
+  (define (down directory stat result) result)
+  (define (up directory stat result) result)
+  (define (skip entry stat result) result)
+  (define (error name stat errno result) result)
+  (define (leaf file stat result)
+    (if (string=? (basename file) "modalias")
+        (let* ((alias   (string-trim-right
+                         (call-with-input-file file get-string-all)))
+               (modules (matching-modules alias aliases)))
+          (load-linux-modules-from-directory modules
+                                             module-directory)
+          (append modules result))
+        result))
+
+  (file-system-fold enter? leaf down up skip error #t "/sys/devices"))
+
 ;;; linux-modules.scm ends here
-- 
2.16.2





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 12:40:03 GMT) Full text and rfc822 format available.

Message #284 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>,
 Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH v10 4/6] vm: Make the virtio-blk uniquely identifiable in /sys.
Date: Mon, 12 Mar 2018 13:39:16 +0100
From: Danny Milosavljevic <dannym <at> scratchpost.org>

* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.

Signed-off-by: Ludovic Courtès <ludo <at> gnu.org>
---
 gnu/build/vm.scm  | 6 +++++-
 gnu/system/vm.scm | 7 ++++++-
 2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..773b738ae 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,11 @@ the #:references-graphs parameter of 'derivation'."
                                            builder)
                   (append
                    (if make-disk-image?
-                       `("-device" "virtio-blk,drive=myhd"
+
+                       ;; QEMU seems to have a bug that it doesn't assign PCI
+                       ;; IDs properly, so force the address of our virtio-blk
+                       ;; device.
+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
                          "-drive" ,(string-append "if=none,file=" output
                                                   ",format=" disk-image-format
                                                   ",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 91ff32ce9..4360adf15 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -677,7 +677,12 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."
 
      #$@(map virtfs-option shared-fs)
      "-vga std"
-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+
+     ;; QEMU seems to have a bug that it doesn't assign PCI IDs properly, so
+     ;; force the address of our virtio-blk device.
+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+
+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
              #$image)))
 
 (define* (system-qemu-image/shared-store-script os
-- 
2.16.2





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 12:40:03 GMT) Full text and rfc822 format available.

Message #287 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 30604 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH v10 6/6] linux-modules: Add "modules.devname" writer.
Date: Mon, 12 Mar 2018 13:39:18 +0100
* gnu/build/linux-modules.scm (aliases->device-tuple)
(write-module-device-database): New procedures.

Co-authored-by: Danny Milosavljevic <dannym <at> scratchpost.org>.
---
 gnu/build/linux-modules.scm | 48 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 9aabe6d37..2ed04e5b4 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -47,6 +47,8 @@
             known-module-aliases
             matching-modules
             write-module-alias-database
+            write-module-device-database
+
             load-needed-linux-modules))
 
 ;;; Commentary:
@@ -418,6 +420,52 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
                              aliases)))
                 aliases))))
 
+(define (aliases->device-tuple aliases)
+  "Traverse ALIASES, a list of module aliases, and search for
+\"char-major-M-N\", \"block-major-M-N\", or \"devname:\" aliases.  When they
+are found, return a tuple (DEVNAME TYPE MAJOR MINOR), otherwise return #f."
+  (define (char/block-major? alias)
+    (or (string-prefix? "char-major-" alias)
+        (string-prefix? "block-major-" alias)))
+
+  (define (char/block-major->tuple alias)
+    (match (string-tokenize alias %not-dash)
+      ((type "major" (= string->number major) (= string->number minor))
+       (list (match type
+               ("char" "c")
+               ("block" "b"))
+             major minor))))
+
+  (let* ((devname     (any (lambda (alias)
+                             (and (string-prefix? "devname:" alias)
+                                  (string-drop alias 8)))
+                           aliases))
+         (major/minor (match (find char/block-major? aliases)
+                        (#f #f)
+                        (str (char/block-major->tuple str)))))
+    (and devname major/minor
+         (cons devname major/minor))))
+
+(define (write-module-device-database directory)
+  "Traverse the '.ko' files in DIRECTORY and create the corresponding
+'modules.devname' file.  This file contains information about modules that can
+be loaded on-demand, such as file system modules."
+  (define aliases
+    (filter-map (lambda (file)
+                  (match (aliases->device-tuple (module-aliases file))
+                    (#f #f)
+                    (tuple (cons (file-name->module-name file) tuple))))
+                (find-files directory "\\.ko$")))
+
+  (call-with-output-file (string-append "/tmp" "/modules.devname")
+    (lambda (port)
+      (display "# Device nodes to trigger on-demand module loading.\n" port)
+      (for-each (match-lambda
+                  ((module devname type major minor)
+                   (format port "~a ~a ~a~a:~a~%"
+                           module devname type major minor)))
+                aliases))))
+
 (define (load-needed-linux-modules module-directory)
   "Examine /sys/devices to find out which modules are needed and load those we
 have in MODULE-DIRECTORY.  Return the list of modules loaded, not including
-- 
2.16.2





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 12:40:04 GMT) Full text and rfc822 format available.

Message #290 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 30604 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH v10 5/6] linux-initrd: Provide our own 'modprobe' program.
Date: Mon, 12 Mar 2018 13:39:17 +0100
This allows us to load modules on demand when the kernel asks for them.

* gnu/system/linux-initrd.scm (modprobe-program): New variable.
(flat-linux-module-directory): Call 'write-module-alias-database'.
(raw-initrd): Pass #:modprobe to 'boot-system'.
(expression->initrd): Copy "closure" to $out/references.
* gnu/build/linux-boot.scm (boot-system): Add #:modprobe and honor it.
Call 'load-needed-linux-modules'.
* gnu/system/vm.scm (qemu-image): Add #:linux parameter.  Define
'modprobe-wrapper' and pass it to 'activate-modprobe'.  Pass #:linux to
'expression->derivation-in-linux-vm'.

Co-authored-by: Danny Milosavljevic <dannym <at> scratchpost.org>
---
 gnu/build/linux-boot.scm    | 13 +++++--
 gnu/system/linux-initrd.scm | 86 +++++++++++++++++++++++++++++++++++++++++----
 gnu/system/vm.scm           | 21 +++++++++++
 3 files changed, 111 insertions(+), 9 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index df0b2b2d1..eedc4bb9d 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -435,6 +435,7 @@ bailing out.~%root contents: ~s~%" (scandir "/"))
 
 
 (define* (boot-system #:key
+                      modprobe
                       (linux-modules '())
                       linux-module-directory
                       qemu-guest-networking?
@@ -449,6 +450,9 @@ QEMU-GUEST-NETWORKING? is true, calling PRE-MOUNT, mounting the file systems
 specified in MOUNTS, and finally booting into the new root if any.  The initrd
 supports kernel command-line options '--load', '--root', and '--repl'.
 
+MODPROBE must be #f or a program to install as the modprobe program that the
+kernel will invoke when it needs to load modules.
+
 Mount the root file system, specified by the '--root' command-line argument,
 if any.
 
@@ -482,9 +486,14 @@ upon error."
        (when (member "--repl" args)
          (start-repl))
 
+       (when modprobe
+         ;; Tell the kernel to invoke MODPROBE.
+         (call-with-output-file "/proc/sys/kernel/modprobe"
+           (lambda (port)
+             (display modprobe port))))
+
        (display "loading kernel modules...\n")
-       (load-linux-modules-from-directory linux-modules
-                                          linux-module-directory)
+       (load-needed-linux-modules linux-module-directory)
 
        (when qemu-guest-networking?
          (unless (configure-qemu-networking)
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 1eb5f5130..6b7883c2c 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -63,6 +63,75 @@
 ;;;
 ;;; Code:
 
+(define* (modprobe-program linux-module-directory #:key
+                           (guile %guile-static-stripped))
+  "Return a minimal implementation of 'modprobe' for our initrd that looks up
+modules in LINUX-MODULE-DIRECTORY.  This program will be invoked by the kernel
+when modules need to be loaded."
+  (define program
+    (with-imported-modules (source-module-closure
+                            '((gnu build linux-modules)))
+      #~(begin
+          (use-modules (gnu build linux-modules)
+                       (ice-9 match)
+                       (srfi srfi-1)
+                       (srfi srfi-26)
+                       (srfi srfi-37))
+
+          (define option-spec
+            (list (option '(#\q "quiet") #f #f
+                          (lambda (opt name arg result)
+                            (alist-cons 'quiet? #t result)))))
+
+          (define options
+            ;; Alist of options and non-option arguments.
+            (args-fold (cdr (program-arguments))
+                       option-spec
+                       (lambda (opt name arg result)
+                         (error "unrecognized option" name))
+                       (lambda (arg result)
+                         (alist-cons 'argument arg result))
+                       '()))
+
+          (define aliases
+            ;; The list of aliases we are asked to load.
+            (filter-map (match-lambda
+                          (('argument . alias) alias)
+                          (_ #f))
+                        options))
+
+          (define linux-module-directory
+            ;; The module directory.  Note: We expect a flat directory here.
+            #$linux-module-directory)
+
+          (define %known-aliases
+            ;; The alias database.
+            (known-module-aliases
+             (string-append linux-module-directory "/modules.alias")))
+
+          (define (device-aliases->module-names aliases)
+            ;; Return the list of module names for the subset of ALIASES that
+            ;; appears in %KNOWN-ALIASES.
+            (append-map (cut matching-modules <> %known-aliases)
+                        aliases))
+
+          (when (assq-ref options 'quiet?)
+            (current-error-port (%make-void-port "w"))
+            (current-output-port (%make-void-port "w")))
+
+          (let ((modules (device-aliases->module-names aliases)))
+            (call-with-output-file "/dev/kmsg"
+              (lambda (port)
+                (format port "modprobe[~a]: aliases ~s; modules ~s; args ~s~%"
+                        (getpid) aliases modules (program-arguments))))
+
+            (when (< (length modules) (length aliases))
+              (error "alias resolution failed" aliases))
+
+            (load-linux-modules-from-directory (reverse modules)
+                                               linux-module-directory)))))
+
+  (program-file "modprobe" program #:guile guile))
 
 (define* (expression->initrd exp
                              #:key
@@ -89,16 +158,16 @@ the derivations referenced by EXP are automatically copied to the initrd."
           (mkdir #$output)
 
           ;; The guile used in the initrd must be present in the store, so
-          ;; that module loading works once the root is switched.
+          ;; that module loading works once the root is switched.  Similarly,
+          ;; the 'modprobe' program installed in /proc/sys/kernel/modprobe by
+          ;; the initrd, if any, must be present after switch root.
           ;;
           ;; To ensure that is the case, add an explicit reference to the
           ;; guile package used in the initrd to the output.
           ;;
-          ;; This fixes guix-patches bug #28399, "Fix mysql activation, and
+          ;; This fixes <https://bugs.gnu.org/28399>, "Fix mysql activation, and
           ;; add a basic test".
-          (call-with-output-file (string-append #$ output "/references")
-            (lambda (port)
-              (simple-format port "~A\n" #$guile)))
+          (copy-file "closure" (string-append #$output "/references"))
 
           (build-initrd (string-append #$output "/initrd")
                         #:guile #$guile
@@ -153,7 +222,9 @@ MODULES and taken from LINUX."
                       (copy-file module
                                  (string-append #$output "/"
                                                 (basename module))))
-                    (delete-duplicates modules)))))
+                    (delete-duplicates modules))
+
+          (write-module-alias-database #$output))))
 
   (computed-file "linux-modules" build-exp))
 
@@ -222,7 +293,8 @@ upon error."
              (set-path-environment-variable "PATH" '("bin" "sbin")
                                             '#$helper-packages)))
 
-         (boot-system #:mounts
+         (boot-system #:modprobe #$(modprobe-program kodir)
+                      #:mounts
                       (map spec->file-system
                            '#$(map file-system->spec file-systems))
                       #:pre-mount (lambda ()
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 4360adf15..ecb7ed506 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -249,6 +249,7 @@ INPUTS is a list of inputs (as for packages)."
 (define* (qemu-image #:key
                      (name "qemu-image")
                      (system (%current-system))
+                     (linux linux-libre)
                      (qemu qemu-minimal)
                      (disk-image-size 'guess)
                      (disk-image-format "qcow2")
@@ -275,18 +276,37 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy
 all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true,
 register INPUTS in the store database of the image so that Guix can be used in
 the image."
+  (define modprobe-wrapper
+    ;; Wrapper for the 'modprobe' command that knows where modules live.
+    (let ((modprobe (file-append kmod "/bin/modprobe")))
+      (program-file "modprobe"
+                    #~(begin
+                        (setenv "LINUX_MODULE_DIRECTORY"
+                                #$(file-append linux "/lib/modules"))
+                        (apply execl #$modprobe
+                               (cons #$modprobe (cdr (command-line))))))))
+
+
   (expression->derivation-in-linux-vm
    name
    (with-imported-modules (source-module-closure '((gnu build bootloader)
                                                    (gnu build vm)
+                                                   (gnu build activation)
                                                    (guix build utils)))
      #~(begin
          (use-modules (gnu build bootloader)
                       (gnu build vm)
+                      ((gnu build activation) #:select (activate-modprobe))
                       (guix build utils)
                       (srfi srfi-26)
                       (ice-9 binary-ports))
 
+         ;; We may need to lazy-load modules.  The initrd installs a
+         ;; 'modprobe' that can only search through the modules available in
+         ;; the initrd, but here we want to be able to use all the modules of
+         ;; LINUX.  Thus, install a real 'modprobe'.
+         (activate-modprobe #$modprobe-wrapper)
+
          (let ((inputs
                 '#$(append (list qemu parted e2fsprogs dosfstools)
                            (map canonical-package
@@ -361,6 +381,7 @@ the image."
                                    #$(bootloader-installer bootloader))
              (reboot)))))
    #:system system
+   #:linux linux
    #:make-disk-image? #t
    #:disk-image-size disk-image-size
    #:disk-image-format disk-image-format
-- 
2.16.2





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 14:46:01 GMT) Full text and rfc822 format available.

Message #293 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v9 3/7] linux-initrd: Provide pure-Guile
 modprobe.
Date: Mon, 12 Mar 2018 15:45:29 +0100
Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

>> +          (define option-spec
>> +           '((quiet    (single-char #\q) (value #f))))
>
> Would make sense to also support "-a" (used when invoking modprobe to load multiple
> kernel modules at once).

I suppose we can do that later because the kernel doesn’t use it in
‘call_modprobe’ and this thing is only ever invoked by the kernel.

WDYT?

Ludo’.





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 17:53:02 GMT) Full text and rfc822 format available.

Message #296 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v9 3/7] linux-initrd: Provide pure-Guile
 modprobe.
Date: Mon, 12 Mar 2018 18:51:55 +0100
[Message part 1 (text/plain, inline)]
Hi Ludo,

On Mon, 12 Mar 2018 15:45:29 +0100
ludo <at> gnu.org (Ludovic Courtès) wrote:

> > Would make sense to also support "-a" (used when invoking modprobe to load multiple
> > kernel modules at once).  
> 
> I suppose we can do that later because the kernel doesn’t use it in
> ‘call_modprobe’ and this thing is only ever invoked by the kernel.

Sure.

I just wanted to comment because I noticed that kmod's

  modprobe a b c

doesn't load three modules a, b, c.  But my v9's pure-Guile modprobe was
implemented in a way that it did load three modules a, b, c - which it
definitely shouldn't be doing.
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 19:28:02 GMT) Full text and rfc822 format available.

Message #299 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v10 2/6] linux-modules: Add
 'load-linux-modules-from-directory'.
Date: Mon, 12 Mar 2018 20:26:54 +0100
[Message part 1 (text/plain, inline)]
LGTM!
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 19:41:01 GMT) Full text and rfc822 format available.

Message #302 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v10 3/6] linux-modules: Add
 'load-needed-linux-modules'.
Date: Mon, 12 Mar 2018 20:40:02 +0100
> +        (let* ((alias   (string-trim-right
> +                         (call-with-input-file file get-string-all)))

Hmm... doesn't this trim other things than newline?

A quick internet search shows that most people do that.  Huh.

LGTM!




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 19:55:02 GMT) Full text and rfc822 format available.

Message #305 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [PATCH v10 1/6] linux-modules: Add "modules.alias" writer.
Date: Mon, 12 Mar 2018 20:54:44 +0100
[Message part 1 (text/plain, inline)]
> +(define %not-dash
> +  (char-set-complement (char-set #\-)))

Is that used?

[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 20:10:02 GMT) Full text and rfc822 format available.

Message #308 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own
 'modprobe' program.
Date: Mon, 12 Mar 2018 21:09:36 +0100
[Message part 1 (text/plain, inline)]
Hi Ludo,

On Mon, 12 Mar 2018 13:39:17 +0100
Ludovic Courtès <ludo <at> gnu.org> wrote:

> +       (when modprobe
> +         ;; Tell the kernel to invoke MODPROBE.
> +         (call-with-output-file "/proc/sys/kernel/modprobe"
> +           (lambda (port)
> +             (display modprobe port))))
> +
>         (display "loading kernel modules...\n")

There's a race because Linux could just start loading modules
immediately before boot-system even starts / executed far enough.

That was why earlier I kept /sbin/modprobe as the name (it's the
default Linux uses) and made sure that the file /sbin/modprobe is already
in the initrd (as opposed to created by boot-system).

The race didn't happen to me yet (and I doubt that Linux has modules
fine-grained enough that that's a problem in practise) - but I would prefer
it not to be possible.

WDYT?

> +          (define aliases
> +            ;; The list of aliases we are asked to load.
> +            (filter-map (match-lambda
> +                          (('argument . alias) alias)
> +                          (_ #f))
> +                        options))

Without "-a", it's not supposed to be a list of aliases.  It's supposed to be
one alias and the remainder are parameters supposed to be passed to that
kernel module.

(I didn't know that before).

When Linux invokes /sbin/modprobe, it doesn't pass "-a" but it doesn't
pass any module parameters either - it just passes one module alias.

For clarity and in order that we don't have weird problems later (trying to
load a module named "verbose=1" or something :) ), let's just accept one alias
for now.

>+(define* (modprobe-program linux-module-directory #:key
>+                           (guile %guile-static-stripped))
[...]
> +            (call-with-output-file "/dev/kmsg"
> +              (lambda (port)
> +                (format port "modprobe[~a]: aliases ~s; modules ~s; args ~s~%"
> +                        (getpid) aliases modules (program-arguments))))

Hmm, logging is nice!

But with /dev/kmsg one write(2) syscall needs to pass the entire message.

What do you think about additionally calling setvbuf with 1024 ?  Because:

kernel/printk/printk.c: #define LOG_LINE_MAX         (1024 - PREFIX_MAX)

> +            (when (< (length modules) (length aliases))
> +              (error "alias resolution failed" aliases))

The original modprobe loads as many modules as it can and possibly sets an
(always the same) error flag (and continues loading the other modules).

Then modprobe's exit status is that error flag.

This is moot because we shouldn't accept more than one alias - see above.

> +          (write-module-alias-database #$output))))

One of the devnames created statically is the one for btrfs, so not writing or
using devnames is not going to end well.

If it did work without it I'd be even more on guard :)

(I'd also copy the modules.builtin (from Linux).
 Also, what happens if we load a module which has as dependency a builtin?
 Will we try to load the builtin as a .ko file and fail the entire thing?)

> +         ;; We may need to lazy-load modules.  The initrd installs a

Nitpick: "kernel modules" (to clarify).
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 21:11:02 GMT) Full text and rfc822 format available.

Message #311 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v10 3/6] linux-modules: Add
 'load-needed-linux-modules'.
Date: Mon, 12 Mar 2018 22:10:39 +0100
Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

>> +        (let* ((alias   (string-trim-right
>> +                         (call-with-input-file file get-string-all)))
>
> Hmm... doesn't this trim other things than newline?

Yes, it trims whitespace.

> A quick internet search shows that most people do that.  Huh.

Yeah I’d think that whitespace is non-significant in these files, but in
practice they’re always generated anyway.

Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 21:13:01 GMT) Full text and rfc822 format available.

Message #314 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own
 'modprobe' program.
Date: Mon, 12 Mar 2018 22:12:26 +0100
[Message part 1 (text/plain, inline)]
> That was why earlier I kept /sbin/modprobe as the name (it's the
> default Linux uses) and made sure that the file /sbin/modprobe is already
> in the initrd (as opposed to created by boot-system).

Note: The simplest change would be to keep this patch as-is but add 

    ;; Make sure that Linux can modprobe even before boot-system started up.
    (when modprobe
      (mkdir-p "sbin")
      (symlink modprobe "sbin/modprobe")
      (compile-to-cache "sbin/modprobe")) ; Note: without this line booting is dead slow. [Maybe in this case that wouldn't be so bad - since boot-system switches away immediately anyway :)]

to gnu/build/linux-initrd.scm
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 21:14:02 GMT) Full text and rfc822 format available.

Message #317 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [PATCH v10 1/6] linux-modules: Add "modules.alias" writer.
Date: Mon, 12 Mar 2018 22:13:26 +0100
Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

>> +(define %not-dash
>> +  (char-set-complement (char-set #\-)))
>
> Is that used?

Oops, not until patch 6/6.  I’ve moved it there.

Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 22:16:02 GMT) Full text and rfc822 format available.

Message #320 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own
 'modprobe' program.
Date: Mon, 12 Mar 2018 23:14:59 +0100
Howdy!

Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

> On Mon, 12 Mar 2018 13:39:17 +0100
> Ludovic Courtès <ludo <at> gnu.org> wrote:
>
>> +       (when modprobe
>> +         ;; Tell the kernel to invoke MODPROBE.
>> +         (call-with-output-file "/proc/sys/kernel/modprobe"
>> +           (lambda (port)
>> +             (display modprobe port))))
>> +
>>         (display "loading kernel modules...\n")
>
> There's a race because Linux could just start loading modules
> immediately before boot-system even starts / executed far enough.
>
> That was why earlier I kept /sbin/modprobe as the name (it's the
> default Linux uses) and made sure that the file /sbin/modprobe is already
> in the initrd (as opposed to created by boot-system).
>
> The race didn't happen to me yet (and I doubt that Linux has modules
> fine-grained enough that that's a problem in practise) - but I would prefer
> it not to be possible.
>
> WDYT?

You’re right, I wondered what would happen.  I’ve restored the
/sbin/modprobe symlink in v11.

>> +          (define aliases
>> +            ;; The list of aliases we are asked to load.
>> +            (filter-map (match-lambda
>> +                          (('argument . alias) alias)
>> +                          (_ #f))
>> +                        options))
>
> Without "-a", it's not supposed to be a list of aliases.  It's supposed to be
> one alias and the remainder are parameters supposed to be passed to that
> kernel module.
>
> (I didn't know that before).
>
> When Linux invokes /sbin/modprobe, it doesn't pass "-a" but it doesn't
> pass any module parameters either - it just passes one module alias.
>
> For clarity and in order that we don't have weird problems later (trying to
> load a module named "verbose=1" or something :) ), let's just accept one alias
> for now.

Indeed, done in v11.

>>+(define* (modprobe-program linux-module-directory #:key
>>+                           (guile %guile-static-stripped))
> [...]
>> +            (call-with-output-file "/dev/kmsg"
>> +              (lambda (port)
>> +                (format port "modprobe[~a]: aliases ~s; modules ~s; args ~s~%"
>> +                        (getpid) aliases modules (program-arguments))))
>
> Hmm, logging is nice!

I didn’t mean to leave it, but after all, it’s pretty useful.  :-)

> But with /dev/kmsg one write(2) syscall needs to pass the entire message.
>
> What do you think about additionally calling setvbuf with 1024 ?  Because:

Sure.

>> +          (write-module-alias-database #$output))))
>
> One of the devnames created statically is the one for btrfs, so not writing or
> using devnames is not going to end well.

We’re fine because btrfs, 9p, overlay, etc. all have an “fs-btrfs”,
“fs-9p”, etc. alias, which shows up in “modules.alias”.  No need for
“modules.devname” AFAICS.

> (I'd also copy the modules.builtin (from Linux).
>  Also, what happens if we load a module which has as dependency a builtin?
>  Will we try to load the builtin as a .ko file and fail the entire thing?)

The dependency of a builtin is necessarily a builtin, and the kernel
won’t invoke modprobe for a builtin, will it?  IOW, I think there’s no
problem here.  :-)

>> +         ;; We may need to lazy-load modules.  The initrd installs a
>
> Nitpick: "kernel modules" (to clarify).

Fixed.

Thanks for the insightful review!

Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 22:17:01 GMT) Full text and rfc822 format available.

Message #323 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v11 1/6] linux-modules: Add "modules.alias" writer.
Date: Mon, 12 Mar 2018 23:15:36 +0100
From: Danny Milosavljevic <dannym <at> scratchpost.org>

* gnu/build/linux-modules.scm (write-module-alias-database): New
procedure.
(%not-dash): New variable.

Co-authored-by: Ludovic Courtès <ludo <at> gnu.org>
---
 gnu/build/linux-modules.scm | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 4fe673cca..b444e6f90 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2014, 2016, 2018 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe <at> gmail.com>
+;;; Copyright © 2018 Danny Milosavljevic <dannym <at> scratchpost.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -21,6 +22,7 @@
   #:use-module (guix elf)
   #:use-module (guix glob)
   #:use-module (guix build syscalls)
+  #:use-module ((guix build utils) #:select (find-files))
   #:use-module (rnrs io ports)
   #:use-module (rnrs bytevectors)
   #:use-module (srfi srfi-1)
@@ -41,7 +43,8 @@
 
             device-module-aliases
             known-module-aliases
-            matching-modules))
+            matching-modules
+            write-module-alias-database))
 
 ;;; Commentary:
 ;;;
@@ -380,4 +383,22 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
                       module)))
               known-aliases))
 
+(define (write-module-alias-database directory)
+  "Traverse the '.ko' files in DIRECTORY and create the corresponding
+'modules.alias' file."
+  (define aliases
+    (map (lambda (file)
+           (cons (file-name->module-name file) (module-aliases file)))
+         (find-files directory "\\.ko$")))
+
+  (call-with-output-file (string-append directory "/modules.alias")
+    (lambda (port)
+      (display "# Aliases extracted from modules themselves.\n" port)
+      (for-each (match-lambda
+                  ((module . aliases)
+                   (for-each (lambda (alias)
+                               (format port "alias ~a ~a\n" alias module))
+                             aliases)))
+                aliases))))
+
 ;;; linux-modules.scm ends here
-- 
2.16.2





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 22:17:02 GMT) Full text and rfc822 format available.

Message #326 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 30604 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH v11 2/6] linux-modules: Add
 'load-linux-modules-from-directory'.
Date: Mon, 12 Mar 2018 23:15:37 +0100
* gnu/build/linux-modules.scm (load-linux-modules-from-directory): New
procedure.
* gnu/build/linux-boot.scm (boot-system)[lookup-module]: Remove.
Use 'load-linux-modules-from-directory' instead.
---
 gnu/build/linux-boot.scm    |  9 ++-------
 gnu/build/linux-modules.scm | 12 ++++++++++++
 2 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index 18d87260a..df0b2b2d1 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -469,10 +469,6 @@ upon error."
              mounts)
         "ext4"))
 
-  (define (lookup-module name)
-    (string-append linux-module-directory "/"
-                   (ensure-dot-ko name)))
-
   (display "Welcome, this is GNU's early boot Guile.\n")
   (display "Use '--repl' for an initrd REPL.\n\n")
 
@@ -487,9 +483,8 @@ upon error."
          (start-repl))
 
        (display "loading kernel modules...\n")
-       (for-each (cut load-linux-module* <>
-                      #:lookup-module lookup-module)
-                 (map lookup-module linux-modules))
+       (load-linux-modules-from-directory linux-modules
+                                          linux-module-directory)
 
        (when qemu-guest-networking?
          (unless (configure-qemu-networking)
diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index b444e6f90..0084f972b 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -38,6 +38,7 @@
             modules-loaded
             module-loaded?
             load-linux-module*
+            load-linux-modules-from-directory
 
             current-module-debugging-port
 
@@ -232,6 +233,17 @@ appears in BLACK-LIST are not loaded."
              (or (and recursive? (= EEXIST (system-error-errno args)))
                  (apply throw args)))))))
 
+(define (load-linux-modules-from-directory modules directory)
+  "Load MODULES and their dependencies from DIRECTORY, a directory containing
+the '.ko' files.  The '.ko' suffix is automatically added to MODULES if
+needed."
+  (define (lookup-module name)
+    (string-append directory "/" (ensure-dot-ko name)))
+
+  (for-each (cut load-linux-module* <>
+                 #:lookup-module lookup-module)
+            (map lookup-module modules)))
+
 
 ;;;
 ;;; Device modules.
-- 
2.16.2





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 22:17:02 GMT) Full text and rfc822 format available.

Message #329 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 30604 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH v11 3/6] linux-modules: Add 'load-needed-linux-modules'.
Date: Mon, 12 Mar 2018 23:15:38 +0100
Based on a patch by Danny Milosavljevic <dannym <at> scratchpost.org>.

* gnu/build/linux-modules.scm (load-needed-linux-modules): New
procedure.
---
 gnu/build/linux-modules.scm | 34 +++++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 0084f972b..55161e026 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -30,6 +30,7 @@
   #:use-module (ice-9 vlist)
   #:use-module (ice-9 match)
   #:use-module (ice-9 rdelim)
+  #:use-module (ice-9 ftw)
   #:export (dot-ko
             ensure-dot-ko
             module-aliases
@@ -45,7 +46,8 @@
             device-module-aliases
             known-module-aliases
             matching-modules
-            write-module-alias-database))
+            write-module-alias-database
+            load-needed-linux-modules))
 
 ;;; Commentary:
 ;;;
@@ -413,4 +415,34 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
                              aliases)))
                 aliases))))
 
+(define (load-needed-linux-modules module-directory)
+  "Examine /sys/devices to find out which modules are needed and load those we
+have in MODULE-DIRECTORY.  Return the list of modules loaded, not including
+dependencies.
+
+Note: loading modules leads to the creation of new entries in /sys/devices,
+which is why we need traversal and loading to be interleaved.  If we walked
+/sys/devices and *then* loaded modules, we'd miss the entries added as a
+side-effect and would thus need to traverse /sys/devices again."
+  (define aliases
+    (known-module-aliases
+     (string-append module-directory "/modules.alias")))
+
+  (define (enter? director stat result) result)
+  (define (down directory stat result) result)
+  (define (up directory stat result) result)
+  (define (skip entry stat result) result)
+  (define (error name stat errno result) result)
+  (define (leaf file stat result)
+    (if (string=? (basename file) "modalias")
+        (let* ((alias   (string-trim-right
+                         (call-with-input-file file get-string-all)))
+               (modules (matching-modules alias aliases)))
+          (load-linux-modules-from-directory modules
+                                             module-directory)
+          (append modules result))
+        result))
+
+  (file-system-fold enter? leaf down up skip error #t "/sys/devices"))
+
 ;;; linux-modules.scm ends here
-- 
2.16.2





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 22:17:03 GMT) Full text and rfc822 format available.

Message #332 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>,
 Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH v11 4/6] vm: Make the virtio-blk uniquely identifiable in /sys.
Date: Mon, 12 Mar 2018 23:15:39 +0100
From: Danny Milosavljevic <dannym <at> scratchpost.org>

* gnu/build/vm.scm (load-in-linux-vm): Set virtio-blk pci addr to 0x10.
* gnu/system/vm.scm (common-qemu-options): Set virtio-blk pci addr to 0x10.

Signed-off-by: Ludovic Courtès <ludo <at> gnu.org>
---
 gnu/build/vm.scm  | 6 +++++-
 gnu/system/vm.scm | 7 ++++++-
 2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/gnu/build/vm.scm b/gnu/build/vm.scm
index fe003ea45..773b738ae 100644
--- a/gnu/build/vm.scm
+++ b/gnu/build/vm.scm
@@ -154,7 +154,11 @@ the #:references-graphs parameter of 'derivation'."
                                            builder)
                   (append
                    (if make-disk-image?
-                       `("-device" "virtio-blk,drive=myhd"
+
+                       ;; QEMU seems to have a bug that it doesn't assign PCI
+                       ;; IDs properly, so force the address of our virtio-blk
+                       ;; device.
+                       `("-device" "virtio-blk-pci,addr=0x10,drive=myhd"
                          "-drive" ,(string-append "if=none,file=" output
                                                   ",format=" disk-image-format
                                                   ",id=myhd"))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 91ff32ce9..4360adf15 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -677,7 +677,12 @@ with '-virtfs' options for the host file systems listed in SHARED-FS."
 
      #$@(map virtfs-option shared-fs)
      "-vga std"
-     (format #f "-drive file=~a,if=virtio,cache=writeback,werror=report,readonly"
+
+     ;; QEMU seems to have a bug that it doesn't assign PCI IDs properly, so
+     ;; force the address of our virtio-blk device.
+     "-device" "virtio-blk-pci,addr=0x10,drive=myhd"
+
+     (format #f "-drive id=myhd,file=~a,if=none,cache=writeback,werror=report,readonly"
              #$image)))
 
 (define* (system-qemu-image/shared-store-script os
-- 
2.16.2





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 22:17:03 GMT) Full text and rfc822 format available.

Message #335 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: [PATCH v11 5/6] linux-initrd: Provide our own 'modprobe' program.
Date: Mon, 12 Mar 2018 23:15:40 +0100
From: Danny Milosavljevic <dannym <at> scratchpost.org>

This allows us to load modules on demand when the kernel asks for them.

* gnu/system/linux-initrd.scm (modprobe-program): New variable.
(flat-linux-module-directory): Call 'write-module-alias-database'.
(raw-initrd): Pass #:modprobe to 'boot-system' and to 'expression->initrd'.
(expression->initrd): Copy "closure" to $out/references.  Add #:modprobe
and pass it to 'build-initrd'.
* gnu/build/linux-initrd.scm (build-initrd): Add #:modprobe and honor
it.
* gnu/build/linux-boot.scm (boot-system): Add #:modprobe and honor it.
Call 'load-needed-linux-modules'.
* gnu/system/vm.scm (qemu-image): Add #:linux parameter.  Define
'modprobe-wrapper' and pass it to 'activate-modprobe'.  Pass #:linux to
'expression->derivation-in-linux-vm'.

Co-authored-by: Ludovic Courtès <ludo <at> gnu.org>
---
 gnu/build/linux-boot.scm    | 13 +++++-
 gnu/build/linux-initrd.scm  | 16 ++++++--
 gnu/system/linux-initrd.scm | 97 ++++++++++++++++++++++++++++++++++++++++-----
 gnu/system/vm.scm           | 21 ++++++++++
 4 files changed, 133 insertions(+), 14 deletions(-)

diff --git a/gnu/build/linux-boot.scm b/gnu/build/linux-boot.scm
index df0b2b2d1..eedc4bb9d 100644
--- a/gnu/build/linux-boot.scm
+++ b/gnu/build/linux-boot.scm
@@ -435,6 +435,7 @@ bailing out.~%root contents: ~s~%" (scandir "/"))
 
 
 (define* (boot-system #:key
+                      modprobe
                       (linux-modules '())
                       linux-module-directory
                       qemu-guest-networking?
@@ -449,6 +450,9 @@ QEMU-GUEST-NETWORKING? is true, calling PRE-MOUNT, mounting the file systems
 specified in MOUNTS, and finally booting into the new root if any.  The initrd
 supports kernel command-line options '--load', '--root', and '--repl'.
 
+MODPROBE must be #f or a program to install as the modprobe program that the
+kernel will invoke when it needs to load modules.
+
 Mount the root file system, specified by the '--root' command-line argument,
 if any.
 
@@ -482,9 +486,14 @@ upon error."
        (when (member "--repl" args)
          (start-repl))
 
+       (when modprobe
+         ;; Tell the kernel to invoke MODPROBE.
+         (call-with-output-file "/proc/sys/kernel/modprobe"
+           (lambda (port)
+             (display modprobe port))))
+
        (display "loading kernel modules...\n")
-       (load-linux-modules-from-directory linux-modules
-                                          linux-module-directory)
+       (load-needed-linux-modules linux-module-directory)
 
        (when qemu-guest-networking?
          (unless (configure-qemu-networking)
diff --git a/gnu/build/linux-initrd.scm b/gnu/build/linux-initrd.scm
index c65b5aacf..4fa2bee7d 100644
--- a/gnu/build/linux-initrd.scm
+++ b/gnu/build/linux-initrd.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2013, 2014, 2015 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2013, 2014, 2015, 2018 Ludovic Courtès <ludo <at> gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -107,12 +107,18 @@ This is similar to what 'compiled-file-name' in (system base compile) does."
 
 (define* (build-initrd output
                        #:key
-                       guile init
+                       guile init modprobe
                        (references-graphs '())
                        (gzip "gzip"))
   "Write an initial RAM disk (initrd) to OUTPUT.  The initrd starts the script
 at INIT, running GUILE.  It contains all the items referred to by
-REFERENCES-GRAPHS."
+REFERENCES-GRAPHS.
+
+When MODPROBE is true, make /sbin/modprobe a symlink to it.  This is useful
+because Linux invokes 'modprobe' when it needs to load a module and its
+default file name is '/sbin/modprobe' (see 'call_modprobe' in kernel/kmod.c).
+Creating this symlink allows us to make sure there's no time window during
+which 'modprobe' is unavailable."
   (mkdir "contents")
 
   ;; Copy the closures of all the items referenced in REFERENCES-GRAPHS.
@@ -131,6 +137,10 @@ REFERENCES-GRAPHS."
     (symlink (string-append guile "/bin/guile") "proc/self/exe")
     (readlink "proc/self/exe")
 
+    (when modprobe
+      (mkdir-p "sbin")
+      (symlink modprobe "sbin/modprobe"))
+
     ;; Reset the timestamps of all the files that will make it in the initrd.
     (for-each (lambda (file)
                 (unless (eq? 'symlink (stat:type (lstat file)))
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 1eb5f5130..7a167146f 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -63,16 +63,87 @@
 ;;;
 ;;; Code:
 
+(define* (modprobe-program linux-module-directory #:key
+                           (guile %guile-static-stripped))
+  "Return a minimal implementation of 'modprobe' for our initrd that looks up
+modules in LINUX-MODULE-DIRECTORY.  This program will be invoked by the kernel
+when modules need to be loaded."
+  (define program
+    (with-imported-modules (source-module-closure
+                            '((gnu build linux-modules)))
+      #~(begin
+          (use-modules (gnu build linux-modules)
+                       (ice-9 match)
+                       (srfi srfi-1)
+                       (srfi srfi-26)
+                       (srfi srfi-37))
+
+          (define option-spec
+            (list (option '(#\q "quiet") #f #f
+                          (lambda (opt name arg result)
+                            (alist-cons 'quiet? #t result)))))
+
+          (define options
+            ;; Alist of options and non-option arguments.
+            (args-fold (cdr (program-arguments))
+                       option-spec
+                       (lambda (opt name arg result)
+                         (error "unrecognized option" name))
+                       (lambda (arg result)
+                         (alist-cons 'argument arg result))
+                       '()))
+
+          (define alias
+            ;; The alias we are asked to load.  The remaining arguments are
+            ;; module parameters.  In practice the kernel doesn't pass module
+            ;; parameters so we ignore them here.
+            (any (match-lambda
+                   (('argument . alias) alias)
+                   (_ #f))
+                 options))
+
+          (define linux-module-directory
+            ;; The module directory.  Note: We expect a flat directory here.
+            #$linux-module-directory)
+
+          (define %known-aliases
+            ;; The alias database.
+            (known-module-aliases
+             (string-append linux-module-directory "/modules.alias")))
+
+          (when (assq-ref options 'quiet?)
+            (current-error-port (%make-void-port "w"))
+            (current-output-port (%make-void-port "w")))
+
+          (let ((modules (matching-modules alias %known-aliases)))
+            (call-with-output-file "/dev/kmsg"
+              (lambda (port)
+                (setvbuf port 'block 1024)
+                (format port "modprobe[~a]: alias ~s; modules ~s; args ~s~%"
+                        (getpid) alias modules (program-arguments))))
+
+            (when (null? modules)
+              (error "alias resolution failed" alias))
+
+            (load-linux-modules-from-directory modules
+                                               linux-module-directory)))))
+
+  (program-file "modprobe" program #:guile guile))
 
 (define* (expression->initrd exp
                              #:key
+                             modprobe
                              (guile %guile-static-stripped)
                              (gzip gzip)
                              (name "guile-initrd")
                              (system (%current-system)))
   "Return a derivation that builds a Linux initrd (a gzipped cpio archive)
 containing GUILE and that evaluates EXP, a G-expression, upon booting.  All
-the derivations referenced by EXP are automatically copied to the initrd."
+the derivations referenced by EXP are automatically copied to the initrd.
+
+When MODPROBE is true, '/sbin/modprobe' is created as a symlink pointing to
+it.  This allows Linux to call out to MODPROBE as soon as it boots if it needs
+to load modules."
 
   ;; General Linux overview in `Documentation/early-userspace/README' and
   ;; `Documentation/filesystems/ramfs-rootfs-initramfs.txt'.
@@ -89,20 +160,21 @@ the derivations referenced by EXP are automatically copied to the initrd."
           (mkdir #$output)
 
           ;; The guile used in the initrd must be present in the store, so
-          ;; that module loading works once the root is switched.
+          ;; that module loading works once the root is switched.  Similarly,
+          ;; the 'modprobe' program installed in /proc/sys/kernel/modprobe by
+          ;; the initrd, if any, must be present after switch root.
           ;;
-          ;; To ensure that is the case, add an explicit reference to the
-          ;; guile package used in the initrd to the output.
+          ;; To ensure that is the case, add an explicit reference to these in
+          ;; the output.
           ;;
-          ;; This fixes guix-patches bug #28399, "Fix mysql activation, and
+          ;; This fixes <https://bugs.gnu.org/28399>, "Fix mysql activation, and
           ;; add a basic test".
-          (call-with-output-file (string-append #$ output "/references")
-            (lambda (port)
-              (simple-format port "~A\n" #$guile)))
+          (copy-file "closure" (string-append #$output "/references"))
 
           (build-initrd (string-append #$output "/initrd")
                         #:guile #$guile
                         #:init #$init
+                        #:modprobe #$modprobe
                         ;; Copy everything INIT refers to into the initrd.
                         #:references-graphs '("closure")
                         #:gzip (string-append #$gzip "/bin/gzip")))))
@@ -153,7 +225,9 @@ MODULES and taken from LINUX."
                       (copy-file module
                                  (string-append #$output "/"
                                                 (basename module))))
-                    (delete-duplicates modules)))))
+                    (delete-duplicates modules))
+
+          (write-module-alias-database #$output))))
 
   (computed-file "linux-modules" build-exp))
 
@@ -196,6 +270,9 @@ upon error."
   (define kodir
     (flat-linux-module-directory linux linux-modules))
 
+  (define modprobe
+    (modprobe-program kodir))
+
   (expression->initrd
    (with-imported-modules (source-module-closure
                            '((gnu build linux-boot)
@@ -229,9 +306,11 @@ upon error."
                                     (and #$@device-mapping-commands))
                       #:linux-modules '#$linux-modules
                       #:linux-module-directory '#$kodir
+                      #:modprobe #$modprobe
                       #:qemu-guest-networking? #$qemu-networking?
                       #:volatile-root? '#$volatile-root?
                       #:on-error '#$on-error)))
+   #:modprobe modprobe
    #:name "raw-initrd"))
 
 (define* (file-system-packages file-systems #:key (volatile-root? #f))
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 4360adf15..7f552bef8 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -249,6 +249,7 @@ INPUTS is a list of inputs (as for packages)."
 (define* (qemu-image #:key
                      (name "qemu-image")
                      (system (%current-system))
+                     (linux linux-libre)
                      (qemu qemu-minimal)
                      (disk-image-size 'guess)
                      (disk-image-format "qcow2")
@@ -275,18 +276,37 @@ INPUTS is a list of inputs (as for packages).  When COPY-INPUTS? is true, copy
 all of INPUTS into the image being built.  When REGISTER-CLOSURES? is true,
 register INPUTS in the store database of the image so that Guix can be used in
 the image."
+  (define modprobe-wrapper
+    ;; Wrapper for the 'modprobe' command that knows where modules live.
+    (let ((modprobe (file-append kmod "/bin/modprobe")))
+      (program-file "modprobe"
+                    #~(begin
+                        (setenv "LINUX_MODULE_DIRECTORY"
+                                #$(file-append linux "/lib/modules"))
+                        (apply execl #$modprobe
+                               (cons #$modprobe (cdr (command-line))))))))
+
+
   (expression->derivation-in-linux-vm
    name
    (with-imported-modules (source-module-closure '((gnu build bootloader)
                                                    (gnu build vm)
+                                                   (gnu build activation)
                                                    (guix build utils)))
      #~(begin
          (use-modules (gnu build bootloader)
                       (gnu build vm)
+                      ((gnu build activation) #:select (activate-modprobe))
                       (guix build utils)
                       (srfi srfi-26)
                       (ice-9 binary-ports))
 
+         ;; We may need to lazy-load Linux modules.  The initrd installs a
+         ;; 'modprobe' that can only search through the modules available in
+         ;; the initrd, but here we want to be able to use all the modules of
+         ;; LINUX.  Thus, install a real 'modprobe'.
+         (activate-modprobe #$modprobe-wrapper)
+
          (let ((inputs
                 '#$(append (list qemu parted e2fsprogs dosfstools)
                            (map canonical-package
@@ -361,6 +381,7 @@ the image."
                                    #$(bootloader-installer bootloader))
              (reboot)))))
    #:system system
+   #:linux linux
    #:make-disk-image? #t
    #:disk-image-size disk-image-size
    #:disk-image-format disk-image-format
-- 
2.16.2





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 22:17:04 GMT) Full text and rfc822 format available.

Message #338 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 30604 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH v11 6/6] linux-modules: Add "modules.devname" writer.
Date: Mon, 12 Mar 2018 23:15:41 +0100
* gnu/build/linux-modules.scm (aliases->device-tuple)
(write-module-device-database): New procedures.

Co-authored-by: Danny Milosavljevic <dannym <at> scratchpost.org>.
---
 gnu/build/linux-modules.scm | 51 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)

diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 55161e026..d3ba2c60a 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -47,6 +47,8 @@
             known-module-aliases
             matching-modules
             write-module-alias-database
+            write-module-device-database
+
             load-needed-linux-modules))
 
 ;;; Commentary:
@@ -415,6 +417,55 @@ ALIAS is a string like \"scsi:t-0x00\" as returned by
                              aliases)))
                 aliases))))
 
+(define (aliases->device-tuple aliases)
+  "Traverse ALIASES, a list of module aliases, and search for
+\"char-major-M-N\", \"block-major-M-N\", or \"devname:\" aliases.  When they
+are found, return a tuple (DEVNAME TYPE MAJOR MINOR), otherwise return #f."
+  (define (char/block-major? alias)
+    (or (string-prefix? "char-major-" alias)
+        (string-prefix? "block-major-" alias)))
+
+  (define (char/block-major->tuple alias)
+    (match (string-tokenize alias %not-dash)
+      ((type "major" (= string->number major) (= string->number minor))
+       (list (match type
+               ("char" "c")
+               ("block" "b"))
+             major minor))))
+
+  (let* ((devname     (any (lambda (alias)
+                             (and (string-prefix? "devname:" alias)
+                                  (string-drop alias 8)))
+                           aliases))
+         (major/minor (match (find char/block-major? aliases)
+                        (#f #f)
+                        (str (char/block-major->tuple str)))))
+    (and devname major/minor
+         (cons devname major/minor))))
+
+(define %not-dash
+  (char-set-complement (char-set #\-)))
+
+(define (write-module-device-database directory)
+  "Traverse the '.ko' files in DIRECTORY and create the corresponding
+'modules.devname' file.  This file contains information about modules that can
+be loaded on-demand, such as file system modules."
+  (define aliases
+    (filter-map (lambda (file)
+                  (match (aliases->device-tuple (module-aliases file))
+                    (#f #f)
+                    (tuple (cons (file-name->module-name file) tuple))))
+                (find-files directory "\\.ko$")))
+
+  (call-with-output-file (string-append "/tmp" "/modules.devname")
+    (lambda (port)
+      (display "# Device nodes to trigger on-demand module loading.\n" port)
+      (for-each (match-lambda
+                  ((module devname type major minor)
+                   (format port "~a ~a ~a~a:~a~%"
+                           module devname type major minor)))
+                aliases))))
+
 (define (load-needed-linux-modules module-directory)
   "Examine /sys/devices to find out which modules are needed and load those we
 have in MODULE-DIRECTORY.  Return the list of modules loaded, not including
-- 
2.16.2





Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 22:49:02 GMT) Full text and rfc822 format available.

Message #341 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [PATCH v11 1/6] linux-modules: Add "modules.alias" writer.
Date: Mon, 12 Mar 2018 23:48:14 +0100
[Message part 1 (text/plain, inline)]
Hi Ludo,

On Mon, 12 Mar 2018 23:15:36 +0100
Ludovic Courtès <ludo <at> gnu.org> wrote:

> (%not-dash): New variable.

Old commit message?

Otherwise LGTM!
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 22:49:02 GMT) Full text and rfc822 format available.

Message #344 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v11 2/6] linux-modules: Add
 'load-linux-modules-from-directory'.
Date: Mon, 12 Mar 2018 23:48:36 +0100
[Message part 1 (text/plain, inline)]
LGTM!
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 22:50:02 GMT) Full text and rfc822 format available.

Message #347 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [PATCH v11 4/6] vm: Make the virtio-blk uniquely identifiable
 in /sys.
Date: Mon, 12 Mar 2018 23:49:22 +0100
[Message part 1 (text/plain, inline)]
LGTM!
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 22:56:02 GMT) Full text and rfc822 format available.

Message #350 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v11 6/6] linux-modules: Add
 "modules.devname" writer.
Date: Mon, 12 Mar 2018 23:55:46 +0100
[Message part 1 (text/plain, inline)]
> +  (call-with-output-file (string-append "/tmp" "/modules.devname")
                                           ^^^^^^ directory

Otherwise LGTM!
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 12 Mar 2018 23:01:02 GMT) Full text and rfc822 format available.

Message #353 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v11 3/6] linux-modules: Add
 'load-needed-linux-modules'.
Date: Tue, 13 Mar 2018 00:00:31 +0100
[Message part 1 (text/plain, inline)]
> +(define (load-needed-linux-modules module-directory)
[...]
> +  (file-system-fold enter? leaf down up skip error #t "/sys/devices"))

Isn't #t a strange initial value?  Shouldn't it be '() ?
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 13 Mar 2018 08:51:01 GMT) Full text and rfc822 format available.

Message #356 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: Re: [bug#30604] [PATCH v11 5/6] linux-initrd: Provide our own
 'modprobe' program.
Date: Tue, 13 Mar 2018 09:50:40 +0100
[Message part 1 (text/plain, inline)]
Hello!

I’m using this on the bare metal.  \o/

I had to make this change so that dm-crypt would be loaded once I’ve
entered my passphrase:

[Message part 2 (text/x-patch, inline)]
diff --git a/gnu/system/linux-initrd.scm b/gnu/system/linux-initrd.scm
index 7a167146f..ef81fe718 100644
--- a/gnu/system/linux-initrd.scm
+++ b/gnu/system/linux-initrd.scm
@@ -115,16 +115,19 @@ when modules need to be loaded."
             (current-error-port (%make-void-port "w"))
             (current-output-port (%make-void-port "w")))
 
-          (let ((modules (matching-modules alias %known-aliases)))
+          ;; Look up ALIAS in %KNOWN-ALIASES first, and then, if it fails,
+          ;; assume that ALIAS is a module name (some modules such as
+          ;; 'dm-crypt' don't have an alias.)  Note that
+          ;; 'kmod_module_new_from_lookup' uses a different search order.
+          (let ((modules (match (matching-modules alias %known-aliases)
+                           (()  (list alias))
+                           (lst lst))))
             (call-with-output-file "/dev/kmsg"
               (lambda (port)
                 (setvbuf port 'block 1024)
                 (format port "modprobe[~a]: alias ~s; modules ~s; args ~s~%"
                         (getpid) alias modules (program-arguments))))
 
-            (when (null? modules)
-              (error "alias resolution failed" alias))
-
             (load-linux-modules-from-directory modules
                                                linux-module-directory)))))
 
[Message part 3 (text/plain, inline)]
FWIW I have:

--8<---------------cut here---------------start------------->8---
modprobe[146]: alias "net-pf-38"; modules ("net-pf-38"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "net-pf-38")
modprobe[152]: alias "dm-crypt"; modules ("dm-crypt"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "dm-crypt")
modprobe[157]: alias "crypto-xts(aes)"; modules ("crypto-xts(aes)"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-xts(aes)")
modprobe[162]: alias "crypto-xts(aes)-all"; modules ("crypto-xts(aes)-all"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-xts(aes)-all")
modprobe[168]: alias "crypto-xts"; modules ("xts"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-xts")
nfo>  [1520929735.4127] manager[0x12950c0]: monitoring kernel firmware directory '/lib/firmware'.
modprobe[173]: alias "crypto-aes"; modules ("crypto-aes"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-aes")
modprobe[178]: alias "crypto-aes-all"; modules ("crypto-aes-all"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-aes-all")
modprobe[183]: alias "crypto-ecb(aes)"; modules ("crypto-ecb(aes)"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-ecb(aes)")
modprobe[188]: alias "crypto-ecb(aes)-all"; modules ("crypto-ecb(aes)-all"); args ("/gnu/store/v0ay302yib6j6ysr0pd1yphm00nlzkxh-modprobe" "-q" "--" "crypto-ecb(aes)-all")
--8<---------------cut here---------------end--------------->8---

Something annoying is that my external USB keyboard doesn’t work while
in the initrd (when I type my passphrase).  I can see that it’s detected
early on, before I type my passphrase:

--8<---------------cut here---------------start------------->8---
[    1.532237] usb 1-4.2: new low-speed USB device number 4 using xhci_hcd
[    1.648836] usb 1-4.2: New USB device found, idVendor=413c, idProduct=2105
[    1.649189] usb 1-4.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    1.649533] usb 1-4.2: Product: Dell USB Keyboard
[    1.649882] usb 1-4.2: Manufacturer: Dell
--8<---------------cut here---------------end--------------->8---

but its modules don’t get loaded until after eudev has been started, and
there’s no trace of a modprobe invocation for it.

--8<---------------cut here---------------start------------->8---
[   20.968938] input: Dell Dell USB Keyboard as /devices/pci0000:00/0000:00:14.0/usb1/1-4/1-4.2/1-4.2:1.0/0003:413C:2105.0001/input/input16
--8<---------------cut here---------------end--------------->8---

Any idea what we’re missing?

Thanks,
Ludo’.

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 13 Mar 2018 08:55:02 GMT) Full text and rfc822 format available.

Message #359 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own
 'modprobe' program.
Date: Tue, 13 Mar 2018 09:54:55 +0100
Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

>> That was why earlier I kept /sbin/modprobe as the name (it's the
>> default Linux uses) and made sure that the file /sbin/modprobe is already
>> in the initrd (as opposed to created by boot-system).
>
> Note: The simplest change would be to keep this patch as-is but add 
>
>     ;; Make sure that Linux can modprobe even before boot-system started up.
>     (when modprobe
>       (mkdir-p "sbin")
>       (symlink modprobe "sbin/modprobe")
>       (compile-to-cache "sbin/modprobe")) ; Note: without this line booting is dead slow. [Maybe in this case that wouldn't be so bad - since boot-system switches away immediately anyway :)]
>
> to gnu/build/linux-initrd.scm

Yeah, done.  Ideally ‘program-file’ would take care of this.  Future
work!

Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 13 Mar 2018 08:56:02 GMT) Full text and rfc822 format available.

Message #362 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v11 3/6] linux-modules: Add
 'load-needed-linux-modules'.
Date: Tue, 13 Mar 2018 09:55:54 +0100
Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

>> +(define (load-needed-linux-modules module-directory)
> [...]
>> +  (file-system-fold enter? leaf down up skip error #t "/sys/devices"))
>
> Isn't #t a strange initial value?  Shouldn't it be '() ?

Indeed!  Fixed, thanks.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 13 Mar 2018 08:57:02 GMT) Full text and rfc822 format available.

Message #365 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v11 6/6] linux-modules: Add "modules.devname"
 writer.
Date: Tue, 13 Mar 2018 09:56:39 +0100
Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

>> +  (call-with-output-file (string-append "/tmp" "/modules.devname")
>                                            ^^^^^^ directory
>
> Otherwise LGTM!

Oops, fixed!




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 13 Mar 2018 09:28:02 GMT) Full text and rfc822 format available.

Message #368 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own
 'modprobe' program.
Date: Tue, 13 Mar 2018 10:27:06 +0100
[Message part 1 (text/plain, inline)]
Hi Ludo,

On Mon, 12 Mar 2018 23:14:59 +0100
ludo <at> gnu.org (Ludovic Courtès) wrote:

> > One of the devnames created statically is the one for btrfs, so not writing or
> > using devnames is not going to end well.  
> 
> We’re fine because btrfs, 9p, overlay, etc. all have an “fs-btrfs”,
> “fs-9p”, etc. alias, which shows up in “modules.alias”.  No need for
> “modules.devname” AFAICS.

The other filesystems are not such a problem - but BTRFS has its own snapshotting/
multivolume functionality - and it's possible that someone accesses /dev/btrfs-control
"too early" for that.

I applied your patches to a fresh clone of guix master now, ran the btrfs-root-os
system check, and indeed I get (tried two rounds, happened both times):

$ make TESTS=btrfs-root-os check-system
[...]
Scanning for Btrfs filesystems
WARNING: failed to open /dev/btrfs-control, skipping device registration: No suy
ERROR: there are 1 errors while registering devices
File system check on /dev/vda2 failed; spawning Bourne-like REPL
GNU Guile 2.2.3
Copyright (C) 1995-2017 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
(can't evaluate anything here)

> > (I'd also copy the modules.builtin (from Linux).
> >  Also, what happens if we load a module which has as dependency a builtin?
> >  Will we try to load the builtin as a .ko file and fail the entire thing?)  
> 
> The dependency of a builtin is necessarily a builtin, 

Yes.

>and the kernel won’t invoke modprobe for a builtin, will it?  

I've read Linux's __request_module and I can't find where they
pre-check anything - neither whether there's already a builtin
nor whether it's loaded already.

They probably don't check.  But I'll read it again, more carefully...

(request_module isn't that popular so it makes sense for them not to complicate
the kernel by these checks when there are like 8 callers in total - and all on init)

>Thanks for the insightful review!

You're welcome :)
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 13 Mar 2018 09:29:01 GMT) Full text and rfc822 format available.

Message #371 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: Re: [bug#30604] [PATCH v11 5/6] linux-initrd: Provide our own
 'modprobe' program.
Date: Tue, 13 Mar 2018 10:28:52 +0100
[Message part 1 (text/plain, inline)]
ludo <at> gnu.org (Ludovic Courtès) skribis:

> Something annoying is that my external USB keyboard doesn’t work while
> in the initrd (when I type my passphrase).  I can see that it’s detected
> early on, before I type my passphrase:

I tried looking up “modalias” right when entering a /sys/devices
directory, like you did in the initial patch, but that didn’t help:

[Message part 2 (text/x-patch, inline)]
diff --git a/gnu/build/linux-modules.scm b/gnu/build/linux-modules.scm
index 4515839e9..213a8bbb9 100644
--- a/gnu/build/linux-modules.scm
+++ b/gnu/build/linux-modules.scm
@@ -480,19 +480,22 @@ side-effect and would thus need to traverse /sys/devices again."
      (string-append module-directory "/modules.alias")))
 
   (define (enter? director stat result) result)
-  (define (down directory stat result) result)
-  (define (up directory stat result) result)
-  (define (skip entry stat result) result)
-  (define (error name stat errno result) result)
-  (define (leaf file stat result)
-    (if (string=? (basename file) "modalias")
+  (define (down directory stat result)
+    ;; Check for "modalias" right when entering DIRECTORY instead of waiting
+    ;; for LEAF to be called.  XXX: Doesn't help!
+    (let ((modalias (string-append directory "/modalias")))
+      (if (file-exists? modalias)
           (let* ((alias   (string-trim-right
-                         (call-with-input-file file get-string-all)))
+                           (call-with-input-file modalias get-string-all)))
                  (modules (matching-modules alias aliases)))
             (load-linux-modules-from-directory modules
                                                module-directory)
             (append modules result))
-        result))
+          result)))
+  (define (up directory stat result) result)
+  (define (skip entry stat result) result)
+  (define (error name stat errno result) result)
+  (define (leaf file stat result) file stat result)
 
   (file-system-fold enter? leaf down up skip error '() "/sys/devices"))
 
[Message part 3 (text/plain, inline)]
Ludo’.

Severity set to 'important' from 'normal' Request was from ludo <at> gnu.org (Ludovic Courtès) to control <at> debbugs.gnu.org. (Tue, 13 Mar 2018 09:30:02 GMT) Full text and rfc822 format available.

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 13 Mar 2018 10:52:02 GMT) Full text and rfc822 format available.

Message #376 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own
 'modprobe' program.
Date: Tue, 13 Mar 2018 11:51:42 +0100
Hello,

Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

> On Mon, 12 Mar 2018 23:14:59 +0100
> ludo <at> gnu.org (Ludovic Courtès) wrote:
>
>> > One of the devnames created statically is the one for btrfs, so not writing or
>> > using devnames is not going to end well.  
>> 
>> We’re fine because btrfs, 9p, overlay, etc. all have an “fs-btrfs”,
>> “fs-9p”, etc. alias, which shows up in “modules.alias”.  No need for
>> “modules.devname” AFAICS.
>
> The other filesystems are not such a problem - but BTRFS has its own snapshotting/
> multivolume functionality - and it's possible that someone accesses /dev/btrfs-control
> "too early" for that.
>
> I applied your patches to a fresh clone of guix master now, ran the btrfs-root-os
> system check, and indeed I get (tried two rounds, happened both times):
>
> $ make TESTS=btrfs-root-os check-system
> [...]
> Scanning for Btrfs filesystems
> WARNING: failed to open /dev/btrfs-control, skipping device registration: No suy
> ERROR: there are 1 errors while registering devices
> File system check on /dev/vda2 failed; spawning Bourne-like REPL
> GNU Guile 2.2.3
> Copyright (C) 1995-2017 Free Software Foundation, Inc.
>
> Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
> This program is free software, and you are welcome to redistribute it
> under certain conditions; type `,show c' for details.
>
> Enter `,help' for help.

Do you see a modprobe invocation for “fs-btrfs” or “block-major-xxx”
before?

>> > (I'd also copy the modules.builtin (from Linux).
>> >  Also, what happens if we load a module which has as dependency a builtin?
>> >  Will we try to load the builtin as a .ko file and fail the entire thing?)  
>> 
>> The dependency of a builtin is necessarily a builtin, 
>
> Yes.
>
>>and the kernel won’t invoke modprobe for a builtin, will it?  
>
> I've read Linux's __request_module and I can't find where they
> pre-check anything - neither whether there's already a builtin
> nor whether it's loaded already.
>
> They probably don't check.  But I'll read it again, more carefully...
>
> (request_module isn't that popular so it makes sense for them not to complicate
> the kernel by these checks when there are like 8 callers in total - and all on init)

Right, and the worst that can happen is that modprobe will fail, but
that’s fine because the functionality is already there anyway.

Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 13 Mar 2018 12:07:01 GMT) Full text and rfc822 format available.

Message #379 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own
 'modprobe' program.
Date: Tue, 13 Mar 2018 13:05:57 +0100
[Message part 1 (text/plain, inline)]
Hi Ludo,

On Tue, 13 Mar 2018 11:51:42 +0100
ludo <at> gnu.org (Ludovic Courtès) wrote:

> Right, and the worst that can happen is that modprobe will fail, but
> that’s fine because the functionality is already there anyway.

Yeah - but what does the kernel do when we return a non-zero exit code for something he wants to use? :)

./fs/filesystems.c:     if (!fs && (request_module("%.*s", len, name) == 0))
                                                                      ^^^^ uh oh



[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 13 Mar 2018 12:29:02 GMT) Full text and rfc822 format available.

Message #382 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v11 5/6] linux-initrd: Provide our own
 'modprobe' program.
Date: Tue, 13 Mar 2018 13:28:02 +0100
[Message part 1 (text/plain, inline)]
> I’m using this on the bare metal.  \o/

Hehe nice!

> -          (let ((modules (matching-modules alias %known-aliases)))
> +          ;; Look up ALIAS in %KNOWN-ALIASES first, and then, if it fails,
> +          ;; assume that ALIAS is a module name (some modules such as
> +          ;; 'dm-crypt' don't have an alias.)  Note that
> +          ;; 'kmod_module_new_from_lookup' uses a different search order.

I also wondered about the resolution order :)

At least module dependencies can't be aliases.

> +          (let ((modules (match (matching-modules alias %known-aliases)
> +                           (()  (list alias))
> +                           (lst lst))))

Yeah, same happened to me.  Did the same (back then in "resolve-alias").

So should be fine.

> Something annoying is that my external USB keyboard doesn’t work while
> in the initrd (when I type my passphrase).

Did it work before?  I remember that Andreas had a similar problem before all this.

Did you implement globs with [0-9] yet?  I remember there were some USB devices
using those :)

What's the modalias of your usb keyboard?

Does your initrd's modules.alias contain any patterns that are supposed to match?

(You can look into those via 
  cat initrd | gunzip -cd |cpio -i

- it will unpack)
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Tue, 13 Mar 2018 19:18:01 GMT) Full text and rfc822 format available.

Message #385 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: [bug#30604] [PATCH v10 5/6] linux-initrd: Provide our own
 'modprobe' program.
Date: Tue, 13 Mar 2018 20:17:02 +0100
[Message part 1 (text/plain, inline)]
Hi Ludo,

> Do you see a modprobe invocation for “fs-btrfs” or “block-major-xxx”
> before?

I can't get the logfile to show any modprobe log message in this case
(which is "make TESTS=btrfs-root-os check-system").

I had that problem before and I think it's because guix is
using some special /tmp/repl pipe for qemu here[1] and so the message
vanishes somewhere where I can't see it (that is, the guest does
everything correctly but the host loses it somewhere). 

The usual trick I use in this case which is "steal the cmdline and
the files from /proc/<qemu> and invoke it myself" doesn't work
either because of the pipe... hrmmmm...

I tried both /dev/kmsg and just not closing current-output-port etc.

I've double-checked that I've applied all v11 patches, including
the resolve-alias one (which I think is necessary anyway).

Help?

Log file attached.

[1] When it uses the VM to create the root fs.
[L.bz2 (application/x-bzip, attachment)]
[Message part 3 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sun, 18 Mar 2018 15:04:01 GMT) Full text and rfc822 format available.

Message #388 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: 30604 <at> debbugs.gnu.org
Cc: Danny Milosavljevic <dannym <at> scratchpost.org>
Subject: Keyboard detection before ‘cryptsetup’ runs
Date: Sun, 18 Mar 2018 16:03:06 +0100
Hi Danny,

ludo <at> gnu.org (Ludovic Courtès) skribis:

> Something annoying is that my external USB keyboard doesn’t work while
> in the initrd (when I type my passphrase).  I can see that it’s detected
> early on, before I type my passphrase:

While experimenting with this on the bare metal, it became clear that
it’s a timing issue: the keyboard is detected right after the cryptsetup
is displayed, so ‘load-needed-linux-modules’ didn’t have a chance to
load the relevant modules.

Similarly, if I boot with ‘--repl’, and I manually type
(load-needed-linux-modules …) from there, then the keyboard’s module
gets loaded as expected.  That’s because the device showed up in the
meantime and the kernel created a /sys entry for it.

In essence, we want a mini-udev and something akin to “udevadm settle”.
Merely calling ‘load-needed-linux-modules’ once isn’t enough.

One way to do that would be to have a separate thread that calls
‘load-needed-linux-modules’ as appropriate.  Ideally it would use
inotify on /sys like udev does, but a poor programmer’s version could
simply call ‘load-needed-linux-modules’ every half a second or so.

Alternately, before passing control to user code (pre-mount actions,
etc.), we could do a “settle” kind of thing: call
‘load-linux-modules-from-directory’ every 0.5 seconds until its result
is the same as before.  There’s still a risk of missing devices, and
those devices will never show up later because nobody’s monitoring /sys.
But then again, “udevadm settle” must have the same problem: it can’t
really know whether things have settled, I guess.

WDYT?

Thanks,
Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 23 Mar 2018 13:03:02 GMT) Full text and rfc822 format available.

Message #391 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: Keyboard detection before ‘cryptsetup’ runs
Date: Fri, 23 Mar 2018 14:02:31 +0100
[Message part 1 (text/plain, inline)]
Hi Ludo,

I thought about it for a bit...

On Sun, 18 Mar 2018 16:03:06 +0100
ludo <at> gnu.org (Ludovic Courtès) wrote:

> One way to do that would be to have a separate thread that calls
> ‘load-needed-linux-modules’ as appropriate.  Ideally it would use
> inotify on /sys like udev does, but a poor programmer’s version could
> simply call ‘load-needed-linux-modules’ every half a second or so.
> 
> Alternately, before passing control to user code (pre-mount actions,
> etc.), we could do a “settle” kind of thing: call
> ‘load-linux-modules-from-directory’ every 0.5 seconds until its result
> is the same as before.  There’s still a risk of missing devices, and
> those devices will never show up later because nobody’s monitoring /sys.
> But then again, “udevadm settle” must have the same problem: it can’t
> really know whether things have settled, I guess.

Yes, but udevd continues running and monitoring /sys anyway - so I guess
they don't really care (if I wrote it, I'd start up the monitoring first
and then traverse /sys for modalias - and make sure the modprobe is
idempotent).

I have to say what we have to do is much more complicated than I thought
it would be - and I kinda regret starting this patchset.  I didn't
want to make it less likely for people to boot than before :-(

I suggest we find out how the other distributions do it - if they do it.

A small bit of research shows that:

- Busybox uses a loop with timeout to try to mount over and over again,
depending on bus type [2].
- Alpine coldplugs twice [1].
- Fatdog has a "waitdev" boot parameter that specifies how long it sleeps
until it tries to mount the root.
- Redhat mkinitrd has an mkinitrd option like "--with=scsi_wait_scan" which
they use after they saw a modalias starting with "scsi:" - or add the
kernel option "scsi_mod.scan=sync" to the command line [3] (but they say that
this is unreliable now and one is supposed to use udev inside the initrd).
Not sure what they do for USB.

The Linux kernel itself has a "rootdelay" parameter which
was specifically introduced because of USB devices that took 15 s (!)
to show up.

All in all, we have alternative ways to continue:

(a) Drop this patchset/feature entirely because it's too unreliable.
(b) Monitor /sys using inotify in an extra thread/process.
(c) Include udev into the initrd and have it do its thing.
(d) Monitor the netlink socket in our own thread and play udev ourselves.
(e) Find out how to, for each bus type, detect when enumeration is finished.
Find out which buses are there.  Wait for them to finish enumerating.
(f) Always modprobe usb-hid first, no matter what.  I think this would
become a whack-a-mole thing fast.

I'd prefer either (a) or (b) for reliability and simplicity.  WDYT?

(inotify is Linux-only, but so is /sys/devices :P)

[1] https://git.alpinelinux.org/cgit/abuild/tree/initramfs-init?id=9154b0d9b7251cca900f9e834ac9af9ae2ae61d9 (scan_drivers)
[2] http://lists.busybox.net/pipermail/busybox/2015-March/082688.html
[3] https://bugzilla.redhat.com/show_bug.cgi?id=466607
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 23 Mar 2018 15:08:01 GMT) Full text and rfc822 format available.

Message #394 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: Keyboard detection before ‘cryptsetup’ runs
Date: Fri, 23 Mar 2018 16:07:02 +0100
[Message part 1 (text/plain, inline)]
> (b) Monitor /sys using inotify in an extra thread/process.

Oh - or loop { load-/sys ; sleep 1 }.  That's what you meant.
[Message part 2 (application/pgp-signature, inline)]

Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sat, 24 Mar 2018 17:08:01 GMT) Full text and rfc822 format available.

Message #397 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: Keyboard detection before ‘cryptsetup’
 runs
Date: Sat, 24 Mar 2018 18:07:44 +0100
Hi Danny!

Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

>> (b) Monitor /sys using inotify in an extra thread/process.
>
> Oh - or loop { load-/sys ; sleep 1 }.  That's what you meant.

Yeah, though we wouldn’t know when to stop, but if I read your message
correctly, nobody does.  :-)

Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Sat, 24 Mar 2018 17:13:02 GMT) Full text and rfc822 format available.

Message #400 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 30604 <at> debbugs.gnu.org
Subject: Re: Keyboard detection before ‘cryptsetup’
 runs
Date: Sat, 24 Mar 2018 18:12:52 +0100
Heya,

Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

> I have to say what we have to do is much more complicated than I thought
> it would be - and I kinda regret starting this patchset.  I didn't
> want to make it less likely for people to boot than before :-(

Yeah, I had a nightmare where I dreamed of all the bug reports due to
devices not showing up quickly enough.  :-)

> A small bit of research shows that:
>
> - Busybox uses a loop with timeout to try to mount over and over again,
> depending on bus type [2].
> - Alpine coldplugs twice [1].
> - Fatdog has a "waitdev" boot parameter that specifies how long it sleeps
> until it tries to mount the root.
> - Redhat mkinitrd has an mkinitrd option like "--with=scsi_wait_scan" which
> they use after they saw a modalias starting with "scsi:" - or add the
> kernel option "scsi_mod.scan=sync" to the command line [3] (but they say that
> this is unreliable now and one is supposed to use udev inside the initrd).
> Not sure what they do for USB.
>
> The Linux kernel itself has a "rootdelay" parameter which
> was specifically introduced because of USB devices that took 15 s (!)
> to show up.

It’s a kludge fest, which is reassuring, in a way.

> All in all, we have alternative ways to continue:
>
> (a) Drop this patchset/feature entirely because it's too unreliable.
> (b) Monitor /sys using inotify in an extra thread/process.
> (c) Include udev into the initrd and have it do its thing.
> (d) Monitor the netlink socket in our own thread and play udev ourselves.
> (e) Find out how to, for each bus type, detect when enumeration is finished.
> Find out which buses are there.  Wait for them to finish enumerating.
> (f) Always modprobe usb-hid first, no matter what.  I think this would
> become a whack-a-mole thing fast.
>
> I'd prefer either (a) or (b) for reliability and simplicity.  WDYT?

Same here!  Sounds like (b) is something we could easily try, though we
won’t know how well it works until people start running it on the bare
metal.

I can adjust the branch I have to do that and push it for people to try.

Thanks a lot of your feedback!

Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Mon, 26 Mar 2018 13:38:01 GMT) Full text and rfc822 format available.

Message #403 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Jan Nieuwenhuizen <janneke <at> gnu.org>
Cc: 30922 <at> debbugs.gnu.org, 30604 <at> debbugs.gnu.org
Subject: Re: bug#30922: LUKS-encrypted root fails using device numbering,
 needs luksUUID
Date: Mon, 26 Mar 2018 15:37:01 +0200
Hello,

Jan Nieuwenhuizen <janneke <at> gnu.org> skribis:

> After booting I get
>
>     Device /dev/nvme0n1p1 doesn't exist or access denied

This message comes from ‘cryptsetup’.

The ‘init’ script for the config you posted does this:

--8<---------------cut here---------------start------------->8---
(boot-system #:mounts (map spec->file-system
                           (quote (("/dev/mapper/guix"
                                    device "/" "ext4" ()
                                    #f #t))))
             #:pre-mount (lambda ()
                           (and (let ((source "/dev/nvme0n1p1"))
                                  (use-modules (rnrs bytevectors)
                                               ((gnu build file-systems)
                                                #:select
                                                (find-partition-by-luks-uuid)))
                                  (zero? (system*
                                          "/gnu/store/v5r0dx5v08847rcy033kyish6m3b3621-cryptsetup-static-1.7.5/sbin/cryptsetup"
                                          "open" "--type" "luks"
                                          (if (bytevector? source)
                                              (or (let loop ((tries-left 10))
                                                    (and (positive? tries-left)
                                                         (or (find-partition-by-luks-uuid source)
                                                             (begin
                                                               (sleep 1)
                                                               (loop (- tries-left 1))))))
                                                  (error "LUKS partition not found" source))
                                              source)
                                          "guix")))))
             …
--8<---------------cut here---------------end--------------->8---

IOW, it does the right thing whether or not you’re using a UUID.

However, when using a UUID, it waits for the device to show up, whereas
otherwise it assumes the /dev node is already there and invokes
‘cryptsetup’ right away.

We should fix that and generalize the wait-for-device loop, which
already exists in a couple of instances.

There are connections with <https://bugs.gnu.org/30604>.

Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#30604; Package guix-patches. (Fri, 16 Aug 2019 21:11:02 GMT) Full text and rfc822 format available.

Message #406 received at 30604 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: Julien Lepiller <julien <at> lepiller.eu>, 34902-done <at> debbugs.gnu.org,
 30604 <at> debbugs.gnu.org
Subject: Re: bug#34902: guix cannot find a module on boot
Date: Fri, 16 Aug 2019 23:10:15 +0200
Hello!

Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

> Maybe I'm too paranoid but can we have "guix" in the file name "modules.name"
> somewhere?  Otherwise I see it coming that upstream uses modules.name for an
> incompatible purpose and then we'd be with a guix interface that's broken
> and/or break their interface.
>
> (So much complexity for something so silly.  Honestly, I feel like E-mailing
> the upstream author and telling him what I think.  WTF :P)
>
> Should we warn when we use the fallback?  I like the defensive programming
> but I feel we shouldn't have it *silently* fall back when the database is
> broken/missing.
>
> Otherwise LGTM!

So I went ahead and pushed these patches, derived from our beautiful
patch set at <https://issues.guix.gnu.org/issue/30604>:

  c85ccf60bf linux-modules: Define and use a module name database.
  e1a9a7f275 linux-modules: Add 'load-linux-modules-from-directory'.
  2a693b69ca linux-modules: Add "modules.devname" writer.
  4f8b9d1a6f linux-modules: Add "modules.alias" writer.

The actual fix for the hyphen/underscore mismatch that Julien reported
is commit c85ccf60bf.  The “modules.devname” and “modules.alias” are
actually unused so far but (1) it was easier to preserve them, and (2)
that’ll give us an incentive to finish
<https://issues.guix.gnu.org/issue/30604>.  :-)

I added an explicit comment that “modules.name” uses a Guix-specific
format.  We can always rename it if the kernel folks decide to acquire
that file name.

Julien, could you please confirm that your initial issue is fixed?

Thanks,
Ludo’.




This bug report was last modified 4 years and 216 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.