GNU bug report logs - #40790
OOM error in graphical installer tests.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 40790 in the body.
You can then email your comments to 40790 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <m.othacehe <at> gmail.com>
To: bug-guix <at> gnu.org
Subject: OOM error in graphical installer tests.
Date: Thu, 23 Apr 2020 11:49:30 +0200

[Message part 1 (text/plain, inline)]

Hello,

The "gui-installed-os" test fails on master because "guix system init"
command is OOM killed. Log of the failure is attached.

Thanks,

Mathieu

[oom.log (application/octet-stream, attachment)]

Message #10 received at 40790 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Mathieu Othacehe <m.othacehe <at> gmail.com>
Cc: 40790 <at> debbugs.gnu.org
Subject: Re: bug#40790: OOM error in graphical installer tests.
Date: Fri, 24 Apr 2020 16:27:13 +0200

Hi Mathieu,

Mathieu Othacehe <m.othacehe <at> gmail.com> skribis:

> The "gui-installed-os" test fails on master because "guix system init"
> command is OOM killed. Log of the failure is attached.

[...]

> Apr 23 11:43:27 localhost installer[230]: command ("guix" "system" "init" "--fallback" "--no-grafts" "--no-substitutes" "/mnt/etc/config.scm" "/mnt") failed with exit code #finstaller[230]: Umounted /remove suc 

Can we throw a little bit more memory at it?  :-)

Is there a commit that trigger this failure?

Thanks,
Ludo’.

Message #15 received at 40790-done <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <m.othacehe <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 40790-done <at> debbugs.gnu.org
Subject: Re: bug#40790: OOM error in graphical installer tests.
Date: Wed, 29 Apr 2020 10:21:13 +0200

Hello,

> Can we throw a little bit more memory at it?  :-)

That's what I did with ae1a0f758 :)

> Is there a commit that trigger this failure?

I didn't have the courage to bisect it, given how long this test
is. Fixing it, I discovered two more issues:

* "sherpherd services" test was failing because a wrong swap device was
  used (fixed with be3abf28a).

* I have some hangs when doing the final testing in marionette. Still
  trying to understand why.

Thanks,

Mathieu

Message #18 received at 40790-done <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <m.othacehe <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 40790-done <at> debbugs.gnu.org
Subject: Re: bug#40790: OOM error in graphical installer tests.
Date: Wed, 29 Apr 2020 17:32:47 +0200

> * I have some hangs when doing the final testing in marionette. Still
>   trying to understand why.

Fixed with c537d2bcb7c652b4776435cd1e2a5119af43dd78. Still an issue
remaining. "gui-installed-desktop-os-encrypted" closure is about 5.5GiB
which makes too much files for libisofs:

--8<---------------cut here---------------start------------->8---
xorriso : UPDATE :  427934 files added in 19 seconds
libisofs: FAILURE : HFS+ map nodes aren't implemented
libisofs: FAILURE : Too much files to mangle, cannot guarantee unique file names
xorriso : FAILURE : Failed to prepare session write run
xorriso : NOTE : -return_with SORRY 32 triggered by problem severity FAILURE
--8<---------------cut here---------------end--------------->8---

Not sure how to fix it. Someone?

Thanks,

Mathieu

Message #21 received at 40790-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Mathieu Othacehe <m.othacehe <at> gmail.com>
Cc: 40790-done <at> debbugs.gnu.org
Subject: Re: bug#40790: OOM error in graphical installer tests.
Date: Thu, 30 Apr 2020 23:10:05 +0200

Hi,

Mathieu Othacehe <m.othacehe <at> gmail.com> skribis:

>> Can we throw a little bit more memory at it?  :-)
>
> That's what I did with ae1a0f758 :)

Oh, good.

Next we should of course find out why ‘guix system init’ is consuming so
much memory.  I haven’t tried GC-profiling that yet, but here’s a rough
figure:

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING=gc GUIX_PROFILING_EVENTS=exit guix system build -d --no-grafts gnu/system/examples/desktop.tmpl
/gnu/store/2by26x4pvvp9vi1f0lmrq39wqqkj0s0f-system.drv
Garbage collection statistics:
  heap size:        98.80 MiB
  allocated:        548.35 MiB
  GC times:         26
  time spent in GC: 1.84 seconds (36% of user time)
$ guix describe
Generacio 140	Apr 29 2020 19:43:22	(nuna)
  guix 19987f5
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 19987f5efe0acb65918802681f3a18cbb6a07c2a
--8<---------------cut here---------------end--------------->8---

So at first sight, it’s surprising that it couldn’t fit in the VM with
800 MiB of RAM.  Oh well, we’ll see!

Nitpick: please include the canonical bug URL in commit logs rather
than a link to the mailing list, like so:

  Fixes <https://bugs.gnu.org/40790>.

That makes it easier to grep for fixed bugs.

> I didn't have the courage to bisect it, given how long this test
> is. Fixing it, I discovered two more issues:
>
> * "sherpherd services" test was failing because a wrong swap device was
>   used (fixed with be3abf28a).

Good catch!

> * I have some hangs when doing the final testing in marionette. Still
>   trying to understand why.

Oh.

Thank you!

Ludo’.

Message #24 received at 40790-done <at> debbugs.gnu.org (full text, mbox):

From: Bengt Richter <bokr <at> bokr.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 40790-done <at> debbugs.gnu.org, Mathieu Othacehe <m.othacehe <at> gmail.com>
Subject: Re: bug#40790: OOM error in graphical installer tests.
Date: Fri, 1 May 2020 12:13:41 +0200

Hi Ludo, et al,

On +2020-04-30 23:10:05 +0200, Ludovic Courtès wrote:
[...]

rc1: best-practice: commit logs: message syntax
--8<---------------cut here---------------start------------->8---
> Nitpick: please include the canonical bug URL in commit logs rather
> than a link to the mailing list, like so:
> 
>   Fixes <https://bugs.gnu.org/40790>.
> 
> That makes it easier to grep for fixed bugs.
--8<---------------cut here---------------end--------------->8---

ISTM your "nit-pick" is a good example of an embedded mail-list nugget.

Here's my wish:
If you and other developers notice (without false modesty ;-) that you
are posting a good example of something, then if you would enclose it
in guix snippet-delimiters as above, with a single prefixed line
starting with "primary-focus-subject:" and optionally followed by other
greppable (or list-matchable) sub-categories or ad-hoc words or names or tags,
then these nuggets would be easy to extract automatically, to
create a treasure-trove of searchable useful goodies.

I would hope to see useful one-liners and examples of package definition,
system configs, debugging methods, small scripts, and good info-URLs etc.
delimited and prefixed.

(The "primary-focus-subject:" of "rc1:" would permit us to invent improved
syntaxes and evolve them as "rc2" etc until we settle on a canonical set
for primary foci, (and PEG grammar for the rest, if we want to get fancy).

Perhaps the servers could have a gem-and-nugget-bot authoring an html
blog of these things, with automatic attibutions etc from email headers?

-- 
Regards,
Bengt Richter

Message #27 received at 40790 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Mathieu Othacehe <m.othacehe <at> gmail.com>
Cc: 40790 <at> debbugs.gnu.org
Subject: Re: bug#40790: OOM error in graphical installer tests.
Date: Sat, 02 May 2020 15:54:38 +0200

[Message part 1 (text/plain, inline)]

Hi,

Mathieu Othacehe <m.othacehe <at> gmail.com> skribis:

>> * I have some hangs when doing the final testing in marionette. Still
>>   trying to understand why.
>
> Fixed with c537d2bcb7c652b4776435cd1e2a5119af43dd78.

Cool, sorry for breaking things.  :-/

> Still an issue remaining. "gui-installed-desktop-os-encrypted" closure
> is about 5.5GiB which makes too much files for libisofs:
>
> xorriso : UPDATE :  427934 files added in 19 seconds
> libisofs: FAILURE : HFS+ map nodes aren't implemented
> libisofs: FAILURE : Too much files to mangle, cannot guarantee unique file names
> xorriso : FAILURE : Failed to prepare session write run
> xorriso : NOTE : -return_with SORRY 32 triggered by problem severity FAILURE
>
> Not sure how to fix it. Someone?

I tried the attached patch, which would select just one desktop
environments, thinking it might appease Xorriso, but actually it still
OOMs at the very last step:

--8<---------------cut here---------------start------------->8---
xorriso : UPDATE :  141879 file filters processed in 848 seconds
xorriso : UPDATE :  142026 file filters processed in 849 seconds
xorriso : UPDATE :  142102 file filters processed in 850 seconds
xorriso : NOTE : Copying to System Area: 512 bytes from file '/gnu/store/dvfl5ax7pj6c62f1ccc27qcrcb8nafr7-grub-hybrid-2.04/lib/grub/i386-pc/boot_hybrid.img'
[13266.747348] xorriso invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[13266.749415] CPU: 0 PID: 251 Comm: xorriso Not tainted 5.4.36-gnu #1
[13266.750777] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014
--8<---------------cut here---------------end--------------->8---

Testing again with this extra patch:

[Message part 2 (text/x-patch, inline)]

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 6f81ac16ff..9f240b50ee 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -356,7 +356,7 @@ INPUTS is a list of inputs (as for packages)."
    #:substitutable? substitutable?
 
    ;; Xorriso seems to be quite memory-hungry, so increase the VM's RAM size.
-   #:memory-size 512))
+   #:memory-size 1024))
 
 (define* (qemu-image #:key
                      (name "qemu-image")

[Message part 3 (text/plain, inline)]

I wonder if why it’s taking so much memory, and whether there’s
something else at play, like temporary files written to the overlaid
tmpfs and thus eating memory.

Thoughts?

Ludo’.

[Message part 4 (text/x-patch, inline)]

diff --git a/gnu/tests/install.scm b/gnu/tests/install.scm
index 019e21fd39..941958d698 100644
--- a/gnu/tests/install.scm
+++ b/gnu/tests/install.scm
@@ -957,7 +957,7 @@ build (current-guix) and then store a couple of full system images.")
 
 (define* (gui-test-program marionette
                            #:key
-                           (desktop? #f)
+                           (desktops '())
                            (encrypted? #f))
   #~(let ()
       (define (screenshot file)
@@ -971,7 +971,9 @@ build (current-guix) and then store a couple of full system images.")
       (setvbuf (current-output-port) 'none)
       (setvbuf (current-error-port) 'none)
 
-      (marionette-eval* '(use-modules (gnu installer tests))
+      (marionette-eval* '(use-modules (gnu installer tests)
+                                      (srfi srfi-1)
+                                      (srfi srfi-26))
                         #$marionette)
 
       ;; Arrange so that 'converse' prints debugging output to the console.
@@ -1016,7 +1018,9 @@ build (current-guix) and then store a couple of full system images.")
 
       (marionette-eval* '(choose-services installer-socket
                                           #:choose-desktop-environment?
-                                          (const #$desktop?)
+                                          (lambda (desktop)
+                                            (any (cut string-contains desktop <>)
+                                                 '#$desktops))
                                           #:choose-network-service?
                                           (const #f))
                         #$marionette)
@@ -1110,7 +1114,7 @@ build (current-guix) and then store a couple of full system images.")
 
 (define* (guided-installation-test name
                                    #:key
-                                   (desktop? #f)
+                                   (desktops '())
                                    (encrypted? #f)
                                    target-os
                                    (install-size 'guess)
@@ -1132,7 +1136,7 @@ build (current-guix) and then store a couple of full system images.")
                                (lambda (marionette)
                                  (gui-test-program
                                   marionette
-                                  #:desktop? desktop?
+                                  #:desktops desktops
                                   #:encrypted? encrypted?))))
          (command (qemu-command/writable-image image)))
       (run-basic-test target-os command name
@@ -1155,7 +1159,7 @@ build (current-guix) and then store a couple of full system images.")
 ;; desktop environments in a single test to reduce the overhead.
 (define %test-gui-installed-desktop-os-encrypted
   (guided-installation-test "gui-installed-desktop-os-encrypted"
-                            #:desktop? #t
+                            #:desktops '("GNOME")
                             #:encrypted? #t
                             #:target-os
                             (installation-target-desktop-os-for-gui-tests

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.