GNU bug report logs - #31015
[PATCH] packages, scripts, utils: Respect (parallel-job-count) when xz-compressing.

Previous Next

Package: guix-patches;

Reported by: Marius Bakke <mbakke <at> fastmail.com>

Date: Sun, 1 Apr 2018 12:55:01 UTC

Severity: normal

Tags: patch, wontfix

Done: ludo <at> gnu.org (Ludovic Courtès)

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 31015 in the body.
You can then email your comments to 31015 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to guix-patches <at> gnu.org:
bug#31015; Package guix-patches. (Sun, 01 Apr 2018 12:55:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Marius Bakke <mbakke <at> fastmail.com>:
New bug report received and forwarded. Copy sent to guix-patches <at> gnu.org. (Sun, 01 Apr 2018 12:55:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: guix-patches <at> gnu.org
Cc: Marius Bakke <mbakke <at> fastmail.com>
Subject: [PATCH] packages, scripts,
 utils: Respect (parallel-job-count) when xz-compressing.
Date: Sun,  1 Apr 2018 14:54:31 +0200
* guix/packages.scm (patch-and-repack): When invoking 'xz', tell it to use the
amount of cores specified by e.g. 'guix build -c'.
* guix/scripts/pack.scm (%compressors, bootstrap-xz): Likewise.  While at it,
use the long form '--threads=' for clarity.
* guix/utils.scm (decompressed-port, compressed-port, compressed-output-port):
Likewise.
---
 guix/packages.scm     |  3 ++-
 guix/scripts/pack.scm |  7 +++++--
 guix/utils.scm        | 18 +++++++++++++++---
 3 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/guix/packages.scm b/guix/packages.scm
index ab4b6278d..cca2c6357 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -604,7 +604,8 @@ specifies modules in scope when evaluating SNIPPET."
                      ;; threaded compression (introduced in
                      ;; 5.2.0), but it ignores the extra flag.
                      (string-append "--use-compress-program="
-                                    #+xz "/bin/xz --threads=0")
+                                    #+xz "/bin/xz --threads="
+                                    (number->string (parallel-job-count)))
                      ;; avoid non-determinism in the archive
                      "--mtime=@0"
                      "--owner=root:0"
diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
index 488638adc..818de0f32 100644
--- a/guix/scripts/pack.scm
+++ b/guix/scripts/pack.scm
@@ -65,7 +65,9 @@
         (compressor "lzip"  ".lz"
                     #~(#+(file-append lzip "/bin/lzip") "-9"))
         (compressor "xz"    ".xz"
-                    #~(#+(file-append xz "/bin/xz") "-e -T0"))
+                    #~(#+(file-append xz "/bin/xz")
+                       (string-append "-e --threads="
+                                      (number->string (parallel-job-count)))))
         (compressor "bzip2" ".bz2"
                     #~(#+(file-append bzip2 "/bin/bzip2") "-9"))
         (compressor "none" "" #f)))
@@ -73,7 +75,8 @@
 ;; This one is only for use in this module, so don't put it in %compressors.
 (define bootstrap-xz
   (compressor "bootstrap-xz" ".xz"
-              #~(#+(file-append %bootstrap-coreutils&co "/bin/xz") "-e -T0")))
+              #~(#+(file-append %bootstrap-coreutils&co "/bin/xz")
+                 (string-append "-e --threads=" (number->string (parallel-job-count))))))
 
 (define (lookup-compressor name)
   "Return the compressor object called NAME.  Error out if it could not be
diff --git a/guix/utils.scm b/guix/utils.scm
index 92e45de61..043034e99 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -172,7 +172,11 @@ a symbol such as 'xz."
   (match compression
     ((or #f 'none) (values input '()))
     ('bzip2        (filtered-port `(,%bzip2 "-dc") input))
-    ('xz           (filtered-port `(,%xz "-dc" "-T0") input))
+    ('xz           (filtered-port
+                    `(,%xz "-dc" (string-append
+                                  "--threads="
+                                  (number->string (parallel-job-count))))
+                    input))
     ('gzip         (filtered-port `(,%gzip "-dc") input))
     (else          (error "unsupported compression scheme" compression))))
 
@@ -182,7 +186,11 @@ a symbol such as 'xz."
   (match compression
     ((or #f 'none) (values input '()))
     ('bzip2        (filtered-port `(,%bzip2 "-c") input))
-    ('xz           (filtered-port `(,%xz "-c" "-T0") input))
+    ('xz           (filtered-port
+                    `(,%xz "-c" (string-append
+                                 "--threads="
+                                 (number->string (parallel-job-count))))
+                    input))
     ('gzip         (filtered-port `(,%gzip "-c") input))
     (else          (error "unsupported compression scheme" compression))))
 
@@ -239,7 +247,11 @@ program--e.g., '(\"--fast\")."
   (match compression
     ((or #f 'none) (values output '()))
     ('bzip2        (filtered-output-port `(,%bzip2 "-c" ,@options) output))
-    ('xz           (filtered-output-port `(,%xz "-c" "-T0" ,@options) output))
+    ('xz           (filtered-output-port
+                    `(,%xz "-c" (string-append
+                                 "--threads="
+                                 (number->string (parallel-job-count)))
+                           ,@options) output))
     ('gzip         (filtered-output-port `(,%gzip "-c" ,@options) output))
     (else          (error "unsupported compression scheme" compression))))
 
-- 
2.16.3





Information forwarded to guix-patches <at> gnu.org:
bug#31015; Package guix-patches. (Wed, 04 Apr 2018 11:43:02 GMT) Full text and rfc822 format available.

Message #8 received at 31015 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Marius Bakke <mbakke <at> fastmail.com>
Cc: 31015 <at> debbugs.gnu.org
Subject: Re: [bug#31015] [PATCH] packages, scripts,
 utils: Respect (parallel-job-count) when xz-compressing.
Date: Wed, 04 Apr 2018 13:42:01 +0200
Hello!

Marius Bakke <mbakke <at> fastmail.com> skribis:

> * guix/packages.scm (patch-and-repack): When invoking 'xz', tell it to use the
> amount of cores specified by e.g. 'guix build -c'.
> * guix/scripts/pack.scm (%compressors, bootstrap-xz): Likewise.  While at it,
> use the long form '--threads=' for clarity.
> * guix/utils.scm (decompressed-port, compressed-port, compressed-output-port):
> Likewise.

Wasn’t there an issue that parallel xz is non-deterministic?  I vaguely
remember something like this when we were producing with glibc tarballs
out of the Git repo.

If so, we should instead use -T1, even if that means getting worse
performance (I hear that lzip “works better” than xz, so we should give
it a try if we want both performance and good compression.)

Besides, the guix/packages.scm part would lead to a full rebuild, so I
think it’s a bit late for ‘core-updates’.

Thanks,
Ludo’.




Information forwarded to guix-patches <at> gnu.org:
bug#31015; Package guix-patches. (Sun, 13 May 2018 20:39:02 GMT) Full text and rfc822 format available.

Message #11 received at 31015 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Marius Bakke <mbakke <at> fastmail.com>
Cc: 31015 <at> debbugs.gnu.org
Subject: Re: [bug#31015] [PATCH] packages, scripts,
 utils: Respect (parallel-job-count) when xz-compressing.
Date: Sun, 13 May 2018 22:37:58 +0200
Hello Marius,

ludo <at> gnu.org (Ludovic Courtès) skribis:

> Marius Bakke <mbakke <at> fastmail.com> skribis:
>
>> * guix/packages.scm (patch-and-repack): When invoking 'xz', tell it to use the
>> amount of cores specified by e.g. 'guix build -c'.
>> * guix/scripts/pack.scm (%compressors, bootstrap-xz): Likewise.  While at it,
>> use the long form '--threads=' for clarity.
>> * guix/utils.scm (decompressed-port, compressed-port, compressed-output-port):
>> Likewise.
>
> Wasn’t there an issue that parallel xz is non-deterministic?  I vaguely
> remember something like this when we were producing with glibc tarballs
> out of the Git repo.

That was indeed the case, so I’m closing this issue.  Let’s reopen it if
we have more info or new ideas!  :-)

Ludo’.




Added tag(s) wontfix. Request was from ludo <at> gnu.org (Ludovic Courtès) to control <at> debbugs.gnu.org. (Sun, 13 May 2018 20:39:02 GMT) Full text and rfc822 format available.

bug closed, send any further explanations to 31015 <at> debbugs.gnu.org and Marius Bakke <mbakke <at> fastmail.com> Request was from ludo <at> gnu.org (Ludovic Courtès) to control <at> debbugs.gnu.org. (Sun, 13 May 2018 20:39:03 GMT) Full text and rfc822 format available.

bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Mon, 11 Jun 2018 11:24:03 GMT) Full text and rfc822 format available.

This bug report was last modified 5 years and 292 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.