GNU bug report logs - #26316
Enable threaded compression of source tarballs

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 26316 in the body.
You can then email your comments to 26316 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: guix-patches <at> gnu.org
Subject: Enable threaded compression of source tarballs
Date: Thu, 30 Mar 2017 18:17:15 -0400

[Message part 1 (text/plain, inline)]

This patch enables multi-threaded xz compression in the repacking stage
of building source tarballs.

With it applied, I get a ~2.2x speedup building the source of
linux-libre with 4 threads, compared to 1 thread.

Specifically, with the upstream linux-libre tarball in the page cache,
`guix build -S linux-libre` took ~366 seconds using a single thread for
xz, and ~164 seconds using 4 threads.

[0001-packages-Enable-threaded-compression-of-source-tarba.patch (text/plain, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 26316 <at> debbugs.gnu.org (full text, mbox):

From: Kei Kebreau <kei <at> openmailbox.org>
To: Leo Famulari <leo <at> famulari.name>
Cc: 26316 <at> debbugs.gnu.org
Subject: Re: bug#26316: Enable threaded compression of source tarballs
Date: Fri, 31 Mar 2017 14:15:36 -0400

[Message part 1 (text/plain, inline)]

Leo Famulari <leo <at> famulari.name> writes:

> This patch enables multi-threaded xz compression in the repacking stage
> of building source tarballs.
>
> With it applied, I get a ~2.2x speedup building the source of
> linux-libre with 4 threads, compared to 1 thread.
>
> Specifically, with the upstream linux-libre tarball in the page cache,
> `guix build -S linux-libre` took ~366 seconds using a single thread for
> xz, and ~164 seconds using 4 threads.
>
> From 78547a09e50440c649c3d28d7691f32fdd47cc25 Mon Sep 17 00:00:00 2001
> From: Leo Famulari <leo <at> famulari.name>
> Date: Thu, 30 Mar 2017 03:10:48 -0400
> Subject: [PATCH] packages: Enable threaded compression of source tarballs.
>
> This provides a ~2x speedup when using 4 threads.
>
> * guix/packages.scm (patch-and-repack)[build]: Invoke xz with
> '--threads=0' when re-packing tarballs.
> ---
>  guix/packages.scm | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/guix/packages.scm b/guix/packages.scm
> index 4bc4b017f..30be5bf38 100644
> --- a/guix/packages.scm
> +++ b/guix/packages.scm
> @@ -573,7 +573,12 @@ specifies modules in scope when evaluating SNIPPET."
>                                                      #:fail-on-error? #t)))))
>                          (zero? (apply system*
>                                        (string-append #+tar "/bin/tar")
> -                                      "cvfa" #$output
> +                                      "cvf" #$output
> +                                      ;; The bootstrap xz does not support
> +                                      ;; threaded compression (introduced in
> +                                      ;; 5.2.0), but it ignores the extra flag.
> +                                      (string-append "--use-compress-program="
> +                                                     #+xz "/bin/xz --threads=0")
>                                        ;; avoid non-determinism in the archive
>                                        "--mtime=@0"
>                                        "--owner=root:0"

LGTM.

[signature.asc (application/pgp-signature, inline)]

Message #11 received at 26316 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Kei Kebreau <kei <at> openmailbox.org>
Cc: 26316 <at> debbugs.gnu.org
Subject: Re: bug#26316: Enable threaded compression of source tarballs
Date: Fri, 31 Mar 2017 19:04:13 -0400

[Message part 1 (text/plain, inline)]

On Fri, Mar 31, 2017 at 02:15:36PM -0400, Kei Kebreau wrote:
> Leo Famulari <leo <at> famulari.name> writes:
> > Subject: [PATCH] packages: Enable threaded compression of source tarballs.

> > -                                      "cvfa" #$output
> > +                                      "cvf" #$output
> > +                                      ;; The bootstrap xz does not support
> > +                                      ;; threaded compression (introduced in
> > +                                      ;; 5.2.0), but it ignores the extra flag.
> > +                                      (string-append "--use-compress-program="
> > +                                                     #+xz "/bin/xz --threads=0")

One question I have is about the removal of 'cvfa'. Here is the relevant
documentation:

-a, --auto-compress
             Use archive suffix to determine the compression
	     program.

AFAICT, we only build XZ archives, so it's okay to remove 'a' and
hard-code the use of xz. Is that correct?

> >                                        ;; avoid non-determinism in the archive
> >                                        "--mtime=@0"
> >                                        "--owner=root:0"
> 
> LGTM.

Thanks! This does mean rebuilding the world, so we'd need to save it for
the next core-updates cycle.

[signature.asc (application/pgp-signature, inline)]

Message #14 received at 26316 <at> debbugs.gnu.org (full text, mbox):

From: Kei Kebreau <kei <at> openmailbox.org>
To: Leo Famulari <leo <at> famulari.name>
Cc: 26316 <at> debbugs.gnu.org
Subject: Re: bug#26316: Enable threaded compression of source tarballs
Date: Fri, 31 Mar 2017 19:56:18 -0400

[Message part 1 (text/plain, inline)]

Leo Famulari <leo <at> famulari.name> writes:

> On Fri, Mar 31, 2017 at 02:15:36PM -0400, Kei Kebreau wrote:
>> Leo Famulari <leo <at> famulari.name> writes:
>> > Subject: [PATCH] packages: Enable threaded compression of source tarballs.
>
>> > -                                      "cvfa" #$output
>> > +                                      "cvf" #$output
>> > +                                      ;; The bootstrap xz does not support
>> > + ;; threaded compression (introduced in
>> > + ;; 5.2.0), but it ignores the extra flag.
>> > + (string-append "--use-compress-program="
>> > + #+xz "/bin/xz --threads=0")
>
> One question I have is about the removal of 'cvfa'. Here is the relevant
> documentation:
>
> -a, --auto-compress
>              Use archive suffix to determine the compression
> 	     program.
>
> AFAICT, we only build XZ archives, so it's okay to remove 'a' and
> hard-code the use of xz. Is that correct?
>

The surrounding code definitely seems to suggest that. Reading "up" the
code from the gexp->derivation call I can find hardcoded xz usage in the
tarxz-name procedure and this bit:

(mlet %store-monad ((tar ->     (lookup-input "tar"))
                  ->(xz ->      (lookup-input "xz"))
  (only xz here!)/  (patch ->   (lookup-input "patch"))
                    (locales -> (lookup-input "locales"))
                    (decomp ->  (lookup-input decompression-type))
                    (patches    (sequence %store-monad
                                          (map instantiate-patch patches))))

>> >                                        ;; avoid non-determinism in
>> > the archive
>> >                                        "--mtime=@0"
>> >                                        "--owner=root:0"
>> 
>> LGTM.
>
> Thanks! This does mean rebuilding the world, so we'd need to save it for
> the next core-updates cycle.

[signature.asc (application/pgp-signature, inline)]

Message #17 received at 26316 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Kei Kebreau <kei <at> openmailbox.org>
Cc: 26316 <at> debbugs.gnu.org
Subject: Re: bug#26316: Enable threaded compression of source tarballs
Date: Fri, 31 Mar 2017 20:15:55 -0400

[Message part 1 (text/plain, inline)]

On Fri, Mar 31, 2017 at 07:56:18PM -0400, Kei Kebreau wrote:
> The surrounding code definitely seems to suggest that. Reading "up" the
> code from the gexp->derivation call I can find hardcoded xz usage in the
> tarxz-name procedure and this bit:
> 
> (mlet %store-monad ((tar ->     (lookup-input "tar"))
>                   ->(xz ->      (lookup-input "xz"))
>   (only xz here!)/  (patch ->   (lookup-input "patch"))
>                     (locales -> (lookup-input "locales"))
>                     (decomp ->  (lookup-input decompression-type))
>                     (patches    (sequence %store-monad
>                                           (map instantiate-patch patches))))

Right, that's what I was looking at.

[signature.asc (application/pgp-signature, inline)]

Message #22 received at 26316-done <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: 26316-done <at> debbugs.gnu.org
Subject: Re: Enable threaded compression of source tarballs
Date: Sat, 15 Apr 2017 18:35:58 -0400

[Message part 1 (text/plain, inline)]

On Thu, Mar 30, 2017 at 06:17:15PM -0400, Leo Famulari wrote:
> This patch enables multi-threaded xz compression in the repacking stage
> of building source tarballs.
> 
> With it applied, I get a ~2.2x speedup building the source of
> linux-libre with 4 threads, compared to 1 thread.
> 
> Specifically, with the upstream linux-libre tarball in the page cache,
> `guix build -S linux-libre` took ~366 seconds using a single thread for
> xz, and ~164 seconds using 4 threads.

> From 78547a09e50440c649c3d28d7691f32fdd47cc25 Mon Sep 17 00:00:00 2001
> From: Leo Famulari <leo <at> famulari.name>
> Date: Thu, 30 Mar 2017 03:10:48 -0400
> Subject: [PATCH] packages: Enable threaded compression of source tarballs.
> 
> This provides a ~2x speedup when using 4 threads.
> 
> * guix/packages.scm (patch-and-repack)[build]: Invoke xz with
> '--threads=0' when re-packing tarballs.

I pushed this to core-updates as
c8a3dea847bb9f87fa1876d0c6c3356d6226f121.

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.