GNU bug report logs - #29181
[PATCH core-updates 0/1] Shrink Mesa

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 29181 in the body.
You can then email your comments to 29181 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: guix-patches <at> gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH core-updates 0/1] Shrink Mesa
Date: Mon,  6 Nov 2017 22:40:57 +0100

Hello Guix!

This patch divides the nar size of ‘mesa’ (as reported by ‘guix size’)
almost by a factor of four.  It does so by replacing hard links created
upon “make install” with symlinks.

The new post-install phase prints things like:

--8<---------------cut here---------------start------------->8---
starting phase `symlinks-instead-of-hard-links'
creating 7 symlinks to '/gnu/store/l1iwgjgp88snlzp9nafdpd0vvsqv7lsi-mesa-17.2.1/lib/dri/i915_dri.so'
creating 3 symlinks to '/gnu/store/l1iwgjgp88snlzp9nafdpd0vvsqv7lsi-mesa-17.2.1/lib/dri/i965_dri.so'
creating 1 symlinks to '/gnu/store/l1iwgjgp88snlzp9nafdpd0vvsqv7lsi-mesa-17.2.1/lib/dri/nouveau_drv_video.so'
creating 1 symlinks to '/gnu/store/l1iwgjgp88snlzp9nafdpd0vvsqv7lsi-mesa-17.2.1/lib/libXvMCnouveau.so.1.0.0'
creating 2 symlinks to '/gnu/store/l1iwgjgp88snlzp9nafdpd0vvsqv7lsi-mesa-17.2.1/lib/vdpau/libvdpau_nouveau.so.1.0.0'
phase `symlinks-instead-of-hard-links' succeeded after 0.0 seconds
--8<---------------cut here---------------end--------------->8---

The nar size does not reflect disk usage since deduplication would
recreate the hard links anyway, but it reflects how much we have to
transfer over the wire since the nar format does not represent hard
links.  A similar problem arose with Git a while back, see
<https://bugs.gnu.org/21949>.

I’d like to push to ‘core-updates’.

Thoughts?

Ludo’.

Ludovic Courtès (1):
  gnu: mesa: Use symlinks instead of hard links.

 gnu/packages/gl.scm | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

-- 
2.14.2

Message #8 received at 29181 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 29181 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 1/1] gnu: mesa: Use symlinks instead of hard links.
Date: Mon,  6 Nov 2017 22:42:52 +0100

This reduces the nar size (as shown by 'guix size') by 124 MiB, from
169 MiB to 45 MiB (almost divided by 4!).

* gnu/packages/gl.scm (mesa)[arguments]: Add #:modules.  Add
'symlinks-instead-of-hard-links' phase.
---
 gnu/packages/gl.scm | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/gnu/packages/gl.scm b/gnu/packages/gl.scm
index 5f96ab303..be0eddfdb 100644
--- a/gnu/packages/gl.scm
+++ b/gnu/packages/gl.scm
@@ -305,6 +305,10 @@ also known as DXTn or DXTC) for Mesa.")
                 "--enable-llvm"))         ; default is x86/x86_64 only
              (_
               '("--with-dri-drivers=nouveau,r200,radeon,swrast"))))
+       #:modules ((ice-9 match)
+                  (srfi srfi-1)
+                  (guix build utils)
+                  (guix build gnu-build-system))
        #:phases
        (modify-phases %standard-phases
          (add-after
@@ -339,6 +343,41 @@ also known as DXTn or DXTC) for Mesa.")
                  ;; egl_gallium support.
                  (("\"gbm_dri\\.so")
                   (string-append "\"" out "/lib/dri/gbm_dri.so")))
+               #t)))
+         (add-after 'install 'symlinks-instead-of-hard-links
+           (lambda* (#:key outputs #:allow-other-keys)
+             ;; All the drivers and gallium targets create hard links upon
+             ;; installation (search for "hardlink each megadriver instance"
+             ;; in the makefiles).  This is no good for us since we'd produce
+             ;; nars that contain several copies of these files.  Thus, turn
+             ;; them into symlinks, which saves ~124 MiB.
+             (let* ((out    (assoc-ref outputs "out"))
+                    (lib    (string-append out "/lib"))
+                    (files  (find-files lib
+                                        (lambda (file stat)
+                                          (and (string-contains file ".so")
+                                               (eq? 'regular
+                                                    (stat:type stat))))))
+                    (inodes (map (compose stat:ino stat) files)))
+               (for-each (lambda (inode)
+                           (match (filter-map (match-lambda
+                                                ((file ino)
+                                                 (and (= ino inode) file)))
+                                              (zip files inodes))
+                             ((_)
+                              #f)
+                             ((reference others ..1)
+                              (format #t "creating ~a symlinks to '~a'~%"
+                                      (length others) reference)
+                              (for-each delete-file others)
+                              (for-each (lambda (file)
+                                          (if (string=? (dirname file)
+                                                        (dirname reference))
+                                              (symlink (basename reference)
+                                                       file)
+                                              (symlink reference file)))
+                                        others))))
+                         (delete-duplicates inodes))
                #t))))))
     (home-page "https://mesa3d.org/")
     (synopsis "OpenGL implementation")
-- 
2.14.2

Message #11 received at 29181 <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>, 29181 <at> debbugs.gnu.org
Subject: Re: [bug#29181] [PATCH 1/1] gnu: mesa: Use symlinks instead of hard
 links.
Date: Tue, 07 Nov 2017 20:15:04 +0100

[Message part 1 (text/plain, inline)]

Ludovic Courtès <ludo <at> gnu.org> writes:

> This reduces the nar size (as shown by 'guix size') by 124 MiB, from
> 169 MiB to 45 MiB (almost divided by 4!).

Wow, nice catch.  If I read the code correctly, it detects hard links
and replaces them with symlinks.  Could we do this unconditionally in
a gnu-build-system phase?  Are there any legitimate uses of hard links
in outputs?

That said, the patch LGTM.

[signature.asc (application/pgp-signature, inline)]

Message #16 received at 29181-done <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Marius Bakke <mbakke <at> fastmail.com>
Cc: 29181-done <at> debbugs.gnu.org
Subject: Re: [bug#29181] [PATCH 1/1] gnu: mesa: Use symlinks instead of hard
 links.
Date: Wed, 08 Nov 2017 09:27:29 +0100

Hi,

Marius Bakke <mbakke <at> fastmail.com> skribis:

> Ludovic Courtès <ludo <at> gnu.org> writes:
>
>> This reduces the nar size (as shown by 'guix size') by 124 MiB, from
>> 169 MiB to 45 MiB (almost divided by 4!).
>
> Wow, nice catch.  If I read the code correctly, it detects hard links
> and replaces them with symlinks.

Yes.

> Could we do this unconditionally in a gnu-build-system phase?  Are
> there any legitimate uses of hard links in outputs?

Good question.  There might be situations where the software relies on
having regular files (not symlinks), so I would rather do it on a
case-by-case basis than have a standard phase.  WDYT?

Besides, a phase that looks at all the files would have to be efficient
than this naïve implementation.

> That said, the patch LGTM.

Pushed as dcc00f54c619118d11982383102d2e9a1b86d080, thanks!

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.