GNU bug report logs - #32942
'guix offload' fails to restore non-ASCII file names when running in C locale

Previous Next

Package: guix;

Reported by: Julien Lepiller <julien <at> lepiller.eu>

Date: Fri, 5 Oct 2018 08:19:02 UTC

Severity: important

Done: Ludovic Courtès <ludo <at> gnu.org>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 32942 in the body.
You can then email your comments to 32942 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-guix <at> gnu.org:
bug#32942; Package guix. (Fri, 05 Oct 2018 08:19:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Julien Lepiller <julien <at> lepiller.eu>:
New bug report received and forwarded. Copy sent to bug-guix <at> gnu.org. (Fri, 05 Oct 2018 08:19:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Julien Lepiller <julien <at> lepiller.eu>
To: bug-guix <at> gnu.org
Subject: nss-certs not deterministic
Date: Fri, 05 Oct 2018 10:17:45 +0200
While updating a profile, I found that nss-certs was not deterministic. 
From ludo:

$ wget -O - -q 
https://mirror.hydra.gnu.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo | 
grep Hash
NarHash: sha256:101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
$ wget -O - -q 
https://berlin.guixsd.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo | 
grep Hash
NarHash: sha256:08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s




Severity set to 'important' from 'normal' Request was from ludo <at> gnu.org (Ludovic Courtès) to control <at> debbugs.gnu.org. (Wed, 05 Dec 2018 11:49:02 GMT) Full text and rfc822 format available.

Information forwarded to bug-guix <at> gnu.org:
bug#32942; Package guix. (Wed, 05 Dec 2018 14:02:02 GMT) Full text and rfc822 format available.

Message #10 received at 32942 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Julien Lepiller <julien <at> lepiller.eu>
Cc: 32942 <at> debbugs.gnu.org
Subject: Re: bug#32942: nss-certs not deterministic
Date: Wed, 05 Dec 2018 15:01:25 +0100
[Message part 1 (text/plain, inline)]
Hello,

Julien Lepiller <julien <at> lepiller.eu> skribis:

> While updating a profile, I found that nss-certs was not
> deterministic. From ludo:
>
> $ wget -O - -q 
> https://mirror.hydra.gnu.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo| grep Hash
> NarHash: sha256:101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
> $ wget -O - -q 
> https://berlin.guixsd.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo |
> grep Hash
> NarHash: sha256:08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s

As shown above, berlin and hydra disagree on nss-certs.

The difference is an encoding bug:

--8<---------------cut here---------------start------------->8---
$ wget -O - https://berlin.guixsd.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.berlin
$ wget -O - https://mirror.hydra.gnu.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.hydra
$ diff -ru /tmp/nss-certs.{hydra,berlin}
Only in /tmp/nss-certs.hydra/etc/ssl/certs: AC_Raíz_Certicámara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
Only in /tmp/nss-certs.berlin/etc/ssl/certs: AC_Ra?z_Certic?mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
Only in /tmp/nss-certs.hydra/etc/ssl/certs: NetLock_Arany_=Class_Gold=_Főtanúsítvány:2.6.73.65.44.228.0.16.pem
Only in /tmp/nss-certs.berlin/etc/ssl/certs: NetLock_Arany_=Class_Gold=_F?tan?s?tv?ny:2.6.73.65.44.228.0.16.pem
--8<---------------cut here---------------end--------------->8---

The problem was already reported as <https://bugs.gnu.org/26948> and
since commit 412701b0e5e073e6767eed162c14698db99df69c (July 2017) ‘guix
publish’ on GuixSD runs in a UTF-8 locale to avoid that problem.

The faulty narinfo/nar on berlin were generated on Oct. 17, 2018, so
clearly the above commit was in effect.  Indeed, after removing them and
regenerating them, I’m still getting
08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s (aka. the wrong
hash).

On closer inspection the problem is elsewhere: the
/gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 directory on
berlin has question marks in file names, so ‘guix publish’ is not to
blame; instead the problem likely comes from ‘guix offload’.

Indeed ‘guix-daemon’ and its child processes such as ‘guix offload’ run
with an empty environment, and thus in the C locale.  Specifically,
‘restore-file-set’ on the build farm front-end must be the one
substituting question marks to the non-ASCII characters.

If this analysis is correct, the patch below should fix it.  I’ll try it
later.

Thanks,
Ludo’.

[Message part 2 (text/x-patch, inline)]
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index cee9898d79..9fe64e8087 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -1603,7 +1603,15 @@ failed to register public key '~a': ~a~%" key status))))))))
                             '())
                      #$@(if tmpdir
                             (list (string-append "TMPDIR=" tmpdir))
-                            '()))
+                            '())
+
+                     ;; Make sure we run in a UTF-8 locale so that 'guix
+                     ;; offload' correctly restores nars that contain UTF-8
+                     ;; file names such as 'nss-certs'.  See
+                     ;; <https://bugs.gnu.org/32942>.
+                     (string-append "GUIX_LOCPATH="
+                                    #$glibc-utf8-locales "/lib/locale")
+                     "LC_ALL=en_US.utf8")
 
                #:log-file #$log-file))
            (stop #~(make-kill-destructor))))))

Changed bug title to ''guix offload' fails to restore non-ASCII file names when running in C locale' from 'nss-certs not deterministic' Request was from ludo <at> gnu.org (Ludovic Courtès) to control <at> debbugs.gnu.org. (Wed, 05 Dec 2018 14:03:01 GMT) Full text and rfc822 format available.

Reply sent to Ludovic Courtès <ludo <at> gnu.org>:
You have taken responsibility. (Sun, 09 Dec 2018 23:30:02 GMT) Full text and rfc822 format available.

Notification sent to Julien Lepiller <julien <at> lepiller.eu>:
bug acknowledged by developer. (Sun, 09 Dec 2018 23:30:02 GMT) Full text and rfc822 format available.

Message #17 received at 32942-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Julien Lepiller <julien <at> lepiller.eu>
Cc: 32942-done <at> debbugs.gnu.org
Subject: Re: bug#32942: nss-certs not deterministic
Date: Mon, 10 Dec 2018 00:29:28 +0100
Hello,

ludo <at> gnu.org (Ludovic Courtès) skribis:

> The difference is an encoding bug:
>
> $ wget -O - https://berlin.guixsd.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.berlin
> $ wget -O - https://mirror.hydra.gnu.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.hydra
> $ diff -ru /tmp/nss-certs.{hydra,berlin}
> Only in /tmp/nss-certs.hydra/etc/ssl/certs: AC_Raíz_Certicámara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
> Only in /tmp/nss-certs.berlin/etc/ssl/certs: AC_Ra?z_Certic?mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
> Only in /tmp/nss-certs.hydra/etc/ssl/certs: NetLock_Arany_=Class_Gold=_Főtanúsítvány:2.6.73.65.44.228.0.16.pem
> Only in /tmp/nss-certs.berlin/etc/ssl/certs: NetLock_Arany_=Class_Gold=_F?tan?s?tv?ny:2.6.73.65.44.228.0.16.pem

[...]

> On closer inspection the problem is elsewhere: the
> /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 directory on
> berlin has question marks in file names, so ‘guix publish’ is not to
> blame; instead the problem likely comes from ‘guix offload’.
>
> Indeed ‘guix-daemon’ and its child processes such as ‘guix offload’ run
> with an empty environment, and thus in the C locale.  Specifically,
> ‘restore-file-set’ on the build farm front-end must be the one
> substituting question marks to the non-ASCII characters.
>
> If this analysis is correct, the patch below should fix it.  I’ll try it
> later.

Pushed as 7e4bc215098f334bc2a11737f2665dd4992fc2da.

Thanks,
Ludo'.




Information forwarded to bug-guix <at> gnu.org:
bug#32942; Package guix. (Wed, 19 Dec 2018 17:36:01 GMT) Full text and rfc822 format available.

Message #20 received at 32942 <at> debbugs.gnu.org (full text, mbox):

From: swedebugia <swedebugia <at> riseup.net>
To: Ludovic Courtès <ludo <at> gnu.org>,
 Julien Lepiller <julien <at> lepiller.eu>
Cc: 32942 <at> debbugs.gnu.org
Subject: Re: bug#32942: nss-certs not deterministic
Date: Wed, 19 Dec 2018 18:42:19 +0100
On 2018-12-05 15:01, Ludovic Courtès wrote:
> Hello,
> 
> Julien Lepiller <julien <at> lepiller.eu> skribis:
> 
>> While updating a profile, I found that nss-certs was not
>> deterministic. From ludo:
>>
>> $ wget -O - -q
>> https://mirror.hydra.gnu.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo| grep Hash
>> NarHash: sha256:101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
>> $ wget -O - -q
>> https://berlin.guixsd.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo |
>> grep Hash
>> NarHash: sha256:08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s
> 
> As shown above, berlin and hydra disagree on nss-certs.
> 
> The difference is an encoding bug:
> 
> --8<---------------cut here---------------start------------->8---
> $ wget -O - https://berlin.guixsd.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.berlin
> $ wget -O - https://mirror.hydra.gnu.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.hydra
> $ diff -ru /tmp/nss-certs.{hydra,berlin}
> Only in /tmp/nss-certs.hydra/etc/ssl/certs: AC_Raíz_Certicámara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
> Only in /tmp/nss-certs.berlin/etc/ssl/certs: AC_Ra?z_Certic?mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
> Only in /tmp/nss-certs.hydra/etc/ssl/certs: NetLock_Arany_=Class_Gold=_Főtanúsítvány:2.6.73.65.44.228.0.16.pem
> Only in /tmp/nss-certs.berlin/etc/ssl/certs: NetLock_Arany_=Class_Gold=_F?tan?s?tv?ny:2.6.73.65.44.228.0.16.pem
> --8<---------------cut here---------------end--------------->8---
> 
> The problem was already reported as <https://bugs.gnu.org/26948> and
> since commit 412701b0e5e073e6767eed162c14698db99df69c (July 2017) ‘guix
> publish’ on GuixSD runs in a UTF-8 locale to avoid that problem.
> 
> The faulty narinfo/nar on berlin were generated on Oct. 17, 2018, so
> clearly the above commit was in effect.  Indeed, after removing them and
> regenerating them, I’m still getting
> 08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s (aka. the wrong
> hash).
> 
> On closer inspection the problem is elsewhere: the
> /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 directory on
> berlin has question marks in file names, so ‘guix publish’ is not to
> blame; instead the problem likely comes from ‘guix offload’.
> 
> Indeed ‘guix-daemon’ and its child processes such as ‘guix offload’ run
> with an empty environment, and thus in the C locale.  Specifically,
> ‘restore-file-set’ on the build farm front-end must be the one
> substituting question marks to the non-ASCII characters.
> 
> If this analysis is correct, the patch below should fix it.  I’ll try it
> later.
> 
> Thanks,
> Ludo’.
> 
> 
> 
> diff --git a/gnu/services/base.scm b/gnu/services/base.scm
> index cee9898d79..9fe64e8087 100644
> --- a/gnu/services/base.scm
> +++ b/gnu/services/base.scm
> @@ -1603,7 +1603,15 @@ failed to register public key '~a': ~a~%" key status))))))))
>                               '())
>                        #$@(if tmpdir
>                               (list (string-append "TMPDIR=" tmpdir))
> -                            '()))
> +                            '())
> +
> +                     ;; Make sure we run in a UTF-8 locale so that 'guix
> +                     ;; offload' correctly restores nars that contain UTF-8
> +                     ;; file names such as 'nss-certs'.  See
> +                     ;; <https://bugs.gnu.org/32942>.
> +                     (string-append "GUIX_LOCPATH="
> +                                    #$glibc-utf8-locales "/lib/locale")
> +                     "LC_ALL=en_US.utf8")
>   
>                  #:log-file #$log-file))
>              (stop #~(make-kill-destructor))))))
> 

Congratulations with the succeded hunt and thanks a lot for showing all 
the steps you took so I can improve my hunting skills and eventually 
begin helping by hunting on my own :D

-- 
Cheers Swedebugia




bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Thu, 17 Jan 2019 12:24:06 GMT) Full text and rfc822 format available.

This bug report was last modified 5 years and 62 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.