GNU bug report logs -
#32942
'guix offload' fails to restore non-ASCII file names when running in C locale
Previous Next
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 32942 in the body.
You can then email your comments to 32942 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-guix <at> gnu.org
:
bug#32942
; Package
guix
.
(Fri, 05 Oct 2018 08:19:02 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Julien Lepiller <julien <at> lepiller.eu>
:
New bug report received and forwarded. Copy sent to
bug-guix <at> gnu.org
.
(Fri, 05 Oct 2018 08:19:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
While updating a profile, I found that nss-certs was not deterministic.
From ludo:
$ wget -O - -q
https://mirror.hydra.gnu.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo |
grep Hash
NarHash: sha256:101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
$ wget -O - -q
https://berlin.guixsd.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo |
grep Hash
NarHash: sha256:08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s
Severity set to 'important' from 'normal'
Request was from
ludo <at> gnu.org (Ludovic Courtès)
to
control <at> debbugs.gnu.org
.
(Wed, 05 Dec 2018 11:49:02 GMT)
Full text and
rfc822 format available.
Information forwarded
to
bug-guix <at> gnu.org
:
bug#32942
; Package
guix
.
(Wed, 05 Dec 2018 14:02:02 GMT)
Full text and
rfc822 format available.
Message #10 received at 32942 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hello,
Julien Lepiller <julien <at> lepiller.eu> skribis:
> While updating a profile, I found that nss-certs was not
> deterministic. From ludo:
>
> $ wget -O - -q
> https://mirror.hydra.gnu.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo| grep Hash
> NarHash: sha256:101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
> $ wget -O - -q
> https://berlin.guixsd.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo |
> grep Hash
> NarHash: sha256:08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s
As shown above, berlin and hydra disagree on nss-certs.
The difference is an encoding bug:
--8<---------------cut here---------------start------------->8---
$ wget -O - https://berlin.guixsd.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.berlin
$ wget -O - https://mirror.hydra.gnu.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.hydra
$ diff -ru /tmp/nss-certs.{hydra,berlin}
Only in /tmp/nss-certs.hydra/etc/ssl/certs: AC_Raíz_Certicámara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
Only in /tmp/nss-certs.berlin/etc/ssl/certs: AC_Ra?z_Certic?mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
Only in /tmp/nss-certs.hydra/etc/ssl/certs: NetLock_Arany_=Class_Gold=_Főtanúsítvány:2.6.73.65.44.228.0.16.pem
Only in /tmp/nss-certs.berlin/etc/ssl/certs: NetLock_Arany_=Class_Gold=_F?tan?s?tv?ny:2.6.73.65.44.228.0.16.pem
--8<---------------cut here---------------end--------------->8---
The problem was already reported as <https://bugs.gnu.org/26948> and
since commit 412701b0e5e073e6767eed162c14698db99df69c (July 2017) ‘guix
publish’ on GuixSD runs in a UTF-8 locale to avoid that problem.
The faulty narinfo/nar on berlin were generated on Oct. 17, 2018, so
clearly the above commit was in effect. Indeed, after removing them and
regenerating them, I’m still getting
08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s (aka. the wrong
hash).
On closer inspection the problem is elsewhere: the
/gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 directory on
berlin has question marks in file names, so ‘guix publish’ is not to
blame; instead the problem likely comes from ‘guix offload’.
Indeed ‘guix-daemon’ and its child processes such as ‘guix offload’ run
with an empty environment, and thus in the C locale. Specifically,
‘restore-file-set’ on the build farm front-end must be the one
substituting question marks to the non-ASCII characters.
If this analysis is correct, the patch below should fix it. I’ll try it
later.
Thanks,
Ludo’.
[Message part 2 (text/x-patch, inline)]
diff --git a/gnu/services/base.scm b/gnu/services/base.scm
index cee9898d79..9fe64e8087 100644
--- a/gnu/services/base.scm
+++ b/gnu/services/base.scm
@@ -1603,7 +1603,15 @@ failed to register public key '~a': ~a~%" key status))))))))
'())
#$@(if tmpdir
(list (string-append "TMPDIR=" tmpdir))
- '()))
+ '())
+
+ ;; Make sure we run in a UTF-8 locale so that 'guix
+ ;; offload' correctly restores nars that contain UTF-8
+ ;; file names such as 'nss-certs'. See
+ ;; <https://bugs.gnu.org/32942>.
+ (string-append "GUIX_LOCPATH="
+ #$glibc-utf8-locales "/lib/locale")
+ "LC_ALL=en_US.utf8")
#:log-file #$log-file))
(stop #~(make-kill-destructor))))))
Changed bug title to ''guix offload' fails to restore non-ASCII file names when running in C locale' from 'nss-certs not deterministic'
Request was from
ludo <at> gnu.org (Ludovic Courtès)
to
control <at> debbugs.gnu.org
.
(Wed, 05 Dec 2018 14:03:01 GMT)
Full text and
rfc822 format available.
Reply sent
to
Ludovic Courtès <ludo <at> gnu.org>
:
You have taken responsibility.
(Sun, 09 Dec 2018 23:30:02 GMT)
Full text and
rfc822 format available.
Notification sent
to
Julien Lepiller <julien <at> lepiller.eu>
:
bug acknowledged by developer.
(Sun, 09 Dec 2018 23:30:02 GMT)
Full text and
rfc822 format available.
Message #17 received at 32942-done <at> debbugs.gnu.org (full text, mbox):
Hello,
ludo <at> gnu.org (Ludovic Courtès) skribis:
> The difference is an encoding bug:
>
> $ wget -O - https://berlin.guixsd.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.berlin
> $ wget -O - https://mirror.hydra.gnu.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.hydra
> $ diff -ru /tmp/nss-certs.{hydra,berlin}
> Only in /tmp/nss-certs.hydra/etc/ssl/certs: AC_Raíz_Certicámara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
> Only in /tmp/nss-certs.berlin/etc/ssl/certs: AC_Ra?z_Certic?mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
> Only in /tmp/nss-certs.hydra/etc/ssl/certs: NetLock_Arany_=Class_Gold=_Főtanúsítvány:2.6.73.65.44.228.0.16.pem
> Only in /tmp/nss-certs.berlin/etc/ssl/certs: NetLock_Arany_=Class_Gold=_F?tan?s?tv?ny:2.6.73.65.44.228.0.16.pem
[...]
> On closer inspection the problem is elsewhere: the
> /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 directory on
> berlin has question marks in file names, so ‘guix publish’ is not to
> blame; instead the problem likely comes from ‘guix offload’.
>
> Indeed ‘guix-daemon’ and its child processes such as ‘guix offload’ run
> with an empty environment, and thus in the C locale. Specifically,
> ‘restore-file-set’ on the build farm front-end must be the one
> substituting question marks to the non-ASCII characters.
>
> If this analysis is correct, the patch below should fix it. I’ll try it
> later.
Pushed as 7e4bc215098f334bc2a11737f2665dd4992fc2da.
Thanks,
Ludo'.
Information forwarded
to
bug-guix <at> gnu.org
:
bug#32942
; Package
guix
.
(Wed, 19 Dec 2018 17:36:01 GMT)
Full text and
rfc822 format available.
Message #20 received at 32942 <at> debbugs.gnu.org (full text, mbox):
On 2018-12-05 15:01, Ludovic Courtès wrote:
> Hello,
>
> Julien Lepiller <julien <at> lepiller.eu> skribis:
>
>> While updating a profile, I found that nss-certs was not
>> deterministic. From ludo:
>>
>> $ wget -O - -q
>> https://mirror.hydra.gnu.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo| grep Hash
>> NarHash: sha256:101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
>> $ wget -O - -q
>> https://berlin.guixsd.org/mbs5mavs3gi4y7xkywcwwjj9g3p1yjmv.narinfo |
>> grep Hash
>> NarHash: sha256:08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s
>
> As shown above, berlin and hydra disagree on nss-certs.
>
> The difference is an encoding bug:
>
> --8<---------------cut here---------------start------------->8---
> $ wget -O - https://berlin.guixsd.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.berlin
> $ wget -O - https://mirror.hydra.gnu.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix archive -x /tmp/nss-certs.hydra
> $ diff -ru /tmp/nss-certs.{hydra,berlin}
> Only in /tmp/nss-certs.hydra/etc/ssl/certs: AC_Raíz_Certicámara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
> Only in /tmp/nss-certs.berlin/etc/ssl/certs: AC_Ra?z_Certic?mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
> Only in /tmp/nss-certs.hydra/etc/ssl/certs: NetLock_Arany_=Class_Gold=_Főtanúsítvány:2.6.73.65.44.228.0.16.pem
> Only in /tmp/nss-certs.berlin/etc/ssl/certs: NetLock_Arany_=Class_Gold=_F?tan?s?tv?ny:2.6.73.65.44.228.0.16.pem
> --8<---------------cut here---------------end--------------->8---
>
> The problem was already reported as <https://bugs.gnu.org/26948> and
> since commit 412701b0e5e073e6767eed162c14698db99df69c (July 2017) ‘guix
> publish’ on GuixSD runs in a UTF-8 locale to avoid that problem.
>
> The faulty narinfo/nar on berlin were generated on Oct. 17, 2018, so
> clearly the above commit was in effect. Indeed, after removing them and
> regenerating them, I’m still getting
> 08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s (aka. the wrong
> hash).
>
> On closer inspection the problem is elsewhere: the
> /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 directory on
> berlin has question marks in file names, so ‘guix publish’ is not to
> blame; instead the problem likely comes from ‘guix offload’.
>
> Indeed ‘guix-daemon’ and its child processes such as ‘guix offload’ run
> with an empty environment, and thus in the C locale. Specifically,
> ‘restore-file-set’ on the build farm front-end must be the one
> substituting question marks to the non-ASCII characters.
>
> If this analysis is correct, the patch below should fix it. I’ll try it
> later.
>
> Thanks,
> Ludo’.
>
>
>
> diff --git a/gnu/services/base.scm b/gnu/services/base.scm
> index cee9898d79..9fe64e8087 100644
> --- a/gnu/services/base.scm
> +++ b/gnu/services/base.scm
> @@ -1603,7 +1603,15 @@ failed to register public key '~a': ~a~%" key status))))))))
> '())
> #$@(if tmpdir
> (list (string-append "TMPDIR=" tmpdir))
> - '()))
> + '())
> +
> + ;; Make sure we run in a UTF-8 locale so that 'guix
> + ;; offload' correctly restores nars that contain UTF-8
> + ;; file names such as 'nss-certs'. See
> + ;; <https://bugs.gnu.org/32942>.
> + (string-append "GUIX_LOCPATH="
> + #$glibc-utf8-locales "/lib/locale")
> + "LC_ALL=en_US.utf8")
>
> #:log-file #$log-file))
> (stop #~(make-kill-destructor))))))
>
Congratulations with the succeded hunt and thanks a lot for showing all
the steps you took so I can improve my hunting skills and eventually
begin helping by hunting on my own :D
--
Cheers Swedebugia
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org
.
(Thu, 17 Jan 2019 12:24:06 GMT)
Full text and
rfc822 format available.
This bug report was last modified 5 years and 62 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.