GNU bug report logs - #28664
'guix publish' fails to guarantee TTL for already-compressed store items

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 28664 in the body.
You can then email your comments to 28664 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: bug-guix <bug-guix <at> gnu.org>
Subject: gst-plugins-base-1.12.3.tar.xz nar on berlin returns 404
Date: Sun, 01 Oct 2017 15:57:06 -0400

Hello,

While attempting to update today I got that:

--8<---------------cut here---------------start------------->8---
Downloading https://berlin.guixsd.org/nar/gzip/3vfvi0qcb5fwwm2pdy3vx06v4jhmwr3d-gtk%2B-2.24.31...
 gtk%2B-2.24.31  10.0MiB                                                                                                                                                                          1.3MiB/s 00:07 [####################] 100.0%

Downloading https://berlin.guixsd.org/nar/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697-gst-plugins-base-1.12.3.tar.xz...
guix substitute: error: download from 'https://berlin.guixsd.org/nar/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697-gst-plugins-base-1.12.3.tar.xz' failed: 404, "Not Found"
guix package: error: build failed: some substitutes for the outputs of derivation `/gnu/store/vcjfcp3va23yqcvg9gbcqwk9fm19hh1c-gst-plugins-base-1.12.3.tar.xz.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
--8<---------------cut here---------------end--------------->8---

Maxim

Message #8 received at 28664 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 28664 <at> debbugs.gnu.org
Subject: Re: bug#28664: gst-plugins-base-1.12.3.tar.xz nar on berlin returns
 404
Date: Mon, 02 Oct 2017 17:15:22 +0200

Hello Maxim,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:

> Downloading https://berlin.guixsd.org/nar/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697-gst-plugins-base-1.12.3.tar.xz...
> guix substitute: error: download from 'https://berlin.guixsd.org/nar/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697-gst-plugins-base-1.12.3.tar.xz' failed: 404, "Not Found"

I think it’s a bug in ‘guix publish’: the narinfo is still here:

--8<---------------cut here---------------start------------->8---
$ wget -q -O - https://berlin.guixsd.org/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697.narinfo
StorePath: /gnu/store/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697-gst-plugins-base-1.12.3.tar.xz
URL: nar/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697-gst-plugins-base-1.12.3.tar.xz
Compression: none
NarHash: sha256:07rx2px28pv6vqvzww4i0sldjray4q4jfggz9x0vp3wqla78rwm5
NarSize: 3117480
References: 
FileSize: 3117480
System: i686-linux
Deriver: mx8rpg1qqbl4244r2xsn3g59lw671sh3-gst-plugins-base-1.12.3.tar.xz.drv
Signature: 1;berlin.guixsd.org;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjMEZFNjA0QjNBRkJGRDYzRjM3RjA3NjJCMjg2MDBGMDFFRUVEQkZCNTE5RTQyNDAwMjNGOEE4NzlBRTA4OTVFOSMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDhEMzhFNkFGM0ZGNzFGNENBMTBENTIwRjU5OTUyREZGNEMxQTREMzREODkwNEYwODVFMDY4Q0FGNTgyNjE4MCMpCiAgIChzICMwNDQwODg2RDEzQTVERkM5RDE4NURFMEJBNzkxMTdGREUxNEQ0OUEyMDE3MUJDQ0M4RjQxRUQ4NDQ5NzcyNDM1IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjOEQxNTZGMjk1RDI0QjBEOUE4NkZBNTc0MUE4NDBGRjJEMjRGNjBGN0I2QzQxMzQ4MTRBRDU1NjI1OTcxQjM5NCMpCiAgICkKICApCiApCg==
--8<---------------cut here---------------end--------------->8---

but the tarball itself is missing.

On berlin, the narinfo is indeed around:

--8<---------------cut here---------------start------------->8---
$ sudo ls -l /var/cache/guix/publish/none/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697*
-rw------- 1 guix-publish guix-publish 998 Sep 19 11:12 /var/cache/guix/publish/none/g2ph0s1bjnzzn3q3h2pb8zw87bdxf697-gst-plugins-base-1.12.3.tar.xz.narinfo
--8<---------------cut here---------------end--------------->8---

So the problem is that for uncompressed store items (‘guix publish’
serves *.tar.xz without any additional compression, so it uses the
‘none’ compression method from its viewpoint), ‘guix publish’ does not
keep the nar in /var/cache/guix/publish, so it cannot guarantee its TTL.

The fix is probably simply to store nars even for uncompressed store
items, even if that increases disk usage.

Ludo’.

Message #15 received at 28664 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 28664 <at> debbugs.gnu.org
Subject: Re: bug#28664: gst-plugins-base-1.12.3.tar.xz nar on berlin returns
 404
Date: Thu, 05 Oct 2017 23:06:00 +0200

ludo <at> gnu.org (Ludovic Courtès) skribis:

> So the problem is that for uncompressed store items (‘guix publish’
> serves *.tar.xz without any additional compression, so it uses the
> ‘none’ compression method from its viewpoint), ‘guix publish’ does not
> keep the nar in /var/cache/guix/publish, so it cannot guarantee its TTL.
>
> The fix is probably simply to store nars even for uncompressed store
> items, even if that increases disk usage.

Done in e5788ebbe1f45a1088249b9138de17b330609712.

(We still need to deploy the fix on berlin.guixsd.org & co.)

Ludo’.

Message #20 received at 28664-done <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 28664-done <at> debbugs.gnu.org
Subject: Re: bug#28664: gst-plugins-base-1.12.3.tar.xz nar on berlin returns
 404
Date: Wed, 18 Oct 2017 23:35:01 +0200

ludo <at> gnu.org (Ludovic Courtès) skribis:

> ludo <at> gnu.org (Ludovic Courtès) skribis:
>
>> So the problem is that for uncompressed store items (‘guix publish’
>> serves *.tar.xz without any additional compression, so it uses the
>> ‘none’ compression method from its viewpoint), ‘guix publish’ does not
>> keep the nar in /var/cache/guix/publish, so it cannot guarantee its TTL.
>>
>> The fix is probably simply to store nars even for uncompressed store
>> items, even if that increases disk usage.
>
> Done in e5788ebbe1f45a1088249b9138de17b330609712.
>
> (We still need to deploy the fix on berlin.guixsd.org & co.)

It’s now deployed on berlin.guixsd.org.

Ludo’.

Message #23 received at 28664-done <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: guix-sysadmin <at> gnu.org
Cc: 28664-done <at> debbugs.gnu.org
Subject: Old narinfos of tarballs purged on berlin
Date: Thu, 19 Oct 2017 12:27:03 +0200

Hello there,

As a followup to <https://bugs.gnu.org/28664>, Ricardo upgraded to the
latest Guix on berlin, and I’ve now removed old .narinfos corresponding
to tarballs for which we potentially lacked the corresponding nar:

  sudo find /var/cache/guix/publish/none -mtime +7 -exec rm -v {} \;

There’s caching in nginx and on the user’s machines, so for a while we
might still get 200 for the narinfo and 404 for the corresponding nar,
but that will become consistent over time.

Apparently hydra.gnu.org is largely unaffected, presumably because we
GC’d less aggressively or have more GC roots.

Ludo’.

Message #26 received at 28664-done <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: guix-sysadmin <at> gnu.org
Cc: 28664-done <at> debbugs.gnu.org
Subject: Re: Old narinfos of tarballs purged on berlin
Date: Mon, 06 Nov 2017 09:40:11 +0100

Hi,

ludo <at> gnu.org (Ludovic Courtès) skribis:

> As a followup to <https://bugs.gnu.org/28664>, Ricardo upgraded to the
> latest Guix on berlin, and I’ve now removed old .narinfos corresponding
> to tarballs for which we potentially lacked the corresponding nar:
>
>   sudo find /var/cache/guix/publish/none -mtime +7 -exec rm -v {} \;

This missed ~100 of them, so I’ve run the following command, which is
more accurate:

  root <at> berlin /var/cache/guix/publish/none# rm -v $(for i in *.narinfo ; do [ -f `echo $i | sed -es/narinfo/nar/g` ] || echo $i ; done)

Ludo’.

Message #29 received at 28664-done <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: guix-sysadmin <at> gnu.org
Cc: 28664-done <at> debbugs.gnu.org
Subject: Re: Old narinfos of tarballs purged on berlin
Date: Tue, 07 Nov 2017 17:37:58 +0100

ludo <at> gnu.org (Ludovic Courtès) skribis:

> ludo <at> gnu.org (Ludovic Courtès) skribis:
>
>> As a followup to <https://bugs.gnu.org/28664>, Ricardo upgraded to the
>> latest Guix on berlin, and I’ve now removed old .narinfos corresponding
>> to tarballs for which we potentially lacked the corresponding nar:
>>
>>   sudo find /var/cache/guix/publish/none -mtime +7 -exec rm -v {} \;
>
> This missed ~100 of them, so I’ve run the following command, which is
> more accurate:
>
>   root <at> berlin /var/cache/guix/publish/none# rm -v $(for i in *.narinfo ; do [ -f `echo $i | sed -es/narinfo/nar/g` ] || echo $i ; done)

I just did the same on hydra.gnu.org, which was starting to have 404 for
these substitutes.

Ludo’.

Message #32 received at 28664-done <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: guix-sysadmin <at> gnu.org
Cc: 28664-done <at> debbugs.gnu.org
Subject: Re: Old narinfos of tarballs purged on berlin
Date: Wed, 08 Nov 2017 21:29:06 +0100

ludo <at> gnu.org (Ludovic Courtès) skribis:

> ludo <at> gnu.org (Ludovic Courtès) skribis:
>
>> ludo <at> gnu.org (Ludovic Courtès) skribis:
>>
>>> As a followup to <https://bugs.gnu.org/28664>, Ricardo upgraded to the
>>> latest Guix on berlin, and I’ve now removed old .narinfos corresponding
>>> to tarballs for which we potentially lacked the corresponding nar:
>>>
>>>   sudo find /var/cache/guix/publish/none -mtime +7 -exec rm -v {} \;
>>
>> This missed ~100 of them, so I’ve run the following command, which is
>> more accurate:
>>
>>   root <at> berlin /var/cache/guix/publish/none# rm -v $(for i in *.narinfo ; do [ -f `echo $i | sed -es/narinfo/nar/g` ] || echo $i ; done)
>
> I just did the same on hydra.gnu.org, which was starting to have 404 for
> these substitutes.

With narinfo caching people were starting to get 404s for
mirror.hydra.gnu.org/guix/nar/xyz.  Based on an incomplete log of the
command above, I’ve repopulated /var/cache/guix/publish/none on
hydra.gnu.org in a pretty crude way:

  for i in $(sed -es/'^removed `\([^-]\+\)-.*/\1/g' < /tmp/file-list ); do wget -q -O /dev/null http://localhost:9999/$i.narinfo || echo $i ; done

That’s 1,135 tarballs back in /var/cache/guix/publish/none.

The remaining missing items will come back eventually as narinfo
requests hit ‘guix publish’ itself (i.e., when narinfo caches on user
machines and in nginx have expired).

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.