GNU bug report logs - #42286
SWH fallback fails (git-fetch)

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 42286 in the body.
You can then email your comments to 42286 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: bug-guix <at> gnu.org
Subject: SWH fallback fails (git-fetch)
Date: Thu, 09 Jul 2020 02:36:55 +0200

Dear,

I think the fallback with SWH is broken.  Here an example to reproduce.

--8<---------------cut here---------------start------------->8---
cd /tmp/
tar -xvf $(guix build -S hello)
cd hello-2.10
git init
git add *
git commit -m 'init'
--8<---------------cut here---------------end--------------->8---

Then I push this fresh repo to my own GitHub account.  Because it is
easy to switch between public and private state option, which mimics
upstream loss.

Let create the well-known "hello" recipe, which is simply a copy/paste
of (gnu packages base).

--8<---------------cut here---------------start------------->8---
(define-module (hello)
  #:use-module (guix packages)
  #:use-module (guix build-system gnu)
  #:use-module (guix git-download)
  #:use-module (guix licenses))

(define-public hi
  (package
    (name "hi")
    (version "2.10")
    (source (origin
              (method git-fetch)
              (uri (git-reference
                    (url "https://github.com/zimoun/hello-example.git")
                    (commit "e1eefd033b8a2c4c81babc6fde08ebb116c6abb8")))
              (sha256
               (base32
                "1im1gglfm4k10bh4mdaqzmx3lm3kivnsmxrvl6vyvmfqqzljq75l"))))
    (build-system gnu-build-system)
    (synopsis "Hello, GNU world: An example GNU package")
    (description
     "GNU Hello prints the message \"Hello, world!\" and then exits.  It
serves as an example of standard GNU coding practices.  As such, it supports
command-line arguments, multiple languages, and so on.")
    (home-page "https://www.gnu.org/software/hello/")
    (license gpl3+)))
--8<---------------cut here---------------end--------------->8---

For the sanity of the test, "guix build -L . hi" builds correctly and
returns the item: /gnu/store/jn8d031zx4znxy7s5zhj4dbr6xjsfq9v-hi-2.10
with Guix b7249aa.

The upstream source is archived on SWH with:

   guix lint -L . hi -c archival

see
https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://github.com/zimoun/hello-example.git
and the logs in https://archive.softwareheritage.org/save/#requests

Note that running again "lint -c archival" returns nothing, which means
that the linter finds it on SWH, I guess.  Well, I have not checked what
'lookup-content' returns yet.  However, I filtered the first ~2000+
'packages-with-sources' with 'lookup-content' and all the 'git-fetch'
sources are returned #f, and some 'url-fetch' not.  Another story,
another thread. :-)


Let clean the previous builds and downloads:

    guix gc -D $(guix gc --list-dead | grep -E '(hi-2|git-checkout)')

And now let turn off upstream and so the maybe bug:

--8<---------------cut here---------------start------------->8---
guix build -L . hi
The following derivations will be built:
   /gnu/store/x5vwhv8sn3l49vysq98j7h2rqqlff5zp-hi-2.10.drv
   /gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv
building /gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv...
guile: warning: failed to install locale
environment variable `PATH' set to `/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-1.10/bin:/gnu/store/sf3rbvb6iqcphgm1afbplcs72hsywg25-tar-1.32/bin'
Initialized empty Git repository in /gnu/store/884nsva9r8wkp40kbqyvpj1ad57jc5dd-git-checkout/.git/
fatal: could not read Username for 'https://github.com': No such device or address
Failed to do a shallow fetch; retrying a full fetch...
fatal: could not read Username for 'https://github.com': No such device or address
git-fetch: '/gnu/store/i2pkspl2vjvas0q85hw46y2li06rcmcg-git-minimal-2.26.2/bin/git fetch origin' failed with exit code 128
Trying content-addressed mirror at berlin.guixsd.org...
Trying content-addressed mirror at berlin.guixsd.org...
Trying to download from Software Heritage...
Backtrace:
           4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
In ./guix/swh.scm:
   573:13  3 (swh-download "https://github.com/zimoun/hello-example?" ?)
   224:22  2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
In web/client.scm:
    563:0  1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
    231:6  0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)

web/client.scm:231:6: In procedure tls-wrap:
Error while printing exception.
builder for `/gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv' failed with exit code 1
build of /gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv failed
View build log at '/var/log/guix/drvs/jn/6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv.bz2'.
cannot build derivation `/gnu/store/x5vwhv8sn3l49vysq98j7h2rqqlff5zp-hi-2.10.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/x5vwhv8sn3l49vysq98j7h2rqqlff5zp-hi-2.10.drv' failed
--8<---------------cut here---------------end--------------->8---


Let me know if moreinfo are required.

All the best,
simon

Message #8 received at 42286 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: 42286 <at> debbugs.gnu.org
Subject: swh-download with https->http
Date: Thu, 09 Jul 2020 03:52:58 +0200

On Thu, 09 Jul 2020 at 02:36, zimoun <zimon.toutoune <at> gmail.com> wrote:

> --8<---------------cut here---------------start------------->8---
> guix build -L . hi

[...]

> Trying to download from Software Heritage...
> Backtrace:
>            4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
> In ./guix/swh.scm:
>    573:13  3 (swh-download "https://github.com/zimoun/hello-example?" ?)
>    224:22  2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
> In web/client.scm:
>     563:0  1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
>     231:6  0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)
>
> web/client.scm:231:6: In procedure tls-wrap:
> Error while printing exception.

[...]

> --8<---------------cut here---------------end--------------->8---

To avoid, this error which should be a missing certificate, I switch to
'http:' instead of 'https:' in %swh-base-url.  The error is then:

--8<---------------cut here---------------start------------->8---
Trying to download from Software Heritage...
Backtrace:
           2 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
In ./guix/swh.scm:
   573:13  1 (swh-download "https://github.com/zimoun/hello-example?" ?)
   243:16  0 (call "http://archive.softwareheritage.org/api/1/revis?" ?)

./guix/swh.scm:243:16: In procedure call:
Throw to key `swh-error' with args `("http://archive.softwareheritage.org/api/1/revision/e1eefd033b8a2c4c81babc6fde08ebb116c6abb8/" #<procedure http-get (uri #:key body port version keep-alive? headers decode-body? verify-certificate? streaming?)> #<<response> version: (1 . 1) code: 302 reason-phrase: "Found" headers: ((date . #<date nanosecond: 0 second: 59 minute: 38 hour: 1 day: 9 month: 7 year: 2020 zone-offset: 0>) (server . "Varnish") (x-varnish . "19881791") (location . #<<uri> scheme: https userinfo: #f host: "archive.softwareheritage.org" port: #f path: "/api/1/revision/e1eefd033b8a2c4c81babc6fde08ebb116c6abb8/" query: #f fragment: #f>) (content-length . 0) (connection close)) port: #<closed: file 7ffff5f1b690>>)'.
builder for `/gnu/store/9f6648zb0p9z6hxr31wz3d7fm7ndcxzx-git-checkout.drv' failed with exit code 1
--8<---------------cut here---------------end--------------->8---


Because I hit the rate limit, I need to wait at least one hour...
Follow-ups soon. :-)

Cheers,
simon

Message #13 received at 42286 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: 42286 <at> debbugs.gnu.org
Subject: Re: bug#42286: SWH fallback fails (git-fetch)
Date: Thu, 9 Jul 2020 20:20:23 +0200

On Thu, 9 Jul 2020 at 02:39, zimoun <zimon.toutoune <at> gmail.com> wrote:

Well, "swh-download" perfectly works directly from the REPL.

> --8<---------------cut here---------------start------------->8---
> guix build -L . hi

[...]

> Trying to download from Software Heritage...
> Backtrace:
>            4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
> In ./guix/swh.scm:
>    573:13  3 (swh-download "https://github.com/zimoun/hello-example?" ?)
>    224:22  2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
> In web/client.scm:
>     563:0  1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
>     231:6  0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)
>
> web/client.scm:231:6: In procedure tls-wrap:
> --8<---------------cut here---------------end--------------->8---

The error is definitively something related to TLS and the gexp.  In
(guix git-download) "git-fetch", the "define build" returns a gexp and
here something is missing, even if the extension gnutls (module-ref
(resolve-interface '(gnu packages tls)) 'gnutls) is provided.  Hum, I
am not sure to understand what.

BTW, if in "git-fetch" from (guix git), I add these lines:

--8<---------------cut here---------------start------------->8---
  (setenv "GIT_SSL_NO_VERIFY" "true")

  (format #t "git-fetch~%")
  (http-get "https://archive.softwareheritage.org/api/1/content/sha256:31e066137a962676e89f69d1b65382de95a7ef7d914b8cb956f41ea72e0f516b/")
  (format #t "ok~%")

  (mkdir-p directory)
--8<---------------cut here---------------end--------------->8---

Then I also hit:

--8<---------------cut here---------------start------------->8---
The following derivations will be built:
   /gnu/store/wam9fca6vj3rifvqlix9c874vpwn5k82-hi-2.10.drv
   /gnu/store/cvp65m4wzmzd8pqdfvah4mrl4zkcw3vz-git-checkout.drv
building /gnu/store/cvp65m4wzmzd8pqdfvah4mrl4zkcw3vz-git-checkout.drv...
guile: warning: failed to install locale
environment variable `PATH' set to
`/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-1.10/bin:/gnu/store/sf3rbvb6iqcphgm1afbplcs72hsywg25-tar-1.32/bin'
git-fetch
Backtrace:
           4 (primitive-load "/gnu/store/gcr6v6p6c5gwr4l6xzqcy6wln33?")
In ice-9/eval.scm:
   293:34  3 (_ #<directory (guile-user) 7ffff5bb8f00>)
In ./guix/build/git.scm:
     44:2  2 (git-fetch "https://github.com/zimoun/hello-example.git" ?)
In web/client.scm:
    563:0  1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
    231:6  0 (tls-wrap #<closed: file 7ffff4c17150> _ # _)

web/client.scm:231:6: In procedure tls-wrap:
Error while printing exception.
builder for `/gnu/store/cvp65m4wzmzd8pqdfvah4mrl4zkcw3vz-git-checkout.drv'
failed with exit code 1
build of /gnu/store/cvp65m4wzmzd8pqdfvah4mrl4zkcw3vz-git-checkout.drv failed
View build log at
'/var/log/guix/drvs/cv/p65m4wzmzd8pqdfvah4mrl4zkcw3vz-git-checkout.drv.bz2'.
cannot build derivation
`/gnu/store/wam9fca6vj3rifvqlix9c874vpwn5k82-hi-2.10.drv': 1
dependencies couldn't be built
guix build: error: build of
`/gnu/store/wam9fca6vj3rifvqlix9c874vpwn5k82-hi-2.10.drv' failed
--8<---------------cut here---------------end--------------->8---

Then I have tried to turn off the certificate verification with
"#:verify-certification #f" for example in "vault-fetch" or "call"
used by "define-query" but nothing works.  Well, I am a bit
circumspect.

Therefore, I am waiting for a hint or the fix. :-)

Cheers,
simon

Message #18 received at 42286-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 42286-done <at> debbugs.gnu.org
Subject: Re: bug#42286: SWH fallback fails (git-fetch)
Date: Fri, 10 Jul 2020 00:24:10 +0200

Hi!

zimoun <zimon.toutoune <at> gmail.com> skribis:

> Trying to download from Software Heritage...
> Backtrace:
>            4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
> In ./guix/swh.scm:
>    573:13  3 (swh-download "https://github.com/zimoun/hello-example?" ?)
>    224:22  2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
> In web/client.scm:
>     563:0  1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
>     231:6  0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)
>
> web/client.scm:231:6: In procedure tls-wrap:
> Error while printing exception.
> builder for `/gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv' failed with exit code 1

Should be fixed with commit a7696b9733d4ede9817a0a0accb5ce5b85d9a2d3.
Let me know if anything’s amiss.

Thanks!

Ludo’.

Message #21 received at 42286-done <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 42286-done <at> debbugs.gnu.org
Subject: Re: bug#42286: SWH fallback fails (git-fetch)
Date: Fri, 10 Jul 2020 05:18:25 +0200

Hi,

On Fri, 10 Jul 2020 at 00:24, Ludovic Courtès <ludo <at> gnu.org> wrote:
> Hi!
>
> zimoun <zimon.toutoune <at> gmail.com> skribis:
>
>> Trying to download from Software Heritage...
>> Backtrace:
>>            4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
>> In ./guix/swh.scm:
>>    573:13  3 (swh-download "https://github.com/zimoun/hello-example?" ?)
>>    224:22  2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
>> In web/client.scm:
>>     563:0  1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
>>     231:6  0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)
>>
>> web/client.scm:231:6: In procedure tls-wrap:
>> Error while printing exception.
>> builder for `/gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv' failed with exit code 1
>
> Should be fixed with commit a7696b9733d4ede9817a0a0accb5ce5b85d9a2d3.
> Let me know if anything’s amiss.

Cool! Works. :-)

I was almost there. :-) The missing trick was because the Guile bug
   <https://bugs.gnu.org/40486>
I was not aware and so the new "http-get*".

Is it worth to add the test in guix-build.sh?


All the best,
simon

Message #24 received at 42286-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: 42286-done <at> debbugs.gnu.org
Subject: Re: bug#42286: SWH fallback fails (git-fetch)
Date: Fri, 10 Jul 2020 10:07:33 +0200

Hi!

zimoun <zimon.toutoune <at> gmail.com> skribis:

> On Fri, 10 Jul 2020 at 00:24, Ludovic Courtès <ludo <at> gnu.org> wrote:
>> Hi!
>>
>> zimoun <zimon.toutoune <at> gmail.com> skribis:
>>
>>> Trying to download from Software Heritage...
>>> Backtrace:
>>>            4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
>>> In ./guix/swh.scm:
>>>    573:13  3 (swh-download "https://github.com/zimoun/hello-example?" ?)
>>>    224:22  2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
>>> In web/client.scm:
>>>     563:0  1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
>>>     231:6  0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)
>>>
>>> web/client.scm:231:6: In procedure tls-wrap:
>>> Error while printing exception.
>>> builder for `/gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv' failed with exit code 1
>>
>> Should be fixed with commit a7696b9733d4ede9817a0a0accb5ce5b85d9a2d3.
>> Let me know if anything’s amiss.
>
> Cool! Works. :-)
>
> I was almost there. :-) The missing trick was because the Guile bug
>    <https://bugs.gnu.org/40486>
> I was not aware and so the new "http-get*".

Yeah.  :-)

> Is it worth to add the test in guix-build.sh?

We don’t add tests that depend on external services, so we can’t really
do that.  Or we would need to mock the original server, SWH, etc. but
that seems tricky.

Thanks,
Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.