GNU bug report logs - #22937
Substitutes cannot be downloaded over HTTPS

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 22937 in the body.
You can then email your comments to 22937 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Chris Marusich <cmmarusich <at> gmail.com>
To: bug-guix <at> gnu.org
Subject: guix package fails when --substitute-urls specifies an HTTPS endpoint
Date: Mon, 07 Mar 2016 22:16:30 -0800

Hi,

I've noticed that "guix package" fails when I specify an HTTPS endpoint
for --substitute-urls. Is that expected behavior?

I recently set up a caching proxy for hydra.gnu.org. The endpoint's DNS
name is hydra-mirror.marusich.info. It's configured to accept both HTTP
and HTTPS requests. When it receives an HTTPS request, it proxies the
request (assuming it's a cache miss) to hydra.gnu.org over HTTP. I've
configured it this way because my understanding is that hydra.gnu.org is
currently only available via HTTP.

Commands like wget, curl, and even "guix download" work fine with the
mirror, even over HTTPS. For example, the following command succeeds:

guix download https://hydra-mirror.marusich.info/nar/8kvb2k0n1jqjd5sa7g6qj5qllq0ckcya-linux-libre-4.4

However, when I try to install a package with "guix package" using the
endpoint, it fails with a backtrace like the following (this command was
invoked using the "guix package" from commit
7b3f2682de38a8e39f052705795ec85fcdfc8a96):

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix package --substitute-urls="https://hydra-mirror.marusich.info" -i graphviz
substitute: Backtrace:
substitute: In ice-9/boot-9.scm:
substitute:   63: 19 [call-with-prompt prompt0 ...]
substitute: In ice-9/eval.scm:
substitute:  432: 18 [eval # #]
substitute: In ice-9/boot-9.scm:
substitute: 2401: 17 [save-module-excursion #<procedure 16ac940 at ice-9/boot-9.scm:4045:3 ()>]
substitute: 4050: 16 [#<procedure 16ac940 at ice-9/boot-9.scm:4045:3 ()>]
substitute: 1724: 15 [%start-stack load-stack ...]
substitute: 1729: 14 [#<procedure 16c4ea0 ()>]
substitute: In unknown file:
substitute:    ?: 13 [primitive-load "/gnu/store/3lg5c1nidbj0kjdz5b63hn3vp29kzf0s-guix-0.9.0.c3f29bc/bin/.guix-real"]
substitute: In guix/ui.scm:
substitute: 1175: 12 [run-guix-command substitute "--query"]
substitute: In ice-9/boot-9.scm:
substitute:  157: 11 [catch getaddrinfo-error ...]
substitute:  157: 10 [catch srfi-34 #<procedure 25850a0 at guix/ui.scm:411:2 ()> ...]
substitute:  157: 9 [catch system-error ...]
substitute: In guix/scripts/substitute.scm:
substitute:  946: 8 [#<procedure 25850c0 at guix/scripts/substitute.scm:939:3 ()>]
substitute:  804: 7 [process-query "info /gnu/store/x29dbk9inqjym79q0907sx4arp1bfp28-graphviz-2.38.0-doc " ...]
substitute:  633: 6 [lookup-narinfos/diverse # #]
substitute:  617: 5 [lookup-narinfos "https://hydra-mirror.marusich.info" #]
substitute:  589: 4 [fetch-narinfos "https://hydra-mirror.marusich.info" #]
substitute:  222: 3 [download-cache-info "https://hydra-mirror.marusich.info"]
substitute: In guix/records.scm:
substitute:  331: 2 [recutils->alist #<unspecified>]
substitute: In ice-9/rdelim.scm:
substitute:  184: 1 [read-line #<unspecified> trim]
substitute: In unknown file:
substitute:    ?: 0 [%read-line #<unspecified>]
substitute: 
substitute: ERROR: In procedure %read-line:
substitute: ERROR: In procedure %read-line: Wrong type argument in position 1 (expecting open input port): #<unspecified>
guix package: error: build failed: substituter `substitute' died unexpectedly
--8<---------------cut here---------------end--------------->8---

When I replaced --substitute-urls="https://hydra-mirror.marusich.info"
with --substitute-urls="http://hydra-mirror.marusich.info", the same
command worked fine.

Chris

Message #8 received at 22937 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: 22937 <at> debbugs.gnu.org
Subject: Re: bug#22937: guix package fails when --substitute-urls specifies an
 HTTPS endpoint
Date: Wed, 09 Mar 2016 23:51:57 +0100

[Message part 1 (text/plain, inline)]

First, the graceless error handling is fixed in
204d34ff961d6dabf18b255decc29712e03afef0.

Second, here’s a preliminary patch that almost works with
<https://hydra-mirror.marusich.info>.

The problem is that sometimes the server closes the connection
unexpectedly, leading to an obscure backtrace like this:

--8<---------------cut here---------------start------------->8---
substitute:  629: 6 [lookup-narinfos "https://hydra-mirror.marusich.info" #]
substitute:  585: 5 [fetch-narinfos "https://hydra-mirror.marusich.info" #]
substitute:  510: 4 [http-multiple-get # ...]
substitute: In web/response.scm:
substitute:  197: 3 [read-response #<input-output: gnutls-session-port 38f59c0>]
substitute: In web/http.scm:
substitute: 1157: 2 [read-response-line #<input-output: gnutls-session-port 38f59c0>]
substitute:  151: 1 [read-header-line #<input-output: gnutls-session-port 38f59c0>]
substitute: In unknown file:
substitute:    ?: 0 [%read-line #<input-output: gnutls-session-port 38f59c0>]
substitute: 
substitute: ERROR: In procedure %read-line:
substitute: ERROR: Throw to key `gnutls-error' with args `(#<gnutls-error-enum Error in the pull function.> fill_session_record_port_input)'.
--8<---------------cut here---------------end--------------->8---

The “error in the pull function” is because ‘gnutls_record_recv’ got
ECONNRESET while reading.

I wonder whether this could be due to the particular configuration of
nginx at Cloudfront, so I’ll try with another server (I’ve set up Let’s
Encrypt on that server but it’s not accessible yet via port 443.)

To be continued!

Ludo’.

[Message part 2 (text/x-patch, inline)]

diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index b82fc17..df95de0 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -32,6 +32,7 @@
   #:use-module ((guix build utils) #:select (mkdir-p dump-port))
   #:use-module ((guix build download)
                 #:select (progress-proc uri-abbreviation
+                          open-connection-for-uri
                           store-path-abbreviation byte-count->string))
   #:use-module (ice-9 rdelim)
   #:use-module (ice-9 regex)
@@ -171,7 +172,7 @@ to the caller without emitting an error message."
      (let ((port (open-file (uri-path uri)
                             (if buffered? "rb" "r0b"))))
        (values port (stat:size (stat port)))))
-    ((http)
+    ((http https)
      (guard (c ((http-get-error? c)
                 (let ((code (http-get-error-code c)))
                   (if (and (= code 404) quiet-404?)
@@ -201,10 +202,10 @@ to the caller without emitting an error message."
                  (close-port port))))
            (begin
              (when (or (not port) (port-closed? port))
-               (set! port (open-socket-for-uri uri))
+               (set! port (open-connection-for-uri uri))
                (unless buffered?
                  (setvbuf port _IONBF)))
-             (http-fetch uri #:text? #f #:port port))))))))
+             (http-fetch uri #:text? #f #:port port))))))
     (else
      (leave (_ "unsupported substitute URI scheme: ~a~%")
             (uri->string uri)))))
@@ -478,20 +479,26 @@ may be #f, in which case it indicates that PATH is unavailable at CACHE-URL."
                             ".narinfo")))
     (build-request (string->uri url) #:method 'GET)))
 
-(define (http-multiple-get base-url proc seed requests)
-  "Send all of REQUESTS to the server at BASE-URL.  Call PROC for each
+(define (http-multiple-get base-uri proc seed requests)
+  "Send all of REQUESTS to the server at BASE-URI.  Call PROC for each
 response, passing it the request object, the response, a port from which to
 read the response body, and the previous result, starting with SEED, à la
 'fold'.  Return the final result."
   (let connect ((requests requests)
                 (result   seed))
-    ;; (format (current-error-port) "connecting (~a requests left)..."
-    ;;         (length requests))
-    (let ((p (open-socket-for-uri base-url)))
+    (format (current-error-port) "connecting (~a requests left)..."
+            (length requests))
+    (let ((p (open-connection-for-uri base-uri)))
+      ;; For HTTPS, P is not a file port and does not support 'setvbuf'.
+      (when (file-port? p)
+        (setvbuf p _IOFBF (expt 2 16)))
+
       ;; Send all of REQUESTS in a row.
-      (setvbuf p _IOFBF (expt 2 16))
-      (for-each (cut write-request <> p) requests)
-      (force-output p)
+      ;; XXX: Do our own caching to work around <http://bugs.gnu.org/22966>.
+      (let-values (((buffer get) (open-bytevector-output-port)))
+        (for-each (cut write-request <> buffer) requests)
+        (put-bytevector p (get))
+        (force-output p))
 
       ;; Now start processing responses.
       (let loop ((requests requests)
@@ -501,6 +508,8 @@ read the response body, and the previous result, starting with SEED, à la
            (reverse result))
           ((head tail ...)
            (let* ((resp   (read-response p))
+                  ;; (xxx    (format (current-error-port)
+                  ;;                 "http response: ~s~%" resp))
                   (body   (response-body-port resp))
                   (result (proc head resp body result)))
              ;; The server can choose to stop responding at any time, in which
@@ -570,10 +579,10 @@ if file doesn't exist, and the narinfo otherwise."
 
   (define (do-fetch uri)
     (case (and=> uri uri-scheme)
-      ((http)
+      ((http https)
        (let ((requests (map (cut narinfo-request url <>) paths)))
          (update-progress!)
-         (let ((result (http-multiple-get url
+         (let ((result (http-multiple-get uri
                                           handle-narinfo-response '()
                                           requests)))
            (newline (current-error-port))

Message #17 received at 22937-done <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: 22937-done <at> debbugs.gnu.org
Subject: Re: bug#22937: guix package fails when --substitute-urls specifies an
 HTTPS endpoint
Date: Thu, 10 Mar 2016 14:08:17 +0100

After some more testing and tweaks, notably with
<https://hydra.gnunet.org> and its brand new Let’s Encrypt certificate,
I pushed the patch as 9b7bd1b160be7c740a798c09e3b8986368b92036.

I can no longer reproduce the ECONNRESET issue I mentioned for
<https://hydra-mirror.marusich.info>, but feel free to open a new bug if
that comes up again!

For the record, the main test I used for that was:

  $ sudo rm -rf /var/guix/substitute/cache
  $ guix build libreoffice --no-grafts -n \
        --substitute-urls=https://hydra.gnunet.org

That forces a redownload of all .narinfo files from hydra.gnunet.org,
which involves HTTP pipelining.

Anyway, thanks for giving me an incentive to address this.  This was
long overdue!

Ludo’.

Message #20 received at 22937 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: 22937 <at> debbugs.gnu.org, ludo <at> gnu.org, cmmarusich <at> gmail.com
Subject: Re: bug#22937: guix package fails when --substitute-urls specifies
 an HTTPS endpoint
Date: Thu, 10 Mar 2016 16:21:33 -0500

On Thu, Mar 10, 2016 at 02:08:17PM +0100, Ludovic Courtès wrote:
> After some more testing and tweaks, notably with
> <https://hydra.gnunet.org> and its brand new Let’s Encrypt certificate,
> I pushed the patch as 9b7bd1b160be7c740a798c09e3b8986368b92036.

With this commit, I still get the backtrace in the original report of
this bug.

I ran `guix pull` and confirmed the changes were in ~/.config/guix/latest.

I cleared the cache at /vat/guix/substitute/cache.

Then, I updated --substitute-urls in the daemon invocation and reloaded the
daemon, and then did `guix build hello`.

Did I miss a step?

> 
> I can no longer reproduce the ECONNRESET issue I mentioned for
> <https://hydra-mirror.marusich.info>, but feel free to open a new bug if
> that comes up again!
> 
> For the record, the main test I used for that was:
> 
>   $ sudo rm -rf /var/guix/substitute/cache
>   $ guix build libreoffice --no-grafts -n \
>         --substitute-urls=https://hydra.gnunet.org
> 
> That forces a redownload of all .narinfo files from hydra.gnunet.org,
> which involves HTTP pipelining.
> 
> Anyway, thanks for giving me an incentive to address this.  This was
> long overdue!
> 
> Ludo’.
> 
> 
> 

Message #23 received at 22937 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Leo Famulari <leo <at> famulari.name>
Cc: cmmarusich <at> gmail.com, 22937 <at> debbugs.gnu.org
Subject: Re: bug#22937: guix package fails when --substitute-urls specifies an
 HTTPS endpoint
Date: Fri, 11 Mar 2016 09:39:10 +0100

Leo Famulari <leo <at> famulari.name> skribis:

> On Thu, Mar 10, 2016 at 02:08:17PM +0100, Ludovic Courtès wrote:
>> After some more testing and tweaks, notably with
>> <https://hydra.gnunet.org> and its brand new Let’s Encrypt certificate,
>> I pushed the patch as 9b7bd1b160be7c740a798c09e3b8986368b92036.
>
> With this commit, I still get the backtrace in the original report of
> this bug.

You need to make sure that ‘guix-daemon’ launches the right ‘guix
substitute’.

A simple way to do that from your Git checkout is:

  sudo ./pre-inst-env guix-daemon --build-users-group=guixbuild \
         --substitute-urls=https://mirror.guixsd.org

Make sure to stop any running guix-daemon before; on GuixSD, this goes
like this:

  sudo herd stop guix-daemon

HTH!

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.