Package: guix;
Reported by: Joshua Randall <jcrandall <at> alum.mit.edu>
Date: Tue, 21 Apr 2015 23:01:01 UTC
Severity: normal
Found in version 0.8.1
To reply to this bug, email your comments to 20402 AT debbugs.gnu.org.
Toggle the display of automated, internal messages from the tracker.
View this report as an mbox folder, status mbox, maintainer mbox
bug-guix <at> gnu.org
:bug#20402
; Package guix
.
(Tue, 21 Apr 2015 23:01:02 GMT) Full text and rfc822 format available.Joshua Randall <jcrandall <at> alum.mit.edu>
:bug-guix <at> gnu.org
.
(Tue, 21 Apr 2015 23:01:02 GMT) Full text and rfc822 format available.Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
From: Joshua Randall <jcrandall <at> alum.mit.edu> To: submit <at> debbugs.gnu.org Subject: guix cannot download via an http proxy Date: Tue, 21 Apr 2015 23:56:02 +0100
[Message part 1 (text/plain, inline)]
Package: guix Version: 0.8.1 I am attempting to use guix from within a network that does not allow outbound http connections except via an http proxy. I am using Guile v2.0.11, which supports http proxies, so my expectation would be that since I have http_proxy and https_proxy environment variables set, guix would use the specified proxy for outbound http connections, but instead it appears to ignore the proxy settings and attempts to contact the http server directly, which results in a timeout. For example, when doing a `guix pull` I got the following: $ guix pull starting download of `/tmp/guix-file.l1zwZ7' from ` http://git.savannah.gnu.org/cgit/guix.git/snapshot/guix-master.tar.gz'... ERROR: In procedure connect: Connection timed out failed to download "/tmp/guix-file.l1zwZ7" from " http://git.savannah.gnu.org/cgit/guix.git/snapshot/guix-master.tar.gz" guix pull: error: failed to download up-to-date source, exiting It appears that Guile has had support for http proxies in the web client package since v2.0.10, and although guix is using the http-get method from Guile, it isn't using the open-socket-for-uri method, which is the one that implements making a proxy connection. Instead, guix seems to have copied and modified the code from an older version of open-socket-for-uri into open-connection-for-uri ( http://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/download.scm?id=v0.8.1#n153) and uses that instead. I suspect what has happened is that the Guile version of open-socket-for-uri has added proxy support since the code was copied into open-connection-for-uri. One fix would be to port over the changes to open-socket-for-uri that were made in Guile 2.0.10. However, it appears from the code comment that that the reason open-connection-for-uri copies the functionality of open-socket-for-uri is to avoid NSS lookups for symbolic port arguments, and it looks to me that since version 2.0.7 of Guile, its open-socket-for-uri can be convinced not to do NSS lookups as long as (uri-port uri) is not #f (see http://git.savannah.gnu.org/cgit/guile.git/tree/module/web/client.scm?id=v2.0.7#n53 ). Rather than porting the new code from Guile's open-socket-for-uri, it might make more sense to just call open-socket-for-uri with a uri that always has a port (i.e. implement the same hard-coding for http and https in the http-fetch function to make sure that uri has the default port set - I notice for some reason Guile's string->uri parser does not set the port for http and https even though it has the default ports for both set in the code. I suppose one could use the existing post-2.0.7? test to keep calling open-connection-for-uri for backwards compatibility with old versions of Guile (which in any case don't have proxy support, so for my use case it doesn't matter). I can try to put together a patch that implements this fix, although I haven't written scheme in quite a while, so someone else may be better suited for it. Cheers, Josh.
[Message part 2 (text/html, inline)]
bug-guix <at> gnu.org
:bug#20402
; Package guix
.
(Wed, 22 Apr 2015 01:17:01 GMT) Full text and rfc822 format available.Message #8 received at 20402 <at> debbugs.gnu.org (full text, mbox):
From: Joshua Randall <jcrandall <at> alum.mit.edu> To: 20402 <at> debbugs.gnu.org Subject: Re: guix cannot download via an http proxy Date: Wed, 22 Apr 2015 02:15:58 +0100
[Message part 1 (text/plain, inline)]
Tags: patch Please find a proposed patch that should fix this issue. I haven't written scheme in many years, so please excuse me if I have failed to follow convention in any way - I won't be offended at all if you want to refactor it. This patch should apply cleanly on the current git master branch and also on the v0.8.1 release. This patch modifies http-fetch (in build/download.scm) such that it calls Guile's open-socket-for-uri after fixing up the uri so that it always has a port specified. I'm not sure how to test the bootstrapping NSS issue that required open-connection-for-uri, but my expectation based on reading the source is that this alternative should work for Guile > 2.0.7, and I've left the original open-connection-for-uri in there for backwards compatibility with Guile < 2.0.7. If someone can test this against a situation known to have needed the NSS workaround, that would be great. I've also changed the only other call to open-connection-for-uri, which is in the probe-uri function in scripts/lint.scm - my suspicion is that won't be an issue because I'm guessing the lint scripts are not used while bootstrapping, so the open-socket-for-uri function will probably be fine for the purpose of probing whether a URL is valid. After applying this patch, guix should use the http proxy support built in to Guile >= 2.0.10. This appears to be working for me - I've just started a bootstrap and it has now started to successfully download packages - so far, so good. Please let me know if there is anything else I can do to assist with getting this issue fixed. Cheers, Josh.
[Message part 2 (text/html, inline)]
[guix-git-dd3a42e6-http-proxy-fix.patch (application/octet-stream, attachment)]
bug-guix <at> gnu.org
:bug#20402
; Package guix
.
(Wed, 22 Apr 2015 09:09:01 GMT) Full text and rfc822 format available.Message #11 received at 20402 <at> debbugs.gnu.org (full text, mbox):
From: Joshua Randall <jcrandall <at> alum.mit.edu> To: 20402 <at> debbugs.gnu.org Subject: Re: guix cannot download via an http proxy Date: Wed, 22 Apr 2015 10:08:27 +0100
[Message part 1 (text/plain, inline)]
After more testing this morning, I see that although the core functionality is working after applying the proposed patch (i.e. the guix daemon itself can download things), the derivations are still failing because the environment is being squashed and so the http_proxy and https_proxy environment variables are not available to the build. It looks like nix has a way around this involving setting imputeEnvVars = "http_proxy https_proxy" so that those variables get copied into the build environment (for fixed-output derivations only, which makes sense to me). I'll try to come up with a patch that inserts the imputeEnvVars, but I'm getting farther from the code I understand so I will probably need some help in getting it in the right place. Cheers, Josh. On Wed, Apr 22, 2015 at 2:15 AM, Joshua Randall <jcrandall <at> alum.mit.edu> wrote: > Tags: patch > > Please find a proposed patch that should fix this issue. I haven't written > scheme in many years, so please excuse me if I have failed to follow > convention in any way - I won't be offended at all if you want to refactor > it. This patch should apply cleanly on the current git master branch and > also on the v0.8.1 release. > > This patch modifies http-fetch (in build/download.scm) such that it calls > Guile's open-socket-for-uri after fixing up the uri so that it always has a > port specified. I'm not sure how to test the bootstrapping NSS issue that > required open-connection-for-uri, but my expectation based on reading the > source is that this alternative should work for Guile > 2.0.7, and I've > left the original open-connection-for-uri in there for backwards > compatibility with Guile < 2.0.7. If someone can test this against a > situation known to have needed the NSS workaround, that would be great. > > I've also changed the only other call to open-connection-for-uri, which is > in the probe-uri function in scripts/lint.scm - my suspicion is that won't > be an issue because I'm guessing the lint scripts are not used while > bootstrapping, so the open-socket-for-uri function will probably be fine > for the purpose of probing whether a URL is valid. > > After applying this patch, guix should use the http proxy support built in > to Guile >= 2.0.10. This appears to be working for me - I've just started a > bootstrap and it has now started to successfully download packages - so > far, so good. > > Please let me know if there is anything else I can do to assist with > getting this issue fixed. > > Cheers, > > Josh. > >
[Message part 2 (text/html, inline)]
bug-guix <at> gnu.org
:bug#20402
; Package guix
.
(Wed, 22 Apr 2015 14:23:02 GMT) Full text and rfc822 format available.Message #14 received at 20402 <at> debbugs.gnu.org (full text, mbox):
From: Joshua Randall <jcrandall <at> alum.mit.edu> To: 20402 <at> debbugs.gnu.org Subject: updated patch to pass http_proxy and https_proxy env vars to derivation builds Date: Wed, 22 Apr 2015 15:22:50 +0100
[Message part 1 (text/plain, inline)]
Tags: patch I've updated my patch to also modify guix/derivations.scm to add a setting for nix's impureEnvVars, such that it will pass http_proxy and https_proxy variables from the guix daemon to the build processes (which nix will do only for fixed-output derivations). I believe this is working, but I've now run into a third problem: the bootstrap guile is only v2.0.9, and v.2.0.10 or later is required for proxy support. What would I need to do to use a later version of bootstrap guile? Cheers, Josh.
[Message part 2 (text/html, inline)]
[guix-git-dd3a42e6-http-proxy-impure-fix.patch (application/octet-stream, attachment)]
bug-guix <at> gnu.org
:bug#20402
; Package guix
.
(Thu, 30 Apr 2015 22:09:03 GMT) Full text and rfc822 format available.Message #17 received at 20402 <at> debbugs.gnu.org (full text, mbox):
From: ludo <at> gnu.org (Ludovic Courtès) To: Joshua Randall <jcrandall <at> alum.mit.edu> Cc: 20402 <at> debbugs.gnu.org Subject: Re: bug#20402: guix cannot download via an http proxy Date: Fri, 01 May 2015 00:07:59 +0200
Hi, Sorry for the delay, and thanks for the investigation and patch! Joshua Randall <jcrandall <at> alum.mit.edu> skribis: > This patch modifies http-fetch (in build/download.scm) such that it calls > Guile's open-socket-for-uri after fixing up the uri so that it always has a > port specified. I'm not sure how to test the bootstrapping NSS issue that > required open-connection-for-uri, but my expectation based on reading the > source is that this alternative should work for Guile > 2.0.7, and I've > left the original open-connection-for-uri in there for backwards > compatibility with Guile < 2.0.7. If someone can test this against a > situation known to have needed the NSS workaround, that would be great. To name lookup with the bootstrap Guile, one way is to run this: $ guix gc -d $(guix build -S -e '(@@ (gnu packages commencement) glibc-final)') $ ./pre-inst-env guix build -S \ -e '(@@ (gnu packages commencement) glibc-final)' --no-substitutes The second command here uses the bootstrap Guile. Another approach is this: --8<---------------cut here---------------start------------->8--- scheme@(guile-user)> ,use(gnu packages bootstrap) scheme@(guile-user)> ,enter-store-monad store-monad@(guile-user) [1]> (mlet %store-monad ((guile (package->derivation %bootstrap-guile))) (gexp->derivation "foo" #~(begin (mkdir #$output) (pk (getaddrinfo "www.gnu.org" "http"))) #:hash-algo 'sha256 #:hash (base32 "0mdqa9w1p6cmli6976v4wi0sw9r4p5prkj7lzfd1877wk11c9c73") #:guile-for-build guile)) $5 = #<derivation /gnu/store/5prfiblj6ddziavk9nz31bkgy8jzaghx-foo.drv => /gnu/store/xf3404zw9kfx6a1gcfk6lmqcx6a53ad5-foo 2ae7960> store-monad@(guile-user) [1]> (built-derivations (list $5)) building path(s) `/gnu/store/xf3404zw9kfx6a1gcfk6lmqcx6a53ad5-foo' ;;; ((#(0 2 1 6 #(2 3497454484 80) #f) #(0 2 2 17 #(2 3497454484 80) #f) #(0 10 1 6 #(10 42541952298791455573290623124440612874 80 0 0) #f) #(0 10 2 17 #(10 42541952298791455573290623124440612874 80 0 0) #f))) --8<---------------cut here---------------end--------------->8--- Here ‘built-derivations’ fails but the build log shows that ‘getaddrinfo’ succeeded. Lastly, one can extract gnu/packages/bootstrap/x86_64-linux/guile-2.0.9.tar.xz and run: strace -o log ./bin/guile -c '(getaddrinfo "www.gnu.org" "http")' Here the log shows that /etc/nsswitch.conf, /etc/services, and /etc/hosts are accessed and things just work; it does not attempt to connect to the nscd. A bit of archeology shows the following timeline: 1. d14ecda introduces the ‘open-connection-for-uri’ hack (Oct. 2012). 2. d3b5972 changes libc used to make bootstrap tarballs to use static NSS modules (Jan. 2013). 3. 0621349 updates the bootstrap guile-2.0.9.tar.xz tarballs (Nov. 2013), meaning that our current bootstrap Guile indeed uses static NSS modules and doesn’t attempt to talk to nscd. In other words, the hack is no longer needed. Thus, ‘open-connection-for-uri’ is almost (see below) unneeded now, which simplifies the solution to the problem you raise. > I've also changed the only other call to open-connection-for-uri, which is > in the probe-uri function in scripts/lint.scm - my suspicion is that won't > be an issue because I'm guessing the lint scripts are not used while > bootstrapping, so the open-socket-for-uri function will probably be fine > for the purpose of probing whether a URL is valid. ‘open-connection-for-uri’ also handles TLS connections, which are also useful for ‘guix lint’, so we cannot completely get rid of it. Commit d17551d simplifies it so that it is just a wrapper around ‘open-socket-for-uri’. After that, ‘guix download’ honors $http_proxy. $https_proxy is not handled yet because that requires more work, and I do not fully understand how that is supposed to work. Patch welcome, though. :-) Thanks! Ludo’.
bug-guix <at> gnu.org
:bug#20402
; Package guix
.
(Thu, 30 Apr 2015 22:13:01 GMT) Full text and rfc822 format available.Message #20 received at 20402 <at> debbugs.gnu.org (full text, mbox):
From: ludo <at> gnu.org (Ludovic Courtès) To: Joshua Randall <jcrandall <at> alum.mit.edu> Cc: 20402 <at> debbugs.gnu.org Subject: Re: bug#20402: updated patch to pass http_proxy and https_proxy env vars to derivation builds Date: Fri, 01 May 2015 00:12:42 +0200
Joshua Randall <jcrandall <at> alum.mit.edu> skribis: > I've updated my patch to also modify guix/derivations.scm to add a setting > for nix's impureEnvVars, such that it will pass http_proxy and https_proxy > variables from the guix daemon to the build processes (which nix will do > only for fixed-output derivations). I took a slightly different approach in commit c046815 (and also not passing “impureEnvVars” by default since that would lead to a full rebuild.) With commit 0d88313, $http_proxy is honored, but note that it’s the $http_proxy value in the daemon’s environment that is used, not in the client’s environment. > I believe this is working, but I've now run into a third problem: the > bootstrap guile is only v2.0.9, and v.2.0.10 or later is required for proxy > support. What would I need to do to use a later version of bootstrap guile? We will need to update the bootstrap Guile, but that won’t happen until a later full-rebuild cycle. You should be able to use substitutes as a workaround in the meantime. How does that sound? Thank you! Ludo’.
bug-guix <at> gnu.org
:bug#20402
; Package guix
.
(Tue, 09 Jun 2015 01:50:04 GMT) Full text and rfc822 format available.Message #23 received at 20402 <at> debbugs.gnu.org (full text, mbox):
From: Claes Wallin (韋嘉誠) <gnu <at> clacke.user.lysator.liu.se> To: 20402 <at> debbugs.gnu.org Subject: Re: guix cannot download via an http proxy Date: Tue, 9 Jun 2015 03:22:38 +0200
Would support for $ftp_proxy as well require the implementation of new functionality, or would it be a case of just letting the environment variable through?
bug-guix <at> gnu.org
:bug#20402
; Package guix
.
(Wed, 10 Jun 2015 02:27:02 GMT) Full text and rfc822 format available.Message #26 received at 20402 <at> debbugs.gnu.org (full text, mbox):
From: Mark H Weaver <mhw <at> netris.org> To: Claes Wallin (韋嘉誠) <gnu <at> clacke.user.lysator.liu.se> Cc: 20402 <at> debbugs.gnu.org Subject: Re: bug#20402: guix cannot download via an http proxy Date: Tue, 09 Jun 2015 22:26:38 -0400
Claes Wallin (韋嘉誠) <gnu <at> clacke.user.lysator.liu.se> writes: > Would support for $ftp_proxy as well require the implementation of new > functionality, or would it be a case of just letting the environment > variable through? It would require new functionality. Our FTP client would have to be enhanced to support proxies. Mark
bug-guix <at> gnu.org
:bug#20402
; Package guix
.
(Sat, 13 Jun 2015 17:45:03 GMT) Full text and rfc822 format available.Message #29 received at 20402 <at> debbugs.gnu.org (full text, mbox):
From: ludo <at> gnu.org (Ludovic Courtès) To: Claes Wallin (韋嘉誠) <gnu <at> clacke.user.lysator.liu.se> Cc: 20402 <at> debbugs.gnu.org Subject: Re: HTTP and FTP proxies for guix package / guix-daemon Date: Sat, 13 Jun 2015 19:44:44 +0200
(Please always keep the bug email address Cc’d.) "Claes Wallin (韋嘉誠)" <gnu <at> clacke.user.lysator.liu.se> skribis: > I suppose if I have a freshly built guix with "guix package -i guix" > or "guix package -u guix" it should be using guile 2.0.11 then? > Actually looking at the head of .../bin/guix it's referring to a > guile-2.0.11. Yes, so that should be OK. And it’s Guix > 0.8.2, right? > But guix-daemon still doesn't seem to be honoring http_proxy. As a first test, could you check whether ‘guix download’ honors it? A second test would be: 1. Define ‘http_proxy’ in your user shell. 2. From the same shell, run ‘guix build -S gcc --no-substitutes’, say. Does that honor ‘http_proxy’? And a third test is for substitutes: Are they also downloaded via the proxy? Here you need to: 1. Define ‘http_proxy’ in a root shell. 2. Run ‘guix-daemon’ from this shell. 3. From a user shell, run ‘guix build gcc’, say, which should normally download a substitute from hydra.gnu.org rather than trying to build things. Does the substitute download honor ‘http_proxy’? Thanks in advance! Ludo’.
bug-guix <at> gnu.org
:bug#20402
; Package guix
.
(Wed, 17 Jun 2015 16:50:04 GMT) Full text and rfc822 format available.Message #32 received at 20402 <at> debbugs.gnu.org (full text, mbox):
From: Claes Wallin (韋嘉誠) <gnu <at> clacke.user.lysator.liu.se> To: Ludovic Courtès <ludo <at> gnu.org> Cc: 20402 <at> debbugs.gnu.org Subject: Re: HTTP and FTP proxies for guix package / guix-daemon Date: Wed, 17 Jun 2015 18:46:17 +0200
Finally got around to this. On Sat, Jun 13, 2015 at 7:44 PM, Ludovic Courtès <ludo <at> gnu.org> wrote: > "Claes Wallin (韋嘉誠)" <gnu <at> clacke.user.lysator.liu.se> skribis: > >> I suppose if I have a freshly built guix with "guix package -i guix" >> or "guix package -u guix" it should be using guile 2.0.11 then? >> Actually looking at the head of .../bin/guix it's referring to a >> guile-2.0.11. > > Yes, so that should be OK. And it’s Guix > 0.8.2, right? # /root/.guix-profile/bin/guix --version guix (GNU Guix) 0.8.3 # ls -l /root/.guix-profile/bin/guix lrwxrwxrwx 3 root 999 71 Jan 1 1970 /root/.guix-profile/bin/guix -> /gnu/store/bja9w08pay8rqfs83f3rzlms9q63kymw-guix-0.8.2.a43b55f/bin/guix >> But guix-daemon still doesn't seem to be honoring http_proxy. > > As a first test, could you check whether ‘guix download’ honors it? It does. > A second test would be: > > 1. Define ‘http_proxy’ in your user shell. > 2. From the same shell, run ‘guix build -S gcc --no-substitutes’, say. > > Does that honor ‘http_proxy’? http_proxy is set (and exported) in user shell and in the daemon process. No. > And a third test is for substitutes: Are they also downloaded via the > proxy? Here you need to: > > 1. Define ‘http_proxy’ in a root shell. > 2. Run ‘guix-daemon’ from this shell. > 3. From a user shell, run ‘guix build gcc’, say, which should normally > download a substitute from hydra.gnu.org rather than trying to build > things. > > Does the substitute download honor ‘http_proxy’? No. Whenever the daemon is doing the downloading, http_proxy is ignored and I get "Temporary failure in name resolution" because the VM doesn't have an internet DNS. As I said earlier, guix has guile-2.0.11 in the hashbang. But guix-daemon seems to be making use of guile-bootstrap. If I provoke it a bit I get this: # find /gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/ -name '*.scm' -exec touch {} + # guix package -i texinfo . . . ;;; note: source file /gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/share/guile/2.0/ice-9/eval.scm ;;; newer than compiled /gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/lib/guile/2.0/ccache/ice-9/eval.go ;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0 ;;; or pass the --no-auto-compile argument to disable. . . . ;;; it seems /gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/share/guile/2.0/ice-9/receive.scm ;;; is part of the compiler; skipping auto-compilation ;;; WARNING: compilation of /gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/share/guile/2.0/ice-9/command-line.scm failed: ;;; ERROR: failed to create path for auto-compiled file "/gnu/store/cyppdaliwllm1mzgka3cxwxnddk7j2cl-guile-bootstrap-2.0/share/guile/2.0/ice-9/command-line.scm" starting download of `/gnu/store/2zkyyw4bq0wzsxdysncrf9lmwl44w5wh-binutils-2.25.tar.bz2' from `http://ftpmirror.gnu.org/binutils/binutils-2.25.tar.bz2'... ERROR: In procedure getaddrinfo: Temporary failure in name resolution ... so it's pretty clear that *something* in there is still referencing the bootstrap guile. But maybe that's expected? Are the guile packages built using only the bootstrap and not themselves? /gnu/store/0aq1kq37b6kr562330xz61rg08l6y5yg-guix-0.8.2.a43b55f.drv has: Derive([("out","/gnu/store/bja9w08pay8rqfs83f3rzlms9q63kymw-guix-0.8.2.a43b55f","","")] . . . ("/gnu/store/4lkrvcnh2y923la3ivsbyv2m2nv3xpjy-guile-2.0.9.tar.xz.drv",["out"]) . . . ("/gnu/store/d3x2skfxhvqj21w39f6bckmxni1q3bbp-guile-2.0.11.tar.xz.drv",["out"]) . . . ("/gnu/store/hvl3gwa57p7k1qnliyy6ymbmz2509azh-guile-2.0.11.drv",["out"]) . . . ("/gnu/store/p4i1inysxx5ra0hvhcrd9liw3a5skcbx-guile-2.0.9.tar.xz.drv",["out"]) . . . ("/gnu/store/wlivhx78f1a05dx0blqy209cb1nqwgf5-guile-2.0.9.tar.xz.drv",["out"]) . . . ["/gnu/store/iylwyjzxcy3hkz1wlgpyk4zmhci97rgk-guix-0.8.2.a43b55f-guile-builder"],"x86_64-linux","/gnu/store/cnqmkmj40jmssnx6fkf9n0n3bqj5x426-guile-2.0.11/bin/guile",["--no-auto-compile","-L","/gnu/store/x2nfxrqhdsvir8l65x19bhnrwd4h41rs-module-import","/gnu/store/iylwyjzxcy3hkz1wlgpyk4zmhci97rgk-guix-0.8.2.a43b55f-guile-builder"],[("GUILE_LOAD_COMPILED_PATH","/gnu/store/07n184fk33b7gqkyrbn6b2ckdfisp09i-module-import-compiled"),("out","/gnu/store/bja9w08pay8rqfs83f3rzlms9q63kymw-guix-0.8.2.a43b55f")]) Why so many different guile drivers? Is that expected? > Thanks in advance! No problem!
bug-guix <at> gnu.org
:bug#20402
; Package guix
.
(Wed, 17 Jun 2015 16:52:02 GMT) Full text and rfc822 format available.Message #35 received at 20402 <at> debbugs.gnu.org (full text, mbox):
From: Claes Wallin (韋嘉誠) <gnu <at> clacke.user.lysator.liu.se> To: Ludovic Courtès <ludo <at> gnu.org> Cc: 20402 <at> debbugs.gnu.org Subject: Re: HTTP and FTP proxies for guix package / guix-daemon Date: Wed, 17 Jun 2015 18:51:06 +0200
On Wed, Jun 17, 2015 at 6:46 PM, Claes Wallin (韋嘉誠) <gnu <at> clacke.user.lysator.liu.se> wrote: > Why so many different guile drivers? Is that expected? ... derivations. Sorry.
bug-guix <at> gnu.org
:bug#20402
; Package guix
.
(Wed, 01 Jul 2015 08:20:04 GMT) Full text and rfc822 format available.Message #38 received at 20402 <at> debbugs.gnu.org (full text, mbox):
From: ludo <at> gnu.org (Ludovic Courtès) To: Claes Wallin (韋嘉誠) <gnu <at> clacke.user.lysator.liu.se> Cc: 20402 <at> debbugs.gnu.org Subject: Re: HTTP and FTP proxies for guix package / guix-daemon Date: Wed, 01 Jul 2015 10:19:14 +0200
"Claes Wallin (韋嘉誠)" <gnu <at> clacke.user.lysator.liu.se> skribis: >>> But guix-daemon still doesn't seem to be honoring http_proxy. >> >> As a first test, could you check whether ‘guix download’ honors it? > > It does. > >> A second test would be: >> >> 1. Define ‘http_proxy’ in your user shell. >> 2. From the same shell, run ‘guix build -S gcc --no-substitutes’, say. >> >> Does that honor ‘http_proxy’? > > http_proxy is set (and exported) in user shell and in the daemon process. Oops, my instructions were incorrect: ‘http_proxy’ needs to be defined in the daemon process. To test, I first run the daemon like this: sudo sh -c 'export http_proxy=foobar ; ./pre-inst-env guix-daemon --build-users-group=guixbuild' and then I run: --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix build -S gcc --no-substitutes accepted connection from pid 14399, user ludo La jenaj derivoj estos konstruataj: /gnu/store/s32s498y88sidfnrppsn33nfmhlacpzx-gcc-5.1.0.tar.xz.drv /gnu/store/z1n2c3hgl7w2cnz9d5szyjwwy8lp0dav-gcc-5.1.0.tar.bz2.drv @ build-started /gnu/store/z1n2c3hgl7w2cnz9d5szyjwwy8lp0dav-gcc-5.1.0.tar.bz2.drv - x86_64-linux /var/log/guix/drvs/z1//n2c3hgl7w2cnz9d5szyjwwy8lp0dav-gcc-5.1.0.tar.bz2.drv.bz2 starting download of `/gnu/store/6a3d39111w0d5n7rzb5p7aliilw0hx1l-gcc-5.1.0.tar.bz2' from `http://ftpmirror.gnu.org/gcc/gcc-5.1.0/gcc-5.1.0.tar.bz2'... ERROR: In procedure struct_vtable: Wrong type argument in position 1 (expecting struct): #f starting download of `/gnu/store/6a3d39111w0d5n7rzb5p7aliilw0hx1l-gcc-5.1.0.tar.bz2' from `ftp://ftp.cs.tu-berlin.de/pub/gnu/gcc/gcc-5.1.0/gcc-5.1.0.tar.bz2'... ftp://ftp.cs.tu-berlin.de/.../gcc-5.1.0.tar.bz2 3.9% of 92728.9 KiB (610. KiB/s) C-c C-cSIGPOLL --8<---------------cut here---------------end--------------->8--- Here the error message comes from the fact that $http_proxy is a malformed URL, showing that it *is* honored. Since the http URL fails, the download logic falls back to the next URL, which happens to be an ftp URL, which is unaffected by ‘http_proxy’. So it’s working as expected here. >> And a third test is for substitutes: Are they also downloaded via the >> proxy? Here you need to: >> >> 1. Define ‘http_proxy’ in a root shell. >> 2. Run ‘guix-daemon’ from this shell. >> 3. From a user shell, run ‘guix build gcc’, say, which should normally >> download a substitute from hydra.gnu.org rather than trying to build >> things. >> >> Does the substitute download honor ‘http_proxy’? This test also works as expected here: --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix build -S inkscape accepted connection from pid 15118, user ludo La jena derivo estos konstruata: /gnu/store/fa6ss1ljwb97ns75332nshskj2bv0dh4-inkscape-0.91.tar.gz.drv La jenaj derivoj estos elŝutataj: /gnu/store/acmzy3nkms38g6vrphh07bczf5s5mbjd-gnutls-3.4.1 /gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1 @ substituter-started /gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1 /home/ludo/src/guix/nix/scripts/substitute found valid signature for '/gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1', from 'http://hydra.gnu.org/nar/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1' downloading `/gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1' (1.7 MiB installed)... [...] In guix/scripts/substitute.scm: 182: 2 [#<procedure 1523210 at guix/scripts/substitute.scm:165:9 ()>] In guix/build/download.scm: 212: 1 [open-socket-for-uri #] In web/client.scm: 73: 0 [open-socket-for-uri #] web/client.scm:73:16: In procedure open-socket-for-uri: web/client.scm:73:16: In procedure struct_vtable: Wrong type argument in position 1 (expecting struct): #f @ substituter-failed /gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1 256 fetching path `/gnu/store/vx82jvj1xbppnphl8nwjdr7cm7f5bk59-nettle-3.1.1' failed with exit code 1 guix build: error: build failed: some substitutes for the outputs of derivation `/gnu/store/2v6lls3f8dj1rkpram4jnl8ckqs77r6m-gnutls-3.4.1.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source --8<---------------cut here---------------end--------------->8--- The backtrace again stems from the bogus $http_proxy that I chose (not nice, but shows that $http_proxy is honored.) Please let me know if you have evidence that something works differently for you. > No. Whenever the daemon is doing the downloading, http_proxy is > ignored and I get "Temporary failure in name resolution" because the > VM doesn't have an internet DNS. [...] > `/gnu/store/2zkyyw4bq0wzsxdysncrf9lmwl44w5wh-binutils-2.25.tar.bz2' > from `http://ftpmirror.gnu.org/binutils/binutils-2.25.tar.bz2'... > ERROR: In procedure getaddrinfo: Temporary failure in name resolution That’s another, unrelated issue, probably in your network or on your machine. Thanks, Ludo’.
bug-guix <at> gnu.org
:bug#20402
; Package guix
.
(Wed, 01 Jul 2015 13:09:02 GMT) Full text and rfc822 format available.Message #41 received at 20402 <at> debbugs.gnu.org (full text, mbox):
From: Claes Wallin (韋嘉誠) <gnu <at> clacke.user.lysator.liu.se> To: Ludovic Courtès <ludo <at> gnu.org> Cc: 20402 <at> debbugs.gnu.org Subject: Re: HTTP and FTP proxies for guix package / guix-daemon Date: Wed, 1 Jul 2015 15:08:31 +0200
On Wed, Jul 1, 2015 at 10:19 AM, Ludovic Courtès <ludo <at> gnu.org> wrote: > To test, I first run the daemon like this: > > sudo sh -c 'export http_proxy=foobar ; ./pre-inst-env guix-daemon --build-users-group=guixbuild' Could there be a difference because you are testing with the checked-out, semi-manually built guix and I was checking with a freshly packaged (pull, package -i guix) guix? Like I said in http://lists.gnu.org/archive/html/bug-guix/2015-06/msg00036.html , guix-daemon as compiled by guix seems to be using guile-2.0.9. Maybe that only happens due to guix rewriting hashbangs etc, and on a build outside guix (even when using guix environment) it uses whatever guile happens to be in the paths? When I have the time again I will check it exactly the way you are doing it. > Here the error message comes from the fact that $http_proxy is a > malformed URL, showing that it *is* honored. Since the http URL fails, > the download logic falls back to the next URL, which happens to be an > ftp URL, which is unaffected by ‘http_proxy’. Ok, that's maybe a clearer way of testing it. > Please let me know if you have evidence that something works differently > for you. I would say that both of us have evidence that it's working and not working. The conclusion must be that we are not testing the exact same program. >> No. Whenever the daemon is doing the downloading, http_proxy is >> ignored and I get "Temporary failure in name resolution" because the >> VM doesn't have an internet DNS. > > [...] > >> `/gnu/store/2zkyyw4bq0wzsxdysncrf9lmwl44w5wh-binutils-2.25.tar.bz2' >> from `http://ftpmirror.gnu.org/binutils/binutils-2.25.tar.bz2'... >> ERROR: In procedure getaddrinfo: Temporary failure in name resolution > > That’s another, unrelated issue, probably in your network or on your > machine. Yes. This is why the proxy is required. There is no internet DNS for the VM, because it is on an isolated lab network. When it does go through the proxy, like when running guix download, no name lookup is done and everything works.
bug-guix <at> gnu.org
:bug#20402
; Package guix
.
(Wed, 01 Jul 2015 14:46:02 GMT) Full text and rfc822 format available.Message #44 received at 20402 <at> debbugs.gnu.org (full text, mbox):
From: ludo <at> gnu.org (Ludovic Courtès) To: Claes Wallin (韋嘉誠) <gnu <at> clacke.user.lysator.liu.se> Cc: 20402 <at> debbugs.gnu.org Subject: Re: HTTP and FTP proxies for guix package / guix-daemon Date: Wed, 01 Jul 2015 16:45:15 +0200
"Claes Wallin (韋嘉誠)" <gnu <at> clacke.user.lysator.liu.se> skribis: > On Wed, Jul 1, 2015 at 10:19 AM, Ludovic Courtès <ludo <at> gnu.org> wrote: > >> To test, I first run the daemon like this: >> >> sudo sh -c 'export http_proxy=foobar ; ./pre-inst-env guix-daemon --build-users-group=guixbuild' > > Could there be a difference because you are testing with the > checked-out, semi-manually built guix and I was checking with a > freshly packaged (pull, package -i guix) guix? Like I said in > http://lists.gnu.org/archive/html/bug-guix/2015-06/msg00036.html , > guix-daemon as compiled by guix seems to be using guile-2.0.9. Maybe > that only happens due to guix rewriting hashbangs etc, and on a build > outside guix (even when using guix environment) it uses whatever guile > happens to be in the paths? Could you send the store file name of Guix that is being used? For instance, I have: --8<---------------cut here---------------start------------->8--- $ sudo deco status guix-daemon Pasvorto: Status of guix-daemon: It is started. Running value is 31316. It is enabled. Provides (guix-daemon). Requires (user-processes). Conflicts with (). Will be respawned. $ ps 31316 PID TTY STAT TIME COMMAND 31316 ? Ss 0:00 /gnu/store/zg6j61inmmk5kyqgyfjnj7rdxl79i9ps-guix-0.8.2.684bf7c/bin/guix-daemon --build-users-group guixbu --8<---------------cut here---------------end--------------->8--- That means I’m using commit 684bf7c of Guix, which has the fix, which came up after the http_proxy commit (commit 0d88313.) >>> No. Whenever the daemon is doing the downloading, http_proxy is >>> ignored and I get "Temporary failure in name resolution" because the >>> VM doesn't have an internet DNS. >> >> [...] >> >>> `/gnu/store/2zkyyw4bq0wzsxdysncrf9lmwl44w5wh-binutils-2.25.tar.bz2' >>> from `http://ftpmirror.gnu.org/binutils/binutils-2.25.tar.bz2'... >>> ERROR: In procedure getaddrinfo: Temporary failure in name resolution >> >> That’s another, unrelated issue, probably in your network or on your >> machine. > > Yes. This is why the proxy is required. There is no internet DNS for > the VM, because it is on an isolated lab network. Hmm, when using Guile 2.0.11 or later, ‘open-socket-for-uri’ connects to the proxy (so its name must be available in DNS lookup), not to the target server. The code in (web client) goes like this: --8<---------------cut here---------------start------------->8--- (define (open-socket-for-uri uri-or-string) "Return an open input/output port for a connection to URI." (define http-proxy (current-http-proxy)) (define uri (ensure-uri (or http-proxy uri-or-string))) ;← proxy (define addresses (let ((port (uri-port uri))) (delete-duplicates (getaddrinfo (uri-host uri) (cond (port => number->string) (else (symbol->string (uri-scheme uri)))) (if port AI_NUMERICSERV 0)) ;; ... --8<---------------cut here---------------end--------------->8--- Do you see the same DNS issue when running: http_proxy=http://... guix download http://... ? Thanks, Ludo’.
bug-guix <at> gnu.org
:bug#20402
; Package guix
.
(Wed, 01 Jul 2015 14:54:02 GMT) Full text and rfc822 format available.Message #47 received at 20402 <at> debbugs.gnu.org (full text, mbox):
From: Claes Wallin (韋嘉誠) <gnu <at> clacke.user.lysator.liu.se> To: Ludovic Courtès <ludo <at> gnu.org> Cc: 20402 <at> debbugs.gnu.org Subject: Re: HTTP and FTP proxies for guix package / guix-daemon Date: Wed, 1 Jul 2015 16:53:15 +0200
[Message part 1 (text/plain, inline)]
Ludovic, sorry for duplicate mail again. On Jul 1, 2015 4:45 PM, "Ludovic Courtès" <ludo <at> gnu.org> wrote: > > Yes. This is why the proxy is required. There is no internet DNS for > > the VM, because it is on an isolated lab network. > > Hmm, when using Guile 2.0.11 or later, ‘open-socket-for-uri’ connects to > the proxy (so its name must be available in DNS lookup), not to the > target server. The code in (web client) goes like this: The proxy is in the DNS. > Do you see the same DNS issue when running: > > http_proxy=http://... guix download http://... Nope. guix download works. It successfully looks up and connects to the proxy. When I next have the chance I will report: 1) Results with guix-compiled guix-daemon 2) Results with dev-compiled guix-daemon 3) /gnu/store hash of 1) On Jul 1, 2015 4:45 PM, "Ludovic Courtès" <ludo <at> gnu.org> wrote: > "Claes Wallin (韋嘉誠)" <gnu <at> clacke.user.lysator.liu.se> skribis: > > > On Wed, Jul 1, 2015 at 10:19 AM, Ludovic Courtès <ludo <at> gnu.org> wrote: > > > >> To test, I first run the daemon like this: > >> > >> sudo sh -c 'export http_proxy=foobar ; ./pre-inst-env guix-daemon > --build-users-group=guixbuild' > > > > Could there be a difference because you are testing with the > > checked-out, semi-manually built guix and I was checking with a > > freshly packaged (pull, package -i guix) guix? Like I said in > > http://lists.gnu.org/archive/html/bug-guix/2015-06/msg00036.html , > > guix-daemon as compiled by guix seems to be using guile-2.0.9. Maybe > > that only happens due to guix rewriting hashbangs etc, and on a build > > outside guix (even when using guix environment) it uses whatever guile > > happens to be in the paths? > > Could you send the store file name of Guix that is being used? > For instance, I have: > > --8<---------------cut here---------------start------------->8--- > $ sudo deco status guix-daemon > Pasvorto: > Status of guix-daemon: > It is started. > Running value is 31316. > It is enabled. > Provides (guix-daemon). > Requires (user-processes). > Conflicts with (). > Will be respawned. > $ ps 31316 > PID TTY STAT TIME COMMAND > 31316 ? Ss 0:00 > /gnu/store/zg6j61inmmk5kyqgyfjnj7rdxl79i9ps-guix-0.8.2.684bf7c/bin/guix-daemon > --build-users-group guixbu > --8<---------------cut here---------------end--------------->8--- > > That means I’m using commit 684bf7c of Guix, which has the fix, which > came up after the http_proxy commit (commit 0d88313.) > > >>> No. Whenever the daemon is doing the downloading, http_proxy is > >>> ignored and I get "Temporary failure in name resolution" because the > >>> VM doesn't have an internet DNS. > >> > >> [...] > >> > >>> `/gnu/store/2zkyyw4bq0wzsxdysncrf9lmwl44w5wh-binutils-2.25.tar.bz2' > >>> from `http://ftpmirror.gnu.org/binutils/binutils-2.25.tar.bz2'... > >>> ERROR: In procedure getaddrinfo: Temporary failure in name resolution > >> > >> That’s another, unrelated issue, probably in your network or on your > >> machine. > > > > Yes. This is why the proxy is required. There is no internet DNS for > > the VM, because it is on an isolated lab network. > > Hmm, when using Guile 2.0.11 or later, ‘open-socket-for-uri’ connects to > the proxy (so its name must be available in DNS lookup), not to the > target server. The code in (web client) goes like this: > > --8<---------------cut here---------------start------------->8--- > (define (open-socket-for-uri uri-or-string) > "Return an open input/output port for a connection to URI." > (define http-proxy (current-http-proxy)) > (define uri (ensure-uri (or http-proxy uri-or-string))) ;← proxy > (define addresses > (let ((port (uri-port uri))) > (delete-duplicates > (getaddrinfo (uri-host uri) > (cond (port => number->string) > (else (symbol->string (uri-scheme uri)))) > (if port > AI_NUMERICSERV > 0)) > ;; ... > --8<---------------cut here---------------end--------------->8--- > > Do you see the same DNS issue when running: > > http_proxy=http://... guix download http://... > > ? > > Thanks, > Ludo’. >
[Message part 2 (text/html, inline)]
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.