GNU bug report logs - #22990
Grafts leads to inefficient substitute info retrieval

Previous Next

Package: guix;

Reported by: ludo <at> gnu.org (Ludovic Courtès)

Date: Fri, 11 Mar 2016 16:54:01 UTC

Severity: important

Merged with 25137

Done: Ludovic Courtès <ludo <at> gnu.org>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 22990 in the body.
You can then email your comments to 22990 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-guix <at> gnu.org:
bug#22990; Package guix. (Fri, 11 Mar 2016 16:54:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to ludo <at> gnu.org (Ludovic Courtès):
New bug report received and forwarded. Copy sent to bug-guix <at> gnu.org. (Fri, 11 Mar 2016 16:54:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: bug-guix <at> gnu.org
Subject: Grafts leads to inefficient substitute info retrieval
Date: Fri, 11 Mar 2016 17:52:59 +0100
As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:

  1. Fetching substitute info about the things being built so that it
     can determine its references, which in turns allows it to determine
     whether they need to be grafted.

  2. Building stuff, as a last resort, so that it can determine its
     references.

Case #1 is hopefully going to be the most common.

The problem with #1 is that when building a profile, we do one
‘package-derivation’ call for each package in the profile, which
translates in one ‘graft-derivation’ call for each relevant package¹,
which translates into one ‘references/substitutes’ call for each.

Concretely, what this means is this:

  $ guix package -u
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
  […]
  The following files would be downloaded:

Each of the initial “updating list” message corresponds to an HTTP
request for a single narinfo file, which can take around 1 second.

Instead, the ideal thing would be to fetch the narinfo files for all the
relevant packages at once; that way, we’d spawn ‘guix substitute’ only
once, and it would benefit from HTTP pipelining (one round-trip instead
of N.)

To achieve this, I’m thinking of extending gexp code such that gexp
compilers can return a list of applicable grafts.  The ‘package’
compiler would do #:graft? #f and instead let ‘gexp->derivation’ call
‘graft-derivation’.

I’ll give it a try and report back.

Ludo’.

¹ A package is “relevant” if ‘package-grafts’ returns a non-empty list.




Information forwarded to bug-guix <at> gnu.org:
bug#22990; Package guix. (Sat, 12 Mar 2016 09:24:01 GMT) Full text and rfc822 format available.

Message #8 received at 22990 <at> debbugs.gnu.org (full text, mbox):

From: Alex Kost <alezost <at> gmail.com>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 22990 <at> debbugs.gnu.org
Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval
Date: Sat, 12 Mar 2016 12:23:35 +0300
Ludovic Courtès (2016-03-11 19:52 +0300) wrote:

> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
>
>   1. Fetching substitute info about the things being built so that it
>      can determine its references, which in turns allows it to determine
>      whether they need to be grafted.
>
>   2. Building stuff, as a last resort, so that it can determine its
>      references.

I noticed that #1 is happening even with --no-substitutes option.  Is it
intended?

I've tried this:

$ guix build --dry-run --no-substitutes mutt
substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
substitute: updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
The following derivations would be built:
   /gnu/store/a2w22xlmfwkgwx4vw11dxc6zrdmww435-mutt-1.5.24.drv
   ...

-- 
Alex




Information forwarded to bug-guix <at> gnu.org:
bug#22990; Package guix. (Sun, 13 Mar 2016 12:12:02 GMT) Full text and rfc822 format available.

Message #11 received at 22990 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Alex Kost <alezost <at> gmail.com>
Cc: 22990 <at> debbugs.gnu.org
Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval
Date: Sun, 13 Mar 2016 13:11:36 +0100
Alex Kost <alezost <at> gmail.com> skribis:

> Ludovic Courtès (2016-03-11 19:52 +0300) wrote:
>
>> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
>>
>>   1. Fetching substitute info about the things being built so that it
>>      can determine its references, which in turns allows it to determine
>>      whether they need to be grafted.
>>
>>   2. Building stuff, as a last resort, so that it can determine its
>>      references.
>
> I noticed that #1 is happening even with --no-substitutes option.  Is it
> intended?

Not really, but I see this is because ‘substitutable-path-info’ (called
from ‘references/substitutes’, called from ‘graft-derivation’) works
regardless of whether substitutes are enabled:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(guix)
scheme@(guile-user)> (define s (open-connection))
scheme@(guile-user)> (set-build-options s #:use-substitutes? #f)
$2 = #t
scheme@(guile-user)> (valid-path? s "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24")
$3 = #f
scheme@(guile-user)> (substitutable-path-info s '("/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24"))
$4 = (#<<substitutable> path: "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" deriver: "/gnu/store/jcl9c3w463xa2g963q5a60rrd97y1g28-mutt-1.5.24.drv" refs: ("/gnu/store/3gmzl5jpk700hqyr8p3kfg0vgcnw8d97-libassuan-2.4.2" "/gnu/store/b02lmk67jq1vcflk2m2bwzc8gmwmndqp-ncurses-6.0" "/gnu/store/d3xdc2w87yw3raafwb9q34gxx4xqci8k-cyrus-sasl-2.1.26" "/gnu/store/pkasxagsa4z4viscfpl6sjszmdmwncl1-gcc-4.9.3-lib" "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" "/gnu/store/qvx4q6lbwi4s3cwr8wqaa7kcva0a5c4b-openssl-1.0.2f" "/gnu/store/sb40mddkia0brc814xkbnhxccfm32q3a-gpgme-1.6.0" "/gnu/store/sgzfawy95pfn7nsw3xvmca58llm5zzbc-glibc-2.22" "/gnu/store/x2p2biyybcb2wac77qz9468asc5fm48i-perl-5.22.1" "/gnu/store/x8dmdlrn5qn0wrbcnngj55y3ab73h0pp-bash-4.3.42" "/gnu/store/zpxg45dq67psrn4wmfk4l635h0si8q63-libgpg-error-1.21") dl-size: 0 nar-size: 6661016>)
--8<---------------cut here---------------end--------------->8---

However, substitutes are not downloaded, so in this regard
--no-substitutes is honored.

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#22990; Package guix. (Tue, 15 Mar 2016 08:25:01 GMT) Full text and rfc822 format available.

Message #14 received at 22990 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: 22990 <at> debbugs.gnu.org
Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval
Date: Tue, 15 Mar 2016 09:24:42 +0100
ludo <at> gnu.org (Ludovic Courtès) skribis:

> Concretely, what this means is this:
>
>   $ guix package -u
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   […]
>   The following files would be downloaded:

This is somewhat mitigated by commits
f09aea1b58b3ef961d3cc712f116fe4617bc8f90 and
264fdedb408ba3620d1e361de6c77e7925025301.

In addition, 026ca50fa4c46a8e280cd51621bbec76b12c0757 slightly improves
‘guix substitute’ performance.

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#22990; Package guix. (Tue, 15 Mar 2016 18:51:01 GMT) Full text and rfc822 format available.

Message #17 received at 22990 <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: Alex Kost <alezost <at> gmail.com>, 22990 <at> debbugs.gnu.org
Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval
Date: Tue, 15 Mar 2016 14:49:55 -0400
ludo <at> gnu.org (Ludovic Courtès) writes:

> Alex Kost <alezost <at> gmail.com> skribis:
>
>> Ludovic Courtès (2016-03-11 19:52 +0300) wrote:
>>
>>> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
>>>
>>>   1. Fetching substitute info about the things being built so that it
>>>      can determine its references, which in turns allows it to determine
>>>      whether they need to be grafted.
>>>
>>>   2. Building stuff, as a last resort, so that it can determine its
>>>      references.
>>
>> I noticed that #1 is happening even with --no-substitutes option.  Is it
>> intended?
>
> Not really, but I see this is because ‘substitutable-path-info’ (called
> from ‘references/substitutes’, called from ‘graft-derivation’) works
> regardless of whether substitutes are enabled:
>
> scheme@(guile-user)> ,use(guix)
> scheme@(guile-user)> (define s (open-connection))
> scheme@(guile-user)> (set-build-options s #:use-substitutes? #f)
> $2 = #t
> scheme@(guile-user)> (valid-path? s "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24")
> $3 = #f
> scheme@(guile-user)> (substitutable-path-info s '("/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24"))
> $4 = (#<<substitutable> path: "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" deriver: "/gnu/store/jcl9c3w463xa2g963q5a60rrd97y1g28-mutt-1.5.24.drv" refs: ("/gnu/store/3gmzl5jpk700hqyr8p3kfg0vgcnw8d97-libassuan-2.4.2" "/gnu/store/b02lmk67jq1vcflk2m2bwzc8gmwmndqp-ncurses-6.0" "/gnu/store/d3xdc2w87yw3raafwb9q34gxx4xqci8k-cyrus-sasl-2.1.26" "/gnu/store/pkasxagsa4z4viscfpl6sjszmdmwncl1-gcc-4.9.3-lib" "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" "/gnu/store/qvx4q6lbwi4s3cwr8wqaa7kcva0a5c4b-openssl-1.0.2f" "/gnu/store/sb40mddkia0brc814xkbnhxccfm32q3a-gpgme-1.6.0" "/gnu/store/sgzfawy95pfn7nsw3xvmca58llm5zzbc-glibc-2.22" "/gnu/store/x2p2biyybcb2wac77qz9468asc5fm48i-perl-5.22.1" "/gnu/store/x8dmdlrn5qn0wrbcnngj55y3ab73h0pp-bash-4.3.42" "/gnu/store/zpxg45dq67psrn4wmfk4l635h0si8q63-libgpg-error-1.21") dl-size: 0 nar-size: 6661016>)

Is the information from the substitute server authenticated by checking
hydra's signature against the list of keys in /etc/guix/acls?

The reason I ask is that if the set of runtime dependencies received is
incomplete, it could lead to incorrect grafting, namely that references
to compromised libraries could be retained.

> However, substitutes are not downloaded, so in this regard
> --no-substitutes is honored.

It depends on the intent of --no-substitutes.  If the intent is to avoid
trusting the substitute server, then by relying on the accuracy of the
runtime dependency data from Hydra, we are failing to honor that intent.

That said, I think it's okay to document that --no-substitutes alone is
not sufficient to avoid trusting a substitute server, and that the
proper way to accomplish that is to make sure its key is not in
/etc/guix/acls.

What do you think?

    Thanks,
      Mark




Information forwarded to bug-guix <at> gnu.org:
bug#22990; Package guix. (Tue, 15 Mar 2016 21:51:01 GMT) Full text and rfc822 format available.

Message #20 received at 22990 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Mark H Weaver <mhw <at> netris.org>
Cc: Alex Kost <alezost <at> gmail.com>, 22990 <at> debbugs.gnu.org
Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval
Date: Tue, 15 Mar 2016 22:50:05 +0100
Mark H Weaver <mhw <at> netris.org> skribis:

> ludo <at> gnu.org (Ludovic Courtès) writes:
>
>> Alex Kost <alezost <at> gmail.com> skribis:
>>
>>> Ludovic Courtès (2016-03-11 19:52 +0300) wrote:
>>>
>>>> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
>>>>
>>>>   1. Fetching substitute info about the things being built so that it
>>>>      can determine its references, which in turns allows it to determine
>>>>      whether they need to be grafted.
>>>>
>>>>   2. Building stuff, as a last resort, so that it can determine its
>>>>      references.
>>>
>>> I noticed that #1 is happening even with --no-substitutes option.  Is it
>>> intended?
>>
>> Not really, but I see this is because ‘substitutable-path-info’ (called
>> from ‘references/substitutes’, called from ‘graft-derivation’) works
>> regardless of whether substitutes are enabled:
>>
>> scheme@(guile-user)> ,use(guix)
>> scheme@(guile-user)> (define s (open-connection))
>> scheme@(guile-user)> (set-build-options s #:use-substitutes? #f)
>> $2 = #t
>> scheme@(guile-user)> (valid-path? s "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24")
>> $3 = #f
>> scheme@(guile-user)> (substitutable-path-info s '("/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24"))
>> $4 = (#<<substitutable> path: "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" deriver: "/gnu/store/jcl9c3w463xa2g963q5a60rrd97y1g28-mutt-1.5.24.drv" refs: ("/gnu/store/3gmzl5jpk700hqyr8p3kfg0vgcnw8d97-libassuan-2.4.2" "/gnu/store/b02lmk67jq1vcflk2m2bwzc8gmwmndqp-ncurses-6.0" "/gnu/store/d3xdc2w87yw3raafwb9q34gxx4xqci8k-cyrus-sasl-2.1.26" "/gnu/store/pkasxagsa4z4viscfpl6sjszmdmwncl1-gcc-4.9.3-lib" "/gnu/store/qf2lm7jpiiyygxz8zq0r1ca1fazv6smn-mutt-1.5.24" "/gnu/store/qvx4q6lbwi4s3cwr8wqaa7kcva0a5c4b-openssl-1.0.2f" "/gnu/store/sb40mddkia0brc814xkbnhxccfm32q3a-gpgme-1.6.0" "/gnu/store/sgzfawy95pfn7nsw3xvmca58llm5zzbc-glibc-2.22" "/gnu/store/x2p2biyybcb2wac77qz9468asc5fm48i-perl-5.22.1" "/gnu/store/x8dmdlrn5qn0wrbcnngj55y3ab73h0pp-bash-4.3.42" "/gnu/store/zpxg45dq67psrn4wmfk4l635h0si8q63-libgpg-error-1.21") dl-size: 0 nar-size: 6661016>)
>
> Is the information from the substitute server authenticated by checking
> hydra's signature against the list of keys in /etc/guix/acls?

Yes.

>> However, substitutes are not downloaded, so in this regard
>> --no-substitutes is honored.
>
> It depends on the intent of --no-substitutes.  If the intent is to avoid
> trusting the substitute server, then by relying on the accuracy of the
> runtime dependency data from Hydra, we are failing to honor that intent.
>
> That said, I think it's okay to document that --no-substitutes alone is
> not sufficient to avoid trusting a substitute server, and that the
> proper way to accomplish that is to make sure its key is not in
> /etc/guix/acls.

The sysadmin gets to choose which principals are trusted; unprivileged
users can only shrink this set.

The weird thing is that in this case, passing --substitute-urls='' on
the client side would effectively disable substitutes entirely.

> What do you think?

We could augment the doc for --no-substitutes, I guess we should first
document that grafting relies on server-provided info.

Ludo’.




Merged 22990 25137. Request was from ludo <at> gnu.org (Ludovic Courtès) to control <at> debbugs.gnu.org. (Fri, 09 Dec 2016 23:26:03 GMT) Full text and rfc822 format available.

Information forwarded to bug-guix <at> gnu.org:
bug#22990; Package guix. (Sun, 08 Jan 2017 21:45:01 GMT) Full text and rfc822 format available.

Message #25 received at 22990 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: 22990 <at> debbugs.gnu.org
Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval
Date: Sun, 08 Jan 2017 22:44:41 +0100
ludo <at> gnu.org (Ludovic Courtès) skribis:

> Concretely, what this means is this:
>
>   $ guix package -u
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   […]
>   The following files would be downloaded:
>
> Each of the initial “updating list” message corresponds to an HTTP
> request for a single narinfo file, which can take around 1 second.
>
> Instead, the ideal thing would be to fetch the narinfo files for all the
> relevant packages at once; that way, we’d spawn ‘guix substitute’ only
> once, and it would benefit from HTTP pipelining (one round-trip instead
> of N.)
>
> To achieve this, I’m thinking of extending gexp code such that gexp
> compilers can return a list of applicable grafts.  The ‘package’
> compiler would do #:graft? #f and instead let ‘gexp->derivation’ call
> ‘graft-derivation’.

The ‘wip-gexp-grafts’ branch does that.  Namely, it’s possible to know
what grafts would apply to a gexp derivation build, so that one can
first build ungrafted, and then apply the grafts to the results.
So for a profile, we’d first build the profile as is, and only then
would we graft it.

Right now the tip of this branch is a hack such that ‘guix package’:

  1. Builds the original (ungrafted) derivation of the profile;

  2. Manually calls ‘graft-derivation’ on that, passing it the list of
     applicable grafts.

Conceptually it’s what we want to do, but the drawback is that the
caller (here ‘guix package’) goes through a lot of hops to get the list
of grafts and to apply it.

I think we should instead have a way to annotate a derivation with a
list of grafts, as well as a procedure to build that derivations in two
phases (first the original derivation, then the grafts).

To be continued…

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#22990; Package guix. (Mon, 09 Jan 2017 22:56:02 GMT) Full text and rfc822 format available.

Message #28 received at 22990 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: 22990 <at> debbugs.gnu.org
Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval
Date: Mon, 09 Jan 2017 23:55:38 +0100
ludo <at> gnu.org (Ludovic Courtès) skribis:

> ludo <at> gnu.org (Ludovic Courtès) skribis:

[...]

>> To achieve this, I’m thinking of extending gexp code such that gexp
>> compilers can return a list of applicable grafts.  The ‘package’
>> compiler would do #:graft? #f and instead let ‘gexp->derivation’ call
>> ‘graft-derivation’.
>
> The ‘wip-gexp-grafts’ branch does that.  Namely, it’s possible to know
> what grafts would apply to a gexp derivation build, so that one can
> first build ungrafted, and then apply the grafts to the results.
> So for a profile, we’d first build the profile as is, and only then
> would we graft it.
>
> Right now the tip of this branch is a hack such that ‘guix package’:
>
>   1. Builds the original (ungrafted) derivation of the profile;
>
>   2. Manually calls ‘graft-derivation’ on that, passing it the list of
>      applicable grafts.
>
> Conceptually it’s what we want to do, but the drawback is that the
> caller (here ‘guix package’) goes through a lot of hops to get the list
> of grafts and to apply it.
>
> I think we should instead have a way to annotate a derivation with a
> list of grafts, as well as a procedure to build that derivations in two
> phases (first the original derivation, then the grafts).

The current iteration introduces “build continuation”: a derivation can
be annotated with a continuation, and the ‘build-things’ procedure will
loop over continuations and return the final results (that only works
for derivations directly passed as an argument to ‘build-things’.)

Example (where Guile 2.0.12 replaced by 2.0.13):

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env  guix package -p foo -i guile-json guile-ssh gdb
The following packages will be installed:
   guile-json	0.5.0	/gnu/store/sd8jm2rw7cp3bnrk421kr97ki6sqxnhz-guile-json-0.5.0
   guile-ssh	0.10.2	/gnu/store/w90isin9pnm9ri8w9njxby8h98lfnkzq-guile-ssh-0.10.2
   gdb	7.12	/gnu/store/d2a4lmycc13ssdf47a9h410knlfqqq41-gdb-7.12

applying 6 grafts to /gnu/store/s7w6r9ih52kdzcl7kprf2rq48s69zh98-profile
3 packages in profile
The following environment variable definitions may be needed:
   export PATH="foo/bin${PATH:+:}$PATH"
$ guix gc -R foo | grep guile
/gnu/store/7fisf4frrgsjzmknjbab1dal23wxrp8d-guile-2.0.13
/gnu/store/w90isin9pnm9ri8w9njxby8h98lfnkzq-guile-ssh-0.10.2
/gnu/store/sd8jm2rw7cp3bnrk421kr97ki6sqxnhz-guile-json-0.5.0
--8<---------------cut here---------------end--------------->8---

That seems to be a good model.

Now we must make sure that it works also for ‘guix build foo’ and ‘guix
system build’.

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#22990; Package guix. (Wed, 11 Jan 2017 09:55:01 GMT) Full text and rfc822 format available.

Message #31 received at 22990 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: 22990 <at> debbugs.gnu.org
Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval
Date: Wed, 11 Jan 2017 10:54:42 +0100
ludo <at> gnu.org (Ludovic Courtès) skribis:

> ludo <at> gnu.org (Ludovic Courtès) skribis:
>
>> ludo <at> gnu.org (Ludovic Courtès) skribis:
>
> [...]
>
>>> To achieve this, I’m thinking of extending gexp code such that gexp
>>> compilers can return a list of applicable grafts.  The ‘package’
>>> compiler would do #:graft? #f and instead let ‘gexp->derivation’ call
>>> ‘graft-derivation’.
>>
>> The ‘wip-gexp-grafts’ branch does that.  Namely, it’s possible to know
>> what grafts would apply to a gexp derivation build, so that one can
>> first build ungrafted, and then apply the grafts to the results.
>> So for a profile, we’d first build the profile as is, and only then
>> would we graft it.
>>
>> Right now the tip of this branch is a hack such that ‘guix package’:
>>
>>   1. Builds the original (ungrafted) derivation of the profile;
>>
>>   2. Manually calls ‘graft-derivation’ on that, passing it the list of
>>      applicable grafts.
>>
>> Conceptually it’s what we want to do, but the drawback is that the
>> caller (here ‘guix package’) goes through a lot of hops to get the list
>> of grafts and to apply it.
>>
>> I think we should instead have a way to annotate a derivation with a
>> list of grafts, as well as a procedure to build that derivations in two
>> phases (first the original derivation, then the grafts).
>
> The current iteration introduces “build continuation”: a derivation can
> be annotated with a continuation, and the ‘build-things’ procedure will
> loop over continuations and return the final results (that only works
> for derivations directly passed as an argument to ‘build-things’.)

On second thought, the whole idea of applying grafts on the final result
(instead of applying grafts at each step like we do now) doesn’t fly.
It works well for things like a profile or the system derivation, but
breaks for less trivial things.

For example, if you’re building a VM image or a binary tarball, you
really need to graft packages early on; trying to graft the VM image or
binary tarball wouldn’t have the desired effect.

I’ve pushed ‘wip-gexp-grafts’ again but it seems to be a dead end.

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#22990; Package guix. (Wed, 11 Jan 2017 10:52:02 GMT) Full text and rfc822 format available.

Message #34 received at 22990 <at> debbugs.gnu.org (full text, mbox):

From: David Craven <david <at> craven.ch>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 22990 <at> debbugs.gnu.org
Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval
Date: Wed, 11 Jan 2017 11:51:29 +0100
> On second thought, the whole idea of applying grafts on the final result
> (instead of applying grafts at each step like we do now) doesn’t fly.
> It works well for things like a profile or the system derivation, but
> breaks for less trivial things.

> For example, if you’re building a VM image or a binary tarball, you
> really need to graft packages early on; trying to graft the VM image or
> binary tarball wouldn’t have the desired effect.

Isn't a system derivation or a profile derivation an intermediate step
to these derivations? Can't there be a flag or something called
#:already-grafted? #t?




Information forwarded to bug-guix <at> gnu.org:
bug#22990; Package guix. (Wed, 11 Jan 2017 21:20:01 GMT) Full text and rfc822 format available.

Message #37 received at 22990 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: David Craven <david <at> craven.ch>
Cc: 22990 <at> debbugs.gnu.org
Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval
Date: Wed, 11 Jan 2017 22:19:19 +0100
David Craven <david <at> craven.ch> skribis:

>> On second thought, the whole idea of applying grafts on the final result
>> (instead of applying grafts at each step like we do now) doesn’t fly.
>> It works well for things like a profile or the system derivation, but
>> breaks for less trivial things.
>
>> For example, if you’re building a VM image or a binary tarball, you
>> really need to graft packages early on; trying to graft the VM image or
>> binary tarball wouldn’t have the desired effect.
>
> Isn't a system derivation or a profile derivation an intermediate step
> to these derivations?

Yes, you’re right.  However the implementation I had come up with relied
on “build continuations”, which only worked for the “top-level”
derivations (those you pass to ‘build-derivations’.)

> Can't there be a flag or something called #:already-grafted? #t?

Yeah, we need something like that.  I need to chew a bit more on this to
find a nice way to achieve that.

Thanks for your feedback, I feel less lonely now.  :-)

Ludo’.




Severity set to 'important' from 'normal' Request was from ludo <at> gnu.org (Ludovic Courtès) to control <at> debbugs.gnu.org. (Sat, 27 May 2017 10:04:02 GMT) Full text and rfc822 format available.

Reply sent to Ludovic Courtès <ludo <at> gnu.org>:
You have taken responsibility. (Mon, 30 Mar 2020 02:36:51 GMT) Full text and rfc822 format available.

Notification sent to ludo <at> gnu.org (Ludovic Courtès):
bug acknowledged by developer. (Mon, 30 Mar 2020 02:36:51 GMT) Full text and rfc822 format available.

Message #44 received at 22990-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 22990-done <at> debbugs.gnu.org
Subject: Re: bug#22990: Grafts leads to inefficient substitute info retrieval
Date: Sun, 29 Mar 2020 15:41:24 +0200
Hi younger self!

ludo <at> gnu.org (Ludovic Courtès) skribis:

> As of right now (v0.9.0-2007-g66a30a3), ‘graft-derivation’ works either by:
>
>   1. Fetching substitute info about the things being built so that it
>      can determine its references, which in turns allows it to determine
>      whether they need to be grafted.
>
>   2. Building stuff, as a last resort, so that it can determine its
>      references.
>
> Case #1 is hopefully going to be the most common.
>
> The problem with #1 is that when building a profile, we do one
> ‘package-derivation’ call for each package in the profile, which
> translates in one ‘graft-derivation’ call for each relevant package¹,
> which translates into one ‘references/substitutes’ call for each.
>
> Concretely, what this means is this:
>
>   $ guix package -u
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   substitute: updating list of substitutes from 'http://mirror.guixsd.org'... 100.0%
>   […]
>   The following files would be downloaded:
>
> Each of the initial “updating list” message corresponds to an HTTP
> request for a single narinfo file, which can take around 1 second.

This is finally fixed with commit
710854304b1ab29332edcb76f3de532e0724c197, as discussed earlier this
week¹.  \o/

The solution is simpler and more orthogonal than what was envisioned
earlier in this thread, which is nice.

I also potentially addresses other uses of “dynamic dependencies”
elsewhere in the code base, such as ‘remote-eval’ calls in ‘guix
deploy’.

Feedback welcome!

Ludo’.

¹ https://lists.gnu.org/archive/html/guix-devel/2020-03/msg00337.html




Reply sent to Ludovic Courtès <ludo <at> gnu.org>:
You have taken responsibility. (Mon, 30 Mar 2020 02:36:51 GMT) Full text and rfc822 format available.

Notification sent to Dave Love <fx <at> gnu.org>:
bug acknowledged by developer. (Mon, 30 Mar 2020 02:36:51 GMT) Full text and rfc822 format available.

bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Mon, 27 Apr 2020 11:24:08 GMT) Full text and rfc822 format available.

This bug report was last modified 3 years and 336 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.