GNU bug report logs - #26608
Provide --only-substitutes flag to "guix package --upgrade"

Previous Next

Package: guix;

Reported by: Christopher Allan Webber <cwebber <at> dustycloud.org>

Date: Sat, 22 Apr 2017 16:04:02 UTC

Severity: important

To reply to this bug, email your comments to 26608 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-guix <at> gnu.org:
bug#26608; Package guix. (Sat, 22 Apr 2017 16:04:02 GMT) Full text and rfc822 format available.

Acknowledgement sent to Christopher Allan Webber <cwebber <at> dustycloud.org>:
New bug report received and forwarded. Copy sent to bug-guix <at> gnu.org. (Sat, 22 Apr 2017 16:04:02 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Christopher Allan Webber <cwebber <at> dustycloud.org>
To: bug-guix <at> gnu.org
Subject: Provide --only-substitutes flag to "guix package --upgrade"
Date: Sat, 22 Apr 2017 11:03:35 -0500
Sometimes I do an upgrade and I want to get the latest security updates,
but I also am not really available to burn through a lot of cpu time,
(especially on my x200).

I'd love it if thre were a flag so that I could specify "only bother to
upgrade the packages where we only upgrade if a substitute is already
available.

Note that I looked at the source to see if this would be an easy thing
to add; I figured that this would be handled in roughly the same place
as --keep-going or --fallback, but it looks to me like those are
actually just passed over to the build daemon, so I'm not so sure how
easy it would be to patch this in while we're still using Nix's C
daemon?  I'm not sure.





Information forwarded to bug-guix <at> gnu.org:
bug#26608; Package guix. (Sat, 22 Apr 2017 23:05:02 GMT) Full text and rfc822 format available.

Message #8 received at 26608 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Christopher Allan Webber <cwebber <at> dustycloud.org>
Cc: 26608 <at> debbugs.gnu.org
Subject: Re: bug#26608: Provide --only-substitutes flag to "guix package
 --upgrade"
Date: Sun, 23 Apr 2017 01:03:52 +0200
Christopher Allan Webber <cwebber <at> dustycloud.org> skribis:

> Sometimes I do an upgrade and I want to get the latest security updates,
> but I also am not really available to burn through a lot of cpu time,
> (especially on my x200).
>
> I'd love it if thre were a flag so that I could specify "only bother to
> upgrade the packages where we only upgrade if a substitute is already
> available.

+1

> Note that I looked at the source to see if this would be an easy thing
> to add; I figured that this would be handled in roughly the same place
> as --keep-going or --fallback, but it looks to me like those are
> actually just passed over to the build daemon, so I'm not so sure how
> easy it would be to patch this in while we're still using Nix's C
> daemon?  I'm not sure.

Clients could check, among the packages that are to be installed, which
ones are substitutable (with the ‘substitute-paths’ RPC or similar) and
filter out those that are not.  No C++ involved.

Cheers,
Ludo’.




Severity set to 'important' from 'normal' Request was from ludo <at> gnu.org (Ludovic Courtès) to control <at> debbugs.gnu.org. (Tue, 30 May 2017 15:14:02 GMT) Full text and rfc822 format available.

Information forwarded to bug-guix <at> gnu.org:
bug#26608; Package guix. (Thu, 30 Aug 2018 22:03:02 GMT) Full text and rfc822 format available.

Message #13 received at 26608 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Alex Sassmannshausen <alex <at> pompo.co>
Cc: 26608 <at> debbugs.gnu.org, Konrad Hinsen <konrad.hinsen <at> fastmail.net>,
 22629 <at> debbugs.gnu.org, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Fri, 31 Aug 2018 00:02:48 +0200
Hi Alex,

(Cc’ing <https://bugs.gnu.org/32022> and <https://bugs.gnu.org/26608>,
which are related.)

Alex Sassmannshausen <alex <at> pompo.co> skribis:

> I don't know if this is what Konrad desires, but from my perspective, a
> desirable part of the definition of stable would be a that the build
> farms have produced a set of binaries/substitutes for a given Guix
> revision that is "good enough".

I just had a bright idea (yes!): this can be addressed by writing
something like this in ~/.config/guix/channels.scm:

  (map latest-commit-with-substitutes-available
       %default-channels)

The hypothetical ‘latest-commit-with-substitutes-available’ would use
(git) and (guix ci) to find the latest commit for which substitutes of
interest are available, and would return:

  (channel
    ;; …
    (commit "cabbag3"))   ;the ideal commit

This has to be done with great care to prevent a downgrade attack and to
make sure the user doesn’t miss out on security updates, but maybe we
could provide a procedure that makes reasonable choices.

Food for thought…

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#26608; Package guix. (Fri, 31 Aug 2018 09:41:02 GMT) Full text and rfc822 format available.

Message #16 received at 26608 <at> debbugs.gnu.org (full text, mbox):

From: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
To: Ludovic Courtès <ludo <at> gnu.org>, Alex Sassmannshausen
 <alex <at> pompo.co>
Cc: 26608 <at> debbugs.gnu.org, 22629 <at> debbugs.gnu.org, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Fri, 31 Aug 2018 11:39:55 +0200
Hi Ludo,

> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
>   (map latest-commit-with-substitutes-available
>        %default-channels)
>
> The hypothetical ‘latest-commit-with-substitutes-available’ would use
> (git) and (guix ci) to find the latest commit for which substitutes of
> interest are available, and would return:

I really like that idea, but it's a pity to limit it to channels.
Two scenarii I'd like to see covered are:

 1) Find the latest commit with all substitutes required by a given
    manifest.

 2) Find the latest commit with all substitutes required for updating a
    given profile.

This is in fact only one problem with two user interfaces.

Konrad.




Information forwarded to bug-guix <at> gnu.org:
bug#26608; Package guix. (Fri, 31 Aug 2018 09:59:02 GMT) Full text and rfc822 format available.

Message #19 received at 26608 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
Cc: 26608 <at> debbugs.gnu.org, 22629 <at> debbugs.gnu.org,
 Alex Sassmannshausen <alex <at> pompo.co>, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Fri, 31 Aug 2018 11:58:15 +0200
Hi Konrad,

Konrad Hinsen <konrad.hinsen <at> fastmail.net> skribis:

>> I just had a bright idea (yes!): this can be addressed by writing
>> something like this in ~/.config/guix/channels.scm:
>>
>>   (map latest-commit-with-substitutes-available
>>        %default-channels)
>>
>> The hypothetical ‘latest-commit-with-substitutes-available’ would use
>> (git) and (guix ci) to find the latest commit for which substitutes of
>> interest are available, and would return:
>
> I really like that idea, but it's a pity to limit it to channels.

What do you mean by “limit it to channels”?  ‘%default-channels’ is an
alias for the official Guix channel (IOW, Guix itself.)

> Two scenarii I'd like to see covered are:
>
>  1) Find the latest commit with all substitutes required by a given
>     manifest.
>
>  2) Find the latest commit with all substitutes required for updating a
>     given profile.
>
> This is in fact only one problem with two user interfaces.

Yes, we could do that, and even maybe more sophisticated things (e.g.,
looking at the commit log to determine whether security fixes are
available, and adjusting the strategy accordingly.)

What I find interesting is that we can provide the tools to support such
policies, and then users can choose or implement the policy they want
directly in ~/.config/guix/channels.scm.

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#26608; Package guix. (Fri, 31 Aug 2018 10:34:02 GMT) Full text and rfc822 format available.

Message #22 received at 26608 <at> debbugs.gnu.org (full text, mbox):

From: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 26608 <at> debbugs.gnu.org, 22629 <at> debbugs.gnu.org,
 Alex Sassmannshausen <alex <at> pompo.co>, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Fri, 31 Aug 2018 12:33:19 +0200
Hi Ludo,

> What do you mean by “limit it to channels”?  ‘%default-channels’ is an
> alias for the official Guix channel (IOW, Guix itself.)

Fine, but I rarely care about all of Guix, or all of any other channel.
I care about the small subset of packages that I actually use.

Better yet, with a per-manifest/profile approach, I could put my most
critical packages in a special profile and get updates for them more
quickly, while still working only with substitutes.

BTW, just out of curiosity: for how many commits in Guix history all
packages could be built successfully? Is that the rule of the exception?

> Yes, we could do that, and even maybe more sophisticated things (e.g.,
> looking at the commit log to determine whether security fixes are
> available, and adjusting the strategy accordingly.)

Nice!

> What I find interesting is that we can provide the tools to support such
> policies, and then users can choose or implement the policy they want
> directly in ~/.config/guix/channels.scm.

I agree, it's nice to give people the tools they need to implement their
own policy.

Konrad.




Information forwarded to bug-guix <at> gnu.org:
bug#26608; Package guix. (Fri, 31 Aug 2018 11:25:02 GMT) Full text and rfc822 format available.

Message #25 received at 26608 <at> debbugs.gnu.org (full text, mbox):

From: Jan Nieuwenhuizen <janneke <at> gnu.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 26608 <at> debbugs.gnu.org, 22629 <at> debbugs.gnu.org,
 Alex Sassmannshausen <alex <at> pompo.co>, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Fri, 31 Aug 2018 13:24:25 +0200
Ludovic Courtès writes:

> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
>   (map latest-commit-with-substitutes-available
>        %default-channels)

This is a nice idea and it makes me remember that it would be useful to
provide a way to avoid installing something that is cricitally broken,
like Debian's apt-listbugs package/facility
(https://packages.debian.org/sid/apt-listbugs).

janneke

-- 
Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com




Information forwarded to bug-guix <at> gnu.org:
bug#26608; Package guix. (Fri, 31 Aug 2018 11:47:02 GMT) Full text and rfc822 format available.

Message #28 received at 26608 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 26608 <at> debbugs.gnu.org, 22629 <at> debbugs.gnu.org,
 Alex Sassmannshausen <alex <at> pompo.co>, 32022 <at> debbugs.gnu.org
Subject: Re: bug#32022: bug#22629: “Stable” branch
Date: Fri, 31 Aug 2018 13:45:29 +0200
Ludovic Courtès <ludo <at> gnu.org> writes:

> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
>   (map latest-commit-with-substitutes-available
>        %default-channels)
>
> The hypothetical ‘latest-commit-with-substitutes-available’ would use
> (git) and (guix ci) to find the latest commit for which substitutes of
> interest are available, and would return:
>
>   (channel
>     ;; …
>     (commit "cabbag3"))   ;the ideal commit
>
> This has to be done with great care to prevent a downgrade attack and to
> make sure the user doesn’t miss out on security updates, but maybe we
> could provide a procedure that makes reasonable choices.

This is a great idea.  Any kind of fetch policy could be implemented
with this, including one that considers the contents of a manifest.

This is another of these instances where having a general purpose
programming language underpinning it all really pays off.

--
Ricardo





Information forwarded to bug-guix <at> gnu.org:
bug#26608; Package guix. (Fri, 31 Aug 2018 13:03:02 GMT) Full text and rfc822 format available.

Message #31 received at 26608 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
Cc: 26608 <at> debbugs.gnu.org, 22629 <at> debbugs.gnu.org,
 Alex Sassmannshausen <alex <at> pompo.co>, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Fri, 31 Aug 2018 15:01:52 +0200
Hello,

Konrad Hinsen <konrad.hinsen <at> fastmail.net> skribis:

>> What do you mean by “limit it to channels”?  ‘%default-channels’ is an
>> alias for the official Guix channel (IOW, Guix itself.)
>
> Fine, but I rarely care about all of Guix, or all of any other channel.
> I care about the small subset of packages that I actually use.
>
> Better yet, with a per-manifest/profile approach, I could put my most
> critical packages in a special profile and get updates for them more
> quickly, while still working only with substitutes.

Sure!  The hypothetical procedure I gave can perform arbitrary checks;
it could be passed a manifest.

> BTW, just out of curiosity: for how many commits in Guix history all
> packages could be built successfully? Is that the rule of the exception?

We never have 100% of successful builds.  Of course we do our best to
keep the failure rate low, but sometimes there are unpopular packages
that remain broken for some time, or there are packages for which we
forgot to exclude some systems via ‘supported-systems’, and of course
there’s unintended breakage.

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#26608; Package guix. (Mon, 03 Sep 2018 14:12:03 GMT) Full text and rfc822 format available.

Message #34 received at 26608 <at> debbugs.gnu.org (full text, mbox):

From: Alex Sassmannshausen <alex <at> pompo.co>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 26608 <at> debbugs.gnu.org, Konrad Hinsen <konrad.hinsen <at> fastmail.net>,
 22629 <at> debbugs.gnu.org, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Mon, 03 Sep 2018 16:10:36 +0200
Hi Ludo,

Ludovic Courtès writes:

> Hi Alex,
>
> (Cc’ing <https://bugs.gnu.org/32022> and <https://bugs.gnu.org/26608>,
> which are related.)
>
> Alex Sassmannshausen <alex <at> pompo.co> skribis:
>
>> I don't know if this is what Konrad desires, but from my perspective, a
>> desirable part of the definition of stable would be a that the build
>> farms have produced a set of binaries/substitutes for a given Guix
>> revision that is "good enough".
>
> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
>   (map latest-commit-with-substitutes-available
>        %default-channels)
>
> The hypothetical ‘latest-commit-with-substitutes-available’ would use
> (git) and (guix ci) to find the latest commit for which substitutes of
> interest are available, and would return:
>
>   (channel
>     ;; …
>     (commit "cabbag3"))   ;the ideal commit

This sounds incredibly interesting — and it is testament once again to
the power of Guix that this kind of solution could be feasible!

Thinking this through in my head somewhat, I had the following thoughts:
- This procedure is invoked client side, where the channel is defined
- That means the git searching is done client side, on every invocation
of guix (I guess this might be cacheable?)
- So the downside vis-a-vis a maintained "stable branch" would be a
price in performance as experienced by the end user
- The upside of course would be automatic curation of a stable branch
that saves a ton of volunteer effort and work

I have no idea what the performance cost would be.  I guess you would
use "guix weather" to turn the set of requested packages into a manifest
which can then be checked with it.

So the cost would be one of the following scenarios:
Option a)
- fetch set of packages in a given commit
- query guix weather for 100% substitutes
- iterate until a match
- then perform the appropriate guix pull

Option b)
- perform a guix pull to the latest commit
- query guix weather for 100% substitutes
- until success, step back one step at a time through guix pull

(because of the cost of guix pull this seems unfeasible)

Option c)
Implement some form of substitute cache set querying on build farms, as
part of guix weather, so the 100% match is done on the build farm
instead of the client.

Dunno.  There may be some things that already exist in Guix land that
I'm missing.

It's a super exciting approach for sure.

> This has to be done with great care to prevent a downgrade attack and to
> make sure the user doesn’t miss out on security updates, but maybe we
> could provide a procedure that makes reasonable choices.

Right — so at the very least it would have to prevent us going "back in
time" from the guix pull commit we are currently at.

The question of security updates is tricky at the moment already — I
would hazard a guess that many people bail out of upgrading when they
can't get substitutes for their entire profile / system right now, which
means they are not getting security upgrades for package (a) when a
substitute for (b) fails.

Thanks for your thoughts — super intriguing!

Alex




Information forwarded to bug-guix <at> gnu.org:
bug#26608; Package guix. (Mon, 03 Sep 2018 19:53:02 GMT) Full text and rfc822 format available.

Message #37 received at 26608 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Alex Sassmannshausen <alex <at> pompo.co>
Cc: 26608 <at> debbugs.gnu.org, Konrad Hinsen <konrad.hinsen <at> fastmail.net>,
 22629 <at> debbugs.gnu.org, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Mon, 03 Sep 2018 21:52:00 +0200
Hi Alex,

Alex Sassmannshausen <alex <at> pompo.co> skribis:

> Ludovic Courtès writes:

[...]

>> I just had a bright idea (yes!): this can be addressed by writing
>> something like this in ~/.config/guix/channels.scm:
>>
>>   (map latest-commit-with-substitutes-available
>>        %default-channels)
>>
>> The hypothetical ‘latest-commit-with-substitutes-available’ would use
>> (git) and (guix ci) to find the latest commit for which substitutes of
>> interest are available, and would return:
>>
>>   (channel
>>     ;; …
>>     (commit "cabbag3"))   ;the ideal commit
>
> This sounds incredibly interesting — and it is testament once again to
> the power of Guix that this kind of solution could be feasible!

Just to be clear: I don’t think this would be a substitute for a
“stable” branch; rather, I view as a way to have user-defined policies
such as “pull up to the latest commit for which there’s a substitute for
IceCat.”

> Thinking this through in my head somewhat, I had the following thoughts:
> - This procedure is invoked client side, where the channel is defined
> - That means the git searching is done client side, on every invocation
> of guix (I guess this might be cacheable?)

On every invocation of ‘guix pull’ only.

> I have no idea what the performance cost would be.  I guess you would
> use "guix weather" to turn the set of requested packages into a manifest
> which can then be checked with it.

As I imagine it, the cost would be a few HTTP queries to the Cuirass
API.  I should try to come up with an example to better explain what I
had in mind!

> The question of security updates is tricky at the moment already — I
> would hazard a guess that many people bail out of upgrading when they
> can't get substitutes for their entire profile / system right now, which
> means they are not getting security upgrades for package (a) when a
> substitute for (b) fails.

That’s probably true, and I agree it’s problematic.

What I typically do is “guix pull && guix package -n -u”.  Then I look
at things that would be built; if, say, LibreOffice is among them, I
wait for a little while and try again later, until I can get enough
substitutes.  That usually works okay, but it fails if it turns out that
one of the dependencies fails to build: substitutes never become
available in that case.

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#26608; Package guix. (Mon, 03 Sep 2018 20:28:02 GMT) Full text and rfc822 format available.

Message #40 received at 26608 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Alex Sassmannshausen <alex <at> pompo.co>
Cc: 26608 <at> debbugs.gnu.org, Konrad Hinsen <konrad.hinsen <at> fastmail.net>,
 22629 <at> debbugs.gnu.org, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Mon, 03 Sep 2018 22:27:25 +0200
[Message part 1 (text/plain, inline)]
Alex Sassmannshausen <alex <at> pompo.co> skribis:

> Ludovic Courtès writes:

[...]

>> I just had a bright idea (yes!): this can be addressed by writing
>> something like this in ~/.config/guix/channels.scm:
>>
>>   (map latest-commit-with-substitutes-available
>>        %default-channels)
>>
>> The hypothetical ‘latest-commit-with-substitutes-available’ would use
>> (git) and (guix ci) to find the latest commit for which substitutes of
>> interest are available, and would return:
>>
>>   (channel
>>     ;; …
>>     (commit "cabbag3"))   ;the ideal commit

The code below is an illustration of that.  If you install it as
~/.config/guix/channels.scm, ‘guix pull’ will pull the latest commit
that was fully built on berlin.guixsd.org (see
<https://berlin.guixsd.org/jobset/guix-modular-master>), meaning that
substitutes for Guix itself should be available, unless ‘guix publish’
hasn’t “baked” them yet.

It takes two GETs and ~1s to do that here.

Ludo’.

[channels-test.scm (text/plain, inline)]
(use-modules (guix http-client)
             (json)
             (srfi srfi-1)
             (ice-9 match))

(define (latest-evaluations jobset)
  "Return the latest evaluations of JOBSET."
  (filter (lambda (json)
            (string=? (hash-ref json "specification") jobset))
          (json->scm
           (http-fetch
            "https://berlin.guixsd.org/api/evaluations?nr=30"))))

(define (evaluation-complete? number)
  "Return true if evaluation NUMBER completed and all its builds were
successful."
  (let ((builds (json->scm
                 (http-fetch
                  (string-append
                   "https://berlin.guixsd.org/api/latestbuilds?nr=30&evaluation="
                   (number->string number))))))
    (every (lambda (build)
             ;; Zero means build success.
             (= (hash-ref build "buildstatus") 0))
           builds)))

(define (latest-commit-successfully-built)
  "Return the latest commit for which substitutes are (potentially)
available."
  (let* ((evaluations (latest-evaluations "guix-modular-master"))
         (candidates  (filter-map (lambda (json)
                                    (match (hash-ref json "checkouts")
                                      ((checkout)
                                       (cons (hash-ref json "id")
                                             (hash-ref checkout "commit")))
                                      (_ #f)))
                                  evaluations)))
    (any (match-lambda
            ((evaluation . commit)
             (and (evaluation-complete? evaluation)
                  commit)))
          candidates)))

;; Pull the latest commit fully built on berlin.guixsd.org.
;; WARNING: This could downgrade your system!
(list (channel
       (name 'guix)
       (url "https://git.savannah.gnu.org/git/guix.git")
       (commit (pk 'commit (latest-commit-successfully-built)))))

Information forwarded to bug-guix <at> gnu.org:
bug#26608; Package guix. (Tue, 04 Sep 2018 08:03:02 GMT) Full text and rfc822 format available.

Message #43 received at 26608 <at> debbugs.gnu.org (full text, mbox):

From: Alex Sassmannshausen <alex <at> pompo.co>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 26608 <at> debbugs.gnu.org, Konrad Hinsen <konrad.hinsen <at> fastmail.net>,
 22629 <at> debbugs.gnu.org, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Tue, 04 Sep 2018 10:02:31 +0200
Ludovic Courtès writes:

> Hi Alex,
>
> Alex Sassmannshausen <alex <at> pompo.co> skribis:
>
>> Ludovic Courtès writes:
>
> [...]
>
>>> I just had a bright idea (yes!): this can be addressed by writing
>>> something like this in ~/.config/guix/channels.scm:
>>>
>>>   (map latest-commit-with-substitutes-available
>>>        %default-channels)
>>>
>>> The hypothetical ‘latest-commit-with-substitutes-available’ would use
>>> (git) and (guix ci) to find the latest commit for which substitutes of
>>> interest are available, and would return:
>>>
>>>   (channel
>>>     ;; …
>>>     (commit "cabbag3"))   ;the ideal commit
>>
>> This sounds incredibly interesting — and it is testament once again to
>> the power of Guix that this kind of solution could be feasible!
>
> Just to be clear: I don’t think this would be a substitute for a
> “stable” branch; rather, I view as a way to have user-defined policies
> such as “pull up to the latest commit for which there’s a substitute for
> IceCat.”

Ah, I understand now.

So the example you provided is a user-defined policy to install the
latest version of Guix that is downloadable using substitutes (if guix
publish has published those already).

As you say, in a similar vein, the end user could for themselves define
a policy that searches for a commit containing a specific successful
build, or a set of specific successful builds.

>> Thinking this through in my head somewhat, I had the following thoughts:
>> - This procedure is invoked client side, where the channel is defined
>> - That means the git searching is done client side, on every invocation
>> of guix (I guess this might be cacheable?)
>
> On every invocation of ‘guix pull’ only.

That makes sense, and is way better than I feared :-)

>> I have no idea what the performance cost would be.  I guess you would
>> use "guix weather" to turn the set of requested packages into a manifest
>> which can then be checked with it.
>
> As I imagine it, the cost would be a few HTTP queries to the Cuirass
> API.  I should try to come up with an example to better explain what I
> had in mind!

Your example helps visualize this, thanks.

Your example depends on there being a jobset that comprises the set of
packages you are interested in testing.

I imagine it is possible to do the same for an individual package / job.

The situation would be different if the end user wanted to perform a
similar operation for an arbitrary set of packages on their end.

It would probably involve something like this (probably naive):

(define (latest-commit-successfully-built-pkg pkg)
  "Return the latest commit for the pkg for which substitutes are
(potentially) available."
  ;;                  Like your version, but magically performs query
  ;;                  for pkg, not the guix-modular-master evaluation
  (let* ((evaluations (latest-evaluations pkg))
         (candidates  (filter-map (lambda (json)
                                    (match (hash-ref json "checkouts")
                                      ((checkout)
                                       (cons (hash-ref json "id")
                                             (hash-ref checkout "commit")))
                                      (_ #f)))
                                  evaluations)))
    (map (match-lambda
            ((evaluation . commit)
             (and (evaluation-complete? evaluation)
                  commit)))
          candidates)))

(any (match-lambda
        ((evaluation . commit) commit)
     (apply lset-intersection equal?
     ;;   Like latest-commit-successfully-built, but takes an
     ;;   individual package name for which we return the
     ;;   commit
     (map latest-commit-successfully-built-pkg
          %set-of-packages))))

Obviously the larger the set, the more requests are required, and the
lower the chance of a commit being available / a downgrade occuring

>> The question of security updates is tricky at the moment already — I
>> would hazard a guess that many people bail out of upgrading when they
>> can't get substitutes for their entire profile / system right now, which
>> means they are not getting security upgrades for package (a) when a
>> substitute for (b) fails.
>
> That’s probably true, and I agree it’s problematic.
>
> What I typically do is “guix pull && guix package -n -u”.  Then I look
> at things that would be built; if, say, LibreOffice is among them, I
> wait for a little while and try again later, until I can get enough
> substitutes.  That usually works okay, but it fails if it turns out that
> one of the dependencies fails to build: substitutes never become
> available in that case.

Interesting.  Do you think this kind of thing might be useful to have in
the Guix manual? Like, in a section about a "typical" desktop end-user
might manage their system day to day?

Alex




Information forwarded to bug-guix <at> gnu.org:
bug#26608; Package guix. (Tue, 04 Sep 2018 12:23:01 GMT) Full text and rfc822 format available.

Message #46 received at 26608 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Alex Sassmannshausen <alex <at> pompo.co>
Cc: 26608 <at> debbugs.gnu.org, Konrad Hinsen <konrad.hinsen <at> fastmail.net>,
 22629 <at> debbugs.gnu.org, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Tue, 04 Sep 2018 14:22:35 +0200
Hi Alex,

Alex Sassmannshausen <alex <at> pompo.co> skribis:

> So the example you provided is a user-defined policy to install the
> latest version of Guix that is downloadable using substitutes (if guix
> publish has published those already).
>
> As you say, in a similar vein, the end user could for themselves define
> a policy that searches for a commit containing a specific successful
> build, or a set of specific successful builds.

Exactly.

>> As I imagine it, the cost would be a few HTTP queries to the Cuirass
>> API.  I should try to come up with an example to better explain what I
>> had in mind!
>
> Your example helps visualize this, thanks.
>
> Your example depends on there being a jobset that comprises the set of
> packages you are interested in testing.

Yes, and it’s hacky in that the substitute server and jobset names are
hard-coded, but you get the idea.

> I imagine it is possible to do the same for an individual package / job.

Yes.

> The situation would be different if the end user wanted to perform a
> similar operation for an arbitrary set of packages on their end.

It would be quite similar: you would query the set of builds of an
evaluation of the “guix-modular” jobset and check whether the packages
of interest were built.

>> What I typically do is “guix pull && guix package -n -u”.  Then I look
>> at things that would be built; if, say, LibreOffice is among them, I
>> wait for a little while and try again later, until I can get enough
>> substitutes.  That usually works okay, but it fails if it turns out that
>> one of the dependencies fails to build: substitutes never become
>> available in that case.
>
> Interesting.  Do you think this kind of thing might be useful to have in
> the Guix manual? Like, in a section about a "typical" desktop end-user
> might manage their system day to day?

It would make sense to have such a section I guess.  However, before
teaching users how to work around deficiencies of our infrastructure our
processes ;-), I’d like us to improve them much as possible.  I’m sure
we have room for improvement for instance in Cuirass.

Thanks,
Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#26608; Package guix. (Tue, 03 Dec 2019 19:25:01 GMT) Full text and rfc822 format available.

Message #49 received at 26608 <at> debbugs.gnu.org (full text, mbox):

From: Nome Grey <greynome72 <at> gmail.com>
To: 26608 <at> debbugs.gnu.org, 32022 <at> debbugs.gnu.org, 22629 <at> debbugs.gnu.org
Subject: channels.scm supporting substitutes
Date: Tue, 3 Dec 2019 09:55:41 -0800
[Message part 1 (text/plain, inline)]
Ludovic posted some channels.scm code in September 2018 supporting using
more substitutes.  Unfortunately his code no longer functions due to an
upgrade of guile-json in guix.

I've tried to learn enough guile to upgrade the code to the newer json
structures, and posted my changes to github at
https://github.com/nomr72/guix-substitutes-channel .  It seems to run again
now.

Maybe I can learn enough to upgrade it to check the 'guix-master'
evaluations to find the latest build of key packages.  We'll see.
[Message part 2 (text/html, inline)]

Information forwarded to bug-guix <at> gnu.org:
bug#26608; Package guix. (Tue, 10 Dec 2019 16:43:02 GMT) Full text and rfc822 format available.

Message #52 received at 26608 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Nome Grey <greynome72 <at> gmail.com>
Cc: 26608 <at> debbugs.gnu.org, 22629 <at> debbugs.gnu.org, 32022 <at> debbugs.gnu.org
Subject: Re: bug#26608: channels.scm supporting substitutes
Date: Tue, 10 Dec 2019 17:41:54 +0100
Hi,

Nome Grey <greynome72 <at> gmail.com> skribis:

> Ludovic posted some channels.scm code in September 2018 supporting using
> more substitutes.  Unfortunately his code no longer functions due to an
> upgrade of guile-json in guix.
>
> I've tried to learn enough guile to upgrade the code to the newer json
> structures, and posted my changes to github at
> https://github.com/nomr72/guix-substitutes-channel .  It seems to run again
> now.

Nice, thanks for sharing!

Ludo’.




Information forwarded to bug-guix <at> gnu.org:
bug#26608; Package guix. (Fri, 20 Aug 2021 11:34:03 GMT) Full text and rfc822 format available.

Message #55 received at 26608 <at> debbugs.gnu.org (full text, mbox):

From: zimoun <zimon.toutoune <at> gmail.com>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 26608 <at> debbugs.gnu.org, Konrad Hinsen <konrad.hinsen <at> fastmail.net>,
 Alex Sassmannshausen <alex <at> pompo.co>, 22629 <at> debbugs.gnu.org,
 32022 <at> debbugs.gnu.org
Subject: Re: bug#26608: bug#22629: “Stable” branch
Date: Fri, 20 Aug 2021 13:09:18 +0200
Hi,

On Fri, 31 Aug 2018 at 00:02, ludo <at> gnu.org (Ludovic Courtès) wrote:

> (Cc’ing <https://bugs.gnu.org/32022> and <https://bugs.gnu.org/26608>,
> which are related.)
>
> Alex Sassmannshausen <alex <at> pompo.co> skribis:
>
>> I don't know if this is what Konrad desires, but from my perspective, a
>> desirable part of the definition of stable would be a that the build
>> farms have produced a set of binaries/substitutes for a given Guix
>> revision that is "good enough".
>
> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
>   (map latest-commit-with-substitutes-available
>        %default-channels)
>
> The hypothetical ‘latest-commit-with-substitutes-available’ would use
> (git) and (guix ci) to find the latest commit for which substitutes of
> interest are available, and would return:
>
>   (channel
>     ;; …
>     (commit "cabbag3"))   ;the ideal commit
>
> This has to be done with great care to prevent a downgrade attack and to
> make sure the user doesn’t miss out on security updates, but maybe we
> could provide a procedure that makes reasonable choices.

From my understanding, this bug can be close because:

<https://guix.gnu.org/manual/devel/en/guix.html#Channels-with-Substitutes>
<https://guix.gnu.org/manual/devel/en/guix.html#Channel-Authentication>

WDYT?

All the best,
simon




This bug report was last modified 2 years and 250 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.