GNU bug report logs - #32022
(new feature) Change guix pull to choose commits for which substitutes is already built by default

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 32022 in the body.
You can then email your comments to 32022 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: swedebugia <swedebugia <at> riseup.net>
To: "Bug-guix <at> gnu org" <Bug-guix <at> gnu.org>
Subject: (new feature) Change guix pull to choose commits for which
 substitutes is already built by default 
Date: Sat, 30 Jun 2018 21:06:55 +0200

[Message part 1 (text/plain, inline)]

Hi

I would like it to be easy for newcomers to do the right thing when pulling.

I suggest we by default give guix pullers a bit more information and interaction.
Something like this:
"Checking for newly built derivations of guix at hydra.gnu.org...
(table)
Date            Commit         Succeeded for x64?
28/6            54d84d8        yes
... 
... 
... 

Pulling based on default commit selection scheme (see --help for details) 
Most recently built commit  is x4678x85 built x hours ago. 
Continuing in 10 seconds...
Press ctrl+c to abort. "

Guix pull gets a new argument " -d" allowing you to specify the days old commit you want to select. 
E. G. Guix pull -d 5 selects a commit build successfully 5 days ago by hydra. 

Guix weather could show the table only, if the argument --guix-pull was given. 
-- 
Cheers Swedebugia 

[Message part 2 (text/html, inline)]

Message #8 received at 32022 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: swedebugia <swedebugia <at> riseup.net>
Cc: 32022 <at> debbugs.gnu.org
Subject: Re: bug#32022: (new feature) Change guix pull to choose commits for
 which substitutes is already built by default
Date: Mon, 02 Jul 2018 15:45:37 +0200

Hello swedebugia,

swedebugia <swedebugia <at> riseup.net> skribis:

> I would like it to be easy for newcomers to do the right thing when pulling.
>
> I suggest we by default give guix pullers a bit more information and interaction.
> Something like this:
> "Checking for newly built derivations of guix at hydra.gnu.org...
> (table)
> Date            Commit         Succeeded for x64?
> 28/6            54d84d8        yes
> ... 
> ... 
> ... 
>
> Pulling based on default commit selection scheme (see --help for details) 
> Most recently built commit  is x4678x85 built x hours ago. 
> Continuing in 10 seconds...
> Press ctrl+c to abort. "

Any strategy that makes it easy to update to an “old” Guix revision is
risky: one could easily tweak the user into using an old revision that
lacks an important security fix.  So that’s not an option to me.

Furthermore, note that people may choose not to use substitutes.  The
proposed changes wouldn’t help them.

The problem we’re trying to address here is ‘guix pull’ slowness.  We
can address it in several ways:

  1. Provide substitutes in a timely fashion.  We do that on
     berlin.guixsd.org; hydra.gnu.org now provides substitutes, but not
     in a timely fashion.  We should arrange to make them more reactive
     so that you’re unlikely to have to build things from source, though
     we can’t entirely eliminate situations where you do have to build
     part of the stuff from source.

  2. Break up the build work that ‘guix pull’ does into reasonably-sized
     derivations.  (guix self) is a step in that direction, but as you
     know there are still big derivations that have to compile a whole
     lot of package modules.  We could split that further.

  3. Make Guile’s compiler faster.  I think there’s still room for
     improvement and that would benefit everyone.

IMO we should really work on these fronts rather than putting users at
risk just to paper over these performance issues.

Thanks,
Ludo’.

Message #11 received at 32022 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Alex Sassmannshausen <alex <at> pompo.co>
Cc: 26608 <at> debbugs.gnu.org, Konrad Hinsen <konrad.hinsen <at> fastmail.net>,
 22629 <at> debbugs.gnu.org, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Fri, 31 Aug 2018 00:02:48 +0200

Hi Alex,

(Cc’ing <https://bugs.gnu.org/32022> and <https://bugs.gnu.org/26608>,
which are related.)

Alex Sassmannshausen <alex <at> pompo.co> skribis:

> I don't know if this is what Konrad desires, but from my perspective, a
> desirable part of the definition of stable would be a that the build
> farms have produced a set of binaries/substitutes for a given Guix
> revision that is "good enough".

I just had a bright idea (yes!): this can be addressed by writing
something like this in ~/.config/guix/channels.scm:

  (map latest-commit-with-substitutes-available
       %default-channels)

The hypothetical ‘latest-commit-with-substitutes-available’ would use
(git) and (guix ci) to find the latest commit for which substitutes of
interest are available, and would return:

  (channel
    ;; …
    (commit "cabbag3"))   ;the ideal commit

This has to be done with great care to prevent a downgrade attack and to
make sure the user doesn’t miss out on security updates, but maybe we
could provide a procedure that makes reasonable choices.

Food for thought…

Ludo’.

Message #14 received at 32022 <at> debbugs.gnu.org (full text, mbox):

From: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
To: Ludovic Courtès <ludo <at> gnu.org>, Alex Sassmannshausen
 <alex <at> pompo.co>
Cc: 26608 <at> debbugs.gnu.org, 22629 <at> debbugs.gnu.org, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Fri, 31 Aug 2018 11:39:55 +0200

Hi Ludo,

> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
>   (map latest-commit-with-substitutes-available
>        %default-channels)
>
> The hypothetical ‘latest-commit-with-substitutes-available’ would use
> (git) and (guix ci) to find the latest commit for which substitutes of
> interest are available, and would return:

I really like that idea, but it's a pity to limit it to channels.
Two scenarii I'd like to see covered are:

 1) Find the latest commit with all substitutes required by a given
    manifest.

 2) Find the latest commit with all substitutes required for updating a
    given profile.

This is in fact only one problem with two user interfaces.

Konrad.

Message #17 received at 32022 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
Cc: 26608 <at> debbugs.gnu.org, 22629 <at> debbugs.gnu.org,
 Alex Sassmannshausen <alex <at> pompo.co>, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Fri, 31 Aug 2018 11:58:15 +0200

Hi Konrad,

Konrad Hinsen <konrad.hinsen <at> fastmail.net> skribis:

>> I just had a bright idea (yes!): this can be addressed by writing
>> something like this in ~/.config/guix/channels.scm:
>>
>>   (map latest-commit-with-substitutes-available
>>        %default-channels)
>>
>> The hypothetical ‘latest-commit-with-substitutes-available’ would use
>> (git) and (guix ci) to find the latest commit for which substitutes of
>> interest are available, and would return:
>
> I really like that idea, but it's a pity to limit it to channels.

What do you mean by “limit it to channels”?  ‘%default-channels’ is an
alias for the official Guix channel (IOW, Guix itself.)

> Two scenarii I'd like to see covered are:
>
>  1) Find the latest commit with all substitutes required by a given
>     manifest.
>
>  2) Find the latest commit with all substitutes required for updating a
>     given profile.
>
> This is in fact only one problem with two user interfaces.

Yes, we could do that, and even maybe more sophisticated things (e.g.,
looking at the commit log to determine whether security fixes are
available, and adjusting the strategy accordingly.)

What I find interesting is that we can provide the tools to support such
policies, and then users can choose or implement the policy they want
directly in ~/.config/guix/channels.scm.

Ludo’.

Message #20 received at 32022 <at> debbugs.gnu.org (full text, mbox):

From: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 26608 <at> debbugs.gnu.org, 22629 <at> debbugs.gnu.org,
 Alex Sassmannshausen <alex <at> pompo.co>, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Fri, 31 Aug 2018 12:33:19 +0200

Hi Ludo,

> What do you mean by “limit it to channels”?  ‘%default-channels’ is an
> alias for the official Guix channel (IOW, Guix itself.)

Fine, but I rarely care about all of Guix, or all of any other channel.
I care about the small subset of packages that I actually use.

Better yet, with a per-manifest/profile approach, I could put my most
critical packages in a special profile and get updates for them more
quickly, while still working only with substitutes.

BTW, just out of curiosity: for how many commits in Guix history all
packages could be built successfully? Is that the rule of the exception?

> Yes, we could do that, and even maybe more sophisticated things (e.g.,
> looking at the commit log to determine whether security fixes are
> available, and adjusting the strategy accordingly.)

Nice!

> What I find interesting is that we can provide the tools to support such
> policies, and then users can choose or implement the policy they want
> directly in ~/.config/guix/channels.scm.

I agree, it's nice to give people the tools they need to implement their
own policy.

Konrad.

Message #23 received at 32022 <at> debbugs.gnu.org (full text, mbox):

From: Jan Nieuwenhuizen <janneke <at> gnu.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 26608 <at> debbugs.gnu.org, 22629 <at> debbugs.gnu.org,
 Alex Sassmannshausen <alex <at> pompo.co>, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Fri, 31 Aug 2018 13:24:25 +0200

Ludovic Courtès writes:

> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
>   (map latest-commit-with-substitutes-available
>        %default-channels)

This is a nice idea and it makes me remember that it would be useful to
provide a way to avoid installing something that is cricitally broken,
like Debian's apt-listbugs package/facility
(https://packages.debian.org/sid/apt-listbugs).

janneke

-- 
Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Message #26 received at 32022 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 26608 <at> debbugs.gnu.org, 22629 <at> debbugs.gnu.org,
 Alex Sassmannshausen <alex <at> pompo.co>, 32022 <at> debbugs.gnu.org
Subject: Re: bug#32022: bug#22629: “Stable” branch
Date: Fri, 31 Aug 2018 13:45:29 +0200

Ludovic Courtès <ludo <at> gnu.org> writes:

> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
>   (map latest-commit-with-substitutes-available
>        %default-channels)
>
> The hypothetical ‘latest-commit-with-substitutes-available’ would use
> (git) and (guix ci) to find the latest commit for which substitutes of
> interest are available, and would return:
>
>   (channel
>     ;; …
>     (commit "cabbag3"))   ;the ideal commit
>
> This has to be done with great care to prevent a downgrade attack and to
> make sure the user doesn’t miss out on security updates, but maybe we
> could provide a procedure that makes reasonable choices.

This is a great idea.  Any kind of fetch policy could be implemented
with this, including one that considers the contents of a manifest.

This is another of these instances where having a general purpose
programming language underpinning it all really pays off.

--
Ricardo

Message #29 received at 32022 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
Cc: 26608 <at> debbugs.gnu.org, 22629 <at> debbugs.gnu.org,
 Alex Sassmannshausen <alex <at> pompo.co>, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Fri, 31 Aug 2018 15:01:52 +0200

Hello,

Konrad Hinsen <konrad.hinsen <at> fastmail.net> skribis:

>> What do you mean by “limit it to channels”?  ‘%default-channels’ is an
>> alias for the official Guix channel (IOW, Guix itself.)
>
> Fine, but I rarely care about all of Guix, or all of any other channel.
> I care about the small subset of packages that I actually use.
>
> Better yet, with a per-manifest/profile approach, I could put my most
> critical packages in a special profile and get updates for them more
> quickly, while still working only with substitutes.

Sure!  The hypothetical procedure I gave can perform arbitrary checks;
it could be passed a manifest.

> BTW, just out of curiosity: for how many commits in Guix history all
> packages could be built successfully? Is that the rule of the exception?

We never have 100% of successful builds.  Of course we do our best to
keep the failure rate low, but sometimes there are unpopular packages
that remain broken for some time, or there are packages for which we
forgot to exclude some systems via ‘supported-systems’, and of course
there’s unintended breakage.

Ludo’.

Message #32 received at 32022 <at> debbugs.gnu.org (full text, mbox):

From: Alex Sassmannshausen <alex <at> pompo.co>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 26608 <at> debbugs.gnu.org, Konrad Hinsen <konrad.hinsen <at> fastmail.net>,
 22629 <at> debbugs.gnu.org, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Mon, 03 Sep 2018 16:10:36 +0200

Hi Ludo,

Ludovic Courtès writes:

> Hi Alex,
>
> (Cc’ing <https://bugs.gnu.org/32022> and <https://bugs.gnu.org/26608>,
> which are related.)
>
> Alex Sassmannshausen <alex <at> pompo.co> skribis:
>
>> I don't know if this is what Konrad desires, but from my perspective, a
>> desirable part of the definition of stable would be a that the build
>> farms have produced a set of binaries/substitutes for a given Guix
>> revision that is "good enough".
>
> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
>   (map latest-commit-with-substitutes-available
>        %default-channels)
>
> The hypothetical ‘latest-commit-with-substitutes-available’ would use
> (git) and (guix ci) to find the latest commit for which substitutes of
> interest are available, and would return:
>
>   (channel
>     ;; …
>     (commit "cabbag3"))   ;the ideal commit

This sounds incredibly interesting — and it is testament once again to
the power of Guix that this kind of solution could be feasible!

Thinking this through in my head somewhat, I had the following thoughts:
- This procedure is invoked client side, where the channel is defined
- That means the git searching is done client side, on every invocation
of guix (I guess this might be cacheable?)
- So the downside vis-a-vis a maintained "stable branch" would be a
price in performance as experienced by the end user
- The upside of course would be automatic curation of a stable branch
that saves a ton of volunteer effort and work

I have no idea what the performance cost would be.  I guess you would
use "guix weather" to turn the set of requested packages into a manifest
which can then be checked with it.

So the cost would be one of the following scenarios:
Option a)
- fetch set of packages in a given commit
- query guix weather for 100% substitutes
- iterate until a match
- then perform the appropriate guix pull

Option b)
- perform a guix pull to the latest commit
- query guix weather for 100% substitutes
- until success, step back one step at a time through guix pull

(because of the cost of guix pull this seems unfeasible)

Option c)
Implement some form of substitute cache set querying on build farms, as
part of guix weather, so the 100% match is done on the build farm
instead of the client.

Dunno.  There may be some things that already exist in Guix land that
I'm missing.

It's a super exciting approach for sure.

> This has to be done with great care to prevent a downgrade attack and to
> make sure the user doesn’t miss out on security updates, but maybe we
> could provide a procedure that makes reasonable choices.

Right — so at the very least it would have to prevent us going "back in
time" from the guix pull commit we are currently at.

The question of security updates is tricky at the moment already — I
would hazard a guess that many people bail out of upgrading when they
can't get substitutes for their entire profile / system right now, which
means they are not getting security upgrades for package (a) when a
substitute for (b) fails.

Thanks for your thoughts — super intriguing!

Alex

Message #35 received at 32022 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Alex Sassmannshausen <alex <at> pompo.co>
Cc: 26608 <at> debbugs.gnu.org, Konrad Hinsen <konrad.hinsen <at> fastmail.net>,
 22629 <at> debbugs.gnu.org, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Mon, 03 Sep 2018 21:52:00 +0200

Hi Alex,

Alex Sassmannshausen <alex <at> pompo.co> skribis:

> Ludovic Courtès writes:

[...]

>> I just had a bright idea (yes!): this can be addressed by writing
>> something like this in ~/.config/guix/channels.scm:
>>
>>   (map latest-commit-with-substitutes-available
>>        %default-channels)
>>
>> The hypothetical ‘latest-commit-with-substitutes-available’ would use
>> (git) and (guix ci) to find the latest commit for which substitutes of
>> interest are available, and would return:
>>
>>   (channel
>>     ;; …
>>     (commit "cabbag3"))   ;the ideal commit
>
> This sounds incredibly interesting — and it is testament once again to
> the power of Guix that this kind of solution could be feasible!

Just to be clear: I don’t think this would be a substitute for a
“stable” branch; rather, I view as a way to have user-defined policies
such as “pull up to the latest commit for which there’s a substitute for
IceCat.”

> Thinking this through in my head somewhat, I had the following thoughts:
> - This procedure is invoked client side, where the channel is defined
> - That means the git searching is done client side, on every invocation
> of guix (I guess this might be cacheable?)

On every invocation of ‘guix pull’ only.

> I have no idea what the performance cost would be.  I guess you would
> use "guix weather" to turn the set of requested packages into a manifest
> which can then be checked with it.

As I imagine it, the cost would be a few HTTP queries to the Cuirass
API.  I should try to come up with an example to better explain what I
had in mind!

> The question of security updates is tricky at the moment already — I
> would hazard a guess that many people bail out of upgrading when they
> can't get substitutes for their entire profile / system right now, which
> means they are not getting security upgrades for package (a) when a
> substitute for (b) fails.

That’s probably true, and I agree it’s problematic.

What I typically do is “guix pull && guix package -n -u”.  Then I look
at things that would be built; if, say, LibreOffice is among them, I
wait for a little while and try again later, until I can get enough
substitutes.  That usually works okay, but it fails if it turns out that
one of the dependencies fails to build: substitutes never become
available in that case.

Ludo’.

Message #38 received at 32022 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Alex Sassmannshausen <alex <at> pompo.co>
Cc: 26608 <at> debbugs.gnu.org, Konrad Hinsen <konrad.hinsen <at> fastmail.net>,
 22629 <at> debbugs.gnu.org, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Mon, 03 Sep 2018 22:27:25 +0200

[Message part 1 (text/plain, inline)]

Alex Sassmannshausen <alex <at> pompo.co> skribis:

> Ludovic Courtès writes:

[...]

>> I just had a bright idea (yes!): this can be addressed by writing
>> something like this in ~/.config/guix/channels.scm:
>>
>>   (map latest-commit-with-substitutes-available
>>        %default-channels)
>>
>> The hypothetical ‘latest-commit-with-substitutes-available’ would use
>> (git) and (guix ci) to find the latest commit for which substitutes of
>> interest are available, and would return:
>>
>>   (channel
>>     ;; …
>>     (commit "cabbag3"))   ;the ideal commit

The code below is an illustration of that.  If you install it as
~/.config/guix/channels.scm, ‘guix pull’ will pull the latest commit
that was fully built on berlin.guixsd.org (see
<https://berlin.guixsd.org/jobset/guix-modular-master>), meaning that
substitutes for Guix itself should be available, unless ‘guix publish’
hasn’t “baked” them yet.

It takes two GETs and ~1s to do that here.

Ludo’.

[channels-test.scm (text/plain, inline)]

(use-modules (guix http-client)
             (json)
             (srfi srfi-1)
             (ice-9 match))

(define (latest-evaluations jobset)
  "Return the latest evaluations of JOBSET."
  (filter (lambda (json)
            (string=? (hash-ref json "specification") jobset))
          (json->scm
           (http-fetch
            "https://berlin.guixsd.org/api/evaluations?nr=30"))))

(define (evaluation-complete? number)
  "Return true if evaluation NUMBER completed and all its builds were
successful."
  (let ((builds (json->scm
                 (http-fetch
                  (string-append
                   "https://berlin.guixsd.org/api/latestbuilds?nr=30&evaluation="
                   (number->string number))))))
    (every (lambda (build)
             ;; Zero means build success.
             (= (hash-ref build "buildstatus") 0))
           builds)))

(define (latest-commit-successfully-built)
  "Return the latest commit for which substitutes are (potentially)
available."
  (let* ((evaluations (latest-evaluations "guix-modular-master"))
         (candidates  (filter-map (lambda (json)
                                    (match (hash-ref json "checkouts")
                                      ((checkout)
                                       (cons (hash-ref json "id")
                                             (hash-ref checkout "commit")))
                                      (_ #f)))
                                  evaluations)))
    (any (match-lambda
            ((evaluation . commit)
             (and (evaluation-complete? evaluation)
                  commit)))
          candidates)))

;; Pull the latest commit fully built on berlin.guixsd.org.
;; WARNING: This could downgrade your system!
(list (channel
       (name 'guix)
       (url "https://git.savannah.gnu.org/git/guix.git")
       (commit (pk 'commit (latest-commit-successfully-built)))))

Message #41 received at 32022 <at> debbugs.gnu.org (full text, mbox):

From: Alex Sassmannshausen <alex <at> pompo.co>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 26608 <at> debbugs.gnu.org, Konrad Hinsen <konrad.hinsen <at> fastmail.net>,
 22629 <at> debbugs.gnu.org, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Tue, 04 Sep 2018 10:02:31 +0200

Ludovic Courtès writes:

> Hi Alex,
>
> Alex Sassmannshausen <alex <at> pompo.co> skribis:
>
>> Ludovic Courtès writes:
>
> [...]
>
>>> I just had a bright idea (yes!): this can be addressed by writing
>>> something like this in ~/.config/guix/channels.scm:
>>>
>>>   (map latest-commit-with-substitutes-available
>>>        %default-channels)
>>>
>>> The hypothetical ‘latest-commit-with-substitutes-available’ would use
>>> (git) and (guix ci) to find the latest commit for which substitutes of
>>> interest are available, and would return:
>>>
>>>   (channel
>>>     ;; …
>>>     (commit "cabbag3"))   ;the ideal commit
>>
>> This sounds incredibly interesting — and it is testament once again to
>> the power of Guix that this kind of solution could be feasible!
>
> Just to be clear: I don’t think this would be a substitute for a
> “stable” branch; rather, I view as a way to have user-defined policies
> such as “pull up to the latest commit for which there’s a substitute for
> IceCat.”

Ah, I understand now.

So the example you provided is a user-defined policy to install the
latest version of Guix that is downloadable using substitutes (if guix
publish has published those already).

As you say, in a similar vein, the end user could for themselves define
a policy that searches for a commit containing a specific successful
build, or a set of specific successful builds.

>> Thinking this through in my head somewhat, I had the following thoughts:
>> - This procedure is invoked client side, where the channel is defined
>> - That means the git searching is done client side, on every invocation
>> of guix (I guess this might be cacheable?)
>
> On every invocation of ‘guix pull’ only.

That makes sense, and is way better than I feared :-)

>> I have no idea what the performance cost would be.  I guess you would
>> use "guix weather" to turn the set of requested packages into a manifest
>> which can then be checked with it.
>
> As I imagine it, the cost would be a few HTTP queries to the Cuirass
> API.  I should try to come up with an example to better explain what I
> had in mind!

Your example helps visualize this, thanks.

Your example depends on there being a jobset that comprises the set of
packages you are interested in testing.

I imagine it is possible to do the same for an individual package / job.

The situation would be different if the end user wanted to perform a
similar operation for an arbitrary set of packages on their end.

It would probably involve something like this (probably naive):

(define (latest-commit-successfully-built-pkg pkg)
  "Return the latest commit for the pkg for which substitutes are
(potentially) available."
  ;;                  Like your version, but magically performs query
  ;;                  for pkg, not the guix-modular-master evaluation
  (let* ((evaluations (latest-evaluations pkg))
         (candidates  (filter-map (lambda (json)
                                    (match (hash-ref json "checkouts")
                                      ((checkout)
                                       (cons (hash-ref json "id")
                                             (hash-ref checkout "commit")))
                                      (_ #f)))
                                  evaluations)))
    (map (match-lambda
            ((evaluation . commit)
             (and (evaluation-complete? evaluation)
                  commit)))
          candidates)))

(any (match-lambda
        ((evaluation . commit) commit)
     (apply lset-intersection equal?
     ;;   Like latest-commit-successfully-built, but takes an
     ;;   individual package name for which we return the
     ;;   commit
     (map latest-commit-successfully-built-pkg
          %set-of-packages))))

Obviously the larger the set, the more requests are required, and the
lower the chance of a commit being available / a downgrade occuring

>> The question of security updates is tricky at the moment already — I
>> would hazard a guess that many people bail out of upgrading when they
>> can't get substitutes for their entire profile / system right now, which
>> means they are not getting security upgrades for package (a) when a
>> substitute for (b) fails.
>
> That’s probably true, and I agree it’s problematic.
>
> What I typically do is “guix pull && guix package -n -u”.  Then I look
> at things that would be built; if, say, LibreOffice is among them, I
> wait for a little while and try again later, until I can get enough
> substitutes.  That usually works okay, but it fails if it turns out that
> one of the dependencies fails to build: substitutes never become
> available in that case.

Interesting.  Do you think this kind of thing might be useful to have in
the Guix manual? Like, in a section about a "typical" desktop end-user
might manage their system day to day?

Alex

Message #44 received at 32022 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Alex Sassmannshausen <alex <at> pompo.co>
Cc: 26608 <at> debbugs.gnu.org, Konrad Hinsen <konrad.hinsen <at> fastmail.net>,
 22629 <at> debbugs.gnu.org, 32022 <at> debbugs.gnu.org
Subject: Re: bug#22629: “Stable” branch
Date: Tue, 04 Sep 2018 14:22:35 +0200

Hi Alex,

Alex Sassmannshausen <alex <at> pompo.co> skribis:

> So the example you provided is a user-defined policy to install the
> latest version of Guix that is downloadable using substitutes (if guix
> publish has published those already).
>
> As you say, in a similar vein, the end user could for themselves define
> a policy that searches for a commit containing a specific successful
> build, or a set of specific successful builds.

Exactly.

>> As I imagine it, the cost would be a few HTTP queries to the Cuirass
>> API.  I should try to come up with an example to better explain what I
>> had in mind!
>
> Your example helps visualize this, thanks.
>
> Your example depends on there being a jobset that comprises the set of
> packages you are interested in testing.

Yes, and it’s hacky in that the substitute server and jobset names are
hard-coded, but you get the idea.

> I imagine it is possible to do the same for an individual package / job.

Yes.

> The situation would be different if the end user wanted to perform a
> similar operation for an arbitrary set of packages on their end.

It would be quite similar: you would query the set of builds of an
evaluation of the “guix-modular” jobset and check whether the packages
of interest were built.

>> What I typically do is “guix pull && guix package -n -u”.  Then I look
>> at things that would be built; if, say, LibreOffice is among them, I
>> wait for a little while and try again later, until I can get enough
>> substitutes.  That usually works okay, but it fails if it turns out that
>> one of the dependencies fails to build: substitutes never become
>> available in that case.
>
> Interesting.  Do you think this kind of thing might be useful to have in
> the Guix manual? Like, in a section about a "typical" desktop end-user
> might manage their system day to day?

It would make sense to have such a section I guess.  However, before
teaching users how to work around deficiencies of our infrastructure our
processes ;-), I’d like us to improve them much as possible.  I’m sure
we have room for improvement for instance in Cuirass.

Thanks,
Ludo’.

Message #49 received at 32022-done <at> debbugs.gnu.org (full text, mbox):

From: swedebugia <at> riseup.net
To: ludo <at> gnu.org
Cc: 32022-done <at> debbugs.gnu.org
Subject: bug#32022: (new feature) Change guix pull to choose commits for which
 substitutes is already built by default
Date: Mon, 17 Dec 2018 02:53:29 -0800

Closed as we now have channels and ludo' had good arguments for why to
not implement my suggestion.

Message #56 received at 32022 <at> debbugs.gnu.org (full text, mbox):

From: Nome Grey <greynome72 <at> gmail.com>
To: 32022 <at> debbugs.gnu.org
Subject: Adding link to closed bug linked from reddit
Date: Thu, 5 Dec 2019 12:30:47 -0800

[Message part 1 (text/plain, inline)]

This bug is linked from a reddit discussion at
https://www.reddit.com/r/GUIX/comments/9un20j/question_about_guix_pull
mentioning ludo's solution.

If anybody finds this needing it like I did, I've tried to upgrade his code
to work on a modern guix at
https://github.com/nomr72/guix-substitutes-channel .  Changes in guile-json
stopped the old code from working.

I'm very new to guile, but it's working now.

Once the build infrastructure is developed enough to provide substitutes
very reliably, this will be unneeded.

[Message part 2 (text/html, inline)]

Message #59 received at 32022 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Nome Grey <greynome72 <at> gmail.com>
Cc: 26608 <at> debbugs.gnu.org, 22629 <at> debbugs.gnu.org, 32022 <at> debbugs.gnu.org
Subject: Re: bug#26608: channels.scm supporting substitutes
Date: Tue, 10 Dec 2019 17:41:54 +0100

Hi,

Nome Grey <greynome72 <at> gmail.com> skribis:

> Ludovic posted some channels.scm code in September 2018 supporting using
> more substitutes.  Unfortunately his code no longer functions due to an
> upgrade of guile-json in guix.
>
> I've tried to learn enough guile to upgrade the code to the newer json
> structures, and posted my changes to github at
> https://github.com/nomr72/guix-substitutes-channel .  It seems to run again
> now.

Nice, thanks for sharing!

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.