GNU bug report logs - #33378
offload: Avoid hosts with little or no free disk space

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 33378 in the body.
You can then email your comments to 33378 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Bug Guix <bug-guix <at> gnu.org>
Subject: offload: Avoid hosts with little or no free disk space
Date: Wed, 14 Nov 2018 14:50:49 +0100

‘choose-build-machine’ in (guix scripts offload) could/should exclude
build machines that don’t have at least, say, 100 MiB available in their
store.  That would avoid ENOSPC build failures such as
<https://berlin.guixsd.org/log/0hvhrfdjwnmdm2rdf58knshbbikm4ryv-r-bsgenome-hsapiens-ucsc-hg19-masked-1.3.99>.

Thanks,
Ludo’.

Message #10 received at 33378-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 33378-done <at> debbugs.gnu.org
Subject: Re: bug#33378: offload: Avoid hosts with little or no free disk space
Date: Fri, 21 Dec 2018 23:51:35 +0100

ludo <at> gnu.org (Ludovic Courtès) skribis:

> ‘choose-build-machine’ in (guix scripts offload) could/should exclude
> build machines that don’t have at least, say, 100 MiB available in their
> store.  That would avoid ENOSPC build failures such as
> <https://berlin.guixsd.org/log/0hvhrfdjwnmdm2rdf58knshbbikm4ryv-r-bsgenome-hsapiens-ucsc-hg19-masked-1.3.99>.

Done in commit 63b0c3eaccdf1816b419632cd7fe721934d2eb27.

Ludo'.

Message #13 received at 33378 <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: 33378 <at> debbugs.gnu.org
Cc: ludo <at> gnu.org
Subject: Re: bug#33378: offload: Avoid hosts with little or no free disk space
Date: Sun, 23 Dec 2018 12:20:17 -0500

Hi Ludovic,

Ludovic Courtès <ludo <at> gnu.org> writes:

> ludo <at> gnu.org (Ludovic Courtès) skribis:
>
>> ‘choose-build-machine’ in (guix scripts offload) could/should exclude
>> build machines that don’t have at least, say, 100 MiB available in their
>> store.  That would avoid ENOSPC build failures such as
>> <https://berlin.guixsd.org/log/0hvhrfdjwnmdm2rdf58knshbbikm4ryv-r-bsgenome-hsapiens-ucsc-hg19-masked-1.3.99>.
>
> Done in commit 63b0c3eaccdf1816b419632cd7fe721934d2eb27.

Many builds require more free space than 100 MB, so this is only a
partial solution.  Also, if multiple builds are happening on the same
machine, the amount of free space available for each build will be less
than 100 MB.

Finally, the other related commit that attempts to detect build failures
caused by ENOSPC by checking if there's at least 10 MB free after the
failure won't often work in practice, I suspect.  The problem is, by the
time you perform that check, the build directory has already been
deleted.

       Mark

Message #16 received at 33378 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Mark H Weaver <mhw <at> netris.org>
Cc: 33378 <at> debbugs.gnu.org
Subject: Re: bug#33378: offload: Avoid hosts with little or no free disk space
Date: Sun, 23 Dec 2018 21:58:49 +0100

Hi Mark,

Mark H Weaver <mhw <at> netris.org> skribis:

> Ludovic Courtès <ludo <at> gnu.org> writes:
>
>> ludo <at> gnu.org (Ludovic Courtès) skribis:
>>
>>> ‘choose-build-machine’ in (guix scripts offload) could/should exclude
>>> build machines that don’t have at least, say, 100 MiB available in their
>>> store.  That would avoid ENOSPC build failures such as
>>> <https://berlin.guixsd.org/log/0hvhrfdjwnmdm2rdf58knshbbikm4ryv-r-bsgenome-hsapiens-ucsc-hg19-masked-1.3.99>.
>>
>> Done in commit 63b0c3eaccdf1816b419632cd7fe721934d2eb27.
>
> Many builds require more free space than 100 MB, so this is only a
> partial solution.  Also, if multiple builds are happening on the same
> machine, the amount of free space available for each build will be less
> than 100 MB.

You’re right, but this at least avoids the most obvious problems.

> Finally, the other related commit that attempts to detect build failures
> caused by ENOSPC by checking if there's at least 10 MB free after the
> failure won't often work in practice, I suspect.  The problem is, by the
> time you perform that check, the build directory has already been
> deleted.

Correct.  (I wasn’t sure the build directory would already be deleted at
that point but rereading build.cc confirms this.)

I suppose the only way to solve this correctly would be to keep the
build tree and then delete it from (guix scripts offload).  WDYT?

Thanks for your feedback!

Ludo’.

Message #19 received at 33378 <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 33378 <at> debbugs.gnu.org
Subject: Re: bug#33378: offload: Avoid hosts with little or no free disk space
Date: Mon, 24 Dec 2018 13:45:08 -0500

Hi Ludovic,

Ludovic Courtès <ludo <at> gnu.org> writes:
> I suppose the only way to solve this correctly would be to keep the
> build tree and then delete it from (guix scripts offload).  WDYT?

Generally, this sounds like a reasonable approach.  However, one issue
is that it will likely lead to build directories being left around
occasionally when transient network outages occur.  Those would need to
be cleaned up manually, or by cron jobs on the build slaves.

Also, does the daemon API provide a way to find the name of the build
directory?  What if there's more than one build directory with the same
name and version number, differing only by the numeric suffix?

Ideally, there would be a way for the build-side code to return a result
code indicating that the build "aborted" rather than "failed", for this
to be recognized by the daemon as a transient failure (i.e. not cached),
and to propagate this information back to the client who requested the
build.  Do the relevant daemon APIs and offloading protocols allow for
this information to be propagated to the client?

       Thanks,
         Mark

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.