GNU bug report logs - #29674
Ceph creates Btrfs subvolumes on Btrfs during tests.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 29674 in the body.
You can then email your comments to 29674 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Rutger Helling <rhelling <at> mykolab.com>
To: bug-guix <at> gnu.org
Subject: Ceph creates Btrfs subvolumes on Btrfs during tests.
Date: Tue, 12 Dec 2017 11:28:41 +0100

Hey Guix,

I was surprised to learn that Ceph creates Btrfs subvolumes during its 
tests.
This is problematic because on Btrfs regular users can create 
subvolumes, but they cannot delete them.
This means I had to manually delete the following subvolumes as root:

ID 1744 gen 819598 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/current
ID 1745 gen 819598 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/snap_1
ID 1746 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/test_subvol
ID 1747 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/sync_snap_test
ID 1748 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/async_snap_test
ID 1749 gen 819598 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd1/current
ID 1750 gen 819598 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd1/snap_1
ID 1751 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd1/test_subvol
ID 1752 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd1/sync_snap_test
ID 1753 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd1/async_snap_test
ID 1754 gen 819598 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd2/current
ID 1755 gen 819598 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd2/snap_1
ID 1756 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd2/test_subvol
ID 1757 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd2/sync_snap_test
ID 1758 gen 819564 top level 257 path 
tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd2/async_snap_test

Message #8 received at 29674 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Rutger Helling <rhelling <at> mykolab.com>
Cc: 29674 <at> debbugs.gnu.org
Subject: Re: bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.
Date: Tue, 12 Dec 2017 13:52:32 +0100

Hi Rutger,

Rutger Helling <rhelling <at> mykolab.com> skribis:

> I was surprised to learn that Ceph creates Btrfs subvolumes during its
> tests.
> This is problematic because on Btrfs regular users can create
> subvolumes, but they cannot delete them.
> This means I had to manually delete the following subvolumes as root:
>
> ID 1744 gen 819598 top level 257 path
> tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/current
> ID 1745 gen 819598 top level 257 path
> tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/snap_1

Does that mean that wiping out /tmp/guix-build-ceph-*, as happens when
build finishes, isn’t enough to get rid of those volumes?

Ludo’.

Message #11 received at 29674 <at> debbugs.gnu.org (full text, mbox):

From: Rutger Helling <rhelling <at> mykolab.com>
To: ludo <at> gnu.org
Cc: 29674 <at> debbugs.gnu.org
Subject: Re: bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.
Date: Tue, 12 Dec 2017 14:10:16 +0100

Hi Ludo,

I'll try building again later (could be a while, Ceph takes a really 
long time to build), but I highly doubt that that's the case.
Btrfs subvolumes cannot be deleted with tools such as rm/rmdir, even as 
root. You have to run 'btrfs subvolume delete /subvolume' as root.
Although a regular user may create subvolumes, they cannot delete it 
(unless the USER_SUBVOL_RM_ALLOWED mount option is on).
Since the building is done as a guixbuilder user, I don't see how that 
would work.
And even if it did, if a build failed for example you'd still be left 
with a bunch of subvolumes in /tmp that can only be deleted as root.

On 2017-12-12 13:52, ludo <at> gnu.org wrote:

> Hi Rutger,
> 
> Rutger Helling <rhelling <at> mykolab.com> skribis:
> 
>> I was surprised to learn that Ceph creates Btrfs subvolumes during its
>> tests.
>> This is problematic because on Btrfs regular users can create
>> subvolumes, but they cannot delete them.
>> This means I had to manually delete the following subvolumes as root:
>> 
>> ID 1744 gen 819598 top level 257 path
>> tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/current
>> ID 1745 gen 819598 top level 257 path
>> tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/snap_1
> 
> Does that mean that wiping out /tmp/guix-build-ceph-*, as happens when
> build finishes, isn't enough to get rid of those volumes?
> 
> Ludo'.

Message #14 received at 29674 <at> debbugs.gnu.org (full text, mbox):

From: Rutger Helling <rhelling <at> mykolab.com>
To: ludo <at> gnu.org
Cc: 29674 <at> debbugs.gnu.org
Subject: Re: bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.
Date: Tue, 12 Dec 2017 14:23:56 +0100

Actually, I just tested it with the following:

$ cd /tmp
$ sudo -s -u guixbuilder01
bash-4.4$ guix environment -C --ad-hoc btrfs-progs
guixbuilder01 <at> guixsd /tmp [env]#  btrfs subvol create test-snapshot
Create subvolume './test-snapshot'
guixbuilder01 <at> guixsd /tmp [env]#  btrfs subvol del test-snapshot
Delete subvolume (no-commit): '/tmp/test-snapshot'
ERROR: cannot delete '/tmp/test-snapshot': Operation not permitted
guixbuilder01 <at> guixsd /tmp [env]# exit
exit
bash-4.4$ guix environment -C --ad-hoc coreutils
guixbuilder01 <at> guixsd /tmp [env]#  rm -rfv test-snapshot/
rm: cannot remove 'test-snapshot/': Operation not permitted
guixbuilder01 <at> guixsd /tmp [env]#  rmdir  -v test-snapshot/
rmdir: removing directory, 'test-snapshot/'
rmdir: failed to remove 'test-snapshot/': Operation not permitted
guixbuilder01 <at> guixsd /tmp [env]#

On 2017-12-12 14:10, Rutger Helling wrote:

> Hi Ludo,
> 
> I'll try building again later (could be a while, Ceph takes a really 
> long time to build), but I highly doubt that that's the case.
> Btrfs subvolumes cannot be deleted with tools such as rm/rmdir, even as 
> root. You have to run 'btrfs subvolume delete /subvolume' as root.
> Although a regular user may create subvolumes, they cannot delete it 
> (unless the USER_SUBVOL_RM_ALLOWED mount option is on).
> Since the building is done as a guixbuilder user, I don't see how that 
> would work.
> And even if it did, if a build failed for example you'd still be left 
> with a bunch of subvolumes in /tmp that can only be deleted as root.
> 
> On 2017-12-12 13:52, ludo <at> gnu.org wrote:
> 
> Hi Rutger,
> 
> Rutger Helling <rhelling <at> mykolab.com> skribis:
> 
> I was surprised to learn that Ceph creates Btrfs subvolumes during its
> tests.
> This is problematic because on Btrfs regular users can create
> subvolumes, but they cannot delete them.
> This means I had to manually delete the following subvolumes as root:
> 
> ID 1744 gen 819598 top level 257 path
> tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/current
> ID 1745 gen 819598 top level 257 path
> tmp/guix-build-ceph-12.0.2.drv-0/td/t-7201/dev/osd0/snap_1
> Does that mean that wiping out /tmp/guix-build-ceph-*, as happens when
> build finishes, isn't enough to get rid of those volumes?
> 
> Ludo'.

Message #17 received at 29674 <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Rutger Helling <rhelling <at> mykolab.com>, 29674 <at> debbugs.gnu.org
Subject: Re: bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.
Date: Tue, 12 Dec 2017 16:58:11 +0100

[Message part 1 (text/plain, inline)]

Rutger Helling <rhelling <at> mykolab.com> writes:

> Hey Guix,
>
> I was surprised to learn that Ceph creates Btrfs subvolumes during its 
> tests.
> This is problematic because on Btrfs regular users can create 
> subvolumes, but they cannot delete them.

Ugh.  My build machines do not use Btrfs so I hadn't noticed.  Will look
into disabling these tests.

Actually I'm tempted to just disable tests, since they are extremely
I/O intensive and some are flaky (about a 50% success rate on Hydra).
I've also been unable to update to the latest versions due to test suite
changes.

Maybe a system test is a decent compromise, although that will require
writing service definitions for the various daemons.  It's probably a
better use of our time than fighting the comprehensive test suite.

Do you intend to use Ceph on Guix, or is it for a dependent package?

[signature.asc (application/pgp-signature, inline)]

Message #20 received at 29674 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Rutger Helling <rhelling <at> mykolab.com>
Cc: 29674 <at> debbugs.gnu.org
Subject: Re: bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.
Date: Tue, 12 Dec 2017 17:03:19 +0100

Rutger Helling <rhelling <at> mykolab.com> skribis:

> Actually, I just tested it with the following:
>
> $ cd /tmp
> $ sudo -s -u guixbuilder01
> bash-4.4$ guix environment -C --ad-hoc btrfs-progs
> guixbuilder01 <at> guixsd /tmp [env]#  btrfs subvol create test-snapshot
> Create subvolume './test-snapshot'
> guixbuilder01 <at> guixsd /tmp [env]#  btrfs subvol del test-snapshot
> Delete subvolume (no-commit): '/tmp/test-snapshot'
> ERROR: cannot delete '/tmp/test-snapshot': Operation not permitted
> guixbuilder01 <at> guixsd /tmp [env]# exit
> exit
> bash-4.4$ guix environment -C --ad-hoc coreutils
> guixbuilder01 <at> guixsd /tmp [env]#  rm -rfv test-snapshot/
> rm: cannot remove 'test-snapshot/': Operation not permitted
> guixbuilder01 <at> guixsd /tmp [env]#  rmdir  -v test-snapshot/
> rmdir: removing directory, 'test-snapshot/'
> rmdir: failed to remove 'test-snapshot/': Operation not permitted

So does guix-daemon systematically leave /tmp/guix-build-ceph* behind it?

Ludo’.

Message #23 received at 29674 <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: ludo <at> gnu.org, rhelling <at> mykolab.com
Cc: 29674 <at> debbugs.gnu.org
Subject: Re: bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.
Date: Tue, 12 Dec 2017 17:13:25 +0100

Ludovic Courtès wrote on 12/12/17 at 17:03:
> So does guix-daemon systematically leave /tmp/guix-build-ceph* behind it?

Almost certainly. I can confirm several hundred previously unknown
subvolumes filling up my btrfs substitute server. Time to clean up.

Kind regards,

T G-R

Message #26 received at 29674 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Tobias Geerinckx-Rice <me <at> tobias.gr>
Cc: 29674 <at> debbugs.gnu.org, rhelling <at> mykolab.com
Subject: Re: bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.
Date: Tue, 12 Dec 2017 21:19:07 +0100

Tobias Geerinckx-Rice <me <at> tobias.gr> skribis:

> Ludovic Courtès wrote on 12/12/17 at 17:03:
>> So does guix-daemon systematically leave /tmp/guix-build-ceph* behind it?
>
> Almost certainly. I can confirm several hundred previously unknown
> subvolumes filling up my btrfs substitute server. Time to clean up.

Ouch.  I find this Btrfs behavior highly questionable.

Let’s make sure our packages don’t leak Btrfs subvolumes then!

Ludo’.

Message #29 received at 29674 <at> debbugs.gnu.org (full text, mbox):

From: Rutger Helling <rhelling <at> mykolab.com>
To: ludo <at> gnu.org
Cc: 29674 <at> debbugs.gnu.org, mbakke <at> fastmail.com,
 Tobias Geerinckx-Rice <me <at> tobias.gr>
Subject: Re: bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.
Date: Wed, 13 Dec 2017 09:44:50 +0100

Hey everyone,

Thanks for all the replies.

@Marius: It's a dependency for multipath-tools.

@Ludo: Like Tobias mentioned it does indeed systematically leave behind 
subvolumes that can only be manually deleted as root.
Thankfully this is the only package in which I've noticed this behavior.

I've disabled the tests entirely myself. This allowed me to build 
multipath-tools without any problems.
Maybe it's better to do that for now to make sure Btrfs users don't run 
into trouble and later disable the individual offending tests?

On 2017-12-12 21:19, ludo <at> gnu.org wrote:

> Tobias Geerinckx-Rice <me <at> tobias.gr> skribis:
> 
> Ludovic Courtès wrote on 12/12/17 at 17:03: So does guix-daemon 
> systematically leave /tmp/guix-build-ceph* behind it?
> Almost certainly. I can confirm several hundred previously unknown
> subvolumes filling up my btrfs substitute server. Time to clean up.

Ouch.  I find this Btrfs behavior highly questionable.

Let's make sure our packages don't leak Btrfs subvolumes then!

Ludo'.

Message #32 received at 29674 <at> debbugs.gnu.org (full text, mbox):

From: Rutger Helling <rhelling <at> mykolab.com>
To: 29674 <at> debbugs.gnu.org
Subject: Re: bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.
Date: Wed, 27 Dec 2017 23:50:39 +0100

[Message part 1 (text/plain, inline)]

This bug has been open for quite a while now.
If there are no objections in the next few days, I'd like to disable the
tests entirely for now with a FIXME on finding and disabling only the
offending tests.

On Tue, 12 Dec 2017 21:19:07 +0100
ludo <at> gnu.org (Ludovic Courtès) wrote:

> Tobias Geerinckx-Rice <me <at> tobias.gr> skribis:
> 
> > Ludovic Courtès wrote on 12/12/17 at 17:03:  
> >> So does guix-daemon systematically leave /tmp/guix-build-ceph*
> >> behind it?  
> >
> > Almost certainly. I can confirm several hundred previously unknown
> > subvolumes filling up my btrfs substitute server. Time to clean
> > up.  
> 
> Ouch.  I find this Btrfs behavior highly questionable.
> 
> Let’s make sure our packages don’t leak Btrfs subvolumes then!
> 
> Ludo’.

[Message part 2 (application/pgp-signature, inline)]

Message #35 received at 29674 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Rutger Helling <rhelling <at> mykolab.com>
Cc: 29674 <at> debbugs.gnu.org
Subject: Re: bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.
Date: Wed, 27 Dec 2017 18:07:40 -0500

[Message part 1 (text/plain, inline)]

On Wed, Dec 27, 2017 at 11:50:39PM +0100, Rutger Helling wrote:
> This bug has been open for quite a while now.
> If there are no objections in the next few days, I'd like to disable the
> tests entirely for now with a FIXME on finding and disabling only the
> offending tests.

That sounds fine for now.

> 
> On Tue, 12 Dec 2017 21:19:07 +0100
> ludo <at> gnu.org (Ludovic Courtès) wrote:
> 
> > Tobias Geerinckx-Rice <me <at> tobias.gr> skribis:
> > 
> > > Ludovic Courtès wrote on 12/12/17 at 17:03:  
> > >> So does guix-daemon systematically leave /tmp/guix-build-ceph*
> > >> behind it?  
> > >
> > > Almost certainly. I can confirm several hundred previously unknown
> > > subvolumes filling up my btrfs substitute server. Time to clean
> > > up.  
> > 
> > Ouch.  I find this Btrfs behavior highly questionable.
> > 
> > Let’s make sure our packages don’t leak Btrfs subvolumes then!
> > 
> > Ludo’.
> 

[signature.asc (application/pgp-signature, inline)]

Message #38 received at 29674 <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: rhelling <at> mykolab.com, 29674 <at> debbugs.gnu.org
Subject: Re: bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.
Date: Thu, 28 Dec 2017 00:16:04 +0100

[Message part 1 (text/plain, inline)]

Rutger,

Rutger Helling wrote on 27/12/17 at 23:50:
> This bug has been open for quite a while now.
> If there are no objections in the next few days, I'd like to disable the
> tests entirely for now with a FIXME on finding and disabling only the
> offending tests.
No objections if it's in a few days.

Kind regards,

T G-R

[signature.asc (application/pgp-signature, attachment)]

Message #41 received at 29674 <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Rutger Helling <rhelling <at> mykolab.com>, 29674 <at> debbugs.gnu.org
Subject: Re: bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.
Date: Thu, 28 Dec 2017 01:04:02 +0100

[Message part 1 (text/plain, inline)]

Rutger Helling <rhelling <at> mykolab.com> writes:

> This bug has been open for quite a while now.
> If there are no objections in the next few days, I'd like to disable the
> tests entirely for now with a FIXME on finding and disabling only the
> offending tests.

I dug pretty deep into the Ceph test code a few days ago without
being able to pinpoint which test(s) are the culprit for this.  Btrfs
subvolume operations only occur in FileStore.cc and objectstoretool.py
IIRC, but attempts at disabling their tests was unfruitful.

So, disabling tests altogether sounds good to me.  Make sure to add a
link to this bug report with the FIXME.

Thanks!

[signature.asc (application/pgp-signature, inline)]

Message #44 received at 29674 <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: rhelling <at> mykolab.com, 29674 <at> debbugs.gnu.org
Subject: Re: bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.
Date: Thu, 28 Dec 2017 05:30:25 +0100

[Message part 1 (text/plain, inline)]

Tobias Geerinckx-Rice wrote on 28/12/17 at 00:16:
> No objections if it's in a few days.

Actually, the fix I was hacking on appears as dead an end as all the others.

No objections any day.

Kind regards,

T G-R

[signature.asc (application/pgp-signature, attachment)]

Message #47 received at 29674 <at> debbugs.gnu.org (full text, mbox):

From: Rutger Helling <rhelling <at> mykolab.com>
To: 29674 <at> debbugs.gnu.org
Subject: Re: bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.
Date: Thu, 28 Dec 2017 08:31:00 +0100

[Message part 1 (text/plain, inline)]

Thanks for all the replies. I've disabled the tests.

[Message part 2 (application/pgp-signature, inline)]

Message #52 received at 29674-done <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Rutger Helling <rhelling <at> mykolab.com>
Cc: 29674-done <at> debbugs.gnu.org
Subject: Re: bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.
Date: Mon, 08 Jan 2018 15:29:35 +0100

Rutger Helling <rhelling <at> mykolab.com> skribis:

> Thanks for all the replies. I've disabled the tests.

Thank you.  Remember to close the bug by emailing
NUMBER-done <at> debbugs.gnu.org as I did here.  :-)

Ludo’.

Message #55 received at 29674-done <at> debbugs.gnu.org (full text, mbox):

From: Rutger Helling <rhelling <at> mykolab.com>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 29674-done <at> debbugs.gnu.org
Subject: Re: bug#29674: Ceph creates Btrfs subvolumes on Btrfs during tests.
Date: Mon, 8 Jan 2018 16:38:59 +0100

[Message part 1 (text/plain, inline)]

I left it open because I was under the impression that some people were
still working on a fix for the tests. But since there doesn't seem to be
a short-term solution it's probably better to close it. There's a TODO
comment now anyway.

On Mon, 08 Jan 2018 15:29:35 +0100
ludo <at> gnu.org (Ludovic Courtès) wrote:

> Rutger Helling <rhelling <at> mykolab.com> skribis:
> 
> > Thanks for all the replies. I've disabled the tests.  
> 
> Thank you.  Remember to close the bug by emailing
> NUMBER-done <at> debbugs.gnu.org as I did here.  :-)
> 
> Ludo’.

[Message part 2 (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.