GNU bug report logs - #42883
Selecting a subset of tests in recursive make causes unnecessary failures

Previous Next

Package: automake;

Reported by: development <at> efficientek.com

Date: Sun, 16 Aug 2020 06:24:01 UTC

Severity: normal

Tags: confirmed, help

To reply to this bug, email your comments to 42883 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-automake <at> gnu.org:
bug#42883; Package automake. (Sun, 16 Aug 2020 06:24:01 GMT) Full text and rfc822 format available.

Acknowledgement sent to development <at> efficientek.com:
New bug report received and forwarded. Copy sent to bug-automake <at> gnu.org. (Sun, 16 Aug 2020 06:24:01 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Glenn Washburn <development <at> efficientek.com>
To: bug-automake <at> gnu.org
Subject: Selecting a subset of tests in recursive make causes unnecessary
 failures
Date: Sun, 16 Aug 2020 00:44:41 -0500
[Message part 1 (text/plain, inline)]
Hi,

I've been working on some automake tests for grub, and ran in to a bug.
I noticed that when specifying a subset of tests to run via the TESTS
environment variable, I could have all tests complete successfully and
yet still have make return an error.

Here's a sequence of commands that should reproduce the error:
   git clone https://git.savannah.gnu.org/git/grub.git
   cd grub
   ./bootstrap
   ./configure
   make
   env TESTS=grub_cmd_echo make -e check VERBOSE=yes

It turns out that make check is being recursively run in the grub-core
directory, as expected.  The TESTS environment variable is the same in
both invocations, also as expected.  However, the test program is not in
the grub-core subdirectory.  So the test is run successfully in the root
make, and not found in the subdirectory, thus generating an error. In
fact, grub has no tests in the grub-core subdirectory, but the harness
is run because its part of automake, even though I do not need or want
it to.

I've attached a patch which fixes this issue by ignoring tests which do
not exist. I'm not sure this patch takes care of everything needed, for
instance EXEEXT, to check for the existence of the test program. Grub
does not use any TEST_EXTENSIONS, so this works.

This will change make it so TESTS which should exist, but do not, will
not cause an error, whereas currently it will.  I'm not sure how this
situation would arise, because make should require the existence as a
dependency.  If there is an error in the TESTS variable (eg. a typo in
a test name), this will not cause an error either, which I deem
undesirable. However, I think this is an acceptable trade-off.

Perhaps a better solution would be to not recursively run make check if
TESTS is defined and run all given tests from the root make. Tests
which are run in recursive make invocations when TESTS is not defined
can be run via TESTS by specifying a path component (eg.
TESTS="grub-core/test-program").  This could be a problem if test
programs assume they are run from the directory in which they reside
(in which case, we could chdir to the directory and run the program).

Any thoughts?
Glenn

[fix-recursive-subset-test.patch (text/x-patch, attachment)]
[grub-tests-error.txt (text/plain, attachment)]

Information forwarded to bug-automake <at> gnu.org:
bug#42883; Package automake. (Thu, 20 Aug 2020 01:18:02 GMT) Full text and rfc822 format available.

Message #8 received at 42883 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: development <at> efficientek.com
Cc: 42883 <at> debbugs.gnu.org
Subject: Re: bug#42883: Selecting a subset of tests in recursive make causes
 unnecessary failures
Date: Wed, 19 Aug 2020 19:17:13 -0600
Hi Glenn,

    I noticed that when specifying a subset of tests to run via the TESTS
    environment variable, I could have all tests complete successfully

Thanks for the report. That does seem suboptimal, but on the other hand,
subsets of tests are a complication. Does it work to override SUBDIRS
along with TESTS? As in (thanks to Jim):
  env TESTS=grub_cmd_echo make -e check VERBOSE=yes SUBDIRS=

Thanks,
Karl




Information forwarded to bug-automake <at> gnu.org:
bug#42883; Package automake. (Fri, 21 Aug 2020 08:08:01 GMT) Full text and rfc822 format available.

Message #11 received at 42883 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Washburn <development <at> efficientek.com>
To: Karl Berry <karl <at> freefriends.org>
Cc: development <at> efficientek.com, 42883 <at> debbugs.gnu.org
Subject: Re: bug#42883: Selecting a subset of tests in recursive make causes
 unnecessary failures
Date: Fri, 21 Aug 2020 03:07:21 -0500
Hi Karl,

On Wed, 19 Aug 2020 19:17:13 -0600
Karl Berry <karl <at> freefriends.org> wrote:

> Hi Glenn,
> 
>     I noticed that when specifying a subset of tests to run via the
> TESTS environment variable, I could have all tests complete
> successfully
> 
> Thanks for the report. That does seem suboptimal, but on the other
> hand, subsets of tests are a complication. Does it work to override
> SUBDIRS along with TESTS? As in (thanks to Jim):
>   env TESTS=grub_cmd_echo make -e check VERBOSE=yes SUBDIRS=

Yes this does work in the special case of grub, thanks for the
suggestion. I think at a minimum this behavior should be documented
with the test subsets documentation and this work around mentioned.
Ideally, the subset tests feature design should be updated to play nice
with makefile recursion.  Setting SUBDIRS= is just disabling the
makefile recusion.

If I had a test in grub-core/unittest1 and I wanted to run just the
test subset of unittest1 and grub_cmd_echo, how would I do that such
that make returns success?  If I run `TESTS="unittest1 grub_cmd_echo"
make -e check', make will fail even if both test individually succeed
because the recursive make check in grub-core will try to run
grub_cmd_echo (in addition to unittest1) and fail.

Another issue is that if there is a test X in the root makefile and a
test X in SUBDIR, how does one choose to run the X in SUBDIR and not the
X in root?  Right now TESTS=X will run both tests.  Sure one could do
`TESTS=X make -e -C SUBDIR check', but then again you're just bypassing
the makefile recursion.

And thanks Karl for helping to maintain this venerable and essential,
yet not super glamorous project.

Glenn





Information forwarded to bug-automake <at> gnu.org:
bug#42883; Package automake. (Fri, 21 Aug 2020 22:06:01 GMT) Full text and rfc822 format available.

Message #14 received at 42883 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: development <at> efficientek.com
Cc: 42883 <at> debbugs.gnu.org
Subject: Re: bug#42883: Selecting a subset of tests in recursive make causes
 unnecessary failures
Date: Fri, 21 Aug 2020 16:05:24 -0600
    at a minimum this behavior should be documented
    with the test subsets documentation and this work around mentioned.

Yes, I agree. Contemplating.

    Ideally, the subset tests feature design should be updated to play nice
    with makefile recursion.

I think I agree again, but ... not sure ...

    Another issue is that if there is a test X in the root makefile and a
    test X in SUBDIR

Isn't that fundamentally not supported, or supportable? I admit I don't
actually know if there's any history about it, but using the same name
for two tests in different places just seems like a basic source of
confusion. As far as I know, the TESTS mechanism just isn't set up to
be able to specify arbitrary test names in arbitrary directories.

Admittedly if it is not supported, it could at least be detected and
complained about. On the other hand, I suppose it works when not
explicitly specifying tests to run, which is 99% of the time. So, once
again ... I don't know. Nothing is ever simple :).

Jim, Paul, anyone? --thanks, karl.




Information forwarded to bug-automake <at> gnu.org:
bug#42883; Package automake. (Mon, 24 Aug 2020 15:55:02 GMT) Full text and rfc822 format available.

Message #17 received at 42883 <at> debbugs.gnu.org (full text, mbox):

From: Glenn Washburn <development <at> efficientek.com>
To: Karl Berry <karl <at> freefriends.org>
Cc: 42883 <at> debbugs.gnu.org
Subject: Re: bug#42883: Selecting a subset of tests in recursive make causes
 unnecessary failures
Date: Mon, 24 Aug 2020 10:54:48 -0500
On Fri, 21 Aug 2020 16:05:24 -0600
Karl Berry <karl <at> freefriends.org> wrote:

>> Another issue is that if there is a test X in the root makefile
>> and a test X in SUBDIR
> 
> Isn't that fundamentally not supported, or supportable? I admit I
> don't actually know if there's any history about it, but using the
> same name for two tests in different places just seems like a basic
> source of confusion. As far as I know, the TESTS mechanism just isn't
> set up to be able to specify arbitrary test names in arbitrary
> directories.

I'm not sure if I was sufficiently clear in my example. The example is
meant to be such that the test X in root and SUBDIR are tests that
would be run when not explicitly specifying tests.  So its to be able
to run any test that would be run when not explicitly specifying tests.

I think the way to support this is to have TESTS be a list of test
paths relative to the root makefile.  So in the example above, I would
specify TESTS="X SUBDIR/X".  Before running the recursive make in
SUBDIR, the TESTS variable would be filtered to only include tests with
a relative path of the sub directory.  Also, the root makefile would
not run tests that have a directory component to the path.

I would like to get a rough idea of if this would be an approved of
approach so that I or someone else could take a stab at implementing it
with the hopes of the changes being accepted.

Glenn




Information forwarded to bug-automake <at> gnu.org:
bug#42883; Package automake. (Tue, 25 Aug 2020 01:40:02 GMT) Full text and rfc822 format available.

Message #20 received at 42883 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: development <at> efficientek.com
Cc: 42883 <at> debbugs.gnu.org
Subject: Re: bug#42883: Selecting a subset of tests in recursive make causes
 unnecessary failures
Date: Mon, 24 Aug 2020 19:39:25 -0600
Hi Glenn,

    The example is meant to be such that the test X in root and SUBDIR
    are tests that would be run when not explicitly specifying tests.

Yep, I got that.

    So its to be able to run any test that would be run when not
    explicitly specifying tests.

Clearly yes, it would be desirable in principle to be able to name tests
anything you want, in any directory you want, and then be able to
specify exactly the individual tests you want to run, regardless of
names or (sub)directories. I'm just not sure how feasible it is to
implement.

    I think the way to support this is to have TESTS be a list of test
    paths relative to the root makefile.  

Isn't it already that way?

    So in the example above, I would
    specify TESTS="X SUBDIR/X".

To run a specific test in automake now, what I came up with is 
make check TESTS=t/sometest.sh

Although the manual says it can be overridden in the environment
(Parallel Test Harness node):
  env TESTS="foo.test bar.test" make -e check
this has never worked for me. (Neither foo.sh or t/foo.sh.) I have not
tried to discern what I was doing wrong. (I feel pretty sure it was me
doing something wrong and not Automake, at least.)

I realize the case you're concerned about is when both ./sometest.sh and
t/sometest.sh are present. My gut reaction is that it would be more
reliable to detect that case and complain about it than try to support
it. But ok, never mind, proceeding along the ideal lines ...

    Before running the recursive make in SUBDIR, the TESTS variable
    would be filtered to only include tests with a relative path of the
    sub directory.  Also, the root makefile would not run tests that
    have a directory component to the path.

It sounds sensible to me, but regrettably, I just don't have a great
overall grip on the code, having come to the implementation side of
Automake rather late. Jim or Paul would have better things to say.
Unfortunately I think they are busy with other things (which is why I'm
here in the first place ... anyway ...).

    I would like to get a rough idea of if this would be an approved of
    approach so that I or someone else could take a stab at implementing it
    with the hopes of the changes being accepted.

In general, the top priority for any change in Automake is to not break
existing practice. If you can devise a change that, for starters, still
passes all the existing Automake tests, then I surmise there's a good
shot.

As you may infer from my TESTS= stuff above, exactly what invariants
have to be maintained to keep compatibility in this case is not clear to
me :(.

(Also, as one would expect, the change has to be comprehensible and
maintainable, but that's nothing special to Automake.)

Regarding your previous patch, it actually seemed pretty plausible to me
(modulo EXEEXT and other such, as you noted; not a big deal), except
that it's a shame to lose the typo checking. There are over a thousand
tests in Automake, and no doubt that many or more in other packages. So
typos are inevitable. Seems like some sort of separate check for them
would need to be available, going down that road.

Thanks for following up. I hope you are sufficiently interested to spend
some more time on it. --best, karl.




Added tag(s) help and confirmed. Request was from Karl Berry <karl <at> freefriends.org> to control <at> debbugs.gnu.org. (Sun, 22 Nov 2020 02:10:03 GMT) Full text and rfc822 format available.

This bug report was last modified 3 years and 127 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.