GNU bug report logs - #49309
Feature Request: Automake script based tests to print the test name before running it

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 49309 in the body.
You can then email your comments to 49309 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Kasper k <kasperkantz <at> outlook.com>
To: "bug-automake <at> gnu.org" <bug-automake <at> gnu.org>
Subject: Feature Request: Automake script based tests to print the test name
 before running it
Date: Thu, 1 Jul 2021 03:01:30 +0000

Hello automake devs,

In script based testsuites (https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html#Scripts_002dbased-Testsuites), when we run `make check` it prints one of the following string for each test: PASS <testname>, FAIL <testname>, XFAIL <testname> or SKIP <testname>.

However, in some cases (long running test, flaky test, other inconclusive ones for more involved reasons), it is essential to know which test is currently in flight. Especially in CI systems, where flakiness every now and then, due to external factors, is almost unavoidable.

It would be very helpful if a new option AM_TESTS_NAME=1 is provided, which would change the stdout output from:

    <blank stdout and wait for test execution to complete>
     PASS (or FAIL/XFAIL/SKIP) $name

to something like:

    RUNNING $name
    <blank stdout and wait for test execution to complete>
     PASS (or FAIL/XFAIL/SKIP) $name

this way we can easily find out the name of test (or multiple of them; in case of parallel execution) which is (are) currently running.

Thanks,
/K

Message #8 received at 49309 <at> debbugs.gnu.org (full text, mbox):

From: Peter Johansson <trojkan <at> gmail.com>
To: Kasper k <kasperkantz <at> outlook.com>, 49309 <at> debbugs.gnu.org
Subject: Re: bug#49309: Feature Request: Automake script based tests to print
 the test name before running it
Date: Thu, 1 Jul 2021 23:30:02 +1000

Hi Kasper,

I leave to the maintainers to comment on the suggestion, but in the 
meantime...

On 1/7/21 1:01 pm, Kasper k wrote:
> Hello automake devs,
>
> In script based testsuites (https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html#Scripts_002dbased-Testsuites), when we run `make check` it prints one of the following string for each test: PASS <testname>, FAIL <testname>, XFAIL <testname> or SKIP <testname>.
>
> However, in some cases (long running test, flaky test, other inconclusive ones for more involved reasons), it is essential to know which test is currently in flight. Especially in CI systems, where flakiness every now and then, due to external factors, is almost unavoidable.
>
> It would be very helpful if a new option AM_TESTS_NAME=1 is provided, which would change the stdout output from:
>
>      <blank stdout and wait for test execution to complete>
>       PASS (or FAIL/XFAIL/SKIP) $name
>
> to something like:
>
>      RUNNING $name
>      <blank stdout and wait for test execution to complete>
>       PASS (or FAIL/XFAIL/SKIP) $name

Please note that that output comes from the test-driver script. If 
you're not happy with the default test-driver, you can declare to use 
your own my-test-driver (see 
https://www.gnu.org/software/automake/manual/automake.html#Declaring-Custom-Test-Drivers), 
and for example this could be a modified version of 'test-driver' script 
provided by Automake. Adding a line outputting 'Running $name' is 
trivial if you know any shell, or otherwise, I'm sure people here can help.

>
> this way we can easily find out the name of test (or multiple of them; in case of parallel execution) which is (are) currently running.

I usually find out which tests are running by looking at the timestamps 
of log and trace files with something like 'ls -lrt *.log *.trs'.

Cheers,

Peter

Message #11 received at 49309 <at> debbugs.gnu.org (full text, mbox):

From: Kasper k <kasperkantz <at> outlook.com>
To: Peter Johansson <trojkan <at> gmail.com>, "49309 <at> debbugs.gnu.org"
 <49309 <at> debbugs.gnu.org>
Subject: Re: bug#49309: Feature Request: Automake script based tests to print
 the test name before running it
Date: Thu, 1 Jul 2021 18:59:10 +0000

[Message part 1 (text/plain, inline)]

Hi Peter,

Thanks for your reply.

As to the workarounds, they are just that and I had coined my own for "my machine" ™. The standard solution for "display test name before starting it" (controlled by a new AC option) would be much more palatable for consumer to add to their Makefile.am and check into their source tree.  Then if some intermittent bad situation happens in unattended/disposable CI environment, as an example, then we won't be needing to opt into any kind of workaround, it'd be just there right in the "regular" make-check logs; without introducing crazy amount of verbosity (which set -x, VERBOSE=1 etc. renders).

/K
________________________________
From: Peter Johansson <trojkan <at> gmail.com>
Sent: Thursday, July 1, 2021 1:30 PM
To: Kasper k <kasperkantz <at> outlook.com>; 49309 <at> debbugs.gnu.org <49309 <at> debbugs.gnu.org>
Subject: Re: bug#49309: Feature Request: Automake script based tests to print the test name before running it

Hi Kasper,

I leave to the maintainers to comment on the suggestion, but in the
meantime...

On 1/7/21 1:01 pm, Kasper k wrote:
> Hello automake devs,
>
> In script based testsuites (https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html#Scripts_002dbased-Testsuites), when we run `make check` it prints one of the following string for each test: PASS <testname>, FAIL <testname>, XFAIL <testname> or SKIP <testname>.
>
> However, in some cases (long running test, flaky test, other inconclusive ones for more involved reasons), it is essential to know which test is currently in flight. Especially in CI systems, where flakiness every now and then, due to external factors, is almost unavoidable.
>
> It would be very helpful if a new option AM_TESTS_NAME=1 is provided, which would change the stdout output from:
>
>      <blank stdout and wait for test execution to complete>
>       PASS (or FAIL/XFAIL/SKIP) $name
>
> to something like:
>
>      RUNNING $name
>      <blank stdout and wait for test execution to complete>
>       PASS (or FAIL/XFAIL/SKIP) $name

Please note that that output comes from the test-driver script. If
you're not happy with the default test-driver, you can declare to use
your own my-test-driver (see
https://www.gnu.org/software/automake/manual/automake.html#Declaring-Custom-Test-Drivers),
and for example this could be a modified version of 'test-driver' script
provided by Automake. Adding a line outputting 'Running $name' is
trivial if you know any shell, or otherwise, I'm sure people here can help.

>
> this way we can easily find out the name of test (or multiple of them; in case of parallel execution) which is (are) currently running.

I usually find out which tests are running by looking at the timestamps
of log and trace files with something like 'ls -lrt *.log *.trs'.

Cheers,

Peter

[Message part 2 (text/html, inline)]

Message #14 received at 49309 <at> debbugs.gnu.org (full text, mbox):

From: Hans-Bernhard Bröker <HBBroeker <at> t-online.de>
To: Kasper k <kasperkantz <at> outlook.com>, Peter Johansson <trojkan <at> gmail.com>,
 "49309 <at> debbugs.gnu.org" <49309 <at> debbugs.gnu.org>
Subject: Re: bug#49309: Feature Request: Automake script based tests to print
 the test name before running it
Date: Fri, 2 Jul 2021 19:02:38 +0200

Am 01.07.2021 um 20:59 schrieb Kasper k:

> Then if some intermittent bad situation happens in 
> unattended/disposable CI environment, as an example, then we won't be 
> needing to opt into any kind of workaround, it'd be just there right in 
> the "regular" make-check logs; without introducing crazy amount of 
> verbosity (which set -x, VERBOSE=1 etc. renders).

IMHO that whole idea loses a most, if not all of its appeal when tests 
are run in parallel (which CI builds, always pressed for time, will 
rather likely want to do).

As soon as there is more than one line of output, you're back to where 
you were without this change, or worse.

* -Otarget or -Orecurse will undo the desired effect, because make will 
keep the entire output to itself until the test is done.  That's the 
back to square one part.

* -Oline or -Onone modes will print outputs from different tests 
intermixed, making it virtually impossible to know which result output 
belongs to which testcase.  That's considerably worse than single-line 
outputs.

Message #17 received at 49309 <at> debbugs.gnu.org (full text, mbox):

From: Kasper k <kasperkantz <at> outlook.com>
To: Hans-Bernhard Bröker <HBBroeker <at> t-online.de>, Peter
 Johansson <trojkan <at> gmail.com>, "49309 <at> debbugs.gnu.org"
 <49309 <at> debbugs.gnu.org>
Subject: Re: bug#49309: Feature Request: Automake script based tests to print
 the test name before running it
Date: Fri, 2 Jul 2021 21:05:39 +0000

[Message part 1 (text/plain, inline)]

Hi Hans,

Thanks for taking interest. But I am not sure how would it break the parallelism of tests or anything in the test infra. The ask here is to optionally print an additional extra line before running the test regardless of order or medium in which they will eventually get rendered.

AM script check supports these options:

  *   AM_TESTS_FD_REDIRECT
  *   AM_TESTS_ENVIRONMENT
  *   AM_TESTSUITE_SUMMARY_HEADER

they solve similar problems which are more useful in one case, but not the other one. With the proposed AM_TESTS_SHOW_NAME, you can have parallelism:

    # with AM_TESTS_SHOW_NAME=1
    #
    RUN Foo
    RUN Bar
    PASS Foo
    RUN Baz
    FAILED Baz
    PASS Bar

now as to your point for my specific example where it could be useful, I agree it will not solve that particular problem (of revealing the hanging tests) perfectly in all modes and neither I am looking for the requested option to do so (and neither do the other three supported options are meant to solve everything...)

/K

________________________________
From: Hans-Bernhard Bröker <HBBroeker <at> t-online.de>
Sent: Friday, July 2, 2021 5:02 PM
To: Kasper k <kasperkantz <at> outlook.com>; Peter Johansson <trojkan <at> gmail.com>; 49309 <at> debbugs.gnu.org <49309 <at> debbugs.gnu.org>
Subject: Re: bug#49309: Feature Request: Automake script based tests to print the test name before running it

Am 01.07.2021 um 20:59 schrieb Kasper k:

> Then if some intermittent bad situation happens in
> unattended/disposable CI environment, as an example, then we won't be
> needing to opt into any kind of workaround, it'd be just there right in
> the "regular" make-check logs; without introducing crazy amount of
> verbosity (which set -x, VERBOSE=1 etc. renders).

IMHO that whole idea loses a most, if not all of its appeal when tests
are run in parallel (which CI builds, always pressed for time, will
rather likely want to do).

As soon as there is more than one line of output, you're back to where
you were without this change, or worse.

* -Otarget or -Orecurse will undo the desired effect, because make will
keep the entire output to itself until the test is done.  That's the
back to square one part.

* -Oline or -Onone modes will print outputs from different tests
intermixed, making it virtually impossible to know which result output
belongs to which testcase.  That's considerably worse than single-line
outputs.

[Message part 2 (text/html, inline)]

Message #20 received at 49309 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: kasperkantz <at> outlook.com
Cc: HBBroeker <at> t-online.de, trojkan <at> gmail.com, 49309 <at> debbugs.gnu.org
Subject: Re: bug#49309: Feature Request: Automake script based tests to print
 the test name before running it
Date: Sat, 3 Jul 2021 16:58:47 -0600

      *   AM_TESTS_FD_REDIRECT
      *   AM_TESTS_ENVIRONMENT
      *   AM_TESTSUITE_SUMMARY_HEADER

(Aside: I see now I should have made that name
AM_TESTS_SUMMARY_HEADER. Oh well, too late now.)

    # with AM_TESTS_SHOW_NAME=1

I guess I have nothing against it, for the reasons you give. Can you (or
anyone) provide a patch? --thanks, karl.

Message #23 received at 49309 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: kasperkantz <at> outlook.com
Cc: 49309 <at> debbugs.gnu.org
Subject: Re: bug#49309: Feature Request: Automake script based tests to print
 the test name before running it
Date: Sun, 26 Sep 2021 19:40:33 -0600

Back on this suggestion ...

    It would be very helpful if a new option AM_TESTS_NAME=1 
    ...
        RUNNING $name

Looking into it now, I believe this can be done already using
AM_TESTS_ENVIRONMENT (or by an individual user with
TESTS_ENVIRONMENT). The test name is the shell variable $f at that
point. Thus:

  make AM_TESTS_ENVIRONMENT='echo RUNNING: "$$f";' check 

seems to give the desired output for me.  Although the use of $f here is
not documented, in practice it is never going to change.

Seems way simpler than adding this to Automake ... --best, karl.

Message #26 received at 49309 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: 49309 <at> debbugs.gnu.org
Subject: Re: bug#49309: Feature Request: Automake script based tests to print
 the test name before running it
Date: Tue, 28 Sep 2021 19:28:13 -0600

  make AM_TESTS_ENVIRONMENT='echo RUNNING: "$$f";' check 

I added this to the manual as another example of using
[AM_]TESTS_ENVIRONMENT.  Closing the bug ... --thanks, karl.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.