GNU bug report logs - #70638
Automake custom-driver interface documentation doesn't match implementation

Previous Next

To reply to this bug, email your comments to 70638 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Mark Wooding <mdw <at> distorted.org.uk>
To: bug-automake <at> gnu.org
Subject: Automake custom-driver interface documentation doesn't match
 implementation
Date: Mon, 29 Apr 2024 00:04:42 +0100

This is sort of a follow-up on #22445; alas, nobody involved with that
bug noticed the actual scale of the problem.  I'm sorry to be the bearer
of bad news.

In essence, #22445 observed that the usage message printed by
`test-driver' showed option arguments attached to their options with an
`=' sign, while the program itself required the argument in the next
argument word.  The bug was resolved by fixing the usage message to
match the program's behaviour.

The command-line syntax accepted by `test-driver' is not idiosyncratic.
Rather, it's part of a defined protocol, also implemented by
`tap-driver.sh', and `check.am' and `check2.am', and defined in the
manual at (automake)Command-line arguments for test drivers.  The usage
message for `tap-driver.sh' and the manual agree with the old usage for
`test-driver' that option arguments should be attached to their options
with `='.  However, the implementation of `tap-driver.sh' uses code
almost identical to `test-driver', and requires option arguments to be
passed in separate words:

        [gibson ~/src/ext/automake (master)]git describe
        v1.16i-41-gefb9f7b29
        [gibson ~/src/ext/automake (master)]lib/tap-driver.sh --test-name=splat
        tap-driver.sh: invalid option: '--test-name=splat'
        Usage:
          tap-driver.sh --test-name=NAME --log-file=PATH --trs-file=PATH
        ...

And the code to invoke test drivers, in `check2.am', assumes that this
is acceptable:

        ?GENERIC?       $(am__set_b); \
                $(am__check_pre) %DRIVER% --test-name "$$f" \
                --log-file $$b.log --trs-file $$b.trs \
                $(am__common_driver_flags) %DRIVER_FLAGS% -- %COMPILE% \
                "$$tst" $(AM_TESTS_FD_REDIRECT)

The default setting of `am__common_driver_flags' in `check.am' concurs,
unsurprisingly:

        # Default flags passed to test drivers.
        am__common_driver_flags = \
          --color-tests "$$am__color_tests" \
          --enable-hard-errors "$$am__enable_hard_errors" \
          --expect-failure "$$am__expect_failure"

But all of this is contrary to the manual:

        A custom driver can rely on various command-line options and
        arguments being passed to it automatically by the
        Automake-generated test harness.  It is _mandatory_ that it
        understands all of them (even if the exact interpretation of the
        associated semantics can legitimately change between a test
        driver and another, and even be a no-op in some drivers).

        Here is the list of options:

        `--test-name=NAME'
             The name of the test, with VPATH prefix (if any) removed.
             This can have a suffix and a directory component (as in
             e.g., `sub/foo.test'), and is mostly meant to be used in
             console reports about testsuite advancements and results
             (see Testsuite progress output).
        ...

It seems that the documentation has never matched the implementation
since the former was written in 2011.

I don't see any particularly satisfactory resolution to this.  Usually,
I'd consider the manual as normative, especially when it's written in
such a proscriptive style.  But, as far as I know, Automake itself is
the only software which invokes test drivers to any significant extent,
and its current behaviour is to pass arguments in separate words, it
seems to me that the least bad approach is to correct the documentation
and usage messages to describe it accurately, with a historical note in
the manual explaining the long-standing error in previous versions of
the documentation.

It seems that I'm the only person in nearly 13 years to have noticed
this.  I guess that hardly anyone writes Automake test drivers. :-/
(Yes, I am considering doing this.)

-- [mdw]

Message #8 received at 70638 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: mdw <at> distorted.org.uk
Cc: 70638 <at> debbugs.gnu.org
Subject: Re: bug#70638: Automake custom-driver interface documentation doesn't
 match implementation
Date: Mon, 29 Apr 2024 15:29:20 -0600

Hi Mark - thanks for the report.

    it seems to me that the least bad approach is to correct the
    documentation and usage messages to describe it accurately, with a
    historical note in the manual explaining the long-standing error in
    previous versions of the documentation.

I agree. Would you be up for writing a draft patch? Given the research
you've already done to send this report, you're certainly more familiar
with the relevant places than me (by several miles :).

    I guess that hardly anyone writes Automake test drivers. :-/

I guess so too.

    (Yes, I am considering doing this.)

If so ... happy driving :). --thanks again, karl.

Message #11 received at 70638 <at> debbugs.gnu.org (full text, mbox):

From: Mark Wooding <mdw <at> distorted.org.uk>
To: Karl Berry <karl <at> freefriends.org>
Cc: 70638 <at> debbugs.gnu.org
Subject: [PATCH] doc: Put test-driver option args in separate words, not
 joined with `='
Date: Tue, 30 Apr 2024 00:06:39 +0100

Karl Berry <karl <at> freefriends.org> writes:

> Hi Mark - thanks for the report.
>
>     it seems to me that the least bad approach is to correct the
>     documentation and usage messages to describe it accurately, with a
>     historical note in the manual explaining the long-standing error in
>     previous versions of the documentation.
>
> I agree. Would you be up for writing a draft patch? Given the research
> you've already done to send this report, you're certainly more familiar
> with the relevant places than me (by several miles :).

Sure, I can give that a try.

------8<------8<------8<------8<------8<------8<------8<------8<------

This addresses #70638.  As now mentioned in the manual, the description
of the test-driver command-line interface has been wrong since 2011 when
it was first added.  The manual, and the driver usage messages
themselves, have consistently shown option arguments joined to the
option name with `=' characters, while the implementation of Automake
has always provided option arguments in the following argument word, and
the provided test drivers only accepted option arguments in the
following argument word.

Because Automake has never used the `=' syntax, there can't possibly be
a working driver which actually follows the specification as written, so,
despite the fierce language of the manual, we agreed that changing the
specification was the right approach.  The `test-driver' program's usage
message has already been fixed (#22445), but the full extent of the
problem wasn't noticed at that time.

So...

* doc/automake.texi (Command-line arguments for test drivers): Fix the
table of options to show arguments passed as separate words; add
footnote explaining this rather sorry situation.
* doc/automake.texi (Use TAP with the Automake test harness): Remove `='
from documentation of `--diagnostic-string', because that was never
acceptable either.
* lib/tap-driver.sh: Fix usage message.
* contrib/tap-driver.pl: Change usage message to match the defined
protocol.  (This implementation parses options using Perl's
`GetOpt::Long' module, so it accepts the `=' syntax as specified, but
this program isn't actually used.)

Signed-off-by: Mark Wooding <mdw <at> distorted.org.uk>
---
 contrib/tap-driver.pl |  8 ++++----
 doc/automake.texi     | 22 ++++++++++++++--------
 lib/tap-driver.sh     |  8 ++++----
 3 files changed, 22 insertions(+), 16 deletions(-)

diff --git a/contrib/tap-driver.pl b/contrib/tap-driver.pl
index c48593984..787741e38 100755
--- a/contrib/tap-driver.pl
+++ b/contrib/tap-driver.pl
@@ -38,10 +38,10 @@ my $ME = "tap-driver.pl";
 
 my $USAGE = <<'END';
 Usage:
-  tap-driver --test-name=NAME --log-file=PATH --trs-file=PATH
-             [--expect-failure={yes|no}] [--color-tests={yes|no}]
-             [--enable-hard-errors={yes|no}] [--ignore-exit]
-             [--diagnostic-string=STRING] [--merge|--no-merge]
+  tap-driver --test-name NAME --log-file PATH --trs-file PATH
+             [--expect-failure {yes|no}] [--color-tests {yes|no}]
+             [--enable-hard-errors {yes|no}] [--ignore-exit]
+             [--diagnostic-string STRING] [--merge|--no-merge]
              [--comments|--no-comments] [--] TEST-COMMAND
 The '--test-name', '--log-file' and '--trs-file' options are mandatory.
 END
diff --git a/doc/automake.texi b/doc/automake.texi
index d30905e0f..a9fa601da 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -10125,34 +10125,40 @@ A custom driver can rely on various command-line options and arguments
 being passed to it automatically by the Automake-generated test harness.
 It is @emph{mandatory} that it understands all of them (even if the exact
 interpretation of the associated semantics can legitimately change
-between a test driver and another, and even be a no-op in some drivers).
+between a test driver and another, and even be a no-op in some
+drivers).@footnote{Regrettably, older versions of this manual stated that
+option arguments should be joined to their options with a @code{=}
+character, rather than passed as in the following argument word as shown
+here.  The syntax with @code{=} has never been accepted by the test drivers
+supplied with Automake, and has never been produced by @file{Makefile}s
+generated by Automake.}
 
 @noindent
 Here is the list of options:
 
 @table @option
-@item --test-name=@var{NAME}
+@item --test-name @var{NAME}
 The name of the test, with VPATH prefix (if any) removed.  This can have a
 suffix and a directory component (as in e.g., @file{sub/foo.test}), and is
 mostly meant to be used in console reports about testsuite advancements and
 results (@pxref{Testsuite progress output}).
-@item --log-file=@file{@var{PATH}.log}
+@item --log-file @file{@var{PATH}.log}
 The @file{.log} file the test driver must create (@pxref{Basics of
 test metadata}).  If it has a directory component (as in e.g.,
 @file{sub/foo.log}), the test harness will ensure that such directory
 exists @emph{before} the test driver is called.
-@item --trs-file=@file{@var{PATH}.trs}
+@item --trs-file @file{@var{PATH}.trs}
 The @file{.trs} file the test driver must create (@pxref{Basics of
 test metadata}).  If it has a directory component (as in e.g.,
 @file{sub/foo.trs}), the test harness will ensure that such directory
 exists @emph{before} the test driver is called.
-@item --color-tests=@{yes|no@}
+@item --color-tests @{yes|no@}
 Whether the console output should be colorized or not (@pxref{Simple
 tests and color-tests}, to learn when this option gets activated and
 when it doesn't).
-@item --expect-failure=@{yes|no@}
+@item --expect-failure @{yes|no@}
 Whether the tested program is expected to fail.
-@item --enable-hard-errors=@{yes|no@}
+@item --enable-hard-errors @{yes|no@}
 Whether ``hard errors'' in the tested program should be treated differently
 from normal failures or not (the default should be @code{yes}).  The exact
 meaning of ``hard error'' is highly dependent from the test protocols or
@@ -10378,7 +10384,7 @@ harness to get confused if anything that appears on standard error
 looks like a test result.
 @item --no-merge
 Revert the effects of @option{--merge}.
-@item --diagnostic-string=@var{STRING}
+@item --diagnostic-string @var{STRING}
 Change the string that introduces TAP diagnostics from the default value
 of ``@code{#}'' to @code{@var{STRING}}.  This can be useful if your
 TAP-based test scripts produce verbose output on which they have limited
diff --git a/lib/tap-driver.sh b/lib/tap-driver.sh
index f16f44df3..69121683a 100755
--- a/lib/tap-driver.sh
+++ b/lib/tap-driver.sh
@@ -48,10 +48,10 @@ print_usage ()
 {
   cat <<END
 Usage:
-  tap-driver.sh --test-name=NAME --log-file=PATH --trs-file=PATH
-                [--expect-failure={yes|no}] [--color-tests={yes|no}]
-                [--enable-hard-errors={yes|no}] [--ignore-exit]
-                [--diagnostic-string=STRING] [--merge|--no-merge]
+  tap-driver.sh --test-name NAME --log-file PATH --trs-file PATH
+                [--expect-failure {yes|no}] [--color-tests {yes|no}]
+                [--enable-hard-errors {yes|no}] [--ignore-exit]
+                [--diagnostic-string STRING] [--merge|--no-merge]
                 [--comments|--no-comments] [--] TEST-COMMAND
 The '--test-name', '-log-file' and '--trs-file' options are mandatory.
 END
-- 
[mdw]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.