GNU bug report logs - #32703
echo_man_error

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 32703 in the body.
You can then email your comments to 32703 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "1064240043" <szq1064240043 <at> qq.com>
To: "bug-coreutils" <bug-coreutils <at> gnu.org>
Subject: echo_man_error
Date: Tue, 11 Sep 2018 19:32:54 +0800

[Message part 1 (text/plain, inline)]

In Manual page echo(1)


it says '-E disable interpretation of backslash escapes (default)'


but I tried and found out that enable(-e) is default.

[Message part 2 (text/html, inline)]

Message #8 received at 32703 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: 1064240043 <szq1064240043 <at> qq.com>, 32703 <at> debbugs.gnu.org
Subject: Re: bug#32703: echo_man_error
Date: Tue, 11 Sep 2018 11:01:59 -0500

tag 32703 needinfo
thanks

On 9/11/18 6:32 AM, 1064240043 wrote:
> In Manual page echo(1)
> 
> 
> it says '-E disable interpretation of backslash escapes (default)'
> 
> 
> but I tried and found out that enable(-e) is default.

What exactly did you try? And are you sure you are testing the same echo 
in your various tests?  Remember that many shells ship echo as a 
built-in (both bash and dash, which are the two most common choices for 
/bin/sh on a GNU/Linux installation), so you have to first determine 
whether you are testing the behavior of your shell instead of the echo 
built as part of coreutils (which is often installed at /bin/echo).

By the way, POSIX requires echo to behave with backslash escapes enabled 
by default (dash does this, and bash can be convinced to do it via 'set 
-o posix && shopt -s xpg_echo').  But oddly enough, coreutils does NOT 
quite seem to obey POSIX, even under POSIXLY_CORRECT.  Note:

$ /bin/echo --version | head -n1
echo (GNU coreutils) 8.29
$ POSIXLY_CORRECT=1 /bin/echo --version
--version

(so far, so good - printing --help/--version is a useful GNU extension, 
but violates POSIX, so we turn it off under POSIXLY_CORRECT)

$ bash -c 'echo a\\nb'
a\nb
$ bash -c 'set -o posix; shopt -s xpg_echo; echo a\\nb'
a
b
$ dash -c 'echo a\\nb'
a
b

(okay, we've demonstrated that bash defaults to non-POSIX behavior, but 
that it can be coerced into obeying; and dash defaults to POSIX behavior)

$ /bin/echo a\\nb
a\nb

(so far, so good - we've chosen to NOT expand backslash by default, 
which matches bash's defaults even if it violates POSIX)

$ POSIXLY_CORRECT=1 /bin/echo a\\nb
a\nb

Yikes!  Even though we asked for POSIX correctness, we are NOT 
interpreting backslashes.  I think this is a bug in GNU coreutils' echo, 
and could be fixed by the patch below (but the testsuite would also need 
updates).

But also note that POSIX has some rather wonky wording - it permits -n 
to have implementation-defined behavior, but forgot to give special 
exceptions to -e or -E.  In short, the POSIX recommendation is that you 
NEVER use 'echo' if you are trying to print a string that might start 
with '-' or contain '\' - use printf instead.


diff --git i/src/echo.c w/src/echo.c
index 9958a8ec6..b3dfdb54a 100644
--- i/src/echo.c
+++ w/src/echo.c
@@ -189,7 +189,7 @@ main (int argc, char **argv)

 just_echo:

-  if (do_v9)
+  if (do_v9 || getenv ("POSIXLY_CORRECT"))
     {
       while (argc > 0)
         {


-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Message #13 received at 32703 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: 1064240043 <szq1064240043 <at> qq.com>, 32703 <at> debbugs.gnu.org
Subject: Re: bug#32703: echo_man_error
Date: Tue, 11 Sep 2018 13:07:07 -0500

On 9/11/18 11:01 AM, Eric Blake wrote:

> $ POSIXLY_CORRECT=1 /bin/echo a\\nb
> a\nb
> 
> Yikes!  Even though we asked for POSIX correctness, we are NOT 
> interpreting backslashes.  I think this is a bug in GNU coreutils' echo, 
> and could be fixed by the patch below (but the testsuite would also need 
> updates).

And it might even be a regression.  Reading through NEWS, I found this 
back in 5.3.0:

  echo now conforms to POSIX better.  It supports the \0ooo syntax for
  octal escapes, and \c now terminates printing immediately.  If
  POSIXLY_CORRECT is set and the first argument is not "-n", echo now
  outputs all option-like arguments instead of treating them as options.

although I haven't actually tested prior versions to see if behavior has 
changed over time.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Message #16 received at 32703 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Eric Blake <eblake <at> redhat.com>, 1064240043 <szq1064240043 <at> qq.com>,
 32703 <at> debbugs.gnu.org
Subject: Re: bug#32703: echo_man_error
Date: Sat, 15 Sep 2018 23:43:34 -0700

On 11/09/18 11:07, Eric Blake wrote:
> On 9/11/18 11:01 AM, Eric Blake wrote:
> 
>> $ POSIXLY_CORRECT=1 /bin/echo a\\nb
>> a\nb
>>
>> Yikes!  Even though we asked for POSIX correctness, we are NOT 
>> interpreting backslashes.  I think this is a bug in GNU coreutils' echo, 
>> and could be fixed by the patch below (but the testsuite would also need 
>> updates).
> 
> And it might even be a regression.  Reading through NEWS, I found this 
> back in 5.3.0:
> 
>    echo now conforms to POSIX better.  It supports the \0ooo syntax for
>    octal escapes, and \c now terminates printing immediately.  If
>    POSIXLY_CORRECT is set and the first argument is not "-n", echo now
>    outputs all option-like arguments instead of treating them as options.
> 
> although I haven't actually tested prior versions to see if behavior has 
> changed over time.

I agree that we shouldn't be divergent here.
POSIXLY_CORRECT should enable interpretation of escapes.

I don't think it's a regression though.
From a quick review, default interpretation of escapes
was only controllable at compile time, until -e allowed
interpretation as of commit b7bfc2d2 (textutils-1.22).

  Author: Jim Meyering <jim <at> meyering.net>
  Date:   Thu Sep 25 12:58:50 1997 +0000
    Make echo conform to POSIX.  By default, don't
    interpret backslash escape sequences.

POSIX may have changed in the meantime, but POSIXLY_CORRECT
was never used as a condition for enabling interpretation of escapes.

cheers,
Pádraig

Message #19 received at 32703 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pádraig Brady <P <at> draigBrady.com>,
 Eric Blake <eblake <at> redhat.com>, 1064240043 <szq1064240043 <at> qq.com>,
 32703 <at> debbugs.gnu.org
Subject: Re: bug#32703: echo_man_error
Date: Sun, 16 Sep 2018 07:42:19 -0700

Pádraig Brady wrote:
>    Author: Jim Meyering<jim <at> meyering.net>
>    Date:   Thu Sep 25 12:58:50 1997 +0000
>      Make echo conform to POSIX.  By default, don't
>      interpret backslash escape sequences.
> 
> POSIX may have changed in the meantime, but POSIXLY_CORRECT
> was never used as a condition for enabling interpretation of escapes.

POSIX certainly did change in the meantime. In 1997 it was POSIX.2 (1992), 
whereas the requirements we're talking about weren't added until POSIX-2001. So 
the bug is that coreutils echo hasn't been upgraded to the new standard since 
2001 and nobody has reported this until now.

Message #24 received at 32703-done <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Eric Blake <eblake <at> redhat.com>, 1064240043 <szq1064240043 <at> qq.com>,
 32703-done <at> debbugs.gnu.org
Subject: Re: bug#32703: echo_man_error
Date: Sun, 23 Sep 2018 19:48:12 -0700

[Message part 1 (text/plain, inline)]

On 11/09/18 11:07, Eric Blake wrote:
> On 9/11/18 11:01 AM, Eric Blake wrote:
> 
>> $ POSIXLY_CORRECT=1 /bin/echo a\\nb
>> a\nb
>>
>> Yikes!  Even though we asked for POSIX correctness, we are NOT 
>> interpreting backslashes.  I think this is a bug in GNU coreutils' echo, 
>> and could be fixed by the patch below (but the testsuite would also need 
>> updates).

I'll push the attached later.
Interesting there were no echo tests until now.

Marking this as done.

cheers,
Pádraig.

[echo-posix.patch (text/x-patch, attachment)]

Message #27 received at 32703-done <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Pádraig Brady <P <at> draigBrady.com>,
 1064240043 <szq1064240043 <at> qq.com>, 32703-done <at> debbugs.gnu.org
Subject: Re: bug#32703: echo_man_error
Date: Mon, 24 Sep 2018 08:52:43 -0500

On 9/23/18 9:48 PM, Pádraig Brady wrote:
> I'll push the attached later.
> Interesting there were no echo tests until now.
> 
> Marking this as done.

Thanks.

> +++ b/tests/misc/echo.sh

> +
> +# Verify the methods of specifying "Escape":
> +# Note 4 octal digits are allows (unlike printf which uses up to 3)

allowed

> +++ b/tests/misc/printf.sh
> @@ -60,9 +60,8 @@ $prog '5 % +d\n' 234  >> out || fail=1
>   # coreutils-5.0.1, it would print six bytes: "6 \41\n".
>   $prog '6 \41\n' | tr '\41' '!' >> out
>   
> -# Note that as of coreutils-5.0.1, printf with a format of '\0002x'
> -# prints a NUL byte followed by the digit '2' and an 'x'.
> -# By contrast bash's printf outputs the same thing as $(printf '\2x') does.
> +# Note that as of coreutils-5.0.1, printf with a format of '\0002y'
> +# prints a NUL byte followed by the digit '2' and an 'y'.

s/an/a/

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Message #30 received at 32703 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: 32703 <at> debbugs.gnu.org, P <at> draigBrady.com, szq1064240043 <at> qq.com
Subject: Re: bug#32703: echo_man_error
Date: Mon, 24 Sep 2018 23:47:39 +0200

[Message part 1 (text/plain, inline)]

On 9/24/18 4:48 AM, Pádraig Brady wrote:
> I'll push the attached later.
> Interesting there were no echo tests until now.

Nice work, indeed.

Additionally to Eric's comment, I'd suggest to increase the coverage
for 'echo' to about 100% [1] with the attached patch:

*  src/echo.c (usage): Assert that STATUS is always EXIT_SUCCESS.
* tests/misc/echo.sh: Add further tests for all hex and escape and
escape characters.

You may simply squash it in.

[1] To get coverage statistic, run:
  make coverage -j 4 TESTS=tests/misc/echo.sh SUBDIRS=.
  xdg-open file:doc/coverage/src/echo.c.gcov.frameset.html


Have a nice day,
Berny

[echo-test-coverage.diff (text/x-patch, attachment)]

Message #33 received at 32703-done <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Pádraig Brady <P <at> draigBrady.com>,
 1064240043 <szq1064240043 <at> qq.com>, 32703-done <at> debbugs.gnu.org
Subject: Re: bug#32703: echo_man_error
Date: Mon, 24 Sep 2018 17:15:56 -0500

On 9/23/18 9:48 PM, Pádraig Brady wrote:

> +++ b/NEWS
> @@ -5,6 +5,12 @@ GNU coreutils NEWS                                    -*- outline -*-
>   ** Bug fixes
>   
>     df no longer corrupts displayed multibyte characters on macOS.
> +  [bug introduced with coreutils-8.18]
> +
> +** Changes in behavior
> +
> +  echo now always processes backslash escapes when the POSIXLY_CORRECT
> +  environment variable is set.

Having re-read the POSIX wording, I'm not quite sure we are still accurate.

> +++ b/src/echo.c
> @@ -108,8 +108,9 @@ int
>   main (int argc, char **argv)
>   {
>     bool display_return = true;
> +  bool posixly_correct = getenv ("POSIXLY_CORRECT");
>     bool allow_options =
> -    (! getenv ("POSIXLY_CORRECT")
> +    (! posixly_correct
>        || (! DEFAULT_ECHO_TO_XPG && 1 < argc && STREQ (argv[1], "-n")));

This special-cases a literal "-n" as the first argument, while POSIX 
states (http://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html)

Implementations shall not support any options.

A string to be written to standard output. If the first operand is -n, 
or if any of the operands contain a <backslash> character, the results 
are implementation-defined.

and then with XSI shading states:

On XSI-conformant systems, if the first operand is -n, it shall be 
treated as a string, not an option.

So my initial patch (which you took and improved) still isn't quite right.

> +
> +# With POSIXLY_CORRECT:
> +#   only -n as the first (separate) option enables option processing

Rather, not even -n should get special treatment as an option.

> +#   -E is ignored
> +#   escapes are processed by default
> +POSIXLY_CORRECT=1 $prog -n -E 'foo\n' > out || fail=1

So I would argue that this should output $'-n -E foo\n'.

> +POSIXLY_CORRECT=1 $prog -nE 'foo' >> out || fail=1
> +POSIXLY_CORRECT=1 $prog -E -n 'foo' >> out || fail=1
> +POSIXLY_CORRECT=1 $prog --version >> out || fail=1
> +cat <<\EOF > exp
> +foo
> +-nE foo
> +-E -n foo
> +--version
> +EOF
> +compare exp out || fail=1
-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Message #36 received at 32703 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>, 32703 <at> debbugs.gnu.org,
 szq1064240043 <at> qq.com
Subject: Re: bug#32703: echo_man_error
Date: Mon, 24 Sep 2018 21:10:54 -0700

On 24/09/18 14:47, Bernhard Voelker wrote:
> On 9/24/18 4:48 AM, Pádraig Brady wrote:
>> I'll push the attached later.
>> Interesting there were no echo tests until now.
> 
> Nice work, indeed.
> 
> Additionally to Eric's comment, I'd suggest to increase the coverage
> for 'echo' to about 100% [1] with the attached patch:
> 
> *  src/echo.c (usage): Assert that STATUS is always EXIT_SUCCESS.
> * tests/misc/echo.sh: Add further tests for all hex and escape and
> escape characters.
> 
> You may simply squash it in.
> 
> [1] To get coverage statistic, run:
>   make coverage -j 4 TESTS=tests/misc/echo.sh SUBDIRS=.
>   xdg-open file:doc/coverage/src/echo.c.gcov.frameset.html
> 
> 
> Have a nice day,
> Berny
> 

Excellent!
I'll push this as its own patch.

thanks!
Pádraig

Message #39 received at 32703-done <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Eric Blake <eblake <at> redhat.com>, 1064240043 <szq1064240043 <at> qq.com>,
 32703-done <at> debbugs.gnu.org
Subject: Re: bug#32703: echo_man_error
Date: Mon, 24 Sep 2018 21:51:30 -0700

On 24/09/18 15:15, Eric Blake wrote:
> On 9/23/18 9:48 PM, Pádraig Brady wrote:
> 
>> +++ b/NEWS
>> @@ -5,6 +5,12 @@ GNU coreutils NEWS                                    -*- outline -*-
>>   ** Bug fixes
>>   
>>     df no longer corrupts displayed multibyte characters on macOS.
>> +  [bug introduced with coreutils-8.18]
>> +
>> +** Changes in behavior
>> +
>> +  echo now always processes backslash escapes when the POSIXLY_CORRECT
>> +  environment variable is set.
> 
> Having re-read the POSIX wording, I'm not quite sure we are still accurate.
> 
>> +++ b/src/echo.c
>> @@ -108,8 +108,9 @@ int
>>   main (int argc, char **argv)
>>   {
>>     bool display_return = true;
>> +  bool posixly_correct = getenv ("POSIXLY_CORRECT");
>>     bool allow_options =
>> -    (! getenv ("POSIXLY_CORRECT")
>> +    (! posixly_correct
>>        || (! DEFAULT_ECHO_TO_XPG && 1 < argc && STREQ (argv[1], "-n")));
> 
> This special-cases a literal "-n" as the first argument, while POSIX 
> states (http://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html)
> 
> Implementations shall not support any options.
> 
> A string to be written to standard output. If the first operand is -n, 
> or if any of the operands contain a <backslash> character, the results 
> are implementation-defined.
> 
> and then with XSI shading states:
> 
> On XSI-conformant systems, if the first operand is -n, it shall be 
> treated as a string, not an option.
> 
> So my initial patch (which you took and improved) still isn't quite right.

I noticed that POSIX wording too,
but erred on the side of consistency:

  $ bash -c 'set -o posix; shopt -s xpg_echo; echo -n foo'
  -n foo

  $ bash -c 'set -o posix; echo -n foo'
  foo
  $ dash -c 'echo -n foo'
  foo

Now we do have compile time control over xpg mode so
should probably restrict the no option processing
to that mode?
In summary the handling of -n specially is weird
and inconsistent with bash, so in a separate patch
I propose we handle all (leading) options also
with POSIXLY_CORRECT to be consistent with bash.

Oh hang on. It's the "xpg_echo" bash setting that enables -e by default :/
We don't have a runtime setting for that at present.

$ bash -c 'set -o posix; echo -ne a\\nb\\n'
a
b
bash -c 'set -o posix; echo -E a\\nb'
a\nb

$ bash -c 'set -o posix; shopt -s xpg_echo; echo a\\nb'
a
b
$ bash -c 'set -o posix; echo a\\nb'
a\nb
$ bash -c 'set -o posix; echo -e a\\nb'
a
b

So there is no way to be consistent with bash and dash I think.
Let's have bash and coreutils consistent at least.
The only question is whether POSIXLY_CORRECT in coreutils
is equiv to `shopt -s xpg_echo` in bash or not.
Assuming it is, then we'll need to echo -n when set as you say.

Le sigh. What a mess

cheers,
Pádraig

Message #46 received at 32703 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: 32703 <at> debbugs.gnu.org
Subject: Re: bug#32703: echo_man_error
Date: Sat, 27 Oct 2018 04:55:28 -0700

On 24/09/18 21:51, Pádraig Brady wrote:
> On 24/09/18 15:15, Eric Blake wrote:
>> On 9/23/18 9:48 PM, Pádraig Brady wrote:
>>
>>> +++ b/NEWS
>>> @@ -5,6 +5,12 @@ GNU coreutils NEWS                                    -*- outline -*-
>>>   ** Bug fixes
>>>   
>>>     df no longer corrupts displayed multibyte characters on macOS.
>>> +  [bug introduced with coreutils-8.18]
>>> +
>>> +** Changes in behavior
>>> +
>>> +  echo now always processes backslash escapes when the POSIXLY_CORRECT
>>> +  environment variable is set.
>>
>> Having re-read the POSIX wording, I'm not quite sure we are still accurate.
>>
>>> +++ b/src/echo.c
>>> @@ -108,8 +108,9 @@ int
>>>   main (int argc, char **argv)
>>>   {
>>>     bool display_return = true;
>>> +  bool posixly_correct = getenv ("POSIXLY_CORRECT");
>>>     bool allow_options =
>>> -    (! getenv ("POSIXLY_CORRECT")
>>> +    (! posixly_correct
>>>        || (! DEFAULT_ECHO_TO_XPG && 1 < argc && STREQ (argv[1], "-n")));
>>
>> This special-cases a literal "-n" as the first argument, while POSIX 
>> states (http://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html)
>>
>> Implementations shall not support any options.
>>
>> A string to be written to standard output. If the first operand is -n, 
>> or if any of the operands contain a <backslash> character, the results 
>> are implementation-defined.
>>
>> and then with XSI shading states:
>>
>> On XSI-conformant systems, if the first operand is -n, it shall be 
>> treated as a string, not an option.
>>
>> So my initial patch (which you took and improved) still isn't quite right.
> 
> I noticed that POSIX wording too,
> but erred on the side of consistency:
> 
>   $ bash -c 'set -o posix; shopt -s xpg_echo; echo -n foo'
>   -n foo
> 
>   $ bash -c 'set -o posix; echo -n foo'
>   foo
>   $ dash -c 'echo -n foo'
>   foo
> 
> Now we do have compile time control over xpg mode so
> should probably restrict the no option processing
> to that mode?
> In summary the handling of -n specially is weird
> and inconsistent with bash, so in a separate patch
> I propose we handle all (leading) options also
> with POSIXLY_CORRECT to be consistent with bash.
> 
> Oh hang on. It's the "xpg_echo" bash setting that enables -e by default :/
> We don't have a runtime setting for that at present.
> 
> $ bash -c 'set -o posix; echo -ne a\\nb\\n'
> a
> b
> bash -c 'set -o posix; echo -E a\\nb'
> a\nb
> 
> $ bash -c 'set -o posix; shopt -s xpg_echo; echo a\\nb'
> a
> b
> $ bash -c 'set -o posix; echo a\\nb'
> a\nb
> $ bash -c 'set -o posix; echo -e a\\nb'
> a
> b
> 
> So there is no way to be consistent with bash and dash I think.
> Let's have bash and coreutils consistent at least.
> The only question is whether POSIXLY_CORRECT in coreutils
> is equiv to `shopt -s xpg_echo` in bash or not.
> Assuming it is, then we'll need to echo -n when set as you say.
> 
> Le sigh. What a mess

Thinking a bit more about this,

basically for -e, in echo(1), bash, and dash,
where -e is not interpreted as an option
we should interpret backslashes by default.
That's what this patch now does.

For -n, dash and bash (set -o posix), do give the
facility to outputting without the trailing newline
by interpreting the -n option, and POSIX says this
is implementation defined, so let's continue to
provide this facility.

We've compile time support for the less
frequently needed facility to output a leading -n,
dash doesn't currently have this facility.
bash supports this with `shopt -s xpg_echo`.
I suggest we don't change this with POSIXLY_CORRECT,
or if we do with a subsequent patch, to do
do it with a separate env var/value.

Pushing these two patches now...

cheers,
Pádraig

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.