GNU bug report logs - #25388
ls-quotes: kills existing scripts reading "ls" -1 as input

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 25388 in the body.
You can then email your comments to 25388 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: bug-coreutils <at> gnu.org
Subject: Bug in ls, kills existing scripts reading "ls" -1 as input
Date: Fri, 06 Jan 2017 19:52:22 -0800

The new ls doesn't maintain backwards compatibility.
The default options now add extraneous quotes to
some filenames.

Anyplace one uses 'ls -1' to read 1 file/line
now breaks.

This is a regression as it breaks existing
scripts and behavior.

Message #8 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Schwab <schwab <at> linux-m68k.org>
To: L A Walsh <coreutils <at> tlinx.org>
Cc: 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls,
 kills existing scripts reading "ls" -1 as input
Date: Sun, 08 Jan 2017 10:02:03 +0100

On Jan 06 2017, L A Walsh <coreutils <at> tlinx.org> wrote:

> The new ls doesn't maintain backwards compatibility.
> The default options now add extraneous quotes to
> some filenames.
>
> Anyplace one uses 'ls -1' to read 1 file/line
> now breaks.
>
> This is a regression as it breaks existing
> scripts and behavior.

Does it?

$ touch 'a b'
$ ls -1 | grep "'"

Andreas.

-- 
Andreas Schwab, schwab <at> linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Message #11 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Andreas Schwab <schwab <at> linux-m68k.org>
Cc: 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1
 as input
Date: Sun, 08 Jan 2017 21:21:52 -0800

Andreas Schwab wrote:
> On Jan 06 2017, L A Walsh <coreutils <at> tlinx.org> wrote:
>> The new ls doesn't maintain backwards compatibility.
>> The default options now add extraneous quotes to
>> some filenames.
---
> Does it?
> 
> $ touch 'a b'
> $ ls -1 | grep "'"
----
	Sorry, you are right.

	The problem is 'ls' doesn't show the same
information on the screen when it is passed through
a pager like "less" or "more".  

	In all but the smallest of directories, it's virtually
a requirement to page directory output using 'more' or 'less'.

	Having 'ls' show different output through a pager
than it does when scrolling off the screen is not a standard
feature.  The user has no idea how to page through 'ls's 
output as they saw it scroll by on the screen.  It's 
inconsistent with any other tools that work with pagers as
well as being inconsistent with itself in how it handles 
COLOR options.

	'ls' should default to its regular behavior and
only engage quoting when asked to.  

	Let distros and users decide what to enable
to add new behaviors -- like the various color options.
My distro added color by default in their defaults
w/it only on when output is not a pipe.

	I changed it on my machine to put out color
even if through a pipe as I usually am piping it through
'less' and I want to see the color.  If I want no color --
I can get the normal text by adding a backslash before
the command.

	The new quoting also messes up copy/pasting into
any place other than the shell.  I tend to copy/paste 
between a shell and gvim when doing script development.
I wouldn't mind having such a switch as an option, 
but it is usually the case that I can copy/paste between
windows and have it pick up whether or not a tab or a space
was used.  Having tabs converted to a shell code would
prevent them being expanded anyplace but in the same shell.

Follow the example of COLOR and let users (including distros)
decide what to enable. It will let those who want such
to have it on, but not cause confusion or compatibility
problems.

Message #14 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: L A Walsh <coreutils <at> tlinx.org>, Andreas Schwab <schwab <at> linux-m68k.org>
Cc: 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1 as
 input
Date: Sun, 8 Jan 2017 21:25:23 -0800

L A Walsh wrote:

>     Having 'ls' show different output through a pager
> than it does when scrolling off the screen is not a standard
> feature

Sure it is. 'ls' has done that since then 1980s. 'ls' shows multicolumn output 
when the output is a tty, and single-column output when piped into a pager.

Message #17 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1
 as input
Date: Mon, 09 Jan 2017 10:48:39 -0800

[Message part 1 (text/plain, inline)]

Paul Eggert wrote:
> L A Walsh wrote:
> 
>>     Having 'ls' show different output through a pager
>> than it does when scrolling off the screen is not a standard
>> feature
> 
> Sure it is. 'ls' has done that since then 1980s. 'ls' shows multicolumn 
> output when the output is a tty, and single-column output when piped 
> into a pager.
> 
----
	That's not what I'm used to:

Ishtar:/tmp/tmp> ls
12345678                     a.exe*        cccccccccccc.tar  ssh-3LKwkRvOQM/
UiGUI_log_20161216-1442.htm  bbbbbbbbb.gz  ddddddddddddddd   tall/
Ishtar:/tmp/tmp> ls |more        
12345678                     a.exe*        cccccccccccc.tar  ssh-3LKwkRvOQM/
UiGUI_log_20161216-1442.htm  bbbbbbbbb.gz  ddddddddddddddd   tall/
Ishtar:/tmp/tmp> ls >../tmpdir.txt
Ishtar:/tmp/tmp> wc ../tmpdir.txt
 2   8 235 ../tmpdir.txt

i.e. they look the same either way -- even down to the same
colors (attaching tmpdir.txt which should display as above
but with color)... depends on your LS_COLOR settings... 
Will include that here (very long line):

export LS_COLORS='*~=01;30:*.bak=01;30:*.orig=01;30:no=00:fi=00:di=01;35:ln=01;36:pi=40;33:so=04;01;34:do=04;01;34:bd=40;33;01:cd=40;33;01:or=41;33;01:su=01;04;36:sg=01;04;35:ca=01;04;36:tw=30;42:ow=34;42:st=37;44:ex=00;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tlzma=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.rpm*=01\:30:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;3
5:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:'

ls alias on my machines:
alias ls='ls -CF --show-control-chars --color=always'

[tmpdir.txt (text/plain, inline)]

12345678                     a.exe*        cccccccccccc.tar  ssh-3LKwkRvOQM/
UiGUI_log_20161216-1442.htm  bbbbbbbbb.gz  ddddddddddddddd   tall/

Message #20 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1
 as input
Date: Mon, 09 Jan 2017 11:00:20 -0800

Paul Eggert wrote:
> L A Walsh wrote:
>>     Having 'ls' show different output through a pager than it
>> does when scrolling off the screen is not a standard feature.
> 
> Sure it is. 'ls' has done that since then 1980s. 'ls' shows multicolumn 
> output when the output is a tty, and single-column output when piped 
> into a pager.
----
That would be really annoying if it were true.  It doesn't do that
on any of my *nix terms.  When I put it through a pager I want it to page
the output I just saw.  That's what it's for.

I.e. using a pager, should not, by default, change the output
of programs.  It's purpose is to moderate the output stream
so a user can read what was output when they ran it without
the pager.  Changing the output, by default, depending on
pager usage defeats the primary purpose of pagers.

If it became common place for programs to detect pager use
and generate different output, it would seriously harm the
intended use of pagers, since you would no longer be able to
page the output of those programs (since what you are paging
isn't the original output).

FWIW: If I want 1-column output, I usually use "-1".  
That's what it is for, no?

-l

Message #23 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: L A Walsh <coreutils <at> tlinx.org>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1 as
 input
Date: Mon, 9 Jan 2017 11:07:41 -0800

On 01/09/2017 10:48 AM, L A Walsh wrote:
>     That's not what I'm used to: 

I couldn't parse your email. I was describing the common behavior in BSD
and GNU/Linux for ages, like this:

$ touch a b
$ ls
a  b
$ ls | more
a
b

Message #26 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: L A Walsh <coreutils <at> tlinx.org>, Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1 as
 input
Date: Mon, 9 Jan 2017 13:33:19 -0600

[Message part 1 (text/plain, inline)]

On 01/09/2017 01:00 PM, L A Walsh wrote:
> Paul Eggert wrote:
>> L A Walsh wrote:
>>>     Having 'ls' show different output through a pager than it
>>> does when scrolling off the screen is not a standard feature.
>>
>> Sure it is. 'ls' has done that since then 1980s. 'ls' shows
>> multicolumn output when the output is a tty, and single-column output
>> when piped into a pager.
> ----
> That would be really annoying if it were true.  It doesn't do that
> on any of my *nix terms.  When I put it through a pager I want it to page
> the output I just saw.  That's what it's for.

But that's EXACTLY what POSIX has specified, because it has been
existing practice for YEARS.

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/ls.html
"STDOUT
    The default format shall be to list one entry per line to standard
output; the exceptions are to terminals or when one of the -C, -m, or -x
options is specified. If the output is to a terminal, the format is
implementation-defined.
"

And POSIX merely codified existing practice (this is nothing new - it
has been this way since the 70's) - the earliest versions of ls used
isatty(), and implicitly turned on at least -q if stdout was a terminal
or -1 if it was not, which means that you have ALWAYS had different
behavior between 'ls' and 'ls | more', at least when ls is not some
alias that explicitly specifies other command line options to override
the defaults.

The GNU Coding Standards explicitly discourage NEW programs from
changing default behavior based solely on whether stdout is a tty or
not, but existing programs like ls are grandfathered in to have their
historical behavior, which predates the GNU Coding Standards.

> 
> I.e. using a pager, should not, by default, change the output
> of programs.  It's purpose is to moderate the output stream
> so a user can read what was output when they ran it without
> the pager.  Changing the output, by default, depending on
> pager usage defeats the primary purpose of pagers.

I agree that it is not nice to change output merely based on the type of
fd that stdout is, and GNU Coding Standards agrees in general.  But like
it or not, that's precisely what happens in ls due to
backwards-compatibility - using a pager through a pipeline changes
stdout from being a tty to instead being a pipeline, and therefore
changes default output of at least 'ls', and such change in behavior is
POSIX-compliant.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #29 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1
 as input
Date: Mon, 09 Jan 2017 11:38:29 -0800

Paul Eggert wrote:
> On 01/09/2017 10:48 AM, L A Walsh wrote:
>>     That's not what I'm used to: 
> 
> I couldn't parse your email. 
---
1) I ran the ls command on a directory, shows 4 columns 
  (that were in color).

Ishtar:/tmp/tmp> ls
12345678                     a.exe*        cccccccccccc.tar  ssh-3LKwkRvOQM/
UiGUI_log_20161216-1442.htm  bbbbbbbbb.gz  ddddddddddddddd   tall/

2) Next I ran the same ls command through more (really 'less' -- something
  set @ distro level, but found convenient, so kept it).  Output is
  identical.

Ishtar:/tmp/tmp> ls |more        
12345678                     a.exe*        cccccccccccc.tar  ssh-3LKwkRvOQM/
UiGUI_log_20161216-1442.htm  bbbbbbbbb.gz  ddddddddddddddd   tall/

3) Then I ran the same command and sent it to a file (which I attached
  to the email) and ran 'wc' on it -- which showed it had
  2 lines and 8 "words" (filenames in this case).

Ishtar:/tmp/tmp> ls >../tmpdir.txt
Ishtar:/tmp/tmp> wc ../tmpdir.txt
2   8 235 ../tmpdir.txt 

I was describing the common behavior in BSD
> and GNU/Linux for ages, like this:
> 
> $ touch a b
> $ ls
> a  b
> $ ls | more
> a
> b
----
When I do the same, I get this:

> ls
a  b
> ls | more
a  b

Which is how it has been for over 15 years on linux on my machines.

If my problem is that the 'ls' listing scrolled off the screen
too fast for me to read, then the "default" behavior (that you 
describe) doesn't seem well thought out.  If a user couldn't
read it in multi-column mode, what would prompt anyone to think that
they would automatically want it in 1-column mode when they
put it through a pager -- doesn't make sense.

If I wanted 1 column format, why wouldn't I use "-1".  
That's the entire purpose of the switch.

Again, I state that showing different output through a pager vs.
what was shot out to the screen could allow a security problem
in the form of a stenographic attack.

While dumping the output to a screen, maybe, a SW hack couldn't
be covered up, but if they altered the sources of 'more/less', 
to hide their tracks, it would be noticeably harder to find.

It's not good practice, IMO, to be altering output based on
what (or who) is reading it (at least not by default).

Message #32 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: L A Walsh <coreutils <at> tlinx.org>, Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1 as
 input
Date: Mon, 9 Jan 2017 13:44:41 -0600

[Message part 1 (text/plain, inline)]

On 01/09/2017 12:48 PM, L A Walsh wrote:

>> Sure it is. 'ls' has done that since then 1980s. 'ls' shows
>> multicolumn output when the output is a tty, and single-column output
>> when piped into a pager.
>>
> ----
>     That's not what I'm used to:

> ls alias on my machines:
> alias ls='ls -CF --show-control-chars --color=always'

That's because you are using an alias to alter 'ls' to non-default
behavior for your personal default.

Try again with '\ls' everywhere you previously used 'ls', and you will
see that output to the terminal is different than output to a file or
pipeline.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #35 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: L A Walsh <coreutils <at> tlinx.org>, Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1 as
 input
Date: Mon, 9 Jan 2017 13:47:16 -0600

[Message part 1 (text/plain, inline)]

On 01/09/2017 01:38 PM, L A Walsh wrote:
> 
> 
> Paul Eggert wrote:
>> On 01/09/2017 10:48 AM, L A Walsh wrote:
>>>     That's not what I'm used to: 
>>
>> I couldn't parse your email. 
> ---
> 1) I ran the ls command on a directory, shows 4 columns   (that were in
> color).

That right there is a clue that your 'ls' is a non-standard alias that
is adding additional command lines.  'ls' by default does not output in
color.

> 2) Next I ran the same ls command through more (really 'less' -- something
>   set @ distro level, but found convenient, so kept it).  Output is
>   identical.

Again, output is identical BECAUSE of your alias.  But if you run:

\ls | more

you will get DIFFERENT output.

> It's not good practice, IMO, to be altering output based on
> what (or who) is reading it (at least not by default).

Good or not, it IS what happens, and we can't change it due to years of
usage.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #38 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Eric Blake <eblake <at> redhat.com>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, Andreas Schwab <schwab <at> linux-m68k.org>,
 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1
 as input
Date: Mon, 09 Jan 2017 11:53:12 -0800

Eric Blake wrote:
> But that's EXACTLY what POSIX has specified, because it has been
> existing practice for YEARS.
---
	A bit louder Eric, you can't be heard.


> 
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/ls.html
> "STDOUT
>     The default format shall be to list one entry per line to standard
> output; the exceptions are to terminals or when one of the -C, -m, or -x
> options is specified. If the output is to a terminal, the format is
> implementation-defined.
---
	Perhaps you could read my email -- especially the part
where I listed the alias I use for running 'ls'.


> And POSIX merely codified existing practice (this is nothing new - it
> has been this way since the 70's)
---
	Not anymore.

	Breaking "rm -fr ." wasn't an existing practice except
at BSD-using dists (like BSD & SunOS).  While Solaris was SysV, 
since it was bought up, it has changed.


> The GNU Coding Standards explicitly discourage NEW programs from
> changing default behavior based solely on whether stdout is a tty or
> not, but existing programs like ls are grandfathered in to have their
> historical behavior, which predates the GNU Coding Standards.
---
	Good -- that makes the issue pretty black & white.

	Given those standards, then adding a NEW behavior that 
does change the output depending on destination being tty or not
would be "explicitly discouraged".  Making the matter more clear
is that it is NOT a case of grandfathering in a "historic behavior"


> I agree that it is not nice to change output merely based on the type of
> fd that stdout is, and GNU Coding Standards agrees in general.  But like
> it or not, that's precisely what happens in ls due to
> backwards-compatibility.
---
	The old behavior of multi or single columns may be
historical and backwards compat -- I have no problem with that.

*However*, the new behavior of adding new-shell-encoded
output as the default is not historical.  It encodes output
for bash and other new shells -- hindering cut&paste to other
programs.

Message #41 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Eric Blake <eblake <at> redhat.com>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, Andreas Schwab <schwab <at> linux-m68k.org>,
 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1
 as input
Date: Mon, 09 Jan 2017 11:57:29 -0800

Eric Blake wrote:
> On 01/09/2017 12:48 PM, L A Walsh wrote:
> 
>>> Sure it is. 'ls' has done that since then 1980s. 'ls' shows
>>> multicolumn output when the output is a tty, and single-column output
>>> when piped into a pager.
>>>
>> ----
>>     That's not what I'm used to:
> 
>> ls alias on my machines:
>> alias ls='ls -CF --show-control-chars --color=always'
> 
> That's because you are using an alias to alter 'ls' to non-default
> behavior for your personal default.
> 
> Try again with '\ls' everywhere you previously used 'ls', and you will
> see that output to the terminal is different than output to a file or
> pipeline.
---

Gee guess you missed my 1st response to Andreas:

>     I changed it on my machine to put out color
> even if through a pipe as I usually am piping it through
> 'less' and I want to see the color.  If I want no color --
> I can get the normal text by adding a backslash before
> the command.  
---

So 'backslash' usage -- got that -- disables alias usage.

Thanks.
:-)

Message #44 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Eric Blake <eblake <at> redhat.com>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, Andreas Schwab <schwab <at> linux-m68k.org>,
 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1
 as input
Date: Mon, 09 Jan 2017 12:00:45 -0800

Eric Blake wrote:
> Good or not, it IS what happens, and we can't change it due to years of
> usage.
----
	You are coming into the middle of the discussion.
	I wasn't asking for a change to _that_ specific behavior.
	I'm against adding more/new behaviors that also change
output and do it by default.

Message #47 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: L A Walsh <coreutils <at> tlinx.org>, Eric Blake <eblake <at> redhat.com>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1 as
 input
Date: Mon, 9 Jan 2017 12:07:51 -0800

On 01/09/2017 12:00 PM, L A Walsh wrote:
> I wasn't asking for a change to _that_ specific behavior.

You argued that it's not standard for 'ls' to behave differently when
piping output to a pager, and that the recent change was therefore
undesirable. As the premise for this argument was incorrect, it
shouldn't be surprising if readers disagree with its conclusion.

There are obviously advantages and disadvantages of the recent change,
and we thought that the former outweighed the latter for typical users.
If this isn't the case for you, you can change your personal alias to
reflect your preferences. You're already using such an alias, since you
already prefer alternatives to the default in other areas, so this
shouldn't be much trouble.

Message #50 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: L A Walsh <coreutils <at> tlinx.org>, Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1 as
 input
Date: Mon, 9 Jan 2017 20:10:30 +0000

tag 25388 notabug
close 25388
stop

On 09/01/17 18:48, L A Walsh wrote:
> ls alias on my machines:
> alias ls='ls -CF --show-control-chars --color=always'

Note a caveat of the above is that it changes the display width
depending on whether you pipe to more or not.
You could avoid that by changing the alias to:

  alias ls='COLUMNS=$COLUMNS ls -CF --show-control-chars --color'

Also you can stick a -N option in that alias to disable the default quoting.

cheers,
Pádraig.

Message #53 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, Eric Blake <eblake <at> redhat.com>,
 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1
 as input
Date: Mon, 09 Jan 2017 12:35:46 -0800

Paul Eggert wrote:
> On 01/09/2017 12:00 PM, L A Walsh wrote:
>> I wasn't asking for a change to _that_ specific behavior.
> 
> You argued that it's not standard for 'ls' to behave differently when
> piping output to a pager, and that the recent change was therefore
> undesirable.
====
	I stand corrected for the historical behavior.

	I also stand for it being strongly against GNU
standards to add more such behaviors.

> As the premise for this argument was incorrect, it
> shouldn't be surprising if readers disagree with its conclusion.
----
	Now you are deliberately excluding the fact.  While my
initial stance was that it was engaging in non-standard behavior
(which is true) -- it is also the case that an exception was
made for 'ls' for historical behavior.

	This is not historical behavior.  'ls' doesn't get
a carte blanc to change output anyway it wants because of 
a specific historical behavior.  

Message #56 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, Andreas Schwab <schwab <at> linux-m68k.org>,
 25388 <at> debbugs.gnu.org
Subject: Bug in ls, kills existing scripts reading "ls" -1 as input
Date: Mon, 09 Jan 2017 12:37:00 -0800

How do you reopen a bug that was closed before I even replied?


Pádraig Brady wrote:
> tag 25388 notabug
> close 25388
> stop
> 
> On 09/01/17 18:48, L A Walsh wrote:
>> ls alias on my machines:
>> alias ls='ls -CF --show-control-chars --color=always'
> 
> Note a caveat of the above is that it changes the display width
> depending on whether you pipe to more or not.
> You could avoid that by changing the alias to:
> 
>   alias ls='COLUMNS=$COLUMNS ls -CF --show-control-chars --color'
> 
> Also you can stick a -N option in that alias to disable the default quoting.
> 
> cheers,
> Pádraig.
> 

Message #59 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, Andreas Schwab <schwab <at> linux-m68k.org>,
 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1
 as input
Date: Mon, 09 Jan 2017 12:39:22 -0800

reopen 25388
thanks

It's still a bug as it goes against adding new behaviors that
generate different output based on destination = tty or not.



Pádraig Brady wrote:
> tag 25388 notabug
> close 25388
> stop
> 
> On 09/01/17 18:48, L A Walsh wrote:
>> ls alias on my machines:
>> alias ls='ls -CF --show-control-chars --color=always'
> 
> Note a caveat of the above is that it changes the display width
> depending on whether you pipe to more or not.
> You could avoid that by changing the alias to:
> 
>   alias ls='COLUMNS=$COLUMNS ls -CF --show-control-chars --color'
> 
> Also you can stick a -N option in that alias to disable the default quoting.
> 
> cheers,
> Pádraig.
> 

Message #62 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: L A Walsh <coreutils <at> tlinx.org>, Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1 as
 input
Date: Mon, 9 Jan 2017 14:50:53 -0600

[Message part 1 (text/plain, inline)]

On 01/09/2017 02:35 PM, L A Walsh wrote:
>     I also stand for it being strongly against GNU
> standards to add more such behaviors.

'ls' did not recently add any more cases where tty output differs from
non-tty output when all other things are equal in the default state.
All that changed was that tty output is formatted differently than it
has been in the past.  And, as always, if you don't like the default,
you can override it.

> 
>> As the premise for this argument was incorrect, it
>> shouldn't be surprising if readers disagree with its conclusion.
> ----
>     Now you are deliberately excluding the fact.  While my
> initial stance was that it was engaging in non-standard behavior
> (which is true)

No, your claim that ls' behavior is non-standard has been proven to be
false; I quoted the line from POSIX that explicitly permits ls to use
implementation-defined behavior when outputting to a terminal, and if it
is permitted by the standard, then the behavior can't be considered
non-standard.  It may not be what you want, but that does not make it
non-standard.

> -- it is also the case that an exception was
> made for 'ls' for historical behavior.
> 
>     This is not historical behavior.  'ls' doesn't get
> a carte blanc to change output anyway it wants because of a specific
> historical behavior. 

Now you are excluding a fact.  POSIX _does_ state that the output of
'ls' when targetting a tty is implementation-defined, so we _do_ get
carte blanche permission to define our implementation to do what we
think is the sanest default, even if our definition of sanest changes
over time as we gain more experience on what types of problems trip up
the most default users.  The fact that ls output to a tty differs from
what you get when outputting to a pipeline or file is irrelevant - ls
output to a tty has ALWAYS differed from pipeline and file, when relying
on the defaults.  And if you don't LIKE the difference, then don't use
the defaults - write your alias to override the defaults to ask
explicitly for a format that does not add quoting you don't like.

And remember, scripts are UNLIKELY to be broken by the recent change in
what gets output to a tty, because scripts don't usually direct 'ls'
output to a tty, at least not if that output is going to affect the
control flow of the script (as it is, scripts that try to parse 'ls'
output are generally already broken; there are much better tools for
handling directory contents than trying to parse 'ls' output).

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #65 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: L A Walsh <coreutils <at> tlinx.org>, Pádraig Brady
 <P <at> draigBrady.com>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, Andreas Schwab <schwab <at> linux-m68k.org>,
 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1 as
 input
Date: Mon, 9 Jan 2017 14:52:17 -0600

[Message part 1 (text/plain, inline)]

On 01/09/2017 02:39 PM, L A Walsh wrote:
> reopen 25388
> thanks
> 
> It's still a bug as it goes against adding new behaviors that
> generate different output based on destination = tty or not.

This is NOT adding a new behavior that is based on isatty() checks.  The
output may differ, but there were no new isatty() checks added,
therefore the new behavior is NOT a violation of GNU's stance that
program output should not differ based solely on whether stdout is a tty.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #68 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: L A Walsh <coreutils <at> tlinx.org>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, Andreas Schwab <schwab <at> linux-m68k.org>,
 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1 as
 input
Date: Mon, 9 Jan 2017 14:55:39 -0600

[Message part 1 (text/plain, inline)]

On 01/09/2017 01:53 PM, L A Walsh wrote:
>> And POSIX merely codified existing practice (this is nothing new - it
>> has been this way since the 70's)
> ---
>     Not anymore.
> 
>     Breaking "rm -fr ." wasn't an existing practice except
> at BSD-using dists (like BSD & SunOS).  While Solaris was SysV, since it
> was bought up, it has changed.

Please quit trying to change the topic.  My sentence about POSIX
codifying the existing practice of 'ls' using different output to tty
than it does to a pipeline is unrelated to your attempt to drag in 'rm'
behavior to every single bug report.  Whether or not 'rm' behavior
changes have occurred over time, and whether such changes were codified
by POSIX as existing practice, are not relevant to the current topic of
'ls' changing behavior.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #71 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Eric Blake <eblake <at> redhat.com>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, Andreas Schwab <schwab <at> linux-m68k.org>,
 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1
 as input
Date: Mon, 09 Jan 2017 13:22:09 -0800

Eric Blake wrote:
> On 01/09/2017 02:35 PM, L A Walsh wrote:
>>     I also stand for it being strongly against GNU
>> standards to add more such behaviors.
> 
> 'ls' did not recently add any more cases where tty output differs from
> non-tty output when all other things are equal in the default state.
> All that changed was that tty output is formatted differently than it
> has been in the past. 
----
	That is not the case.

Created dir with common files w/spaces in them from people's
Doc dirs.  I also replaced the space in "My Pictures w/a tab to
see the effect -- though it wasn't necessary.


> ls
All Settings        My  Pictures    Show Desktop.scf  browser - logitech
Bluetooth Software  My Documents  Start Menu
Local Settings      Saved Games   Virtual Machines

Then I did ls the standard way:

All Settings
Bluetooth Software
Local Settings
My  Pictures
My Documents
Saved Games
Show Desktop.scf
Start Menu
Virtual Machines
browser - logitech
===
And quoted way:

'All Settings'
'Bluetooth Software'
'Local Settings'
'My'$'\t''Pictures'
'My Documents'
'Saved Games'
'Show Desktop.scf'
'Start Menu'
'Virtual Machines'
'browser - logitech'

I cut & pasted into bash, and read them into array vars using

readarray -t std
<paste>
<ctl-d>

and the same for the quoted version but 
to "readarray -t quoted".

Now I look for the files 1 at a time in a loop:
First old way (using -gG to shorten line):
-rw-rw-r-- 1 0 Jan  9 13:06 All Settings
-rw-rw-r-- 1 0 Jan  9 13:06 Bluetooth Software
-rw-rw-r-- 1 0 Jan  9 13:06 Local Settings
-rw-rw-r-- 1 0 Jan  9 13:06 My  Pictures
-rw-rw-r-- 1 0 Jan  9 13:06 My Documents
-rw-rw-r-- 1 0 Jan  9 13:06 Saved Games
-rw-rw-r-- 1 0 Jan  9 13:06 Show Desktop.scf
-rw-rw-r-- 1 0 Jan  9 13:06 Start Menu
-rw-rw-r-- 1 0 Jan  9 13:06 Virtual Machines
-rw-rw-r-- 1 0 Jan  9 13:06 browser - logitech
---
And now the new way that you claim is "the same output":

ls: cannot access 'All Settings': No such file or directory
ls: cannot access 'Bluetooth Software': No such file or directory
ls: cannot access 'Local Settings': No such file or directory
ls: cannot access 'My'$'t''Pictures': No such file or directory
ls: cannot access 'My Documents': No such file or directory
ls: cannot access 'Saved Games': No such file or directory
ls: cannot access 'Show Desktop.scf': No such file or directory
ls: cannot access 'Start Menu': No such file or directory
ls: cannot access 'Virtual Machines': No such file or directory
ls: cannot access 'browser - logitech': No such file or directory


> your claim that ls' behavior is non-standard has been proven to be
> false; I quoted the line from POSIX that explicitly permits ls
---
	By permitting it as an exception, you are admitting it
is non-standard compared to other programs.

> implementation-defined behavior when outputting to a terminal, and if it
> is permitted by the standard, then the behavior can't be considered
> non-standard.  It may not be what you want, but that does not make it
> non-standard.
---
	
> Now you are excluding a fact.  POSIX _does_ state that the output of
> 'ls' when targetting a tty is implementation-defined,
---
	I'm not quoting posix -- I'm quoting the GNU
standards you mention.


Anyway, as demonstrated above, the output is not the same and
varies based on whether or not the destination is a tty.

Your claim that the output is the same is demonstrably false.
They are not functionally equivalent.

Message #74 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Eric Blake <eblake <at> redhat.com>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, Andreas Schwab <schwab <at> linux-m68k.org>,
 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1
 as input
Date: Mon, 09 Jan 2017 13:23:52 -0800

Eric Blake wrote:
> On 01/09/2017 01:53 PM, L A Walsh wrote:
>>> And POSIX merely codified existing practice (this is nothing new - it
>>> has been this way since the 70's)
>> ---
>>     Not anymore.
>>
>>     Breaking "rm -fr ." wasn't an existing practice except
>> at BSD-using dists (like BSD & SunOS).  While Solaris was SysV, since it
>> was bought up, it has changed.
> 
> Please quit trying to change the topic.
---
You said that posix codified existing practice and that it has been
this way since the 70's.  This is not true.

You are making overly broad statements, or, more likely, 
using indeterminant pronouns.

Message #77 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: L A Walsh <coreutils <at> tlinx.org>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, Andreas Schwab <schwab <at> linux-m68k.org>,
 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1 as
 input
Date: Mon, 9 Jan 2017 15:53:20 -0600

[Message part 1 (text/plain, inline)]

On 01/09/2017 03:23 PM, L A Walsh wrote:
> 
> 
> Eric Blake wrote:
>> On 01/09/2017 01:53 PM, L A Walsh wrote:
>>>> And POSIX merely codified existing practice (this is nothing new - it
>>>> has been this way since the 70's)
>>> ---
>>>     Not anymore.
>>>
>>>     Breaking "rm -fr ." wasn't an existing practice except
>>> at BSD-using dists (like BSD & SunOS).  While Solaris was SysV, since it
>>> was bought up, it has changed.
>>
>> Please quit trying to change the topic.
> ---
> You said that posix codified existing practice and that it has been
> this way since the 70's.  This is not true.

You're quoting me out of context.

When coupled with the sentences prior to what got quoted in this email,
I was trying to emphasize only that that POSIX codified the existing
practice of ls outputting different data to a tty than to a non-tty, and
that the behavior of different ls output to a tty has been around since
the 70s, which explains why POSIX codified that particular aspect of ls
as standard-conforming behavior.

By eliding the context, you are then attempting to broaden my statement
into me making a blanket claim that POSIX can only ever codify existing
practice (which may are may not be the case, but it was not the point I
was trying to make), by twisting my claim into something unrelated to ls
and dragging rm into the mix.

> 
> You are making overly broad statements, or, more likely, using
> indeterminant pronouns.

That was not my intent.  Email is a lousy medium for communications.

Here's the full context, for reference:

>>>
>>> Sure it is. 'ls' has done that since then 1980s. 'ls' shows
>>> multicolumn output when the output is a tty, and single-column output
>>> when piped into a pager.
>> ----
>> That would be really annoying if it were true.  It doesn't do that
>> on any of my *nix terms.  When I put it through a pager I want it to page
>> the output I just saw.  That's what it's for.
>
> But that's EXACTLY what POSIX has specified, because it has been
> existing practice for YEARS.

So in my sentence, as originally written, the pronoun of "that's
EXACTLY" is referring only to the earlier wording "'ls' has done that
since then 1980s", and not a sweeping generalization to all of POSIX.

And in my other sentence:

>>>> And POSIX merely codified existing practice (this is nothing new - it
>>>> has been this way since the 70's)

the "it" is referring, again, to the behavior of 'ls' outputting
something different for ttys.  That is, read it as "(this is nothing new
- ls having different tty output has been this way since the 70's)", and
NOT as "(this is nothing new - POSIX has been codifying existing
practice in this way since the 70's)" - particularly since POSIX did not
exist in the 70's and therefore could not have been codifying existing
practice for a duration of time that long.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #80 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Eric Blake <eblake <at> redhat.com>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, Andreas Schwab <schwab <at> linux-m68k.org>,
 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1
 as input
Date: Mon, 09 Jan 2017 14:53:10 -0800

Eric Blake wrote:
> You're quoting me out of context.

Your context was not clear.

> 
> When coupled with the sentences
----
Your "But" started that sentence responding to my statement.

While you may have thought you included all previous context, 
it was interwoven with my statements, so the context was
muddied.

>> You are making overly broad statements, or, more likely, using
>> indeterminant pronouns.
> 
> That was not my intent.  Email is a lousy medium for communications.
---
	That was how I took it, especially since in other areas
posix has had no problem being more than descriptive.

	As soon as you claimed I was dragging in unrelated items,
I figured out what you had meant.  But the thing about posix only
describing -- which was their original mission statement, to 
becoming a prescriptive body, is more than a bit of a sore spot, especially
since one or more BSDer's have gone off on me in the past about how BSD
won the standards war against SysV, since now the BSD'ers were able to
be a majority in posix and change the standards to along BSD lines
vs. SysV lines.  That made me realize what a waste of energy was
going to go into making bunches of tools incompatible with how they 
have been.  ...And it progresses...  

    I liked BSD, but now I see its followers as more than a bit 
religious, and unwilling to see anyone else's view.  It's probably no
coincidence that Apple chose a BSD-system as their base.

	As for email being a lousy medium... yeah, it's not the best,
but not the worst.  You went off on me because you thought I was changing
ls's historical behavior, when my issue is compounding those incompatibilities
by adding flaky features that can't be cut/pasted into anything.  It's
a poor option to do by default.  

Message #83 received at 25388 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: L A Walsh <coreutils <at> tlinx.org>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, Andreas Schwab <schwab <at> linux-m68k.org>,
 25388 <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1 as
 input
Date: Mon, 9 Jan 2017 16:57:22 -0600

[Message part 1 (text/plain, inline)]

On 01/09/2017 03:22 PM, L A Walsh wrote:
> 
> 
> Eric Blake wrote:
>> On 01/09/2017 02:35 PM, L A Walsh wrote:
>>>     I also stand for it being strongly against GNU
>>> standards to add more such behaviors.
>>
>> 'ls' did not recently add any more cases where tty output differs from
>> non-tty output when all other things are equal in the default state.
>> All that changed was that tty output is formatted differently than it
>> has been in the past. 
> ----
>     That is not the case.
> 
> Created dir with common files w/spaces in them from people's
> Doc dirs.  I also replaced the space in "My Pictures w/a tab to
> see the effect -- though it wasn't necessary.

Okay, I'm going to follow along, using two different versions of ls:
8.25 (Fedora 25) and 8.4 (RHEL 6).

$ mkdir -p /tmp/ls
$ cd /tmp/ls
$ touch 'a b' c

Now, let's check whether output differed based on tty in the old
version, 8.4:

$ \ls
a b  c
$ \ls | cat
a b
c
$ \ls -1 | cat
a b
c
$ \ls --version | head -n1
ls (GNU coreutils) 8.4

Yep, the output is different based on whether stdout was a tty.

Now, let's check again with a modern version:

$ \ls
'a b'  c
$ \ls | cat
a b
c
$ \ls -1 | cat
a b
c
$ \ls --version | head -n1
ls (GNU coreutils) 8.25

Yep, the output STILL differs based on whether stdout was a tty.  But
note, that in BOTH cases, the output of '\ls -1' is identical when
stdout is NOT a tty.  If you are parsing the output of ls -1 in a
script, chances are you are NOT creating a pty, and therefore the output
you are parsing did NOT change.  The only content that changed between
ls versions is the CONTENT of the tty output, but NOT the logic of
whether to use tty output, nor the content when tty output is not involved.

The GNU Coding Standards do NOT forbid the content of tty output to
change between versions (that is, the tty output of version x.1 and x.2
need not be the same), they only state that if version x.1 outputs a
particular text to a non-tty, it should default to outputting that same
text to a tty, IF that difference in behavior was not already
grandfathered in.

Let's go read the actual GNU Coding Standards, instead of paraphrasing it:

https://www.gnu.org/prep/standards/standards.html#User-Interfaces

"Likewise, please don’t make the behavior of a command-line program
depend on the type of output device it gets as standard output or
standard input. Device independence is an important principle of the
system’s design; do not compromise it merely to save someone from typing
an option now and then. (Variation in error message syntax when using a
terminal is ok, because that is a side issue that people do not depend on.)

If you think one behavior is most useful when the output is to a
terminal, and another is most useful when the output is a file or a
pipe, then it is usually best to make the default behavior the one that
is useful with output to a terminal, and have an option for the other
behavior. You can also build two different versions of the program with
different names.

There is an exception for programs whose output in certain cases is
binary data. Sending such output to a terminal is useless and can cause
trouble. If such a program normally sends its output to stdout, it
should detect, in these cases, when the output is a terminal and give an
error message instead. The -f option should override this exception,
thus permitting the output to go to the terminal.

Compatibility requires certain programs to depend on the type of output
device. It would be disastrous if ls or sh did not do so in the way all
users expect. In some of these cases, we supplement the program with a
preferred alternate version that does not depend on the output device
type. For example, we provide a dir program much like ls except that its
default output format is always multi-column format."


Nothing in there says that we can't change what ls outputs to a termianl
between versions of ls, merely that the reason that ls DEFAULTS to
different output to terminal than to non-terminal is historical.
Furthermore, it _encourages_ output to a terminal to default to being
safe in the presence of binary data (filenames can contain control
characters, which if not properly quoted are indeed disastrous to the
terminal), and does not forbid us from using a version of quoting that
is less ambiguous in newer versions of ls than it was in previous versions.

>> your claim that ls' behavior is non-standard has been proven to be
>> false; I quoted the line from POSIX that explicitly permits ls
> ---
>     By permitting it as an exception, you are admitting it
> is non-standard compared to other programs.

Huh? The fact that ls is allowed to have implementation-defined output
when writing to a tty is standard.  It may not be consistent with other
applications, but the point being discussed here wasn't consistency, but
standards-compliant.  You can't be non-standard by comparing to another
app - either the behavior of ls is permitted by the standard or it is
not permitted, regardless of what other apps do.

>     
>> Now you are excluding a fact.  POSIX _does_ state that the output of
>> 'ls' when targetting a tty is implementation-defined,
> ---
>     I'm not quoting posix -- I'm quoting the GNU
> standards you mention.

I actually quoted it above.  So far, you have only paraphrased it, and I
think you are trying to claim that the GNU Coding Standards do not let
us change the tty-only output between versions, but that is not what it
says.

> 
> 
> Anyway, as demonstrated above, the output is not the same and
> varies based on whether or not the destination is a tty.

Indeed - and we've been telling you that default ls output has ALWAYS
varied based on whether or not the destination is a tty.  The fact that
it STILL varies (although with different content than in older versions
of ls) is nothing new.  We are not violating the requirement that no NEW
code should have a difference based on tty, because we already HAVE a
difference based on tty.

> 
> Your claim that the output is the same is demonstrably false.

No, you have only managed to prove that the tty-only output of ls has
changed between versions, and NOT that the tty-only and non-tty output
used to be the same but now differ.  The tty-only and non-tty output
have ALWAYS been different, and I was not claiming that they have been
the same.

> They are not functionally equivalent.

You are trying to feed tty output into a shell script. In general, this
is not portable.  But this is not ls' fault.  It has NEVER been portable
to try and parse the tty-only output of ls in a shell script.  Just
because the outcome of trying to parse non-portable output has changed
doesn't mean that the output is broken - rather, your attempt to feed
non-portable output to the shell in the first place is broken.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #90 received at 25388-done <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: L A Walsh <coreutils <at> tlinx.org>
Cc: Andreas Schwab <schwab <at> linux-m68k.org>, 25388-done <at> debbugs.gnu.org
Subject: Re: bug#25388: Bug in ls, kills existing scripts reading "ls" -1 as
 input
Date: Thu, 13 Dec 2018 13:29:08 -0700

Hello,

>>> 'ls' did not recently add any more cases where tty output differs from
>>> non-tty output when all other things are equal in the default state.
>>> All that changed was that tty output is formatted differently than it
>>> has been in the past.
>> ----

We created a summary of common issues and FAQs
regarding the quoting change in ls(1):
  https://www.gnu.org/software/coreutils/quotes.html

If there is an issue that is not addressed there,
please send an email to coreutils <at> gnu.org .

regards,
 - assaf

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.