GNU bug report logs - #26574
v4.4: POSIX violation with respect to output of a trailing newline, even with --posix

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 26574 in the body.
You can then email your comments to 26574 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Michael Klement <michael.klement <at> usa.net>
To: bug-sed <at> gnu.org
Subject: v4.4: POSIX violation with respect to output of a trailing newline,
 even with --posix
Date: Wed, 19 Apr 2017 21:43:15 -0400

[Message part 1 (text/plain, inline)]

$ sed --version
sed (GNU sed) 4.4

The POSIX spec. <http://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html> states:
"Whenever the pattern space is written to standard output or a named file, sed shall immediately follow it with a <newline>."

While GNU Sed's default behavior of preserving the trailing-newline status of the input's last line is defensible and can be helpful,
it should exhibit POSIX-compliant behavior when invoked with --posix.

# Acceptable default behavior - the no-trailing-newline status of the input is preserved.
$ printf 'a' | sed '' | od -t x1
0000000    61


# SHOULD include a trailing newline, per POSIX, but currently doesn't.
$ printf 'a' | sed --posix '' | od -t x1
0000000    61



Regards,

Michael

[Message part 2 (text/html, inline)]

Message #12 received at 26574-done <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Michael Klement <michael.klement <at> usa.net>, 26574-done <at> debbugs.gnu.org
Subject: Re: bug#26574: v4.4: POSIX violation with respect to output of a
 trailing newline, even with --posix
Date: Thu, 20 Apr 2017 05:42:08 -0500

[Message part 1 (text/plain, inline)]

tag 26574 notabug
thanks

On 04/19/2017 08:43 PM, Michael Klement wrote:
> $ sed --version
> sed (GNU sed) 4.4
> 
> The POSIX spec. <http://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html> states:
> "Whenever the pattern space is written to standard output or a named file, sed shall immediately follow it with a <newline>."
> 
> While GNU Sed's default behavior of preserving the trailing-newline status of the input's last line is defensible and can be helpful,
> it should exhibit POSIX-compliant behavior when invoked with --posix.

POSIX also requires that input given to sed be text files:

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html
"The input files shall be text files."

And per the definition of text file, ALL input lines must have a
trailing newline in the first place:
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html
"3.403 Text File
A file that contains characters organized into zero or more lines. The
lines do not contain NUL characters and none can exceed {LINE_MAX} bytes
in length, including the <newline> character. Although POSIX.1-2008 does
not distinguish between text files and binary files (see the ISO C
standard), many utilities only produce predictable or meaningful output
when operating on text files. The standard utilities that have such
restrictions always specify "text files" in their STDIN or INPUT FILES
sections."

"3.206 Line
A sequence of zero or more non- <newline> characters plus a terminating
<newline> character."

Input that does NOT end in a trailing newline is NOT a text file, and
therefore is NOT a POSIX-compliant use of sed, and therefore, sed
--posix need not do anything different with it because you are already
outside the bounds of what POSIX requires.

Therefore, I don't think you have a case for changing any behavior, at
least not on the grounds of appealing to POSIX, so I'm marking this as
not a bug, but feel free to continue discussion.  If anything, the only
change I would make is have 'sed --posix' error out on non-text input,
to call attention to the user's attempt to feed non-posix-compliant data
to sed.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #15 received at 26574-done <at> debbugs.gnu.org (full text, mbox):

From: Michael Klement <michael.klement <at> usa.net>
To: Eric Blake <eblake <at> redhat.com>
Cc: 26574-done <at> debbugs.gnu.org
Subject: Re: bug#26574: v4.4: POSIX violation with respect to output of a
 trailing newline, even with --posix
Date: Thu, 20 Apr 2017 12:36:03 -0400

[Message part 1 (text/plain, inline)]

Thanks for the detailed feedback, Eric.

The POSIX spec. is, unfortunately, vague on this topic:

The definition of a line (which you quote) is complemented with the definition of an incomplete line <http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_195>:

> A sequence of one or more non- <newline> characters at the end of the file.


So while the standard is aware of this possibility and gives it a name that suggests it is a kind of line, but something's missing, there is precious little behavior prescribed with respect to such incomplete lines.

So we have:

sed's "input files shall be text files."
a text file contains "characters organized into zero or more lines"

Beyond the "zero or more lines", the only restrictions placed on what constitutes a text file <http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_403> are:
" The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character. "

If you interpret the word "lines" in the phrase "zero or more lines" to mean complete lines only (which is reasonable), then indeed any file that ends in an incomplete line is not a text file.

I really wish the spec. were more explicit about incomplete lines.

>   If anything, the only
> change I would make is have 'sed --posix' error out on non-text input,
> to call attention to the user's attempt to feed non-posix-compliant data
> to sed.


That is definitely an option, but perhaps intuitive understanding and historical practice / other implementations could be considered instead:

Intuitively, a file containing text with an incomplete line is obviously still a text file - just one that has no trailing \n, so treating incomplete lines (mostly) like lines makes sense.
In practice, most utilities still read the incomplete line - the shell's read builtin being a notable exception.
wc is an interesting case, which doesn't count an incomplete line as a line (the spec <http://pubs.opengroup.org/onlinepubs/9699919799/utilities/wc.html>. is actually unambiguous there and mandates counting the newlines), yet still counts its words and characters/bytes.
BSD/macOS Sed is a mostly POSIX-features-only implementation, and it always appends a trailing \n, even when encountering an incomplete line. (On the flip side, that makes it fundamentally unsuited to operating on binary files - unlike GNU Sed).
I'm not sure about other implementations (or even if there are any that still matter today).

So, as a compromise, GNU sed --posix could treat files with an incomplete line as text files, as long as the incomplete line contains no NULs and contains at most getconf LINE_MAX - 1 characters.

Maybe the issue at hand is rarely of concern in the real world, but I've stumbled over it on several occasions when writing portable Sed commands (at least portable between Linux and macOS).
This issue and the infamous -i option incompatibility (which probably will never go away) are what get in the way of writing such commands.

Thanks,

Michael






> On Apr 20, 2017, at 6:42 AM, Eric Blake <eblake <at> redhat.com> wrote:
> 
> tag 26574 notabug
> thanks
> 
> On 04/19/2017 08:43 PM, Michael Klement wrote:
>> $ sed --version
>> sed (GNU sed) 4.4
>> 
>> The POSIX spec. <http://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html> states:
>> "Whenever the pattern space is written to standard output or a named file, sed shall immediately follow it with a <newline>."
>> 
>> While GNU Sed's default behavior of preserving the trailing-newline status of the input's last line is defensible and can be helpful,
>> it should exhibit POSIX-compliant behavior when invoked with --posix.
> 
> POSIX also requires that input given to sed be text files:
> 
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html
> "The input files shall be text files."
> 
> And per the definition of text file, ALL input lines must have a
> trailing newline in the first place:
> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html
> "3.403 Text File
> A file that contains characters organized into zero or more lines. The
> lines do not contain NUL characters and none can exceed {LINE_MAX} bytes
> in length, including the <newline> character. Although POSIX.1-2008 does
> not distinguish between text files and binary files (see the ISO C
> standard), many utilities only produce predictable or meaningful output
> when operating on text files. The standard utilities that have such
> restrictions always specify "text files" in their STDIN or INPUT FILES
> sections."
> 
> "3.206 Line
> A sequence of zero or more non- <newline> characters plus a terminating
> <newline> character."
> 
> Input that does NOT end in a trailing newline is NOT a text file, and
> therefore is NOT a POSIX-compliant use of sed, and therefore, sed
> --posix need not do anything different with it because you are already
> outside the bounds of what POSIX requires.
> 
> Therefore, I don't think you have a case for changing any behavior, at
> least not on the grounds of appealing to POSIX, so I'm marking this as
> not a bug, but feel free to continue discussion.  If anything, the only
> change I would make is have 'sed --posix' error out on non-text input,
> to call attention to the user's attempt to feed non-posix-compliant data
> to sed.
> 
> -- 
> Eric Blake, Principal Software Engineer
> Red Hat, Inc.           +1-919-301-3266
> Virtualization:  qemu.org | libvirt.org
> 

[Message part 2 (text/html, inline)]

Message #18 received at 26574-done <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Michael Klement <michael.klement <at> usa.net>
Cc: 26574-done <at> debbugs.gnu.org
Subject: Re: bug#26574: v4.4: POSIX violation with respect to output of a
 trailing newline, even with --posix
Date: Thu, 20 Apr 2017 11:46:15 -0500

[Message part 1 (text/plain, inline)]

On 04/20/2017 11:36 AM, Michael Klement wrote:
> Thanks for the detailed feedback, Eric.
> 
> The POSIX spec. is, unfortunately, vague on this topic:
> 
> The definition of a line (which you quote) is complemented with the definition of an incomplete line <http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_195>:
> 
>> A sequence of one or more non- <newline> characters at the end of the file.
> 
> 
> So while the standard is aware of this possibility and gives it a name that suggests it is a kind of line, but something's missing, there is precious little behavior prescribed with respect to such incomplete lines.
> 

You're welcome to submit a bug report to get POSIX to more clearly word
its intentions that a file with an incomplete line is NOT a text file
(http://austingroupbugs.net/main_page.php), but everyone on the Austin
Group (myself included) has already agreed that the intention is there
(even if the wording could be improved): Omitting a trailing newline
causes sed to enter into the realm of undefined behavior - and this is
BECAUSE there are existing sed implementations that behave differently
when a trailing newline is omitted.  Some do not do anything with an
incomplete line (sed behaves as though the file were truncated at the
last newline).

> So we have:
> 
> sed's "input files shall be text files."
> a text file contains "characters organized into zero or more lines"
> 
> Beyond the "zero or more lines", the only restrictions placed on what constitutes a text file <http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_403> are:
> " The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character. "
> 
> If you interpret the word "lines" in the phrase "zero or more lines" to mean complete lines only (which is reasonable), then indeed any file that ends in an incomplete line is not a text file.
> 
> I really wish the spec. were more explicit about incomplete lines.

As I said, you're welcome to propose a bug report with suggested wording
improvements.

> 
>>   If anything, the only
>> change I would make is have 'sed --posix' error out on non-text input,
>> to call attention to the user's attempt to feed non-posix-compliant data
>> to sed.
> 
> 
> That is definitely an option, but perhaps intuitive understanding and historical practice / other implementations could be considered instead:
> 
> Intuitively, a file containing text with an incomplete line is obviously still a text file

Not per the POSIX definition of a text file.

It is still a file, but no longer a text file.

It wouldn't be the first time intuition has been wrong.

> wc is an interesting case, which doesn't count an incomplete line as a line (the spec <http://pubs.opengroup.org/onlinepubs/9699919799/utilities/wc.html>. is actually unambiguous there and mandates counting the newlines),

Indeed, wc is a good example of how the POSIX writers specifically went
out of their way to describe behaviors of programs that MUST be
consistent when presented with a non-text file; as well as the escape
clause that for all other programs (including sed) that require text
file inputs, the behavior is intentionally unspecified if the trailing
newline is not present.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #21 received at 26574-done <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: Eric Blake <eblake <at> redhat.com>
Cc: Michael Klement <michael.klement <at> usa.net>, 26574-done <at> debbugs.gnu.org
Subject: Re: bug#26574: v4.4: POSIX violation with respect to output of a
 trailing newline, even with --posix
Date: Thu, 20 Apr 2017 18:32:22 +0000

Hello,

On Thu, Apr 20, 2017 at 11:46:15AM -0500, Eric Blake wrote:
>On 04/20/2017 11:36 AM, Michael Klement wrote:
>> Thanks for the detailed feedback, Eric.
>>
>> The POSIX spec. is, unfortunately, vague on this topic:
>>
>> The definition of a line (which you quote) is complemented with the definition of an incomplete line <http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_195>:
>>
>>> A sequence of one or more non- <newline> characters at the end of the file.
>>
>>
>> So while the standard is aware of this possibility and gives it a name that suggests it is a kind of line, but something's missing, there is precious little behavior prescribed with respect to such incomplete lines.
>>
>
>You're welcome to submit a bug report to get POSIX to more clearly word
>its intentions that a file with an incomplete line is NOT a text file
>(http://austingroupbugs.net/main_page.php), but everyone on the Austin
>Group (myself included) has already agreed that the intention is there
>(even if the wording could be improved): Omitting a trailing newline
>causes sed to enter into the realm of undefined behavior - and this is
>BECAUSE there are existing sed implementations that behave differently
>when a trailing newline is omitted.  Some do not do anything with an
>incomplete line (sed behaves as though the file were truncated at the
>last newline).
>

For completeness, here's the behaviour of several implementaions:

sed implementations that do not add a newline (like gnu sed):
  FreeBSD 10
  OpenBSD 5.9
  BusyBox 1.22
  ToyBox 7.2
  AIX 7

sed implementations that do add a new line:
  NetBSD 7.0
  Heirloom

SunOS 5.11's sed prints nothing if there is no newline:
  $ printf 'a' | sed '' | od -tx1
  0000000
  $ printf 'a\n' | sed '' | od -tx1
  0000000 61 0a
  0000002
  $ uname -a
  SunOS unstable11s 5.11 11.2 sun4u sparc SUNW,SPARC-Enterprise
  $ which sed
  /usr/bin/sed


The behaviour (of processing a file without newline at the last 
line) also differs in other programs/languages/implementations:

  $ printf a | perl -npe '' | od -tx1
  0000000 61
  0000001

  $ printf a | perl -lnpe '' | od -tx1
  0000000 61 0a
  0000002

  $ printf a | awk '{print}' | od -tx1
  0000000 61 0a
  0000002

  $ printf 'a' | sh -c 'while read A ; do echo $A ; done' | od -tx1
  0000000

  $ printf 'a' \
     | python3 -c 'import sys; [print(x,end="") for x in sys.stdin]' \
     | od -tx1
  0000000 61
  0000001

  $ printf a | uniq-gnu | od -t x1
  0000000 61 0a
  0000002

  $ printf a | uniq-freebsd-11 | od -t x1
  0000000    61
  0000001

  $ printf a | cut-gnu -f1 | od -tx1
  0000000 61 0a
  0000002

  $ printf a | cut-freebsd-11 -f1 | od -tx1
  0000000    61
  0000001

  $ printf a | sort | od -t x1
  0000000 61 0a
  0000002


And this reinforces what Eric wrote: there is simply no
'one correct' (or agreed-upon) way to deal with files without newlines 
on the last line.


regards,
- assaf

Message #24 received at 26574-done <at> debbugs.gnu.org (full text, mbox):

From: Michael Klement <michael.klement <at> usa.net>
To: Assaf Gordon <assafgordon <at> gmail.com>
Cc: Eric Blake <eblake <at> redhat.com>, 26574-done <at> debbugs.gnu.org
Subject: Re: bug#26574: v4.4: POSIX violation with respect to output of a
 trailing newline, even with --posix
Date: Thu, 20 Apr 2017 15:32:13 -0400

[Message part 1 (text/plain, inline)]

Thanks for digging into this, it indeed illustrates the point well.

Just for the record:

Here's what I get on FreeBSD 10.1.2 and on macOS 10.12.4:

$ printf 'a' | sed '' | od -tx1
0000000    61  0a                                                        
0000002

macOS typically comes with an older version of the BSD implementation (which doesn't support --version, but the man pages are dated June 20, 2014 and May 10, 2005, respectively).

Another (minor) point of interest:

On macOS 10.12.4 (but not FreeBSD 10.1.2), Sed chokes on bytes that aren't valid in UTF-8 encoding, when using regex-based functionality:

$ printf '\xfc\n' | sed  -n '/./p'
sed: RE error: illegal byte sequence




> On Apr 20, 2017, at 2:32 PM, Assaf Gordon <assafgordon <at> gmail.com> wrote:
> 
> Hello,
> 
> On Thu, Apr 20, 2017 at 11:46:15AM -0500, Eric Blake wrote:
>> On 04/20/2017 11:36 AM, Michael Klement wrote:
>>> Thanks for the detailed feedback, Eric.
>>> 
>>> The POSIX spec. is, unfortunately, vague on this topic:
>>> 
>>> The definition of a line (which you quote) is complemented with the definition of an incomplete line <http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_195>:
>>> 
>>>> A sequence of one or more non- <newline> characters at the end of the file.
>>> 
>>> 
>>> So while the standard is aware of this possibility and gives it a name that suggests it is a kind of line, but something's missing, there is precious little behavior prescribed with respect to such incomplete lines.
>>> 
>> 
>> You're welcome to submit a bug report to get POSIX to more clearly word
>> its intentions that a file with an incomplete line is NOT a text file
>> (http://austingroupbugs.net/main_page.php), but everyone on the Austin
>> Group (myself included) has already agreed that the intention is there
>> (even if the wording could be improved): Omitting a trailing newline
>> causes sed to enter into the realm of undefined behavior - and this is
>> BECAUSE there are existing sed implementations that behave differently
>> when a trailing newline is omitted.  Some do not do anything with an
>> incomplete line (sed behaves as though the file were truncated at the
>> last newline).
>> 
> 
> For completeness, here's the behaviour of several implementaions:
> 
> sed implementations that do not add a newline (like gnu sed):
>  FreeBSD 10
>  OpenBSD 5.9
>  BusyBox 1.22
>  ToyBox 7.2
>  AIX 7
> 
> sed implementations that do add a new line:
>  NetBSD 7.0
>  Heirloom
> 
> SunOS 5.11's sed prints nothing if there is no newline:
>  $ printf 'a' | sed '' | od -tx1
>  0000000
>  $ printf 'a\n' | sed '' | od -tx1
>  0000000 61 0a
>  0000002
>  $ uname -a
>  SunOS unstable11s 5.11 11.2 sun4u sparc SUNW,SPARC-Enterprise
>  $ which sed
>  /usr/bin/sed
> 
> 
> The behaviour (of processing a file without newline at the last line) also differs in other programs/languages/implementations:
> 
>  $ printf a | perl -npe '' | od -tx1
>  0000000 61
>  0000001
> 
>  $ printf a | perl -lnpe '' | od -tx1
>  0000000 61 0a
>  0000002
> 
>  $ printf a | awk '{print}' | od -tx1
>  0000000 61 0a
>  0000002
> 
>  $ printf 'a' | sh -c 'while read A ; do echo $A ; done' | od -tx1
>  0000000
> 
>  $ printf 'a' \
>     | python3 -c 'import sys; [print(x,end="") for x in sys.stdin]' \
>     | od -tx1
>  0000000 61
>  0000001
> 
>  $ printf a | uniq-gnu | od -t x1
>  0000000 61 0a
>  0000002
> 
>  $ printf a | uniq-freebsd-11 | od -t x1
>  0000000    61
>  0000001
> 
>  $ printf a | cut-gnu -f1 | od -tx1
>  0000000 61 0a
>  0000002
> 
>  $ printf a | cut-freebsd-11 -f1 | od -tx1
>  0000000    61
>  0000001
> 
>  $ printf a | sort | od -t x1
>  0000000 61 0a
>  0000002
> 
> 
> And this reinforces what Eric wrote: there is simply no
> 'one correct' (or agreed-upon) way to deal with files without newlines on the last line.
> 
> 
> regards,
> - assaf

[Message part 2 (text/html, inline)]

Message #27 received at 26574-done <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Michael Klement <michael.klement <at> usa.net>,
 Assaf Gordon <assafgordon <at> gmail.com>
Cc: 26574-done <at> debbugs.gnu.org
Subject: Re: bug#26574: v4.4: POSIX violation with respect to output of a
 trailing newline, even with --posix
Date: Thu, 20 Apr 2017 14:36:39 -0500

[Message part 1 (text/plain, inline)]

On 04/20/2017 02:32 PM, Michael Klement wrote:

> On macOS 10.12.4 (but not FreeBSD 10.1.2), Sed chokes on bytes that aren't valid in UTF-8 encoding, when using regex-based functionality:
> 
> $ printf '\xfc\n' | sed  -n '/./p'
> sed: RE error: illegal byte sequence
> 

That's locale dependent (should not happen with LC_ALL=C) - but it
illustrates another nice point about POSIX text files: a text file may
not have encoding errors, but as a corollary of that fact, there exist
files which are text files in some locales but binary files in others!

The behavior of sed is only specified when you have no encoding errors,
so your choice of locale can indeed affect whether you get output that
you wanted.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.