GNU bug report logs - #18808
tail: implement 'tail -r' as synonym for 'tac'

Previous Next

Package: coreutils;

Reported by: Eric Blake <eblake <at> redhat.com>

Date: Thu, 23 Oct 2014 16:40:03 UTC

Severity: wishlist

To reply to this bug, email your comments to 18808 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-coreutils <at> gnu.org:
bug#18808; Package coreutils. (Thu, 23 Oct 2014 16:40:04 GMT) Full text and rfc822 format available.

Acknowledgement sent to Eric Blake <eblake <at> redhat.com>:
New bug report received and forwarded. Copy sent to bug-coreutils <at> gnu.org. (Thu, 23 Oct 2014 16:40:05 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: bug-coreutils <at> gnu.org
Subject: implement 'tail -r' as synonym for 'tac'
Date: Thu, 23 Oct 2014 10:39:26 -0600
[Message part 1 (text/plain, inline)]
POSIX is considering standardizing the Solaris/BSD behavior of 'tail -r'
rather than adding the GNU extension of 'tac' as a separate utility:
http://austingroupbugs.net/view.php?id=877

This bug report serves as a reminder that we need to add 'tail -r'
support to coreutils, as well as a request for reviewers for the current
state of the proposal to make sure that it is not an onerous burden for
adding that code.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Information forwarded to bug-coreutils <at> gnu.org:
bug#18808; Package coreutils. (Thu, 23 Oct 2014 16:54:02 GMT) Full text and rfc822 format available.

Message #8 received at 18808 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: 18808 <at> debbugs.gnu.org
Subject: Re: bug#18808: implement 'tail -r' as synonym for 'tac'
Date: Thu, 23 Oct 2014 10:53:13 -0600
[Message part 1 (text/plain, inline)]
On 10/23/2014 10:39 AM, Eric Blake wrote:
> POSIX is considering standardizing the Solaris/BSD behavior of 'tail -r'
> rather than adding the GNU extension of 'tac' as a separate utility:
> http://austingroupbugs.net/view.php?id=877
> 
> This bug report serves as a reminder that we need to add 'tail -r'
> support to coreutils, as well as a request for reviewers for the current
> state of the proposal to make sure that it is not an onerous burden for
> adding that code.
> 

Implementation-wise, I'm thinking the easiest thing might be to turn
tac.c into a shared library-style file (similar to how copy.c is shared
among multiple programs), add a 'tac -n' option that limits output to a
fixed number of "lines" (or rather, a fixed number of occurrences of
'--separator'-delimited segments), then have the new 'tail -r' call into
the library code with separator hard-coded to newline, along with
sufficient checks that tail's new -r can only be used with -n with no
sign on the number (at least, as a first cut; we can certainly support
more combinations than what POSIX wants to require, if we can determine
sane semantics and easy implementations for those combinations).

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Information forwarded to bug-coreutils <at> gnu.org:
bug#18808; Package coreutils. (Thu, 23 Oct 2014 18:36:01 GMT) Full text and rfc822 format available.

Message #11 received at 18808 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Eric Blake <eblake <at> redhat.com>
Cc: 18808 <at> debbugs.gnu.org
Subject: Re: bug#18808: implement 'tail -r' as synonym for 'tac'
Date: Thu, 23 Oct 2014 19:35:30 +0100
On 10/23/2014 05:53 PM, Eric Blake wrote:
> On 10/23/2014 10:39 AM, Eric Blake wrote:
>> POSIX is considering standardizing the Solaris/BSD behavior of 'tail -r'
>> rather than adding the GNU extension of 'tac' as a separate utility:
>> http://austingroupbugs.net/view.php?id=877
>>
>> This bug report serves as a reminder that we need to add 'tail -r'
>> support to coreutils, as well as a request for reviewers for the current
>> state of the proposal to make sure that it is not an onerous burden for
>> adding that code.
>>
> 
> Implementation-wise, I'm thinking the easiest thing might be to turn
> tac.c into a shared library-style file (similar to how copy.c is shared
> among multiple programs), add a 'tac -n' option that limits output to a
> fixed number of "lines" (or rather, a fixed number of occurrences of
> '--separator'-delimited segments), then have the new 'tail -r' call into
> the library code with separator hard-coded to newline, along with
> sufficient checks that tail's new -r can only be used with -n with no
> sign on the number (at least, as a first cut; we can certainly support
> more combinations than what POSIX wants to require, if we can determine
> sane semantics and easy implementations for those combinations).

The above approach sounds sensible if we were to do this.

Now the only reason I see would be for compat with solaris and BSD.

I don't think they should have included that functionality though
as there isn't much complementary with the other tail functions
(being incompat with -f for example) and so is better split out as a separate util.

The only small advantage of integration is a tiny perf benefit
when specifying a number of lines, but really `tail -r -n$num $file`
is of minimal improvement over `tac $file | head -n$num`
(though would be better than `tail -n$num | tac` as it avoids buffering).

As a side note I see that solaris doesn't even support non seekable input mode
(though BSD does):

  $ printf "%s\n" 1 2 3 | tail -r -2
  tail: cannot open input

Pádraig.




Information forwarded to bug-coreutils <at> gnu.org:
bug#18808; Package coreutils. (Thu, 23 Oct 2014 19:04:02 GMT) Full text and rfc822 format available.

Message #14 received at 18808 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Pádraig Brady <P <at> draigbrady.com>
Cc: 18808 <at> debbugs.gnu.org
Subject: Re: bug#18808: implement 'tail -r' as synonym for 'tac'
Date: Thu, 23 Oct 2014 13:03:54 -0600
[Message part 1 (text/plain, inline)]
On 10/23/2014 12:35 PM, Pádraig Brady wrote:
> 
> Now the only reason I see would be for compat with solaris and BSD.
> 
> I don't think they should have included that functionality though
> as there isn't much complementary with the other tail functions
> (being incompat with -f for example) and so is better split out as a separate util.

Yeah, today's Austin Group call included a debate about whether
standardizing 'tac' would be smarter than 'tail -r', but the consensus
among those on the call was that since:
1. more implementations already have 'tail -r' than 'tac'
2. standardizing new requirements to an existing utility is easier than
adding a completely new utility (if only slightly, due to less paperwork)
3. GNU code tends to be the most adaptable

therefore, 'tail -r' won the vote, and the result of the meeting was an
action to me to see how adaptable GNU really is, and whether we can
indeed quickly implement 'tail -r'.

> 
> The only small advantage of integration is a tiny perf benefit
> when specifying a number of lines, but really `tail -r -n$num $file`
> is of minimal improvement over `tac $file | head -n$num`
> (though would be better than `tail -n$num | tac` as it avoids buffering).

The fact that we DON'T have 'tac -n' is what intrigues me most; that's
the one thing that 'tail -r -n' can do in a single process that GNU
can't.  Of course, it is trivial to implement it by doing what tac has
always done and then stop output after -n is exceeded, and easy on
seekable files as well.  But on non-seekable files, it may lend to an
interesting optimization where knowing that the output will be limited
to n lines makes it easier to set up a rolling window of the most recent
n lines seen rather than having to dump out to a temporary file when the
file proves to be larger than the current hueristics of when tac stays
in memory.

> 
> As a side note I see that solaris doesn't even support non seekable input mode
> (though BSD does):
> 
>   $ printf "%s\n" 1 2 3 | tail -r -2
>   tail: cannot open input

Careful, which tail are you testing?  From your output, I'm guessing you
are trying to open the file named './-2' rather than limiting -r to a
line count.  We made several observations during the Austin Group that
/usr/bin/tail and /usr/xpg4/bin/tail behave differently.

$ /usr/bin/tail -rc
usage: tail [+/-[n][lbc][f]] [file]
       tail [+/-[n][l][r|f]] [file]
$ printf "%s\n" 1 2 3 | /usr/bin/tail -2r
3
2
$ printf "%s\n" 1 2 3 | /usr/xpg4/bin/tail -r -n2
3
2
$ printf "%s\n" 1 2 3 | /usr/xpg4/bin/tail -r -2
3
2

Curiously, Solaris usage also has a bug, as this command line SHOULD be
supported, per the output:

$ printf "%s\n" 1 2 3 | /usr/xpg4/bin/tail -r -c2
usage: tail [-f|-r] [-c number | -n number] [file]
       tail [+/-[number][lbc][f]] [file]
       tail [+/-[number][l][r|f]] [file]

But since it is not supported, the POSIX proposal does not require it,
and therefore, we don't have to worry about it either :)


-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Information forwarded to bug-coreutils <at> gnu.org:
bug#18808; Package coreutils. (Thu, 23 Oct 2014 22:00:03 GMT) Full text and rfc822 format available.

Message #17 received at 18808 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Eric Blake <eblake <at> redhat.com>
Cc: 18808 <at> debbugs.gnu.org
Subject: Re: bug#18808: implement 'tail -r' as synonym for 'tac'
Date: Thu, 23 Oct 2014 22:59:11 +0100
On 10/23/2014 08:03 PM, Eric Blake wrote:
> On 10/23/2014 12:35 PM, Pádraig Brady wrote:
>>
>> Now the only reason I see would be for compat with solaris and BSD.
>>
>> I don't think they should have included that functionality though
>> as there isn't much complementary with the other tail functions
>> (being incompat with -f for example) and so is better split out as a separate util.
> 
> Yeah, today's Austin Group call included a debate about whether
> standardizing 'tac' would be smarter than 'tail -r', but the consensus
> among those on the call was that since:

I understand these 3 arguments, though don't necessarily agree with any.

> 1. more implementations already have 'tail -r' than 'tac'

More implementations but definitely not more installations.
POSIX should prefer the most popular interface in my biased opinion.

> 2. standardizing new requirements to an existing utility is easier than
> adding a completely new utility (if only slightly, due to less paperwork)

meh, paperwork should not impact on interface

> 3. GNU code tends to be the most adaptable

True, though while pragmatic should also not dictate interface.

Saying that, I'm 50:50 for implementing `tail -r` for compat reasons.

> therefore, 'tail -r' won the vote, and the result of the meeting was an
> action to me to see how adaptable GNU really is, and whether we can
> indeed quickly implement 'tail -r'.
> 
>>
>> The only small advantage of integration is a tiny perf benefit
>> when specifying a number of lines, but really `tail -r -n$num $file`
>> is of minimal improvement over `tac $file | head -n$num`
>> (though would be better than `tail -n$num | tac` as it avoids buffering).
> 
> The fact that we DON'T have 'tac -n' is what intrigues me most; that's
> the one thing that 'tail -r -n' can do in a single process that GNU
> can't.  Of course, it is trivial to implement it by doing what tac has
> always done and then stop output after -n is exceeded, and easy on
> seekable files as well.  But on non-seekable files, it may lend to an
> interesting optimization where knowing that the output will be limited
> to n lines makes it easier to set up a rolling window of the most recent
> n lines seen rather than having to dump out to a temporary file when the
> file proves to be larger than the current hueristics of when tac stays
> in memory.

-n wouldn't really have any impact on that better implementation.
I.E. ideally we should have an adaptive buffer internally
that after 128KiB or so (the current io_blksize()) would
start buffering to file, irrespective of -n.
That would deal with large lines and large -n.
Generally such an adaptive switch is optimal in various utils,
to operate efficiently with small inputs, but scalably with large ones.

>> As a side note I see that solaris doesn't even support non seekable input mode
>> (though BSD does):
>>
>>   $ printf "%s\n" 1 2 3 | tail -r -2
>>   tail: cannot open input
> 
> Careful, which tail are you testing?  From your output, I'm guessing you
> are trying to open the file named './-2' rather than limiting -r to a
> line count.  We made several observations during the Austin Group that
> /usr/bin/tail and /usr/xpg4/bin/tail behave differently.
> 
> $ /usr/bin/tail -rc
> usage: tail [+/-[n][lbc][f]] [file]
>        tail [+/-[n][l][r|f]] [file]
> $ printf "%s\n" 1 2 3 | /usr/bin/tail -2r
> 3
> 2
> $ printf "%s\n" 1 2 3 | /usr/xpg4/bin/tail -r -n2
> 3
> 2
> $ printf "%s\n" 1 2 3 | /usr/xpg4/bin/tail -r -2
> 3
> 2

Ugh right, that's the case here too.
How confusing.

thanks,
Pádraig.




Information forwarded to bug-coreutils <at> gnu.org:
bug#18808; Package coreutils. (Thu, 23 Oct 2014 22:53:02 GMT) Full text and rfc822 format available.

Message #20 received at 18808 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Pádraig Brady <P <at> draigbrady.com>
Cc: 18808 <at> debbugs.gnu.org
Subject: Re: bug#18808: implement 'tail -r' as synonym for 'tac'
Date: Thu, 23 Oct 2014 16:52:28 -0600
[Message part 1 (text/plain, inline)]
On 10/23/2014 03:59 PM, Pádraig Brady wrote:
>> 1. more implementations already have 'tail -r' than 'tac'
> 
> More implementations but definitely not more installations.
> POSIX should prefer the most popular interface in my biased opinion.

Hey, I like being biased towards free software, as well :)  Popularity
does have some impact when considering what to standardize, and the
Austin Group has been very good at considering GNU behavior to at least
ensure that new requirements in the standard do not conflict with GNU.

But to be pedantic, probably a majority of POSIX-certified installations
have 'tail -r' (maybe as many as 100%? 'tail -r' dates back to at least
1980, before POSIX was even a dream, and is on more flavors of Unix than
just Solaris) while very few POSIX-certified installations have 'tac'
(no one has yet certified a GNU/Linux platform, to my knowledge; and
while some POSIX systems like Solaris include various GNU tools in a
default installation, GNU tac doesn't tend to be one of them).

Sadly, even though many pieces of GNU software try hard to comply with
POSIX, no one has yet ironed out all the remaining warts to use GNU as a
POSIX-certified system (and it's not a trivial task; some of the warts
are embedded fairly deeply in the kernel, such as behavior of
unlink("symlink-to-dir/") being different than the current POSIX
requirements).  So among existing POSIX-certified systems, 'tail -r' is
the more popular extension at the moment.

> Saying that, I'm 50:50 for implementing `tail -r` for compat reasons.

Yes, this alone is reason enough to implement it in GNU, regardless of
the direction POSIX takes.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Information forwarded to bug-coreutils <at> gnu.org:
bug#18808; Package coreutils. (Fri, 24 Oct 2014 06:12:02 GMT) Full text and rfc822 format available.

Message #23 received at 18808 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: Eric Blake <eblake <at> redhat.com>, Pádraig Brady
 <P <at> draigbrady.com>
Cc: 18808 <at> debbugs.gnu.org
Subject: Re: bug#18808: implement 'tail -r' as synonym for 'tac'
Date: Fri, 24 Oct 2014 08:11:11 +0200
On 10/24/2014 12:52 AM, Eric Blake wrote:
> On 10/23/2014 03:59 PM, Pádraig Brady wrote:
> [...].  So among existing POSIX-certified systems, 'tail -r' is
> the more popular extension at the moment.
> 
>> Saying that, I'm 50:50 for implementing `tail -r` for compat reasons.
> 
> Yes, this alone is reason enough to implement it in GNU, regardless of
> the direction POSIX takes.

I'm also worried about such an extension.  'tail -r' might be a useful
thing - when -r is the only option being used, but looking at all the
other options GNU tail already got over the years, adding -r doesn't
fit well.  You proposed to make -n available to tac and then jump into
it from 'tail -r'.  Well, what about -c? --follow={name|descriptor}?
--retry, --pid=PID?  To make -r fit well into tail, we'd have to add
every single feature of it to tac.  This sounds not like a good idea.
tail does a good job, and so does tac.  I'd almost prefer to fork+pipe
internally instead, similar to what e.g. 'sort --compress-program=PROG'
does.

Have a nice day,
Berny





Information forwarded to bug-coreutils <at> gnu.org:
bug#18808; Package coreutils. (Fri, 24 Oct 2014 12:13:02 GMT) Full text and rfc822 format available.

Message #26 received at 18808 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>, Pádraig Brady <P <at> draigbrady.com>
Cc: 18808 <at> debbugs.gnu.org
Subject: Re: bug#18808: implement 'tail -r' as synonym for 'tac'
Date: Fri, 24 Oct 2014 06:11:55 -0600
[Message part 1 (text/plain, inline)]
On 10/24/2014 12:11 AM, Bernhard Voelker wrote:
> On 10/24/2014 12:52 AM, Eric Blake wrote:
>> On 10/23/2014 03:59 PM, Pádraig Brady wrote:
>> [...].  So among existing POSIX-certified systems, 'tail -r' is
>> the more popular extension at the moment.
>>
>>> Saying that, I'm 50:50 for implementing `tail -r` for compat reasons.
>>
>> Yes, this alone is reason enough to implement it in GNU, regardless of
>> the direction POSIX takes.
> 
> I'm also worried about such an extension.  'tail -r' might be a useful
> thing - when -r is the only option being used, but looking at all the
> other options GNU tail already got over the years, adding -r doesn't
> fit well.  You proposed to make -n available to tac and then jump into
> it from 'tail -r'.  Well, what about -c? --follow={name|descriptor}?

Reject them both.  Solaris 'tail -r' rejects use with -c or -f; reverse
mode is strictly limited to lines of a text file, and is strictly
limited to the existing end of the file at the time the command starts.

> --retry, --pid=PID?  To make -r fit well into tail, we'd have to add
> every single feature of it to tac.  This sounds not like a good idea.

No, to support 'tail -r', we make it refuse to operate with any option
other than -n, because that is all the more other implementations do,
and all the more POSIX is considering to require.  At least, as our
first cut at an implementation.  We can add more option interactions
later over time if desired as extensions, but we are not required to add
them, especially if we cannot come up with good semantics for those
interactions.

> tail does a good job, and so does tac.  I'd almost prefer to fork+pipe
> internally instead, similar to what e.g. 'sort --compress-program=PROG'
> does.

That's why my proposal is limited to just librar-ifying the tac.c code
and calling into it (no need to fork+pipe when we can just do the work
directly).


-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Severity set to 'wishlist' from 'normal' Request was from Assaf Gordon <assafgordon <at> gmail.com> to control <at> debbugs.gnu.org. (Fri, 19 Oct 2018 01:07:02 GMT) Full text and rfc822 format available.

Changed bug title to 'tail: implement 'tail -r' as synonym for 'tac'' from 'implement 'tail -r' as synonym for 'tac'' Request was from Assaf Gordon <assafgordon <at> gmail.com> to control <at> debbugs.gnu.org. (Fri, 19 Oct 2018 01:07:02 GMT) Full text and rfc822 format available.

This bug report was last modified 5 years and 162 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.