GNU bug report logs - #18750
--line-buffered option doesn't work with --files-with-matches

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 18750 in the body.
You can then email your comments to 18750 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Louis S." <sautier.louis <at> gmail.com>
To: bug-grep <at> gnu.org
Subject: --line-buffered option doesn't work with --files-with-matches
Date: Thu, 16 Oct 2014 23:16:30 +0200

Hi, I'm not quite sure this is a bug but it appears that the
--line-buffered option has no effect when used in conjunction with
--files-with-matches.
Here are some tests I ran with grep 2.20:
$ time for i in {1..100}; do grep --line-buffered -Rl gnome /usr/ |
head -n1 >& /dev/null; done
real    0m54.831s
user    0m37.077s
sys     0m17.550s

$ time for i in {1..100}; do grep -Rl gnome /usr/ | head -n1 >& /dev/null; done
real    0m54.858s
user    0m36.987s
sys     0m17.657s

$ time for i in {1..100}; do grep --line-buffered -R gnome /usr/ |
head -n1 >& /dev/null; done
real    0m7.414s
user    0m4.727s
sys     0m2.613s

Please let me know if line-buffering is supposed to work in that case.

Regards,

Louis

Message #8 received at 18750 <at> debbugs.gnu.org (full text, mbox):

From: Norihiro Tanaka <noritnk <at> kcn.ne.jp>
To: "Louis S." <sautier.louis <at> gmail.com>
Cc: 18750 <at> debbugs.gnu.org
Subject: Re: bug#18750: --line-buffered option doesn't work with
 --files-with-matches
Date: Fri, 17 Oct 2014 22:30:44 +0900

[Message part 1 (text/plain, inline)]

"Louis S." <sautier.louis <at> gmail.com> wrote:
> Hi, I'm not quite sure this is a bug but it appears that the
> --line-buffered option has no effect when used in conjunction with
> --files-with-matches.
> Here are some tests I ran with grep 2.20:
> 
> Please let me know if line-buffering is supposed to work in that case.

I see that it's designed, but I think that it's reasonable that output
of file names is also line buffered in grep --line-buffered.

[0001-grep-line-buffered-for-output-of-file-names-in-grep-.patch (text/plain, attachment)]

Message #11 received at 18750 <at> debbugs.gnu.org (full text, mbox):

From: Norihiro Tanaka <noritnk <at> kcn.ne.jp>
To: "Louis S." <sautier.louis <at> gmail.com>
Cc: 18750 <at> debbugs.gnu.org
Subject: Re: bug#18750: --line-buffered option doesn't work with
 --files-with-matches
Date: Sat, 18 Oct 2014 13:01:03 +0900

[Message part 1 (text/plain, inline)]

If remove fflush() as the previous patch, unexpected result will be
caused in grep -Z.  I fixed it.

[0001-grep-line-buffered-for-output-of-file-names-in-grep-.patch (text/plain, attachment)]

Message #14 received at 18750 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Norihiro Tanaka <noritnk <at> kcn.ne.jp>
Cc: 18750 <at> debbugs.gnu.org, "Louis S." <sautier.louis <at> gmail.com>
Subject: Re: bug#18750: --line-buffered option doesn't work with
 --files-with-matches
Date: Sat, 18 Oct 2014 16:11:01 -0700

On Fri, Oct 17, 2014 at 9:01 PM, Norihiro Tanaka <noritnk <at> kcn.ne.jp> wrote:
> If remove fflush() as the previous patch, unexpected result will be
> caused in grep -Z.  I fixed it.

Thanks, but as a stand-alone patch, that makes grep fail to compile,
because there is one remaining use of the variable whose declaration
is removed.

In any case, there is some ambiguity in the documentation
about what "line" buffering means with -Z. With -l and -Z, should
grep --line-buffered flush after printing each <file_name, NUL> pair?
That's what I'd expect.

Message #17 received at 18750 <at> debbugs.gnu.org (full text, mbox):

From: Norihiro Tanaka <noritnk <at> kcn.ne.jp>
To: Jim Meyering <jim <at> meyering.net>
Cc: 18750 <at> debbugs.gnu.org, "Louis S." <sautier.louis <at> gmail.com>
Subject: Re: bug#18750: --line-buffered option doesn't work with
 --files-with-matches
Date: Sun, 19 Oct 2014 09:43:12 +0900

[Message part 1 (text/plain, inline)]

Jim Meyering <jim <at> meyering.net> wrote:
> Thanks, but as a stand-alone patch, that makes grep fail to compile,
> because there is one remaining use of the variable whose declaration
> is removed.

Sorry, I fixed it, and compiled and tested.

> In any case, there is some ambiguity in the documentation
> about what "line" buffering means with -Z. With -l and -Z, should
> grep --line-buffered flush after printing each <file_name, NUL> pair?
> That's what I'd expect.

I interprete the means as line buffered option `_IOLBF' of setvbuf, and
I think that many users will expect the behavior.  However, different
interpretations might also be present.

[0001-grep-line-buffered-for-output-of-file-names-in-grep-.patch (text/plain, attachment)]

Message #20 received at 18750 <at> debbugs.gnu.org (full text, mbox):

From: Louis Sautier <sautier.louis <at> gmail.com>
To: Norihiro Tanaka <noritnk <at> kcn.ne.jp>
Cc: 18750 <at> debbugs.gnu.org, Jim Meyering <jim <at> meyering.net>
Subject: Re: bug#18750: --line-buffered option doesn't work with
 --files-with-matches
Date: Mon, 20 Oct 2014 09:29:23 +0200

I have no issue with this behaviour, I just wish one could have
buffered null-separated output as well.

On 19 October 2014 02:43, Norihiro Tanaka <noritnk <at> kcn.ne.jp> wrote:
> Jim Meyering <jim <at> meyering.net> wrote:
>> Thanks, but as a stand-alone patch, that makes grep fail to compile,
>> because there is one remaining use of the variable whose declaration
>> is removed.
>
> Sorry, I fixed it, and compiled and tested.
>
>> In any case, there is some ambiguity in the documentation
>> about what "line" buffering means with -Z. With -l and -Z, should
>> grep --line-buffered flush after printing each <file_name, NUL> pair?
>> That's what I'd expect.
>
> I interprete the means as line buffered option `_IOLBF' of setvbuf, and
> I think that many users will expect the behavior.  However, different
> interpretations might also be present.

Message #23 received at 18750 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Louis Sautier <sautier.louis <at> gmail.com>
Cc: 18750 <at> debbugs.gnu.org, Norihiro Tanaka <noritnk <at> kcn.ne.jp>
Subject: Re: bug#18750: --line-buffered option doesn't work with
 --files-with-matches
Date: Mon, 20 Oct 2014 16:54:07 -0700

On Mon, Oct 20, 2014 at 12:29 AM, Louis Sautier <sautier.louis <at> gmail.com> wrote:
> I have no issue with this behaviour, I just wish one could have
> buffered null-separated output as well.
>
> On 19 October 2014 02:43, Norihiro Tanaka <noritnk <at> kcn.ne.jp> wrote:
>> Jim Meyering <jim <at> meyering.net> wrote:
>>> Thanks, but as a stand-alone patch, that makes grep fail to compile,
>>> because there is one remaining use of the variable whose declaration
>>> is removed.
>>
>> Sorry, I fixed it, and compiled and tested.
>>
>>> In any case, there is some ambiguity in the documentation
>>> about what "line" buffering means with -Z. With -l and -Z, should
>>> grep --line-buffered flush after printing each <file_name, NUL> pair?
>>> That's what I'd expect.
>>
>> I interprete the means as line buffered option `_IOLBF' of setvbuf, and
>> I think that many users will expect the behavior.  However, different
>> interpretations might also be present.

Thanks for the feedback.  What is your use case?
As I said above, I too would have expected -Z -l --line-buffered to flush
after each <file_name, NUL> pair.

Though note that the larger change would
have to update documentation clarifying that this also applies
to output that may not contain any newline.  It might even prompt
the addition of a new option name, --buffered, as a
more-aptly-named synonym for --line-buffered.

P.S., please do not "top post", i.e., reply after the bottom of the
quoted text, as I have done.

Message #26 received at 18750 <at> debbugs.gnu.org (full text, mbox):

From: Louis Sautier <sautier.louis <at> gmail.com>
To: Jim Meyering <jim <at> meyering.net>
Cc: 18750 <at> debbugs.gnu.org, Norihiro Tanaka <noritnk <at> kcn.ne.jp>
Subject: Re: bug#18750: --line-buffered option doesn't work with
 --files-with-matches
Date: Tue, 21 Oct 2014 11:27:38 +0200

On 21 October 2014 01:54, Jim Meyering <jim <at> meyering.net> wrote:
> On Mon, Oct 20, 2014 at 12:29 AM, Louis Sautier <sautier.louis <at> gmail.com> wrote:
>> I have no issue with this behaviour, I just wish one could have
>> buffered null-separated output as well.
>>
>> On 19 October 2014 02:43, Norihiro Tanaka <noritnk <at> kcn.ne.jp> wrote:
>>> Jim Meyering <jim <at> meyering.net> wrote:
>>>> Thanks, but as a stand-alone patch, that makes grep fail to compile,
>>>> because there is one remaining use of the variable whose declaration
>>>> is removed.
>>>
>>> Sorry, I fixed it, and compiled and tested.
>>>
>>>> In any case, there is some ambiguity in the documentation
>>>> about what "line" buffering means with -Z. With -l and -Z, should
>>>> grep --line-buffered flush after printing each <file_name, NUL> pair?
>>>> That's what I'd expect.
>>>
>>> I interprete the means as line buffered option `_IOLBF' of setvbuf, and
>>> I think that many users will expect the behavior.  However, different
>>> interpretations might also be present.
>
> Thanks for the feedback.  What is your use case?
> As I said above, I too would have expected -Z -l --line-buffered to flush
> after each <file_name, NUL> pair.
>
> Though note that the larger change would
> have to update documentation clarifying that this also applies
> to output that may not contain any newline.  It might even prompt
> the addition of a new option name, --buffered, as a
> more-aptly-named synonym for --line-buffered.
>
> P.S., please do not "top post", i.e., reply after the bottom of the
> quoted text, as I have done.
I think --buffered would be more clear, yes. I would use such an
option to run commands on batches of files found by grep such as:
grep --buffered -RlZ pattern /path | xargs -0 -n 100 rm

Message #31 received at 18750-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: 18750-done <at> debbugs.gnu.org
Cc: "Louis S." <sautier.louis <at> gmail.com>, Norihiro Tanaka <noritnk <at> kcn.ne.jp>
Subject: Re: --line-buffered option doesn't work with --files-with-matches
Date: Thu, 31 Dec 2015 00:29:46 -0800

[Message part 1 (text/plain, inline)]

Thanks for the bug report. It also seems to me that --line-buffered should apply 
to nulls, not newlines, when -z is used. Also, there are a couple of other 
places where stdout should get flushed after a newline. Perhaps we should use 
setvbuf instead (is it faster?) when -z is not also used, but right now I just 
want to get the bug fixed for the next release, so I installed the attached, 
more-conservative, patch, and am marking this bug as done.

[0001-grep-fix-l-line-buffer-bug.patch (text/x-diff, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.