GNU bug report logs - #25459
sed bug: native sed-4.3 appends excessive carriage-returns when run in binary mode on windows

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 25459 in the body.
You can then email your comments to 25459 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Michael Builov <mbuilov <at> gmail.com>
To: bug-sed <at> gnu.org
Subject: sed bug: native sed-4.3 appends excessive carriage-returns when run
 in binary mode on windows
Date: Mon, 16 Jan 2017 13:03:01 +0300

[Message part 1 (text/plain, inline)]

Steps to reproduce.

First it's needed to build native sed.exe on windows.

It's hard, but possible, I've done this:
https://github.com/mbuilov/sed-windows/blob/master/sed-4.3-build-patch-howto.txt

(Note: don't apply sed-4.3-src.patch in these instructions - it fixes
reported bug and build problems with Visual Studio 2008)


Now, if we have working native sed.exe, from cmd.exe

1) create a file containing a line terminated with '\n':

echo.1> test

check "test" contents, dump it:

c:\cygwin64\bin\xxd test
00000000: 310d 0a                                  1..


2) process file "test" with sed in binary mode:

sed\sed.exe -b "" test > test1

dump resulting file "test1":

c:\cygwin64\bin\xxd test1
00000000: 310d 0d0a                                1...


As we can see, there is excessive '0d' in result.


To fix this bug, stdout should be set to binary mode, for example by
applying attached patch.



Best Regards,
Michael M. Builov

[Message part 2 (text/html, inline)]

[sed-4.3-src-stdout.patch (application/octet-stream, attachment)]

Message #8 received at 25459 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: Michael Builov <mbuilov <at> gmail.com>, 25459 <at> debbugs.gnu.org
Subject: Re: bug#25459: sed bug: native sed-4.3 appends excessive
 carriage-returns when run in binary mode on windows
Date: Tue, 17 Jan 2017 19:25:43 -0500

Hello,

Thank you for the report.

I do not have access to MSVC, but I can see similar issues when building
with mingw on cygwin64.

Before jumping to the fix, I'd like to show what I'm experiencing,
because there might be a larger issue here.

I'm testing on a Windows R2008 with cygwin64.
I've compared 3 binaries (below):
  sed-4.2.2-cygwin.exe:
      The default that's installed by cygwin itself.

  sed-4.3-cygwin-gcc-5.4.0.exe:
      Built from tarball using cygwin's default gcc compiler.

  sed-4.3-mingw-64bit.exe:
      Built from tarball using:
      ./configure --host=x86_64-w64-mingw32 --disable-nls


First,
sed-4.2.2 seems to produce incorrect output without "-b":
The output file is not set to text mode, and only '\n' is written
(without '\r').

second,
It seems the new sed-4.3-mingw does output extra '\r',
but only if the input is a file (and not stdin).

So there's some unclear interplay here (unclear to me).

The proposed patch does produce the correct output,
but I'd like to verify it's for the right reasons.

Lastly,
There's also testing needed with in-place writing of files,
and adding tests.

I hope to get to all of these soon.

regards,
 - assaf

$ env printf '1\x0D\x0A' > test

$ ./sed-4.2.2-cygwin.exe -b "" < test | od -tx1 -An
 31 0d 0a
$ ./sed-4.2.2-cygwin.exe -b ""   test | od -tx1 -An
 31 0d 0a
$ ./sed-4.2.2-cygwin.exe    "" < test | od -tx1 -An
 31 0a
$ ./sed-4.2.2-cygwin.exe    ""   test | od -tx1 -An
 31 0a


$ ./sed-4.3-cygwin-gcc-5.4.0.exe -b "" < test | od -tx1 -An
 31 0d 0a
$ ./sed-4.3-cygwin-gcc-5.4.0.exe -b ""   test | od -tx1 -An
 31 0d 0a
$ ./sed-4.3-cygwin-gcc-5.4.0.exe    "" < test | od -tx1 -An
 31 0d 0a
$ ./sed-4.3-cygwin-gcc-5.4.0.exe    ""   test | od -tx1 -An
 31 0d 0a



$ ./sed-4.3-mingw-64bit.exe -b "" < test | od -tx1 -An
 31 0d 0a
$ ./sed-4.3-mingw-64bit.exe -b ""   test | od -tx1 -An
 31 0d 0d 0a
$ ./sed-4.3-mingw-64bit.exe    "" < test | od -tx1 -An
 31 0d 0a
$ ./sed-4.3-mingw-64bit.exe    ""   test | od -tx1 -An
 31 0d 0a

Message #11 received at 25459 <at> debbugs.gnu.org (full text, mbox):

From: Assaf Gordon <assafgordon <at> gmail.com>
To: Michael Builov <mbuilov <at> gmail.com>, 25459 <at> debbugs.gnu.org
Subject: Re: bug#25459: sed bug: native sed-4.3 appends excessive
 carriage-returns when run in binary mode on windows
Date: Mon, 24 Dec 2018 01:35:22 -0700

tags 25459 fixed
close 25459
stop

Hello,

On 2017-01-17 5:25 p.m., Assaf Gordon wrote:
> I'm testing on a Windows R2008 with cygwin64.
> I've compared 3 binaries (below):
>    sed-4.2.2-cygwin.exe:
>        The default that's installed by cygwin itself.
> 
>    sed-4.3-cygwin-gcc-5.4.0.exe:
>        Built from tarball using cygwin's default gcc compiler.
> 
>    sed-4.3-mingw-64bit.exe:
>        Built from tarball using:
>        ./configure --host=x86_64-w64-mingw32 --disable-nls
> 

sed version 4.7 (released 4 days ago) includes improved
and consistent CR/LF handling under cygwin and mingw.

Please give it a try (and of course, if there are other issues,
do report them).

I'm closing this item as "fixed", but discussion can continue by
replying to this thread.

-assaf

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.