GNU bug report logs - #17422
dd giving different results between UNIX and LINUX

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 17422 in the body.
You can then email your comments to 17422 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Baggett, Don" <don.baggett <at> atlanticus.com>
To: "'bug-coreutils <at> gnu.org'" <bug-coreutils <at> gnu.org>
Cc: "Cart, Eric" <eric.cart <at> atlanticus.com>
Subject: dd giving different results between UNIX and LINUX
Date: Tue, 6 May 2014 09:13:37 -0400

[Message part 1 (text/plain, inline)]

Hello,

I am converting some UNIX programs to Linux.

One of the programs does the following:
 dd if=20140430.1.EBC.TXT of=tmpUNIX.dat cbs=152 conv=ascii

In Solaris I get this:
461614+1 records in
368585+1 records out

In Linux I get this:
461614+1 records in
461614+1 records out
236346472 bytes (236 MB) copied, 11.4938 seconds, 20.6 MB/s

The cksum done on 20140430.1.EBC.TXT yields the same result on both Linux and UNIX.


uname -a
SunOS dsa0401 5.10 Generic_142900-03 sun4v sparc SUNW,SPARC-Enterprise-T5220

Linux dla0614.ccrt.us 2.6.18-371.1.2.el5 #1 SMP Tue Oct 22 11:40:34 CDT 2013 x86_64 x86_64 x86_64 GNU/Linux

Any help would be appreciated.

Thanks,
Don




----- CONFIDENTIALITY NOTICE -----
This message contains PRIVILEGED and CONFIDENTIAL information that is intended
only for use by the named recipient. If you are not the named recipient, any
disclosure, dissemination, or action based on the contents of this message is
prohibited. In such case please notify us and destroy and delete all copies of
this transmission. Thank you.

[Message part 2 (text/html, inline)]

Message #8 received at 17422 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: "Baggett, Don" <don.baggett <at> atlanticus.com>, 17422 <at> debbugs.gnu.org
Cc: "Cart, Eric" <eric.cart <at> atlanticus.com>
Subject: Re: bug#17422: dd giving different results between UNIX and LINUX
Date: Tue, 06 May 2014 22:52:27 +0200

tags 17442 + moreinfo
thanks

On 05/06/2014 03:13 PM, Baggett, Don wrote:
> Hello,

Hi, thanks for the report.

> I am converting some UNIX programs to Linux.

... whatever this means ...

> One of the programs does the following:
>  dd if=20140430.1.EBC.TXT of=tmpUNIX.dat cbs=152 conv=ascii

... ah, converting a file from EBCDIC to ASCII.

> In Solaris I get this:
> 461614+1 records in
> 368585+1 records out
> 
> In Linux I get this:
> 461614+1 records in
> 461614+1 records out
> 236346472 bytes (236 MB) copied, 11.4938 seconds, 20.6 MB/s
> 
> The cksum done on 20140430.1.EBC.TXT yields the same result on both Linux and UNIX.

Okay, this means that the *input* files on Solaris and Linux
were the same. You only showed us the transfer status (for
which I'm not sure how to interpret the one from Solaris).
But what is the actual difference between the output files?
And what did you expect?

I'm therefore marking this bug as "+ moreinfo".

> uname -a
> SunOS dsa0401 5.10 Generic_142900-03 sun4v sparc SUNW,SPARC-Enterprise-T5220
> 
> Linux dla0614.ccrt.us 2.6.18-371.1.2.el5 #1 SMP Tue Oct 22 11:40:34 CDT 2013 x86_64 x86_64 x86_64 GNU/Linux

The above only gives us the Linux kernel version and architecture,
but not the actual version of 'dd' being used. For dd from GNU
coreutils, you can determine the version by "dd --version".

Thanks & have a nice day,
Berny

Message #13 received at 17422-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: "Baggett, Don" <don.baggett <at> atlanticus.com>, 
 17422-done <at> debbugs.gnu.org
Cc: "Cart, Eric" <eric.cart <at> atlanticus.com>
Subject: Re: bug#17422: dd giving different results between UNIX and LINUX
Date: Tue, 06 May 2014 18:43:50 -0700

[Message part 1 (text/plain, inline)]

Thanks for reporting that bug.  Who would have thought that after all 
these years, GNU 'dd' would still be mishandling EBCDIC conversion?  The 
GNU dd implementation of conv=ascii is clearly incorrect.  I installed 
the attached patch.  Please give it a try if you can.  I am closing the 
bug report for convenience, but we can reopen it if the patch does not 
work for you.

[0001-dd-fix-conv-ascii-conv-ebcdic-conv-ibm-to-match-POSI.patch (text/plain, attachment)]

Message #16 received at 17422 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: 17422 <at> debbugs.gnu.org, eggert <at> cs.ucla.edu, 
 don.baggett <at> atlanticus.com
Subject: Re: bug#17422: dd giving different results between UNIX and LINUX
Date: Wed, 07 May 2014 09:20:16 +0200

[Message part 1 (text/plain, inline)]

On 05/07/2014 03:43 AM, Paul Eggert wrote:
> Thanks for reporting that bug.  Who would have thought that after all 
> these years, GNU 'dd' would still be mishandling EBCDIC conversion?  The 
> GNU dd implementation of conv=ascii is clearly incorrect.  I installed 
> the attached patch.

ooh, thanks!

There is a testing artifact left in the new test:

  > cp ./in ./out ./exp /tmp

I corrected this besides other minor nits in the attached patch.
Pushing later.

Have a nice day,
Berny

[0001-tests-improve-new-ascii-test-of-dd.patch (text/x-patch, attachment)]

Message #19 received at 17422 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>
Cc: 17422 <at> debbugs.gnu.org, don.baggett <at> atlanticus.com, eggert <at> cs.ucla.edu
Subject: Re: bug#17422: dd giving different results between UNIX and LINUX
Date: Wed, 07 May 2014 12:40:34 +0100

On 05/07/2014 08:20 AM, Bernhard Voelker wrote:
> On 05/07/2014 03:43 AM, Paul Eggert wrote:
>> Thanks for reporting that bug.  Who would have thought that after all 
>> these years, GNU 'dd' would still be mishandling EBCDIC conversion?  The 
>> GNU dd implementation of conv=ascii is clearly incorrect.  I installed 
>> the attached patch.
> 
> ooh, thanks!
> 
> There is a testing artifact left in the new test:
> 
>   > cp ./in ./out ./exp /tmp
> 
> I corrected this besides other minor nits in the attached patch.
> Pushing later.
> 
> Have a nice day,
> Berny
> 

Wow nice catch on the incorrect tables Paul.

Bernhard the test adjustments look good.

thanks!
Pádraig.

Message #22 received at 17422 <at> debbugs.gnu.org (full text, mbox):

From: "Baggett, Don" <don.baggett <at> atlanticus.com>
To: 'Pádraig Brady' <P <at> draigBrady.com>, Bernhard Voelker
 <mail <at> bernhard-voelker.de>
Cc: "17422 <at> debbugs.gnu.org" <17422 <at> debbugs.gnu.org>,
 "eggert <at> cs.ucla.edu" <eggert <at> cs.ucla.edu>
Subject: RE: bug#17422: dd giving different results between UNIX and LINUX
Date: Wed, 7 May 2014 07:44:28 -0400

Hello,

Thanks to all. I will work the UNIX administrators to get the patch put in.

This is the dd version:
dd --version
dd (coreutils) 5.97
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.

Written by Paul Rubin, David MacKenzie, and Stuart Kemp.

Thanks,
Don

-----Original Message-----
From: Pádraig Brady [mailto:P <at> draigBrady.com] 
Sent: Wednesday, May 07, 2014 7:41 AM
To: Bernhard Voelker
Cc: 17422 <at> debbugs.gnu.org; eggert <at> cs.ucla.edu; Baggett, Don
Subject: Re: bug#17422: dd giving different results between UNIX and LINUX

On 05/07/2014 08:20 AM, Bernhard Voelker wrote:
> On 05/07/2014 03:43 AM, Paul Eggert wrote:
>> Thanks for reporting that bug.  Who would have thought that after all 
>> these years, GNU 'dd' would still be mishandling EBCDIC conversion?  The 
>> GNU dd implementation of conv=ascii is clearly incorrect.  I installed 
>> the attached patch.
> 
> ooh, thanks!
> 
> There is a testing artifact left in the new test:
> 
>   > cp ./in ./out ./exp /tmp
> 
> I corrected this besides other minor nits in the attached patch.
> Pushing later.
> 
> Have a nice day,
> Berny
> 

Wow nice catch on the incorrect tables Paul.

Bernhard the test adjustments look good.

thanks!
Pádraig.




----- CONFIDENTIALITY NOTICE -----
This message contains PRIVILEGED and CONFIDENTIAL information that is intended
only for use by the named recipient. If you are not the named recipient, any
disclosure, dissemination, or action based on the contents of this message is
prohibited. In such case please notify us and destroy and delete all copies of
this transmission. Thank you.

Message #25 received at 17422 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>, 17422 <at> debbugs.gnu.org, 
 don.baggett <at> atlanticus.com
Subject: Re: bug#17422: dd giving different results between UNIX and LINUX
Date: Wed, 07 May 2014 06:24:28 -0700

Thanks for those improvements to the patch; please push.

Message #28 received at 17422 <at> debbugs.gnu.org (full text, mbox):

From: "Baggett, Don" <don.baggett <at> atlanticus.com>
To: 'Paul Eggert' <eggert <at> cs.ucla.edu>, Bernhard Voelker
 <mail <at> bernhard-voelker.de>, "17422 <at> debbugs.gnu.org" <17422 <at> debbugs.gnu.org>
Subject: RE: bug#17422: dd giving different results between UNIX and LINUX
Date: Wed, 7 May 2014 09:25:41 -0400

Hello,

I apologize for my ignorance. What exactly am I supposed to give the UNIX administrators?

Thanks,
Don

-----Original Message-----
From: Paul Eggert [mailto:eggert <at> cs.ucla.edu] 
Sent: Wednesday, May 07, 2014 9:24 AM
To: Bernhard Voelker; 17422 <at> debbugs.gnu.org; Baggett, Don
Subject: Re: bug#17422: dd giving different results between UNIX and LINUX

Thanks for those improvements to the patch; please push.






----- CONFIDENTIALITY NOTICE -----
This message contains PRIVILEGED and CONFIDENTIAL information that is intended
only for use by the named recipient. If you are not the named recipient, any
disclosure, dissemination, or action based on the contents of this message is
prohibited. In such case please notify us and destroy and delete all copies of
this transmission. Thank you.

Message #31 received at 17422 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: "Baggett, Don" <don.baggett <at> atlanticus.com>, 
 Bernhard Voelker <mail <at> bernhard-voelker.de>,
 "17422 <at> debbugs.gnu.org" <17422 <at> debbugs.gnu.org>
Subject: Re: bug#17422: dd giving different results between UNIX and LINUX
Date: Wed, 07 May 2014 06:32:19 -0700

Baggett, Don wrote:
> What exactly am I supposed to give the UNIX administrators?

I'd give them this URL:

http://bugs.gnu.org/17422

and ask them to install my patch, followed by Bernhard's patch.  The 
patches themselves are in messages #13 and #16 in that thread; here are 
separate URLs if that helps:

http://debbugs.gnu.org/cgi/bugreport.cgi?msg=13;filename=0001-dd-fix-conv-ascii-conv-ebcdic-conv-ibm-to-match-POSI.patch;att=1;bug=17422

http://debbugs.gnu.org/cgi/bugreport.cgi?msg=16;filename=0001-tests-improve-new-ascii-test-of-dd.patch;att=1;bug=17422

Message #34 received at 17422 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: "Baggett, Don" <don.baggett <at> atlanticus.com>
Cc: 'Paul Eggert' <eggert <at> cs.ucla.edu>,
 Bernhard Voelker <mail <at> bernhard-voelker.de>,
 "17422 <at> debbugs.gnu.org" <17422 <at> debbugs.gnu.org>
Subject: Re: bug#17422: dd giving different results between UNIX and LINUX
Date: Wed, 07 May 2014 14:33:24 +0100

On 05/07/2014 02:25 PM, Baggett, Don wrote:
> Hello,
> 
> I apologize for my ignorance. What exactly am I supposed to give the UNIX administrators?

They'll have to adjust this to apply on version 5.97:
http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=patch;h=3974c09
The only needed part is the change to dd.c

thanks,
Pádraig.

Message #37 received at 17422 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: Paul Eggert <eggert <at> cs.ucla.edu>, 17422 <at> debbugs.gnu.org, 
 don.baggett <at> atlanticus.com
Subject: Re: bug#17422: dd giving different results between UNIX and LINUX
Date: Wed, 07 May 2014 17:05:10 +0200

On 05/07/2014 03:24 PM, Paul Eggert wrote:
> Thanks for those improvements to the patch; please push.

Thanks for the review.

Furthermore, I was puzzled that there doesn't seem to be
an easier way to create the 0..255 file as this:

  printf $(for i in $(seq 0 255); do printf '\\%03o' $i; done; echo '');

At least, the following would avoid the for-loop:

diff --git a/tests/dd/ascii.sh b/tests/dd/ascii.sh
index f2cc375..da6e128 100755
--- a/tests/dd/ascii.sh
+++ b/tests/dd/ascii.sh
@@ -22,7 +22,7 @@ print_ver_ dd
 {
   # Two lines, EBCDIC " A A" and " A  ", followed by all the bytes in order.
   printf '\100\301\100\301\100\301\100\100' &&
-  printf $(for i in $(seq 0 255); do printf '\\%03o' $i; done; echo '');
+  printf $(printf "\\%03o" $(seq 0 255 ));
 } >in || framework_failure_

 {

WDYT?

Thanks & have a nice day,
Berny

Message #40 received at 17422 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>, 17422 <at> debbugs.gnu.org, 
 don.baggett <at> atlanticus.com
Subject: Re: bug#17422: dd giving different results between UNIX and LINUX
Date: Wed, 07 May 2014 08:17:11 -0700

Bernhard Voelker wrote:
> At least, the following would avoid the for-loop:

Sure, that works for me.

Message #43 received at 17422 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: Paul Eggert <eggert <at> cs.ucla.edu>, 17422 <at> debbugs.gnu.org, 
 don.baggett <at> atlanticus.com
Subject: Re: bug#17422: dd giving different results between UNIX and LINUX
Date: Wed, 07 May 2014 18:43:24 +0200

On 05/07/2014 05:17 PM, Paul Eggert wrote:
> Sure, that works for me.

Thanks, pushed.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.