GNU bug report logs - #9734
[solaris] `dd if=/dev/urandom of=file bs=1024k count=1' gets a file of 133120 bytes

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 9734 in the body.
You can then email your comments to 9734 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Clark J. Wang" <dearvoid <at> gmail.com>
To: bug-coreutils <at> gnu.org
Subject: [solaris] `dd if=/dev/urandom of=file bs=1024k count=1' gets a file
	of 133120 bytes
Date: Wed, 12 Oct 2011 16:22:12 +0800

[Message part 1 (text/plain, inline)]

I'm not sure if it's a bug but it's not reasonable to me. On Solaris 11
(SunOS 5.11 snv_174, i86pc):

$ uname -a
SunOS sollab-242.cn.oracle.com 5.11 snv_174 i86pc i386 i86pc
$ pkg list gnu-coreutils
NAME (PUBLISHER)                                  VERSION
IFO
file/gnu-coreutils                                8.5-0.174.0.0.0.0.504
i--
$ /usr/gnu/bin/dd if=/dev/urandom of=file bs=1024k count=1
0+1 records in
0+1 records out
133120 bytes (133 kB) copied, 0.00290536 s, 45.8 MB/s
$ ls -l file
-rw-r--r-- 1 root root 133120 2011-10-12 16:12 file
$

I'm new to Solaris but I've never seen this problem whe I use Linux so it
really suprises me.

I found this in the man page of /dev/urandom on Solaris: "The limitation per
read for /dev/random is 1040 bytes. The limit for /dev/urandom is (128 *
1040 = 133120)." That seems to be the reason but I think dd should handle
that and check the return value of the read() system call and make sure
1024k bytes have really been read from /dev/urandom.

Any idea?

Thanks.

-Clark

[Message part 2 (text/html, inline)]

Message #12 received at 9734-done <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: "Clark J. Wang" <dearvoid <at> gmail.com>
Cc: 9734-done <at> debbugs.gnu.org
Subject: Re: bug#9734: [solaris] `dd if=/dev/urandom of=file bs=1024k count=1'
	gets a file of 133120 bytes
Date: Wed, 12 Oct 2011 08:14:15 -0600

tag 9734 notabug
thanks

On 10/12/2011 02:22 AM, Clark J. Wang wrote:
> I'm not sure if it's a bug but it's not reasonable to me. On Solaris 11
> (SunOS 5.11 snv_174, i86pc):
>
> $ uname -a
> SunOS sollab-242.cn.oracle.com 5.11 snv_174 i86pc i386 i86pc
> $ pkg list gnu-coreutils
> NAME (PUBLISHER)                                  VERSION
> IFO
> file/gnu-coreutils                                8.5-0.174.0.0.0.0.504
> i--
> $ /usr/gnu/bin/dd if=/dev/urandom of=file bs=1024k count=1
> 0+1 records in

Notice that this means you read a partial record - read() tried to read 
1024k bytes, but the read ended short at only 133120 bytes.

> 0+1 records out

And because you didn't request dd to group multiple short reads before 
doing a full write, you got a single (short) record written.

> I'm new to Solaris but I've never seen this problem whe I use Linux so it
> really suprises me.

Solaris and Linux kernels differ on when you will get short reads, and 
magic files like /dev/urandom are more likely to display the issue than 
regular files.  That said, Linux also has the "problem" of short reads; 
it's especially noticeable when passing the output of dd to a pipe.

You probably wanted to use this GNU extension:

dd if=/dev/urandom of=file bs=1024k count=1 iconv=fullblock

where the iconv flag requests that dd pile together multiple read()s 
until it has a full block, so that you no longer have a partial block 
output.

>
> I found this in the man page of /dev/urandom on Solaris: "The limitation per
> read for /dev/random is 1040 bytes. The limit for /dev/urandom is (128 *
> 1040 = 133120)." That seems to be the reason but I think dd should handle
> that and check the return value of the read() system call and make sure
> 1024k bytes have really been read from /dev/urandom.

Only if the iconv=fullblock flag is specified, since it is a violation 
of POSIX to do more than one read() without an explicit flag requesting 
multiple reads per block.

-- 
Eric Blake   eblake <at> redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

Message #15 received at submit <at> debbugs.gnu.org (full text, mbox):

From: John Reiser <jreiser <at> bitwagon.com>
To: bug-coreutils <at> gnu.org
Subject: Re: bug#9734: [solaris] `dd if=/dev/urandom of=file bs=1024k count=1'
	gets a file of 133120 bytes
Date: Wed, 12 Oct 2011 08:10:18 -0700

> I found this in the man page of /dev/urandom on Solaris: "The limitation per
> read for /dev/random is 1040 bytes. The limit for /dev/urandom is (128 *
> 1040 = 133120)." That seems to be the reason but I think dd should handle
> that and check the return value of the read() system call and make sure
> 1024k bytes have really been read from /dev/urandom.

"bs=N" means "write(,,read(,,N))" [error checking has been elided for
presentation here] so if the read is short then the write will be also.
Check the manual page.  Consider ibs=, obs=, iflag=fullblock.

-- 

Message #18 received at 9734 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: 9734 <at> debbugs.gnu.org, eblake <at> redhat.com
Subject: Re: bug#9734: [solaris] `dd if=/dev/urandom of=file bs=1024k count=1'
	gets a file of 133120 bytes
Date: Wed, 12 Oct 2011 16:42:49 +0100

On 10/12/2011 03:14 PM, Eric Blake wrote:
> 
> On 10/12/2011 02:22 AM, Clark J. Wang wrote:
>> I'm not sure if it's a bug but it's not reasonable to me. On Solaris 11
>> (SunOS 5.11 snv_174, i86pc):
>>
>> $ uname -a
>> SunOS sollab-242.cn.oracle.com 5.11 snv_174 i86pc i386 i86pc
>> $ pkg list gnu-coreutils
>> NAME (PUBLISHER)                                  VERSION
>> IFO
>> file/gnu-coreutils                                8.5-0.174.0.0.0.0.504
>> i--
>> $ /usr/gnu/bin/dd if=/dev/urandom of=file bs=1024k count=1
>> 0+1 records in
> 
> Notice that this means you read a partial record - read() tried to read 1024k bytes, but the read ended short at only 133120 bytes.
> 
>> 0+1 records out
> 
> And because you didn't request dd to group multiple short reads before doing a full write, you got a single (short) record written.
> 
>> I'm new to Solaris but I've never seen this problem whe I use Linux so it
>> really suprises me.
> 
> Solaris and Linux kernels differ on when you will get short reads, and magic files like /dev/urandom are more likely to display the issue than regular files.  That said, Linux also has the "problem" of short reads; it's especially noticeable when passing the output of dd to a pipe.
> 
> You probably wanted to use this GNU extension:
> 
> dd if=/dev/urandom of=file bs=1024k count=1 iconv=fullblock
> 
> where the iconv flag requests that dd pile together multiple read()s until it has a full block, so that you no longer have a partial block output.

Right, but for the record it's iflag=fullblock (available since coreutils 7.0)

This common issue is warned about by coreutils >= 8.11,
where it will suggest using iflag=fullblock
Note the particular case where count=1 is not warned about,
as with a single read, one doesn't know if we're just at EOF.
Also it's probably a quite common idiom to, consume available data
up to $bs bytes.

cheers,
Pádraig.

Message #19 received at 9734-done <at> debbugs.gnu.org (full text, mbox):

From: "Clark J. Wang" <dearvoid <at> gmail.com>
To: Eric Blake <eblake <at> redhat.com>
Cc: 9734-done <at> debbugs.gnu.org
Subject: Re: bug#9734: [solaris] `dd if=/dev/urandom of=file bs=1024k count=1'
	gets a file of 133120 bytes
Date: Thu, 13 Oct 2011 10:39:58 +0800

[Message part 1 (text/plain, inline)]

On Wed, Oct 12, 2011 at 10:14 PM, Eric Blake <eblake <at> redhat.com> wrote:

> tag 9734 notabug
> thanks
>
>
> On 10/12/2011 02:22 AM, Clark J. Wang wrote:
>
>> I'm not sure if it's a bug but it's not reasonable to me. On Solaris 11
>> (SunOS 5.11 snv_174, i86pc):
>>
>> $ uname -a
>> SunOS sollab-242.cn.oracle.com 5.11 snv_174 i86pc i386 i86pc
>> $ pkg list gnu-coreutils
>> NAME (PUBLISHER)                                  VERSION
>> IFO
>> file/gnu-coreutils                                8.5-0.174.0.0.0.0.504
>> i--
>> $ /usr/gnu/bin/dd if=/dev/urandom of=file bs=1024k count=1
>> 0+1 records in
>>
>
> Notice that this means you read a partial record - read() tried to read
> 1024k bytes, but the read ended short at only 133120 bytes.
>
>  0+1 records out
>>
>
> And because you didn't request dd to group multiple short reads before
> doing a full write, you got a single (short) record written.


I've never noticed what "0+1 records in/out" really means since dd always
does what I mean. Thanks a lot for all the clarification.


>
>  I'm new to Solaris but I've never seen this problem whe I use Linux so it
>> really suprises me.
>>
>
> Solaris and Linux kernels differ on when you will get short reads, and
> magic files like /dev/urandom are more likely to display the issue than
> regular files.  That said, Linux also has the "problem" of short reads; it's
> especially noticeable when passing the output of dd to a pipe.
>
> You probably wanted to use this GNU extension:
>
> dd if=/dev/urandom of=file bs=1024k count=1 iconv=fullblock
>
> where the iconv flag requests that dd pile together multiple read()s until
> it has a full block, so that you no longer have a partial block output.
>
>
>
>> I found this in the man page of /dev/urandom on Solaris: "The limitation
>> per
>> read for /dev/random is 1040 bytes. The limit for /dev/urandom is (128 *
>> 1040 = 133120)." That seems to be the reason but I think dd should handle
>> that and check the return value of the read() system call and make sure
>> 1024k bytes have really been read from /dev/urandom.
>>
>
> Only if the iconv=fullblock flag is specified, since it is a violation of
> POSIX to do more than one read() without an explicit flag requesting
> multiple reads per block.
>
> --
> Eric Blake   eblake <at> redhat.com    +1-801-349-2682
> Libvirt virtualization library http://libvirt.org
>

[Message part 2 (text/html, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.