GNU bug report logs - #13537
split: memory exhausted when putting at most T/P/E bytes of lines per output file

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 13537 in the body.
You can then email your comments to 13537 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Lei Zhang <lei.zhang <at> uwaterloo.ca>
To: bug-coreutils <at> gnu.org
Subject: split: memory exhausted when putting at most T/P/E bytes of lines per
	output file
Date: Wed, 23 Jan 2013 18:58:23 -0500

[Message part 1 (text/plain, inline)]

Hi All,

In the `split' program of coreutils 8.20, we found a bug which is similar
to the bug reported in bug#13530:

Invoking `split -C T' or `split -C P' or `split -C E' will cause memory
exhaustion.

However, smaller units (e.g., K, M, G) work fine; bigger units (e.g., Z, Y)
fail properly (by outputing "invalid number of bytes").

A bit dig reveals that the bug is introduced at line 1412 of split.c
(coreutils 8.20):

1400: switch (split_type)
1401: {
      ...
1411:   case type_byteslines:
1412:     line_bytes_split (n_units); /* this function calls
xmalloc(n_bytes), which fails. */
1413:     break;
      ...
1434: }

`n_units' before that statement can be very large. In the case of `split -C
T', it is 1099511627776.

We think this is a bug similar to the one reported in bug#13530, so we
bring this issue to your attention. Thanks!


Environments:

$ uname -a
Linux anti-think 3.7.3-1-ARCH #1 SMP PREEMPT Thu Jan 17 18:52:30 CET 2013
x86_64 GNU/Linux

$ pacman -Qi coreutils
Name           : coreutils
Version        : 8.20-1
URL            : http://www.gnu.org/software/coreutils
Licenses       : GPL3
Groups         : base
Provides       : None
Depends On     : glibc  pam  acl  gmp  libcap
Optional Deps  : None
Required By    : ca-certificates  dbus  filesystem  linux  mkinitcpio  perl
 sysvinit-tools  util-linux
Conflicts With : None
Replaces       : None
Installed Size : 13820.00 KiB
Packager       : Allan McRae <allan <at> archlinux.org>
Architecture   : x86_64
Build Date     : Wed 24 Oct 2012 03:57:11 AM EDT
Install Date   : Sun 28 Oct 2012 01:51:06 PM EDT
Install Reason : Explicitly installed
Install Script : Yes
Description    : The basic file, shell and text manipulation utilities of
the GNU operating system

CPU: Intel(R) Core(TM) i5-3320M CPU @ 2.60GHz CPU
memory: 4GB

Best regards,

-- 
Lei Zhang
Department of Electrical and Computer Engineering
University of Waterloo
 200 University Avenue West
Waterloo, Ontario, Canada N2L 3G1

[Message part 2 (text/html, inline)]

Message #8 received at 13537 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bernhard Voelker <mail <at> bernhard-voelker.de>
Cc: 13537 <at> debbugs.gnu.org, coreutils <at> gnu.org,
	Ondrej Oprala <ooprala <at> redhat.com>
Subject: Re: [PATCH] delaying dd allocation
Date: Mon, 11 Feb 2013 01:39:17 +0000

On 02/10/2013 10:35 PM, Bernhard Voelker wrote:
> On 01/23/2013 11:48 AM, Ondrej Oprala wrote:
>> Hi Bernhard,
>> thanks for the remarks. Also, you were right about the corner case, the
>> succession of commands
>> $ touch a
>> $ dd if=a seek=1 bs=1
>> $ ^D
>> actually caused a segfault. Sorry, I didn't notice it before.
>>
>> I added a few lines to alloc_obuf, which cover the case when the initial
>> ibuf allocation is skipped but
>> obuf needs to be allocated and is to be set to ibuf and removed the
>> newly redundant check before memset.
>>
>> Thanks,
>> Ondrej
>
> Hi Padraig,
>
> I'm afraid this one slipped from my/our list. ;-/
>
> AFAIR that last patch was okay, but we should have a look
> on it again. Do you think we should include it in 8.21?

It has not slipped from my radar,
though I thought I has responded with my plans for that, sorry.

There is a whole class of heap allocation avoidance bugs
that I intend to merge/address early in the next cycle.
This dd one, one for head (13530), one for split (13537).

thanks,
Pádraig.

Message #13 received at 13537-done <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Lei Zhang <lei.zhang <at> uwaterloo.ca>
Cc: 13537-done <at> debbugs.gnu.org
Subject: Re: bug#13537: split: memory exhausted when putting at most T/P/E
	bytes of lines per output file
Date: Thu, 25 Apr 2013 02:34:41 +0100

[Message part 1 (text/plain, inline)]

On 01/23/2013 11:58 PM, Lei Zhang wrote:
> Hi All,
> 
> In the `split' program of coreutils 8.20, we found a bug which is similar
> to the bug reported in bug#13530:
> 
> Invoking `split -C T' or `split -C P' or `split -C E' will cause memory
> exhaustion.
> 
> However, smaller units (e.g., K, M, G) work fine; bigger units (e.g., Z, Y)
> fail properly (by outputing "invalid number of bytes").
> 
> A bit dig reveals that the bug is introduced at line 1412 of split.c
> (coreutils 8.20):
> 
> 1400: switch (split_type)
> 1401: {
>       ...
> 1411:   case type_byteslines:
> 1412:     line_bytes_split (n_units); /* this function calls
> xmalloc(n_bytes), which fails. */
> 1413:     break;
>       ...
> 1434: }
> 
> `n_units' before that statement can be very large. In the case of `split -C
> T', it is 1099511627776.
> 
> We think this is a bug similar to the one reported in bug#13530, so we
> bring this issue to your attention. Thanks!

I intend to push the attached soon to fix this.

thanks,
Pádraig.

[split-C-mem.patch (text/x-patch, attachment)]

Message #16 received at 13537 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: 13537 <at> debbugs.gnu.org, lei.zhang <at> uwaterloo.ca
Subject: Re: bug#13537: split: memory exhausted when putting at most T/P/E
	bytes of lines per output file
Date: Thu, 23 May 2013 11:08:26 +0100

On 04/25/2013 02:34 AM, Pádraig Brady wrote:
> On 01/23/2013 11:58 PM, Lei Zhang wrote:
>> Hi All,
>>
>> In the `split' program of coreutils 8.20, we found a bug which is similar
>> to the bug reported in bug#13530:
>>
>> Invoking `split -C T' or `split -C P' or `split -C E' will cause memory
>> exhaustion.
>>
>> However, smaller units (e.g., K, M, G) work fine; bigger units (e.g., Z, Y)
>> fail properly (by outputing "invalid number of bytes").
>>
>> A bit dig reveals that the bug is introduced at line 1412 of split.c
>> (coreutils 8.20):
>>
>> 1400: switch (split_type)
>> 1401: {
>>       ...
>> 1411:   case type_byteslines:
>> 1412:     line_bytes_split (n_units); /* this function calls
>> xmalloc(n_bytes), which fails. */
>> 1413:     break;
>>       ...
>> 1434: }
>>
>> `n_units' before that statement can be very large. In the case of `split -C
>> T', it is 1099511627776.
>>
>> We think this is a bug similar to the one reported in bug#13530, so we
>> bring this issue to your attention. Thanks!
> 
> I intend to push the attached soon to fix this.

Bernhard Voelker kindly reviewed this and noticed
I had an '|| fail=1' inside a subshell in the test.

Also he questioned the open coding of:
  error (EXIT_FAILURE, 0, "%s", _("memory exhausted"));
It's better to use the more abstract xalloc_die()
in this case (and another in lines_rr()).

I'm pushing now with these two adjustments.

thanks,
Pádraig.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.