Package: coreutils;
Reported by: "Mike Gerth" <m.gerth <at> avm.de>
Date: Mon, 15 Oct 2012 15:56:01 UTC
Severity: normal
Tags: fixed
Done: Assaf Gordon <assafgordon <at> gmail.com>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 12656 in the body.
You can then email your comments to 12656 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
View this report as an mbox folder, status mbox, maintainer mbox
bug-coreutils <at> gnu.org:bug#12656; Package coreutils.
(Mon, 15 Oct 2012 15:56:02 GMT) Full text and rfc822 format available."Mike Gerth" <m.gerth <at> avm.de>:bug-coreutils <at> gnu.org.
(Mon, 15 Oct 2012 15:56:02 GMT) Full text and rfc822 format available.Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
From: "Mike Gerth" <m.gerth <at> avm.de> To: bug-coreutils <at> gnu.org Subject: cp since 8.11 corrupts files Date: Mon, 15 Oct 2012 17:05:56 +0200
[Message part 1 (text/plain, inline)]
Hello,
I noticed an error in cp (8.11 upwards till 8.19) when copying many files
from an fragmented ext3fs (to an empty partition).
ERROR:
The copied file has exact length.
The SourceFile has in the middle a 4K-block filled with ZERO, the
DestFile this 4k-block and the previous 4k-block also filled with ZERO.
I can reproduce this:
Conditions:
*I have a strong fragmented 110GB ext3-partition with >300
Files
*I have to copy about 100 files (ca. 15GB) to another
HDD-Partition or ramdisk
*cp must be 8.11 or higher (compiled under SLES11SP2
kernel 3.0) (everything is ok with 6.11. 8.05..8.10)
*Command:
cp819 $FILES $DEST #here the corruption comes
*$FILES is a list of about 100 DatabaseFiles. The
last one is the file that becomes corrupted
I can _NOT_ reproduce this:
*with cp <8.11
*Command:
for F in $FILES; do cp -dp $SRC$F $DEST$F ;done
#no corruption
Reproduce:
*tested with SLES11SP1, SLES11SP2, Ubuntu 12.04
(cp less 8.11 is good, cp 8.11 upwards corrupts the file)
*when I bulk copy the about 100 files, the corruption always
happen with the same file and same position
*if I copy less files or the single file there is no corruption
*if I copy all files there are more corruptions
Additional info:
# ./fragments /mnt/Source/
Summary statistics for /mnt/Source/ including subdirectories:
blocks files fragmented percent fragm./file
---------------------------------------------------
1 1 0 0.00 1.00
<= 12 0 0 0.00 0.00
<= 524 0 0 0.00 0.00
<= 1036 0 0 0.00 0.00
<= 4108 11 11 100.00 54.00
> 4108 307 307 100.00 159.71
---------------------------------------------------
all 319 318 99.69 155.57
> 1 318 318 100.00 156.05
Fragmentation index: 0.428 percent.
frag: 49625 n: 318 n_blocks: 11513587 (n_frag: 318)
#cmp Julius.nsf /mnt/ramdisk/Julius.nsf
Julius.nsf /mnt/ramdisk/Julius.nsf differ: char 18395137, line 345000
18395137=4491*4K + 1
=======================================================================================
#dd if=JuliusGOOD.nsf bs=4096 skip=4491 |hexdump -C |more
00000000 16 00 00 ff 00 10 00 00 00 00 00 00 00 00 02 00
|................|
00000010 00 00 00 00 00 00 00 00 10 0f 00 00 00 00 00 00
|................|
00000020 af 2f 06 00 1d 00 16 0a 00 00 00 00 40 09 88 00
|./..........@...|
....
00000a30 00 00 00 6e 64 65 00 00 00 00 00 00 00 00 00 00
|...nde..........|
00000a40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00000c00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00000c10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00000e00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00000e10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00002000 18 00 00 ff 00 10 00 00 00 00 00 00 00 00 02 00
|................|
---------------
#dd if=/mnt/ramdisk/JuliusBAD.nsf bs=4096 skip=4491 |hexdump -C |more
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
00002000 18 00 00 ff 00 10 00 00 00 00 00 00 00 00 02 00
|................|
=======================================================================================
# filefrag JuliusGood.nsf -v
Filesystem type is: ef53
Filesystem cylinder groups is approximately 881
File size of JuliusGood.nsf is 175935488 (42953 blocks, blocksize 4096)
ext logical physical expected length flags
0 0 2008262 2 merged
1 2 2008281 2008263 7 merged
2 9 2008303 2008287 1 merged
3 10 2008307 2008303 2 merged
4 12 2008310 2008308 2 merged
5 14 2008319 2008311 1 merged
6 15 2008321 2008319 4 merged
7 19 2031582 2008324 2 merged
8 21 2033663 2031583 41 merged
9 62 2033706 2033703 1 merged
10 63 2033704 2033706 2 merged
11 65 2033707 2033705 22 merged
12 87 2033736 2033728 23 merged
13 110 7961027 2033758 1 merged
14 111 7961587 7961027 1 merged
15 112 7943051 7961587 2 merged
16 114 7932925 7943052 1 merged
17 115 7939414 7932925 1 merged
18 116 2053842 7939414 2 merged
19 119 7961063 2053843 1 merged
20 120 7961059 7961063 2 merged
21 122 2033759 7961060 160 merged
22 282 7943053 2033918 1 merged
23 283 7961035 7943053 1 merged
24 284 7961030 7961035 1 merged
25 285 2033919 7961030 108 merged
26 393 7961054 2034026 1 merged
27 394 7961039 7961054 1 merged
28 395 2034027 7961039 104 merged
29 499 2034143 2034130 3 merged
30 502 2034159 2034145 8 merged
31 510 7961047 2034166 1 merged
32 511 7961033 7961047 1 merged
33 512 2034167 7961033 494 merged
34 1006 2034666 2034660 1 merged
35 1007 2034670 2034666 29 merged
36 1036 2034701 2034698 478 merged
37 1514 2035182 2035178 510 merged
38 2024 2035697 2035691 24 merged
39 2048 2038674 2035720 1 merged
40 2049 2035721 2038674 11 merged
41 2060 2035733 2035731 385 merged
42 2445 2036120 2036117 2 merged
43 2447 2036128 2036121 1 merged
44 2448 2036136 2036128 1 merged
45 2449 2036518 2036136 1 merged
46 2450 2037030 2036518 634 merged
47 3084 2037665 2037663 117 merged
48 3201 2037784 2037781 886 merged
49 4087 7961058 2038669 1 merged
50 4088 7961049 7961058 1 merged
51 4089 2038670 7961049 4 merged
52 4093 2040616 2038673 1 merged
53 4094 2038675 2040616 14 merged
54 4108 2038690 2038688 291 merged
55 4399 7961036 2038980 1 merged
56 4400 7961050 7961036 1 merged
57 4401 2038981 7961050 10 merged
58 4411 2038993 2038990 9 merged
59 4431 2038991 2039001 2 merged
60 4433 7961064 2038992 9 merged
61 4453 2039002 7961072 10 merged
62 4463 2053899 2039011 1 merged
63 4464 7932921 2053899 1 merged
64 4465 7936120 7932921 1 merged
65 4466 2039012 7936120 18 merged
66 4484 7943055 2039029 1 merged
67 4485 7961056 7943055 1 merged
68 4486 7961041 7961056 1 merged
69 4487 7961045 7961041 2 merged
70 4489 7961053 7961046 1 merged
71 4490 7961031 7961053 1 merged
72 4491 7961034 7961031 1 merged
73 4493 7961044 7961034 1 merged
74 4494 2039030 7961044 285 merged
75 4779 2054871 2039314 1 merged
76 4780 2053844 2054871 1 merged
....
Copied corrupted File:
# filefrag /mnt/DEST/JuliusBad.nsf -v
Filesystem type is: ef53
Filesystem cylinder groups is approximately 881
File size of /mnt/DEST/gschouten.nsf is 175935488 (42953 blocks, blocksize
4096)
ext logical physical expected length flags
0 0 3146242 6 merged
1 6 3155036 3146247 6 merged
2 12 3155043 3155041 106 merged
3 119 3155149 917 merged
4 1036 3156068 3156065 1024 merged
5 2060 3157093 3157091 1024 merged
6 3084 3158118 3158116 1024 merged
7 4108 3159143 3159141 312 merged
8 4431 3159455 11 merged
9 4453 3159466 38 merged
10 4493 3159504 639 merged
11 5132 3160144 3160142 417 merged
12 5550 3160561 441 merged
......
###This is the fragmented,good SourceFile:
debugfs 1.41.9 (22-Aug-2009)
debugfs: stat JuliusGood.nsf
Inode: 102 Type: regular Mode: 0644 Flags: 0x0
Generation: 1499014015 Version: 0x00000000
User: 1001 Group: 1001 Size: 175935488
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 343576
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x505e346a -- Sat Sep 22 23:58:02 2012
atime: 0x506041bf -- Mon Sep 24 13:19:27 2012
mtime: 0x505e346a -- Sat Sep 22 23:58:02 2012
Size of extra inode fields: 4
BLOCKS:
(0-1):2008262-2008263, (2-8):2008281-2008287, (9):2008303,
(10-11):2008307-2008308, (IND):2008309, (12-13):2008310-2008311,
(14):2008319, (15-18):2008321-2008324, (19-20):2031582
-2031583, (21-61):2033663-2033703, (62):2033706, (63-64):2033704-2033705,
(65-86):2033707-2033728, (87-109):2033736-2033758, (110):7961027,
(111):7961587, (112-113):7943051-79430
52, (114):7932925, (115):7939414, (116-117):2053842-2053843,
(119):7961063, (120-121):7961059-7961060, (122-281):2033759-2033918,
(282):7943053, (283):7961035, (284):7961030, (28
5-392):2033919-2034026, (393):7961054, (394):7961039,
(395-498):2034027-2034130, (499-501):2034143-2034145,
(502-509):2034159-2034166, (510):7961047, (511):7961033, (512-1005):20
34167-2034660, (1006):2034666, (1007-1035):2034670-2034698,
(DIND):2034699, (IND):2034700, (1036-1513):2034701-2035178,
(1514-2023):2035182-2035691, (2024-2047):2035697-2035720,
(2048):2038674, (2049-2059):2035721-2035731, (IND):2035732,
(2060-2444):2035733-2036117, (2445-2446):2036120-2036121, (2447):2036128,
(2448):2036136, (2449):2036518, (2450-3083):
2037030-2037663, (IND):2037664, (3084-3200):2037665-2037781,
(3201-4086):2037784-2038669, (4087):7961058, (4088):7961049,
(4089-4092):2038670-2038673, (4093):2040616, (4094-4107)
:2038675-2038688, (IND):2038689, (4108-4398):2038690-2038980,
(4399):7961036, (4400):7961050, (4401-4410):2038981-2038990,
(4411-4419):2038993-2039001, (4431-4432):2038991-203899
2, (4433-4441):7961064-7961072, (4453-4462):2039002-2039011,
(4463):2053899, (4464):7932921, (4465):7936120,
(4466-4483):2039012-2039029, (4484):7943055, (4485):7961056, (4486):7
961041, (4487-4488):7961045-7961046, (4489):7961053, (4490):7961031,
(4491):7961034, (4493):7961044, (4494-4778):2039030-2039314,
(4779):2054871, (4780):2053844, (4781-4800):2039
###Here comes the corrupted File:
debugfs 1.41.9 (22-Aug-2009)
debugfs: stat JuliusBad.nsf
Inode: 114 Type: regular Mode: 0644 Flags: 0x0
Generation: 3620008692 Version: 0x00000000
User: 0 Group: 0 Size: 175935488
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 343568
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x507831f2 -- Fri Oct 12 17:06:26 2012
atime: 0x507831f0 -- Fri Oct 12 17:06:24 2012
mtime: 0x507831f2 -- Fri Oct 12 17:06:26 2012
Size of extra inode fields: 4
BLOCKS:
(0-5):3146242-3146247, (6-11):3155036-3155041, (IND):3155042,
(12-117):3155043-3155148, (119-1035):3155149-3156065, (DIND):3156066,
(IND):3156067, (1036-2059):3156068-3157091, (IND):315709
2, (2060-3083):3157093-3158116, (IND):3158117,
(3084-4107):3158118-3159141, (IND):3159142, (4108-4419):3159143-3159454,
(4431-4441):3159455-3159465, (4453-4490):3159466-3159503, (4493-5131
):3159504-3160142,
I am 95% sure that in previous Tests if have also seen a corrupted file
with several 100k filled with ZERO.
Surely it would be most simple to reproduce the error if I could provide
the whole partition as is.
But since the Files are Database files I can not provide it. Are there any
tests that I can do?
Best regards and thanks in advance for your help,
Mike
Mit freundlichen Grüßen
Mike Gerth
Dipl. Inf. Mike Gerth
+49 30 399 76 136
IT-Systemadministrator
AVM Audiovisuelles Marketing und Computersysteme GmbH
Alt-Moabit 95, 10559 Berlin
HRB 23075 AG Charlottenburg
Geschäftsführer: Johannes Nill
[Message part 2 (text/html, inline)]
bug-coreutils <at> gnu.org:bug#12656; Package coreutils.
(Mon, 15 Oct 2012 17:53:02 GMT) Full text and rfc822 format available.Message #8 received at 12656 <at> debbugs.gnu.org (full text, mbox):
From: Jim Meyering <jim <at> meyering.net> To: "Mike Gerth" <m.gerth <at> avm.de> Cc: 12656 <at> debbugs.gnu.org Subject: Re: bug#12656: cp since 8.11 corrupts files Date: Mon, 15 Oct 2012 19:51:35 +0200
Mike Gerth wrote:
> I noticed an error in cp (8.11 upwards till 8.19) when copying many files
> from an fragmented ext3fs (to an empty partition).
>
> ERROR:
> The copied file has exact length.
> The SourceFile has in the middle a 4K-block filled with ZERO, the
> DestFile this 4k-block and the previous 4k-block also filled with ZERO.
>
>
> I can reproduce this:
> Conditions:
> *I have a strong fragmented 110GB ext3-partition with >300
> Files
> *I have to copy about 100 files (ca. 15GB) to another
> HDD-Partition or ramdisk
> *cp must be 8.11 or higher (compiled under SLES11SP2
> kernel 3.0) (everything is ok with 6.11. 8.05..8.10)
> *Command:
> cp819 $FILES $DEST #here the corruption comes
> *$FILES is a list of about 100 DatabaseFiles. The
> last one is the file that becomes corrupted
Thanks a lot for the detailed report.
...
> But since the Files are Database files I can not provide it. Are there any
> tests that I can do?
Can you run this command, and then send us the part of the LOG file
that reads and writes the file that is corrupted, i.e., only the
presumably small fraction at the end:
strace -s8 -oLOG cp819 $FILES $DEST
The interesting-to-us part will start with "open" syscalls that
operate on the src and dest corrupted files.
However, note that the read syscalls will expose a few bytes (8 per
block read -- per the -s8 option above) of the contents of your
final file. If you're not comfortable with that, let us know
and we'll try to come up with something else.
If LOG is larger than say 40KB, please compress and attach e.g., LOG.xz.
bug-coreutils <at> gnu.org:bug#12656; Package coreutils.
(Mon, 15 Oct 2012 22:03:01 GMT) Full text and rfc822 format available.Message #11 received at 12656 <at> debbugs.gnu.org (full text, mbox):
From: m.gerth <at> avm.de To: Jim Meyering <jim <at> meyering.net> Cc: 12656 <at> debbugs.gnu.org Subject: Re[2]: bug#12656: cp since 8.11 corrupts files Date: Tue, 16 Oct 2012 00:00:54 +0200
[Message part 1 (text/plain, inline)]
Hi Jim,
please find attached the output (compressed) of
strace -s8 -oLOG cp819 $FILES $DEST
about 2700 writes a 64kB.
Thanks,
Mike
Mit freundlichen Grüßen
Mike Gerth
Dipl. Inf. Mike Gerth
+49 30 399 76 136
IT-Systemadministrator
AVM Audiovisuelles Marketing und Computersysteme GmbH
Alt-Moabit 95, 10559 Berlin
HRB 23075 AG Charlottenburg
Geschäftsführer: Johannes Nill
Von: Jim Meyering <jim <at> meyering.net>
An: "Mike Gerth" <m.gerth <at> avm.de>
Kopie: 12656 <at> debbugs.gnu.org
Datum: 15.10.2012 19:51
Betreff: Re: bug#12656: cp since 8.11 corrupts files
Mike Gerth wrote:
> I noticed an error in cp (8.11 upwards till 8.19) when copying many
files
> from an fragmented ext3fs (to an empty partition).
>
> ERROR:
> The copied file has exact length.
> The SourceFile has in the middle a 4K-block filled with ZERO,
the
> DestFile this 4k-block and the previous 4k-block also filled with ZERO.
>
>
> I can reproduce this:
> Conditions:
> *I have a strong fragmented 110GB ext3-partition with
>300
> Files
> *I have to copy about 100 files (ca. 15GB) to another
> HDD-Partition or ramdisk
> *cp must be 8.11 or higher (compiled under SLES11SP2
> kernel 3.0) (everything is ok with 6.11. 8.05..8.10)
> *Command:
> cp819 $FILES $DEST #here the corruption comes
> *$FILES is a list of about 100 DatabaseFiles.
The
> last one is the file that becomes corrupted
Thanks a lot for the detailed report.
...
> But since the Files are Database files I can not provide it. Are there
any
> tests that I can do?
Can you run this command, and then send us the part of the LOG file
that reads and writes the file that is corrupted, i.e., only the
presumably small fraction at the end:
strace -s8 -oLOG cp819 $FILES $DEST
The interesting-to-us part will start with "open" syscalls that
operate on the src and dest corrupted files.
However, note that the read syscalls will expose a few bytes (8 per
block read -- per the -s8 option above) of the contents of your
final file. If you're not comfortable with that, let us know
and we'll try to come up with something else.
If LOG is larger than say 40KB, please compress and attach e.g., LOG.xz.
[Message part 2 (text/html, inline)]
[LOG.txt.bz2 (application/octet-stream, attachment)]
bug-coreutils <at> gnu.org:bug#12656; Package coreutils.
(Tue, 16 Oct 2012 08:26:02 GMT) Full text and rfc822 format available.Message #14 received at 12656 <at> debbugs.gnu.org (full text, mbox):
From: "Mike Gerth" <m.gerth <at> avm.de> To: "Alan Curry" <pacman-cu <at> kosh.dhis.org> Cc: Jim Meyering <jim <at> meyering.net>, 12656 <at> debbugs.gnu.org Subject: Re[3]: bug#12656: cp since 8.11 corrupts files Date: Tue, 16 Oct 2012 10:24:22 +0200
[Message part 1 (text/plain, inline)]
Hi Alan,
the difference between bulk copy (then the corruption comes) and single
file copy (everything ok) is:
diff /PatchLogBulk.txt /PatchLogSingle.txt
635c635
< Julius.nsf merged extent#3 off=18239488 len=155648 flags=0x1000
---
> Julius.nsf merged extent#3 off=18239488 len=159744 flags=0x1000
Thanks,
Mike
Von: "Alan Curry" <pacman-cu <at> kosh.dhis.org>
An: m.gerth <at> avm.de
Kopie: jim <at> meyering.net (Jim Meyering), 12656 <at> debbugs.gnu.org
Datum: 16.10.2012 03:14
Betreff: Re: bug#12656: Re[2]: bug#12656: cp since 8.11 corrupts
files
The critical piece of the trace is here:
lseek(3, 18239488, SEEK_SET) = 18239488
lseek(4, 18239488, SEEK_SET) = 18239488
read(3, "\0\0\0\0\0\0\0\0"..., 65536) = 65536
write(4, "\0\0\0\0\0\0\0\0"..., 65536) = 65536
read(3, "?\0\r\0\20\0\0\37"..., 65536) = 65536
write(4, "?\0\r\0\20\0\0\37"..., 65536) = 65536
read(3, "\20\0\0\377\0\20\0\0"..., 24576) = 24576
write(4, "\20\0\0\377\0\20\0\0"..., 24576) = 24576
lseek(3, 18403328, SEEK_SET) = 18403328
lseek(4, 18403328, SEEK_SET) = 18403328
The 24576-byte read and write should be 4096 bytes longer. cp has got the
extents mixed up somehow.
Here are some quick and dirty patches to make cp dump what it thinks the
extents are at a couple of different stages. It would be interesting to
see
what they say about the troublesome file.
--- src/extent-scan.c 2012-10-15 19:24:55.112096929 -0500
+++ src/extent-scan.c 2012-10-15 19:56:22.533209742 -0500
@@ -131,6 +131,12 @@
sizeof (struct extent_info));
unsigned int i = 0;
+ for(i=0; i<fiemap->fm_mapped_extents; ++i)
+ fprintf(stderr, "fd %d extent#%d log=%llu len=%llu
flags=0x%llx\n",
+ scan->fd, i,
+ (unsigned long long)fm_extents[i].fe_logical,
+ (unsigned long long)fm_extents[i].fe_length,
+ (unsigned long long)fm_extents[i].fe_flags);
for (i = 0; i < fiemap->fm_mapped_extents; i++)
{
assert (fm_extents[i].fe_logical
--- src/copy.c 2012-10-15 18:55:39.397899266 -0500
+++ src/copy.c 2012-10-15 19:56:23.417209716 -0500
@@ -317,6 +317,12 @@
unsigned int i;
bool empty_extent = false;
+ for(i=0; i<scan.ei_count; ++i)
+ fprintf(stderr, "%s merged extent#%d off=%llu len=%llu
flags=0x%llx\n",
+ src_name, i,
+ (unsigned long long)scan.ext_info[i].ext_logical,
+ (unsigned long long)scan.ext_info[i].ext_length,
+ (unsigned long long)scan.ext_info[i].ext_flags);
for (i = 0; i < scan.ei_count || empty_extent; i++)
{
off_t ext_start;
--
Alan Curry
[Message part 2 (text/html, inline)]
[PatchlogSingle.txt.bz2 (application/octet-stream, attachment)]
[PatchlogBulk.txt.bz2 (application/octet-stream, attachment)]
bug-coreutils <at> gnu.org:bug#12656; Package coreutils.
(Tue, 16 Oct 2012 15:48:02 GMT) Full text and rfc822 format available.Message #17 received at 12656 <at> debbugs.gnu.org (full text, mbox):
From: Jim Meyering <jim <at> meyering.net> To: "Mike Gerth" <m.gerth <at> avm.de> Cc: Alan Curry <pacman-cu <at> kosh.dhis.org>, 12656 <at> debbugs.gnu.org Subject: Re: bug#12656: Re[3]: bug#12656: cp since 8.11 corrupts files Date: Tue, 16 Oct 2012 17:46:35 +0200
Mike Gerth wrote:
> Hi Alan,
> the difference between bulk copy (then the corruption comes) and single
> file copy (everything ok) is:
>
> diff /PatchLogBulk.txt /PatchLogSingle.txt
> 635c635
> < Julius.nsf merged extent#3 off=18239488 len=155648 flags=0x1000
> ---
>> Julius.nsf merged extent#3 off=18239488 len=159744 flags=0x1000
...
> Von: "Alan Curry" <pacman-cu <at> kosh.dhis.org>
> An: m.gerth <at> avm.de
> Kopie: jim <at> meyering.net (Jim Meyering), 12656 <at> debbugs.gnu.org
> Datum: 16.10.2012 03:14
> Betreff: Re: bug#12656: Re[2]: bug#12656: cp since 8.11 corrupts
> files
>
> The critical piece of the trace is here:
>
> lseek(3, 18239488, SEEK_SET) = 18239488
> lseek(4, 18239488, SEEK_SET) = 18239488
> read(3, "\0\0\0\0\0\0\0\0"..., 65536) = 65536
> write(4, "\0\0\0\0\0\0\0\0"..., 65536) = 65536
> read(3, "?\0\r\0\20\0\0\37"..., 65536) = 65536
> write(4, "?\0\r\0\20\0\0\37"..., 65536) = 65536
> read(3, "\20\0\0\377\0\20\0\0"..., 24576) = 24576
> write(4, "\20\0\0\377\0\20\0\0"..., 24576) = 24576
> lseek(3, 18403328, SEEK_SET) = 18403328
> lseek(4, 18403328, SEEK_SET) = 18403328
>
> The 24576-byte read and write should be 4096 bytes longer. cp has got the
> extents mixed up somehow.
>
> Here are some quick and dirty patches to make cp dump what it thinks the
> extents are at a couple of different stages. It would be interesting to
> see
> what they say about the troublesome file.
>
> --- src/extent-scan.c 2012-10-15 19:24:55.112096929 -0500
> +++ src/extent-scan.c 2012-10-15 19:56:22.533209742 -0500
> @@ -131,6 +131,12 @@
> sizeof (struct extent_info));
>
> unsigned int i = 0;
> + for(i=0; i<fiemap->fm_mapped_extents; ++i)
> + fprintf(stderr, "fd %d extent#%d log=%llu len=%llu
> flags=0x%llx\n",
> + scan->fd, i,
> + (unsigned long long)fm_extents[i].fe_logical,
> + (unsigned long long)fm_extents[i].fe_length,
> + (unsigned long long)fm_extents[i].fe_flags);
> for (i = 0; i < fiemap->fm_mapped_extents; i++)
> {
> assert (fm_extents[i].fe_logical
> --- src/copy.c 2012-10-15 18:55:39.397899266 -0500
> +++ src/copy.c 2012-10-15 19:56:23.417209716 -0500
> @@ -317,6 +317,12 @@
>
> unsigned int i;
> bool empty_extent = false;
> + for(i=0; i<scan.ei_count; ++i)
> + fprintf(stderr, "%s merged extent#%d off=%llu len=%llu
> flags=0x%llx\n",
> + src_name, i,
> + (unsigned long long)scan.ext_info[i].ext_logical,
> + (unsigned long long)scan.ext_info[i].ext_length,
> + (unsigned long long)scan.ext_info[i].ext_flags);
> for (i = 0; i < scan.ei_count || empty_extent; i++)
> {
> off_t ext_start;
Thanks for the additional info, Alan.
With that, I have deduced that cp was using freed memory
in this unusual case. How?
When a file was so fragmented that it took two or more FIEMAP ioctls
to obtain the list of extents, our "last_ei" pointer would be pointing
into a malloc'd buffer from the first ioctl, when that buffer has to
be enlarged (realloc'd) to accommodate the results of the second call.
If that realloc actually freed the first buffer, our last_ei pointer
(which we're about to use) was left pointing to that freed memory.
Mike, can you test this preliminary patch?
If it does the job, I'll add something like the above to
the log, write a NEWS entry and work on adding a test.
This can happen only with a FIEMAP-enabled copy.
From 25046bda75bba889e800a2df10c8c4393b106990 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering <at> redhat.com>
Date: Tue, 16 Oct 2012 17:43:49 +0200
Subject: [PATCH] cp: avoid data-corrupting free-memory-read
* src/extent-scan.c (extent_scan_read): Reset our last_ei
pointer whenever the parent buffer might have just been freed.
Reported by Mike Gerth in http://bugs.gnu.org/12656, and with
help from Alan Curry.
---
src/extent-scan.c | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/src/extent-scan.c b/src/extent-scan.c
index 0c25c57..5b47295 100644
--- a/src/extent-scan.c
+++ b/src/extent-scan.c
@@ -89,7 +89,7 @@ extern bool
extent_scan_read (struct extent_scan *scan)
{
unsigned int si = 0;
- struct extent_info *last_ei IF_LINT ( = scan->ext_info);
+ struct extent_info *last_ei = scan->ext_info;
while (true)
{
@@ -127,8 +127,12 @@ extent_scan_read (struct extent_scan *scan)
assert (scan->ei_count <= SIZE_MAX - fiemap->fm_mapped_extents);
scan->ei_count += fiemap->fm_mapped_extents;
- scan->ext_info = xnrealloc (scan->ext_info, scan->ei_count,
- sizeof (struct extent_info));
+ {
+ size_t prev_idx = last_ei - scan->ext_info;
+ scan->ext_info = xnrealloc (scan->ext_info, scan->ei_count,
+ sizeof (struct extent_info));
+ last_ei = scan->ext_info + prev_idx;
+ }
unsigned int i = 0;
for (i = 0; i < fiemap->fm_mapped_extents; i++)
--
1.8.0.rc2
bug-coreutils <at> gnu.org:bug#12656; Package coreutils.
(Tue, 16 Oct 2012 17:40:01 GMT) Full text and rfc822 format available.Message #20 received at 12656 <at> debbugs.gnu.org (full text, mbox):
From: m.gerth <at> avm.de To: Jim Meyering <jim <at> meyering.net> Cc: Alan Curry <pacman-cu <at> kosh.dhis.org>, 12656 <at> debbugs.gnu.org Subject: Re[4]: bug#12656: cp since 8.11 corrupts files Date: Tue, 16 Oct 2012 19:37:54 +0200
[Message part 1 (text/plain, inline)]
Hi Jim,
Congratulation - the patch works for me. Result looks good - no errors.
Tomorror I will test more extensive (copy all files and compare all
files).
The debug output from the first patch is now identical when using the bulk
copy and the single copy.
Best regards,
Mike
Von: Jim Meyering <jim <at> meyering.net>
An: "Mike Gerth" <m.gerth <at> avm.de>
Kopie: "Alan Curry" <pacman-cu <at> kosh.dhis.org>, 12656 <at> debbugs.gnu.org
Datum: 16.10.2012 17:46
Betreff: Re: bug#12656: Re[3]: bug#12656: cp since 8.11 corrupts
files
Mike Gerth wrote:
> Hi Alan,
> the difference between bulk copy (then the corruption comes) and single
> file copy (everything ok) is:
>
> diff /PatchLogBulk.txt /PatchLogSingle.txt
> 635c635
> < Julius.nsf merged extent#3 off=18239488 len=155648 flags=0x1000
> ---
>> Julius.nsf merged extent#3 off=18239488 len=159744 flags=0x1000
...
> Von: "Alan Curry" <pacman-cu <at> kosh.dhis.org>
> An: m.gerth <at> avm.de
> Kopie: jim <at> meyering.net (Jim Meyering), 12656 <at> debbugs.gnu.org
> Datum: 16.10.2012 03:14
> Betreff: Re: bug#12656: Re[2]: bug#12656: cp since 8.11 corrupts
> files
>
> The critical piece of the trace is here:
>
> lseek(3, 18239488, SEEK_SET) = 18239488
> lseek(4, 18239488, SEEK_SET) = 18239488
> read(3, "\0\0\0\0\0\0\0\0"..., 65536) = 65536
> write(4, "\0\0\0\0\0\0\0\0"..., 65536) = 65536
> read(3, "?\0\r\0\20\0\0\37"..., 65536) = 65536
> write(4, "?\0\r\0\20\0\0\37"..., 65536) = 65536
> read(3, "\20\0\0\377\0\20\0\0"..., 24576) = 24576
> write(4, "\20\0\0\377\0\20\0\0"..., 24576) = 24576
> lseek(3, 18403328, SEEK_SET) = 18403328
> lseek(4, 18403328, SEEK_SET) = 18403328
>
> The 24576-byte read and write should be 4096 bytes longer. cp has got
the
> extents mixed up somehow.
>
> Here are some quick and dirty patches to make cp dump what it thinks the
> extents are at a couple of different stages. It would be interesting to
> see
> what they say about the troublesome file.
>
> --- src/extent-scan.c 2012-10-15 19:24:55.112096929 -0500
> +++ src/extent-scan.c 2012-10-15 19:56:22.533209742 -0500
> @@ -131,6 +131,12 @@
> sizeof (struct extent_info));
>
> unsigned int i = 0;
> + for(i=0; i<fiemap->fm_mapped_extents; ++i)
> + fprintf(stderr, "fd %d extent#%d log=%llu len=%llu
> flags=0x%llx\n",
> + scan->fd, i,
> + (unsigned long long)fm_extents[i].fe_logical,
> + (unsigned long long)fm_extents[i].fe_length,
> + (unsigned long long)fm_extents[i].fe_flags);
> for (i = 0; i < fiemap->fm_mapped_extents; i++)
> {
> assert (fm_extents[i].fe_logical
> --- src/copy.c 2012-10-15 18:55:39.397899266 -0500
> +++ src/copy.c 2012-10-15 19:56:23.417209716 -0500
> @@ -317,6 +317,12 @@
>
> unsigned int i;
> bool empty_extent = false;
> + for(i=0; i<scan.ei_count; ++i)
> + fprintf(stderr, "%s merged extent#%d off=%llu len=%llu
> flags=0x%llx\n",
> + src_name, i,
> + (unsigned long long)scan.ext_info[i].ext_logical,
> + (unsigned long long)scan.ext_info[i].ext_length,
> + (unsigned long long)scan.ext_info[i].ext_flags);
> for (i = 0; i < scan.ei_count || empty_extent; i++)
> {
> off_t ext_start;
Thanks for the additional info, Alan.
With that, I have deduced that cp was using freed memory
in this unusual case. How?
When a file was so fragmented that it took two or more FIEMAP ioctls
to obtain the list of extents, our "last_ei" pointer would be pointing
into a malloc'd buffer from the first ioctl, when that buffer has to
be enlarged (realloc'd) to accommodate the results of the second call.
If that realloc actually freed the first buffer, our last_ei pointer
(which we're about to use) was left pointing to that freed memory.
Mike, can you test this preliminary patch?
If it does the job, I'll add something like the above to
the log, write a NEWS entry and work on adding a test.
This can happen only with a FIEMAP-enabled copy.
From 25046bda75bba889e800a2df10c8c4393b106990 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering <at> redhat.com>
Date: Tue, 16 Oct 2012 17:43:49 +0200
Subject: [PATCH] cp: avoid data-corrupting free-memory-read
* src/extent-scan.c (extent_scan_read): Reset our last_ei
pointer whenever the parent buffer might have just been freed.
Reported by Mike Gerth in http://bugs.gnu.org/12656, and with
help from Alan Curry.
---
src/extent-scan.c | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/src/extent-scan.c b/src/extent-scan.c
index 0c25c57..5b47295 100644
--- a/src/extent-scan.c
+++ b/src/extent-scan.c
@@ -89,7 +89,7 @@ extern bool
extent_scan_read (struct extent_scan *scan)
{
unsigned int si = 0;
- struct extent_info *last_ei IF_LINT ( = scan->ext_info);
+ struct extent_info *last_ei = scan->ext_info;
while (true)
{
@@ -127,8 +127,12 @@ extent_scan_read (struct extent_scan *scan)
assert (scan->ei_count <= SIZE_MAX - fiemap->fm_mapped_extents);
scan->ei_count += fiemap->fm_mapped_extents;
- scan->ext_info = xnrealloc (scan->ext_info, scan->ei_count,
- sizeof (struct extent_info));
+ {
+ size_t prev_idx = last_ei - scan->ext_info;
+ scan->ext_info = xnrealloc (scan->ext_info, scan->ei_count,
+ sizeof (struct extent_info));
+ last_ei = scan->ext_info + prev_idx;
+ }
unsigned int i = 0;
for (i = 0; i < fiemap->fm_mapped_extents; i++)
--
1.8.0.rc2
[Message part 2 (text/html, inline)]
bug-coreutils <at> gnu.org:bug#12656; Package coreutils.
(Tue, 16 Oct 2012 18:44:01 GMT) Full text and rfc822 format available.Message #23 received at 12656 <at> debbugs.gnu.org (full text, mbox):
From: Jim Meyering <jim <at> meyering.net> To: m.gerth <at> avm.de Cc: Alan Curry <pacman-cu <at> kosh.dhis.org>, 12656 <at> debbugs.gnu.org Subject: Re: bug#12656: cp since 8.11 corrupts files Date: Tue, 16 Oct 2012 20:41:47 +0200
m.gerth <at> avm.de wrote:
> Hi Jim,
>
> Congratulation - the patch works for me. Result looks good - no errors.
> Tomorror I will test more extensive (copy all files and compare all files).
> The debug output from the first patch is now identical when using the bulk copy
> and the single copy.
Great. Thanks for the confirmation.
I've reproduced the problem on an ext4 file system.
This shows the free-memory read, though the two files compare equal.
$ perl -e 'for (1..3333) { sysseek (*STDOUT, 4096, 1)' \
-e '&& syswrite (*STDOUT, "a" x 1024) or die "$!"}' > j
$ valgrind cp j j2
Memcheck, a memory error detector
Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
Command: cp j j2
Invalid read of size 4
at 0x4093D7: extent_scan_read (extent-scan.c:139)
by 0x407D46: copy_internal (copy.c:301)
by 0x40904B: copy (copy.c:2691)
by 0x404B93: do_copy (cp.c:762)
by 0x4037F7: main (cp.c:1170)
Address 0x4c4c170 is 1,696 bytes inside a block of size 1,728 free'd
at 0x4A0892E: realloc (vg_replace_malloc.c:632)
by 0x410700: xrealloc (xmalloc.c:63)
by 0x409392: extent_scan_read (xalloc.h:126)
by 0x407D46: copy_internal (copy.c:301)
by 0x40904B: copy (copy.c:2691)
by 0x404B93: do_copy (cp.c:762)
by 0x4037F7: main (cp.c:1170)
Invalid read of size 8
at 0x4094E8: extent_scan_read (extent-scan.c:141)
by 0x407D46: copy_internal (copy.c:301)
by 0x40904B: copy (copy.c:2691)
by 0x404B93: do_copy (cp.c:762)
by 0x4037F7: main (cp.c:1170)
Address 0x4c4c168 is 1,688 bytes inside a block of size 1,728 free'd
at 0x4A0892E: realloc (vg_replace_malloc.c:632)
by 0x410700: xrealloc (xmalloc.c:63)
by 0x409392: extent_scan_read (xalloc.h:126)
by 0x407D46: copy_internal (copy.c:301)
by 0x40904B: copy (copy.c:2691)
by 0x404B93: do_copy (cp.c:762)
by 0x4037F7: main (cp.c:1170)
Invalid read of size 8
at 0x4094EF: extent_scan_read (extent-scan.c:141)
by 0x407D46: copy_internal (copy.c:301)
by 0x40904B: copy (copy.c:2691)
by 0x404B93: do_copy (cp.c:762)
by 0x4037F7: main (cp.c:1170)
Address 0x4c4c160 is 1,680 bytes inside a block of size 1,728 free'd
at 0x4A0892E: realloc (vg_replace_malloc.c:632)
by 0x410700: xrealloc (xmalloc.c:63)
by 0x409392: extent_scan_read (xalloc.h:126)
by 0x407D46: copy_internal (copy.c:301)
by 0x40904B: copy (copy.c:2691)
by 0x404B93: do_copy (cp.c:762)
by 0x4037F7: main (cp.c:1170)
Now, let's see if I can reproduce the actual data corruption part.
This obviously reinforces my desire to switch from FIEMAP to
using SEEK_HOLE/SEEK_DATA as soon as possible.
bug-coreutils <at> gnu.org:bug#12656; Package coreutils.
(Tue, 16 Oct 2012 22:23:01 GMT) Full text and rfc822 format available.Message #26 received at 12656 <at> debbugs.gnu.org (full text, mbox):
From: m.gerth <at> avm.de To: Jim Meyering <jim <at> meyering.net> Cc: Alan Curry <pacman-cu <at> kosh.dhis.org>, 12656 <at> debbugs.gnu.org Subject: WG: Re[4]: bug#12656: cp since 8.11 corrupts files Date: Wed, 17 Oct 2012 00:20:47 +0200
[Message part 1 (text/plain, inline)]
Hi Jim,
different cp version produce different corruptions:
8.11: unpatched: 14 corrupted files; patched == OK
8.19: unpatched: 3 corrupted files; patched == OK
Regards,
Mike
[Message part 2 (text/html, inline)]
bug-coreutils <at> gnu.org:bug#12656; Package coreutils.
(Wed, 17 Oct 2012 08:46:02 GMT) Full text and rfc822 format available.Message #29 received at 12656 <at> debbugs.gnu.org (full text, mbox):
From: Jim Meyering <jim <at> meyering.net> To: m.gerth <at> avm.de Cc: Alan Curry <pacman-cu <at> kosh.dhis.org>, 12656 <at> debbugs.gnu.org Subject: Re: WG: Re[4]: bug#12656: cp since 8.11 corrupts files Date: Wed, 17 Oct 2012 10:44:33 +0200
m.gerth <at> avm.de wrote:
> different cp version produce different corruptions:
>
> 8.11: unpatched: 14 corrupted files; patched == OK
> 8.19: unpatched: 3 corrupted files; patched == OK
Thanks for the thorough testing.
Here's a complete patch:
Changes:
- added a comment to the code
- added NEWS
- added a test case
[testing note:
I would have preferred to create a test case that made the pre-patch
cp produce a corrupted result rather than just a valgrind-detected FMR,
but that would have required being able to create an input file with
multiple adjacent extents, and at least with ext4, I don't know off hand
how to do that (without resorting to creating/using a nearly full
partition?), and prefer to keep this a user-runnable (not root-only) test.
Suggestions welcome. ]
From a5365003c88f4fce6293827c13f90acd0b5bd0cc Mon Sep 17 00:00:00 2001
From: Jim Meyering <jim <at> meyering.net>
Date: Tue, 16 Oct 2012 17:43:49 +0200
Subject: [PATCH] cp: avoid data-corrupting free-memory-read
* src/extent-scan.c (extent_scan_read): Reset our last_ei
pointer whenever the parent buffer might have just been freed.
* tests/cp/fiemap-extent-FMR.sh: New test.
* tests/local.mk (all_tests): Add it.
* NEWS (Bug fixes): Mention it.
Reported by Mike Gerth in http://bugs.gnu.org/12656, and with
help from Alan Curry. Bug introduced in commit v8.10-60-g18f5a85.
---
NEWS | 4 ++++
src/extent-scan.c | 12 +++++++++---
tests/cp/fiemap-FMR.sh | 31 +++++++++++++++++++++++++++++++
tests/local.mk | 1 +
4 files changed, 45 insertions(+), 3 deletions(-)
create mode 100755 tests/cp/fiemap-FMR.sh
diff --git a/NEWS b/NEWS
index aff5bf1..46ce698 100644
--- a/NEWS
+++ b/NEWS
@@ -12,6 +12,10 @@ GNU coreutils NEWS -*- outline -*-
** Bug fixes
+ cp could read from freed memory and could even make corrupt copies.
+ This could happen only with a very fragmented input file and when using
+ its FIEMAP/extent-based copying code. [bug introduced in coreutils-8.11]
+
cp --no-preserve=mode now no longer preserves the original file's
permissions but correctly sets mode specified by 0666 & ~umask
diff --git a/src/extent-scan.c b/src/extent-scan.c
index 0c25c57..f962298 100644
--- a/src/extent-scan.c
+++ b/src/extent-scan.c
@@ -89,7 +89,7 @@ extern bool
extent_scan_read (struct extent_scan *scan)
{
unsigned int si = 0;
- struct extent_info *last_ei IF_LINT ( = scan->ext_info);
+ struct extent_info *last_ei = scan->ext_info;
while (true)
{
@@ -127,8 +127,14 @@ extent_scan_read (struct extent_scan *scan)
assert (scan->ei_count <= SIZE_MAX - fiemap->fm_mapped_extents);
scan->ei_count += fiemap->fm_mapped_extents;
- scan->ext_info = xnrealloc (scan->ext_info, scan->ei_count,
- sizeof (struct extent_info));
+ {
+ /* last_ei points into a buffer that may be freed via xnrealloc.
+ Record its offset and adjust after allocation. */
+ size_t prev_idx = last_ei - scan->ext_info;
+ scan->ext_info = xnrealloc (scan->ext_info, scan->ei_count,
+ sizeof (struct extent_info));
+ last_ei = scan->ext_info + prev_idx;
+ }
unsigned int i = 0;
for (i = 0; i < fiemap->fm_mapped_extents; i++)
diff --git a/tests/cp/fiemap-FMR.sh b/tests/cp/fiemap-FMR.sh
new file mode 100755
index 0000000..10c9e05
--- /dev/null
+++ b/tests/cp/fiemap-FMR.sh
@@ -0,0 +1,31 @@
+#!/bin/sh
+# Trigger a free-memory read bug in cp from coreutils-[8.11..8.19]
+
+# Copyright (C) 2012 Free Software Foundation, Inc.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+. "${srcdir=.}/tests/init.sh"; path_prepend_ ./src
+print_ver_ cp
+
+require_valgrind_
+require_perl_
+: ${PERL=perl}
+
+$PERL -e 'for (1..600) { sysseek (*STDOUT, 4096, 1)' \
+ -e '&& syswrite (*STDOUT, "a" x 1024) or die "$!"}' > j || fail=1
+valgrind --quiet --error-exitcode=3 cp j j2 || fail=1
+cmp j j2 || fail=1
+
+Exit $fail
diff --git a/tests/local.mk b/tests/local.mk
index 486bf31..5a237fa 100644
--- a/tests/local.mk
+++ b/tests/local.mk
@@ -418,6 +418,7 @@ all_tests = \
tests/cp/existing-perm-race.sh \
tests/cp/fail-perm.sh \
tests/cp/fiemap-empty.sh \
+ tests/cp/fiemap-FMR.sh \
tests/cp/fiemap-perf.sh \
tests/cp/fiemap-2.sh \
tests/cp/file-perm-race.sh \
--
1.8.0.rc2.11.gd25c58c
bug-coreutils <at> gnu.org:bug#12656; Package coreutils.
(Sun, 21 Oct 2012 09:44:02 GMT) Full text and rfc822 format available.Message #32 received at 12656 <at> debbugs.gnu.org (full text, mbox):
From: Bernhard Voelker <mail <at> bernhard-voelker.de> To: Jim Meyering <jim <at> meyering.net> Cc: m.gerth <at> avm.de, Alan Curry <pacman-cu <at> kosh.dhis.org>, 12656 <at> debbugs.gnu.org Subject: Re: bug#12656: WG: Re[4]: bug#12656: cp since 8.11 corrupts files Date: Sun, 21 Oct 2012 11:41:20 +0200
On 10/17/2012 10:44 AM, Jim Meyering wrote:
> From a5365003c88f4fce6293827c13f90acd0b5bd0cc Mon Sep 17 00:00:00 2001
> From: Jim Meyering <jim <at> meyering.net>
> Date: Tue, 16 Oct 2012 17:43:49 +0200
> Subject: [PATCH] cp: avoid data-corrupting free-memory-read
>
> * src/extent-scan.c (extent_scan_read): Reset our last_ei
> pointer whenever the parent buffer might have just been freed.
> * tests/cp/fiemap-extent-FMR.sh: New test.
> * tests/local.mk (all_tests): Add it.
> * NEWS (Bug fixes): Mention it.
> Reported by Mike Gerth in http://bugs.gnu.org/12656, and with
> help from Alan Curry. Bug introduced in commit v8.10-60-g18f5a85.
> ---
> NEWS | 4 ++++
> src/extent-scan.c | 12 +++++++++---
> tests/cp/fiemap-FMR.sh | 31 +++++++++++++++++++++++++++++++
> tests/local.mk | 1 +
> 4 files changed, 45 insertions(+), 3 deletions(-)
> create mode 100755 tests/cp/fiemap-FMR.sh
>
> diff --git a/NEWS b/NEWS
> index aff5bf1..46ce698 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -12,6 +12,10 @@ GNU coreutils NEWS -*- outline -*-
>
> ** Bug fixes
>
> + cp could read from freed memory and could even make corrupt copies.
> + This could happen only with a very fragmented input file and when using
> + its FIEMAP/extent-based copying code. [bug introduced in coreutils-8.11]
As the bug is in extent-scan.c, doesn't it also affect mv (and ginstall)?
In src/local.mk:
copy_sources = \
src/copy.c \
src/cp-hash.c \
src/extent-scan.c \
src/extent-scan.h
...
src_ginstall_SOURCES = src/install.c src/prog-fprintf.c $(copy_sources)
...
src_cp_SOURCES = src/cp.c $(copy_sources)
...
src_mv_SOURCES = src/mv.c src/remove.c $(copy_sources)
Have a nice day,
Berny
bug-coreutils <at> gnu.org:bug#12656; Package coreutils.
(Sun, 21 Oct 2012 10:01:01 GMT) Full text and rfc822 format available.Message #35 received at 12656 <at> debbugs.gnu.org (full text, mbox):
From: Jim Meyering <jim <at> meyering.net> To: Bernhard Voelker <mail <at> bernhard-voelker.de> Cc: m.gerth <at> avm.de, Alan Curry <pacman-cu <at> kosh.dhis.org>, 12656 <at> debbugs.gnu.org Subject: Re: bug#12656: WG: Re[4]: bug#12656: cp since 8.11 corrupts files Date: Sun, 21 Oct 2012 11:58:15 +0200
Bernhard Voelker wrote:
> On 10/17/2012 10:44 AM, Jim Meyering wrote:
...
>> ** Bug fixes
>>
>> + cp could read from freed memory and could even make corrupt copies.
>> + This could happen only with a very fragmented input file and when using
>> + its FIEMAP/extent-based copying code. [bug introduced in coreutils-8.11]
>
> As the bug is in extent-scan.c, doesn't it also affect mv (and ginstall)?
>
> In src/local.mk:
>
> copy_sources = \
> src/copy.c \
> src/cp-hash.c \
> src/extent-scan.c \
> src/extent-scan.h
>
> ...
> src_ginstall_SOURCES = src/install.c src/prog-fprintf.c $(copy_sources)
> ...
> src_cp_SOURCES = src/cp.c $(copy_sources)
> ...
> src_mv_SOURCES = src/mv.c src/remove.c $(copy_sources)
Good point. It does, not just by virtue of linking to copy.c,
but because each also enables sparse_mode:
$ grep sparse_mode.= src/{mv,install}.c
src/mv.c: x->sparse_mode = SPARSE_AUTO; /* FIXME: maybe make this an option */
src/install.c: x->sparse_mode = SPARSE_AUTO;
Thanks!
From d001e65bb1f218b7208d65b38e5f854e58a11e62 Mon Sep 17 00:00:00 2001
From: Jim Meyering <jim <at> meyering.net>
Date: Sun, 21 Oct 2012 11:56:44 +0200
Subject: [PATCH] doc: NEWS: the cp bug affects mv and install, too
* NEWS (Bug fixes): Note that the copy-induced corruption affects
mv (sometimes) and install, too. Pointed out by Bernhard Voelker.
---
NEWS | 1 +
1 file changed, 1 insertion(+)
diff --git a/NEWS b/NEWS
index 0fc2a94..bbd8f59 100644
--- a/NEWS
+++ b/NEWS
@@ -15,6 +15,7 @@ GNU coreutils NEWS -*- outline -*-
cp could read from freed memory and could even make corrupt copies.
This could happen with a very fragmented and sparse input file,
on GNU/Linux file systems supporting fiemap extent scanning.
+ This bug also affects mv when it resorts to copying, and install.
[bug introduced in coreutils-8.11]
cp --no-preserve=mode now no longer preserves the original file's
--
1.8.0.rc2.11.gd25c58c
Jim Meyering <meyering <at> hx.meyering.net>
to control <at> debbugs.gnu.org.
(Thu, 25 Oct 2012 20:01:02 GMT) Full text and rfc822 format available.Assaf Gordon <assafgordon <at> gmail.com>
to control <at> debbugs.gnu.org.
(Thu, 11 Oct 2018 22:35:02 GMT) Full text and rfc822 format available.Debbugs Internal Request <help-debbugs <at> gnu.org>
to internal_control <at> debbugs.gnu.org.
(Fri, 09 Nov 2018 12:24:08 GMT) Full text and rfc822 format available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.