GNU bug report logs - #35283
ISO images are not reproducible

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 35283 in the body.
You can then email your comments to 35283 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Bug Guix <bug-guix <at> gnu.org>
Cc: "\"pelzflorian \(Florian Pelz\)\"" <pelzflorian <at> pelzflorian.de>
Subject: ISO images are not reproducible
Date: Mon, 15 Apr 2019 09:46:37 +0200

As noted by Florian in <https://issues.guix.info/issue/33639>, images
created by ‘guix system disk-image --file-system-type=iso9660’ are not
bit-reproducible.

Ludo’.

Message #10 received at 35283 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: "Thomas Schmitt" <scdbackup <at> gmx.net>
Cc: bug-xorriso <at> gnu.org, 35283 <at> debbugs.gnu.org, pelzflorian <at> pelzflorian.de
Subject: ISO images are not reproducible
Date: Fri, 19 Apr 2019 13:40:24 +0200

Hi,

(Moving discussion to <https://issues.guix.info/issue/35283>, which is
specifically about ISO image reproducibility issues.)

"Thomas Schmitt" <scdbackup <at> gmx.net> skribis:

> Florian Pelz wrote:
>>  The content is different at the beginning of the ISO image
>> (maybe padding or timestamps in the file system)
>
> That's to expect if not environment SOURCE_DATE_EPOCH is set and exported.
>
> SOURCE_DATE_EPOCH belongs to the specs of reproducible-builds.org. It
> is supposed to be either undefined or to contain a decimal number which
> tells the seconds since january 1st 1970. If it contains a number, then
> it is used for all timestamps and as seed of pseudo-random numbers like
> MBR id or GPT UUIDs.
>
> If all files and directories have the same names and the same content,
> then xorriso runs with the same arguments and the same SOURCE_DATE_EPOCH
> value are supposed to create byte-identical result ISOs.

By mounting the ISO image, I found that some files didn’t have their
timestamp reset: some files in /var/guix (easily fixed), but more
importantly those added by GRUB in /boot and /System.

Files added by ‘grub-mkrescue’ are “out of our control” so we would need
to patch ‘grub-mkrescue’ to honor SOURCE_DATE_EPOCH, for example.

However, after rereading the Xorriso manual, it seemed to me that if we
set SOURCE_DATE_EPOCH and pass:

  -volume_date all_file_dates set_to_mtime

then all the files would have the mtime specified by SOURCE_DATE_EPOCH,
which would solve the problem.

I tried it, but that’s not what happened.  What am I missing, Thomas?

Thanks,
Ludo’.

Message #13 received at 35283 <at> debbugs.gnu.org (full text, mbox):

From: "Thomas Schmitt" <scdbackup <at> gmx.net>
To: bug-xorriso <at> gnu.org
Cc: 35283 <at> debbugs.gnu.org, pelzflorian <at> pelzflorian.de
Subject: Re: ISO images are not reproducible
Date: Fri, 19 Apr 2019 14:46:48 +0200

Hi,

> Files added by ‘grub-mkrescue’ are “out of our control” so we would need
> to patch ‘grub-mkrescue’ to honor SOURCE_DATE_EPOCH, for example.

Google shows that patches have been proposed. But they seem not to
have made it into the source.

Vladimir Serbinko's answer here
  https://lists.gnu.org/archive/html/grub-devel/2015-12/msg00046.html
might be the reason. I understand that he demands uniqueness of UUIDs.

But that's not really a problem with reproducible ISOs. If pseudo-random
UUIDs depend deterministically on SOURCE_DATE_EPOCH, then collisions are
only to expect between ISOs made with the same seconds value.
This can also happen if non-reproducible ISOs are made while their
systems' clocks show the same time by mere incident.

So one should use SOURCE_DATE_EPOCH values with best possible entropy.
Not one humanly invented lucky number for all ISOs of a distro.

If ever two identical ISOs are offered to GRUB at boot time, it needs
some imagination to construct a problem if GRUB operates on the one
which was not used by the EFI firmware to start GRUB.


So when a reproducible ISO is made for the first time, its SOURCE_DATE_EPOCH
should be taken from "date +%s" and recorded for further runs.
The ISO will bear it as "Creation Time", like "2019021612165300".
The last two digits "00" are centiseconds and should be ignored even
if not "00".
If decoding that time back to seconds-since-1970 is cumbersome, one may
store the seconds value in a data file in the input tree of the ISO
before packing up by a xorriso run with SOURCE_DATE_EPOCH having that
value.


> after rereading the Xorriso manual, it seemed to me that if we
> set SOURCE_DATE_EPOCH and pass:
>   -volume_date all_file_dates set_to_mtime
> then all the files would have the mtime specified by SOURCE_DATE_EPOCH,
> which would solve the problem.

This is the support for ignoring atime and ctime changes of input files
but respecting their mtime changes.

If you want a fixed time for all three timestamps in all files, do:

  -volume_date all_file_dates ="$SOURCE_DATE_EPOCH"

The "=" announces seconds-since-1970 as time format. See -alter_date.

Note that in this proposal $SOURCE_DATE_EPOCH is evaluated by the shell,
not by xorriso. Depending on the way how xorriso is started, you need to
insert the actual number.


Have a nice day :)

Thomas

Message #18 received at 35283-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: "Thomas Schmitt" <scdbackup <at> gmx.net>
Cc: bug-xorriso <at> gnu.org, 35283-done <at> debbugs.gnu.org, pelzflorian <at> pelzflorian.de
Subject: Re: bug#35283: ISO images are not reproducible
Date: Sun, 21 Apr 2019 00:57:19 +0200

Hi Thomas,

"Thomas Schmitt" <scdbackup <at> gmx.net> skribis:

> If you want a fixed time for all three timestamps in all files, do:
>
>   -volume_date all_file_dates ="$SOURCE_DATE_EPOCH"

Thanks, that’s what I was missing.

It was still not the end of the story, but I have some good news: the
series of commits below allow me to build ISO images reproducibly!  \o/

  1b0b1651b1 gnu: mtools: 'mformat' initializes boot sector before writing it.
  5502fbd7fd gnu: valgrind: Add 3.15.0.
  605815023c vm: Use a fixed FAT serial number for 'efi.img' in ISO images.
  52b5fe5bcf gnu: grub: 'grub-mkrescue' honors 'GRUB_FAT_SERIAL_NUMBER'.
  6901b9248e vm: Reset file timestamps of the EFI image in ISO images.
  833480cc1f vm: Reset file timestamps in ISO images.

To check by yourself you can do, say:

  guix system disk-image --file-system-type=iso9660 \
     gnu/system/examples/bare-bones.tmpl

and then check the ISO derivation that was built as the last step above:

  guix build --check -K /gnu/store/…-image.iso.drv

Ludo’.

Message #21 received at 35283 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: info-mtools <at> gnu.org
Cc: 35283 <at> debbugs.gnu.org
Subject: [PATCH] mformat: initialize boot sector before writing it
Date: Sun, 21 Apr 2019 01:03:57 +0200

[Message part 1 (text/plain, inline)]

Hello,

While investigating reproducible ISO images for Guix¹, I found that
‘mformat’ would not initialize the boot sector before writing it.  This
led to non-deterministic FAT image contents.

The attached patch fixes that.

Thanks,
Ludo’.

¹ https://issues.guix.info/issue/35283

[mtools-mformat-uninitialized.patch (text/x-patch, inline)]

Fix a bug whereby 'mformat' could end up passing uninitialized bytes
to write(2).  This could be reproduced with:

  mformat -C -f 1440 -L 16  -N 77777777 -i  /tmp/x ::

where the output of /tmp/x would be non-deterministic.

Patch by Ludovic Courtès <ludo <at> gnu.org>.

--- mtools-4.0.23/mformat.c	2019-04-21 00:12:01.496116195 +0200
+++ mtools-4.0.23/mformat.c	2019-04-21 00:12:36.675967157 +0200
@@ -927,6 +927,7 @@ void mformat(int argc, char **argv, int
 	
 	char *endptr;
 
+	memset(&boot.bytes, '\0', sizeof boot);
 	hs = hs_set = 0;
 	argtracks = 0;
 	argheads = 0;

Message #24 received at 35283 <at> debbugs.gnu.org (full text, mbox):

From: "Thomas Schmitt" <scdbackup <at> gmx.net>
To: bug-xorriso <at> gnu.org
Cc: 35283 <at> debbugs.gnu.org, pelzflorian <at> pelzflorian.de
Subject: Re: ISO images are not reproducible
Date: Sun, 21 Apr 2019 10:17:59 +0200

Hi,

>   833480cc1f vm: Reset file timestamps in ISO images.

That's also a big solution for the problem of timestamps of synthetic files.

I understand that your plan for reproducibility is to make timestamps
completely insignificant. Radical but effective.

But since you set in commit 6901b9248e SOURCE_DATE_EPOCH to 1980, why not
use the same seconds value for the ISO file objects ?


>  6901b9248e vm: Reset file timestamps of the EFI image in ISO images.

Maybe the commit message should have mentioned that setting SOURCE_DATE_EPOCH
not only influences mformat underneath grub-mkrescue, but also the run
of xorriso, where it determines volume date timestamps and GPT individual
UUIDs.
(Other impacts of the variable get overridden by the
   -volume_date "all_file_dates"
 command in commit 833480cc1f.)


>   52b5fe5bcf gnu: grub: 'grub-mkrescue' honors 'GRUB_FAT_SERIAL_NUMBER'.

I still riddle why /efi.img in the 0.16.0 ISO has 1.4 MB of size
but grub-mkrescue.c uses mformat -f 2880, which is supposed to produce
a 2.8 MB FAT image.


>   1b0b1651b1 gnu: mtools: 'mformat' initializes boot sector before writing

How good are chances to bring such changes into upstream ?

I ask in the advance assumption that we find a way to make the mformat
image digestible for Florian's Macbook.
(It is clear now that the difference between failure and success is in
 mformat versus mkfs.fat. But the exact point of failure is not found yet.
 I place my bet on the partition entry with start LBA 0.)


Have a nice day :)

Thomas

Message #27 received at 35283 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: bug-grub <at> gnu.org
Cc: 35283 <at> debbugs.gnu.org
Subject: [PATCH] grub-mkrescue: Allow users to specify a FAT serial number
Date: Sun, 21 Apr 2019 18:32:21 +0200

[Message part 1 (text/plain, inline)]

Hello,

While investigating reproducible ISO images for Guix¹, I found that
‘grub-mkrescue’ would invoke ’mformat’ without the ‘-N’ option.
Consequently, ‘mformat’ would pick a random serial number, thereby
making the ‘efi.img’ build process non-deterministic.

I came up with the gross hack attached: the ‘grub-mkrescue’ caller can
set the ‘GRUB_FAT_SERIAL_NUMBER’ environment variable, which
‘grub-mkrescue’ translates into a ‘-N’ flag for ‘mformat’.

We could perhaps achieve the same result differently, for instance by
adding an option to ‘grub-mkrescue’.

WDYT?

Thanks,
Ludo’.

¹ https://issues.guix.info/issue/35283

[grub-efi-fat-serial-number.patch (text/x-patch, inline)]

Change 'grub-mkrescue' to honor the 'GRUB_FAT_SERIAL_NUMBER'
environment variable.  That way, the caller can specify a fixed
serial number (instead of the randomly chosen one) to create EFI
images (the 'efi.img' file) that are reproducible bit-for-bit.

Patch by Ludovic Courtès <ludo <at> gnu.org>.

--- grub-2.02/util/grub-mkrescue.c	2019-04-20 19:15:26.180242812 +0200
+++ grub-2.02/util/grub-mkrescue.c	2019-04-20 21:56:34.672370849 +0200
@@ -788,8 +788,15 @@ main (int argc, char *argv[])
 
       efiimgfat = grub_util_path_concat (2, iso9660_dir, "efi.img");
       int rv;
-      rv = grub_util_exec ((const char * []) { "mformat", "-C", "-f", "2880", "-L", "16", "-i",
-	    efiimgfat, "::", NULL });
+
+      const char *fat_serial_number = getenv ("GRUB_FAT_SERIAL_NUMBER");
+      const char *mformat_args[] =
+	{ "mformat", "-C", "-f", "2880", "-L", "16",
+	  fat_serial_number != NULL ? "-N" : "-C",
+	  fat_serial_number != NULL ? fat_serial_number : "-C",
+	  "-i", efiimgfat, "::", NULL };
+
+      rv = grub_util_exec (mformat_args);
       if (rv != 0)
 	grub_util_error ("`%s` invocation failed\n", "mformat");
       rv = grub_util_exec ((const char * []) { "mcopy", "-s", "-i", efiimgfat, efidir_efi, "::/", NULL });

Message #30 received at 35283 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: "Thomas Schmitt" <scdbackup <at> gmx.net>
Cc: bug-xorriso <at> gnu.org, 35283 <at> debbugs.gnu.org, pelzflorian <at> pelzflorian.de
Subject: Re: bug#35283: ISO images are not reproducible
Date: Sun, 21 Apr 2019 18:42:28 +0200

Hi,

"Thomas Schmitt" <scdbackup <at> gmx.net> skribis:

>>   833480cc1f vm: Reset file timestamps in ISO images.
>
> That's also a big solution for the problem of timestamps of synthetic files.
>
> I understand that your plan for reproducibility is to make timestamps
> completely insignificant. Radical but effective.
>
> But since you set in commit 6901b9248e SOURCE_DATE_EPOCH to 1980, why not
> use the same seconds value for the ISO file objects ?

Files in /gnu/store, by convention, all have their mtime set to 1 (one
second after the epoch).

>>  6901b9248e vm: Reset file timestamps of the EFI image in ISO images.
>
> Maybe the commit message should have mentioned that setting SOURCE_DATE_EPOCH
> not only influences mformat underneath grub-mkrescue, but also the run
> of xorriso, where it determines volume date timestamps and GPT individual
> UUIDs.
> (Other impacts of the variable get overridden by the
>    -volume_date "all_file_dates"
>  command in commit 833480cc1f.)

AFAICS, setting SOURCE_DATE_EPOCH didn’t have a noticeable impact on
Xorriso, or at least it was overridden by the “-volume_date” options
that I pass.

It’s crucial for me to have the mtime set to 1 for all the files on the
ISO; I wanted the 1980 setting to apply only to ‘efi.img’.

>>   52b5fe5bcf gnu: grub: 'grub-mkrescue' honors 'GRUB_FAT_SERIAL_NUMBER'.
>
> I still riddle why /efi.img in the 0.16.0 ISO has 1.4 MB of size
> but grub-mkrescue.c uses mformat -f 2880, which is supposed to produce
> a 2.8 MB FAT image.

I haven’t dig deep enough to provide a satisfactory answer.  :-)

>>   1b0b1651b1 gnu: mtools: 'mformat' initializes boot sector before writing
>
> How good are chances to bring such changes into upstream ?

I’ve emailed them (actually tried to, their mailing list rejected my
message.)  We’ll see!

Ludo’.

Message #33 received at 35283 <at> debbugs.gnu.org (full text, mbox):

From: "Thomas Schmitt" <scdbackup <at> gmx.net>
To: bug-xorriso <at> gnu.org
Cc: 35283 <at> debbugs.gnu.org, pelzflorian <at> pelzflorian.de
Subject: Re: ISO images are not reproducible
Date: Sun, 21 Apr 2019 20:44:47 +0200

Hi,

Ludovic Courtès wrote:
> AFAICS, setting SOURCE_DATE_EPOCH didn’t have a noticeable impact on
> Xorriso, or at least it was overridden by the “-volume_date” options
> that I pass.

Probably. Among the automatic grub-mkrescue options for xorriso's mkisofs
emulation is

  --modification-date=2019042117165600

The equivalent native command is

  -volume_date uuid 2019042117165600

So you indeed have to override this by an own subsequent command.

(SOURCE_DATE_EPOCH overrides defaults of xorriso. But commands or options
 override the overridden defaults.)


Have a nice day :)

Thomas

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.