GNU bug report logs - #26734
Snippets (even empty ones) of tar sources reset the timestamps of all files

Previous Next

To reply to this bug, email your comments to 26734 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Clément Lassieur <clement <at> lassieur.org>
To: bug-guix <at> gnu.org
Subject: Snippets (even empty ones) of tar sources reset the timestamps of all
 files
Date: Mon, 01 May 2017 15:57:00 +0200

Please consider the following package:

--8<---------------cut here---------------start------------->8---
(define-public hello
  (package
    (name "hello")
    (version "2.10")
    (source (origin
              (method url-fetch)
              (uri (string-append "mirror://gnu/hello/hello-" version
                                  ".tar.gz"))
              (sha256
               (base32
                "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))
              (modules '((guix build utils)))
              (snippet '(begin #t))))
    (build-system gnu-build-system)
    (synopsis "Hello, GNU world: An example GNU package")
    (description
     "GNU Hello prints the message \"Hello, world!\" and then exits.  It
serves as an example of standard GNU coding practices.  As such, it supports
command-line arguments, multiple languages, and so on.")
    (home-page "https://www.gnu.org/software/hello/")
    (license gpl3+)))
--8<---------------cut here---------------end--------------->8---

It is the 'hello' package with two extra lines that add an empty
snippet.

--8<---------------cut here---------------start------------->8---
(modules '((guix build utils)))
(snippet '(begin #t))
--8<---------------cut here---------------end--------------->8---

Now here is the output of 'ls -l' within the original (extracted)
'hello':

--8<---------------cut here---------------start------------->8---
-rw-r--r-- 1 clement users  93787 Nov 16  2014 ABOUT-NLS
-rw-r--r-- 1 clement users  43830 Nov 16  2014 aclocal.m4
-rw-r--r-- 1 clement users    593 Jul 19  2014 AUTHORS
drwxr-xr-x 3 clement users   4096 Nov 16  2014 build-aux/
-rw-r--r-- 1 clement users  12988 Nov 16  2014 ChangeLog
-rw-r--r-- 1 clement users  30632 Jul 19  2014 ChangeLog.O
-rw-r--r-- 1 clement users  32871 Nov 16  2014 config.in
-rwxr-xr-x 1 clement users 449922 Nov 16  2014 configure
-rw-r--r-- 1 clement users   2408 Oct  5  2014 configure.ac
drwxr-xr-x 2 clement users   4096 Nov 16  2014 contrib/
-rw-r--r-- 1 clement users  35147 Dec 12  2013 COPYING
drwxr-xr-x 2 clement users   4096 Nov 16  2014 doc/
-rw-r--r-- 1 clement users   4573 Nov 16  2014 GNUmakefile
-r--r--r-- 1 clement users   1400 Nov 16  2014 hello.1
-rw-r--r-- 1 clement users  15752 Dec 29  2013 INSTALL
drwxr-xr-x 2 clement users   4096 Nov 16  2014 lib/
drwxr-xr-x 2 clement users   4096 Nov 16  2014 m4/
-rw-r--r-- 1 clement users  63223 Nov 16  2014 maint.mk
-rw-r--r-- 1 clement users   4367 Aug  5  2014 Makefile.am
-rw-r--r-- 1 clement users 143282 Nov 16  2014 Makefile.in
drwxr-xr-x 2 clement users   4096 Nov 16  2014 man/
-rw-r--r-- 1 clement users   4023 Nov 16  2014 NEWS
drwxr-xr-x 2 clement users   4096 Nov 16  2014 po/
-rw-r--r-- 1 clement users   3582 Jul 19  2014 README
-rw-r--r-- 1 clement users   1571 Jul 19  2014 README-dev
-rw-r--r-- 1 clement users   3014 Nov 16  2014 README-release
drwxr-xr-x 2 clement users   4096 Nov 16  2014 src/
drwxr-xr-x 2 clement users   4096 Nov 16  2014 tests/
-rw-r--r-- 1 clement users    898 Jul 19  2014 THANKS
-rw-r--r-- 1 clement users     75 Jul 19  2014 TODO
--8<---------------cut here---------------end--------------->8---

And here is the same output within the modified 'hello' (with the empty
snippet):

--8<---------------cut here---------------start------------->8---
-rw-r--r-- 1 clement users  93787 Jan  1  1970 ABOUT-NLS
-rw-r--r-- 1 clement users  43830 Jan  1  1970 aclocal.m4
-rw-r--r-- 1 clement users    593 Jan  1  1970 AUTHORS
drwxr-xr-x 3 clement users   4096 Jan  1  1970 build-aux/
-rw-r--r-- 1 clement users  12988 Jan  1  1970 ChangeLog
-rw-r--r-- 1 clement users  30632 Jan  1  1970 ChangeLog.O
-rw-r--r-- 1 clement users  32871 Jan  1  1970 config.in
-rwxr-xr-x 1 clement users 449922 Jan  1  1970 configure
-rw-r--r-- 1 clement users   2408 Jan  1  1970 configure.ac
drwxr-xr-x 2 clement users   4096 Jan  1  1970 contrib/
-rw-r--r-- 1 clement users  35147 Jan  1  1970 COPYING
drwxr-xr-x 2 clement users   4096 Jan  1  1970 doc/
-rw-r--r-- 1 clement users   4573 Jan  1  1970 GNUmakefile
-r--r--r-- 1 clement users   1400 Jan  1  1970 hello.1
-rw-r--r-- 1 clement users  15752 Jan  1  1970 INSTALL
drwxr-xr-x 2 clement users   4096 Jan  1  1970 lib/
drwxr-xr-x 2 clement users   4096 Jan  1  1970 m4/
-rw-r--r-- 1 clement users  63223 Jan  1  1970 maint.mk
-rw-r--r-- 1 clement users   4367 Jan  1  1970 Makefile.am
-rw-r--r-- 1 clement users 143282 Jan  1  1970 Makefile.in
drwxr-xr-x 2 clement users   4096 Jan  1  1970 man/
-rw-r--r-- 1 clement users   4023 Jan  1  1970 NEWS
drwxr-xr-x 2 clement users   4096 Jan  1  1970 po/
-rw-r--r-- 1 clement users   3582 Jan  1  1970 README
-rw-r--r-- 1 clement users   1571 Jan  1  1970 README-dev
-rw-r--r-- 1 clement users   3014 Jan  1  1970 README-release
drwxr-xr-x 2 clement users   4096 Jan  1  1970 src/
drwxr-xr-x 2 clement users   4096 Jan  1  1970 tests/
-rw-r--r-- 1 clement users    898 Jan  1  1970 THANKS
-rw-r--r-- 1 clement users     75 Jan  1  1970 TODO
--8<---------------cut here---------------end--------------->8---

As you can see, the timestamps are not the same, although there is
nothing in the snippet.

I think the code is in guix/packages.scm (patch-and-repack).  Here is my
understanding of what happens: the presence of the snippet leads to 3
things:

1. extraction of the archive,
2. modification of some files (because of patches or snippet),
3. compression of the archive.

Step 3 sets all timestamps to zero so to avoid non-determinism in the
archive.  We obviously don't want archives to depend on the time of
their creation.  But I believe we should only reset the timestamps of
the files we modified in step 2.  Unmodified files should stay
unmodified.

What's the point?  Well, while working on the 0ad package, I realized
that building with 32 cores would always fail with a snippet (even if it
is empty) and always succeed without a snippet.  It took me a few hours
to understand this, and I put a comment in the 0ad package.  Maybe it is
a 0ad bug (some tricky race condition that depends of file timestamps),
but I believe it is also a Guix bug: an empty snippet should not change
in any way the binary output of a package.

I tried to patch 'patch-and-repack', but it triggers a full
rebuild...  WDYT?

Message #8 received at 26734 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Clément Lassieur <clement <at> lassieur.org>
Cc: 26734 <at> debbugs.gnu.org
Subject: Re: bug#26734: Snippets (even empty ones) of tar sources reset the
 timestamps of all files
Date: Tue, 02 May 2017 15:00:16 +0200

Hi!

Clément Lassieur <clement <at> lassieur.org> skribis:

> I think the code is in guix/packages.scm (patch-and-repack).  Here is my
> understanding of what happens: the presence of the snippet leads to 3
> things:
>
> 1. extraction of the archive,
> 2. modification of some files (because of patches or snippet),
> 3. compression of the archive.
>
> Step 3 sets all timestamps to zero so to avoid non-determinism in the
> archive.  We obviously don't want archives to depend on the time of
> their creation.  But I believe we should only reset the timestamps of
> the files we modified in step 2.  Unmodified files should stay
> unmodified.

I agree.

> What's the point?  Well, while working on the 0ad package, I realized
> that building with 32 cores would always fail with a snippet (even if it
> is empty) and always succeed without a snippet.  It took me a few hours
> to understand this, and I put a comment in the 0ad package.  Maybe it is
> a 0ad bug (some tricky race condition that depends of file timestamps),
> but I believe it is also a Guix bug: an empty snippet should not change
> in any way the binary output of a package.

Yeah I was bitten by the same problem recently.

> I tried to patch 'patch-and-repack', but it triggers a full
> rebuild...  WDYT?

Right, it’s expected to trigger a full rebuild, so this should be fixed
in ‘core-updates’.

I guess we’ll have to collect the timestamps of all non-symlink files¹
in step #1 and to reapply them with ‘set-file-time’ from (guix build
utils) after step #2.

Thoughts?  Would you like to do that?

Thanks,
Ludo’.

¹ Because Guile provides bindings for ‘utime’, which does not support
  setting timestamps on symlinks.

Message #11 received at 26734 <at> debbugs.gnu.org (full text, mbox):

From: Clément Lassieur <clement <at> lassieur.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 26734 <at> debbugs.gnu.org
Subject: Re: bug#26734: Snippets (even empty ones) of tar sources reset the
 timestamps of all files
Date: Tue, 02 May 2017 15:17:29 +0200

Ludovic Courtès <ludo <at> gnu.org> writes:

> Clément Lassieur <clement <at> lassieur.org> skribis:
>
>> I tried to patch 'patch-and-repack', but it triggers a full
>> rebuild...  WDYT?
>
> Right, it’s expected to trigger a full rebuild, so this should be fixed
> in ‘core-updates’.

Yes, but is there a way to test the patch on one package without having
first to rebuild everything?

> I guess we’ll have to collect the timestamps of all non-symlink files¹
> in step #1 and to reapply them with ‘set-file-time’ from (guix build
> utils) after step #2.

Does that mean that symlinks will still have their timestamps changed?
To me that is a half-solution...  Wouldn't it be easier to collect all
recently modified files (those modified by snippet and patches), and set
their timestamp to "1 January 1970", without changing the other files?
That means removing the --mtime option from tar at step 3.

> Thoughts?  Would you like to do that?

Sure :-)

> ¹ Because Guile provides bindings for ‘utime’, which does not support
>   setting timestamps on symlinks.

If the guile binding doesn't support setting timestamps on symlinks, I
guess we can still use another way, like a system touch.

Message #14 received at 26734 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Clément Lassieur <clement <at> lassieur.org>
Cc: 26734 <at> debbugs.gnu.org
Subject: Re: bug#26734: Snippets (even empty ones) of tar sources reset the
 timestamps of all files
Date: Wed, 03 May 2017 10:58:59 +0200

Clément Lassieur <clement <at> lassieur.org> skribis:

> Ludovic Courtès <ludo <at> gnu.org> writes:
>
>> Clément Lassieur <clement <at> lassieur.org> skribis:
>>
>>> I tried to patch 'patch-and-repack', but it triggers a full
>>> rebuild...  WDYT?
>>
>> Right, it’s expected to trigger a full rebuild, so this should be fixed
>> in ‘core-updates’.
>
> Yes, but is there a way to test the patch on one package without having
> first to rebuild everything?

I would add a snippet in ‘gnu-make-boot0’, which is the first package
being built, and then run:

  ./pre-inst-env guix build -S --rounds=2 \
    -e '(@@ (gnu packages commencement) gnu-make-boot0)'

>> I guess we’ll have to collect the timestamps of all non-symlink files¹
>> in step #1 and to reapply them with ‘set-file-time’ from (guix build
>> utils) after step #2.
>
> Does that mean that symlinks will still have their timestamps changed?

No, that means symlinks will still have their timestamps unchanged.  :-)

BTW, what timestamps to we put on the modified files?  We want that to
be deterministic so we cannot use the build time.  We cannot use a date
in the future, either.  We cannot use Jan. 1 1970 either because that
means that modified files may now be older than the unmodified files,
which may break build systems; for the same reason, we cannot leave the
mtime of modified files unchanged.

Now that I think about it, it’s not clear to me what can be done without
breaking something.

Thoughts?

>> ¹ Because Guile provides bindings for ‘utime’, which does not support
>>   setting timestamps on symlinks.
>
> If the guile binding doesn't support setting timestamps on symlinks, I
> guess we can still use another way, like a system touch.

Or we could add bindings for ‘futimes’.

Thanks,
Ludo’.

Message #17 received at 26734 <at> debbugs.gnu.org (full text, mbox):

From: Mark H Weaver <mhw <at> netris.org>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 26734 <at> debbugs.gnu.org,
 Clément Lassieur <clement <at> lassieur.org>
Subject: Re: bug#26734: Snippets (even empty ones) of tar sources reset the
 timestamps of all files
Date: Wed, 03 May 2017 17:45:20 -0400

ludo <at> gnu.org (Ludovic Courtès) writes:

> BTW, what timestamps to we put on the modified files?  We want that to
> be deterministic so we cannot use the build time.  We cannot use a date
> in the future, either.  We cannot use Jan. 1 1970 either because that
> means that modified files may now be older than the unmodified files,
> which may break build systems; for the same reason, we cannot leave the
> mtime of modified files unchanged.
>
> Now that I think about it, it’s not clear to me what can be done without
> breaking something.
>
> Thoughts?

We could set the timestamp of modified files to be 1 second newer than
the newest file in the original source archive.

      Mark

Message #20 received at 26734 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Mark H Weaver <mhw <at> netris.org>
Cc: 26734 <at> debbugs.gnu.org,
 Clément Lassieur <clement <at> lassieur.org>
Subject: Re: bug#26734: Snippets (even empty ones) of tar sources reset the
 timestamps of all files
Date: Thu, 04 May 2017 00:01:21 +0200

Mark H Weaver <mhw <at> netris.org> skribis:

> ludo <at> gnu.org (Ludovic Courtès) writes:
>
>> BTW, what timestamps to we put on the modified files?  We want that to
>> be deterministic so we cannot use the build time.  We cannot use a date
>> in the future, either.  We cannot use Jan. 1 1970 either because that
>> means that modified files may now be older than the unmodified files,
>> which may break build systems; for the same reason, we cannot leave the
>> mtime of modified files unchanged.
>>
>> Now that I think about it, it’s not clear to me what can be done without
>> breaking something.
>>
>> Thoughts?
>
> We could set the timestamp of modified files to be 1 second newer than
> the newest file in the original source archive.

Sounds like a good idea.

Thanks,
Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.