GNU bug report logs - #22010
Python 2.7.10 not deterministic

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 22010 in the body.
You can then email your comments to 22010 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: bug-guix <at> gnu.org
Subject: Python 2.7.10 not deterministic
Date: Wed, 25 Nov 2015 15:38:54 +0100

As of v0.9.0-238-gd7572b4 (currently ‘tk-update’), Python 2.7.10 is not
reproducible:

--8<---------------cut here---------------start------------->8---
$ diff -r --no-dereference /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10 ,py
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/libpython2.7.so.1.0 and ,py/lib/libpython2.7.so.1.0 differ
diff -r --no-dereference /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/config/Makefile ,py/lib/python2.7/config/Makefile
31,32c31,32
< abs_srcdir=	/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
< abs_builddir=	/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
---
> abs_srcdir=	/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
> abs_builddir=	/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
146c146
< RUNSHARED=       LD_LIBRARY_PATH=/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
---
> RUNSHARED=       LD_LIBRARY_PATH=/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/getopt.pyc and ,py/lib/python2.7/getopt.pyc differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/getopt.pyo and ,py/lib/python2.7/getopt.pyo differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/lib2to3/Grammar2.7.10.final.0.pickle and ,py/lib/python2.7/lib2to3/Grammar2.7.10.final.0.pickle differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/lib-dynload/array.so and ,py/lib/python2.7/lib-dynload/array.so differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/lib-dynload/gdbm.so and ,py/lib/python2.7/lib-dynload/gdbm.so differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/lib-dynload/pyexpat.so and ,py/lib/python2.7/lib-dynload/pyexpat.so differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/py_compile.pyc and ,py/lib/python2.7/py_compile.pyc differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/py_compile.pyo and ,py/lib/python2.7/py_compile.pyo differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/struct.pyc and ,py/lib/python2.7/struct.pyc differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/struct.pyo and ,py/lib/python2.7/struct.pyo differ
diff -r --no-dereference /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/_sysconfigdata.py ,py/lib/python2.7/_sysconfigdata.py
458c458
<  'RUNSHARED': 'LD_LIBRARY_PATH=/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10',
---
>  'RUNSHARED': 'LD_LIBRARY_PATH=/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10',
502c502
<  'TESTPYTHON': 'LD_LIBRARY_PATH=/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10 ./python -Wd -3 -E -tt',
---
>  'TESTPYTHON': 'LD_LIBRARY_PATH=/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10 ./python -Wd -3 -E -tt',
526,527c526,527
<  'abs_builddir': '/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10',
<  'abs_srcdir': '/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10',
---
>  'abs_builddir': '/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10',
>  'abs_srcdir': '/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10',
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/_sysconfigdata.pyc and ,py/lib/python2.7/_sysconfigdata.pyc differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/_sysconfigdata.pyo and ,py/lib/python2.7/_sysconfigdata.pyo differ
--8<---------------cut here---------------end--------------->8---

The pyc/pyo differ by one 32-bit value starting on the 5th byte of the
file.

Ludo’.

Message #8 received at 22010 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: 22010 <at> debbugs.gnu.org
Subject: Re: bug#22010: Python 2.7.10 not deterministic
Date: Sun, 13 Dec 2015 19:17:54 +0100

ludo <at> gnu.org (Ludovic Courtès) skribis:

> As of v0.9.0-238-gd7572b4 (currently ‘tk-update’), Python 2.7.10 is not
> reproducible:
>
> $ diff -r --no-dereference /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10 ,py
> Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/libpython2.7.so.1.0 and ,py/lib/libpython2.7.so.1.0 differ
> diff -r --no-dereference /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/config/Makefile ,py/lib/python2.7/config/Makefile
> 31,32c31,32
> < abs_srcdir=	/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
> < abs_builddir=	/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
> ---
>> abs_srcdir=	/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
>> abs_builddir=	/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10

This problem is sidestepped with commit cb96010, where the daemon always
use /tmp/nix-build*.drv-0 inside the build environment.

It will disappear when we have upgraded the daemon on the build
machines.

Ludo’.

Message #11 received at 22010 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: 22010 <at> debbugs.gnu.org
Subject: bytecode timestamps
Date: Sun, 7 Feb 2016 18:58:39 -0500

Those differing bytes are the timestamps of the .py sources files that
correspond to the compiled .pyc / .pyo files. So, the
python-2.7-source-date-epoch.patch is working for software compiled by
the patched compiler, but not on python-2.7 itself.

It's related to <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22533>.

Message #16 received at 22010 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: Leo Famulari <leo <at> famulari.name>
Cc: 22010 <at> debbugs.gnu.org
Subject: Re: bug#22010: bytecode timestamps
Date: Tue, 05 Feb 2019 13:25:54 +0100

Leo Famulari <leo <at> famulari.name> writes:

> Those differing bytes are the timestamps of the .py sources files that
> correspond to the compiled .pyc / .pyo files. So, the
> python-2.7-source-date-epoch.patch is working for software compiled by
> the patched compiler, but not on python-2.7 itself.
>
> It's related to <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22533>.

It looks easy to fix this:

--8<---------------cut here---------------start------------->8---
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 1c4ea720f..34c01bd50 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -276,6 +276,38 @@ data types.")
                         (search-patches "python2-CVE-2018-14647.patch"
                                         "python2-CVE-2018-1000802.patch")))))))
 
+(define-public python-2-rekado
+  (package
+    (inherit python-2)
+    (name "python2-rekado")
+    (arguments
+     `(#:tests? #f ; XXX
+       ,@(substitute-keyword-arguments (package-arguments python-2)
+           ((#:phases phases)
+            `(modify-phases ,phases
+               (add-after 'remove-tests 'rebuild-bytecode
+                 (lambda* (#:key outputs #:allow-other-keys)
+                   (let ((out (assoc-ref outputs "out")))
+                     ;; Disable hash randomization to ensure the generated .pycs
+                     ;; are reproducible.
+                     (setenv "PYTHONHASHSEED" "0")
+                     (for-each
+                      (lambda (opt)
+                        (format #t "Compiling with optimization level: ~a\n"
+                                (if (null? opt) "none" (car opt)))
+                        (for-each (lambda (file)
+                                    (apply invoke
+                                           `(,(string-append out "/bin/python")
+                                             ,@opt
+                                             "-m" "compileall"
+                                             "-f" ; force rebuild
+                                             ;; Don't build lib2to3, because it contains Python 3 code.
+                                             "-x" "lib2to3/.*"
+                                             ,file)))
+                                  (find-files out "\\.py$")))
+                      (list '() '("-O") '("-OO")))
+                     #t))))))))))
+
 (define-public python2-called-python
   ;; Both 2.x and 3.x used to be called "python".  In commit
   ;; a7714d42de2c3082f3609d1e63c83d703fb39cf9 (March 2018), we renamed
   the
--8<---------------cut here---------------end--------------->8---

This Python variant builds reproducibly (I disabled the tests for speed
and did this in a separate package to avoid grafting shenanigans and
rebuilds).  The phase is virtually the same as the phase of the same
name in the Python 3 package; only difference is the use of
“/bin/python” instead of “/bin/python3”.

I wonder if we can simplify this with (setenv "PYTHONHASHSEED" "0")
before any pyc files are built, but I expect that to be problematic.

~~ Ricardo

Message #19 received at 22010 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: Leo Famulari <leo <at> famulari.name>
Cc: 22010 <at> debbugs.gnu.org
Subject: Re: bug#22010: bytecode timestamps
Date: Tue, 05 Feb 2019 21:29:59 +0100

Ricardo Wurmus <rekado <at> elephly.net> writes:

> I wonder if we can simplify this with (setenv "PYTHONHASHSEED" "0")
> before any pyc files are built, but I expect that to be problematic.

Doing this has no discernable effect.

I’ll push the change to the core-updates branch.

-- 
Ricardo

Message #24 received at 22010-done <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: Leo Famulari <leo <at> famulari.name>
Cc: 22010-done <at> debbugs.gnu.org
Subject: Re: bug#22010: bytecode timestamps
Date: Fri, 08 Feb 2019 10:45:19 +0100

Ricardo Wurmus <rekado <at> elephly.net> writes:

> I’ll push the change to the core-updates branch.

This is now on core-updates.  Python 2 now builds reproducibly.

--
Ricardo

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.