GNU bug report logs - #20765
Python .egg files must not be compressed

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 20765 in the body.
You can then email your comments to 20765 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: bug-guix <at> gnu.org
Cc: Ricardo Wurmus <rekado <at> elephly.net>, iyzsong <at> gmail.com
Subject: Python .egg files must not be compressed
Date: Sun, 07 Jun 2015 22:37:19 +0200

[Message part 1 (text/plain, inline)]

The other day on IRC Ricardo and 宋文武 noted that python-pillow
currently comes with a compressed egg.  Because it is compressed, the
daemon’s conservative scanning fails to see what store items it refers
to; in particular Ricardo noted that on his machine, python-pillow
refers to a non-existent store item for OpenJPEG.

To fix that, python-build-system must be tweaked to ask for uncompressed
eggs.  I tried the attached patch, which uses the ‘--always-unzip’
option of easyinstall.

Unfortunately, that option is unsupported by some setup.py, such as that
of setuptools itself.

What would be the right way to do that?

TIA.  :-)

Ludo’.

[Message part 2 (text/x-patch, inline)]

--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -60,14 +60,20 @@
                   #:allow-other-keys)
   "Install a given Python package."
   (let* ((out (assoc-ref outputs "out"))
-         (params (append (list (string-append "--prefix=" out))
+         (params (cons* (string-append "--prefix=" out)
+
+                        ;; Make sure the .egg files are kept unzipped,
+                        ;; otherwise the daemon's conservative scanning may
+                        ;; not find store references embedded in it.
+                        "--always-unzip"
+
                         configure-flags))
          (python-version (get-python-version (assoc-ref inputs "python")))
          (old-path (getenv "PYTHONPATH"))
          (add-path (string-append out "/lib/python" python-version
                                   "/site-packages/")))
-        ;; create the module installation directory and add it to PYTHONPATH
-        ;; to make setuptools happy
+    ;; Create the module installation directory and add it to PYTHONPATH
+    ;; to make setuptools happy.
     (mkdir-p add-path)
     (setenv "PYTHONPATH"
             (string-append (if old-path

Message #10 received at 20765 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: 20765 <at> debbugs.gnu.org
Cc: Ricardo Wurmus <rekado <at> elephly.net>, iyzsong <at> gmail.com
Subject: Re: bug#20765: Python .egg files must not be compressed
Date: Mon, 20 Jul 2015 23:22:35 +0200

Commit e5358a6 disables egg compression by augmenting ‘setup.cfg’ for
python-pillow only.

After:

--8<---------------cut here---------------start------------->8---
$ guix gc --references /gnu/store/jx7dmdrsgndzic0jqqs9hsljgxgs95kw-python-pillow-2.8.1
/gnu/store/1hcg2k1lfz7z64p27cjm156jj8y6ia1s-python-3.4.3
/gnu/store/29s2a1hfc47qa292pf9kd0k8m9djd3hf-libwebp-0.4.3
/gnu/store/61c1h5zlxcjq55z13i1n7inm0d1nfv0w-libtiff-4.0.3
/gnu/store/cpx9iibpdwi3wb81glpnnlxr9zra2iiv-bash-4.3.39
/gnu/store/fsy5s8wn2l542q05jn54lzp9wx5l9fdj-freetype-2.6
/gnu/store/j4w3bddy32p14mrd1a9d80435y50s3yz-zlib-1.2.7
/gnu/store/jawh8qxz9s9pn7mafq1gw9khpmq14qvl-python-setuptools-12.1
/gnu/store/jx7dmdrsgndzic0jqqs9hsljgxgs95kw-python-pillow-2.8.1
/gnu/store/jyk2nxd922wmls7xarx0npy11cvi2848-lcms-2.6
/gnu/store/lcsm84qj11fbg3wqwnpsl3x0p3b22f8s-python-wrapper-3.4.3
/gnu/store/mq8kb3z2qga5jb8vc3s832yp9pr4knm6-openjpeg-2.1.0
/gnu/store/n3xi1hzh43hz2i1qq78ajr1mrapif2pa-libjpeg-9a
/gnu/store/v7bvaklf40mmm9pq9bc3hilwvyj7dlwk-python-nose-1.3.4
/gnu/store/w29667jfv02s1hgmv0yp7nqyywvdv1fz-glibc-2.21
/gnu/store/z8xw06ns2xjc9v5iza2n7gprjhi7dda0-gcc-4.9.3-lib
--8<---------------cut here---------------end--------------->8---

Before:

--8<---------------cut here---------------start------------->8---
$ guix gc --references $(guix build python-pillow)
/gnu/store/1hcg2k1lfz7z64p27cjm156jj8y6ia1s-python-3.4.3
/gnu/store/cpx9iibpdwi3wb81glpnnlxr9zra2iiv-bash-4.3.39
/gnu/store/imi7gz1779h6559d5kh8w94p7kqx5fqv-python-pillow-2.8.1
/gnu/store/jawh8qxz9s9pn7mafq1gw9khpmq14qvl-python-setuptools-12.1
/gnu/store/lcsm84qj11fbg3wqwnpsl3x0p3b22f8s-python-wrapper-3.4.3
/gnu/store/v7bvaklf40mmm9pq9bc3hilwvyj7dlwk-python-nose-1.3.4
--8<---------------cut here---------------end--------------->8---

So clearly, I’d expect python-pillow to be more reliable now.

Now the question is how many packages end up installing compressed eggs,
and whether we can generalize that solution.

I think we should try doing the ‘setup.cfg’ dance in
python-build-system.scm after 0.8.3.

Thoughts?

Ludo’.

Message #13 received at 20765 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
To: <20765 <at> debbugs.gnu.org>
Cc: ludo <at> gnu.org
Subject: Python .egg files must not be compressed
Date: Tue, 13 Oct 2015 15:38:38 +0200

I stumbled upon a similar problem when building other Python packages.
For ‘python-patsy’, for example, I found that an egg archive is produced
no matter what I put in ‘setup.cfg’.  As far as I can tell, “zip_ok”
defaults to “0” in recent versions of setuptools, so I don’t think the
fix you used on ‘python-pillow’ would have much effect in general.

I could only coerce the build system to install plain files by adding a
build phase like this:

         (add-after 'unpack 'prevent-generation-of-egg-archive
          (lambda _
            (substitute* "setup.py"
              (("from setuptools import setup")
               "from distutils.core import setup"))
            #t))

It turns out that when ‘setuptools.setup’ is used egg archives are
built, when ‘distutils.core.setup’ is used, however, this does not
happen.

~~ Ricardo

Message #16 received at 20765 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
Cc: 20765 <at> debbugs.gnu.org
Subject: Re: Python .egg files must not be compressed
Date: Tue, 13 Oct 2015 16:10:14 +0200

Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de> skribis:

> I stumbled upon a similar problem when building other Python packages.
> For ‘python-patsy’, for example, I found that an egg archive is produced
> no matter what I put in ‘setup.cfg’.  As far as I can tell, “zip_ok”
> defaults to “0” in recent versions of setuptools, so I don’t think the
> fix you used on ‘python-pillow’ would have much effect in general.

OK.

> I could only coerce the build system to install plain files by adding a
> build phase like this:
>
>          (add-after 'unpack 'prevent-generation-of-egg-archive
>           (lambda _
>             (substitute* "setup.py"
>               (("from setuptools import setup")
>                "from distutils.core import setup"))
>             #t))
>
> It turns out that when ‘setuptools.setup’ is used egg archives are
> built, when ‘distutils.core.setup’ is used, however, this does not
> happen.

But are distutils.core and setuptools the same thing?  Replacing one by
the other sounds a bit scary, no?

Still it would be nice to have a generic solution.  Maybe we should
patch setuptools itself to change its defaults?

Ludo’.

Message #19 received at 20765 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 20765 <at> debbugs.gnu.org
Subject: Re: Python .egg files must not be compressed
Date: Tue, 13 Oct 2015 16:32:19 +0200

Ludovic Courtès <ludo <at> gnu.org> writes:

>> I could only coerce the build system to install plain files by adding a
>> build phase like this:
>>
>>          (add-after 'unpack 'prevent-generation-of-egg-archive
>>           (lambda _
>>             (substitute* "setup.py"
>>               (("from setuptools import setup")
>>                "from distutils.core import setup"))
>>             #t))
>>
>> It turns out that when ‘setuptools.setup’ is used egg archives are
>> built, when ‘distutils.core.setup’ is used, however, this does not
>> happen.
>
> But are distutils.core and setuptools the same thing?  Replacing one by
> the other sounds a bit scary, no?

I’m not familiar with Python packaging.  From what I understand,
distutils has much fewer features than setuptools; but in the few cases
that I tried replacing it did work out just fine.

> Still it would be nice to have a generic solution.  Maybe we should
> patch setuptools itself to change its defaults?

This would probably be best, but I don’t understand setuptools enough to
propose a patch.

~~ Ricardo

Message #22 received at 20765 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: 20765 <at> debbugs.gnu.org
Date: Mon, 8 Feb 2016 23:54:42 -0500

Another reason to not compress Python eggs is that the contents of the
zip archives introduce non-determinism through their timestamps.

You can see it in action:
$ git checkout fe17fb4a2c897fd9186f91887f5af63dd00d227a
$ ./pre-inst-env guix build --rounds=2 acme

If you save the output between rounds, you can use diffoscope to confirm
that the differences between the rounds are the timestamps of the
contents of the eggs.

Message #25 received at 20765 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Leo Famulari <leo <at> famulari.name>
Cc: 20765 <at> debbugs.gnu.org
Subject: Re: bug#20765: (no subject)
Date: Tue, 09 Feb 2016 22:37:35 +0100

Leo Famulari <leo <at> famulari.name> skribis:

> Another reason to not compress Python eggs is that the contents of the
> zip archives introduce non-determinism through their timestamps.

Indeed, good catch!

Ludo’.

Message #28 received at 20765 <at> debbugs.gnu.org (full text, mbox):

From: Andreas Enge <andreas <at> enge.fr>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 20765 <at> debbugs.gnu.org, Leo Famulari <leo <at> famulari.name>
Subject: Re: bug#20765: (no subject)
Date: Wed, 6 Apr 2016 20:17:24 +0200

I have stumbled over the same issue for another python package:
python-pkgconfig (just committed) makes a system call to pkg-config.
I rewrote the system call to use an absolute path, but the non-scanning
of the compressed egg means that no reference to pkg-config is returned.

Spending a week with Python wizards, I asked around and got the following
advice, implemented in the package as
  #:configure-flags '("--single-version-externally-managed" "--root=/")
Our Python build system adds these flags only to the install phase.

We could do this automatically in our Python build system, but we would
need to make sure to add these flags only when python-setuptools or
python2-setuptools are part of the input, since the flags are not recognised
by the setup.py from distutils.

Another advice was to replace the call to "python setup.py install"
by "pip install .", which apparently is becoming the standard approach
of installing Python packages, not only over the Internet, but also locally
(for which the "." stands).

What do you think?

Andreas

Message #31 received at 20765 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Andreas Enge <andreas <at> enge.fr>
Cc: 20765 <at> debbugs.gnu.org, Leo Famulari <leo <at> famulari.name>
Subject: Compressed eggs (Python)
Date: Wed, 06 Apr 2016 22:52:04 +0200

Andreas Enge <andreas <at> enge.fr> skribis:

> Spending a week with Python wizards, I asked around and got the following
> advice, implemented in the package as
>   #:configure-flags '("--single-version-externally-managed" "--root=/")

Woow, I wouldn’t have guessed that this has something to do with .egg
compression.  :-)

> Our Python build system adds these flags only to the install phase.
>
> We could do this automatically in our Python build system, but we would
> need to make sure to add these flags only when python-setuptools or
> python2-setuptools are part of the input, since the flags are not recognised
> by the setup.py from distutils.

OK.  If the above flags do the trick, then why not.  The difficulty will
be to find a nice way to determine whether those flags can be passed.

> Another advice was to replace the call to "python setup.py install"
> by "pip install .", which apparently is becoming the standard approach
> of installing Python packages, not only over the Internet, but also locally
> (for which the "." stands).

Dunno, but that’s a different story.  :-)

Thanks!

Ludo’.

Message #34 received at 20765 <at> debbugs.gnu.org (full text, mbox):

From: Hartmut Goebel <h.goebel <at> crazy-compilers.com>
To: 20765 <at> debbugs.gnu.org
Subject: Compressed eggs (Python)
Date: Mon, 18 Apr 2016 21:21:19 +0200

Hi,

I support the proposal for switching to pip.

Switching to pip should be save, since this is the proposed standard way
for installing - and thus it should be save in the long run, too.

I suggest calling pip with this options (valid as of pip 8.1.1)

--no-deps                        Don't install package dependencies.
--no-binary :all:                Do not use binary packages.
--no-index                        Ignore package index
--disable-pip-version-check

Since in the internet one can find issues with
--single-version-externally-managed I verified that pip will even work
for packages using distutils instead of setuptools. I did a test and
inspecting the software.

Here is what I did: I used a simply Python package which imports
distutils instead of setuptools. Installing the via pip in verbose-mode
shows this line (you do not need to look at the details):

    Running command /tmp/myvenv/bin/python2.7 -u -c "import setuptools,
tokenize;__file__='/tmp/pip-Biwi3y-build/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__,
'exec'))" install --record /tmp/pip-KcE_9O-record/install-record.txt
--single-version-externally-managed --compile --install-headers
/tmp/myvenv/include/site/python2.7/sample

This basically runs the setup.py “wrapped” into setuptools [1]. The
source code of setuptools reviled that it is monkey-pathing distutils
(esp. distutils.dist.Distribution) to objects from setuptools. So the
more elaborate stuff of setuptools is used even if the setup.py only
import distutils.

Additionally I checked what --single-version-externally-managed does
(since the documentation [2] is not that clear): it simply makes
`install` use the "original" distutils install command, which was/is not
able to create zipped eggs. So here we are on the save side, too.

Tested with pip 8.1.1 and setuptools 20.3, but this same code is in pip
6.0 and setuptools 18.0 already (which are older than what is in guix
0.10.0)


[1] Technically this line is executing a command which import setuptools
and then executes setup.py the it the same scope/context.
[2]
https://pythonhosted.org/setuptools/setuptools.html#install-run-easy-install-or-old-style-installation.


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel <at> crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Message #39 received at 20765-done <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Hartmut Goebel <h.goebel <at> crazy-compilers.com>
Cc: 20765-done <at> debbugs.gnu.org
Subject: Re: bug#20765: Compressed eggs (Python)
Date: Sat, 17 Dec 2016 13:43:34 -0500

From what I can see, this bug is fixed by the new Python build system
introduced in commit 03e856ddf5d28bc61144effb1a393b73cb4a2d9f.

Please re-open the bug if I am mistaken.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.