GNU bug report logs -
#20765
Python .egg files must not be compressed
Previous Next
Reported by: ludo <at> gnu.org (Ludovic Courtès)
Date: Sun, 7 Jun 2015 20:38:02 UTC
Severity: serious
Done: Leo Famulari <leo <at> famulari.name>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 20765 in the body.
You can then email your comments to 20765 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-guix <at> gnu.org
:
bug#20765
; Package
guix
.
(Sun, 07 Jun 2015 20:38:02 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
ludo <at> gnu.org (Ludovic Courtès)
:
New bug report received and forwarded. Copy sent to
bug-guix <at> gnu.org
.
(Sun, 07 Jun 2015 20:38:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
The other day on IRC Ricardo and 宋文武 noted that python-pillow
currently comes with a compressed egg. Because it is compressed, the
daemon’s conservative scanning fails to see what store items it refers
to; in particular Ricardo noted that on his machine, python-pillow
refers to a non-existent store item for OpenJPEG.
To fix that, python-build-system must be tweaked to ask for uncompressed
eggs. I tried the attached patch, which uses the ‘--always-unzip’
option of easyinstall.
Unfortunately, that option is unsupported by some setup.py, such as that
of setuptools itself.
What would be the right way to do that?
TIA. :-)
Ludo’.
[Message part 2 (text/x-patch, inline)]
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -60,14 +60,20 @@
#:allow-other-keys)
"Install a given Python package."
(let* ((out (assoc-ref outputs "out"))
- (params (append (list (string-append "--prefix=" out))
+ (params (cons* (string-append "--prefix=" out)
+
+ ;; Make sure the .egg files are kept unzipped,
+ ;; otherwise the daemon's conservative scanning may
+ ;; not find store references embedded in it.
+ "--always-unzip"
+
configure-flags))
(python-version (get-python-version (assoc-ref inputs "python")))
(old-path (getenv "PYTHONPATH"))
(add-path (string-append out "/lib/python" python-version
"/site-packages/")))
- ;; create the module installation directory and add it to PYTHONPATH
- ;; to make setuptools happy
+ ;; Create the module installation directory and add it to PYTHONPATH
+ ;; to make setuptools happy.
(mkdir-p add-path)
(setenv "PYTHONPATH"
(string-append (if old-path
Severity set to 'serious' from 'normal'
Request was from
ludo <at> gnu.org (Ludovic Courtès)
to
control <at> debbugs.gnu.org
.
(Wed, 01 Jul 2015 08:24:05 GMT)
Full text and
rfc822 format available.
Information forwarded
to
bug-guix <at> gnu.org
:
bug#20765
; Package
guix
.
(Mon, 20 Jul 2015 21:24:02 GMT)
Full text and
rfc822 format available.
Message #10 received at 20765 <at> debbugs.gnu.org (full text, mbox):
Commit e5358a6 disables egg compression by augmenting ‘setup.cfg’ for
python-pillow only.
After:
--8<---------------cut here---------------start------------->8---
$ guix gc --references /gnu/store/jx7dmdrsgndzic0jqqs9hsljgxgs95kw-python-pillow-2.8.1
/gnu/store/1hcg2k1lfz7z64p27cjm156jj8y6ia1s-python-3.4.3
/gnu/store/29s2a1hfc47qa292pf9kd0k8m9djd3hf-libwebp-0.4.3
/gnu/store/61c1h5zlxcjq55z13i1n7inm0d1nfv0w-libtiff-4.0.3
/gnu/store/cpx9iibpdwi3wb81glpnnlxr9zra2iiv-bash-4.3.39
/gnu/store/fsy5s8wn2l542q05jn54lzp9wx5l9fdj-freetype-2.6
/gnu/store/j4w3bddy32p14mrd1a9d80435y50s3yz-zlib-1.2.7
/gnu/store/jawh8qxz9s9pn7mafq1gw9khpmq14qvl-python-setuptools-12.1
/gnu/store/jx7dmdrsgndzic0jqqs9hsljgxgs95kw-python-pillow-2.8.1
/gnu/store/jyk2nxd922wmls7xarx0npy11cvi2848-lcms-2.6
/gnu/store/lcsm84qj11fbg3wqwnpsl3x0p3b22f8s-python-wrapper-3.4.3
/gnu/store/mq8kb3z2qga5jb8vc3s832yp9pr4knm6-openjpeg-2.1.0
/gnu/store/n3xi1hzh43hz2i1qq78ajr1mrapif2pa-libjpeg-9a
/gnu/store/v7bvaklf40mmm9pq9bc3hilwvyj7dlwk-python-nose-1.3.4
/gnu/store/w29667jfv02s1hgmv0yp7nqyywvdv1fz-glibc-2.21
/gnu/store/z8xw06ns2xjc9v5iza2n7gprjhi7dda0-gcc-4.9.3-lib
--8<---------------cut here---------------end--------------->8---
Before:
--8<---------------cut here---------------start------------->8---
$ guix gc --references $(guix build python-pillow)
/gnu/store/1hcg2k1lfz7z64p27cjm156jj8y6ia1s-python-3.4.3
/gnu/store/cpx9iibpdwi3wb81glpnnlxr9zra2iiv-bash-4.3.39
/gnu/store/imi7gz1779h6559d5kh8w94p7kqx5fqv-python-pillow-2.8.1
/gnu/store/jawh8qxz9s9pn7mafq1gw9khpmq14qvl-python-setuptools-12.1
/gnu/store/lcsm84qj11fbg3wqwnpsl3x0p3b22f8s-python-wrapper-3.4.3
/gnu/store/v7bvaklf40mmm9pq9bc3hilwvyj7dlwk-python-nose-1.3.4
--8<---------------cut here---------------end--------------->8---
So clearly, I’d expect python-pillow to be more reliable now.
Now the question is how many packages end up installing compressed eggs,
and whether we can generalize that solution.
I think we should try doing the ‘setup.cfg’ dance in
python-build-system.scm after 0.8.3.
Thoughts?
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org
:
bug#20765
; Package
guix
.
(Tue, 13 Oct 2015 13:39:02 GMT)
Full text and
rfc822 format available.
Message #13 received at 20765 <at> debbugs.gnu.org (full text, mbox):
I stumbled upon a similar problem when building other Python packages.
For ‘python-patsy’, for example, I found that an egg archive is produced
no matter what I put in ‘setup.cfg’. As far as I can tell, “zip_ok”
defaults to “0” in recent versions of setuptools, so I don’t think the
fix you used on ‘python-pillow’ would have much effect in general.
I could only coerce the build system to install plain files by adding a
build phase like this:
(add-after 'unpack 'prevent-generation-of-egg-archive
(lambda _
(substitute* "setup.py"
(("from setuptools import setup")
"from distutils.core import setup"))
#t))
It turns out that when ‘setuptools.setup’ is used egg archives are
built, when ‘distutils.core.setup’ is used, however, this does not
happen.
~~ Ricardo
Information forwarded
to
bug-guix <at> gnu.org
:
bug#20765
; Package
guix
.
(Tue, 13 Oct 2015 14:11:02 GMT)
Full text and
rfc822 format available.
Message #16 received at 20765 <at> debbugs.gnu.org (full text, mbox):
Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de> skribis:
> I stumbled upon a similar problem when building other Python packages.
> For ‘python-patsy’, for example, I found that an egg archive is produced
> no matter what I put in ‘setup.cfg’. As far as I can tell, “zip_ok”
> defaults to “0” in recent versions of setuptools, so I don’t think the
> fix you used on ‘python-pillow’ would have much effect in general.
OK.
> I could only coerce the build system to install plain files by adding a
> build phase like this:
>
> (add-after 'unpack 'prevent-generation-of-egg-archive
> (lambda _
> (substitute* "setup.py"
> (("from setuptools import setup")
> "from distutils.core import setup"))
> #t))
>
> It turns out that when ‘setuptools.setup’ is used egg archives are
> built, when ‘distutils.core.setup’ is used, however, this does not
> happen.
But are distutils.core and setuptools the same thing? Replacing one by
the other sounds a bit scary, no?
Still it would be nice to have a generic solution. Maybe we should
patch setuptools itself to change its defaults?
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org
:
bug#20765
; Package
guix
.
(Tue, 13 Oct 2015 14:33:02 GMT)
Full text and
rfc822 format available.
Message #19 received at 20765 <at> debbugs.gnu.org (full text, mbox):
Ludovic Courtès <ludo <at> gnu.org> writes:
>> I could only coerce the build system to install plain files by adding a
>> build phase like this:
>>
>> (add-after 'unpack 'prevent-generation-of-egg-archive
>> (lambda _
>> (substitute* "setup.py"
>> (("from setuptools import setup")
>> "from distutils.core import setup"))
>> #t))
>>
>> It turns out that when ‘setuptools.setup’ is used egg archives are
>> built, when ‘distutils.core.setup’ is used, however, this does not
>> happen.
>
> But are distutils.core and setuptools the same thing? Replacing one by
> the other sounds a bit scary, no?
I’m not familiar with Python packaging. From what I understand,
distutils has much fewer features than setuptools; but in the few cases
that I tried replacing it did work out just fine.
> Still it would be nice to have a generic solution. Maybe we should
> patch setuptools itself to change its defaults?
This would probably be best, but I don’t understand setuptools enough to
propose a patch.
~~ Ricardo
Information forwarded
to
bug-guix <at> gnu.org
:
bug#20765
; Package
guix
.
(Tue, 09 Feb 2016 04:55:01 GMT)
Full text and
rfc822 format available.
Message #22 received at 20765 <at> debbugs.gnu.org (full text, mbox):
Another reason to not compress Python eggs is that the contents of the
zip archives introduce non-determinism through their timestamps.
You can see it in action:
$ git checkout fe17fb4a2c897fd9186f91887f5af63dd00d227a
$ ./pre-inst-env guix build --rounds=2 acme
If you save the output between rounds, you can use diffoscope to confirm
that the differences between the rounds are the timestamps of the
contents of the eggs.
Information forwarded
to
bug-guix <at> gnu.org
:
bug#20765
; Package
guix
.
(Tue, 09 Feb 2016 21:38:02 GMT)
Full text and
rfc822 format available.
Message #25 received at 20765 <at> debbugs.gnu.org (full text, mbox):
Leo Famulari <leo <at> famulari.name> skribis:
> Another reason to not compress Python eggs is that the contents of the
> zip archives introduce non-determinism through their timestamps.
Indeed, good catch!
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org
:
bug#20765
; Package
guix
.
(Wed, 06 Apr 2016 18:18:01 GMT)
Full text and
rfc822 format available.
Message #28 received at 20765 <at> debbugs.gnu.org (full text, mbox):
I have stumbled over the same issue for another python package:
python-pkgconfig (just committed) makes a system call to pkg-config.
I rewrote the system call to use an absolute path, but the non-scanning
of the compressed egg means that no reference to pkg-config is returned.
Spending a week with Python wizards, I asked around and got the following
advice, implemented in the package as
#:configure-flags '("--single-version-externally-managed" "--root=/")
Our Python build system adds these flags only to the install phase.
We could do this automatically in our Python build system, but we would
need to make sure to add these flags only when python-setuptools or
python2-setuptools are part of the input, since the flags are not recognised
by the setup.py from distutils.
Another advice was to replace the call to "python setup.py install"
by "pip install .", which apparently is becoming the standard approach
of installing Python packages, not only over the Internet, but also locally
(for which the "." stands).
What do you think?
Andreas
Information forwarded
to
bug-guix <at> gnu.org
:
bug#20765
; Package
guix
.
(Wed, 06 Apr 2016 20:53:02 GMT)
Full text and
rfc822 format available.
Message #31 received at 20765 <at> debbugs.gnu.org (full text, mbox):
Andreas Enge <andreas <at> enge.fr> skribis:
> Spending a week with Python wizards, I asked around and got the following
> advice, implemented in the package as
> #:configure-flags '("--single-version-externally-managed" "--root=/")
Woow, I wouldn’t have guessed that this has something to do with .egg
compression. :-)
> Our Python build system adds these flags only to the install phase.
>
> We could do this automatically in our Python build system, but we would
> need to make sure to add these flags only when python-setuptools or
> python2-setuptools are part of the input, since the flags are not recognised
> by the setup.py from distutils.
OK. If the above flags do the trick, then why not. The difficulty will
be to find a nice way to determine whether those flags can be passed.
> Another advice was to replace the call to "python setup.py install"
> by "pip install .", which apparently is becoming the standard approach
> of installing Python packages, not only over the Internet, but also locally
> (for which the "." stands).
Dunno, but that’s a different story. :-)
Thanks!
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org
:
bug#20765
; Package
guix
.
(Mon, 18 Apr 2016 19:29:01 GMT)
Full text and
rfc822 format available.
Message #34 received at 20765 <at> debbugs.gnu.org (full text, mbox):
Hi,
I support the proposal for switching to pip.
Switching to pip should be save, since this is the proposed standard way
for installing - and thus it should be save in the long run, too.
I suggest calling pip with this options (valid as of pip 8.1.1)
--no-deps Don't install package dependencies.
--no-binary :all: Do not use binary packages.
--no-index Ignore package index
--disable-pip-version-check
Since in the internet one can find issues with
--single-version-externally-managed I verified that pip will even work
for packages using distutils instead of setuptools. I did a test and
inspecting the software.
Here is what I did: I used a simply Python package which imports
distutils instead of setuptools. Installing the via pip in verbose-mode
shows this line (you do not need to look at the details):
Running command /tmp/myvenv/bin/python2.7 -u -c "import setuptools,
tokenize;__file__='/tmp/pip-Biwi3y-build/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__,
'exec'))" install --record /tmp/pip-KcE_9O-record/install-record.txt
--single-version-externally-managed --compile --install-headers
/tmp/myvenv/include/site/python2.7/sample
This basically runs the setup.py “wrapped” into setuptools [1]. The
source code of setuptools reviled that it is monkey-pathing distutils
(esp. distutils.dist.Distribution) to objects from setuptools. So the
more elaborate stuff of setuptools is used even if the setup.py only
import distutils.
Additionally I checked what --single-version-externally-managed does
(since the documentation [2] is not that clear): it simply makes
`install` use the "original" distutils install command, which was/is not
able to create zipped eggs. So here we are on the save side, too.
Tested with pip 8.1.1 and setuptools 20.3, but this same code is in pip
6.0 and setuptools 18.0 already (which are older than what is in guix
0.10.0)
[1] Technically this line is executing a command which import setuptools
and then executes setup.py the it the same scope/context.
[2]
https://pythonhosted.org/setuptools/setuptools.html#install-run-easy-install-or-old-style-installation.
--
Regards
Hartmut Goebel
| Hartmut Goebel | h.goebel <at> crazy-compilers.com |
| www.crazy-compilers.com | compilers which you thought are impossible |
Reply sent
to
Leo Famulari <leo <at> famulari.name>
:
You have taken responsibility.
(Sat, 17 Dec 2016 18:44:01 GMT)
Full text and
rfc822 format available.
Notification sent
to
ludo <at> gnu.org (Ludovic Courtès)
:
bug acknowledged by developer.
(Sat, 17 Dec 2016 18:44:02 GMT)
Full text and
rfc822 format available.
Message #39 received at 20765-done <at> debbugs.gnu.org (full text, mbox):
From what I can see, this bug is fixed by the new Python build system
introduced in commit 03e856ddf5d28bc61144effb1a393b73cb4a2d9f.
Please re-open the bug if I am mistaken.
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org
.
(Sun, 15 Jan 2017 12:24:04 GMT)
Full text and
rfc822 format available.
This bug report was last modified 7 years and 101 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.