GNU bug report logs -
#35322
AM_PATH_PYTHON incompatible with Python frameworks on macOS
Previous Next
Reported by: Chih-Hsuan Yen <yan12125 <at> gmail.com>
Date: Fri, 19 Apr 2019 15:40:02 UTC
Severity: normal
Tags: confirmed, help
Done: Karl Berry <karl <at> freefriends.org>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 35322 in the body.
You can then email your comments to 35322 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-automake <at> gnu.org
:
bug#35322
; Package
automake
.
(Fri, 19 Apr 2019 15:40:02 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Chih-Hsuan Yen <yan12125 <at> gmail.com>
:
New bug report received and forwarded. Copy sent to
bug-automake <at> gnu.org
.
(Fri, 19 Apr 2019 15:40:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
Hi Automake experts,
On macOS, if CPython is configured with --enable-framework (default
for MacPorts [1]), Python packages should be installed in
${prefix}/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages.
However, AM_PATH_PYTHON gives
pythondir=${prefix}/lib/python3.7/site-packages.
Here's a minimal example configure.ac:
AC_PREREQ(2.57)
AC_INIT([automake-python-test], 0.1)
AM_INIT_AUTOMAKE([foreign])
AM_PATH_PYTHON([2.7])
AC_CONFIG_FILES([Makefile])
The issue appears to be that AM_PATH_PYTHON overwrites the 'base'
variable. If we remove this overwrite like [2], pythondir contains the
correct path.
Related bug report in MacPorts: https://trac.macports.org/ticket/57329
Regards,
Chih-Hsuan Yen
[1] https://github.com/macports/macports-ports/blob/834d761210be0d989076000c454a4a1e32209af5/lang/python37/Portfile#L57
[2] https://github.com/macports/macports-ports/blob/master/devel/automake/files/patch-issue57329.diff
Added tag(s) help and confirmed.
Request was from
Karl Berry <karl <at> freefriends.org>
to
control <at> debbugs.gnu.org
.
(Sat, 13 Feb 2021 02:47:02 GMT)
Full text and
rfc822 format available.
Information forwarded
to
bug-automake <at> gnu.org
:
bug#35322
; Package
automake
.
(Sun, 14 Feb 2021 22:17:01 GMT)
Full text and
rfc822 format available.
Message #10 received at 35322 <at> debbugs.gnu.org (full text, mbox):
Hi - thanks for this report from almost two years ago (sorry).
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35322
On macOS, if CPython is configured with --enable-framework (default
for MacPorts [1]), Python packages should be installed in ...
Ok.
Here's a minimal example configure.ac:
Thanks for sending.
The issue appears to be that AM_PATH_PYTHON overwrites the 'base'
variable. If we remove this overwrite like [2], pythondir contains
the correct path.
Unfortunately, simply removing the change to the `base' assignment as
you did in your change, for instance the first hunk,
- sitedir = sysconfig.get_path('purelib', vars={'base':'$am_py_prefix'})
+ sitedir = sysconfig.get_path('purelib')
will affect any use of AM_PATH_PYTHON in an (it seems to me)
incompatible way. That does not seem viable. It's not even OS-dependent,
since other Mac users might not use --enable-framework.
Evidently the current default for am_py_prefix is analogous to how
Automake treats "lispdir" and others, to make for a coherent system. So
it's not desirable to change these variables' default values in a
system-dependent way.
I don't know enough about Python to know if there is some clever way to
handle this. In my naive way, it seems like you just have to override
the directory for your case. I think setting the shell variable
$am_cv_python_pythondir before running configure would do this, although
I confess I haven't tried it.
Sorry not to have a helpful reply. If you (or anyone) can think of a
patch that will fix your case while not breaking others, I'd like to see.
Anyway, I'll leave this open, marked as needing help to resolve. --best, karl.
[2] https://github.com/macports/macports-ports/blob/master/devel/automake/files/patch-issue57329.diff
Information forwarded
to
bug-automake <at> gnu.org
:
bug#35322
; Package
automake
.
(Mon, 15 Feb 2021 03:31:01 GMT)
Full text and
rfc822 format available.
Message #13 received at 35322 <at> debbugs.gnu.org (full text, mbox):
On 2021-2-15 09:16 , Karl Berry wrote:
> Unfortunately, simply removing the change to the `base' assignment as
> you did in your change, for instance the first hunk,
>
> - sitedir = sysconfig.get_path('purelib', vars={'base':'$am_py_prefix'})
> + sitedir = sysconfig.get_path('purelib')
>
> will affect any use of AM_PATH_PYTHON in an (it seems to me)
> incompatible way. That does not seem viable. It's not even OS-dependent,
> since other Mac users might not use --enable-framework.
>
> Evidently the current default for am_py_prefix is analogous to how
> Automake treats "lispdir" and others, to make for a coherent system. So
> it's not desirable to change these variables' default values in a
> system-dependent way.
A python installation configured with --disable-framework will have a
different value for base. I guess it's a question of whether the goal is
installing modules in a consistent location across different systems, or
installing them somewhere where the python on this system will be able
to find them.
The documentation says that pythondir is "The directory name for the
site-packages subdirectory of the standard Python install tree." I
suppose that could be interpreted different ways depending on your
chosen value of "standard". Bear in mind that macOS may be unique in
that it's normal for python's sys.prefix to be different (a few
subdirectories deeper) than the --prefix passed to a configure script. I
do wonder if the current approach was designed with the assumption that
they would always be the same.
> I don't know enough about Python to know if there is some clever way to
> handle this. In my naive way, it seems like you just have to override
> the directory for your case. I think setting the shell variable
> $am_cv_python_pythondir before running configure would do this, although
> I confess I haven't tried it.
Just to be clear, --enable-framework is not just the default for
MacPorts, it's the default on macOS in general in all recent python
versions. It's how the Mac binary installers from python.org are
configured. Python won't be able to find modules installed in the
location currently chosen by AM_PATH_PYTHON unless its configuration is
changed to look there.
I admit I seem not to totally understand the design here based on what
you've said above, but it seems like it would be better to have defaults
that result in a working installation.
- Josh
Information forwarded
to
bug-automake <at> gnu.org
:
bug#35322
; Package
automake
.
(Sat, 20 Feb 2021 02:27:01 GMT)
Full text and
rfc822 format available.
Message #16 received at 35322 <at> debbugs.gnu.org (full text, mbox):
Thanks for the follow-up.
a question of whether the goal is installing modules in a consistent
location across different systems, or installing them somewhere
where the python on this system will be able to find them.
Yes. Ordinarily, install directories in Automake are supposed to be the
same across systems. But from what you quoted from the manual:
The documentation says that pythondir is "The directory name for the
site-packages subdirectory of the standard Python install tree."
.. I tend to think pythondir was intended to be, or at least could
reasonably be, an exception.
Looking at the patch again (content pasted below for the record):
https://github.com/macports/macports-ports/blob/master/devel/automake/files/patch-issue57329.diff
It seems too dangerous to me to apply as is, i.e., simply removing all
the $am_py_prefix values. That would change the results of
AM_PYTHON_PATH on every system. Maybe in a good way, but maybe not, and
I, at least, am not prepared to induce that level of incompatible
change. People could have their build systems set up to define the
prefix to be their desired directory.
What I think would be safer to apply is a patch that eliminates those
am_py_prefix values only on Macs. I'm not sure if am_py_prefix can be
defined (only on Macs) to something in the code above that sets it:
[if test "x$prefix" = xNONE
then
am_py_prefix=$ac_default_prefix
else
am_py_prefix=$prefix
fi
i.e.: if <on mac> then am_py_prefix=$something ...
or if it's necessary to make all those sysconfig.* calls conditional on
the system.
Wdyt? Can you provide a modified patch in whatever way seems best?
--thanks, karl.
diff --git m4/python.m4 m4/python.m4
index 63c0a0e04..f81684378 100644
--- m4/python.m4
+++ m4/python.m4
@@ -147,10 +147,10 @@ except ImportError:
am_cv_python_pythondir=`$PYTHON -c "
$am_python_setup_sysconfig
if can_use_sysconfig:
- sitedir = sysconfig.get_path('purelib', vars={'base':'$am_py_prefix'})
+ sitedir = sysconfig.get_path('purelib')
else:
from distutils import sysconfig
- sitedir = sysconfig.get_python_lib(0, 0, prefix='$am_py_prefix')
+ sitedir = sysconfig.get_python_lib(0, 0)
sys.stdout.write(sitedir)"`
case $am_cv_python_pythondir in
$am_py_prefix*)
@@ -189,10 +189,10 @@ sys.stdout.write(sitedir)"`
am_cv_python_pyexecdir=`$PYTHON -c "
$am_python_setup_sysconfig
if can_use_sysconfig:
- sitedir = sysconfig.get_path('platlib', vars={'platbase':'$am_py_prefix'})
+ sitedir = sysconfig.get_path('platlib')
else:
from distutils import sysconfig
- sitedir = sysconfig.get_python_lib(1, 0, prefix='$am_py_prefix')
+ sitedir = sysconfig.get_python_lib(1, 0)
sys.stdout.write(sitedir)"`
case $am_cv_python_pyexecdir in
$am_py_exec_prefix*)
Information forwarded
to
bug-automake <at> gnu.org
:
bug#35322
; Package
automake
.
(Fri, 26 Feb 2021 19:20:02 GMT)
Full text and
rfc822 format available.
Message #19 received at 35322 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
How about this? It bases the default values of PYTHON_PREFIX and
PYTHON_EXEC_PREFIX on sys.prefix and sys.exec_prefix respectively, which
the documentation has long said might happen. Behaviour should not
change in the case that sys.prefix = $prefix. The variables can still be
overridden; e.g. PYTHON_PREFIX is defined with reference to $prefix if
the former is a subdir of (or the same as) the latter, and pythondir is
defined with reference to $PYTHON_PREFIX.
There's also a fix for an apparent bug where $am_py_prefix was used
instead of $am_py_exec_prefix when setting up the value of pyexecdir.
- Josh
[use-sys.prefix-for-PYTHON_PREFIX.patch (text/plain, attachment)]
Information forwarded
to
bug-automake <at> gnu.org
:
bug#35322
; Package
automake
.
(Mon, 01 Mar 2021 23:15:01 GMT)
Full text and
rfc822 format available.
Message #22 received at 35322 <at> debbugs.gnu.org (full text, mbox):
Hi Joshua,
How about this? It bases the default values of PYTHON_PREFIX and
PYTHON_EXEC_PREFIX on sys.prefix and sys.exec_prefix respectively, ...
Thanks much for the patch (including fixing the tests). I don't see any
immediate comments. Jim and I will ponder for a couple days. --best, karl.
Information forwarded
to
bug-automake <at> gnu.org
:
bug#35322
; Package
automake
.
(Tue, 02 Mar 2021 06:26:02 GMT)
Full text and
rfc822 format available.
Message #25 received at 35322 <at> debbugs.gnu.org (full text, mbox):
On 2021-3-2 10:14 , Karl Berry wrote:
> Thanks much for the patch (including fixing the tests). I don't see any
> immediate comments. Jim and I will ponder for a couple days. --best, karl.
Thanks Karl. I'd certainly appreciate input from other packagers as well.
I wonder if it might also make sense to offer a configure option to set
PYTHON_PREFIX?
- Josh
Information forwarded
to
bug-automake <at> gnu.org
:
bug#35322
; Package
automake
.
(Tue, 02 Mar 2021 23:55:02 GMT)
Full text and
rfc822 format available.
Message #28 received at 35322 <at> debbugs.gnu.org (full text, mbox):
I wonder if it might also make sense to offer a configure option to set
PYTHON_PREFIX?
I think that would be great (--pythondir, --pythonexecdir or some
such). Would you mind looking into that? I've never played around with
the options Automake adds and have unfortunately limited time to
research the code. But I will eventually if needed.
I'd certainly appreciate input from other packagers as well.
Indeed yes. --thanks, karl.
Information forwarded
to
bug-automake <at> gnu.org
:
bug#35322
; Package
automake
.
(Fri, 05 Mar 2021 23:27:02 GMT)
Full text and
rfc822 format available.
Message #31 received at 35322 <at> debbugs.gnu.org (full text, mbox):
Hi Joshua - did you run the full make check on your Mac?
I saw you patched some of the tests, which is great, so I'm guessing
you did, but just asking ...
On Jim's Mac system, many more tests (35) fail, but haven't tracked that
down yet. --thanks, karl.
Information forwarded
to
bug-automake <at> gnu.org
:
bug#35322
; Package
automake
.
(Fri, 05 Mar 2021 23:38:01 GMT)
Full text and
rfc822 format available.
Message #34 received at 35322 <at> debbugs.gnu.org (full text, mbox):
On 2021-3-6 10:26 , Karl Berry wrote:
> Hi Joshua - did you run the full make check on your Mac?
> I saw you patched some of the tests, which is great, so I'm guessing
> you did, but just asking ...
>
> On Jim's Mac system, many more tests (35) fail, but haven't tracked that
> down yet. --thanks, karl.
>
I did run a full make check. There were indeed quite a few tests that
failed; I only fixed the python related ones.
- Josh
Information forwarded
to
bug-automake <at> gnu.org
:
bug#35322
; Package
automake
.
(Sat, 13 Mar 2021 11:22:01 GMT)
Full text and
rfc822 format available.
Message #37 received at 35322 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On 2021-3-3 10:54 , Karl Berry wrote:
> I wonder if it might also make sense to offer a configure option to set
> PYTHON_PREFIX?
>
> I think that would be great (--pythondir, --pythonexecdir or some
> such). Would you mind looking into that? I've never played around with
> the options Automake adds and have unfortunately limited time to
> research the code. But I will eventually if needed.
OK, here's a patch to add --with-python_prefix and
--with-python_exec_prefix options. It applies on top of my previous patch.
Note that the paths given to these options aren't edited to use the
${prefix} and ${exec_prefix} variables when they are subdirs thereof,
since I figure if you set a specific value with these options you really
mean it. Note also that if --with-python_prefix is given but
--with-python_exec_prefix is not, PYTHON_EXEC_PREFIX takes the value
that was given to --with-python_prefix. I think this is more convenient
on the whole, since setting a custom PYTHON_PREFIX but wanting to keep
the automatically determined value of PYTHON_EXEC_PREFIX seems like it
would be a pretty rare scenario.
- Josh
[add-with-python_prefix-option.patch (text/plain, attachment)]
Information forwarded
to
bug-automake <at> gnu.org
:
bug#35322
; Package
automake
.
(Sat, 13 Mar 2021 23:43:02 GMT)
Full text and
rfc822 format available.
Message #40 received at 35322 <at> debbugs.gnu.org (full text, mbox):
OK, here's a patch to add --with-python_prefix and
--with-python_exec_prefix options. It applies on top of my previous patch.
Thanks! I think the option names would be normal with all hyphens
(e.g., the std option is --exec-prefix, not --exec_prefix), but
I'll just tweak that when I install, obviously trivial.
setting a custom PYTHON_PREFIX but wanting to keep the automatically
determined value of PYTHON_EXEC_PREFIX seems like it would be a
pretty rare scenario.
Fully agreed. Thanks again. -k
Reply sent
to
Karl Berry <karl <at> freefriends.org>
:
You have taken responsibility.
(Tue, 18 May 2021 22:16:02 GMT)
Full text and
rfc822 format available.
Notification sent
to
Chih-Hsuan Yen <yan12125 <at> gmail.com>
:
bug acknowledged by developer.
(Tue, 18 May 2021 22:16:02 GMT)
Full text and
rfc822 format available.
Message #45 received at 35322-done <at> debbugs.gnu.org (full text, mbox):
Hi Joshua and all - regarding https://bugs.gnu.org/35322, I have
(finally) pushed your patches. I tweaked the doc update a little bit and
created a new test to try out the --with-python_prefix and
--with-python_exec_prefix options. (I left the underscores in the name
since I saw there were other such cases already. Confusing either way,
so fine.)
Thanks much for doing all the work. --karl
Information forwarded
to
bug-automake <at> gnu.org
:
bug#35322
; Package
automake
.
(Tue, 18 May 2021 22:17:02 GMT)
Full text and
rfc822 format available.
Information forwarded
to
bug-automake <at> gnu.org
:
bug#35322
; Package
automake
.
(Wed, 19 May 2021 03:09:01 GMT)
Full text and
rfc822 format available.
Message #51 received at 35322 <at> debbugs.gnu.org (full text, mbox):
On 2021-5-19 08:15 , Karl Berry wrote:
> Hi Joshua and all - regarding https://bugs.gnu.org/35322, I have
> (finally) pushed your patches. I tweaked the doc update a little bit and
> created a new test to try out the --with-python_prefix and
> --with-python_exec_prefix options. (I left the underscores in the name
> since I saw there were other such cases already. Confusing either way,
> so fine.)
>
> Thanks much for doing all the work. --karl
Thanks Karl, much appreciated.
- Josh
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org
.
(Wed, 16 Jun 2021 11:24:05 GMT)
Full text and
rfc822 format available.
This bug report was last modified 2 years and 286 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.