Opened 3 months ago

Last modified 6 weeks ago

#58882 assigned defect

Python37 configure script segfaults in generate-posix-vars on macOS 10.15

Reported by: vdoublie Owned by: jmroot (Joshua Root)
Priority: Normal Milestone:
Component: ports Version:
Keywords: catalina Cc: colons (colons), trainman75
Port: python37

Description

During the build of Python37 under macOS10.15, generate-posix-vars fails. This does not happens under 10.14

just "forcing" the MACOSX_DEPLOYMENT_TARGET variable to 10.14 during the build of python37 instead of 10.15 "fixes" the problem.

See main_macports.log for errors

See main_patched.log

Attachments (4)

main_macport.log (164.9 KB) - added by vdoublie 3 months ago.
main_patched.log (6.3 MB) - added by vdoublie 3 months ago.
forcing MACOSX_DEPLOYMENT_TARGET to 10.14 so generate-posix-vars knows what to do
main.log (170.8 KB) - added by trainman75 2 months ago.
Build failure on Catalina/Xcode11
python.exe_2019-10-12-002028_MacBook-Pro.crash (10.2 KB) - added by trainman75 2 months ago.

Change History (42)

Changed 3 months ago by vdoublie

Attachment: main_macport.log added

comment:1 Changed 3 months ago by mf2k (Frank Schima)

Keywords: catalina added; upcoming macOS release build rules removed
Owner: set to jmroot
Status: newassigned

In the future, please add the port maintainer(s) to Cc (port info --maintainers python37), if any.

comment:2 Changed 3 months ago by vdoublie

sorry

comment:3 Changed 3 months ago by Schamschula (Marius Schamschula)

Are you sure you uploaded the main_patched.log. I see the same MACOSX_DEPLOYMENT_TARGET and the same error.

comment:4 Changed 3 months ago by vdoublie

oops forgot about the "autoclean" done at the end of the successful install here it is

Changed 3 months ago by vdoublie

Attachment: main_patched.log added

forcing MACOSX_DEPLOYMENT_TARGET to 10.14 so generate-posix-vars knows what to do

comment:5 Changed 3 months ago by mike-savory

Note this affect all python builds, not just 3.7.

comment:6 Changed 3 months ago by jmroot (Joshua Root)

Please be aware of wiki:FAQ#prerelease

comment:7 Changed 3 months ago by jmroot (Joshua Root)

Summary: Python37 configure script requires an update of MACOSX_DEPLOYMENT_TARGET rules for upcoming macOS 10.15Python37 configure script segfaults in generate-posix-vars on macOS 10.15

Upstream python developers who are disclosed on Catalina would probably like to see the crash log.

comment:8 Changed 3 months ago by josh-blake (Josh Blake)

Python will build if you use the Xcode 10.3 toolchain - either select 10.3 (10G8) from within Xcode, or point appropriately using

xcode-select -s /Applications/Xcode.app/Contents/Developer

from the command line. You would also be wise to run a

sudo port select clang none
sudo port select llvm none

just to force macports to use Apples toolchain.

comment:9 Changed 2 months ago by pmetzger (Perry E. Metzger)

jmroot, do we have any progress on this? Catalina is out, and python 3.7 is a critical package...

comment:10 Changed 2 months ago by jmroot (Joshua Root)

I have no idea if it's still an issue with the final release of Catalina and current Xcode.

comment:11 Changed 2 months ago by colons (colons)

Running the App Store version of Xcode 11.1, python37 runs into this. So does python27.

:info:build DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python37/python37/work/Python-3.7.4 ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
:info:build     if test $? -ne 0 ; then \
:info:build             echo "generate-posix-vars failed" ; \
:info:build             rm -f ./pybuilddir.txt ; \
:info:build             exit 1 ; \
:info:build     fi
:info:build /bin/sh: line 1: 25214 Segmentation fault: 11  DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python37/python37/work/Python-3.7.4 ./python.exe -E -S -m sysconfig --generate-posix-vars
:info:build DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27
/work/Python-2.7.16 ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
:info:build     if test $? -ne 0 ; then \
:info:build             echo "generate-posix-vars failed" ; \
:info:build             rm -f ./pybuilddir.txt ; \
:info:build             exit 1 ; \
:info:build     fi
:info:build /bin/sh: line 1: 31630 Segmentation fault: 11  DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.16 ./python.exe -E -S -m sysconfig --generate-posix-vars
:info:build generate-posix-vars failed

I've tried selecting Xcode 10.3, cleaning, and rebuilding, but that doesn't seem to change anything.

comment:12 Changed 2 months ago by jmroot (Joshua Root)

Did anyone ever report it upstream?

comment:13 Changed 2 months ago by colons (colons)

With Xcode 10.3 and 11.1, I'm able to build, install, and run from the 3.7.4 tarball successfully, with no modifications. The below step goes by with no segfault:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: libpython3.7m.a(pymath.o) has no symbols
gcc     -Wl,-stack_size,1000000  -framework CoreFoundation -o python.exe Programs/python.o libpython3.7m.a -ldl  -framework CoreFoundation     
./python.exe -E -S -m sysconfig --generate-posix-vars ;\
	if test $? -ne 0 ; then \
		echo "generate-posix-vars failed" ; \
		rm -f ./pybuilddir.txt ; \
		exit 1 ; \
	fi
gcc -c  -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration   -I. -I./Include    -DPy_BUILD_CORE -o Modules/_math.o Modules/_math.c

I can't find anyone reporting this particular issue in Python's issue tracker. 38295 isn't the same issue and I can't see anything that looks like this in their issue tracker.

Last edited 2 months ago by colons (colons) (previous) (diff)

comment:14 Changed 2 months ago by colons (colons)

Cc: colons added

comment:15 in reply to:  12 Changed 2 months ago by yan12125 (Chih-Hsuan Yen)

Cc: yan12125 added

Replying to jmroot:

Did anyone ever report it upstream?

Here is one related comment: https://bugs.python.org/issue38014#msg351312. It says --enable-framework is related to the crash.

comment:16 Changed 2 months ago by jmroot (Joshua Root)

OK, so can someone with the final version of Catalina please show a build log with the failure, and the crash log?

comment:17 Changed 2 months ago by trainman75

Cc: trainman75 added

comment:18 Changed 2 months ago by trainman75

Build log added. If you need anything else, let me know (and preferably how to get it in the form most useful to you!)

comment:19 Changed 2 months ago by jmroot (Joshua Root)

I still need the crash log please. Should be in either ~/Library/Logs/DiagnosticReports or /Library/Logs/DiagnosticReports.

comment:20 Changed 2 months ago by mf2k (Frank Schima)

I cannot reproduce this on the release versions of Catalina and Xcode 11.1.

$ port -v installed python37
The following ports are currently installed:
  python37 @3.7.4_1 (active) platform='darwin 19' archs='x86_64' date='2019-10-11T03:29:47-0600'

comment:21 Changed 2 months ago by jmroot (Joshua Root)

Interesting. Have those seeing the crash started fresh with 10.15 final, or could there be bits left over from the beta?

comment:22 Changed 2 months ago by vdoublie

I have been able to build python37 on Catalina Xcode 11.1 without crash.

comment:23 Changed 2 months ago by vdoublie

without the "hack"

Changed 2 months ago by trainman75

Attachment: main.log added

Build failure on Catalina/Xcode11

Changed 2 months ago by trainman75

comment:24 Changed 2 months ago by kencu (Ken)

my guess is that this is another manifestation of

6e78e5c9495b4dc4e7e050fae2b41dd5b9accfdd/macports-ports

Last edited 7 weeks ago by ryandesign (Ryan Schmidt) (previous) (diff)

comment:25 Changed 2 months ago by 19h

I'm on 19A582a and definitely still seeing this issue.

comment:26 Changed 2 months ago by jmroot (Joshua Root)

I can't repro either.

comment:27 Changed 2 months ago by jmroot (Joshua Root)

@trainman75 the only significant difference between your log and mine that I can see up to the point where yours fails is:

-:info:configure checking for python3.7... python3.7
+:info:configure checking for python3.7... no
+:info:configure checking for python3... python3

How is configure finding python3.7 on your system? Also you have coreutils installed; clutching at straws but I don't suppose deactivating that port makes any difference?

comment:28 Changed 2 months ago by jmroot (Joshua Root)

Well, having coreutils installed made no difference here. So I guess we're back to the admittedly vague "beta pollution" hypothesis, unless someone can repro on a clean never-beta 10.15 installation.

comment:29 Changed 2 months ago by vdoublie

I sort of did: I kept my "old port installed under Mojave + Xcode 10 stuff" and it is still working under Catalina... as long as I don't touch it. SO I moved out of the way the "old Mojave" stuff (Macports directory, and /opt/local the rebuild macports using GitHub (so I am up to date port wise" then just port install python37 - so it has to rebuild everything from scratch and it builds no sweat... but it is only very recent. So it suspect that this is a port that does not like to "see beta or pre-release" or anything that is not "user/public release" as I submitted forcing MACOSX_DEPLOYMENT_TARGETto 10.14 (so mascaraing it) in the makefile.pre did the trick... for just that part of the code... if the target is 10.15 up until this special part of the make then reset just after ... it build also as should be... so It really a PB of the generate_posix_var that cannot find a rule somewhere for "unknown forward versions of the platform"

comment:30 Changed 2 months ago by trainman75

I just tried adding the lines mentioned in the commit from comment:24, and success. Python37 builds where it didn't before.

With nothing more than an educated guess, I placed them just before "build.target all"

I had been running Catalina beta and Xcode11 beta, but the build issue was present in that configuration. Updated both to public release in the last few days, still no luck. Reinstalled MacPorts from binary, still no go. After that stage, I had uninstalled all ports, but realise now I'd forgotten to do the various directory removals from the uninstall guide, so I may still have had a dirty setup.

Hope this helps track down a fix.

Last edited 2 months ago by trainman75 (previous) (diff)

comment:31 Changed 2 months ago by kencu (Ken)

It could be processor dependent as well, so perhaps not everyone would see it. Or some other random variation to be discovered.

Last edited 2 months ago by kencu (Ken) (previous) (diff)

comment:32 Changed 2 months ago by 19h

I can verify that -fno-stack-check successfully works around the segmentation fault, for both python37 and python27. Device: MacBookPro13,3 using build 19A582a (Catalina GM).

comment:33 Changed 2 months ago by colons (colons)

That fixed it for me, too.

comment:34 in reply to:  31 Changed 2 months ago by jmroot (Joshua Root)

Replying to kencu:

It could be processor dependent as well, so perhaps not everyone would see it. Or some other random variation to be discovered.

It's very easy to disprove my hypothesis if false: someone who sees the issue just needs to make a fresh Catalina installation on the same hardware and repro.

comment:35 Changed 8 weeks ago by jmroot (Joshua Root)

Python upstream can't repro either. https://bugs.python.org/issue38014#msg354598

They've asked that if the issue is still present, someone who can reproduce it should open a new issue on their bug tracker and show them how to do it.

comment:36 Changed 8 weeks ago by dlamija (Muhammed Ramiza)

Bulid sucessfully on full release Catalina & XCode 11. No crash. For those whose failed maybe there is some leftover from beta version

comment:37 Changed 7 weeks ago by pcallycat

Still unable to build, Catalina, Xcode 11. Clean install of MacPorts and Xcode production releases on general release of MacOS, with a new user just to cover any missed directories. Tomorrow I'll do a full clean install of MacOS. MacBook Pro late 2013 model.

edit: rather than reinstall my whole machine, I just added -fno-stack-check as mentioned prior to build. Not sure why this is necessary.

Last edited 6 weeks ago by pcallycat (previous) (diff)

comment:38 Changed 6 weeks ago by yan12125 (Chih-Hsuan Yen)

Cc: yan12125 removed
Note: See TracTickets for help on using tickets.