Opened 5 years ago

Closed 4 years ago

Last modified 4 years ago

#58882 closed defect (worksforme)

Python37 configure script segfaults in generate-posix-vars on macOS 10.15

Reported by: vdoublie Owned by: jmroot (Joshua Root)
Priority: Normal Milestone:
Component: ports Version:
Keywords: catalina Cc: colons (colons), trainman75
Port: python37

Description

During the build of Python37 under macOS10.15, generate-posix-vars fails. This does not happens under 10.14

just "forcing" the MACOSX_DEPLOYMENT_TARGET variable to 10.14 during the build of python37 instead of 10.15 "fixes" the problem.

See main_macports.log for errors

See main_patched.log

Attachments (4)

main_macport.log (164.9 KB) - added by vdoublie 5 years ago.
main_patched.log (6.3 MB) - added by vdoublie 5 years ago.
forcing MACOSX_DEPLOYMENT_TARGET to 10.14 so generate-posix-vars knows what to do
main.log (170.8 KB) - added by trainman75 4 years ago.
Build failure on Catalina/Xcode11
python.exe_2019-10-12-002028_MacBook-Pro.crash (10.2 KB) - added by trainman75 4 years ago.

Change History (51)

Changed 5 years ago by vdoublie

Attachment: main_macport.log added

comment:1 Changed 5 years ago by mf2k (Frank Schima)

Keywords: catalina added; upcoming macOS release build rules removed
Owner: set to jmroot
Status: newassigned

In the future, please add the port maintainer(s) to Cc (port info --maintainers python37), if any.

comment:2 Changed 5 years ago by vdoublie

sorry

comment:3 Changed 5 years ago by Schamschula (Marius Schamschula)

Are you sure you uploaded the main_patched.log. I see the same MACOSX_DEPLOYMENT_TARGET and the same error.

comment:4 Changed 5 years ago by vdoublie

oops forgot about the "autoclean" done at the end of the successful install here it is

Changed 5 years ago by vdoublie

Attachment: main_patched.log added

forcing MACOSX_DEPLOYMENT_TARGET to 10.14 so generate-posix-vars knows what to do

comment:5 Changed 5 years ago by mike-savory

Note this affect all python builds, not just 3.7.

comment:6 Changed 5 years ago by jmroot (Joshua Root)

Please be aware of wiki:FAQ#prerelease

comment:7 Changed 5 years ago by jmroot (Joshua Root)

Summary: Python37 configure script requires an update of MACOSX_DEPLOYMENT_TARGET rules for upcoming macOS 10.15Python37 configure script segfaults in generate-posix-vars on macOS 10.15

Upstream python developers who are disclosed on Catalina would probably like to see the crash log.

comment:8 Changed 5 years ago by josh-blake (Josh Blake)

Python will build if you use the Xcode 10.3 toolchain - either select 10.3 (10G8) from within Xcode, or point appropriately using

xcode-select -s /Applications/Xcode.app/Contents/Developer

from the command line. You would also be wise to run a

sudo port select clang none
sudo port select llvm none

just to force macports to use Apples toolchain.

comment:9 Changed 4 years ago by pmetzger (Perry E. Metzger)

jmroot, do we have any progress on this? Catalina is out, and python 3.7 is a critical package...

comment:10 Changed 4 years ago by jmroot (Joshua Root)

I have no idea if it's still an issue with the final release of Catalina and current Xcode.

comment:11 Changed 4 years ago by colons (colons)

Running the App Store version of Xcode 11.1, python37 runs into this. So does python27.

:info:build DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python37/python37/work/Python-3.7.4 ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
:info:build     if test $? -ne 0 ; then \
:info:build             echo "generate-posix-vars failed" ; \
:info:build             rm -f ./pybuilddir.txt ; \
:info:build             exit 1 ; \
:info:build     fi
:info:build /bin/sh: line 1: 25214 Segmentation fault: 11  DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python37/python37/work/Python-3.7.4 ./python.exe -E -S -m sysconfig --generate-posix-vars
:info:build DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27
/work/Python-2.7.16 ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
:info:build     if test $? -ne 0 ; then \
:info:build             echo "generate-posix-vars failed" ; \
:info:build             rm -f ./pybuilddir.txt ; \
:info:build             exit 1 ; \
:info:build     fi
:info:build /bin/sh: line 1: 31630 Segmentation fault: 11  DYLD_FRAMEWORK_PATH=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_python27/python27/work/Python-2.7.16 ./python.exe -E -S -m sysconfig --generate-posix-vars
:info:build generate-posix-vars failed

I've tried selecting Xcode 10.3, cleaning, and rebuilding, but that doesn't seem to change anything.

comment:12 Changed 4 years ago by jmroot (Joshua Root)

Did anyone ever report it upstream?

comment:13 Changed 4 years ago by colons (colons)

With Xcode 10.3 and 11.1, I'm able to build, install, and run from the 3.7.4 tarball successfully, with no modifications. The below step goes by with no segfault:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: libpython3.7m.a(pymath.o) has no symbols
gcc     -Wl,-stack_size,1000000  -framework CoreFoundation -o python.exe Programs/python.o libpython3.7m.a -ldl  -framework CoreFoundation     
./python.exe -E -S -m sysconfig --generate-posix-vars ;\
	if test $? -ne 0 ; then \
		echo "generate-posix-vars failed" ; \
		rm -f ./pybuilddir.txt ; \
		exit 1 ; \
	fi
gcc -c  -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration   -I. -I./Include    -DPy_BUILD_CORE -o Modules/_math.o Modules/_math.c

I can't find anyone reporting this particular issue in Python's issue tracker. 38295 isn't the same issue and I can't see anything that looks like this in their issue tracker.

Last edited 4 years ago by colons (colons) (previous) (diff)

comment:14 Changed 4 years ago by colons (colons)

Cc: colons added

comment:15 in reply to:  12 Changed 4 years ago by yan12125 (Chih-Hsuan Yen)

Cc: yan12125 added

Replying to jmroot:

Did anyone ever report it upstream?

Here is one related comment: https://bugs.python.org/issue38014#msg351312. It says --enable-framework is related to the crash.

comment:16 Changed 4 years ago by jmroot (Joshua Root)

OK, so can someone with the final version of Catalina please show a build log with the failure, and the crash log?

comment:17 Changed 4 years ago by trainman75

Cc: trainman75 added

comment:18 Changed 4 years ago by trainman75

Build log added. If you need anything else, let me know (and preferably how to get it in the form most useful to you!)

comment:19 Changed 4 years ago by jmroot (Joshua Root)

I still need the crash log please. Should be in either ~/Library/Logs/DiagnosticReports or /Library/Logs/DiagnosticReports.

comment:20 Changed 4 years ago by mf2k (Frank Schima)

I cannot reproduce this on the release versions of Catalina and Xcode 11.1.

$ port -v installed python37
The following ports are currently installed:
  python37 @3.7.4_1 (active) platform='darwin 19' archs='x86_64' date='2019-10-11T03:29:47-0600'

comment:21 Changed 4 years ago by jmroot (Joshua Root)

Interesting. Have those seeing the crash started fresh with 10.15 final, or could there be bits left over from the beta?

comment:22 Changed 4 years ago by vdoublie

I have been able to build python37 on Catalina Xcode 11.1 without crash.

comment:23 Changed 4 years ago by vdoublie

without the "hack"

Changed 4 years ago by trainman75

Attachment: main.log added

Build failure on Catalina/Xcode11

Changed 4 years ago by trainman75

comment:24 Changed 4 years ago by kencu (Ken)

my guess is that this is another manifestation of

6e78e5c9495b4dc4e7e050fae2b41dd5b9accfdd/macports-ports

Last edited 4 years ago by ryandesign (Ryan Carsten Schmidt) (previous) (diff)

comment:25 Changed 4 years ago by 19h

I'm on 19A582a and definitely still seeing this issue.

comment:26 Changed 4 years ago by jmroot (Joshua Root)

I can't repro either.

comment:27 Changed 4 years ago by jmroot (Joshua Root)

@trainman75 the only significant difference between your log and mine that I can see up to the point where yours fails is:

-:info:configure checking for python3.7... python3.7
+:info:configure checking for python3.7... no
+:info:configure checking for python3... python3

How is configure finding python3.7 on your system? Also you have coreutils installed; clutching at straws but I don't suppose deactivating that port makes any difference?

comment:28 Changed 4 years ago by jmroot (Joshua Root)

Well, having coreutils installed made no difference here. So I guess we're back to the admittedly vague "beta pollution" hypothesis, unless someone can repro on a clean never-beta 10.15 installation.

comment:29 Changed 4 years ago by vdoublie

I sort of did: I kept my "old port installed under Mojave + Xcode 10 stuff" and it is still working under Catalina... as long as I don't touch it. SO I moved out of the way the "old Mojave" stuff (Macports directory, and /opt/local the rebuild macports using GitHub (so I am up to date port wise" then just port install python37 - so it has to rebuild everything from scratch and it builds no sweat... but it is only very recent. So it suspect that this is a port that does not like to "see beta or pre-release" or anything that is not "user/public release" as I submitted forcing MACOSX_DEPLOYMENT_TARGETto 10.14 (so mascaraing it) in the makefile.pre did the trick... for just that part of the code... if the target is 10.15 up until this special part of the make then reset just after ... it build also as should be... so It really a PB of the generate_posix_var that cannot find a rule somewhere for "unknown forward versions of the platform"

comment:30 Changed 4 years ago by trainman75

I just tried adding the lines mentioned in the commit from comment:24, and success. Python37 builds where it didn't before.

With nothing more than an educated guess, I placed them just before "build.target all"

I had been running Catalina beta and Xcode11 beta, but the build issue was present in that configuration. Updated both to public release in the last few days, still no luck. Reinstalled MacPorts from binary, still no go. After that stage, I had uninstalled all ports, but realise now I'd forgotten to do the various directory removals from the uninstall guide, so I may still have had a dirty setup.

Hope this helps track down a fix.

Last edited 4 years ago by trainman75 (previous) (diff)

comment:31 Changed 4 years ago by kencu (Ken)

It could well be processor dependent as well, so not everyone would see it.

Version 0, edited 4 years ago by kencu (Ken) (next)

comment:32 Changed 4 years ago by 19h

I can verify that -fno-stack-check successfully works around the segmentation fault, for both python37 and python27. Device: MacBookPro13,3 using build 19A582a (Catalina GM).

comment:33 Changed 4 years ago by colons (colons)

That fixed it for me, too.

comment:34 in reply to:  31 Changed 4 years ago by jmroot (Joshua Root)

Replying to kencu:

It could be processor dependent as well, so perhaps not everyone would see it. Or some other random variation to be discovered.

It's very easy to disprove my hypothesis if false: someone who sees the issue just needs to make a fresh Catalina installation on the same hardware and repro.

comment:35 Changed 4 years ago by jmroot (Joshua Root)

Python upstream can't repro either. https://bugs.python.org/issue38014#msg354598

They've asked that if the issue is still present, someone who can reproduce it should open a new issue on their bug tracker and show them how to do it.

comment:36 Changed 4 years ago by dlamija (Muhammed Ramiza)

Bulid sucessfully on full release Catalina & XCode 11. No crash. For those whose failed maybe there is some leftover from beta version

comment:37 Changed 4 years ago by pcallycat

Still unable to build, Catalina, Xcode 11. Clean install of MacPorts and Xcode production releases on general release of MacOS, with a new user just to cover any missed directories. Tomorrow I'll do a full clean install of MacOS. MacBook Pro late 2013 model.

edit: rather than reinstall my whole machine, I just added -fno-stack-check as mentioned prior to build. Not sure why this is necessary.

Last edited 4 years ago by pcallycat (previous) (diff)

comment:38 Changed 4 years ago by yan12125 (Chih-Hsuan Yen)

Cc: yan12125 removed

comment:39 Changed 4 years ago by mouse07410 (Mouse)

Pardon my ignorance, but (a) where did you add -fno-stack-check, and (b) does build with Xcode-11.3 still exhibit this problem?

comment:40 in reply to:  39 Changed 4 years ago by jmroot (Joshua Root)

Replying to mouse07410:

(b) does build with Xcode-11.3 still exhibit this problem?

Anyone able to answer this? If there's no issue with 11.3 then I think we can just close the ticket.

comment:41 Changed 4 years ago by jmroot (Joshua Root)

Resolution: worksforme
Status: assignedclosed

comment:42 Changed 4 years ago by mouse07410 (Mouse)

I've installed (via Macports) python37 and python38 on Catalina 10.15.3 with Xcode-11.3.1. I'd say this problem has been remedied. Would appreciate others confirm though...

Last edited 4 years ago by mouse07410 (Mouse) (previous) (diff)

comment:43 Changed 4 years ago by fredowski

I have exactly the same problem but with python38 on MacOS 10.13.6 with MACOSX_DEPLOYMENT_TARGET set to 10.7. It works with target set to 10.8

:info:build DYLD_FRAMEWORK_PATH=/opt/macports/install/var/macports/build/_opt_macports_macp
orts-ports_lang_python38/python38/work/Python-3.8.5 ./python.exe -E -S -m sysconfig --gener
ate-posix-vars ;\
:info:build     if test $? -ne 0 ; then \
:info:build             echo "generate-posix-vars failed" ; \
:info:build             rm -f ./pybuilddir.txt ; \
:info:build             exit 1 ; \
:info:build     fi
:info:build /bin/sh: line 1: 32838 Killed: 9               DYLD_FRAMEWORK_PATH=/opt/macport
s/install/var/macports/build/_opt_macports_macports-ports_lang_python38/python38/work/Pytho
n-3.8.5 ./python.exe -E -S -m sysconfig --generate-posix-vars
:info:build generate-posix-vars failed
:info:build make: *** [pybuilddir.txt] Error 1

comment:44 Changed 4 years ago by fredowski

I checked some other MACOSX_DEPLOYMENT_TARGETS and this is the result on MacOS 10.13 with Xcode 10.1:

  • 10.5 works
  • 10.6 failure
  • 10.7 failure
  • 10.8 works

So you can build python38 with deployment target 10.5 but not 10.6 and not 10.7. Strange.

comment:45 Changed 4 years ago by kencu (Ken)

I wonder if this might be an xcode clang thing. Perhaps building it with one of MacPorts' clang compilers would work.

python38 does build on 10.6 and 10.7, of course, when building directly on those systems.

comment:46 Changed 4 years ago by fredowski

I tried with gcc10 but that fails also. Maybe it is related to the sdk.

comment:47 Changed 4 years ago by kencu (Ken)

always still with the same mysterious Catalina error?

Error Code:      0x00000014 (no mapping for user instruction write)
Note: See TracTickets for help on using tickets.