New Ticket     Tickets     Wiki     Browse Source     Timeline     Roadmap     Ticket Reports     Search

Ticket #24421 (new defect)

Opened 4 years ago

Last modified 3 years ago

python dies using multiprocessing/urllib2 from mac ports, but works with stock python

Reported by: vkuznet@… Owned by: mcalhoun@…
Priority: Normal Milestone:
Component: ports Version: 1.8.2
Keywords: Cc: craig.hawco@…, scott.moynes@…
Port: python26

Description

Hi, I observed very tricky problem. I have application written in python which uses multiprocessing module. So using the python 2.6 from macports cause python die in a thread when urllib2 module is invoked. To test I installed stock python 2.6.5 from python.org and everything works. I also verify that the problem lies with urllib2/multiprocessing modules, since if I replace urllib2 to httplib, everything works too. I attach apple traceback which shows the problem.

My guess that macports uses either special patch or enable flags which didn't come with stock python source and that cause a problem.

Valentin.

Attachments

das.bug (36.6 KB) - added by vkuznet@… 4 years ago.

Change History

Changed 4 years ago by vkuznet@…

comment:1 Changed 4 years ago by jmr@…

  • Cc mcalhoun@… added
  • Keywords python crash, urllib2, multiprocessing removed
  • Port set to python26
  • Owner changed from macports-tickets@… to blb@…

Please remember to fill in the Port field and cc the maintainers.

comment:2 Changed 4 years ago by jmr@…

Looks like _scproxy.so is passing a null pointer to SystemConfiguration. I don't see anything in the patches that looks like it would cause this.

comment:3 Changed 4 years ago by jmr@…

Except SCDynamicStoreCopyProxies is meant to work with a NULL argument, which raises the possibility of this being a bug in SystemConfiguration or CF.

comment:4 Changed 4 years ago by blb@…

Do you have a simple example to reproduce this?

comment:5 Changed 4 years ago by vkuznet@…

I tried to make stand-alone version, but so far I can't reproduce the problem with it. Is it possible to get configuration/patches which went into mac-port. If I can re-apply all configuration steps into the one I download from python.org I can have a look if I can reproduce the problem with manual build of python.

comment:6 Changed 4 years ago by vkuznet@…

I don't know if it helps, I found that _scproxy.so is linked differently for python from macports and the stock one.

Here is the stock python _scproxy.so

otool -L /Users/vk/Work/Python/Python-2.6.5//install/lib/python2.6/lib-dynload/_scproxy.so
/Users/vk/Work/Python/Python-2.6.5//install/lib/python2.6/lib-dynload/_scproxy.so:
	/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 293.4.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 550.19.0)
	/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 246.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.0.1)

while the macports version is

otool -L $PWD/lib-dynload/_scproxy.so 
/opt/local/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/lib-dynload/_scproxy.so:
	/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 293.4.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 550.19.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.0.1)

Actually all _XXX.so from macport build are missing /usr/lib/libgcc_s.1.dylib link. Also I commented out the

if sys.platform == 'darwin':

...

in urllib.py in macport version and my code works just fine when there is no _scproxy.so objects are loaded, while it crash if _scproxy.so is there.

comment:7 Changed 4 years ago by blb@…

The way MacPorts builds python26 (and the patches applied) can be see in the trac browser for the python26 port (patches are in the files/ subdir).

Note that not linking to libgcc_s.1.dylib should be fine since, at least on 10.6, it is just a symlink to libSystem anyway:

$ ls -l /usr/lib/libgcc_s.1.dylib 
lrwxr-xr-x  1 root  wheel  17 Apr 16 18:54 /usr/lib/libgcc_s.1.dylib -> libSystem.B.dylib

comment:8 Changed 4 years ago by blb@…

Looking at your crash log, I see

...
       0x1007f2000 -        0x1007f6ff7 +_ssl.so ??? (???) <BC4937DC-E257-4B12-88BA-58678A48F931> /Users/vk/CMS/COMP/DAS/mylib/_ssl.so
...
       0x1012c9000 -        0x101308fff +libssl.0.9.8.dylib 0.9.8 (compatibility 0.9.8) <98F9160D-1D42-9189-CE14-204FDB2B62E7>  /Users/vk/Work/openssl-0.9.8k/lib/libssl.0.9.8.dylib
       0x10131c000 -        0x10143bfe7 +libcrypto.0.9.8.dylib 0.9.8 (compatibility 0.9.8) <4BB4F468-D80A-B3A1-AE43-DB544191E975> /Users/vk/Work/openssl-0.9.8k/lib/libcrypto.0.9.8.dylib

Not sure if that would be causing the crash since it's in SSL and not where the crash is happening but it might be an interesting experiment to see if using the MacPorts-provided SSL module crashes as well.

comment:9 Changed 4 years ago by vkuznet@…

The openssl module was compiled intentionally, since in our environment we still use renegotiation feature on front-end, which is disabled by default since 0.9.8m release of OpenSSL. In addition, both python (stock one and from mac ports) used my local module, so it should not be a problem.

comment:10 Changed 4 years ago by scott.moynes@…

I am also experiencing this bug. Downgrading to python26 version 2.6.4 has corrected it for me.

I will work on a simple reproduction recipe.

comment:11 Changed 4 years ago by craig.hawco@…

  • Cc craig.hawco@… added

Cc Me!

comment:12 Changed 4 years ago by scott.moynes@…

  • Cc scott.moynes@… added

Cc Me!

comment:13 Changed 4 years ago by craig.hawco@…

As a workaround setting any environment variable ending with "_proxy" to a non-empty value prevents urllib from calling SystemConfiguration, and thus prevents this particular crash. That does not resolve this central issue.

comment:14 Changed 4 years ago by blb@…

  • Owner changed from blb@… to mcalhoun@…
  • Cc mcalhoun@… removed

comment:15 Changed 3 years ago by jmr@…

Does 2.6.6 still have this problem? In any case, it's pretty hard to do anything about it without a test case.

Note: See TracTickets for help on using tickets.