BUG: python26 bus error

[jabronson@…]

When running my  Pylons web app with macports python 2.6, requesting the front page in a browser triggers a bus error. This does not happen when running under Mac OS X 10.5's system Python (2.5.1) or the  Mac Python 2.6.2 distribution from python.org.

I attached gdb to the macports python process and triggered the error. A transcript (including backtrace) is attached.

[santagada@…]

I've got a bus error with python from /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python when running trac. The error report can be seen here:  http://paste.pocoo.org/show/133699/ and there is another user with the same problem here:  http://lists.macosforge.org/pipermail/macports-users/2009-June/015563.html

I found that if I use /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6 instead it works. The binaries are different but I don't know why or how would it affect the execution, but maybe it is a library link problem. Try to run it with python2.6 and see if it fix this.

[santagada@…]

Cc Me!

[blb@…]

Replying to santagada@…:

I found that if I use /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6 instead it works. The binaries are different but I don't know why or how would it affect the execution, but maybe it is a library link problem. Try to run it with python2.6 and see if it fix this.

Odd, tracd crashes for me whether I use the python in bin/ or in Resources/Python.app/Contents/MacOS.

[santagada@…]

Forget what I said before, I tried again and it breaks also. I don't know why it worked twice with it.

[santagada@…]

Replying to blb@…:

Replying to santagada@…:

I found that if I use /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6 instead it works. The binaries are different but I don't know why or how would it affect the execution, but maybe it is a library link problem. Try to run it with python2.6 and see if it fix this.

Odd, tracd crashes for me whether I use the python in bin/ or in Resources/Python.app/Contents/MacOS.

So you can reproduce it, great :). Any idea what is going on here? I tried to run gdb on the core dump, but apparently it can't find the debug symbols for most stuff so I can't really do much. Apparently the bus errors occur on lookdict_string which is a function of the dict object in python. Is there any way to get the object files and/or debug symbols? GDB seems to be looking at the macports temp build diretory for them like this:

warning: Could not find object file "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_lang_python26/work/Python-2.6.2/build/temp.macosx-10.5-i386-2.6/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_lang_python26/work/Python-2.6.2/Modules/_sqlite/prepare_protocol.o" - no debug information available for "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_lang_python26/work/Python-2.6.2/Modules/_sqlite/prepare_protocol.c".

[blb@…]

Yes, I definitely was able to reproduce it. The odd thing, from jabronson@'s gdb trace, is it looks almost like some sort of recursion issue; the backtrace is over 2000 calls deep, with lots of calls between PyEval_EvalFrameEx() and PyEval_EvalCodeEx(). Since I'm not familiar with the internals of python's source, I'm not sure why it is happening though as far as I can tell, MacPorts doesn't seem to be messing with any code in that part of the source.

[santagada@…]

PyEval_EvalFrameEx is a frame and there is *only* 736 of those in the backtrace (recursionlimit is 1k) so it is not that I think.

I tried to get a python level trace of what was going on, by in the case of trac to edit wsgy.py to insert a trace function (so each thread would have its own trace) and the results are that the failure is in the file mimetypes.py in the function guess_type.

I think this is a race condition because if I print too much on my traces the problem vanishes, the same thing is I turn off threads in trac standalone.py.

Here is the code for the trace:

--- wsgi.py 2009-08-21 09:25:38.000000000 -0300 +++ wsgi.py 2009-08-21 09:25:14.000000000 -0300 @@ -178,7 +178,12 @@

return environ

def handle_one_request(self):

+ def trac_func(frame, type, blabla): + if type=='call': + print frame.f_code.co_filename[-20:], frame.f_code.co_name

environ = self.setup_environ()

+ if '.' in self.path: + import sys; sys.settrace(trac_func)

if environ:

gateway = self.server.gateway(self, environ) gateway.run(self.server.application)

[santagada@…]

The diff got destroied by trac... sorry

Can we move this bug to confirmed or something?

--- wsgi.py     2009-08-21 09:25:38.000000000 -0300
+++ wsgi.py     2009-08-21 09:25:14.000000000 -0300
@@ -178,7 +178,12 @@
         return environ
 
     def handle_one_request(self):
+        def trac_func(frame, type, blabla):
+            if type=='call':
+                print frame.f_code.co_filename[-20:], frame.f_code.co_name
         environ = self.setup_environ()
+        if '.' in self.path:
+            import sys; sys.settrace(trac_func)
         if environ: 
             gateway = self.server.gateway(self, environ)
             gateway.run(self.server.application)

[santagada@…]

Asking around pythonmac-sig I was pointed at this:

"I'm a bit surprised. Having used the mimetypes module myself, I never bothered to call init() The fact that users call init() may have to do with the fact that until recently, not doing so explicitly could be thread-unsafe." from  http://codereview.appspot.com/107042

Now trac apparently doesn't call mimetypes.init ever so this could be the race condition that I thought was the problem. But even then python has the GIL that should be making this not crash python, and the reporter of this bug says it doesn't happen with vanilla python.org so I still think there is something wrong with the way python is compiled in macports. This particular bug probably bite a lot of other web frameworks so if this race condition crashed other python versions (and I know it doesn't crash macports python 2.5) we would have heard about it right?

Hoping that someone that knows cpython internals helps in the fixing of this, sincerely yours, -- Leonardo Santagada santagada at gmail.com

[santagada@…]

Minimal test case that show the bus error

[santagada@…]

This is a bug with both the python 2.6.2 source distribution and the current 2.6 maintenance branch. I filled a bug report on python issue tracker:  http://bugs.python.org/issue6763

Why doesn't this affects the python.org .dmg I don't know.

[jeremykarlson@…]

Cc Me!

[blb@…]

Is anyone able to reproduce this with python26 2.6.3_1? I can't here, though I don't know if it's from python 2.6.3 or upgrading to 10.6...

[jabronson@…]

Replying to blb@…:

Is anyone able to reproduce this with python26 2.6.3_1? I can't here, though I don't know if it's from python 2.6.3 or upgrading to 10.6...

I'm on 10.5.8 and just upgraded from python 2.6.3_0 to 2.6.3_1 and threadboom gave me "Your interpreter didn't blow up". Unfortunately I was an idiot and forgot to try running threadboom *before* port -u upgradeing python26. Anyone else on 10.5 have a minute to try a before and after?

[santagada@…]

The new python has a new mimetypes lib so threadboom.py will not fail at all on python >= 2.6.3. I also upgraded to 10.6 so I can't try it.

You could download the python 2.6.2 source and compile it, it does give the same error as macports python. I don't understand why the .dmg doesn't and that is one of the reasons why I think this bug is so interesting. Someone should build an universal python 2.6.2 and see if that is the difference.

[jabronson@…]

Replying to santagada@…:

You could download the python 2.6.2 source and compile it, it does give the same error as macports python.

indeed, threadboom gives me the bus error on Leopard with 2.6.2 compiled from source:

~/tmp/Python-2.6.2/build/universal $ bin/python2.6 ~/tmp/threadboom.py 
Run Run Run
Bus error

I don't understand why the .dmg doesn't and that is one of the reasons why I think this bug is so interesting. Someone should build an universal python 2.6.2 and see if that is the difference.

I passed --with-universal-archs=all to configure in order to try to compile a universal binary, but it must not have worked because at the end of make install I got "No rule to make target update4wayuniversal', needed by altinstall'. Stop.", and file python2.6 gives only "python2.6: Mach-O executable i386". Should I have passed enable-universalsdk or enable-framework as well?

I also upgraded to 10.6 so I can't try it.

What changed in 10.6 that would prevent you from seeing the error?

[santagada@…]

nothing actually, just tested and using python 2.6.2 from source gives the same problem, I'm really starting to think that a universal build would not show this problem.

[eijkhout@…]

Is there any progress on this bug? It looks very much like what I'm running in right now, installing py26-nltk. Note I did install this before, without problems, but upgrading gave an error, so I uninstalled & installed it.

 %% sudo port install py26-nltk
Password:
--->  Deactivating py26-pyobjc2 @2.0_2
--->  Computing dependencies for py26-pyobjc
--->  Activating py26-pyobjc2 @2.0_2
--->  Cleaning py26-pyobjc2
--->  Cleaning py26-pyobjc
--->  Deactivating py26-pyobjc2-cocoa @2.0_2
--->  Computing dependencies for py26-pyobjc-cocoa
--->  Activating py26-pyobjc2-cocoa @2.0_2
--->  Cleaning py26-pyobjc2-cocoa
--->  Cleaning py26-pyobjc-cocoa
--->  Computing dependencies for py26-nltk
--->  Building py26-nltk
Error: Target org.macports.build returned: shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py26-nltk/work/nltk-2.0b8" && /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6 setup.py --no-user-cfg build " returned error 138
Command output: sh: line 1: 27615 Bus error               /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6 setup.py --no-user-cfg build

Error: Status 1 encountered during processing.
Before reporting a bug, first run the command again with the -d flag to get complete output.

[blb@…]

Replying to eijkhout@…:

Is there any progress on this bug? It looks very much like what I'm running in right now, installing py26-nltk. Note I did install this before, without problems, but upgrading gave an error, so I uninstalled & installed it.

I think it requires someone with some good low-level knowledge of python to debug it since the crash is definitely deep within its source, and unless I'm missing it, not in an area any of the patches applied by the port.