py27-pandas crashes with blosc

[dershow]

I am trying to create an HDF5 file, and if I use blosc compression it segmentation faults every time when I try to close the file. Here is an example:

$ python
Python 2.7.10 (default, May 26 2015, 17:31:25) 
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> a=pd.DataFrame([1,2])
>>> store=pd.HDFStore("test.h5",complib='blosc')
>>> store['test']=a
>>> store.close()
Segmentation fault: 11

I listed this as a py27-pandas port problem, because that is where I am seeing it. But, I suppose it could be, pandas, python, hdf5, pytables, or blosc related. If I use no compression, or zlib, the store closes without an error.

Here is the thread that crashed:

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_platform.dylib      	0x00007fff978a9b50 _platform_strcmp + 80
1   libblosc.1.dylib              	0x000000011275dc78 blosc_compname_to_compcode + 21
2   libblosc.1.dylib              	0x000000011275e0ca blosc_compress_ctx + 75
3   utilsextension.so             	0x00000001124f0da8 blosc_filter + 504
4   libhdf5.10.dylib              	0x00000001126dc084 H5Z_pipeline + 956
5   libhdf5.10.dylib              	0x0000000112555413 H5D__chunk_flush_entry + 355
6   libhdf5.10.dylib              	0x00000001125536da H5D__chunk_flush + 113
7   libhdf5.10.dylib              	0x0000000112564aed H5D__flush_real + 192
8   libhdf5.10.dylib              	0x000000011256458f H5D_close + 208
9   libhdf5.10.dylib              	0x00000001125e0478 H5I_dec_ref + 132
10  libhdf5.10.dylib              	0x00000001125e0331 H5I_dec_app_ref + 35
11  libhdf5.10.dylib              	0x00000001125e05ed H5I_dec_app_ref_always_close + 35
12  libhdf5.10.dylib              	0x000000011254db28 H5Dclose + 110
13  hdf5extension.so              	0x00000001128838a9 __pyx_pw_6tables_13hdf5extension_4Leaf_9_g_close + 57
14  org.python.python             	0x000000010f58a3a8 PyEval_EvalFrameEx + 15944
15  org.python.python             	0x000000010f586102 PyEval_EvalCodeEx + 1666
16  org.python.python             	0x000000010f58da86 fast_function + 118
17  org.python.python             	0x000000010f589c4f PyEval_EvalFrameEx + 14063
18  org.python.python             	0x000000010f586102 PyEval_EvalCodeEx + 1666
19  org.python.python             	0x000000010f58da86 fast_function + 118
20  org.python.python             	0x000000010f589c4f PyEval_EvalFrameEx + 14063
21  org.python.python             	0x000000010f586102 PyEval_EvalCodeEx + 1666
22  org.python.python             	0x000000010f58da86 fast_function + 118
23  org.python.python             	0x000000010f589c4f PyEval_EvalFrameEx + 14063
24  org.python.python             	0x000000010f58db66 fast_function + 342
25  org.python.python             	0x000000010f589c4f PyEval_EvalFrameEx + 14063
26  org.python.python             	0x000000010f58db66 fast_function + 342
27  org.python.python             	0x000000010f589c4f PyEval_EvalFrameEx + 14063
28  org.python.python             	0x000000010f586102 PyEval_EvalCodeEx + 1666
29  org.python.python             	0x000000010f58da86 fast_function + 118
30  org.python.python             	0x000000010f589c4f PyEval_EvalFrameEx + 14063
31  org.python.python             	0x000000010f58db66 fast_function + 342
32  org.python.python             	0x000000010f589c4f PyEval_EvalFrameEx + 14063
33  org.python.python             	0x000000010f586102 PyEval_EvalCodeEx + 1666
34  org.python.python             	0x000000010f585a76 PyEval_EvalCode + 54
35  org.python.python             	0x000000010f5af3d6 PyRun_InteractiveOneFlags + 406
36  org.python.python             	0x000000010f5aee8e PyRun_InteractiveLoopFlags + 206
37  org.python.python             	0x000000010f5aed38 PyRun_AnyFileExFlags + 136
38  org.python.python             	0x000000010f5c515c Py_Main + 3084
39  libdyld.dylib                 	0x00007fff928b55c9 start + 1

[mamoll (Mark Moll)]

I can reproduce the problem with py34-pandas, but have no idea what the problem might be.

[mf2k (Frank Schima)]

The port is still "py-pandas" so that searches can find it. The description, as I have modified it, should specify the python version that failed. Although based on mmoll's comment, this does not seem to be specific to python 2.7.

[stromnov (Andrey Stromnov)]

1) "pyx_pw_6tables_13hdf5extension_4Leaf_9_g_close" - so it happens inside of py-tables

2) py-tables port depends on external blosc library (directly or via hdf5)

Note: py-tables also contains specific version (even specially downgraded) of blosc library

3) Since version 1.7.0 blosc library no longer supports HDF5 filter (excerpt from blosc 1.7.0 ChangeLog: The HDF5 filter has been removed from c-blosc and moved into its own repo at: ​https://github.com/Blosc/hdf5)

[dershow]

I am not sure if this is the same bug or a different one. But, there is a known bug in pytables 3.2 that has been fixed in 3.2.1. I have created a separate ticket for it: ​https://trac.macports.org/ticket/48566 Perhaps that upgrade will help with this, and even if not, it will help with another crash that keeps happening with pandas.

[stromnov (Andrey Stromnov)]

Example from pytables documentation (leads to crash in py-tables 3.2 and 3.2.1):

import numpy
from tables import *

fileh = open_file('test5.h5', mode='w')
atom = Float32Atom()
filters = Filters(complevel=1, complib='blosc', fletcher32=True)
arr = fileh.create_earray(fileh.root, 'earray', atom, (0,2),
                         "A growable array", filters=filters)

# Append several rows in only one call
arr.append(numpy.array([[1., 2.],
                        [2., 3.],
                        [3., 4.]], dtype=numpy.float32))

# Print information on that enlargeable array
print("Result Array:")
print(repr(arr))
fileh.close()

[stromnov (Andrey Stromnov)]

Explanation: ​https://github.com/PyTables/PyTables/issues/451

[mamoll (Mark Moll)]

Fixed in r139337.

[petrrr]

Cc Me!