apache-arrow: segmentation fault, potentially related to jemalloc

[essandess (Steve Smith)]

When compiled with ​https://github.com/apache/arrow, the py*-pyarrow packages create a segfault on import at a jemalloc call. Both arm64 and x86_64 architectures are affected.

The issue disappears using the cmake directive -DARROW_JEMALLOC:BOOL=OFF.

lldb trace

/opt/local/bin/lldb-mp-16
(lldb) command script import pyarrow
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: /opt/local/libexec/llvm-16/bin/lldb
1.	HandleCommand(command = "command script import pyarrow")
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  libLLVM.dylib               0x0000000113a91c6c llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 56
1  libLLVM.dylib               0x0000000113a90c38 llvm::sys::RunSignalHandlers() + 112
2  libLLVM.dylib               0x0000000113a922f8 SignalHandler(int) + 344
3  libsystem_platform.dylib    0x000000018d386a24 _sigtramp + 56
4  libjemalloc.2.dylib         0x000000010f5940e0 je_free_default + 1104
5  libprotobuf.3.21.12.0.dylib 0x000000010f2471f0 google::protobuf::internal::ArenaStringPtr::Destroy() + 56
6  libprotobuf.3.21.12.0.dylib 0x000000010f2d0000 google::protobuf::FileDescriptorProto::SharedDtor() + 144
7  libprotobuf.3.21.12.0.dylib 0x000000010f2d00e4 google::protobuf::FileDescriptorProto::~FileDescriptorProto() + 48
8  libprotobuf.3.21.12.0.dylib 0x000000010f2ea634 google::protobuf::EncodedDescriptorDatabase::Add(void const*, int) + 168
9  libprotobuf.3.21.12.0.dylib 0x000000010f29e878 google::protobuf::DescriptorPool::InternalAddGeneratedFile(void const*, int) + 40
10 libprotobuf.3.21.12.0.dylib 0x000000010f3056a8 google::protobuf::(anonymous namespace)::AddDescriptors(google::protobuf::internal::DescriptorTable const*) + 136
11 libprotobuf.3.21.12.0.dylib 0x000000010f3056dc google::protobuf::internal::AddDescriptorsRunner::AddDescriptorsRunner(google::protobuf::internal::DescriptorTable const*) + 24
12 dyld                        0x000000018d01c1d8 invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const::$_0::operator()() const + 168
13 dyld                        0x000000018d05de94 invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 340
14 dyld                        0x000000018d0511a4 invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 528
15 dyld                        0x000000018cffc2d8 dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 296
16 dyld                        0x000000018d0501cc dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 192
17 dyld                        0x000000018d05d958 dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 516
18 dyld                        0x000000018d01885c dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 448
19 dyld                        0x000000018d018c10 dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 220
20 dyld                        0x000000018d018bec dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
21 dyld                        0x000000018d018bec dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
22 dyld                        0x000000018d018bec dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
23 dyld                        0x000000018d018bec dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
24 dyld                        0x000000018d018bec dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 184
25 dyld                        0x000000018d01c264 dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const::$_1::operator()() const + 112
26 dyld                        0x000000018d018d90 dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const + 304
27 dyld                        0x000000018d036d58 dyld4::APIs::dlopen_from(char const*, int, void*) + 1440
28 Python                      0x000000010378ae64 _imp_create_dynamic + 616
29 Python                      0x00000001036ce5d0 cfunction_vectorcall_FASTCALL + 80
30 Python                      0x0000000103760d94 _PyEval_EvalFrameDefault + 50396
31 Python                      0x0000000103763424 _PyEval_Vector + 116
32 Python                      0x000000010368859c object_vacall + 224
33 Python                      0x0000000103688458 PyObject_CallMethodObjArgs + 92
34 Python                      0x00000001037873a8 PyImport_ImportModuleLevelObject + 1224
35 Python                      0x000000010375b8f0 _PyEval_EvalFrameDefault + 28728
36 Python                      0x0000000103753d74 PyEval_EvalCode + 168
37 Python                      0x0000000103750174 builtin_exec + 332
38 Python                      0x00000001036ce6cc cfunction_vectorcall_FASTCALL_KEYWORDS + 76
39 Python                      0x0000000103760d94 _PyEval_EvalFrameDefault + 50396
40 Python                      0x0000000103763424 _PyEval_Vector + 116
41 Python                      0x000000010368859c object_vacall + 224
42 Python                      0x0000000103688458 PyObject_CallMethodObjArgs + 92
43 Python                      0x00000001037873a8 PyImport_ImportModuleLevelObject + 1224
44 Python                      0x000000010375b8f0 _PyEval_EvalFrameDefault + 28728
45 Python                      0x0000000103753d74 PyEval_EvalCode + 168
46 Python                      0x00000001037a5d38 run_eval_code_obj + 84
47 Python                      0x00000001037a5c9c run_mod + 112
48 Python                      0x00000001037a8084 PyRun_StringFlags + 112
49 liblldb.16.0.6.dylib        0x000000010469effc lldb_private::python::runStringMultiLine(llvm::Twine const&, lldb_private::python::PythonDictionary const&, lldb_private::python::PythonDictionary const&) + 120
50 liblldb.16.0.6.dylib        0x00000001046a5838 lldb_private::ScriptInterpreterPythonImpl::ExecuteMultipleLines(char const*, lldb_private::ExecuteScriptOptions const&) + 960
51 liblldb.16.0.6.dylib        0x00000001046ab314 lldb_private::ScriptInterpreterPythonImpl::LoadScriptingModule(char const*, lldb_private::LoadScriptOptions const&, lldb_private::Status&, std::__1::shared_ptr<lldb_private::StructuredData::Object>*, lldb_private::FileSpec) + 2092
52 liblldb.16.0.6.dylib        0x0000000104777158 CommandObjectCommandsScriptImport::DoExecute(lldb_private::Args&, lldb_private::CommandReturnObject&) + 328
53 liblldb.16.0.6.dylib        0x00000001043469c0 lldb_private::CommandObjectParsed::Execute(char const*, lldb_private::CommandReturnObject&) + 484
54 liblldb.16.0.6.dylib        0x000000010433d2e8 lldb_private::CommandInterpreter::HandleCommand(char const*, lldb_private::LazyBool, lldb_private::CommandReturnObject&) + 2136
55 liblldb.16.0.6.dylib        0x00000001043409a0 lldb_private::CommandInterpreter::IOHandlerInputComplete(lldb_private::IOHandler&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&) + 852
56 liblldb.16.0.6.dylib        0x00000001042731a0 lldb_private::IOHandlerEditline::Run() + 304
57 liblldb.16.0.6.dylib        0x0000000104256968 lldb_private::Debugger::RunIOHandlers() + 140
58 liblldb.16.0.6.dylib        0x0000000104341c14 lldb_private::CommandInterpreter::RunCommandInterpreter(lldb_private::CommandInterpreterRunOptions&) + 156
59 liblldb.16.0.6.dylib        0x00000001040a0614 lldb::SBDebugger::RunCommandInterpreter(bool, bool) + 124
60 lldb                        0x0000000102ff48e0 Driver::MainLoop() + 2784
61 lldb                        0x0000000102ff541c main + 2168
62 dyld                        0x000000018cffff28 start + 2236
LLDB diagnostics will be written to /var/folders/2s/r9z6gsps7gn5wd89c0sqs8k80000gq/T/diagnostics-5f4976
Please include the directory content when filing a bug report
Segmentation fault: 11

Upstream issue link:

• ​https://github.com/jemalloc/jemalloc/issues/2511

Downstream issue links

• ​https://github.com/apache/arrow/issues/37010
• ​https://trac.macports.org/ticket/67900
• ​https://trac.macports.org/ticket/67896

[jmroot (Joshua Root)]

It's not at all clear from the backtrace that jemalloc is at fault here. It could just as easily be protobuf freeing an invalid pointer.

[essandess (Steve Smith)]

Perhaps, but the issue disappears without jemalloc.

[jmroot (Joshua Root)]

Maybe it disappears, or maybe it just isn't detected. Could go either way; I'm just cautioning against drawing conclusions without sufficient evidence.

[mascguy (Christopher Nielsen)]

Steve, now that jemalloc is no longer in play for apache-arrow (as of the following PR), should this be closed?

​https://github.com/macports/macports-ports/pull/19664

Or would you prefer to keep it open, until/unless upstream discussion helps to clear this up...?

[essandess (Steve Smith)]

This is still an issue, as upstream apache-arrow says that jemalloc is preferred: ​https://github.com/apache/arrow/issues/37010#issuecomment-1668601173

The latest merged MacPorts PR that removes jemalloc is just a stop-gap so that all the downstream stuff like py-pandas that will attempt to limport pyarrow if it’s installed won’t crash.