Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#63443 closed defect (fixed)

ffmpeg-4.4_3+gpl2: build failures: libffi dylib version bump breaks core ports like LLVM

Reported by: Tommaso93 Owned by: mascguy (Christopher Nielsen)
Priority: Normal Milestone:
Component: ports Version: 2.7.1
Keywords: Cc: dbevans (David B. Evans), hapaguy (Brian Kurt Fujikawa), splaisan (Stephane Plaisance), JDLH (Jim DeLaHunt), ATL-Flaneur (Andreas Yankopolus), dsavransky (Dmitry Savransky), amagela (Anthony M. Agelastos), jeremyhu (Jeremy Huddleston Sequoia)
Port: ffmpeg libffi

Description

Dear experts,

I am having some issues trying to upgrade ffmpeg to the last version. In attachment you can find the main log from the build.

Best, Tommaso

Attachments (2)

main.log (4.3 MB) - added by Tommaso93 3 years ago.
main.2.log (4.1 MB) - added by sambthompson (Sam Thompson) 3 years ago.
xcode 8.2.1 on 10.11.6 main.log

Change History (32)

Changed 3 years ago by Tommaso93

Attachment: main.log added

comment:1 Changed 3 years ago by sambthompson (Sam Thompson)

Cc: sambthompson added

comment:2 Changed 3 years ago by sambthompson (Sam Thompson)

Seeing the same errors on 10.11.6 building variant +gpl2+nonfree; message of interest is:

Undefined symbols for architecture x86_64

while linking symbols with _ff_ prefixes.

Changed 3 years ago by sambthompson (Sam Thompson)

Attachment: main.2.log added

xcode 8.2.1 on 10.11.6 main.log

comment:3 Changed 3 years ago by ATL-Flaneur (Andreas Yankopolus)

I'm getting the same slew of Undefined symbols for architecture x86_64 error messages on my system (MacOS 11.5.2 Intel, Xcode 12.5.1).

comment:4 Changed 3 years ago by hapaguy (Brian Kurt Fujikawa)

Cc: hapaguy added

comment:5 Changed 3 years ago by JDLH (Jim DeLaHunt)

Build failed for me. ffmpeg @4.4_4+gpl2+x11 on macOS 10.14 Mojave.

It is interesting that the log says arch i386. Doesn't that mean 32-bit, and isn't 32-bit unsupported on macOS 10.14 Mojave?

Excerpt of the interesting parts of the main.log:

ffmpeg @4.4_4+gpl2+x11
:debug:sysinfo macOS 10.14.6 (darwin/18.7.0) arch i386
:debug:sysinfo MacPorts 2.7.1
:debug:sysinfo Xcode 11.2.1
:debug:sysinfo SDK 10.14
:debug:sysinfo MACOSX_DEPLOYMENT_TARGET: 10.14
… [elided] …
:info:build Undefined symbols for architecture x86_64:
:info:build   "_ff_butterflies_fixed_sse2", referenced from:
:info:build       _ff_fixed_dsp_init_x86 in fixed_dsp_init.o
:info:build   "_ff_butterflies_float_sse", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_cpu_cpuid", referenced from:
:info:build       _ff_get_cpu_flags_x86 in cpu.o
:info:build   "_ff_cpu_xgetbv", referenced from:
:info:build       _ff_get_cpu_flags_x86 in cpu.o
:info:build   "_ff_evaluate_lls_sse2", referenced from:
:info:build       _ff_init_lls_x86 in lls_init.o
:info:build   "_ff_image_copy_plane_uc_from_sse4", referenced from:
:info:build       _ff_image_copy_plane_uc_from_x86 in imgutils_init.o
:info:build   "_ff_pixelutils_sad_16x16_mmxext", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_16x16_sse2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_32x32_avx2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_32x32_sse2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_8x8_mmx", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_8x8_mmxext", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_a_16x16_sse2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_a_32x32_avx2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_a_32x32_sse2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_u_16x16_sse2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_u_32x32_avx2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_u_32x32_sse2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_scalarproduct_float_sse", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_update_lls_avx", referenced from:
:info:build       _ff_init_lls_x86 in lls_init.o
:info:build   "_ff_update_lls_fma3", referenced from:
:info:build       _ff_init_lls_x86 in lls_init.o
:info:build   "_ff_update_lls_sse2", referenced from:
:info:build       _ff_init_lls_x86 in lls_init.o
:info:build   "_ff_vector_dmac_scalar_avx", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_dmac_scalar_fma3", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_dmac_scalar_sse2", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_dmul_avx", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_dmul_scalar_avx", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_dmul_scalar_sse2", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_dmul_sse2", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmac_scalar_avx", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmac_scalar_fma3", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmac_scalar_sse", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_add_avx", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_add_fma3", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_add_sse", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_avx", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_reverse_avx", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_reverse_avx2", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_reverse_sse", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_scalar_sse", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_sse", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_window_3dnowext", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_window_sse", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build ld: symbol(s) not found for architecture x86_64
:info:build clang: error: linker command failed with exit code 1 (use -v to see invocation)
:info:build gmake: *** [ffbuild/library.mak:103: libavutil/libavutil.56.dylib] Error 1
… [elided] …

comment:6 in reply to:  5 ; Changed 3 years ago by sambthompson (Sam Thompson)

Replying to JDLH:

Build failed for me. ffmpeg @4.4_4+gpl2+x11 on macOS 10.14 Mojave.

It is interesting that the log says arch i386. Doesn't that mean 32-bit, and isn't 32-bit unsupported on macOS 10.14 Mojave?

It could be a mismatch in arch, anyway; I'm on El Capitan which still supports 32-bit.

Same message re: i386 in my build log.

comment:7 Changed 3 years ago by reneeotten (Renee Otten)

Cc: splaisan added

has duplicate 63473

Last edited 3 years ago by reneeotten (Renee Otten) (previous) (diff)

comment:8 Changed 3 years ago by JDLH (Jim DeLaHunt)

Cc: JDLH added

comment:9 Changed 3 years ago by JDLH (Jim DeLaHunt)

Possible duplicate: #63482

comment:10 Changed 3 years ago by ATL-Flaneur (Andreas Yankopolus)

Cc: ATL-Flaneur added

comment:11 in reply to:  6 Changed 3 years ago by sambthompson (Sam Thompson)

Replying to sambthompson:

Replying to JDLH:

Build failed for me. ffmpeg @4.4_4+gpl2+x11 on macOS 10.14 Mojave.

It is interesting that the log says arch i386. Doesn't that mean 32-bit, and isn't 32-bit unsupported on macOS 10.14 Mojave?

It could be a mismatch in arch, anyway; I'm on El Capitan which still supports 32-bit.

Same message re: i386 in my build log.

Hmm. 4.4_4 just built OK for me during a port upgrade outdated.

comment:12 Changed 3 years ago by sambthompson (Sam Thompson)

Cc: sambthompson removed

comment:13 Changed 3 years ago by dsavransky (Dmitry Savransky)

Cc: dsavransky added

comment:14 Changed 3 years ago by msbit (Tom Sullivan)

I had the same issue while running port upgrade outdated, and while I am not claiming it's the same for every other report, the cause for mine may be relevant.

When troubleshooting the issue, I had tried to run nm on some of the output .o files, but that had failed with:

dyld: Library not loaded: /opt/local/lib/libffi.7.dylib
  Referenced from: /opt/local/libexec/llvm-10/lib/libLLVM.dylib
  Reason: image not found

which seems to be due to a big migration from libffi.7.dylib to libffi.8.dylib that happened while I'd been neglecting this machine? Anyhow, I didn't think much of it, reinstalling with libffi, then later on performing a rebuild of ffmpeg directly (which worked fine). Poring over the output files under /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_ffmpeg/ffmpeg/work I discovered one big difference in the config.logfiles, namely:

-check_inline_asm inline_asm_direct_symbol_refs "movl _test, %eax"
-    1  void foo(void){ __asm__ volatile("movl _test, %eax"); }
-void foo(void){ __asm__ volatile("movl _test, %eax"); }
+check_inline_asm inline_asm_direct_symbol_refs "movl test, %eax"
+    1  void foo(void){ __asm__ volatile("movl test, %eax"); }
+void foo(void){ __asm__ volatile("movl test, %eax"); }
-check_inline_asm inline_asm_direct_symbol_refs "movl _test(%rip), %eax"
+check_inline_asm inline_asm_direct_symbol_refs "movl test(%rip), %eax"
-    1  void foo(void){ __asm__ volatile("movl _test(%rip), %eax"); }
+    1  void foo(void){ __asm__ volatile("movl test(%rip), %eax"); }

The big giveaway is the presence of the _ prefix for the symbol on the separate build (post libffi installation).

In the configure script, nm is used to determine what the external prefix should be for name mangling. If nm fails like it did for me, the prefix is empty, but if it succeeds the prefix is _. This is also used to toggle defining PREFIX when invoking nasm, which ensures that the mangled symbols have the same _ prefix.

So, finally, looking over just _ff_butterflies_fixed_sse2, I've got:

nm ./libavutil/x86/fixed_dsp{,_init}.o | grep ff_butterflies_fixed_sse2$
0000000000000000 T ff_butterflies_fixed_sse2
                 U _ff_butterflies_fixed_sse2

in the MacPorts ffmpeg work directory and:

nm ./libavutil/x86/fixed_dsp{,_init}.o | grep ff_butterflies_fixed_sse2$
0000000000000000 T _ff_butterflies_fixed_sse2
                 U _ff_butterflies_fixed_sse2

in the standalone ffmpeg build.

I'm currently re-running port upgrade outdated and the built object file has the correct symbol, with mangling:

nm ./libavutil/x86/fixed_dsp{,_init}.o | grep ff_butterflies_fixed_sse2$
0000000000000000 T _ff_butterflies_fixed_sse2
                 U _ff_butterflies_fixed_sse2
Last edited 3 years ago by msbit (Tom Sullivan) (previous) (diff)

comment:15 Changed 3 years ago by ATL-Flaneur (Andreas Yankopolus)

Same set of build errors after updating to Xcode 13.

comment:16 Changed 3 years ago by amagela (Anthony M. Agelastos)

Cc: amagela added

comment:17 Changed 3 years ago by mascguy (Christopher Nielsen)

Cc: mascguy added

comment:18 Changed 3 years ago by mascguy (Christopher Nielsen)

Owner: set to jeremyhu
Status: newassigned

comment:19 Changed 3 years ago by mascguy (Christopher Nielsen)

Cc: jeremyhu removed

comment:20 Changed 3 years ago by mascguy (Christopher Nielsen)

Summary: ffmpeg-4.4_3+gpl2: Build Failedffmpeg-4.4_3+gpl2: build failures: possibly related to libffi

Updated summary, based on the investigative work in comment:14

comment:21 Changed 3 years ago by amagela (Anthony M. Agelastos)

FWIW, I did the following and was able to upgrade ffmpeg. This may help point to libffi and its recent update as the item to focus upon.

$ sudo port deactivate libffi @3.4.2_0
$ sudo port activate libffi @3.3_1
$ sudo port upgrade ffmpeg
--->  Computing dependencies for ffmpeg
--->  Fetching archive for ffmpeg
--->  Attempting to fetch ffmpeg-4.4_4+gpl2.darwin_19.x86_64.tbz2 from https://packages.macports.org/ffmpeg
--->  Attempting to fetch ffmpeg-4.4_4+gpl2.darwin_19.x86_64.tbz2 from https://nue.de.packages.macports.org/ffmpeg
--->  Attempting to fetch ffmpeg-4.4_4+gpl2.darwin_19.x86_64.tbz2 from http://atl.us.packages.macports.org/ffmpeg
--->  Fetching distfiles for ffmpeg
--->  Attempting to fetch ffmpeg-4.4.tar.xz from https://ffmpeg.org/releases/
--->  Verifying checksums for ffmpeg
--->  Extracting ffmpeg
--->  Applying patches to ffmpeg
--->  Configuring ffmpeg
--->  Building ffmpeg
--->  Staging ffmpeg into destroot
--->  Installing ffmpeg @4.4_4+gpl2
--->  Cleaning ffmpeg
--->  Computing dependencies for ffmpeg
--->  Deactivating ffmpeg @4.4_2+gpl2
--->  Cleaning ffmpeg
--->  Activating ffmpeg @4.4_4+gpl2
--->  Cleaning ffmpeg

comment:22 in reply to:  14 ; Changed 3 years ago by mascguy (Christopher Nielsen)

Summary: ffmpeg-4.4_3+gpl2: build failures: possibly related to libffiffmpeg-4.4_3+gpl2: build failures: libffi dylib version bump breaks core ports like LLVM

Replying to msbit:

When troubleshooting the issue, I had tried to run nm on some of the output .o files, but that had failed with:

dyld: Library not loaded: /opt/local/lib/libffi.7.dylib
  Referenced from: /opt/local/libexec/llvm-10/lib/libLLVM.dylib
  Reason: image not found

which seems to be due to a big migration from libffi.7.dylib to libffi.8.dylib that happened while I'd been neglecting this machine?

Just experienced a similar situation, with some macOS VMs that were at least a month out-of-date [in terms of MacPorts].

What's clear is that a libffi upgrade temporarily breaks foundational ports like LLVM, when the dylib major version changes. And that seems like a big problem.

To avoid this situation, I'd like to update libffi to create a dylib symlink, with the previous major version. So when it's updated to libffi.8.dylib, for example, symlink libffi.7.dylib to it. That will prevent breakage of LLVM, along with these related headaches. (The port would determine the previous relative major version, from the current built version. So nothing would be hard-coded.)

Thoughts?

Longer-term, perhaps LLVM components should utilize a separate libffi. I'm thinking in terms of how we avoid circular dependencies among toolchain components, by utilizing separate "bootstrap" ports. (So perhaps utilize a libffi-bootstrap for all LLVM/toolchain components.) Or alternatively, statically link with libffi... assuming that doesn't cause issues?

Last edited 3 years ago by mascguy (Christopher Nielsen) (previous) (diff)

comment:23 Changed 3 years ago by mascguy (Christopher Nielsen)

Cc: jeremyhu added; mascguy removed
Owner: changed from jeremyhu to mascguy
Port: libffi added

comment:24 Changed 3 years ago by mascguy (Christopher Nielsen)

Related ticket: issue:63378

comment:25 Changed 3 years ago by Christopher Nielsen <mascguy@…>

Resolution: fixed
Status: assignedclosed

In 6fb2ae6631b642ed4fcf13d0531348e51d29f36c/macports-ports (master):

libffi: symlink via prev major version

  • Absolutely critical, to avoid temporary toolchain breakage

Closes: #63443
Closes: #63453
See: #63378

comment:26 in reply to:  22 Changed 3 years ago by jmroot (Joshua Root)

Replying to mascguy:

What's clear is that a libffi upgrade temporarily breaks foundational ports like LLVM, when the dylib major version changes. And that seems like a big problem.

All that's clear from this ticket is that ffmpeg is using cctools' nm without declaring a dependency. It should either ensure that the Xcode nm is used, or depend on cctools. In the latter case, llvm would then be upgraded to the revision linked with the new libffi, before ffmpeg is built.

comment:27 Changed 3 years ago by mascguy (Christopher Nielsen)

Great point, and perhaps that's enough to fix ffmpeg.

But are you certain that we don't have a fair number of other ports, which opportunistically utilize cctools components?

comment:28 in reply to:  27 Changed 3 years ago by jmroot (Joshua Root)

Replying to mascguy:

Great point, and perhaps that's enough to fix ffmpeg.

But are you certain that we don't have a fair number of other ports, which opportunistically utilize cctools components?

If you want to look at installing cctools differently to prevent accidental usage, that's fine.

comment:29 Changed 3 years ago by Christopher Nielsen <mascguy@…>

In a4617245e09a39d4c873cce84a5bc98307830cc7/macports-ports (master):

ffmpeg: add build dep for cctools
See: #63443

comment:30 Changed 3 years ago by mascguy (Christopher Nielsen)

In retrospect, perhaps we all need to utilize trace mode more often, particularly when tackling major upgrades from upstream projects. (I realize that trace mode occasionally causes issues, and won't work 100% of the time. But still not a bad thing to try.)

More recently, I've been handling this by closely scrutinizing output from configure (or the equivalent, for the build system used by a given project), and/or running configure --help. It's more time-consuming to be sure, but pays off longer-term.

Last edited 3 years ago by mascguy (Christopher Nielsen) (previous) (diff)
Note: See TracTickets for help on using tickets.