ffmpeg-4.4_3+gpl2: build failures: libffi dylib version bump breaks core ports like LLVM

[Tommaso93]

Dear experts,

I am having some issues trying to upgrade ffmpeg to the last version. In attachment you can find the main log from the build.

Best, Tommaso

[sambthompson (Sam Thompson)]

Seeing the same errors on 10.11.6 building variant +gpl2+nonfree; message of interest is:

Undefined symbols for architecture x86_64

while linking symbols with _ff_ prefixes.

[sambthompson (Sam Thompson)]

xcode 8.2.1 on 10.11.6 main.log

[ATL-Flaneur (Andreas Yankopolus)]

I'm getting the same slew of Undefined symbols for architecture x86_64 error messages on my system (MacOS 11.5.2 Intel, Xcode 12.5.1).

[JDLH (Jim DeLaHunt)]

Build failed for me. ffmpeg @4.4_4+gpl2+x11 on macOS 10.14 Mojave.

It is interesting that the log says arch i386. Doesn't that mean 32-bit, and isn't 32-bit unsupported on macOS 10.14 Mojave?

Excerpt of the interesting parts of the main.log:

ffmpeg @4.4_4+gpl2+x11
:debug:sysinfo macOS 10.14.6 (darwin/18.7.0) arch i386
:debug:sysinfo MacPorts 2.7.1
:debug:sysinfo Xcode 11.2.1
:debug:sysinfo SDK 10.14
:debug:sysinfo MACOSX_DEPLOYMENT_TARGET: 10.14
… [elided] …
:info:build Undefined symbols for architecture x86_64:
:info:build   "_ff_butterflies_fixed_sse2", referenced from:
:info:build       _ff_fixed_dsp_init_x86 in fixed_dsp_init.o
:info:build   "_ff_butterflies_float_sse", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_cpu_cpuid", referenced from:
:info:build       _ff_get_cpu_flags_x86 in cpu.o
:info:build   "_ff_cpu_xgetbv", referenced from:
:info:build       _ff_get_cpu_flags_x86 in cpu.o
:info:build   "_ff_evaluate_lls_sse2", referenced from:
:info:build       _ff_init_lls_x86 in lls_init.o
:info:build   "_ff_image_copy_plane_uc_from_sse4", referenced from:
:info:build       _ff_image_copy_plane_uc_from_x86 in imgutils_init.o
:info:build   "_ff_pixelutils_sad_16x16_mmxext", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_16x16_sse2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_32x32_avx2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_32x32_sse2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_8x8_mmx", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_8x8_mmxext", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_a_16x16_sse2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_a_32x32_avx2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_a_32x32_sse2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_u_16x16_sse2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_u_32x32_avx2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_pixelutils_sad_u_32x32_sse2", referenced from:
:info:build       _ff_pixelutils_sad_init_x86 in pixelutils_init.o
:info:build   "_ff_scalarproduct_float_sse", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_update_lls_avx", referenced from:
:info:build       _ff_init_lls_x86 in lls_init.o
:info:build   "_ff_update_lls_fma3", referenced from:
:info:build       _ff_init_lls_x86 in lls_init.o
:info:build   "_ff_update_lls_sse2", referenced from:
:info:build       _ff_init_lls_x86 in lls_init.o
:info:build   "_ff_vector_dmac_scalar_avx", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_dmac_scalar_fma3", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_dmac_scalar_sse2", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_dmul_avx", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_dmul_scalar_avx", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_dmul_scalar_sse2", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_dmul_sse2", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmac_scalar_avx", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmac_scalar_fma3", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmac_scalar_sse", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_add_avx", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_add_fma3", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_add_sse", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_avx", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_reverse_avx", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_reverse_avx2", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_reverse_sse", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_scalar_sse", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_sse", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_window_3dnowext", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build   "_ff_vector_fmul_window_sse", referenced from:
:info:build       _ff_float_dsp_init_x86 in float_dsp_init.o
:info:build ld: symbol(s) not found for architecture x86_64
:info:build clang: error: linker command failed with exit code 1 (use -v to see invocation)
:info:build gmake: *** [ffbuild/library.mak:103: libavutil/libavutil.56.dylib] Error 1
… [elided] …

[sambthompson (Sam Thompson)]

Replying to JDLH:

Build failed for me. ffmpeg @4.4_4+gpl2+x11 on macOS 10.14 Mojave.

It is interesting that the log says arch i386. Doesn't that mean 32-bit, and isn't 32-bit unsupported on macOS 10.14 Mojave?

It could be a mismatch in arch, anyway; I'm on El Capitan which still supports 32-bit.

Same message re: i386 in my build log.

[reneeotten (Renee Otten)]

has duplicate 63473

[JDLH (Jim DeLaHunt)]

Possible duplicate: #63482

[sambthompson (Sam Thompson)]

Replying to sambthompson:

Replying to JDLH:

Build failed for me. ffmpeg @4.4_4+gpl2+x11 on macOS 10.14 Mojave.

It is interesting that the log says arch i386. Doesn't that mean 32-bit, and isn't 32-bit unsupported on macOS 10.14 Mojave?

It could be a mismatch in arch, anyway; I'm on El Capitan which still supports 32-bit.

Same message re: i386 in my build log.

Hmm. 4.4_4 just built OK for me during a port upgrade outdated.

[msbit (Tom Sullivan)]

I had the same issue while running port upgrade outdated, and while I am not claiming it's the same for every other report, the cause for mine may be relevant.

When troubleshooting the issue, I had tried to run nm on some of the output .o files, but that had filed with:

dyld: Library not loaded: /opt/local/lib/libffi.7.dylib
  Referenced from: /opt/local/libexec/llvm-10/lib/libLLVM.dylib
  Reason: image not found

which seems to be due to a big migration from libffi.7.dylib to libffi.8.dylib that happened while I'd been neglecting this machine? Anyhow, I didn't think much of it, reinstalling with libffi, then later on performing a rebuild of ffmpeg directly (which worked fine). Poring over the output files under /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_multimedia_ffmpeg/ffmpeg/work I discovered one big difference in the config.logfiles, namely:

-check_inline_asm inline_asm_direct_symbol_refs "movl _test, %eax"
-    1  void foo(void){ __asm__ volatile("movl _test, %eax"); }
-void foo(void){ __asm__ volatile("movl _test, %eax"); }
+check_inline_asm inline_asm_direct_symbol_refs "movl test, %eax"
+    1  void foo(void){ __asm__ volatile("movl test, %eax"); }
+void foo(void){ __asm__ volatile("movl test, %eax"); }
-check_inline_asm inline_asm_direct_symbol_refs "movl _test(%rip), %eax"
+check_inline_asm inline_asm_direct_symbol_refs "movl test(%rip), %eax"
-    1  void foo(void){ __asm__ volatile("movl _test(%rip), %eax"); }
+    1  void foo(void){ __asm__ volatile("movl test(%rip), %eax"); }

The big giveaway is the presence of the _ prefix for the symbol on the separate build (post libffi installation).

In the configure script, nm is used to determine what the external prefix should be for name mangling. If nm fails like it did for me, the prefix is empty, but if it succeeds the prefix is _. This is also used to toggle defining PREFIX when invoking nasm, which ensures that the mangled symbols have the same _ prefix.

So, finally, looking over just _ff_butterflies_fixed_sse2, I've got:

nm ./libavutil/x86/fixed_dsp{,_init}.o | grep ff_butterflies_fixed_sse2$
0000000000000000 T ff_butterflies_fixed_sse2
                 U _ff_butterflies_fixed_sse2

in the MacPorts ffmpeg work directory and:

nm ./libavutil/x86/fixed_dsp{,_init}.o | grep ff_butterflies_fixed_sse2$
0000000000000000 T _ff_butterflies_fixed_sse2
                 U _ff_butterflies_fixed_sse2

in the standalone ffmpeg build.

I'm currently re-running port upgrade outdated and the built object file has the correct symbol, with mangling:

nm ./libavutil/x86/fixed_dsp{,_init}.o | grep ff_butterflies_fixed_sse2$
0000000000000000 T _ff_butterflies_fixed_sse2
                 U _ff_butterflies_fixed_sse2

[ATL-Flaneur (Andreas Yankopolus)]

Same set of build errors after updating to Xcode 13.

[mascguy (Christopher Nielsen)]

Updated summary, based on the investigative work in comment:14

[amagela (Anthony M. Agelastos)]

FWIW, I did the following and was able to upgrade ffmpeg. This may help point to libffi and its recent update as the item to focus upon.

$ sudo port deactivate libffi @3.4.2_0
$ sudo port activate libffi @3.3_1
$ sudo port upgrade ffmpeg
--->  Computing dependencies for ffmpeg
--->  Fetching archive for ffmpeg
--->  Attempting to fetch ffmpeg-4.4_4+gpl2.darwin_19.x86_64.tbz2 from https://packages.macports.org/ffmpeg
--->  Attempting to fetch ffmpeg-4.4_4+gpl2.darwin_19.x86_64.tbz2 from https://nue.de.packages.macports.org/ffmpeg
--->  Attempting to fetch ffmpeg-4.4_4+gpl2.darwin_19.x86_64.tbz2 from http://atl.us.packages.macports.org/ffmpeg
--->  Fetching distfiles for ffmpeg
--->  Attempting to fetch ffmpeg-4.4.tar.xz from https://ffmpeg.org/releases/
--->  Verifying checksums for ffmpeg
--->  Extracting ffmpeg
--->  Applying patches to ffmpeg
--->  Configuring ffmpeg
--->  Building ffmpeg
--->  Staging ffmpeg into destroot
--->  Installing ffmpeg @4.4_4+gpl2
--->  Cleaning ffmpeg
--->  Computing dependencies for ffmpeg
--->  Deactivating ffmpeg @4.4_2+gpl2
--->  Cleaning ffmpeg
--->  Activating ffmpeg @4.4_4+gpl2
--->  Cleaning ffmpeg

[mascguy (Christopher Nielsen)]

Replying to msbit:

When troubleshooting the issue, I had tried to run nm on some of the output .o files, but that had failed with:

dyld: Library not loaded: /opt/local/lib/libffi.7.dylib
  Referenced from: /opt/local/libexec/llvm-10/lib/libLLVM.dylib
  Reason: image not found

which seems to be due to a big migration from libffi.7.dylib to libffi.8.dylib that happened while I'd been neglecting this machine?

Just experienced a similar situation, with some macOS VMs that were at least a month out-of-date [in terms of MacPorts].

What's clear is that a libffi upgrade temporarily breaks foundational ports like LLVM, when the dylib major version changes. And that seems like a big problem.

To avoid this situation, I'd like to update libffi to create a dylib symlink, with the previous major version. So when it's updated to libffi.8.dylib, for example, symlink libffi.7.dylib to it. That will prevent breakage of LLVM, along with these related headaches. (The port would determine the previous relative major version, from the current built version. So nothing would be hard-coded.)

Thoughts?

Longer-term, perhaps LLVM components should utilize a separate libffi. I'm thinking in terms of how we avoid circular dependencies among toolchain components, by utilizing separate "bootstrap" ports. (So perhaps utilize a libffi-bootstrap for all LLVM/toolchain components.) Or alternatively, statically link with libffi... assuming that doesn't cause issues?

[mascguy (Christopher Nielsen)]

Related ticket: issue:63378

[Christopher Nielsen <mascguy@…>]

In 6fb2ae6631b642ed4fcf13d0531348e51d29f36c/macports-ports (master):

libffi: symlink via prev major version

• Absolutely critical, to avoid temporary toolchain breakage

Closes: #63443
Closes: #63453
See: #63378

[jmroot (Joshua Root)]

Replying to mascguy:

What's clear is that a libffi upgrade temporarily breaks foundational ports like LLVM, when the dylib major version changes. And that seems like a big problem.

All that's clear from this ticket is that ffmpeg is using cctools' nm without declaring a dependency. It should either ensure that the Xcode nm is used, or depend on cctools. In the latter case, llvm would then be upgraded to the revision linked with the new libffi, before ffmpeg is built.

[mascguy (Christopher Nielsen)]

Great point, and perhaps that's enough to fix ffmpeg.

But are you certain that we don't have a fair number of other ports, which opportunistically utilize cctools components?

[jmroot (Joshua Root)]

Replying to mascguy:

Great point, and perhaps that's enough to fix ffmpeg.

But are you certain that we don't have a fair number of other ports, which opportunistically utilize cctools components?

If you want to look at installing cctools differently to prevent accidental usage, that's fine.

[Christopher Nielsen <mascguy@…>]

In a4617245e09a39d4c873cce84a5bc98307830cc7/macports-ports (master):

ffmpeg: add build dep for cctools
See: #63443

[mascguy (Christopher Nielsen)]

In retrospect, perhaps we all need to utilize trace mode more often, particularly when tackling major upgrades from upstream projects. (I realize that trace mode occasionally causes issues, and won't work 100% of the time. But still not a bad thing to try.)

More recently, I've been handling this by closely scrutinizing output from configure (or the equivalent, for the build system used by a given project), and/or running configure --help. It's more time-consuming to be sure, but pays off longer-term.