cctools @927.0.2: not fully compatible with llvm after llvm10: Unknown command line argument '-macho'.

[blair (Blair Zajac)]

When cctools +llvm11 is installed than gobject-introspection fails to build:

CPPFLAGS="-I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk" CFLAGS="-pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -arch x86_64 -Wno-duplicated-branches -Wno-switch-enum" LDFLAGS="-L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -arch x86_64" CC="/usr/bin/clang -arch x86_64" PKG_CONFIG="/opt/local/bin/pkg-config" GI_HOST_OS="" DLLTOOL="false"  env PATH=".libs:/opt/local/bin:/opt/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin" LPATH=.libs CC="/usr/bin/clang -arch x86_64" PYTHONPATH=".:.:" UNINSTALLED_INTROSPECTION_SRCDIR=. UNINSTALLED_INTROSPECTION_BUILDDIR=. ./g-ir-scanner  --verbose -I. --add-include-path=. --add-include-path=./gir --add-include-path=. --add-include-path=. --add-include-path=./gir --namespace=GLib --nsversion=2.0 --libtool="/bin/sh ./libtool" --pkg=glib-2.0     --library=glib-2.0 --library=gobject-2.0 --external-library --reparse-validate --identifier-prefix=G --symbol-prefix=g --symbol-prefix=glib --c-include="glib.h"  --cflags-begin -I/opt/local/include/glib-2.0 -I/opt/local/lib/glib-2.0/include -DGETTEXT_PACKAGE=Dummy -DGLIB_COMPILATION -D__G_I18N_LIB_H__ --cflags-end  /opt/local/lib/glib-2.0/include/glibconfig.h /opt/local/include/glib-2.0/glib/galloca.h /opt/local/include/glib-2.0/glib/garray.h /opt/local/include/glib-2.0/glib/gasyncqueue.h /opt/local/include/glib-2.0/glib/gatomic.h /opt/local/include/glib-2.0/glib/gbacktrace.h /opt/local/include/glib-2.0/glib/gbase64.h /opt/local/include/glib-2.0/glib/gbitlock.h /opt/local/include/glib-2.0/glib/gbookmarkfile.h /opt/local/include/glib-2.0/glib/gbytes.h /opt/local/include/glib-2.0/glib/gcharset.h /opt/local/include/glib-2.0/glib/gchecksum.h /opt/local/include/glib-2.0/glib/gconvert.h /opt/local/include/glib-2.0/glib/gdataset.h /opt/local/include/glib-2.0/glib/gdate.h /opt/local/include/glib-2.0/glib/gdatetime.h /opt/local/include/glib-2.0/glib/gdir.h /opt/local/include/glib-2.0/glib/genviron.h /opt/local/include/glib-2.0/glib/gerror.h /opt/local/include/glib-2.0/glib/gfileutils.h /opt/local/include/glib-2.0/glib/ggettext.h /opt/local/include/glib-2.0/glib/ghash.h /opt/local/include/glib-2.0/glib/ghmac.h /opt/local/include/glib-2.0/glib/ghook.h /opt/local/include/glib-2.0/glib/ghostutils.h /opt/local/include/glib-2.0/glib/gi18n-lib.h /opt/local/include/glib-2.0/glib/gi18n.h /opt/local/include/glib-2.0/glib/giochannel.h /opt/local/include/glib-2.0/glib/gkeyfile.h /opt/local/include/glib-2.0/glib/glist.h /opt/local/include/glib-2.0/glib/gmacros.h /opt/local/include/glib-2.0/glib/gmain.h /opt/local/include/glib-2.0/glib/gmappedfile.h /opt/local/include/glib-2.0/glib/gmarkup.h /opt/local/include/glib-2.0/glib/gmem.h /opt/local/include/glib-2.0/glib/gmessages.h /opt/local/include/glib-2.0/glib/gnode.h /opt/local/include/glib-2.0/glib/goption.h /opt/local/include/glib-2.0/glib/gpattern.h /opt/local/include/glib-2.0/glib/gpoll.h /opt/local/include/glib-2.0/glib/gprimes.h /opt/local/include/glib-2.0/glib/gprintf.h /opt/local/include/glib-2.0/glib/gqsort.h /opt/local/include/glib-2.0/glib/gquark.h /opt/local/include/glib-2.0/glib/gqueue.h /opt/local/include/glib-2.0/glib/grand.h /opt/local/include/glib-2.0/glib/grcbox.h /opt/local/include/glib-2.0/glib/grefcount.h /opt/local/include/glib-2.0/glib/grefstring.h /opt/local/include/glib-2.0/glib/gregex.h /opt/local/include/glib-2.0/glib/gscanner.h /opt/local/include/glib-2.0/glib/gsequence.h /opt/local/include/glib-2.0/glib/gshell.h /opt/local/include/glib-2.0/glib/gslice.h /opt/local/include/glib-2.0/glib/gslist.h /opt/local/include/glib-2.0/glib/gspawn.h /opt/local/include/glib-2.0/glib/gstdio.h /opt/local/include/glib-2.0/glib/gstrfuncs.h /opt/local/include/glib-2.0/glib/gstring.h /opt/local/include/glib-2.0/glib/gstringchunk.h /opt/local/include/glib-2.0/glib/gtestutils.h /opt/local/include/glib-2.0/glib/gthread.h /opt/local/include/glib-2.0/glib/gthreadpool.h /opt/local/include/glib-2.0/glib/gtimer.h /opt/local/include/glib-2.0/glib/gtimezone.h /opt/local/include/glib-2.0/glib/gtrashstack.h /opt/local/include/glib-2.0/glib/gtree.h /opt/local/include/glib-2.0/glib/gtypes.h /opt/local/include/glib-2.0/glib/gunicode.h /opt/local/include/glib-2.0/glib/gurifuncs.h /opt/local/include/glib-2.0/glib/gutils.h /opt/local/include/glib-2.0/glib/guuid.h /opt/local/include/glib-2.0/glib/gvariant.h /opt/local/include/glib-2.0/glib/gvarianttype.h /opt/local/include/glib-2.0/glib/gversion.h /opt/local/include/glib-2.0/glib/gversionmacros.h /opt/local/include/glib-2.0/glib/gwin32.h /opt/local/include/glib-2.0/gobject/glib-types.h gir/glib-2.0.c /opt/local/include/glib-2.0/glib-unix.h g-ir-scanner g-ir-compiler --output GLib-2.0.gir
/usr/bin/clang -arch x86_64 -E -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -DGETTEXT_PACKAGE=Dummy -DGLIB_COMPILATION -D__G_I18N_LIB_H__ -U__BLOCKS__ -I. -I/opt/local/include/glib-2.0 -I/opt/local/lib/glib-2.0/include -I/opt/local/var/macports/build/_Users_blairzajac_Code_MacPorts_macports-ports.git_gnome_gobject-introspection/gobject-introspection/work/gobject-introspection-1.60.2 -o g-ir-cpp-ygnxvj6t.i -C /opt/local/var/macports/build/_Users_blairzajac_Code_MacPorts_macports-ports.git_gnome_gobject-introspection/gobject-introspection/work/gobject-introspection-1.60.2/g-ir-cpp-ygnxvj6t.c
/usr/bin/clang -arch x86_64 -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -arch x86_64 -Wno-duplicated-branches -Wno-switch-enum -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -D_REENTRANT -DGETTEXT_PACKAGE=Dummy -DGLIB_COMPILATION -D__G_I18N_LIB_H__ -I/opt/local/var/macports/build/_Users_blairzajac_Code_MacPorts_macports-ports.git_gnome_gobject-introspection/gobject-introspection/work/gobject-introspection-1.60.2 -I/opt/local/include/glib-2.0 -I/opt/local/lib/glib-2.0/include -I/opt/local/include/glib-2.0 -I/opt/local/lib/glib-2.0/include -I/opt/local/include/glib-2.0 -I/opt/local/lib/glib-2.0/include -c /opt/local/var/macports/build/_Users_blairzajac_Code_MacPorts_macports-ports.git_gnome_gobject-introspection/gobject-introspection/work/gobject-introspection-1.60.2/tmp-introspectnchs_fhs/GLib-2.0.c -o /opt/local/var/macports/build/_Users_blairzajac_Code_MacPorts_macports-ports.git_gnome_gobject-introspection/gobject-introspection/work/gobject-introspection-1.60.2/tmp-introspectnchs_fhs/GLib-2.0.o -Wno-deprecated-declarations
warning: unknown warning option '-Wno-duplicated-branches' [-Wunknown-warning-option]
1 warning generated.
g-ir-scanner: link: /bin/sh ./libtool --mode=link --tag=CC /usr/bin/clang -arch x86_64 -o /opt/local/var/macports/build/_Users_blairzajac_Code_MacPorts_macports-ports.git_gnome_gobject-introspection/gobject-introspection/work/gobject-introspection-1.60.2/tmp-introspectnchs_fhs/GLib-2.0 -export-dynamic -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -arch x86_64 -Wno-duplicated-branches -Wno-switch-enum /opt/local/var/macports/build/_Users_blairzajac_Code_MacPorts_macports-ports.git_gnome_gobject-introspection/gobject-introspection/work/gobject-introspection-1.60.2/tmp-introspectnchs_fhs/GLib-2.0.o -lgio-2.0 -lgobject-2.0 -lgmodule-2.0 -lglib-2.0 -lintl -Wl,-framework -Wl,CoreFoundation -lglib-2.0 -lgobject-2.0 -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -arch x86_64
libtool: link: /usr/bin/clang -arch x86_64 -o /opt/local/var/macports/build/_Users_blairzajac_Code_MacPorts_macports-ports.git_gnome_gobject-introspection/gobject-introspection/work/gobject-introspection-1.60.2/tmp-introspectnchs_fhs/GLib-2.0 -I/opt/local/include -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -arch x86_64 -Wno-duplicated-branches -Wno-switch-enum /opt/local/var/macports/build/_Users_blairzajac_Code_MacPorts_macports-ports.git_gnome_gobject-introspection/gobject-introspection/work/gobject-introspection-1.60.2/tmp-introspectnchs_fhs/GLib-2.0.o -Wl,-framework -Wl,CoreFoundation -Wl,-headerpad_max_install_names -Wl,-syslibroot -Wl,/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -arch x86_64  -lgio-2.0 -lgmodule-2.0 -lintl -lglib-2.0 -lgobject-2.0 -L/opt/local/lib
llvm-objdump: Unknown command line argument '-macho'.  Try: '/opt/local/libexec/llvm-11/bin/llvm-objdump --help'
llvm-objdump: Did you mean '-h'?
llvm-objdump: Unknown command line argument '-dylibs-used'.  Try: '/opt/local/libexec/llvm-11/bin/llvm-objdump --help'
llvm-objdump: Did you mean '--dylibs-used'?
llvm-objdump: Unknown command line argument '-non-verbose'.  Try: '/opt/local/libexec/llvm-11/bin/llvm-objdump --help'
llvm-objdump: Did you mean '--non-verbose'?
Traceback (most recent call last):
  File "./g-ir-scanner", line 99, in <module>
    sys.exit(scanner_main(sys.argv))
  File "./giscanner/scannermain.py", line 579, in scanner_main
    shlibs = create_binary(transformer, options, args)
  File "./giscanner/scannermain.py", line 419, in create_binary
    shlibs = resolve_shlibs(options, binary, options.libraries)
  File "./giscanner/shlibs.py", line 183, in resolve_shlibs
    _resolve_non_libtool(options, binary, non_libtool))
  File "./giscanner/shlibs.py", line 103, in _resolve_non_libtool
    output = subprocess.check_output(args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/bin/sh', './libtool', '--mode=execute', 'otool', '-L', '/opt/local/var/macports/build/_Users_blairzajac_Code_MacPorts_macports-ports.git_gnome_gobject-introspection/gobject-introspection/work/gobject-introspection-1.60.2/tmp-introspectnchs_fhs/GLib-2.0']' returned non-zero exit status 1.
make[2]: *** [GLib-2.0.gir] Error 1

When cctools +llvm90 is installed than gobject-introspection builds fine.

Do all the calls to add_execute_list() in otool/main.c need to be updated to add another - to the argument?

[kencu (Ken)]

Hi Blair, this is a known issue, and is not a bug -- they did this on purpose. There are a number of other tickets against various software builds that ran into the same thing.

Newer versions of llvm come with a new version of llvm-objdump, and they decided upstream to change the arguments that llvm-objdump accepts. llvm-objdump will no longer accept -macho. You can use -m and you can use --macho now. All the old scripts that use -macho are broken, and will need to be gradually updated to no longer do that.

If you want to get on with your work, please do this:

sudo port -f uninstall cctools
sudo port -v install cctools

and then you should get a proper working version of cctools for your system, which is cctools +xcode.

If you want to help fix all the scripts that are broken, you can stay on cctools +llvm11 and start submitting PRs (upstream, preferably) to get them all updated.

[kencu (Ken)]

have to do a bit of sleuthing to sort out if it's gobject-introspection or python that needs to be updated here to the new call.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #61901.

[michaelld (Michael Dickens)]

has duplicate #62095

[michaelld (Michael Dickens)]

Copied from #62095 ... where we're telling poppler to save-temps and thus we can test this issue outside the MacPorts' environment.

Looking at the cctools code in cctools-927.0.2/otool/main.c, line 80 reads:

enum bool show_objdump_command = FALSE; /* print the objdump command */

change this to TRUE; build and install. then run the otool command again:

$ otool -L /opt/local/var/macports/build/_opt_sources_MacPorts_ports_github_macports_graphics_poppler/poppler/work/build/glib/tmp-introspectdnhi97mq/Poppler-0.18
+ /opt/local/bin/llvm-objdump-mp-11 -macho -dylibs-used -non-verbose /opt/local/var/macports/build/_opt_sources_MacPorts_ports_github_macports_graphics_poppler/poppler/work/build/glib/tmp-introspectdnhi97mq/Poppler-0.18
llvm-objdump: Unknown command line argument '-macho'.  Try: '/opt/local/libexec/llvm-11/bin/llvm-objdump --help'
llvm-objdump: Did you mean '-h'?
llvm-objdump: Unknown command line argument '-dylibs-used'.  Try: '/opt/local/libexec/llvm-11/bin/llvm-objdump --help'
llvm-objdump: Did you mean '--dylibs-used'?
llvm-objdump: Unknown command line argument '-non-verbose'.  Try: '/opt/local/libexec/llvm-11/bin/llvm-objdump --help'
llvm-objdump: Did you mean '--non-verbose'?

... so ... this is a cctools issue, not GO-I or poppler.

[kencu (Ken)]

Uhm -- this is expected behaviour from cctools/llvm-objdump (this command comes from the llvm tree), it does not accept "-macho" as an argument any more, it is not a bug, they did this on purpose, they won't be fixing it or reverting it.

The error is this:

/opt/local/bin/llvm-objdump-mp-11 -macho

as -macho is not an accepted flag any more.

Am I backwards on this somehow? I'm multitasking here...

[kencu (Ken)]

Oh -- do you mean that somehow the cctools otools driver is calling llvm-objdump and that is what is adding the -macho flag?

If that is what is happening, then yes, 100%, we should patch that out of cctools (or update cctools). Sorry if I missed this first time round.

[michaelld (Michael Dickens)]

Yes it's this latter: cctools otools driver is calling llvm-objdump and that is what is adding the -macho flag! It's easy to fix in the cctools code luckily!

[michaelld (Michael Dickens)]

Given that the current cctools variants are just "llvm10, llvm11, llvmdev, [+]xcode", we can patch the cctools otool code to use -- instead of - for those arguments. I think that's the only change required.

[kencu (Ken)]

those are the only llvm variants on 10.19+, anyway

on 10.4 and up, I need to see if the old llvms accept "--macho"; otherwise we can just use "-m" which is accepted still, for no good reason :>

[kencu (Ken)]

looks like llvm-objdump-3.4 accepts "--macho" as well as "-m" and "-macho".

So I can patch it to "--macho".

[michaelld (Michael Dickens)]

makes sense! make it so!

[kencu (Ken)]

oh oh. Looks like there is a whole mess of single-hyphen arguments that are no longer accepted:

% otool -l /opt/local/bin/mtoc
llvm-objdump: Unknown command line argument '-private-headers'.  Try: '/opt/local/libexec/llvm-11/bin/llvm-objdump --help'
llvm-objdump: Did you mean '--private-headers'?
llvm-objdump: Unknown command line argument '-non-verbose'.  Try: '/opt/local/libexec/llvm-11/bin/llvm-objdump --help'
llvm-objdump: Did you mean '--non-verbose'?

[kencu (Ken)]

I smell my cctools update coming ...

[kencu (Ken)]

In 3c027d9f80b35954bd37e5d715874527643e6bab/macports-ports (master):

cctools: remove llvm11 variant

there is no released version of cctools
that is compatible with the llvm11
calling conventions.

we'll remove the llvm11 variant until such
time as an appropriate version of cctools
becomes available.

users who have cctools with an llvm variant
of < 11 will see an upgrade.

users who have cctools with llvm11 will be
updated to the appropriate version for their
systems

closes: #61967

[kencu (Ken)]

So -- why?

a lot of scripts, and people, use otool. otool is binary driver for llvm-objdump. otool has compiled into it the flags for llvm-objdump that map to the old otool flags. It translates the flags, then calls llvm-objdump.

but the flags that llvm-objdump accepts have changed, and otools now adds flags that llvm-objdump in llvm11 does not support, ergo the errors.

It's not just one -- there are a whole bunch of them.

And, perhaps more such issues, in other parts of cctools. This is the only one we've found so far. The newer version of cctools that is out there, 949, has the same issues, and it has not been fixed there yet either. No doubt some newer open source version of cctools will be coming out eventually that does support the newer llvm calling conventions.

Now sure, I could take a stab at patching otool to emit the new flags. One or two, easy -- but there are lots. And maybe more? And maybe in other tools?

There is no good reason to cause this grief upon our users. AFAIK, llvm-10 was OK, and that makes sense, timeline wise... cctools is about that vintage.

So we will have most users on newer systems using cctools +xcode, as they should be using. And we'll make available llvm10, as that was not visibly broken.

And for those few users out there who really want the newest, know what they are doing, and can hack in on it -- cctools +llvmdev is still available.

I'll re-enable cctools +llvm11 when it works properly.

[michaelld (Michael Dickens)]

Thanks for your detailed thoughts here, @kencu . Makes sense all around IMHO! Glad to get this issue fixed!

[kencu (Ken)]

This issue is still open, until I update cctools to some new version that has the new calling conventions in it. A few folks can hit it when playing with cctools +llvmdev (unless I decide to remove that option to stop that).

[kencu (Ken)]

has duplicate #63448