Opened 17 months ago

Last modified 4 months ago

#60477 assigned defect

stack @2.3.1_0: Undefined symbols: _utimensat

Reported by: macdeport Owned by: essandess (Steve Smith)
Priority: Normal Milestone:
Component: ports Version: 2.6.2
Keywords: sierra elcapitan yosemite mavericks mountainlion lion snowleopard leopard tiger Cc: macdeport, someuser12, kencu (Ken), iEFdev, Ionic (Mihai Moldovan)
Port: stack

Description


Attachments (1)

main.log.tbz2 (40.9 KB) - added by macdeport 17 months ago.
main.log <= stack-2.3.1_0.darwin_14.x86_64.tbz2

Download all attachments as: .zip

Change History (28)

Changed 17 months ago by macdeport

Attachment: main.log.tbz2 added

main.log <= stack-2.3.1_0.darwin_14.x86_64.tbz2

comment:1 Changed 17 months ago by ryandesign (Ryan Schmidt)

Keywords: stack build upgrade removed
Owner: set to essandess
Port: stack added
Status: newassigned
Summary: Failed to build stack (upgrade 2.1.3_1 < 2.3.1_0)stack @2.3.1_0: Undefined symbols: _utimensat

It looks like the error in the log is:

:info:build Linking .stack-work/dist/x86_64-osx/Cabal-2.4.0.1/build/stack/stack ...
:info:build Undefined symbols for architecture x86_64:
:info:build   "_utimensat", referenced from:
:info:build       _cazW_info in libHSdirectory-1.3.3.0.a(Posix.o)
:info:build ld: symbol(s) not found for architecture x86_64
:info:build clang: error: linker command failed with exit code 1 (use -v to see invocation)

comment:2 Changed 17 months ago by ryandesign (Ryan Schmidt)

I believe utimensat is only available on macOS 10.13 and later. You're on OS X 10.10 so I'm not sure why it's trying to use it.

comment:3 Changed 16 months ago by ryandesign (Ryan Schmidt)

Keywords: sierra elcapitan yosemite mavericks mountainlion lion snowleopard leopard tiger added

This problem affects macOS 10.12 and older. Here is the buildbot log from 10.12. Having the stack port unavailable on an OS as recent as Sierra is limiting. Please figure out how to make the port work on older systems.

I recall prior discussion about the fact that stack was made to work by using precompiled binaries of something. If that's still the case, then perhaps that precompiled binary was compiled on High Sierra or newer and thus makes use of symbols available on newer systems. This yet again exemplifies why we need to build from source on each OS version rather than repackage binaries.

comment:4 Changed 16 months ago by essandess (Steve Smith)

Stack is self-compiling—the pre-compiled binaries are necessary to bootstrap. The situation is exactly the same as for ghc. IDK how upstream bootstraps stack (or ghc) for macOS, but I strongly suspect that they’re cross-compiled, which obvs. wouldn’t work within a MacPorts build.

Therefore, this is an upstream issue.

I’d suggest that anyone who needs stack on 10.12 or earlier drive the agenda by experimenting to see if older stack versions that run on these systems work to compile the latest version, or ask upstream to provide bootstrap instructions for macOS-only non-stack compilers.

comment:5 Changed 15 months ago by reneeotten (Renee Otten)

In 8d76d66a4167ca3de6bd95ced08259605ece3ec5/macports-ports (master):

bali-phy: do not build docs on macOS 10.12 and older

See: #60477

comment:6 Changed 15 months ago by kencu (Ken)

I have stack, ghc, pandoc, and cabal working back to 10.6, at the latest versions, for bootstrapping.

comment:7 Changed 14 months ago by someuser12

Cc: someuser12 added

comment:8 in reply to:  6 Changed 13 months ago by ryandesign (Ryan Schmidt)

Cc: kencu added

Replying to kencu:

I have stack, ghc, pandoc, and cabal working back to 10.6, at the latest versions, for bootstrapping.

Why don't you contribute your fixes so everyone can benefit? :)

comment:9 Changed 13 months ago by kencu (Ken)

Mostly, I've been looking for somewhere to host them other my personal google drive, and once that is done, integrating them into the existing bootstrap of our ghc/cabal/stack ports requires essandess on board.

I did send you the links to all of them on my Google drive previously, but I know it's not your issue to fix. Perhaps I can stick them up on GitHub someplace, as they seem willing to host just about anything these days...

comment:10 Changed 13 months ago by iEFdev

Cc: iEFdev added

comment:11 Changed 13 months ago by ryandesign (Ryan Schmidt)

I had hoped that your fixes would involve changes to the portfile to allow it to build stack on older systems, not that you would provide binaries of something that you compiled outside of MacPorts.

comment:12 Changed 13 months ago by kencu (Ken)

well the issue is that each of these, ghc, stack, and cabal, all require bootstrap binaries that the portfiles use to build newer versions.

And all the bootstrap binaries from upstream crash as they are not built with older systems in mind.

SO the next trick is to see how to use the ones I provided instead of the official upstream binaries (on older systems at least), and then the portfiles can build newer ones.

Last edited 12 months ago by kencu (Ken) (previous) (diff)

comment:13 in reply to:  12 Changed 13 months ago by iEFdev

Replying to kencu:

well the issue that each of these, ghc, stack, and cabal, all require bootstrap binaries that the portfiles use to build newer versions.

Is it something that could be built/done i separate ports? and brought in as deps?

comment:14 Changed 13 months ago by kencu (Ken)

It's a year since I last looked in detail -- I mostly wanted pandoc, and made that for myself.

As I recall, ghc/stack/cabal and their respective ports are all wired up to get their bootstrap compilers from the original sources.

We have to position the more compatible sources I made somewhere accessible, and then rewire to portfiles and the actual software to look whereever we put the sources instead of where they are wired to look.

Exactly how that is to be done is TBA.

If you are interested in digging in, I have the sources for 10.6.8+ working versions of all of those roots, that don't have any ofthe various utimensat etc errors.

And then we have to sort our how not to make a total mess of the (sometimes frighteningly complex) existing portfiles in the process.

comment:15 Changed 12 months ago by kencu (Ken)

Yesterday I updated ghc for 10.6.8+ to 8.10.2, cabal for 10.6.8+ to 3.2.0, and pandoc for 10.6.8+ to the latest version. These are the actual binaries, not Portfiles, at present.

I'm just running into a little haskell-source-code hiccup in one dependency (casa-types) while updating stack to the current release.

Once I have that fixed, I'll make them all available for anyone who wants them.

Last edited 12 months ago by kencu (Ken) (previous) (diff)

comment:16 Changed 12 months ago by kencu (Ken)

Got stack 2.3.3 sorted out. It needs legacysupport for the newer defines and symbols. Using an older stack and with the system ghc set to ghc-8.6.5 which is presently the one it needs, editing the stack.yaml in stack works.

Add this:

in the stack.yaml for stack itself, add this:

ghc-options:
  "$locals": -fhide-source-paths
+   "$targets": -isystem/opt/local/include/LegacySupport/ /opt/local/lib/libMacportsLegacySupport.a

and legacysupport links in statically, and seems to work as it should.

comment:17 Changed 12 months ago by kencu (Ken)

I have posted current versions of ghc, cabal, and stack for 10.6.8 and newer here <https://github.com/kencu/ghc-for-older-darwin-systems/releases/tag/8.10.2>.

These are the binaries, all built on 10.6.8, and all seem to be working as expected there. Of course, there are a few details -- you will have to install ghc and use the system-ghc switch for stack on older systems, otherwise you download the ghc version from the ghc website. There are going to be certain builds that need legacysupport added, most likely (see above post).

I'm going to make some Portfiles that use these on older systems, and see how well they integrate with the Portfiles already on MacPorts. No promises there.

If you need pandoc, which seems to be the most-commonly needed binary that ghc is used to build these days, you can build it quite easily using these roots -- took a few minutes, and I had a current pandoc for 10.6.8.

Last edited 12 months ago by kencu (Ken) (previous) (diff)

comment:18 Changed 12 months ago by kencu (Ken)

stack downloads the info file for all the ghc and other software it might want to install from here <https://raw.githubusercontent.com/fpco/stackage-content/master/stack/stack-setup-2.yaml>, as specified in this file:

stack-2.3.3/src/Stack/Setup.hs

So if we were going to override that to point towards our copies of ghc and stack etc, we'd have to replace that with a file that pointed to our software instead.

Not a trivial undertaking, but not overly hard either. At the moment, I think this is all hardcoded into stack, so you'd need to build stack with a patch that points to our yaml file instead. Perhaps there is some option in the 10,000 options that ghc / stack / cabal seem to have that lets you use a different URL without changing it in the stack build.

comment:19 Changed 8 months ago by Ionic (Mihai Moldovan)

Cc: Ionic added

I finally have a working fix (on 10.9, at least) for this issue.

It requires utimensat support in legacy-support and a release (we're currently working on that part), though, but once everything is in place, we can get proper stack builds on older systems again.

comment:20 Changed 8 months ago by kencu (Ken)

making stack build with legacysupport is --- tricky I found :>

comment:21 Changed 8 months ago by Ionic (Mihai Moldovan)

It's tricky and almost impossible because of some bugs.

The general idea is to pass --ghc-options "-optc-I${prefix}/include/LegacySupport -optl${prefix}/lib/libMacportsLegacySupport.dylib" to the build process, which works surprisingly well.

Sadly, turning ${configure.cflags} and ${configure.ldflags} into ghc options (i.e., -optc... and -optl...) won't work, because this will also add -optl-L${prefix}/lib and -optc-I${prefix}/include, which will make packages within stack link against the GNU iconv implementation and fail to find the symbols later on. Not sure why, probably because Apple's iconv implementation is different, but that's definitely an issue to avoid, so we'll just the library and add the legacy-support-related include directory only manually.

That works... beautifully.

Up until the destroot phase, when everything blows up.

For some reason, stack does not pass down --ghc-options in the install command (which is really just an alias for build --copy-bins). I haven't found a way to make it do this, because I really can't wrap my head around Haskell well enough.

That's bad enough already, but here comes #62228: stack install will actually rebuild EVERYTHING a second time because it doesn't respect STACK_ROOT, which is where the stack database is written to. Instead, it seems to use ~/.stack in a hardcoded fashion, re-download GHC and re-build everything. Without any way to pass special GHC options.

Luckily, the workaround for that is an easy one: just define STACK_ROOT as ${workpath}/.home/.stack (which is really ${HOME}/.stack as far as the process is concerned) and this second rebuild, including its issues, will vanish completely.

comment:22 Changed 8 months ago by kencu (Ken)

btw, many versions of ghc, stack, and cabal for systems back to 10.6.8 are here already:

<https://github.com/kencu/ghc-for-older-darwin-systems/releases>

Last edited 8 months ago by kencu (Ken) (previous) (diff)

comment:23 Changed 8 months ago by kencu (Ken)

the only thing needed to make those versions work for MacPorts is a way to tell stack to use those ghc versions as upstream ghc, rather than the ones on the official ghc website.

Unfortunately, that is not trivial. You have to somehow override stack's "portindex" (packaging manifest), and the mechanism to do that in a way that would work for a diffuse set of MacPorts users is not clear.

It's simple to do on a local, case-by-case basis though, as I explained above.

Last edited 8 months ago by kencu (Ken) (previous) (diff)

comment:25 Changed 5 months ago by harens <harensdeveloper@…>

In 02369fddbd17602ac171f1b53e184a5808874fa3/macports-ports (master):

exa: update to 0.10.1, add git and doc variants

  • Add harens as a co-maintainer
  • Do not build docs on macOS 10.12 and older by default
  • Declares libiconv as a dependency

See: #60477
Closes: #60604

comment:26 Changed 5 months ago by macdeport

Feedback: Thank you for your persevering, sharp and efficient efforts: seems to be fixed, at least at this moment for me...

comment:27 in reply to:  21 Changed 4 months ago by kencu (Ken)

Replying to Ionic:

It's tricky and almost impossible because of some bugs.

I actually don't see any bugs in this. Just the way it works.

The general idea is to pass --ghc-options "-optc-I${prefix}/include/LegacySupport -optl${prefix}/lib/libMacportsLegacySupport.dylib" to the build process, which works surprisingly well.

Yeah, that is the command-line equivalent to what I offered here 60477?replyto=21#comment:16 before.

I moved my fix as per the yaml file into my main config.yaml on my older systems, so now legacy support is always used when building with stack. No more troubles ;>

Sadly, turning ${configure.cflags} and ${configure.ldflags} into ghc options (i.e., -optc... and -optl...) won't work, because this will also add -optl-L${prefix}/lib and -optc-I${prefix}/include, which will make packages within stack link against the GNU iconv implementation and fail to find the symbols later on. Not sure why, probably because Apple's iconv implementation is different, but that's definitely an issue to avoid, so we'll just the library and add the legacy-support-related include directory only manually.

This iconv vs libiconv symbol-naming issue goes back many many years. libiconv does that on purpose -- different sympbols for system vs user installations of libiconv to find and flag misbuilt and/or misconfigured software that mixes up headers and libraries, which I guess they found was all too common (and apparently still is ;> ).

I have updated my builds of the haskell ecosystem for 10.6+, including ghc-9.x, stack 2.71, cabal 3.40, and the needed alex and happy versions. All this haskell stuff works great for me back to 10.6.8 still, and I can build anything I want, eg pandoc and shellcheck on Lion:

$ port -v installed pandoc
The following ports are currently installed:
  pandoc @2.13_0 (active) requested_variants='' platform='darwin 11' archs='x86_64' date='2021-05-27T07:31:22-0700'

$ port -v installed shellcheck
The following ports are currently installed:
  shellcheck @0.7.2_0 (active) requested_variants='' platform='darwin 11' archs='x86_64' date='2021-05-27T08:34:45-0700'

The stack that is downloaded from upstream is built on a system that is not friendly to older systems. I just replace that one in the port with my own, and everything is 100%.

I may update the stack port here on MacPorts to use the one I have which actually works on older systems, but perhaps it would be easier to just make the binaries of pandoc and shellcheck available because in the end, that is pretty much all that people want this for anyway.

Last edited 4 months ago by kencu (Ken) (previous) (diff)
Note: See TracTickets for help on using tickets.