Opened 5 years ago

Closed 5 years ago

Last modified 5 years ago

#59217 closed defect (fixed)

Need to delete 10.6-10.8 archives for ports that used the cxx11 1.1 portgroup and rebuild them

Reported by: ryandesign (Ryan Carsten Schmidt) Owned by: ryandesign (Ryan Carsten Schmidt)
Priority: Normal Milestone:
Component: server/hosting Version: 2.6.1
Keywords: snowleopard lion mountainlion Cc: michaelld (Michael Dickens), jmroot (Joshua Root), MarcusCalhoun-Lopez (Marcus Calhoun-Lopez), mojca (Mojca Miklavec), cjones051073 (Chris Jones)
Port:

Description

At least on 10.8 (e.g. boost-1.71.0_0+no_single+no_static+python27.darwin_12.x86_64), boost is linked with MacPorts libstdc++, so of course programs that use libc++ fail to link with it.

I don't understand why gcc got anywhere near this build.

I have seen a similar problem on 10.6-10.8 with graphite2 (e.g. graphite2-1.3.13_0.darwin_12.x86_64.tbz2), and I don't understand why that happened either.

It looks like these archives were not identified for deletion and rebuild during our switchover to libc++. Josh, do you know why?

Attachments (8)

cxx11-1.1-ports.txt (3.8 KB) - added by ryandesign (Ryan Carsten Schmidt) 5 years ago.
cxx11-1.1-subports.txt (18.0 KB) - added by ryandesign (Ryan Carsten Schmidt) 5 years ago.
snowleopard-i386-uninstall.log (10.5 KB) - added by ryandesign (Ryan Carsten Schmidt) 5 years ago.
snowleopard-uninstall.log (11.4 KB) - added by ryandesign (Ryan Carsten Schmidt) 5 years ago.
lion-uninstall.log (15.1 KB) - added by ryandesign (Ryan Carsten Schmidt) 5 years ago.
mountainlion-uninstall.log (17.1 KB) - added by ryandesign (Ryan Carsten Schmidt) 5 years ago.
delete-packages.log (199.8 KB) - added by ryandesign (Ryan Carsten Schmidt) 5 years ago.
delete-packages-private.log (65.9 KB) - added by ryandesign (Ryan Carsten Schmidt) 5 years ago.

Download all attachments as: .zip

Change History (36)

comment:1 Changed 5 years ago by jmroot (Joshua Root)

In the metadata of the graphite2 archive it has cxx_stdlib_overridden 1. So apparently something changed configure.cxx_stdlib to something different to the global cxx_stdlib at the portfile level when it was built. Beyond that no idea sorry.

comment:2 Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)

Cc: MarcusCalhoun-Lopez added

Looks like the cxx11 1.1 portgroup sets cxx_stdlib to macports-libstdc++ if it was libstdc++. The portgroup has done this ever since it was added; I don't know why. Marcus, do you remember?

I think I should identify and remove all 10.6-10.8 binaries for ports that override cxx_stdlib, and rebuild them. Or, restrict it to only those that override it to macports-libstdc++, if that information is available in the metadata.

Last edited 5 years ago by ryandesign (Ryan Carsten Schmidt) (previous) (diff)

comment:3 Changed 5 years ago by jmroot (Joshua Root)

Overriding cxx_stdlib is a perfectly valid thing to do in some cases and rebuilding ports that still do it won't change anything. Just removing the 10.6-10.8 archives for all ports that used the cxx11 portgroup at the time of the changeover and haven't been updated since should be fine.

Last edited 5 years ago by jmroot (Joshua Root) (previous) (diff)

comment:4 in reply to:  3 Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)

Replying to jmroot:

Overriding cxx_stdlib is a perfectly valid thing to do in some cases and rebuilding ports that still do it won't change anything.

I know, I just didn't think of another way to identify the ones that need rebuilds.

Just removing the 10.6-10.8 archives for all ports that use the cxx11 portgroup and haven't been updated since the changeover should be fine.

That's a better way, except I'll have to use an old checkout of ports to identify those that used the portgroup at the time of the changeover, since several ports have had that portgroup removed since then without a revbump that would have rebuilt them.

comment:5 Changed 5 years ago by jmroot (Joshua Root)

Right. And then delete the corresponding archives if their mtime is before the changeover.

comment:6 in reply to:  2 Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)

Replying to ryandesign:

Looks like the cxx11 1.1 portgroup sets cxx_stdlib to macports-libstdc++ if it was libstdc++. The portgroup has done this ever since it was added; I don't know why. Marcus, do you remember?

notEnoughCoffeeErr: Obviously it's because macOS libstdc++ doesn't support C++11 so MacPorts libstdc++ has to be used.

comment:7 Changed 5 years ago by mojca (Mojca Miklavec)

Cc: mojca added

comment:8 Changed 5 years ago by mojca (Mojca Miklavec)

I tried to compile boost with the following patch:

  • devel/boost/Portfile

     
    11# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:filetype=tcl:et:sw=4:ts=4:sts=4
    22
    33PortSystem      1.0
    4 PortGroup       compiler_blacklist_versions 1.0
    54PortGroup       mpi 1.0
    6 PortGroup       cxx11 1.1
    75PortGroup       active_variants 1.1
     6compiler.cxx_standard   2011
     7configure.cxxflags-append -std=c++11
    88
    99name            boost
    1010
    mpi.setup -gcc 
    104104# Further: Building Boost using C++11 compliance does not seem to then
    105105# require ports depending on Boost to also require C++11 compliance,
    106106# and requiring it does make such building easier for those ports.
    107 configure.cxxflags-append -std=gnu++11
     107#configure.cxxflags-append -std=gnu++11
    108108
    109109# It turns out that ccache and distcc can produce boost libraries that, although they
    110110# compile without warning, have all sorts of runtime errors especially with pointer corruption.

but it fails to build due to some errors with (potentially misidentified?) thread-local support:

    "/opt/local/bin/clang++-mp-9.0" -dynamiclib -Wl,-single_module -install_name "/opt/local/lib/libboost_context-mt.dylib"  -o "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/libboost_context-mt.dylib" "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/asm/make_x86_64_sysv_macho_gas.o" "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/asm/jump_x86_64_sysv_macho_gas.o" "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/asm/ontop_x86_64_sysv_macho_gas.o" "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/execution_context.o" "bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/posix/stack_traits.o"      -headerpad_max_install_names -fPIC -m64 -fvisibility=hidden -fvisibility-inlines-hidden -Wl,-dead_strip -no_dead_strip_inits_and_terms -L/opt/local/lib -Wl,-headerpad_max_install_names -stdlib=libc++ -arch x86_64 

Undefined symbols for architecture x86_64:
  "___cxa_thread_atexit", referenced from:
      boost::context::detail::ecv1_activation_record::current() in execution_context.o
      boost::context::detail::ecv1_activation_record_initializer::ecv1_activation_record_initializer() in execution_context.o
      boost::context::detail::ecv1_activation_record_initializer::ecv1_activation_record_initializer() in execution_context.o
      boost::context::detail::ecv1_activation_record_initializer::~ecv1_activation_record_initializer() in execution_context.o
      boost::context::detail::ecv1_activation_record_initializer::~ecv1_activation_record_initializer() in execution_context.o
      boost::context::v1::execution_context::current() in execution_context.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
...failed darwin.link.dll bin.v2/libs/context/build/darwin-9.0.0/release/threading-multi/visibility-hidden/libboost_context-mt.dylib...
...skipped

comment:9 Changed 5 years ago by mojca (Mojca Miklavec)

Cc: kencu cjones051073 added

comment:10 Changed 5 years ago by jmroot (Joshua Root)

Cc: kencu cjones051073 removed

Even though the compiler may support TLS, libcxx probably doesn't.

comment:11 Changed 5 years ago by cjones051073 (Chris Jones)

Cc: cjones051073 added

comment:12 Changed 5 years ago by kencu (Ken)

Cc: cjones051073 removed

Mojca is the first to run into this 58898

comment:13 Changed 5 years ago by cjones051073 (Chris Jones)

why is everyone removing me from the cc list ??

comment:14 Changed 5 years ago by cjones051073 (Chris Jones)

Cc: cjones051073 added

comment:15 Changed 5 years ago by kencu (Ken)

simultaneous posting to the ticket

comment:16 Changed 5 years ago by cjones051073 (Chris Jones)

well when that happens you get a warning.... Its best I find to not ignore that warning and refresh before posting (e.g. copy the message and then retype it again).

comment:17 Changed 5 years ago by kencu (Ken)

Apologies. Everything harder on this ipad (and also I often miss spelling errors as the writing is so small). I'll try to do better!

Last edited 5 years ago by kencu (Ken) (previous) (diff)

comment:18 Changed 5 years ago by cjones051073 (Chris Jones)

no apologies needed, I was just saying what works for me.

(I mostly blame trac here. I'm not a fan of its UI.... Its pretty backward I would say that it cannot properly handle simultaneous posts without losing information ).

Last edited 5 years ago by cjones051073 (Chris Jones) (previous) (diff)

comment:19 Changed 5 years ago by mojca (Mojca Miklavec)

I just wanted to thank you for the quick hint. I was able to compile boost with the above mentioned patches by just running sudo port install libcxx again.

This also means that now liblas works with the above patch for boost (so far reported broken by our buildbots).

Last edited 5 years ago by mojca (Mojca Miklavec) (previous) (diff)

comment:20 Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)

Owner: set to ryandesign
Port: boost graphite2 removed
Status: newaccepted
Summary: boost, graphite2, possibly other ports use macports-libstdc++ instead of libc++Need to delete 10.6-10.8 archives for ports that used the cxx11 1.1 portgroup and rebuild them

This ticket is about the need to identify the ports that used the cxx11 1.1 portgroup at the time of the MacPorts 2.6.0 release; uninstall them from the 10.6-10.8 builders; delete the archives from the server; delete failcache entries for any ports that depend on them; and schedule builds to rebuild them.

Any discussion of proposed modifications to boost or another port need to go elsewhere.

comment:21 Changed 5 years ago by mojca (Mojca Miklavec)

It might also generally be about all the ports that still use the cxx11 1.1 PortGroup which needs to be removed, else the removal itself won't really help. As these ports need to be modified, I suspect that we need to revbump them (and dependencies) anyway, since users do need to upgrade them on their machine (unless revupgrade will do that?). I'm not sure about dependencies. The one dependency I tried failed to build due to stdlib mismatch anyway.

I'm waiting for #58898 to be resolved before being able to change boost / before opening a PR.

comment:22 Changed 5 years ago by jmroot (Joshua Root)

The portgroup doesn't need to be removed, though there's also no reason to use it now. The ports don't need to be modified, the archives just need to be deleted and the builds can be forced. Rev-upgrade will take care of any mismatches on users' machines.

Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)

Attachment: cxx11-1.1-ports.txt added

Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)

Attachment: cxx11-1.1-subports.txt added

Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)

Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)

Attachment: snowleopard-uninstall.log added

Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)

Attachment: lion-uninstall.log added

Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)

Attachment: mountainlion-uninstall.log added

Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)

Attachment: delete-packages.log added

Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)

Attachment: delete-packages-private.log added

comment:23 Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)

Component: portsserver/hosting

I downloaded the 2.6.0 archive of ports, extracted it, ran portindex in it, set up sources.conf to use it as my primary ports tree, and ran these commands in it to find all the portfiles that used the cxx11 1.1 portgroup at the time that MacPorts 2.6.0 was released:

find . -name Portfile \
| xargs grep -E '^[[:space:]]*PortGroup[[:space:]]+cxx11[[:space:]]+1\.1[[:space:]]*$' \
| cut -d / -f 3 \
| tee ~/cxx11-1.1-ports.txt

Output: cxx11-1.1-ports.txt

Then I used these commands to identify all the subports:

xargs -I % port echo % subportof:% < ~/cxx11-1.1-ports.txt \
| sed -E 's/ +//' \
| tee ~/cxx11-1.1-subports.txt

Output: cxx11-1.1-subports.txt

I didn't attempt to distinguish which subports used the portgroup. If the portgroup is mentioned at all in a portfile, I'm counting all of its subports as using it. This means some ports will get uninstalled / deleted that didn't need to be. That's fine. They'll get rebuilt.

I copied cxx11-1.1-subports.txt to the 10.6-10.8 builders and uninstalled those subports using these commands (run once on each builder):

xargs /opt/local/bin/port -q installed < ~/cxx11-1.1-subports.txt \
| sed -E 's/ \(active\)$//' \
| xargs sudo /opt/local/bin/port -f uninstall \
| tee ~/$(hostname -s)-uninstall.log

Output: snowleopard-i386-uninstall.log snowleopard-uninstall.log lion-uninstall.log mountainlion-uninstall.log

Even ports that had already been updated/fixed after the 2.6.0 release were uninstalled. That's fine. They'll get reinstalled automatically as needed by future builds.

Other ports had failed to build because of this problem with these cxx11 ports, so I deleted the entire failcache on the 10.6-10.8 builders. This seemed like the simplest way to ensure no builds are skipped because of this problem.

I copied cxx11-1.1-subports.txt to the packages server and deleted the public and private archives for 10.6-10.8 older than the 2.6.0 release date using these commands (run once in each directory):

# 2.6.0 release date as mmddHHMMccyy (UTC)
release_date=$(date -ju 092102492019 +%s)
while read p; do
  for f in $p/*.darwin_1[012].*.tbz2; do
    if [[ $f != *.noarch.tbz2 && -f $f ]]; then
      mod_date=$(stat -f %m $f)
      if [ $mod_date -lt $release_date ]; then
        echo $f
        echo $f.rmd160
      fi
    fi
  done
done < ~/cxx11-1.1-subports.txt \
| xargs rm -fv \
| tee ~/delete-$(basename $PWD).log

Output: delete-packages.log delete-packages-private.log

I'm now waiting for the deletion of the packages to sync to the public server. After that, I'll purge the MaxCDN cache for the packages zone, reactivate the 10.6-10.8 builders, and schedule new builds of cxx11-1.1-ports.txt on the 10.6-10.8 builders. And that should take care of this ticket.

comment:24 Changed 5 years ago by mojca (Mojca Miklavec)

Thank you very much!

Out of curiosity: what's the status of the 10.5/ppc and 10.6/i386 builders?

comment:25 in reply to:  24 Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)

Replying to mojca:

what's the status of the 10.5/ppc and 10.6/i386 builders?

10.6 i386 is now online along with 10.6-10.8 x86_64. I was bringing them online one at a time to make sure things were working.

10.5 ppc remains offline due to a failed power supply and the power supply is the most ridiculously hard component of a G5 to remove/replace. I have a replacement G5 that I'd like to bring online, but it needs to be vigorously cleaned first.

I'll wait to reschedule the cxx11-1.1-ports.txt​ ports until the queues are clear. I think that will probably help minimize any problems caused by #59286 / #53587 / #59287.

comment:26 Changed 5 years ago by mojca (Mojca Miklavec)

I'll try to speed up my BB 2 setup which should hopefully address most of the issues in the tickets you linked (I really hope that the last two won't even happen).

comment:27 Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)

Resolution: fixed
Status: acceptedclosed

All of the builds have been rescheduled.

comment:28 Changed 5 years ago by jmroot (Joshua Root)

Great; thanks Ryan.

Note: See TracTickets for help on using tickets.