bsdtar crashes with dyld missing symbol when upgrading a dependent port such as libxml2

[PerMildner]

Something like the following often happens when I do sudo port upgrade outdated.

-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               bsdtar [72528]
Path:                  /opt/local/bin/bsdtar
Identifier:            bsdtar
Version:               ???
Code Type:             X86-64 (Native)
Parent Process:        tclsh8.6 [72518]
Responsible:           Emacs-x86_64-10_14 [1074]
User ID:               0

Date/Time:             2023-11-08 10:15:18.1561 +0100
OS Version:            macOS 13.6.1 (22G313)
Report Version:        12
Bridge OS Version:     8.1 (21P1069)
Anonymous UUID:        0B3FD7AE-CD61-E7F1-5BBA-25341183702C

Sleep/Wake UUID:       4A798EDD-FE76-4E04-A49B-2F9F16D0F35F

Time Awake Since Boot: 650000 seconds
Time Since Wake:       856 seconds

System Integrity Protection: enabled

Crashed Thread:        0

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000

Termination Reason:    Namespace DYLD, Code 4 Symbol missing
Symbol not found: _lzma_cputhreads
Referenced from: <EAC1F2A3-C365-3B4B-997E-5AF3244D631F> /opt/local/lib/libarchive.13.dylib
Expected in:     <77FF8999-1AB6-326B-90EE-F96C63299803> /usr/lib/liblzma.5.dylib
(terminated at launch; ignore backtrace)

Thread 0 Crashed:
0   dyld                          	    0x7ff80cd3ac52 __abort_with_payload + 10
1   dyld                          	    0x7ff80cd54fd7 abort_with_payload_wrapper_internal + 82
2   dyld                          	    0x7ff80cd55009 abort_with_payload + 9
3   dyld                          	    0x7ff80ccd98f0 dyld4::halt(char const*) + 375
4   dyld                          	    0x7ff80ccd6b71 dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*) + 4526
5   dyld                          	    0x7ff80ccd53bd start + 1805


Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x0000000002000209  rbx: 0x0000000000000000  rcx: 0x00007ff7be7db268  rdx: 0x00007ff7be7db6d0
  rdi: 0x0000000000000006  rsi: 0x0000000000000004  rbp: 0x00007ff7be7db2b0  rsp: 0x00007ff7be7db268
   r8: 0x00007ff7be7db2d0   r9: 0x0000000000000000  r10: 0x0000000000000061  r11: 0x0000000000000246
  r12: 0x0000000000000061  r13: 0x00007ff7be7db6d0  r14: 0x0000000000000004  r15: 0x0000000000000006
  rip: 0x00007ff80cd3ac52  rfl: 0x0000000000000246  cr2: 0x0000000000000000
  
Logical CPU:     0
Error Code:      0x02000209 
Trap Number:     133


Binary Images:
       0x101723000 -        0x10172efff bsdtar (*) <7d1ba33f-6a74-3323-aef7-ded132795449> /opt/local/bin/bsdtar
       0x101827000 -        0x1018b2fff libarchive.13.dylib (*) <eac1f2a3-c365-3b4b-997e-5af3244d631f> /opt/local/lib/libarchive.13.dylib
       0x1019eb000 -        0x101aeefff libiconv.2.dylib (*) <53f687f6-88a6-3f39-92b2-8a1ae884ab0d> /opt/local/lib/libiconv.2.dylib
       0x10179f000 -        0x1017bafff liblzo2.2.dylib (*) <629c6816-92b6-3323-b99b-8f98db244b4a> /opt/local/lib/liblzo2.2.dylib
       0x101b03000 -        0x101baefff libzstd.1.5.5.dylib (*) <313f9964-ad79-3129-94ca-d5e477daf7f0> /opt/local/lib/libzstd.1.5.5.dylib
       0x1017ea000 -        0x101809fff liblz4.1.9.4.dylib (*) <7206faf2-3cfd-3774-b282-ed4b384a66e1> /opt/local/lib/liblz4.1.9.4.dylib
       0x101789000 -        0x101790fff libb2.1.dylib (*) <1c9c266e-b15f-34e8-a966-7b4c5300a7f4> /opt/local/lib/libb2.1.dylib
       0x1018d3000 -        0x1018e6fff libbz2.1.0.8.dylib (*) <c596b77c-a43e-3592-85ec-94083337d7c2> /opt/local/lib/libbz2.1.0.8.dylib
       0x1018f3000 -        0x101906fff libz.1.3.dylib (*) <29a0416b-29be-3f0f-b445-092ceb18dbd3> /opt/local/lib/libz.1.3.dylib
       0x101cd4000 -        0x101daffff libxml2.2.dylib (*) <f3fa095a-e685-3a41-b435-445590ec2187> /opt/local/lib/libxml2.2.dylib
       0x10208c000 -        0x102213fff libicui18n.73.2.dylib (*) <377e14a3-41a5-341d-a134-e56bb1cf6248> /opt/local/lib/libicui18n.73.2.dylib
       0x102330000 -        0x10245bfff libicuuc.73.2.dylib (*) <5019c2d1-a418-34a7-8e7a-39b5c0490abf> /opt/local/lib/libicuuc.73.2.dylib
       0x10436d000 -        0x1061fcfff libicudata.73.2.dylib (*) <ceca8248-f42c-350e-bc32-da7d4418569e> /opt/local/lib/libicudata.73.2.dylib
    0x7ff80cccf000 -     0x7ff80cd675ef dyld (*) <3df96f32-b9c9-3566-a6b7-4daebc6d6563> /usr/lib/dyld

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=215.0M resident=0K(0%) swapped_out_or_unallocated=215.0M(100%)
Writable regions: Total=9.9M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=9.9M(100%)

                                VIRTUAL   REGION 
REGION TYPE                        SIZE    COUNT (non-coalesced) 
===========                     =======  ======= 
STACK GUARD                       54.4M        1 
Stack                             9788K        1 
VM_ALLOCATE                          8K        2 
__DATA                             328K       44 
__DATA_CONST                       583K       46 
__DATA_DIRTY                        63K       23 
__LINKEDIT                       172.8M       14 
__OBJC_RO                         66.3M        1 
__OBJC_RW                         2013K        2 
__TEXT                            42.2M       55 
dyld private memory                260K        2 
shared memory                        4K        1 
===========                     =======  ======= 
TOTAL                            348.5M      192 



-----------
Full Report
-----------

{"app_name":"bsdtar","timestamp":"2023-11-08 10:15:19.00 +0100","app_version":"","slice_uuid":"7d1ba33f-6a74-3323-aef7-ded132795449","build_version":"","platform":1,"share_with_app_devs":0,"is_first_party":1,"bug_type":"309","os_version":"macOS 13.6.1 (22G313)","roots_installed":0,"incident_id":"892BD8E3-E3BC-4DDE-999B-6FB8B14EDB06","name":"bsdtar"}
{
  "uptime" : 650000,
  "procRole" : "Unspecified",
  "version" : 2,
  "userID" : 0,
  "deployVersion" : 210,
  "modelCode" : "MacBookPro16,1",
  "coalitionID" : 20609,
  "osVersion" : {
    "train" : "macOS 13.6.1",
    "build" : "22G313",
    "releaseType" : "User"
  },
  "captureTime" : "2023-11-08 10:15:18.1561 +0100",
  "incident" : "892BD8E3-E3BC-4DDE-999B-6FB8B14EDB06",
  "pid" : 72528,
  "cpuType" : "X86-64",
  "roots_installed" : 0,
  "bug_type" : "309",
  "procLaunch" : "2023-11-08 10:15:18.1284 +0100",
  "procStartAbsTime" : 659311515649774,
  "procExitAbsTime" : 659311528306549,
  "procName" : "bsdtar",
  "procPath" : "\/opt\/local\/bin\/bsdtar",
  "parentProc" : "tclsh8.6",
  "parentPid" : 72518,
  "coalitionName" : "org.gnu.Emacs",
  "crashReporterKey" : "0B3FD7AE-CD61-E7F1-5BBA-25341183702C",
  "responsiblePid" : 1074,
  "responsibleProc" : "Emacs-x86_64-10_14",
  "codeSigningID" : "",
  "codeSigningTeamID" : "",
  "codeSigningValidationCategory" : 0,
  "codeSigningTrustLevel" : 0,
  "wakeTime" : 856,
  "bridgeVersion" : {"build":"21P1069","train":"8.1"},
  "sleepWakeUUID" : "4A798EDD-FE76-4E04-A49B-2F9F16D0F35F",
  "sip" : "enabled",
  "exception" : {"codes":"0x0000000000000000, 0x0000000000000000","rawCodes":[0,0],"type":"EXC_CRASH","signal":"SIGABRT"},
  "termination" : {"code":4,"flags":518,"namespace":"DYLD","indicator":"Symbol missing","details":["(terminated at launch; ignore backtrace)"],"reasons":["Symbol not found: _lzma_cputhreads","Referenced from: <EAC1F2A3-C365-3B4B-997E-5AF3244D631F> \/opt\/local\/lib\/libarchive.13.dylib","Expected in:     <77FF8999-1AB6-326B-90EE-F96C63299803> \/usr\/lib\/liblzma.5.dylib"]},
  "extMods" : {"caller":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"system":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"targeted":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"warnings":0},
  "faultingThread" : 0,
  "threads" : [{"triggered":true,"id":7132021,"threadState":{"r13":{"value":140702029559504},"rax":{"value":33554953},"rflags":{"value":582},"cpu":{"value":0},"r14":{"value":4},"rsi":{"value":4},"r8":{"value":140702029558480},"cr2":{"value":0},"rdx":{"value":140702029559504},"r10":{"value":97},"r9":{"value":0},"r15":{"value":6},"rbx":{"value":0},"trap":{"value":133},"err":{"value":33554953},"r11":{"value":582},"rip":{"value":140703343815762,"matchesCrashFrame":1},"rbp":{"value":140702029558448},"rsp":{"value":140702029558376},"r12":{"value":97},"rcx":{"value":140702029558376},"flavor":"x86_THREAD_STATE","rdi":{"value":6}},"frames":[{"imageOffset":441426,"symbol":"__abort_with_payload","symbolLocation":10,"imageIndex":13},{"imageOffset":548823,"symbol":"abort_with_payload_wrapper_internal","symbolLocation":82,"imageIndex":13},{"imageOffset":548873,"symbol":"abort_with_payload","symbolLocation":9,"imageIndex":13},{"imageOffset":43248,"symbol":"dyld4::halt(char const*)","symbolLocation":375,"imageIndex":13},{"imageOffset":31601,"symbol":"dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*)","symbolLocation":4526,"imageIndex":13},{"imageOffset":25533,"symbol":"start","symbolLocation":1805,"imageIndex":13}]}],
  "usedImages" : [
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4319227904,
    "size" : 49152,
    "uuid" : "7d1ba33f-6a74-3323-aef7-ded132795449",
    "path" : "\/opt\/local\/bin\/bsdtar",
    "name" : "bsdtar"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4320292864,
    "size" : 573440,
    "uuid" : "eac1f2a3-c365-3b4b-997e-5af3244d631f",
    "path" : "\/opt\/local\/lib\/libarchive.13.dylib",
    "name" : "libarchive.13.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4322144256,
    "size" : 1064960,
    "uuid" : "53f687f6-88a6-3f39-92b2-8a1ae884ab0d",
    "path" : "\/opt\/local\/lib\/libiconv.2.dylib",
    "name" : "libiconv.2.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4319735808,
    "size" : 114688,
    "uuid" : "629c6816-92b6-3323-b99b-8f98db244b4a",
    "path" : "\/opt\/local\/lib\/liblzo2.2.dylib",
    "name" : "liblzo2.2.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4323291136,
    "size" : 704512,
    "uuid" : "313f9964-ad79-3129-94ca-d5e477daf7f0",
    "path" : "\/opt\/local\/lib\/libzstd.1.5.5.dylib",
    "name" : "libzstd.1.5.5.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4320043008,
    "size" : 131072,
    "uuid" : "7206faf2-3cfd-3774-b282-ed4b384a66e1",
    "path" : "\/opt\/local\/lib\/liblz4.1.9.4.dylib",
    "name" : "liblz4.1.9.4.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4319645696,
    "size" : 32768,
    "uuid" : "1c9c266e-b15f-34e8-a966-7b4c5300a7f4",
    "path" : "\/opt\/local\/lib\/libb2.1.dylib",
    "name" : "libb2.1.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4320997376,
    "size" : 81920,
    "uuid" : "c596b77c-a43e-3592-85ec-94083337d7c2",
    "path" : "\/opt\/local\/lib\/libbz2.1.0.8.dylib",
    "name" : "libbz2.1.0.8.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4321128448,
    "size" : 81920,
    "uuid" : "29a0416b-29be-3f0f-b445-092ceb18dbd3",
    "path" : "\/opt\/local\/lib\/libz.1.3.dylib",
    "name" : "libz.1.3.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4325195776,
    "size" : 901120,
    "uuid" : "f3fa095a-e685-3a41-b435-445590ec2187",
    "path" : "\/opt\/local\/lib\/libxml2.2.dylib",
    "name" : "libxml2.2.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4329095168,
    "size" : 1605632,
    "uuid" : "377e14a3-41a5-341d-a134-e56bb1cf6248",
    "path" : "\/opt\/local\/lib\/libicui18n.73.2.dylib",
    "name" : "libicui18n.73.2.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4331864064,
    "size" : 1228800,
    "uuid" : "5019c2d1-a418-34a7-8e7a-39b5c0490abf",
    "path" : "\/opt\/local\/lib\/libicuuc.73.2.dylib",
    "name" : "libicuuc.73.2.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4365668352,
    "size" : 32047104,
    "uuid" : "ceca8248-f42c-350e-bc32-da7d4418569e",
    "path" : "\/opt\/local\/lib\/libicudata.73.2.dylib",
    "name" : "libicudata.73.2.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 140703343374336,
    "size" : 624112,
    "uuid" : "3df96f32-b9c9-3566-a6b7-4daebc6d6563",
    "path" : "\/usr\/lib\/dyld",
    "name" : "dyld"
  }
],
  "sharedCache" : {
  "base" : 140703342751744,
  "size" : 21474836480,
  "uuid" : "6bc51ea1-6f37-3cc3-907e-56f0e3ae6ab0"
},
  "vmSummary" : "ReadOnly portion of Libraries: Total=215.0M resident=0K(0%) swapped_out_or_unallocated=215.0M(100%)\nWritable regions: Total=9.9M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=9.9M(100%)\n\n                                VIRTUAL   REGION \nREGION TYPE                        SIZE    COUNT (non-coalesced) \n===========                     =======  ======= \nSTACK GUARD                       54.4M        1 \nStack                             9788K        1 \nVM_ALLOCATE                          8K        2 \n__DATA                             328K       44 \n__DATA_CONST                       583K       46 \n__DATA_DIRTY                        63K       23 \n__LINKEDIT                       172.8M       14 \n__OBJC_RO                         66.3M        1 \n__OBJC_RW                         2013K        2 \n__TEXT                            42.2M       55 \ndyld private memory                260K        2 \nshared memory                        4K        1 \n===========                     =======  ======= \nTOTAL                            348.5M      192 \n",
  "legacyInfo" : {
  "threadTriggered" : {

  }
},
  "logWritingSignature" : "20de20f8cae021e64842772c4ed38ccb02db7438",
  "trialInfo" : {
  "rollouts" : [
    {
      "rolloutId" : "645c2d2f9e69a025b0a37e29",
      "factorPackIds" : {

      },
      "deploymentId" : 240000003
    },
    {
      "rolloutId" : "5fb4245a1bbfe8005e33a1e1",
      "factorPackIds" : {

      },
      "deploymentId" : 240000021
    }
  ],
  "experiments" : [
    {
      "treatmentId" : "a092db1b-c401-44fa-9c54-518b7d69ca61",
      "experimentId" : "64a844035c85000c0f42398a",
      "deploymentId" : 400000019
    }
  ]
}
}

Model: MacBookPro16,1, BootROM 2020.41.1.0.0 (iBridge: 21.16.1069.0.0,0), 8 processors, 8-Core Intel Core i9, 2,3 GHz, 32 GB, SMC 
Graphics: Intel UHD Graphics 630, Intel UHD Graphics 630, Built-In
Graphics: AMD Radeon Pro 5500M, AMD Radeon Pro 5500M, PCIe, 8 GB
Display: Color LCD, 3072 x 1920 Retina, MirrorOff, Online
Display: PHL 329P1, 6720 x 3780, Main, MirrorOff, Online
Display: DELL U2713HM, 2560 x 1440 (QHD/WQHD - Wide Quad High Definition), MirrorOff, Online
Memory Module: BANK 0/ChannelA-DIMM0, 16 GB, DDR4, 2667 MHz, SK Hynix, HMAA2GS6CMR8K-VK
Memory Module: BANK 2/ChannelB-DIMM0, 16 GB, DDR4, 2667 MHz, SK Hynix, HMAA2GS6CMR8K-VK
AirPort: spairport_wireless_card_type_wifi (0x14E4, 0x7BF), wl0: Dec  9 2022 17:02:25 version 9.30.492.0.32.5.87 FWID 01-e7856862
Bluetooth: Version (null), 0 services, 0 devices, 0 incoming serial ports
Network Service: Wi-Fi, AirPort, en0
PCI Card: pci1b73,1100, USB eXtensible Host Controller, Thunderbolt@72,0,0
PCI Card: ethernet, Ethernet Controller, Thunderbolt@71,0,0
PCI Card: pci8086,15d4, USB eXtensible Host Controller, Thunderbolt@73,0,0
USB Device: PSSD T7 Shield
USB Device: XS2000
USB Device: USB31Bus
USB Device: USB31Bus
USB Device: USB3.2 Hub
USB Device: USB 10/100/1000 LAN
USB Device: USB2.1 Hub
USB Device: USB30Bus
USB Device: USB audio CODEC
USB Device: Keyboard Hub
USB Device: Apple Keyboard
USB Device: T2Bus
USB Device: composite_device
USB Device: Touch Bar Backlight
USB Device: Touch Bar Display
USB Device: Apple Internal Keyboard / Trackpad
USB Device: Headset
USB Device: Ambient Light Sensor
USB Device: FaceTime HD Camera (Built-in)
USB Device: Apple T2 Controller
Thunderbolt Bus: MacBook Pro, Apple Inc., 63.5
Thunderbolt Device: Thunderbolt 3 Express Dock HD, Belkin International, Inc., 3, 19.1
Thunderbolt Bus: MacBook Pro, Apple Inc., 63.5

[ryandesign (Ryan Carsten Schmidt)]

/opt/local/lib/libarchive.13.dylib should not be looking for anything in /usr/lib/liblzma.5.dylib; it should be looking in /opt/local/lib/liblzma.5.dylib, which is provided by the xz port, which is listed as a dependency of the libarchive port, which provides bsdtar, so I don't know why this is happening for you, unless you have done something with the DYLD_LIBRARY_PATH environment variable which tells dyld to use different libraries than it ordinarily would.

[ryandesign (Ryan Carsten Schmidt)]

However, it is also surprising that your MacPorts is using /opt/local/bin/bsdtar to extract ports. Normally it should use /usr/bin/tar. Did you build MacPorts base from source and explicitly tell it to use /opt/local/bin/bsdtar for extraction, or did you have libarchive installed and have /opt/local/bin in your PATH when you built MacPorts base from source? If so, that could explain it, and the recommendation then would be to reinstall the pre-compiled MacPorts base using our installer, or rebuild MacPorts base from source while /opt/local/bin is not in your PATH so that MacPorts does not configure itself to use /opt/local/bin/bsdtar or other programs installed by MacPorts ports.

[PerMildner]

As far as I can recall I have a "boring" install of MacPorts, using the ordinary installer. I have had MacPorts through many OS upgrades on this machine, but as far as I can recall I have always followed the upgrade instructions (install MacPorts for new OS, then do some uninstall-everything magic, then port install all the things I need).

One other interesting thing is that the crash does not result in any obvious failure from the sudo port upgrade outdated. I only notice the crash because macOS puts up a crash dialog.

One theory would be that the problem is more common, but most users do not have a setting that makes them see a crash dialog.

Are there any logs I could upload that would make it easier to troubleshoot this?

[tobypeterson]

• output of otool -L /opt/local/bin/bsdtar
• output of env | grep DYLD
• debug output from the port command that triggers the crash

[PerMildner]

% otool -L /opt/local/bin/bsdtar 
/opt/local/bin/bsdtar:
	/opt/local/lib/libarchive.13.dylib (compatibility version 21.0.0, current version 21.2.0)
	/opt/local/lib/libiconv.2.dylib (compatibility version 9.0.0, current version 9.1.0)
	/opt/local/lib/liblzo2.2.dylib (compatibility version 3.0.0, current version 3.0.0)
	/opt/local/lib/liblzma.5.dylib (compatibility version 10.0.0, current version 10.4.0)
	/opt/local/lib/libzstd.1.dylib (compatibility version 1.0.0, current version 1.5.5)
	/opt/local/lib/liblz4.1.dylib (compatibility version 1.0.0, current version 1.9.4)
	/opt/local/lib/libb2.1.dylib (compatibility version 2.0.0, current version 2.4.0)
	/opt/local/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.8)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.3.0)
	/opt/local/lib/libxml2.2.dylib (compatibility version 14.0.0, current version 14.5.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

env | grep DYLD has empty output.

Could you elaborate on how I obtain the "debug output from the port command that triggers the crash"? As I mentioned I have never noticed any indication, in the shell, of problem from sudo port upgrade outdated, even when the crash dialog appears. Are there some explicit logs I could look for? (Unfortunately I have done a sudo port upgrade outdated since last time, which found nothing to update and did not cause a crash dialog, so perhaps any logs have been overwritten by now.

[tobypeterson]

Linking looks correct, and I don't know of a reason it would use /usr/lib/liblzma without a DYLD environment variable set. Are you running the env command in the exact same environment you're using to run port? I'm suspicious because the "Responsible" process is... Emacs-x86_64-10_14?

For debug output, use -d, e.g. sudo port -d upgrade outdated.

[tobypeterson]

Although I suppose it's actually libarchive referencing /usr/lib/liblzma? So maybe the output of otool -L /opt/local/lib/libarchive.13.dylib will help.

[jmroot (Joshua Root)]

As far as I'm aware, /usr/bin/env is still incapable of showing you whether DYLD_* variables are set, because SIP filters them out.

[PerMildner]

% otool -L /opt/local/lib/libarchive.13.dylib
/opt/local/lib/libarchive.13.dylib:
	/opt/local/lib/libarchive.13.dylib (compatibility version 21.0.0, current version 21.2.0)
	/opt/local/lib/libiconv.2.dylib (compatibility version 9.0.0, current version 9.1.0)
	/opt/local/lib/liblzo2.2.dylib (compatibility version 3.0.0, current version 3.0.0)
	/opt/local/lib/liblzma.5.dylib (compatibility version 10.0.0, current version 10.4.0)
	/opt/local/lib/libzstd.1.dylib (compatibility version 1.0.0, current version 1.5.5)
	/opt/local/lib/liblz4.1.dylib (compatibility version 1.0.0, current version 1.9.4)
	/opt/local/lib/libb2.1.dylib (compatibility version 2.0.0, current version 2.4.0)
	/opt/local/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.8)
	/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.3.0)
	/opt/local/lib/libxml2.2.dylib (compatibility version 14.0.0, current version 14.5.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

set | grep -e DYLD (from a Terminal.app window, outside Emacs) also empty (and I have no reason to think that any of the DYLD variables would be set).

It could very well be that I ran the port upgrade from a shell inside Emacs. This is how I usually use the shell. I do not remember if the latest crash (or all previous crashes) happened from within Emacs or not.

[PerMildner]

In a shell (in Emacs or Terminal.app makes no difference):

% { find /opt/local -name '*.dylib' -exec otool -L '{}' \; | grep -e '/usr/lib/' | sort -u } 2> /dev/null 
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
	/usr/lib/libXplugin.1.dylib (compatibility version 1.0.0, current version 53.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1300.32.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1300.36.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1500.65.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1500.65.0, weak)
	/usr/lib/libc++abi.dylib (compatibility version 1.0.0, current version 1500.65.0)
	/usr/lib/libcompression.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
	/usr/lib/libedit.3.dylib (compatibility version 2.0.0, current version 3.0.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libutil.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)

No /usr/lib/liblzma... among those.

(I will be sure to use the debug output in the future, but so far port upgrade has not found anything to upgrade.)

[kencu (Ken)]

IMHO -- this is kinda fun I guess to see what exactly went wrong for you, but in the real world -- just start fresh.

Whatever is happening to you is not a systemic MacPorts issue.

Get a listing of your requested ports, uninstall all your ports, reset everything in /opt/local/etc/macports to default values, and reinstall the ports you want.

If you start fresh and the problem comes up again on your reinstallation, then ... well something is seriously hosed on your system.

[ryandesign (Ryan Carsten Schmidt)]

Replying to kencu:

Get a listing of your requested ports, uninstall all your ports, reset everything in /opt/local/etc/macports to default values, and reinstall the ports you want.

That's pretty drastic and inconvenient step, and we have no evidence yet that there's anything wrong with the ports as they are currently installed. We only have the information that, when run from within emacs, something unexpected happens, but I have no experience with emacs so I cannot say whether what the user is trying to do within emacs is supposed to work. The problem could be caused by running port within emacs or by DYLD environment variables. If so, uninstalling and reinstalling all ports wouldn't change anything. I would recommend that for further troubleshooting emacs be left out of the equation.

[ryandesign (Ryan Carsten Schmidt)]

Trying from another macOS user account could be another useful diagnostic step.

[PerMildner]

If this problem is indeed only on my machine I think reinstall-and-move-on as suggested above is the cost effective solution.

I will stay silent unless I can reproduce the problem with debug output. Most likely I will reinstall anyway because of OS upgrade to macOS 14 and the problem will either disappear, or reappear in Sonoma and therefore become more interesting to investigate.

[tobypeterson]

Any updates here? Inclined to close...

[PerMildner]

Update: This has continued to happen, also outside Emacs (in a terminal window) _also_ after having upgraded to an Apple Silicon machine.

Unfortunately I have never remembered to enable logging those times when it crashed. I did an upgrade outdated and selfupdate today with logging and it did not crash.

I do not know your policies for closing issues, but I assume I can reopen this ticket if I ever manage to get a logfile and a crash at the same time.

[tobypeterson]

I still think there must be something odd about your configuration, but I can't say what. Did you ever try this on a separate user account?

[PerMildner]

This only happens rarely, so a failure to reproduce this on another account would not say much, unfortunately.

I will continue to try to remember to enable logging when I upgrade/update, and call back if I reproduce the problem with logging.

[PerMildner]

Log from running sudo port -d upgrade outdated which crashes bsdtar

[PerMildner]

the crashlog about bsdtar

[PerMildner]

I finally created a log file when this crashes (from a Terminal.app windows, so unrelated to Emacs). I have attached it and the bsdtar crashlog message.

When the crash dialog opened I opened the log and it it had 74634 lines at the time, which perhaps can narrow down the search for culprits.

Note that there are no other apparent symptoms. In particular the install continues normally, long after the bsdtar crash, with no error messages.

Looking at the log I see several instances of what appears to be invocations of bsdtar using a relative path. I do not know whether that is relevant.

I did:

% sudo port selfupdate
--->  Checking for newer releases of MacPorts
MacPorts base version 2.10.5 installed,
MacPorts base version 2.10.7 available.
--->  MacPorts base is outdated, installing new version 2.10.7
--->  Attempting to fetch MacPorts 2.10.7 source code from https://github.com/macports/macports-base/releases/download/v2.10.7/MacPorts-2.10.7.tar.bz2
--->  Extracting MacPorts 2.10.7
--->  Installing new MacPorts release in /opt/local as root:wheel; permissions 0755
--->  Checking for newer releases of MacPorts
MacPorts base version 2.10.7 installed,
MacPorts base version 2.10.7 available.
--->  MacPorts base is already the latest version
--->  Updating the ports tree

The ports tree has been updated.

140 ports are outdated. Run 'port outdated' for details.
To upgrade your installed ports, you should run
  port upgrade outdated
% sudo port upgrade outdated

and it crashed in port upgrade outdated so I aborted it (the above was done in Emacs) so I could re-run in a terminal.

I then opened a new terminal window, and did

sudo port -d upgrade outdated >port_upgrade_outdated.log 2>&1

and it put up the crash dialog about bsdtar again.

This is MacOS 15.5, Apple Silicon, and "MacPorts base version 2.10.7".

[ryandesign (Ryan Carsten Schmidt)]

I'm sorry you've continued to see this! I'd love to get to the bottom of it.

The log shows /usr/bin/tar is being used to extract ports, which is good.

The crash log says the crash happened at 18:01:32.1954. Your computer is so fast that many things happen within that second and MacPorts logs don't have subsecond timing. (I filed #72600 to add that.) Here are the things that the log shows happened during that second:

% grep -A1 'started at .* 18:01:32' < port_upgrade_outdated.log
DEBUG: install phase started at Wed Jun 11 18:01:32 CEST 2025
--->  Installing libxml2 @2.13.8_0
--
DEBUG: clean phase started at Wed Jun 11 18:01:32 CEST 2025
--->  Cleaning libxml2
--
DEBUG: activate phase started at Wed Jun 11 18:01:32 CEST 2025
DEBUG: Executing org.macports.activate (libxml2)
--
DEBUG: deactivate phase started at Wed Jun 11 18:01:32 CEST 2025
DEBUG: elevating privileges for deactivate: euid changed to 0, egid changed to 0.
--
DEBUG: clean phase started at Wed Jun 11 18:01:32 CEST 2025
--->  Cleaning libxml2
--
DEBUG: load phase started at Wed Jun 11 18:01:32 CEST 2025
DEBUG: Executing org.macports.load (libxml2)
--
DEBUG: clean phase started at Wed Jun 11 18:01:32 CEST 2025
--->  Cleaning libxml2
--
DEBUG: archivefetch phase started at Wed Jun 11 18:01:32 CEST 2025
--->  Fetching archive for libarchive
--
DEBUG: install phase started at Wed Jun 11 18:01:32 CEST 2025
--->  Installing libarchive @3.8.1_0
--
DEBUG: clean phase started at Wed Jun 11 18:01:32 CEST 2025
--->  Cleaning libarchive
--
DEBUG: activate phase started at Wed Jun 11 18:01:32 CEST 2025
DEBUG: Executing org.macports.activate (libarchive)
--
DEBUG: deactivate phase started at Wed Jun 11 18:01:32 CEST 2025
DEBUG: elevating privileges for deactivate: euid changed to 0, egid changed to 0.
--
DEBUG: clean phase started at Wed Jun 11 18:01:32 CEST 2025
--->  Cleaning libarchive
--
DEBUG: load phase started at Wed Jun 11 18:01:32 CEST 2025
DEBUG: Executing org.macports.load (libarchive)
--
DEBUG: clean phase started at Wed Jun 11 18:01:32 CEST 2025
--->  Cleaning libarchive
--
DEBUG: archivefetch phase started at Wed Jun 11 18:01:32 CEST 2025
--->  Fetching archive for cmake

One of the things that happened was that libarchive, which provides /opt/local/bin/bsdtar, got updated. That seems relevant.

I suspect the invocation of bsdtar that causes the crash happens on these lines in MacPorts base, or one of the other places that does something similar:

​https://github.com/macports/macports-base/blob/093d55c49db68404a787b912d19853e97be57525/src/registry2.0/portimage.tcl#L494-L515

Before extracting, MacPorts base runs bsdtar to see if it supports HFS compression. The tar included with old macOS versions does not support HFS compression, but if you install the libarchive port, then all ports use its bsdtar so that the files a port installs are compressed.

I was unable to reproduce the problem by running sudo port -nb upgrade --force --no-rev-upgrade libarchive on my x86_64 macOS 12 machine. I used CrashReporterPrefs to enable showing the dialog but none appeared and no logs were generated in /Library/Logs/DiagnosticReports nor ~/Library/Logs/DiagnosticReports. If you run that command a few times, does the crash occur each time, or sometimes, or not at all?

It still doesn't seem like the problem should be happening. Activation and deactivation affect all of a port's files. Either MacPorts bsdtar and its liblzma should both exist or they both should not.

It almost feels like a variation of a filesystem cache bug we've already encountered (#67336). In this case, while activating libarchive, it seems like the MacPorts bsdtar should not be on disk anymore since the libarchive port has already been deactivated, but for some reason it still exists.

[ryandesign (Ryan Carsten Schmidt)]

I have to revise my assessment a bit after taking a closer look at the crash logs.

The crash originally reported two years ago said:

Path:                  /opt/local/bin/bsdtar
…
Termination Reason:    Namespace DYLD, Code 4 Symbol missing
Symbol not found: _lzma_cputhreads
Referenced from: <EAC1F2A3-C365-3B4B-997E-5AF3244D631F> /opt/local/lib/libarchive.13.dylib
Expected in:     <77FF8999-1AB6-326B-90EE-F96C63299803> /usr/lib/liblzma.5.dylib

which was inexplicable because /opt/local/lib/libarchive.13.dylib should not have been looking in /usr/lib/liblzma.5.dylib.

The new crash today says:

Path:                  /opt/local/bin/bsdtar
…
Termination Reason:    Namespace DYLD, Code 1 Library missing
Library not loaded: /opt/local/lib/libxml2.2.dylib
Referenced from: <18027103-810D-39DE-A1F7-F5CE526D06DD> /opt/local/bin/bsdtar
Reason: tried: '/opt/local/lib/libxml2.2.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/local/lib/libxml2.2.dylib' (no such file), '/opt/local/lib/libxml2.2.dylib' (no such file)

So MacPorts bsdtar couldn't find the MacPorts libxml2 library. So this wouldn't have occurred while updating the libarchive port, but rather while updating the libxml2 port, which did also happen for you during the second of 18:01:32.

However, I can't reproduce it with sudo port -nb upgrade --force --no-rev-upgrade libxml2. Can you?

[jmroot (Joshua Root)]

The code that uses bsdtar checks whether it works (specifically if bsdtar -x --hfsCompression < /dev/null >& /dev/null succeeds) before using it. So it's entirely possible that a crash log will be generated by this check when one of libarchive's dependencies is being upgraded, but the upgrade will still succeed (just without HFS compression being applied to the files).

[tobypeterson]

I think you're right that it's the _test_ of bsdtar's usability that crashes:

--->  Activating ccache @4.11.3_0+doc
DEBUG: Using bsdtar with HFS+ compression (if valid)

--->  Activating libxml2 @2.13.8_0
[presumably the crash is here]
DEBUG: Using /usr/bin/tar

--->  Activating libarchive @3.8.1_0
DEBUG: Using bsdtar with HFS+ compression (if valid)

[tobypeterson]

So I'm not sure this is a real issue, even if a crash log is generated.

[tobypeterson]

It's certainly not a bug in bsdtar itself - when one of its dependencies is uninstalled or deactivated, a crash is expected.

[jmroot (Joshua Root)]

I don't think it can be avoided in the general case if we want to apply HFS compression. On new enough systems we could use /usr/bin/bsdtar, but older ones either don't support HFS compression or have serious bugs with it.

[PerMildner]

If I understand correctly, the installer tests whether bsdtar works (with some options). If it crashes (and brings up the crash dialog) the installer just treats its as the somewhat expected "does not work", which would explain why everything seems to run fine even when the crash dialog appears.

The reason for testing bsdtar is "MacPorts base runs bsdtar to see if it supports HFS compression. The tar included with old macOS versions does not support HFS compression"

But "old macOS version" would be /usr/bin/bsdtar, and could never(?) be /opt/local/bin/bsdtar, and since /opt/local/bin/bsdtar is what crashes, it seems the installer tests the wrong bsdtar.

So, hand waving, one theory is that the tests (that uses PATH lookup to find bsdtar) should either ensure they really test /usr/bin/bsdtar (to see whether it is old) or use dependency information to know whether /opt/local/bsdtar is usable.

I.e., if the above is correct, the behavior sounds like a bug, and fixable. (otherwise it may be possible to suppress the crash dialog by running with lldb --batch --one-line ... but that seems too brittle).

[tobypeterson]

Updated title, changed component to base.

[ryandesign (Ryan Carsten Schmidt)]

When we refer to "the installer", we usually mean the pkg files that install a pre-compiled binary of MacPorts base. That's not the part that detects bsdtar's capabilities. MacPorts base itself tests those capabilities every time it tries to install a port.

It tests bsdtar, using the path, so that if the user has MacPorts libarchive installed it will test that bsdtar, and otherwise it will test the macOS bsdtar.

[PerMildner]

Replying to ryandesign:

When we refer to "the installer", we usually mean the pkg files that install a pre-compiled binary of MacPorts base. That's not the part that detects bsdtar's capabilities. MacPorts base itself tests those capabilities every time it tries to install a port.

It tests bsdtar, using the path, so that if the user has MacPorts libarchive installed it will test that bsdtar, and otherwise it will test the macOS bsdtar.

My point was that, presumably, there is no need to test the bsdtar that "MacPorts libarchive installed" (if any). The test is, if I understand correctly, a way to detect old versions of /usr/bin/bsdtar (and those would never crash during the test, regardless of the state of MacPorts). So, from I read above, it seems the right test would test /usr/bin/bsdtar _only_ which would both test the right executable _and_ fix the crash dialog.

But note that I have no idea how the MacPorts base works or is supposed to work. I am just going by the information in this thread.

[jmroot (Joshua Root)]

The use of /opt/local/bin/bsdtar is deliberate, since the system bsdtar doesn't support HFS compression on older systems.

[PerMildner]

Replying to jmroot:

The use of /opt/local/bin/bsdtar is deliberate, since the system bsdtar doesn't support HFS compression on older systems.

Yes but the problem is not "The use of /opt/local/bin/bsdtar". The problem is the _test_ of /opt/local/bin/bsdtar, which apparently is done in a way that can cause a crash dialog. My point, and my understanding, is that the _test_ is only done in other to determine whether "the system bsdtar doesn't support HFS compression", i.e. the _test_ should only test "the system bsdtar" (/usr/bin/bsdtar) the test should _not_ test /opt/local/bin/bsdtar, so the test should not test whatever it happens to find on PATH (e.g. /opt/local/bin/bsdtar) but should instead explicitly test the system bsdtar.

[jmroot (Joshua Root)]

In the circumstances where the check crashes, using /opt/local/bin/bsdtar without checking whether it works first would mean it crashes when extracting the archive, causing activation of the port to actually fail.

[ryandesign (Ryan Carsten Schmidt)]

If it were very important to avoid the crash during the check, we could first look at bsdtar's list of libraries and see if they all exist. We already have code to do some of this in rev-upgrade; maybe that code can be reused here.

[PerMildner]

About "we could first look at bsdtar's list of libraries": Since in my scenario, the tested bsdtar (/opt/local/bin/bsdtar) crashes, adding a test for its list of libraries would presumably conclude that the tested bsdtar is unusable. This seems undesirable, since, in the common case where /usr/bin/bsdtar is new enough to support HFS compression, this sounds like MacPorts would needlessly decide that HFS compression is unsupported. If MacPorts instead tested /usr/bin/bsdtar, which I assume would never crash, it would discover that it _does_ support HFS compression (if /usr/bin/bsdtar does not support HFS compression MacPorts could either fall back to testing other locations, like /opt/local/bin/bsdtar, or just fall back to a somewhat less optimal format on those old platforms

PS. It is not clear to me whether "HFS+ compression" is relevant on modern, i.e. APFS, file systems but #60230 indicates that perhaps it is (user had problem with that option on an APFS system, running as non-root).

[jmroot (Joshua Root)]

Yes, HFS compression is relevant on APFS. Yes, it would be possible to check /usr/bin/bsdtar as well on sufficiently new OS versions.