Opened 2 years ago
Last modified 4 weeks ago
#66358 reopened defect
sip-workaround / trace mode no longer works on arm64 macOS ≥ 13 due to new security features
Reported by: | reneeotten (Renee Otten) | Owned by: | Clemens Lang <neverpanic@…> |
---|---|---|---|
Priority: | Normal | Milestone: | |
Component: | base | Version: | |
Keywords: | arm64 ventura sonoma sequoia | Cc: | linuxgemini (İlteriş Eroğlu), neverpanic (Clemens Lang), fracai, kurthindenburg (Kurt Hindenburg), mascguy (Christopher Nielsen), jrabinow, fhgwright (Fred Wright), markmentovai (Mark Mentovai), Dave-Allured (Dave Allured), markemer (Mark Anderson), cooljeanius (Eric Gallager) |
Port: |
Description (last modified by reneeotten (Renee Otten))
Trace-mode breaks the extract
phase under Ventura (never seen this on my previous OS, Catalina). As an example I've attached the main.log
when running the command sudo port -dvt extract py310-setuptools
. Anyone else seeing this issue or is it something specific to my installation?
[as an aside: is there an indication when there will be a buildbot for Ventura available?]
Attachments (2)
Change History (72)
Changed 2 years ago by reneeotten (Renee Otten)
comment:1 Changed 2 years ago by reneeotten (Renee Otten)
Description: | modified (diff) |
---|
comment:2 Changed 2 years ago by ryandesign (Ryan Carsten Schmidt)
comment:3 Changed 2 years ago by kencu (Ken)
if anyone knows a workaround for this, I'd like to hear it.
As much as possible, I try to use trace mode to verify builds, but this makes it difficult.
Extracting normally and then building in trace mode fails.
comment:4 Changed 2 years ago by ryandesign (Ryan Carsten Schmidt)
comment:5 follow-ups: 6 7 Changed 2 years ago by ryandesign (Ryan Carsten Schmidt)
In #66421 it was suggested to run xcodebuild -runFirstLaunch
and that appeared to solve it. Please confirm if that works for you too.
comment:6 Changed 2 years ago by jmroot (Joshua Root)
Replying to ryandesign:
In #66421 it was suggested to run
xcodebuild -runFirstLaunch
and that appeared to solve it.
AIUI, that fixed another error that that user was getting after they stopped using -t.
comment:7 Changed 2 years ago by linuxgemini (İlteriş Eroğlu)
Replying to ryandesign:
In #66421 it was suggested to run
xcodebuild -runFirstLaunch
and that appeared to solve it. Please confirm if that works for you too.
This did not fix the issue for me, even after a full uninstall and reinstall of MacPorts didn't help.
comment:9 Changed 23 months ago by linuxgemini (İlteriş Eroğlu)
After looking at system console logs, I have noticed that SIP is kicking in to block sandbox-exec:
default 11:02:43.231827+0300 kernel AMFI: Launch Constraint Violation (enforcing), error info: c[1]p[1]m[1]e[3], (Constraint not matched) launching proc[vc: 1 pid: 38987]: /opt/local/var/macports/sip-workaround/503/usr/bin/sandbox-exec, launch type 0, failure proc [vc: 1 pid: 38987]: /opt/local/var/macports/sip-workaround/503/usr/bin/sandbox-exec default 11:02:43.231920+0300 kernel ASP: Security policy would not allow process: 38987, /opt/local/var/macports/sip-workaround/503/usr/bin/sandbox-exec default 11:02:43.238965+0300 tccd AUTHREQ_ATTRIBUTION: msgID=180.536, attribution={responsible={TCCDProcess: identifier=com.googlecode.iterm2, pid=38987, auid=502, euid=503, responsible_path=/Applications/iTerm.app/Contents/MacOS/iTerm2, binary_path=/opt/local/var/macports/sip-workaround/503/usr/bin/sandbox-exec}, accessing={TCCDProcess: identifier=com.apple.sandbox-exec, pid=38987, auid=502, euid=503, binary_path=/opt/local/var/macports/sip-workaround/503/usr/bin/sandbox-exec}, requesting={TCCDProcess: identifier=com.apple.syspolicyd, pid=180, auid=0, euid=0, binary_path=/usr/libexec/syspolicyd}, }, default 11:02:43.261656+0300 kernel sandbox-exec[38987] Corpse allowed 1 of 5 default 11:02:46.863394+0300 ReportCrash Formulating fatal 309 report for corpse[38987] sandbox-exec default 11:02:46.869412+0300 osanalyticshelper creating type 309 as /Users/ilteris.eroglu/Library/Logs/DiagnosticReports/.sandbox-exec-2023-02-03-110246.ips default 11:02:47.033565+0300 osanalyticshelper Saved type '309(<private>)' report (10 of max 25) at /Users/ilteris.eroglu/Library/Logs/DiagnosticReports/sandbox-exec-2023-02-03-110246.ips default 11:02:47.033731+0300 osanalyticshelper xpc log creation type 309 result success: /Users/ilteris.eroglu/Library/Logs/DiagnosticReports/sandbox-exec-2023-02-03-110246.ips default 11:02:47.033818+0300 ReportCrash client log create type 309 result success: /Users/ilteris.eroglu/Library/Logs/DiagnosticReports/sandbox-exec-2023-02-03-110246.ips default 11:02:47.033899+0300 ReportCrash no MetricKit for process sandbox-exec type 309 bundleId (null) default 11:02:47.034039+0300 ReportCrash Sending event: com.apple.stability.crash {"coalitionName":"com.googlecode.iterm2","exceptionCodes":"0x0000000000000000, 0x0000000000000000(\n 0,\n 0\n)EXC_CRASHSIGKILL (Code Signature Invalid)","incidentID":"CDC1A1CF-829A-48C5-9BD2-6C8D471F2D6A","logwritten":1,"process":"sandbox-exec","terminationReasonExceptionCode":"0x4","terminationReasonNamespace":"CODESIGNING"} default 11:02:47.034143+0300 analyticsd Received event: com.apple.stability.crash {"coalitionName":"com.googlecode.iterm2","exceptionCodes":"0x0000000000000000, 0x0000000000000000(\n 0,\n 0\n)EXC_CRASHSIGKILL (Code Signature Invalid)","incidentID":"CDC1A1CF-829A-48C5-9BD2-6C8D471F2D6A","logwritten":1,"process":"sandbox-exec","terminationReasonExceptionCode":"0x4","terminationReasonNamespace":"CODESIGNING"} default 11:02:47.034575+0300 analyticsd Aggregated. Transform: StabilityCrashNumerator3WithBundleVersion Dirty: 1 Event: com.apple.stability.crash {"coalitionName":"com.googlecode.iterm2","exceptionCodes":"0x0000000000000000, 0x0000000000000000(\n 0,\n 0\n)EXC_CRASHSIGKILL (Code Signature Invalid)","incidentID":"CDC1A1CF-829A-48C5-9BD2-6C8D471F2D6A","logwritten":1,"process":"sandbox-exec","terminationReasonExceptionCode":"0x4","terminationReasonNamespace":"CODESIGNING","timestamp":1675411367033982} default 11:02:47.035773+0300 analyticsd Aggregated. Transform: StabilityCrashNumerator3WithIncidentID Dirty: 1 Event: com.apple.stability.crash {"coalitionName":"com.googlecode.iterm2","exceptionCodes":"0x0000000000000000, 0x0000000000000000(\n 0,\n 0\n)EXC_CRASHSIGKILL (Code Signature Invalid)","incidentID":"CDC1A1CF-829A-48C5-9BD2-6C8D471F2D6A","logwritten":1,"process":"sandbox-exec","terminationReasonExceptionCode":"0x4","terminationReasonNamespace":"CODESIGNING","timestamp":1675411367033982} default 11:02:47.035988+0300 analyticsd Aggregated. Transform: StabilityCrashNumerator3 Dirty: 1 Event: com.apple.stability.crash {"coalitionName":"com.googlecode.iterm2","exceptionCodes":"0x0000000000000000, 0x0000000000000000(\n 0,\n 0\n)EXC_CRASHSIGKILL (Code Signature Invalid)","incidentID":"CDC1A1CF-829A-48C5-9BD2-6C8D471F2D6A","logwritten":1,"process":"sandbox-exec","terminationReasonExceptionCode":"0x4","terminationReasonNamespace":"CODESIGNING","timestamp":1675411367033982}
Looks like there is an entitlement issue, not quite sure.
comment:10 Changed 23 months ago by linuxgemini (İlteriş Eroğlu)
Cc: | linuxgemini added |
---|
comment:11 Changed 23 months ago by linuxgemini (İlteriş Eroğlu)
I have been informed by a friend that this is caused by a new feature at AppleMobileFileIntegrity called Launch Constraints(1) which basically blocks Apple system binaries under the sip-workaround
folder(2).
(2): https://theevilbit.github.io/posts/amfi_launch_constraints/
comment:12 Changed 22 months ago by kencu (Ken)
perhaps then the process of copying the system binaries and running the copies to avoid SIP might need to be revisited:
comment:13 Changed 22 months ago by kencu (Ken)
Cc: | neverpanic added |
---|---|
Summary: | extract phase fails in trace-mode on macOS 13 → sip-workaround no longer works on macOS 13 Ventura due to new security features |
might as well bump this up as it looks like it is going to need some thinking from the team
comment:14 Changed 22 months ago by neverpanic (Clemens Lang)
Chances are this happens because the binary has a signature from Apple. I'm guessing we will end up having to strip the Apple signature when copying the binary, and then run the unsigned copy. This, of course, comes with the risk of breaking some binaries that would otherwise have required entitlements, so we probably have to skip the copy for binaries with entitlements and just run the original binary at the expense of not having trace mode work on those binaries.
Could anybody test this for me on a Ventura system by copying a binary from /usr/bin to a different place, re-signing it using codesign -f -i -, and then running it?
Trace mode also does not work on aarch64 Monterey either, although I don't know why. It doesn't cause failures, but also does not work as expected.
comment:15 Changed 22 months ago by kencu (Ken)
well you can't run the binary if you move it, that is for sure:
cp /usr/bin/make ./make % /usr/bin/make --version GNU Make 3.81 Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. This program built for i386-apple-darwin11.3.0 % ./make --version zsh: killed ./make --version
but I can't (so far) figure out how to codesign it:
1008 codesign -f -i - make 1009 codesign -f -i make 1010 codesign -f -i make 1011 man codesign 1012 codesign -f make 1013 codesign -d make
nothing works (yet).
comment:16 Changed 22 months ago by kencu (Ken)
% codesign -f -i - ./make Usage: codesign -s identity [-fv*] [-o flags] [-r reqs] [-i ident] path ... # sign codesign -v [-v*] [-R=<req string>|-R <req file path>] path|[+]pid ... # verify codesign -d [options] path ... # display contents codesign -h pid ... # display hosting paths
comment:17 Changed 22 months ago by neverpanic (Clemens Lang)
Sorry, should have checked the manpage. It's codesign -s - -f $path
.
comment:18 Changed 22 months ago by kencu (Ken)
thanks.
no happiness yet:
% codesign -s - -f ./make ./make: replacing existing signature % ./make zsh: killed ./make
codesign -s - -f /Users/cunningh/kentest/make /Users/cunningh/kentest/make: replacing existing signature % ./make --version zsh: killed ./make --version % /Users/cunningh/kentest/make --version zsh: killed /Users/cunningh/kentest/make --version
comment:19 Changed 22 months ago by neverpanic (Clemens Lang)
Ugh, so Apple thought of some other mechanism to identify what binaries were produced by them and shouldn't be run from copies.
Until somebody figures out how they did it, we're stuck. Maybe it's a new stat flag, an entitlement, something in the resource fork, or some other metadata? It could be a database of hashes, although the signature modification should have bypassed that already.
comment:20 Changed 22 months ago by linuxgemini (İlteriş Eroğlu)
If I preserve the entitlements already present and sign with my Apple Development certificate, it works:
~ ❯ /usr/bin/make --version GNU Make 3.81 Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. This program built for i386-apple-darwin11.3.0 ~ ❯ ./make --version [1] 39270 killed ./make --version ~ ❯ codesign --preserve-metadata=entitlements --force --verbose --sign "Apple Development" ./make ./make: replacing existing signature ./make: signed Mach-O universal (x86_64 arm64e) [com.apple.dt.xcode_select.tool-shim] ~ ❯ ./make --version GNU Make 3.81 Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. This program built for i386-apple-darwin11.3.0
comment:21 Changed 21 months ago by kurthindenburg (Kurt Hindenburg)
In case this is helpful https://mjtsai.com/blog/2023/03/16/ventura-adds-com-apple-provenance/
comment:22 Changed 21 months ago by neverpanic (Clemens Lang)
Hm, can somebody check on Ventura or later whether a copy of a binary from /usr/bin has this xattr, and if it does, whether it can be removed, and if it can be removed, whether removing it allows the copied binary to run on Ventura? I'd be surprised if that worked, honestly, but it's worth checking.
comment:23 Changed 21 months ago by fracai
Cc: | fracai added |
---|
comment:24 Changed 21 months ago by kencu (Ken)
if it is there, it is hidden in some way:
cp /usr/bin/make ./ xattr -l ./make <nothing> % xattr -px com.apple.provenance ./make xattr: ./make: No such xattr: com.apple.provenance
comment:25 Changed 21 months ago by kencu (Ken)
so I tried making a new codesigning certificate using this as a template:
https://github.com/llvm/llvm-project/blob/main/lldb/scripts/macos-setup-codesign.sh
and renamed it slightly to macports_codesign instead of lldb_codesign. The certificate is generated and is there:
% ./macports-codesign.sh Certificate has already been generated and installed
but no bueno:
% rm make % cp /usr/bin/make ./ % codesign --preserve-metadata=entitlements --force --verbose --sign "macports_codesign" ./make ./make: replacing existing signature ./make: signed Mach-O universal (x86_64 arm64e) [com.apple.dt.xcode_select.tool-shim] % ./make --version zsh: killed ./make --version
Changed 21 months ago by kencu (Ken)
Attachment: | macports-codesign.sh added |
---|
my slightly edited copy of lldb's codesign-certificate-generatings-script to generate a macports signing certificate
comment:26 Changed 21 months ago by kurthindenburg (Kurt Hindenburg)
Cc: | kurthindenburg added |
---|
comment:27 Changed 20 months ago by neverpanic (Clemens Lang)
The issue is in dyld. Apple has added new restrictions that depend on the result of a syscall called amfi_check_dyld_policy_self()
. See
- https://github.com/apple-oss-distributions/dyld/blob/c8a445f88f9fc1713db34674e79b00e30723e79d/dyld/DyldProcessConfig.cpp#L672-L683
- https://github.com/apple-oss-distributions/dyld/blob/c8a445f88f9fc1713db34674e79b00e30723e79d/dyld/DyldProcessConfig.cpp#L702-L724
- https://github.com/apple-oss-distributions/dyld/blob/c8a445f88f9fc1713db34674e79b00e30723e79d/dyld/DyldDelegates.cpp#L120-L142
- https://github.com/apple-oss-distributions/dyld/blob/c8a445f88f9fc1713db34674e79b00e30723e79d/dyld/glue.c#L971-L977
The only two inputs into this syscall are whether the process is restricted or fairPlayEncrypted. Fair Play encryption would be visible as a load command LC_ENCRYPTION_INFO
or LC_ENCRYPTION_INFO_64
, which isn't present on the binaries in /usr/bin as far as I can see. Restriction is enabled if the binary has a __RESTRICT,__restrict
section according to https://github.com/apple-oss-distributions/dyld/blob/c8a445f88f9fc1713db34674e79b00e30723e79d/common/MachOFile.cpp#L1588-L1598. A quick check in /usr/bin suggests that only the fontrestore
binary has such a section, so it's also not the reason for our problems.
However, since the system call is being performed from within the process in question, the kernel probably knows what binary is being executed, so we must also treat the process and its executable file as inputs, and I believe that's where the magic happens: If I try to run a copy of make, it is immediately killed:
$ cd /tmp $ cp /usr/bin/make make $ ./make Killed: 9
If I, however, remove the code signature on that binary using codesign --remove-signature
, it runs and DYLD_*
variables seem to work:
$ codesign --remove-signature make $ ./make make: *** No targets specified and no makefile found. Stop. $ DYLD_INSERT_LIBRARIES=foo ./make dyld[58717]: terminating because inserted dylib 'foo' could not be loaded: tried: 'foo' (no such file), '/System/Volumes/Preboot/Cryptexes/OSfoo' (no such file), 'foo' (no such file), '/private/tmp/foo' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/private/tmp/foo' (no such file), '/private/tmp/foo' (no such file) dyld[58717]: tried: 'foo' (no such file), '/System/Volumes/Preboot/Cryptexes/OSfoo' (no such file), 'foo' (no such file), '/private/tmp/foo' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/private/tmp/foo' (no such file), '/private/tmp/foo' (no such file) Abort trap: 6
It seems to me we need to modify sip_copy_proc
to strip Apple's signature from those binaries.
Signing the binaries with an ad-hoc signature also seems to work, but that's probably a lot more complicated to implement (although we may have to for arm64).
Additionally, those binaries use the arm64e architecture, so we'll probably also have to start building an arm64e slice of darwintrace.dylib.
comment:28 Changed 20 months ago by neverpanic (Clemens Lang)
Just for completeness, there are also solutions out there that use an Endpoint Security entitlement to interpose processes before they are started to patch the amfi_check_dyld_policy_self
syscall from the binary before the loader has a chance to call it (see https://gist.github.com/saagarjha/a70d44951cb72f82efee3317d80ac07f), but that probably requires disabling SIP. If all else fails, that's probably something we could do on a copy of the binaries from /usr/bin, too, although it would look a bit different for us since we wouldn't be patching in memory but essentially doing what would be called a GOT hijack on Linux.
comment:29 Changed 20 months ago by neverpanic (Clemens Lang)
For those where removing the signature, or replacing it with an ad-hoc signature did not work, can you check whether the output of sysctl security | grep launch_constraints
suggests that launch constraints disallow 3rd party binaries?
comment:30 Changed 18 months ago by mascguy (Christopher Nielsen)
Cc: | mascguy added |
---|
comment:31 Changed 18 months ago by jrabinow
macOS 13.3 22E252 arm64 Xcode 14.3 14E222b
As far as I can tell, amfi_check_dyld_policy_self
isn't being issued as per
- /usr/bin/syscallbypid.d
- /usr/bin/syscallbyproc.d
- /usr/bin/syscallbysysc.d
and no new process is being executed as per /usr/bin/newproc.d
. execsnoop
does see make
but when I tried tracing it with dtruss -W
I got failed to grab pid 19387: (null)
which makes me think it's too short-lived and/or doesn't get initalized properly. So somewhere between execsnoop and newproc.d the process gets killed off.
I tried to strip the signature from make as described above, it's still getting killed:
$ codesign --remove-signature make $ codesign -vv make make: code object is not signed at all In architecture: arm64e [Exit 1] $ ./make Killed: 9 [Exit 137 (KILL)]
I also tried signing make with my own custom signature and it still didn't launch.
Moving onto provenance:
I can see the provenance on make. Worth noting it didn't appear immediately (less than 1h of playing around with it, digging into this and a few other things). When I tried checking and copying a make2
to tmp, the provenance appeared right away though, so go figure if I didn't mess up somehow.
$ /bin/ls -l@ /tmp/make -rwxr-xr-x@ 1 julien wheel 132017 Jul 4 16:09 /tmp/make com.apple.provenance 11 $ xattr -px com.apple.provenance ./make 01 00 00 5D 19 F7 5D 19 6C 9E A9
From: https://eclecticlight.co/2023/03/13/ventura-has-changed-app-quarantine-with-a-new-xattr/
The xattr is protected by SIP, so can’t be removed unless the app is copied to another volume.
I haven't tried removing it.
Also for what it's worth, I asked chatgpt for a summary of what are cdhashes and checked it against apple's docs (links below). It had this to say:
A CDHash, short for Code Directory Hash, is a cryptographic hash value associated with a binary executable file on macOS systems. It is used as a security measure to verify the integrity and authenticity of the code before it is executed. When a binary is signed with an Apple Developer certificate, the signing process generates a CDHash for the code. This hash is derived from the code directory, which contains a list of the file offsets and sizes for each segment of the executable. By calculating the CDHash, the system can ensure that the code has not been tampered with or modified since it was signed. The CDHash is stored in the code signature of the binary and is checked by the Gatekeeper service on macOS when an application is launched. If the CDHash does not match the expected value, indicating that the binary has been altered, Gatekeeper will prevent the application from running, providing an additional layer of security against malicious software. It's worth noting that the CDHash is just one component of the code signing process on macOS, which includes other checks.
Also see:
- https://developer.apple.com/documentation/endpointsecurity/es_process_t/3228976-cdhash
- https://developer.apple.com/documentation/technotes/tn3126-inside-code-signing-hashes
If this is what is causing issues, I'd assume the following approximate setup:
- the provenance is stored in the binary's extended attributes and can't be removed because SIP.
- this provenance points to an entry in the ExecPolicy database in provenance_tracking table. I tried finding it but couldn't.
- The execpolicy database would allow the OS (going by opensnoop's output, most likely syspolicyd) to compare the calculated cdhash against the cdhash stored in the provenance_tracking table.
- The cdhash is used to seal the binary. If the calculated values don't match the cdhash, the OS (again, most likely syspolicyd) sends kill -9
Worth noting that I tried finding make in /var/db/SystemPolicyConfiguration/ExecPolicy provenance_tracking table but couldn't find anything that seemed relevant. I rebooted my computer in case the db was only flushed to disk on process restart, still no dice. I didn't see anything relating to /usr/bin in there either, so I'm still missing something and I'm not sure if this ExecPolicy database is relevant. Maybe provenance is used by something else? Maybe, as chatgpt suggests, it's another mechanism entirely?
I'd suggest the way forward on this one would be to open up execsnoop script, repurpose it to have it attach to a new process named make
, gatekeeper
(assuming it runs in its own process) or syspolicyd
and have it observe everything that happens to it. I've barely looked at gatekeeper, and https://eclecticlight.co/2023/03/16/what-is-macos-ventura-doing-tracking-provenance/ suggests to me that gatekeeper might run even before syspolicyd gets involved at all.
Also I'd be curious if anyone can repro the copying of a binary out of /usr/bin without the 'provenance' xattr, like I initially got and like kencu's comment above suggests. Can make run at that time? If you wait for some time, does the provenance appear on its own? If so, does the binary start getting killed then?
Question from a complete newbie, why do we need to use Apple's binaries at all? Could macports not ship its own binaries and bootstrap from there, reminescent of the gcc bootstrapping process?
comment:32 Changed 18 months ago by jrabinow
Cc: | jrabinow added |
---|
comment:33 Changed 18 months ago by jrabinow
I forgot:
$ sysctl security | grep launch_constraints security.mac.amfi.launch_constraints_enforced: 1 security.mac.amfi.launch_constraints_3rd_party_allowed: 1
so 3rd-party sources doesn't seem to be related either.
comment:34 Changed 18 months ago by neverpanic (Clemens Lang)
This is all very weird. On the one machine that I have upgraded to Ventura so far (which is x86_64), I implemented a change that re-signs all binaries before executing them with an ad-hoc signature, and trace mode works just fine with that.
I cannot reproduce the killed process when re-signing with an ad-hoc signature. Did Apple just implement additional security for arm64 machines, and left x86_64 with less security?
On the other hand, there were people above who reported getting killed binaries on x86_64, which I never saw. I don't know what's going on. I can try pushing the re-signing code for somebody else to test, if that helps?
comment:35 Changed 18 months ago by linuxgemini (İlteriş Eroğlu)
I'm the initial reporter, which the issue happens on an x86_64 machine.
Haven't checked the trace mode for a while; I'll check later tomorrow.
comment:36 Changed 18 months ago by linuxgemini (İlteriş Eroğlu)
It is tomorrow, so I tried trace mode again with proxmark3-iceman
; still no dice:
:info:extract Executing: cd "/opt/local/var/macports/build/_Users_ilteris.eroglu_Projects_macports-ports_science_proxmark3-iceman/proxmark3-iceman/work" && /usr/bin/gzip -dc '/opt/local/var/macports/distfiles/proxmark3-iceman/proxmark3-4.16717.tar.gz' | /usr/bin/tar -xf - :debug:extract system: cd "/opt/local/var/macports/build/_Users_ilteris.eroglu_Projects_macports-ports_science_proxmark3-iceman/proxmark3-iceman/work" && /usr/bin/gzip -dc '/opt/local/var/macports/distfiles/proxmark3-iceman/proxmark3-4.16717.tar.gz' | /usr/bin/tar -xf - :info:extract Command failed: cd "/opt/local/var/macports/build/_Users_ilteris.eroglu_Projects_macports-ports_science_proxmark3-iceman/proxmark3-iceman/work" && /usr/bin/gzip -dc '/opt/local/var/macports/distfiles/proxmark3-iceman/proxmark3-4.16717.tar.gz' | /usr/bin/tar -xf - :info:extract Killed by signal: 9 :error:extract Failed to extract proxmark3-iceman: command execution failed :debug:extract Error code: NONE :debug:extract Backtrace: command execution failed :debug:extract while executing :debug:extract "$procedure $targetname" :error:extract See /opt/local/var/macports/logs/_Users_ilteris.eroglu_Projects_macports-ports_science_proxmark3-iceman/proxmark3-iceman/main.log for details.
default 09:56:59.075368+0300 kernel AMFI: Launch Constraint Violation (enforcing), error info: c[1]p[1]m[1]e[3], (Constraint not matched) launching proc[vc: 1 pid: 80195]: /opt/local/var/macports/sip-workaround/503/usr/bin/sandbox-exec, launch type 0, failure proc [vc: 1 pid: 80195]: /opt/local/var/macports/sip-workaround/503/usr/bin/sandbox-exec default 09:56:59.075443+0300 kernel ASP: Security policy would not allow process: 80195, /opt/local/var/macports/sip-workaround/503/usr/bin/sandbox-exec default 09:56:59.075544+0300 kernel sandbox-exec[80195] Corpse allowed 1 of 5 default 09:56:59.093028+0300 ReportCrash Formulating fatal 309 report for corpse[80195] sandbox-exec default 09:56:59.113785+0300 osanalyticshelper creating type 309 as /Users/ilteris.eroglu/Library/Logs/DiagnosticReports/.sandbox-exec-2023-07-07-095659.ips default 09:56:59.138890+0300 osanalyticshelper Saved type '309(<private>)' report (2 of max 25) at /Users/ilteris.eroglu/Library/Logs/DiagnosticReports/sandbox-exec-2023-07-07-095659.ips default 09:56:59.139312+0300 ReportCrash client log create type 309 result success: /Users/ilteris.eroglu/Library/Logs/DiagnosticReports/sandbox-exec-2023-07-07-095659.ips default 09:56:59.139370+0300 ReportCrash no MetricKit for process sandbox-exec type 309 bundleId (null) default 09:56:59.139173+0300 osanalyticshelper xpc log creation type 309 result success: /Users/ilteris.eroglu/Library/Logs/DiagnosticReports/sandbox-exec-2023-07-07-095659.ips
Using an ad-hoc certificate to re-sign still works:
~ ❯ cp /usr/bin/make ./make ~ ❯ ~ ❯ /usr/bin/make --version GNU Make 3.81 Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. This program built for i386-apple-darwin11.3.0 ~ ❯ ~ ❯ ./make --version [1] 80407 killed ./make --version ~ ❯ ~ ❯ codesign --preserve-metadata=entitlements --force --verbose --sign "test_codesign" ./make ./make: replacing existing signature ./make: signed Mach-O universal (x86_64 arm64e) [com.apple.dt.xcode_select.tool-shim] ~ ❯ ~ ❯ ./make --version GNU Make 3.81 Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. This program built for i386-apple-darwin11.3.0 ~ ❯
This is my work Mac, so SIP is always enabled and there's multiple EDR and DLP solutions installed.
comment:37 Changed 18 months ago by jrabinow
I don't know what's going on. I can try pushing the re-signing code for somebody else to test, if that helps?
Assuming I have step-by-step instructions to follow, I'd love to test on my M1 arm64. Also of note is some parts of my SIP are disabled.
comment:39 Changed 15 months ago by neverpanic (Clemens Lang)
On arm64, apple uses a different ABI, arm64e, which includes pointer authentication. The macOS kernel refuses to run arm64e binaries unless they are signed by Apple. A workaround is to run
sudo nvram boot-args=-arm64e_preview_abi
which disables this check. I think what happens here is twofold: On x86_64, resigning the binaries is sufficient, but on arm64, this doesn't work, and the additional boot argument may be required as well.
comment:40 Changed 15 months ago by neverpanic (Clemens Lang)
On x86_64, I'd welcome feedback for https://github.com/neverpanic/macports-base/commits/cal-tracemode-optimizations. I'm assuming those work, since they do for me.
On arm64, it seems one cannot enable the arm64e preview ABI without disabling system integrity protection, which is unfortunate. If somebody wants to test the branch above with that change, that would be helpful.
comment:41 Changed 15 months ago by Clemens Lang <neverpanic@…>
Owner: | set to Clemens Lang <neverpanic@…> |
---|---|
Resolution: | → fixed |
Status: | new → closed |
comment:42 Changed 15 months ago by neverpanic (Clemens Lang)
comment:43 Changed 15 months ago by neverpanic (Clemens Lang)
Resolution: | fixed |
---|---|
Status: | closed → reopened |
Summary: | sip-workaround no longer works on macOS 13 Ventura due to new security features → sip-workaround no longer works on arm64 macOS 13 Ventura due to new security features |
Re-opening since the problem still exists on arm64.
comment:44 follow-up: 45 Changed 14 months ago by mrdomino (Jōshin)
Sorry if this is the wrong place, but I haven't found answers anywhere else and I wound up here, so asking for posterity: is this the root cause for there not being any port builders for Sonoma arm64?
If so, this seems like a rather severe issue, and the verbiage should be clearer about it in e.g. SonomaProblems and Migration. I didn't find this out until after upgrading, and it's at times very painful to not have binary packages.
comment:45 Changed 14 months ago by reneeotten (Renee Otten)
Replying to mrdomino:
Sorry if this is the wrong place, but I haven't found answers anywhere else and I wound up here, so asking for posterity: is this the root cause for there not being any port builders for Sonoma arm64?
If so, this seems like a rather severe issue, and the verbiage should be clearer about it in e.g. SonomaProblems and Migration. I didn't find this out until after upgrading, and it's at times very painful to not have binary packages.
No, this is about trace-mode not working with the released version of MacPorts - it has nothing to do with the fact that there is no buildbot yet for Sonoma. Please do not hijack tickets for unrelated comments.
comment:46 follow-up: 47 Changed 14 months ago by mrdomino (Jōshin)
Ok, sorry. Where is the right place for that issue? Whatever the resource that is short, I might be able to contribute some of it and would like to.
comment:47 Changed 13 months ago by reneeotten (Renee Otten)
Replying to mrdomino:
Ok, sorry. Where is the right place for that issue? Whatever the resource that is short, I might be able to contribute some of it and would like to.
there is a build-bot for Sonoma but it will take a while before it will have build all the ports so that there are binaries available. Feel free to open a ticket to offer resources setting the "component" to "buildbot/mpbb"; Ryan is running the buildbots so perhaps you tag him (i.e., ryandesign) in that ticket.
comment:48 follow-up: 50 Changed 12 months ago by kencu (Ken)
if MacPorts wanted to use it’s own binaries instead of Apple’s binaries to make trace mode work again on arm, what kind of list would we need?
We probably already have most of them. And older systems would often prefer to use them too.
comment:49 Changed 12 months ago by fhgwright (Fred Wright)
Cc: | fhgwright added |
---|
comment:50 Changed 12 months ago by neverpanic (Clemens Lang)
Replying to kencu:
if MacPorts wanted to use it’s own binaries instead of Apple’s binaries to make trace mode work again on arm, what kind of list would we need?
Xcode, probably. There are a bunch of ports that use it to build GUI software, and I'm not sure there are open source alternatives for those.
You can get an approximation by collecting the contents of $prefix/var/macports/sip-workaround
on a machine where trace mode is supported. Everything in there had system integrity protection enabled and was thus copied and executed from a copy in trace mode. On one of the x86_64 systems I own where I haven't done a huge amount of compiling, this list is:
# cd /opt/local/var/macports/sip-workaround && find . -type f | sed -E 's/^\.\/[0-9]+\///g' | sort -u System/Library/Frameworks/Ruby.framework/Versions/Current/usr/bin/ruby bin/bash bin/cat bin/chmod bin/cp bin/date bin/dd bin/echo bin/expr bin/hostname bin/launchctl bin/ln bin/ls bin/mkdir bin/mv bin/pwd bin/rm bin/rmdir bin/sh bin/sleep usr/bin/ar usr/bin/arch usr/bin/awk usr/bin/basename usr/bin/bison usr/bin/clang usr/bin/clang++ usr/bin/cmp usr/bin/codesign usr/bin/cpio usr/bin/ctags usr/bin/cut usr/bin/diff usr/bin/dirname usr/bin/egrep usr/bin/env usr/bin/file usr/bin/find usr/bin/flex usr/bin/git usr/bin/gm4 usr/bin/grep usr/bin/gzip usr/bin/head usr/bin/hostinfo usr/bin/id usr/bin/install usr/bin/install_name_tool usr/bin/ld usr/bin/lipo usr/bin/m4 usr/bin/make usr/bin/mktemp usr/bin/nm usr/bin/otool usr/bin/patch usr/bin/perl usr/bin/perl5.30 usr/bin/python3 usr/bin/ranlib usr/bin/ruby usr/bin/sandbox-exec usr/bin/sed usr/bin/sort usr/bin/sqlite3 usr/bin/strip usr/bin/sw_vers usr/bin/tail usr/bin/tar usr/bin/tclsh usr/bin/touch usr/bin/tr usr/bin/true usr/bin/uname usr/bin/uniq usr/bin/unzip usr/bin/wc usr/bin/which usr/bin/xcode-select usr/bin/xcrun usr/bin/xsltproc usr/libexec/PlistBuddy usr/sbin/chown usr/sbin/sysctl
launchctl, codesign, hostinfo, install_name_tool, lipo, sandbox-exec, sw_vers, xcode-select, xcrun, PlistBuddy are probably specific enough that we don't yet have them all. Note that this is also just the subset that ports I compiled on my machine use.
We probably already have most of them. And older systems would often prefer to use them too.
I'm not sure that's worth the effort it would be, but feel free to beat me to doing that.
comment:51 follow-up: 52 Changed 12 months ago by kencu (Ken)
OK... that is a list.
Comes to mind we don't really care about tracing the things in /usr/bin or /bin anyway...
What we really care about are opportunistically found ports in ${prefix}.
Even just having trace mode work only on the things in ${prefix} would be a huge step forward...
comment:52 Changed 12 months ago by neverpanic (Clemens Lang)
Replying to kencu:
Comes to mind we don't really care about tracing the things in /usr/bin or /bin anyway...
That isn't correct, unless you want to allow binaries in /usr/bin or binaries executed through a binary in /usr/bin to allow arbitrary unfiltered access to the filesystem. Those include /usr/bin/clang (which we really want to trace) as well as /usr/bin/make, which will execute most of our build steps, or /bin/sh, which will run essentially all build scripts.
That's required because running any binary with system integrity protection will remove all DYLD_*
variables, including the DYLD_INSERT_LIBRARIES
we rely on for trace mode. In other words, the moment we run /usr/bin/make
or /bin/sh
, everything started by those will also automatically be untraced.
What we really care about are opportunistically found ports in ${prefix}.
Yes, but those aren't found by programs in $prefix.
Even just having trace mode work only on the things in ${prefix} would be a huge step forward...
No, that will likely just lead to build failures, because the view of the filesystem is suddenly no longer consistent. The same binary would behave different depending on whether it is run directly or through /bin/sh
.
comment:53 follow-up: 54 Changed 12 months ago by kencu (Ken)
I was hoping that instead of needing to modify the now-unavailable binaries, there might instead be a way to put a file system trace on ${prefix}, and only allowing access to files that have been allowed to be accessed.
picture the equivalent of making a virtual /opt/local populated by symlinks to the contents of ports that have been allowed prior to the build.
then you would have the equivalent of trace mode, leaving the binaries alone.
But I don't know enough about how this is done. chroot, etc ... and I haven't explored any of the trace mode code.
comment:54 Changed 12 months ago by neverpanic (Clemens Lang)
Replying to kencu:
I was hoping that instead of needing to modify the now-unavailable binaries, there might instead be a way to put a file system trace on ${prefix}, and only allowing access to files that have been allowed to be accessed.
macOS does not have this functionality, at least not in the fashion we'd need. There is a sandboxing mechanism, but it only allows denying access to files, not hiding them. We've tried that before, and most build systems will fail when you deny access to a file that they would like to use and know exists.
That's why trace mode emulates this by intercepting system calls related to file system access, but that requires doing this interception on all binaries, regardless of their file system location.
picture the equivalent of making a virtual /opt/local populated by symlinks to the contents of ports that have been allowed prior to the build.
then you would have the equivalent of trace mode, leaving the binaries alone.
But I don't know enough about how this is done. chroot, etc ... and I haven't explored any of the trace mode code.
Yes, that would be the equivalent of Linux mount namespaces, or chroots. macOS does not have the former, and while it does have the latter, they require root access, and are known to break Xcode and other macOS core functionality such as DNS lookups, which is why we're not using them.
Unless you can convince Apple to provide a mechanism to selectively hide files using sandboxes, or provide a container-like mount namespace mechanism, library preloading is the only viable option, and that doesn't work on arm64 macOS at the moment, unless you're willing to disable SIP.
comment:55 Changed 12 months ago by kencu (Ken)
OK. Thanks for taking the time to explain all that.
If I understand the current options, for now, folks can install macports-base from master use trace mode on an Intel system, or on an arm system install macports-base from master, disable SIP and use:
sudo nvram boot-args=-arm64e_preview_abi
which probably nobody will really do, but there it is.
comment:56 Changed 7 months ago by ryandesign (Ryan Carsten Schmidt)
Keywords: | arm64 sonoma added |
---|---|
Summary: | sip-workaround no longer works on arm64 macOS 13 Ventura due to new security features → sip-workaround / trace mode no longer works on arm64 macOS ≥ 13 due to new security features |
comment:57 Changed 3 months ago by markmentovai (Mark Mentovai)
Cc: | markmentovai added |
---|
comment:58 follow-up: 60 Changed 3 months ago by markmentovai (Mark Mentovai)
In many cases, it ought to be possible to reset the mach_header_64::cpusubtype
from CPU_SUBTYPE_ARM64E
to CPU_SUBTYPE_ARM64_ALL
(0), and then re-sign. PAC instructions essentially become no-op under arm64 (as opposed to arm64e).
This may be feasible if you’re not trying to run anything with a restricted entitlement.
mark@arm-and-hammer zsh% sw_vers ProductName: macOS ProductVersion: 15.0 BuildVersion: 24A335 mark@arm-and-hammer zsh% uname -orm Darwin 24.0.0 arm64
mark@arm-and-hammer zsh% lipo /bin/ls -thin arm64e -output /tmp/ls mark@arm-and-hammer zsh% otool -hv /tmp/ls /tmp/ls: Mach header magic cputype cpusubtype caps filetype ncmds sizeofcmds flags MH_MAGIC_64 ARM64 E USR00 EXECUTE 20 1712 NOUNDEFS DYLDLINK TWOLEVEL PIE mark@arm-and-hammer zsh% python3 Python 3.12.6 (main, Sep 7 2024, 05:43:35) [Clang 16.0.0 (clang-1600.0.26.3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import struct >>> f = open('/tmp/ls', 'r+b') >>> f.seek(8) 8 >>> hex(struct.unpack('<I', f.read(4))[0]) '0x80000002' >>> f.seek(8) 8 >>> f.write(struct.pack('<I', 0)) 4 >>> f.close() >>> ^D mark@arm-and-hammer zsh% otool -hv /tmp/ls /tmp/ls: Mach header magic cputype cpusubtype caps filetype ncmds sizeofcmds flags MH_MAGIC_64 ARM64 ALL 0x00 EXECUTE 20 1712 NOUNDEFS DYLDLINK TWOLEVEL PIE mark@arm-and-hammer zsh% codesign --sign=- --force --preserve-metadata /tmp/ls warning: default usage of --preserve-metadata implies "resource-rules" (deprecated in Mac OS X >= 10.10)! /tmp/ls: replacing existing signature mark@arm-and-hammer zsh% /tmp/ls -dl /tmp/ls /tmp/ /tmp / drwxr-xr-x 20 root wheel 640 Sep 5 16:54 / lrwxr-xr-x@ 1 root wheel 11 Sep 5 16:54 /tmp -> private/tmp drwxrwxrwt 35 root wheel 1120 Sep 25 15:45 /tmp/ -rwxr-xr-x 1 mark wheel 89088 Sep 25 15:45 /tmp/ls
comment:59 Changed 3 months ago by markmentovai (Mark Mentovai)
I should probably show something with the full end-to-end context demonstrating that DYLD_INSERT_LIBRARIES
now works.
mark@arm-and-hammer zsh% cat /tmp/preload.cc #include <stdio.h> static struct Preload { public: Preload() { fprintf(stderr, "preloaded!\n"); } } preload; mark@arm-and-hammer zsh% clang++ -dynamiclib -o /tmp/libpreload.dylib /tmp/preload.cc mark@arm-and-hammer zsh% DYLD_INSERT_LIBRARIES=/tmp/libpreload.dylib /tmp/ls -dl /tmp/ls /tmp/ /tmp / preloaded! drwxr-xr-x 20 root wheel 640 Sep 5 16:54 / lrwxr-xr-x@ 1 root wheel 11 Sep 5 16:54 /tmp -> private/tmp drwxrwxrwt 37 root wheel 1184 Sep 25 16:04 /tmp/ -rwxr-xr-x 1 mark wheel 89088 Sep 25 15:54 /tmp/ls
comment:60 Changed 3 months ago by neverpanic (Clemens Lang)
Replying to markmentovai:
In many cases, it ought to be possible to reset the
mach_header_64::cpusubtype
fromCPU_SUBTYPE_ARM64E
toCPU_SUBTYPE_ARM64_ALL
(0), and then re-sign. PAC instructions essentially become no-op under arm64 (as opposed to arm64e).
That's actually a great idea. We already have code that copies and re-signs binaries, seeking to an offset of 8 and writing 4 0-bytes should not be all that complicated. I didn't know that the binaries would just run with the changed header, I had expected them to start failing.
This may be feasible if you’re not trying to run anything with a restricted entitlement.
Our mechanism did already break binaries with entitlements, and it hasn't been a problem in practice, so I think we can ignore this.
https://github.com/macports/macports-base/blob/master/src/pextlib1.0/sip_copy_proc.c#L485-L488 is the place where we currently re-sign binaries, if we add something right before that that adjusts the mach-o header if required, trace mode might actually start working again. Note that this does not go the easy way of using a thin binary, so the function would have to understand universal binaries.
comment:61 Changed 3 months ago by ryandesign (Ryan Carsten Schmidt)
Keywords: | sequoia added |
---|
comment:62 Changed 3 months ago by Dave-Allured (Dave Allured)
Cc: | Dave-Allured added |
---|
comment:63 Changed 2 months ago by markemer (Mark Anderson)
Cc: | markemer added |
---|
comment:64 Changed 2 months ago by cooljeanius (Eric Gallager)
Cc: | cooljeanius added |
---|
comment:65 Changed 6 weeks ago by neverpanic (Clemens Lang)
comment:66 follow-up: 67 Changed 6 weeks ago by neverpanic (Clemens Lang)
Looking for testers of this PR on macOS 14; it works fine for me on macOS 15.1, but in CI on macOS 14, I see
dyld[20831]: bad bind opcode 0x1E … Killed by signal: 6
Now, that error message is completely unhelpful, because the code that prints it (https://github.com/opensource-apple/dyld/blob/master/src/ImageLoaderMachOCompressed.cpp#L1027-L1028) does so after moving the pointer that it prints (https://github.com/opensource-apple/dyld/blob/master/src/ImageLoaderMachOCompressed.cpp#L955). I'd still like to know if this happens in regular usage on macOS 14, or whether it's just a corner case I am triggering with my test.
comment:67 follow-up: 68 Changed 5 weeks ago by markmentovai (Mark Mentovai)
Replying to neverpanic:
Looking for testers of this PR on macOS 14; it works fine for me on macOS 15.1, but in CI on macOS 14, I see
dyld[20831]: bad bind opcode 0x1E … Killed by signal: 6Now, that error message is completely unhelpful, because the code that prints it (https://github.com/opensource-apple/dyld/blob/master/src/ImageLoaderMachOCompressed.cpp#L1027-L1028) does so after moving the pointer that it prints (https://github.com/opensource-apple/dyld/blob/master/src/ImageLoaderMachOCompressed.cpp#L955). I'd still like to know if this happens in regular usage on macOS 14, or whether it's just a corner case I am triggering with my test.
The message you’re quoting from the source (bad bind opcode %d in bind info
) doesn’t even match the error you’re seeing (dyld[20831]: bad bind opcode 0x1E
). You’re looking at super-old dyld source from an unofficial dump of Apple’s old opensource site (github opensource-apple), from before Apple pushed source to GitHub directly. It’s showing dyld-360.18 from 10.11.2 (2015-12-08), and it hasn’t been current since 10.11.3 (2016-01-19). I recommend that everyone purge that unmaintained opensource-apple dump from their bookmarks and workflow, and never consult it for any purpose. It’s outdated and misleading.
Current Apple open-source drops are at (github apple-oss-distributions), and dyld in particular is https://github.com/apple-oss-distributions/dyld. You can look at source specific to the macOS version you’re interested in, ranging from dyld-1222.1 from 14.0 (2023-09-26) through from 14.6 (2024-07-29).
Now, that doesn’t change the fact that, even 9 years later, the loop is still structured such that “bad bind opcode” still references the opcode that follows the one it’s actually complaining about. But given the proper current source, and the fact that there are 8 different loops that might result in bad bind opcode 0x%02X
, you may find a loop that’s used for non-arm64e that’s missing handling for an opcode that the arm64e equivalent does handle.
- common/MachOAnalyzer.cpp:1535 in dyld3::MachOAnalyzer::forEachBind, handles 13 cases
- common/MachOAnalyzer.cpp:1700 in dyld3::MachOAnalyzer::forEachBind, handles 13 cases
- common/MachOAnalyzer.cpp:2170 in dyld3::MachOAnalyzer::parseOrgArm64eChainedFixups, handles 10 cases
- common/MachOAnalyzer.cpp:5070 in dyld3::MachOAnalyzer::forEachBind_OpcodesWeak, handles 13 cases
- common/MachOAnalyzer.cpp:5179 in dyld3::MachOAnalyzer::forEachBind_OpcodesRegular, handles 13 cases
- common/MachOLayout.cpp:603 in mach_o::Fixups::parseOrgArm64eChainedFixups, handles 10 cases
- common/MachOLayout.cpp:1062 in mach_o::Fixups::forEachBind_OpcodesWeak, handles 13 cases
- common/MachOLayout.cpp:1171 in mach_o::Fixups::forEachBind_OpcodesRegular, handles 13 cases
The full suite of opcodes is defined in <mach-o/loader.h>
, the 14 #defines
beginning with the terminator BIND_OPCODE_DONE
:
#define BIND_OPCODE_DONE 0x00 #define BIND_OPCODE_SET_DYLIB_ORDINAL_IMM 0x10 #define BIND_OPCODE_SET_DYLIB_ORDINAL_ULEB 0x20 #define BIND_OPCODE_SET_DYLIB_SPECIAL_IMM 0x30 #define BIND_OPCODE_SET_SYMBOL_TRAILING_FLAGS_IMM 0x40 #define BIND_OPCODE_SET_TYPE_IMM 0x50 #define BIND_OPCODE_SET_ADDEND_SLEB 0x60 #define BIND_OPCODE_SET_SEGMENT_AND_OFFSET_ULEB 0x70 #define BIND_OPCODE_ADD_ADDR_ULEB 0x80 #define BIND_OPCODE_DO_BIND 0x90 #define BIND_OPCODE_DO_BIND_ADD_ADDR_ULEB 0xA0 #define BIND_OPCODE_DO_BIND_ADD_ADDR_IMM_SCALED 0xB0 #define BIND_OPCODE_DO_BIND_ULEB_TIMES_SKIPPING_ULEB 0xC0 #define BIND_OPCODE_THREADED 0xD0
The ones that handle 13 cases omit BIND_OPCODE_THREADED
. The ones that handle 10 cases omit BIND_OPCODE_ADD_ADDR_ULEB
, BIND_OPCODE_DO_BIND_ADD_ADDR_ULEB
, BIND_OPCODE_DO_BIND_ADD_ADDR_IMM_SCALED
, and BIND_OPCODE_DO_BIND_ULEB_TIMES_SKIPPING_ULEB
.
Given that we’re potentially putting an arm64e executable through a non-arm64e path, I’d first suspect a use of BIND_OPCODE_THREADED
.
BIND_OPCODE_THREADED
was for an earlier form of chained fixups (“old arm64e”), while the newer form just uses LC_DYLD_CHAINED_FIXUPS
and is available beyond just arm64e. With additional effort, it may be possible to apply further translations to chained fixups in this format to make them work more broadly.
Your log message is too truncated to dig deeper. Do you have a link? What executable is your test operating on when you observe this failure? What specific OS version?
comment:68 follow-up: 69 Changed 5 weeks ago by neverpanic (Clemens Lang)
Replying to markmentovai:
The message you’re quoting from the source (
bad bind opcode %d in bind info
) doesn’t even match the error you’re seeing (dyld[20831]: bad bind opcode 0x1E
). You’re looking at super-old dyld source from an unofficial dump of Apple’s old opensource site (github opensource-apple), from before Apple pushed source to GitHub directly. It’s showing dyld-360.18 from 10.11.2 (2015-12-08), and it hasn’t been current since 10.11.3 (2016-01-19). I recommend that everyone purge that unmaintained opensource-apple dump from their bookmarks and workflow, and never consult it for any purpose. It’s outdated and misleading.
I noticed that after posting the comment, noticed that the correct version still had the same bug, and thus didn't bother to correct it here to avoid adding more noise to the signal.
The ones that handle 13 cases omit
BIND_OPCODE_THREADED
. The ones that handle 10 cases omitBIND_OPCODE_ADD_ADDR_ULEB
,BIND_OPCODE_DO_BIND_ADD_ADDR_ULEB
,BIND_OPCODE_DO_BIND_ADD_ADDR_IMM_SCALED
, andBIND_OPCODE_DO_BIND_ULEB_TIMES_SKIPPING_ULEB
.Given that we’re potentially putting an arm64e executable through a non-arm64e path, I’d first suspect a use of
BIND_OPCODE_THREADED
.
Thanks, that information is very helpful. Is there some documentation that I'm not aware of that explains what BIND_OPCODE_THREADED
actually does, and whether just stripping it from the binaries is a safe operation?
BIND_OPCODE_THREADED
was for an earlier form of chained fixups (“old arm64e”), while the newer form just usesLC_DYLD_CHAINED_FIXUPS
and is available beyond just arm64e. With additional effort, it may be possible to apply further translations to chained fixups in this format to make them work more broadly.Your log message is too truncated to dig deeper. Do you have a link? What executable is your test operating on when you observe this failure? What specific OS version?
I'm seeing this in the macOS 14 CI run for the PR above, which is at https://github.com/macports/macports-base/actions/runs/11673868984/job/32505429280?pr=354.
The OS version is macOS 14.7 23H124. The specific failure occurs close to the end of the "Test MacPorts Base" foldable section with the /bin/ln
binary. For reasons that I haven't yet investigated this does not actually fail the run, even though the test definitely failed and it should have marked the build as failing.
One instance is, for example:
system: /bin/ln -s ../../../../../../../../../..//Users/runner/work/macports-base/macports-base/src/darwintracelib1.0/tests//./stat symlink dyld[20334]: bad bind opcode 0x1E Command failed: /bin/ln -s ../../../../../../../../../..//Users/runner/work/macports-base/macports-base/src/darwintracelib1.0/tests//./stat symlink Killed by signal: 6 Expected violation '/Users/runner/work/macports-base/macports-base/src/darwintracelib1.0/tests/stat' did not occur SANDBOX /Users/runner/work/macports-base/macports-base/src/darwintracelib1.0/tests/symlink=+ /private/var/select/sh=+ /bin/bash=+ /bin=+ /usr=+ /dev=+ /opt/local/var/macports/sip-workaround=+ ==== darwintrace_relative_symlinks Test that resolution of relative symlinks works as expected FAILED ==== Contents of test case: exec -ignorestderr -- ./stat "$cwd/symlink" 2>@1 ---- Test setup failed: command execution failed ---- errorInfo(setup): command execution failed while executing "system "/bin/ln -s $path symlink"" ("uplevel" body line 6) invoked from within "uplevel 1 $setup" ---- errorCode(setup): CHILDKILLED 20334 SIGABRT SIGABRT ==== darwintrace_relative_symlinks FAILED
All invocations that fail use /bin/ln
.
comment:69 Changed 5 weeks ago by markmentovai (Mark Mentovai)
Replying to neverpanic:
Thanks, that information is very helpful. Is there some documentation that I'm not aware of that explains what
BIND_OPCODE_THREADED
actually does, and whether just stripping it from the binaries is a safe operation?
There’s no documentation aside from dyld source. There have been some WWDC sessions that discuss chained fixups, but only at a high level, not at this low implementation level. And in any case BIND_OPCODE_THREADED
was an older form of chained fixups, before the more recent LC_DYLD_CHAINED_FIXUPS
that Apple would have showcased at WWDC. Limited to arm64e, BIND_OPCODE_THREADED
was only ever Apple-private.
It’s definitely not safe to strip in general. You’ll lose (at least some) relocations, which will leave you with a broken executable (except in the very unlikely event that it only covered some relocations, all of which are irrelevant to execution). You’d need to interpret BIND_OPCODE_THREADED
and translate to a different form.
Without resorting to disabling SIP, I do have another recipe to put Apple’s executables more under user control: exec_via_dlopen. I don’t like it as much because it’s not as “pure” or transparent an environment as the recipe I shared in comment:60, but in light of comment:67, it has the property that it leaves arm64e as arm64e, and allows dyld to process BIND_OPCODE_THREADED
wherever it may validly be found in an arm64e image. It has another advantage: you don’t have to rewrite Apple’s executables. There’s no code signature stripping or Mach-O tweaking involved at all.
mark@arm-and-hammer zsh% clang -Wall -Wextra -Werror -o /tmp/exec_via_dlopen exec_via_dlopen.c mark@arm-and-hammer zsh% DYLD_INSERT_LIBRARIES=/tmp/libpreload.dylib /tmp/exec_via_dlopen /bin/ls -dl /bin/ls /bin / preloaded! drwxr-xr-x 20 root wheel 640 Oct 22 03:49 / drwxr-xr-x@ 39 root wheel 1248 Oct 22 03:49 /bin -rwxr-xr-x 1 root wheel 154624 Oct 22 03:49 /bin/ls
comment:70 Changed 4 weeks ago by neverpanic (Clemens Lang)
It seems it was indeed BIND_OPCODE_THREADED
; I've obtained a copy of /bin/ln
from 14.6 and put it into lief, and here's its representation of the bind opcodes:
In [6]: print(ln.dyld_info.show_bind_opcodes) [THREADED] [SET_BIND_ORDINAL_TABLE_SIZE_ULEB] Ordinal table size := 30 [SET_DYLIB_ORDINAL_IMM] Library Ordinal := 1 [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := ___chkstk_darwin Is Weak ? false [SET_TYPE_IMM] Type := POINTER [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := ___error Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := ___memcpy_chk Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := ___stack_chk_fail Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := ___stack_chk_guard Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := ___stderrp Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := ___stdoutp Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := ___strlcpy_chk Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _basename Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _dirname Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _err Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _exit Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _fflush Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _fprintf Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _fwrite Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _getchar Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _getopt Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _linkat Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _lstat Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _optind Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _printf Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _rmdir Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _snprintf Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _stat Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _strcmp Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _strrchr Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _symlink Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _unlink Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _warn Is Weak ? false [SET_SYMBOL_TRAILING_FLAGS_IMM] Symbol name := _warnx Is Weak ? false [SET_SEGMENT_AND_OFFSET_ULEB] Segment := __DATA_CONST Segment Offset := 0 [THREADED] [APPLY] threaded_bind(THREADED_BIND/POINTER, 0x0, __DATA_CONST, ___error, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x8) threaded_bind(THREADED_BIND/POINTER, 0x8, __DATA_CONST, ___memcpy_chk, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x10) threaded_bind(THREADED_BIND/POINTER, 0x10, __DATA_CONST, ___stack_chk_fail, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x18) threaded_bind(THREADED_BIND/POINTER, 0x18, __DATA_CONST, ___strlcpy_chk, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x20) threaded_bind(THREADED_BIND/POINTER, 0x20, __DATA_CONST, _basename, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x28) threaded_bind(THREADED_BIND/POINTER, 0x28, __DATA_CONST, _dirname, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x30) threaded_bind(THREADED_BIND/POINTER, 0x30, __DATA_CONST, _err, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x38) threaded_bind(THREADED_BIND/POINTER, 0x38, __DATA_CONST, _exit, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x40) threaded_bind(THREADED_BIND/POINTER, 0x40, __DATA_CONST, _fflush, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x48) threaded_bind(THREADED_BIND/POINTER, 0x48, __DATA_CONST, _fprintf, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x50) threaded_bind(THREADED_BIND/POINTER, 0x50, __DATA_CONST, _fwrite, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x58) threaded_bind(THREADED_BIND/POINTER, 0x58, __DATA_CONST, _getchar, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x60) threaded_bind(THREADED_BIND/POINTER, 0x60, __DATA_CONST, _getopt, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x68) threaded_bind(THREADED_BIND/POINTER, 0x68, __DATA_CONST, _linkat, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x70) threaded_bind(THREADED_BIND/POINTER, 0x70, __DATA_CONST, _lstat, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x78) threaded_bind(THREADED_BIND/POINTER, 0x78, __DATA_CONST, _printf, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x80) threaded_bind(THREADED_BIND/POINTER, 0x80, __DATA_CONST, _rmdir, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x88) threaded_bind(THREADED_BIND/POINTER, 0x88, __DATA_CONST, _snprintf, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x90) threaded_bind(THREADED_BIND/POINTER, 0x90, __DATA_CONST, _stat, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0x98) threaded_bind(THREADED_BIND/POINTER, 0x98, __DATA_CONST, _strcmp, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0xa0) threaded_bind(THREADED_BIND/POINTER, 0xa0, __DATA_CONST, _strrchr, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0xa8) threaded_bind(THREADED_BIND/POINTER, 0xa8, __DATA_CONST, _symlink, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0xb0) threaded_bind(THREADED_BIND/POINTER, 0xb0, __DATA_CONST, _unlink, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0xb8) threaded_bind(THREADED_BIND/POINTER, 0xb8, __DATA_CONST, _warn, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0xc0) threaded_bind(THREADED_BIND/POINTER, 0xc0, __DATA_CONST, _warnx, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0xc8) threaded_bind(THREADED_BIND/POINTER, 0xc8, __DATA_CONST, ___stack_chk_guard, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0xd0) threaded_bind(THREADED_BIND/POINTER, 0xd0, __DATA_CONST, ___stderrp, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0xd8) threaded_bind(THREADED_BIND/POINTER, 0xd8, __DATA_CONST, ___stdoutp, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0xe0) threaded_bind(THREADED_BIND/POINTER, 0xe0, __DATA_CONST, _lstat, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0xe8) threaded_bind(THREADED_BIND/POINTER, 0xe8, __DATA_CONST, _optind, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0xf0) threaded_bind(THREADED_BIND/POINTER, 0xf0, __DATA_CONST, _stat, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x8 (0xf8) threaded_bind(THREADED_BIND/POINTER, 0xf8, __DATA_CONST, ___chkstk_darwin, library_ordinal=/usr/lib/libSystem.B.dylib, addend=0, is_weak_import=0) Segment Offset += 0x0 (0xf8) [DONE]
The exec via dlopen thing is a nice idea, but I'm not sure it's worth going down that route given the comment https://github.com/apple-oss-distributions/dyld/blob/dyld-940/dyld/DyldAPIs.cpp#L1248 you linked — it may just eventually stop working, at which point we'd be stuck mmap(2)ing the memory ourselves and shipping a patched dyld to do the same work.
I'm considering whether we could possibly use LIEF (see https://lief.re/doc/latest/tutorials/11_macho_modification.html#when-mach-o-makes-thing-harder) to do the required re-writing; I wouldn't want to ship a copy of that in MacPorts base, but we could probably make something happen where trace mode uses a tool installed by a port to do the necessary binary mangling. I'll take a look at that option.
Replying to reneeotten:
I have no information for you about that.