bsdtar --hfsCompression fails for non-root users

[AgilentGCMS]

I recently ran 'port upgrade outdated', and after building some packages it failed with the error message

--->  Activating libarchive @3.4.2_0
Error: Failed to activate libarchive: command execution failed

after which I am unable to do much with port, because it seems that port wants to install libarchive to fix broken packages first, which fails. The 'main.log' for libarchive is attached.

[AgilentGCMS]

I found a similar issue in #56563. Even though a comment there said that the underlying issue was fixed in libarchive 3.4.0, and my libarchive is 3.4.2

$ bsdtar --version
bsdtar 3.4.2 - libarchive 3.4.2 zlib/1.2.11 liblzma/5.2.4 bz2lib/1.0.8 liblz4/1.9.2 libzstd/1.4.4

I can confirm that the problem reported by #56563 is the same one I'm having, and the fix (disabling hfscompression in macports.conf for non-root installs) is the same as well. So perhaps the libarchive bug is back.

[mf2k (Frank Schima)]

In the future, please add the port maintainer(s) to Cc (port info --maintainers libarchive), if any.

[eborisch (Eric A. Borisch)]

You're operating with a custom (not /opt/local) MacPorts location, so it's always possible we've missed something in testing...

A few quick questions:

• When you're in the state where activating fails, what's the output of which bsdtar and bsdtar --version? (Be sure to check after a sudo port deactivate libarchive, as this would happen before activating a new version.)
• What type of filesystem is /Users/sbasu1/packages/macports on?

Thanks!

[iplasma]

This happens to me for cctools. Tried libarchive 3.4.0_1 and 3.4.2_0, both have same problem with activation. Setting "hfscompression no" in macports.conf fixes it.

[eborisch (Eric A. Borisch)]

Replying to iplasma:

This happens to me for cctools. Tried libarchive 3.4.0_1 and 3.4.2_0, both have same problem with activation. Setting "hfscompression no" in macports.conf fixes it.

Can you answer the first of the bulleted questions I had in my comment above? Also what OS version are you on? Thanks!

[AgilentGCMS]

As I mentioned in my original ticket, before trying port upgrade outdated, this was the bsdtar version

$ bsdtar --version
bsdtar 3.4.2 - libarchive 3.4.2 zlib/1.2.11 liblzma/5.2.4 bz2lib/1.0.8 liblz4/1.9.2 libzstd/1.4.4

Also,

$ which bsdtar
/Users/sbasu1/packages/macports/bin/bsdtar

As for filesystem type, diskutil list says APFS Volume Macintosh HD.

[eborisch (Eric A. Borisch)]

Replying to AgilentGCMS:

As I mentioned in my original ticket, before trying port upgrade outdated, this was the bsdtar version

$ bsdtar --version
bsdtar 3.4.2 - libarchive 3.4.2 zlib/1.2.11 liblzma/5.2.4 bz2lib/1.0.8 liblz4/1.9.2 libzstd/1.4.4

Also,

$ which bsdtar
/Users/sbasu1/packages/macports/bin/bsdtar

As for filesystem type, diskutil list says APFS Volume Macintosh HD.

During an upgrade, it first deactivates the (old version) port and then activates the new version. It is the result of which bsdtar during this deactivated state that I’m looking for. Sorry if my original question was not sufficiently clear.

So, sudo port deactivate bsdtar && which bsdtar would be the desired commands to run and report the result from. (It will not be the one from MacPorts as you have reported here unless something is wrong, as it shouldn’t be present at that point.)

[AgilentGCMS]

OK, before I post the results, I need to clear up one thing. I *did* manage to port upgrade outdated 10 days ago by disabling hfscompression. If that makes my answer useless, then I apologize, but I don't know how to go back to the state before I did port upgrade outdated. Having said that, here's what happens currently.

$ port deactivate bsdtar
Error: port deactivate failed: Image error: port bsdtar is not active.
$ bsdtar --version
bsdtar 3.4.2 - libarchive 3.4.2 zlib/1.2.11 liblzma/5.2.4 bz2lib/1.0.8 liblz4/1.9.2 libzstd/1.4.4
$ which bsdtar
/Users/sbasu1/packages/macports/bin/bsdtar

[eborisch (Eric A. Borisch)]

Is this location where it is trying to install into, as well, or do you have multiple macports installations?

If this is where your macports installs into, then your on-disk state and MP’s database aren’t lining up — MacPorts thinks libarchive is not active, yet its contents (including bsdtar) clearly are in place, at least partially. I’m not sure how it got into this state, but it will be difficult to determine what is causing the activate/compression issue until that is resolved.

If you have two macports installations, then all bets are off; this isn’t a supported configuration.

[AgilentGCMS]

No, I only have one macports installation. The 'port' command resolves to the correct location

$ which port
/Users/sbasu1/packages/macports/bin/port

[tobypeterson]

Rebuilt base to force usage of bsdtar; cannot reproduce this issue. I'm using the standard location.

[tobypeterson]

Ok, I _can_ repro this manually by using /usr/bin/bsdtar, which is confusing.

/usr/bin/bsdtar -xvp --hfsCompression -f /opt/local/var/macports/software/libarchive/libarchive-3.4.2_0.darwin_19.x86_64.tbz2

For some reason, it doesn't fail (for me) during activate, even though it's using /usr/bin/bsdtar there as well.

[tobypeterson]

Ah, got it! It's because I run as root. From the attached log: :debug:main Privilege de-escalation not attempted as not running as root.

[tobypeterson]

This is really a base bug; /usr/bin/bsdtar (at least on macOS 10.15) does not appear to correctly support --hfsCompression when running as non-root.

[tobypeterson]

Based on #60176, macOS 10.14's /usr/bin/bsdtar is also unsuitable.

[jmroot (Joshua Root)]

Replying to tobypeterson:

Based on #60176, macOS 10.14's /usr/bin/bsdtar is also unsuitable.

We don't attempt to use it though. The 10.14 user's issue was with the libarchive port's bsdtar. Whereas on 10.15 /usr/bin/bsdtar claims to support hfscompression so we do use it.

All the information so far points to this being a new bug introduced in libarchive 3.4.2. It was confirmed that 3.4.0 worked fine with non-root installs and it seemed OK with 3.4.1 as well AFAIK.

[jmroot (Joshua Root)]

This is both a bug in a port, and an OS bug that base needs to work around. We need to:

1. Fix the libarchive port and get the fix accepted upstream. If someone could bisect the commits since 3.4.1, that would be very helpful.
2. Make base not use hfscompression if it would be using a broken version of /usr/bin/bsdtar and not running as root.

[tobypeterson]

Sorry, I may be misunderstanding; what's the bug in the port?

/usr/bin/bsdtar --hfsCompression generates a ton of errors; /opt/local/bin/bsdtar (libarchive-3.4.2_0) works fine.

edit: I see, some suspicion that libarchive 3.4.2 may be broken on 10.14. I'll poke around in a VM to see if I can repro that.

[tobypeterson]

Reopened #60176 to continue investigating that. Let's track the base bug here.

[eborisch (Eric A. Borisch)]

#60176 and this are both likely due to issues with extended attributes that shouldn't be there (show up with '@' on ls -l listing) combined with an issue within libarchive if:

• the activated file is set read-only
• AND has extended attributes (not compression iteslf, as this is applied separately)
• AND is not being extracted by root

See also my comment here on libarchive.

For #60176, this can be fixed by squashing stray resource forks (as the port already does for a number of files) in the manpages; I haven't looked further at this one yet.

[eborisch (Eric A. Borisch)]

I don't have 10.15 running, but from ​here it appears it has libarchive 3.3.2, which has --hfsCompression, but does not have ​my fix for supporting non-root extraction in it (which went into 3.4.0).

We may just want to change our test to see if our libarchive is installed rather than if it claims to support --hfsCompression.

I still think supporting compression is a big win, especially for developers (header files are excellent compression fodder, and compilers have lots of them) or anyone on a laptop (fixed size storage; every byte counts.)

[jmroot (Joshua Root)]

Replying to eborisch:

We may just want to change our test to see if our libarchive is installed rather than if it claims to support --hfsCompression.

That won't help, the reporter of #60176 was using our libarchive on 10.14.

[jmroot (Joshua Root)]

Upon further investigation, the case in #60176 has been broken all along, it's not a regression since 3.4.0 as I previously thought.

[jmroot (Joshua Root)]

Replying to jmroot:

1. Fix the libarchive port and get the fix accepted upstream. If someone could bisect the commits since 3.4.1, that would be very helpful.

It doesn't look like anyone ever took this upstream, so: ​https://github.com/libarchive/libarchive/issues/1415

[jmroot (Joshua Root)]

In 9d0ebd5b5b1a7fd0f3a05be85bf0e7267c63b70a/macports-base (master):

Disable hfscompression for non-root installs (again)

See: #60230

[jmroot (Joshua Root)]

Workaround is back in base 2.6.3 but the bug in libarchive still needs to be addressed, so leaving this open for that.

[jmroot (Joshua Root)]

This may have been fixed in 3.5.0; release notes mention "support for system extended attributes", but it's hard to check since the original test file has since been purged.