podman machine does not startup on the arm64 (macOS 11-12)

[langerma (DataPoints)]

recently i discovered that the podman machine does not boot up as expected

[kencu (Ken)]

please open up an issue with podman here with your details

​https://github.com/containers/podman

as MacPorts just manages the build for what they publish, but doesn’t write it

[langerma (DataPoints)]

podman itself runs perfectly fine. may there is just sth wrong with the qemu call in podman machine start... i tried it with the version from brew on another mac. and that worked with the same versions.

[judaew (Vadym-Valdis Yudaiev)]

This is my usual test plan for podman updates: ​https://gist.github.com/judaew/85c6e8a62bf0e7f5be5188e020492e21. The first launch takes 2–3 minutes, but everything works correctly. Can you provide more information about what system you have and what errors you are getting with podman?

[kencu (Ken)]

I just tried this on my arm Mac, and it just stayed like this for a few hours, until I cancelled it:

 % podman machine start podman-machine-default 
Starting machine "podman-machine-default"
Waiting for VM ...

[benz0li (Olivier Benz)]

This is due to ​QEMU v8 (currently 8.0.2). I have reverted to ​QEMU v7 (7.1.0) and everything works fine again.

[F30 (Felix Dreissig)]

I can confirm this (tested on two Apple Silicon MacBook Airs, one with M1 and one with M2).

Podman installed using the ​upstream installer works fine. That installer bundles its own QEMU 8.0.0.

I also tried different versions of the Fedora CoreOS QCOW image and running Podman's QEMU call manually like this:

/opt/local/bin/qemu-system-aarch64 -m 2048 -smp 1
  -fw_cfg name=opt/com.coreos/config,file=/Users/felix/.config/containers/podman/machine/qemu/podman-machine-default.ign
  -device virtio-serial -accel hvf -accel tcg -cpu host -M virt,highmem=on
  -drive file=/opt/local/share/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on
  -drive file=/Users/felix/.local/share/containers/podman/machine/qemu/podman-machine-default_ovmf_vars.fd,if=pflash,format=raw
  -drive if=virtio,file=/Users/felix/.local/share/containers/podman/machine/qemu/podman-machine-default_fedora-coreos-38.20230722.2.1-qemu.aarch64.qcow2

However, there is little helpful output, as the Fedora CoreOS image does not log to a serial console and no output is seen even when attaching a VGA device.

[thestr4ng3r (Florian Märkl)]

The issue is that /opt/local/share/qemu/edk2-aarch64-code.fd has transparent APFS compression. Using afscompress (from the port of the same name), it is possible to check the status and decompress it:

$ afscompress -l /opt/local/share/qemu/edk2-aarch64-code.fd
/opt/local/share/qemu/edk2-aarch64-code.fd is HFS+/APFS compressed.
$ sudo afscompress -d /opt/local/share/qemu/edk2-aarch64-code.fd
$ afscompress -l /opt/local/share/qemu/edk2-aarch64-code.fd
/opt/local/share/qemu/edk2-aarch64-code.fd is not HFS+/APFS compressed.

after that, podman works again.

I had started to diagnose this problem over a month ago, but then forgot about it, only to hit the same issue again today. Iirc the difference was the return value of an lseek call in block/file-posix.c of qemu used to determine whether to consider a range from a file as just 0s. I think this one, or the one directly before it: ​https://gitlab.com/qemu-project/qemu/-/blob/0d52116fd82cdd1f4a88837336af5b6290c364a4/block/file-posix.c#L3136.
The return value that was given for the compressed file made qemu map only 0s instead of the actual firmware contents, so it would not boot.

This should probably be reported upstream with a small reproducer, but macports could also make sure the installed firmwares in /opt/local/share/qemu do not use compression.

[ryandesign (Ryan Carsten Schmidt)]

#67549 is apparently the same issue and should be closed when this is fixed.

Replying to thestr4ng3r:

This should probably be reported upstream

By all means, please do that.

[silverbzez]

tl;dr set hfscompression no in $prefix/etc/macports/macports.conf

This should probably be reported upstream

I don't think it's a bug from upstream but rather macports mess it up. From macports' config file I found these lines in $prefix/etc/macports/macports.conf:

# Apply transparent filesystem compression to files on activation.
# Requires bsdtar with support for --hfsCompression in binpath, which can be
# provided by installing the libarchive port. This will work with HFS+ or APFS
# volumes only and will be ignored on other filesystems.
#hfscompression      	yes

Apparently macports default this flag on. As I tested with this flag off, qemu works as expected.

[sothawo (Peter-Josef Meisch)]

Just ran into this issue on a MBP with M3 Pro.

I changed hfscompression to no (removed the comment) and reinstalled qemu and podman, but I still get stuck on the "Waiting for VM ..." message. Am I still missing something here?

[sothawo (Peter-Josef Meisch)]

Ok, did one more uninstall, restart and reinstall, now it is working.

[ryandesign (Ryan Carsten Schmidt)]

Replying to silverbzez:

tl;dr set hfscompression no in $prefix/etc/macports/macports.conf

This should probably be reported upstream

I don't think it's a bug from upstream but rather macports mess it up.

MacPorts doesn't mess anything up. MacPorts makes (optional) use of hfscompression, a feature Apple introduced in Mac OS X 10.6. If podman and/or qemu are not compatible with hfscompression, that is an upstream bug that should be reported to them.

Has duplicate #69017.

[judaew (Vadym-Valdis Yudaiev)]

I pointed out this workaround for ARM in the notes of the port a few months ago.

commit 9347305

[benz0li (Olivier Benz)]

FYI Stay with QEMU v8.0.5 as Podman v4.8.3 currently does not work with v8.2.0.

Cross reference: ​https://github.com/Homebrew/homebrew-core/issues/159971

---

Update: Podman v4.9.0 seems to work with QEMU v8.2.0.

---

comment:14 is still required, though.

[josepheiba]

I have QEMU v8.2.1 and it's not working

[josepheiba]

QEMU v8.2.1, now working after setting hfscompression to no, can anyone explain why hfscompression is even an option set to yes per default, what what are it's advantages, should I set it to no per default from now on?

[F30 (Felix Dreissig)]

This should be obsolete since Podman 5 doesn't use QEMU anymore. I can successfully run Podman 5.0.2 on macOS 13.6 without any workarounds.

[judaew (Vadym-Valdis Yudaiev)]

The Podman 5 and above is not compatible with macOS 12 and below. But all arm64 compatible Macs have the ability to upgrade to macOS 13 and above. Unfortunately, this issues is won't fix.

[judaew (Vadym-Valdis Yudaiev)]

In 4418b52c7b17e5dc53a2cf7c954360c37da8d1c6/macports-ports (master):

podman: Improve notes

See: #67731