Opened 3 years ago

Closed 3 years ago

#61683 closed defect (fixed)

remoted hangs on Big Sur when installing p5* ports, losing network and sometimes panicking kernel

Reported by: BishopBerkeley Owned by:
Priority: High Milestone:
Component: base Version: 2.6.4
Keywords: bigsur Cc: wifiprintguy (Smith Kennedy), libsystem-ethan, krissen (Kristian Niemi), deusama (Amadeus), blair (Blair Zajac), conradwt (Conrad Taylor), nhchan, acheronfail, dsavransky (Dmitry Savransky)
Port:

Description (last modified by ryandesign (Ryan Carsten Schmidt))

When trying to install

p5.28-package-stash            @0.390.0        perl/p5-package-stash

as part of an "upgrade outdated" command, port consistently causes a kernel panic. If someone can direct me to the appropriate log file, I can post the transcript.

I'm running ports on BS. I simply installed the latest ports for BS over the old one. Xcode and command line tools have all been updated, too.

Change History (59)

comment:1 Changed 3 years ago by ryandesign (Ryan Carsten Schmidt)

Component: portsbase
Description: modified (diff)
Keywords: bigsur added
Summary: Kernel panic when trying to install p5.28-package-stash @0.390.0 perl/p5-package-stashKernel panic on Big Sur when trying to install perl modules

You're the third person to report this problem on Big Sur. Please see prior discussion at https://lists.macports.org/pipermail/macports-users/2020-November/049004.html and https://lists.macports.org/pipermail/macports-dev/2020-November/042573.html. I believe it is related to problems previously observed by one user on Lion and by me on 32-bit Snow Leopard; see #60509.

Do you have any third-party software installed that intercepts network traffic, such as Little Snitch or Carbon Black?

comment:2 Changed 3 years ago by BishopBerkeley

I do, indeed, have Little Snitch running. However, even when I turn Little Snitch's network monitor off, the kernel panic still happens.

comment:3 Changed 3 years ago by ryandesign (Ryan Carsten Schmidt)

Try my suggestions from those threads: use MacPorts master (2.6.99) instead. If that doesn't work, change the limit of 50 simultaneous pings to something lower.

comment:4 Changed 3 years ago by ryandesign (Ryan Carsten Schmidt)

Also, try fully uninstalling Little Snitch. Maybe it still inserts itself into your network stack even if you turn its network monitoring off.

comment:5 Changed 3 years ago by jmroot (Joshua Root)

Not saying we shouldn't work around this if possible, but kernel panics are always a bug in the kernel or a kernel extension (and I believe Big Sur no longer supports kernel extensions). So you should definitely report this to Apple.

comment:6 in reply to:  3 Changed 3 years ago by BishopBerkeley

Replying to ryandesign:

Try my suggestions from those threads: use MacPorts master (2.6.99) instead. If that doesn't work, change the limit of 50 simultaneous pings to something lower.

Thank you. I will try to get to that. How do I change the ping limit?

comment:7 in reply to:  1 ; Changed 3 years ago by BishopBerkeley

Replying to ryandesign:

You're the third person to report this problem on Big Sur. Please see prior discussion at https://lists.macports.org/pipermail/macports-users/2020-November/049004.html and https://lists.macports.org/pipermail/macports-dev/2020-November/042573.html. I believe it is related to problems previously observed by one user on Lion and by me on 32-bit Snow Leopard; see #60509.

Do you have any third-party software installed that intercepts network traffic, such as Little Snitch or Carbon Black?

Yes, those two threads describe what I'm experiencing as well. I have upgrade to little Snitch 5.0.3, which ditches the kext and implements its monitoring another way. I have activated LS's Endpoint security option, which pulls data from the Berkeley packet filter. This did not change the outcome. But, it seems as if the problem is not dependent on LS since the other two reports don't report having a network monitor installed.

comment:8 in reply to:  7 ; Changed 3 years ago by ryandesign (Ryan Carsten Schmidt)

Replying to BishopBerkeley:

How do I change the ping limit?

See https://lists.macports.org/pipermail/macports-dev/2020-November/042592.html

Replying to BishopBerkeley:

it seems as if the problem is not dependent on LS since the other two reports don't report having a network monitor installed.

Perhaps not. But it is probably dependent on something nonstandard that you and those other users have on your systems, since most Big Sur users do not experience the problem. If you can think of what that might be, let us know.

comment:9 Changed 3 years ago by BishopBerkeley

I removed Little Snitch and successfully installed all outdated ports. Upon uninstalling LS, the software informed me that it was "hosting" certain extensions. I guess this is the new way the process goes under BS. I will apprise Objective Software of the bug in LS that persists up to version 5.0.4.

Thank you for your help.

comment:10 in reply to:  8 ; Changed 3 years ago by BishopBerkeley

Can you provide the full path to the folder that contains the number of simultaneous pings? I cannot follow the path provided in the thread below.

Replying to ryandesign:

Replying to BishopBerkeley:

How do I change the ping limit?

See https://lists.macports.org/pipermail/macports-dev/2020-November/042592.html

comment:11 Changed 3 years ago by BishopBerkeley

It seems as if this bug is a kernel bug that will be fixed by Apple in BS 11.1. Details on Objective Development's web site. https://obdev.at/support/littlesnitch Expand the "troubleshooting" header.

comment:12 Changed 3 years ago by BishopBerkeley

I upgraded to 11.1 last night, but the problem persisted. I usually install the latest command line tools by launching Xcode, but since they were not installed on launching xcode this time, I installed them manually with xcode-select. This solved the problem.

So, this problem is solved by ensuring that you have updated to MacOS 11.1 AND installed or rinstalled the latest command line tools. This resolves the kernel panic caused by Little Snitch.

comment:13 Changed 3 years ago by jtrtik (Jan Trtík)

I have still this problem even after upgrading to MacOS 11.1 and reinstalling command line tools. The problem is less frequent, but still present... I sent report to Apple via Feedback Assistant and output of spindump command to Objective Development for further inspection.

comment:14 Changed 3 years ago by mf2k (Frank Schima)

Works fine for me now. Are you running version 5.0.4 of Little Snitch?

comment:15 Changed 3 years ago by jtrtik (Jan Trtík)

Yes, i am running the latest version of Little Snitch. The issue is less frequent, but still present. The system gets restarted because /usr/libexec/remoted hangs. It is clearly bug in MacOS.

Folks at Objective Development recommended to try to kill the hanging process as soon as i notice there will be a panic soon. Noz sure the system will fully recover, but it is worth a try.

comment:16 in reply to:  10 Changed 3 years ago by ryandesign (Ryan Carsten Schmidt)

Replying to BishopBerkeley:

Replying to ryandesign:

See https://lists.macports.org/pipermail/macports-dev/2020-November/042592.html

Can you provide the full path to the folder that contains the number of simultaneous pings? I cannot follow the path provided in the thread below.

The default installation path of the file is /opt/local/libexec/macports/lib/port1.0/fetch_common.tcl

comment:17 in reply to:  12 Changed 3 years ago by ryandesign (Ryan Carsten Schmidt)

Replying to BishopBerkeley:

I usually install the latest command line tools by launching Xcode, but since they were not installed on launching xcode this time, I installed them manually with xcode-select. This solved the problem.

Opening Xcode never installs or updates the command line tools. It does install or update "additional components" but that is a different thing (for mobile device development).

comment:18 Changed 3 years ago by wifiprintguy (Smith Kennedy)

Cc: wifiprintguy added

comment:19 Changed 3 years ago by thermalecology

I get a Kernel panic when trying to upload the URI perl module. I'm using 11.2. It's been happening for weeks now.

comment:20 Changed 3 years ago by libsystem-ethan

Cc: libsystem-ethan added

comment:21 Changed 3 years ago by bosmacs (Craig Bosma)

Ran into this problem after upgrading directly to 11.2 from Catalina -- installing/upgrading anything that hit the CPAN mirrors would stop local networking from working at all, applications would hang, etc. Uninstalled Little Snitch 5.0.4 and the problem is gone, at least for the moment.

comment:22 Changed 3 years ago by krissen (Kristian Niemi)

Cc: krissen added

comment:23 Changed 3 years ago by deusama (Amadeus)

Cc: deusama added

comment:24 Changed 3 years ago by deusama (Amadeus)

Same issue here after upgrading to 11.2 directly from Catalina. I have never installed Little Snitch tho. As soon as I hit the "Fetching distfiles ..." part it starts crashing.

comment:25 in reply to:  24 Changed 3 years ago by krissen (Kristian Niemi)

Replying to deusama:

Same issue here after upgrading to 11.2 directly from Catalina. I have never installed Little Snitch tho. As soon as I hit the "Fetching distfiles ..." part it starts crashing.

Just echoing what deusama said. Was working fine on Catalina. Having upgraded to 11.2 the kernel started panicking when fetching distfiles for perl ports. I do not have Little Snitch or some such installed.

Is changing ping times as described here the recommended workaround? 

comment:26 Changed 3 years ago by krissen (Kristian Niemi)

  • Installed port from git master (port reports v. 2.6.99).
  • (Now have the line set max_hosts_to_ping 50 in /opt/local/libexec/macports/lib/port1.0/fetch_common.tcl.)
  • Set max_hosts_to_ping to 10.
  • Can now fetch perl ports without kernel panics. 🎉

comment:27 Changed 3 years ago by nortcele

Same issue here during installation of port @p5.28-mozilla-ca@.

Macports 2.6.4

macOS 11.2.2 20D80

Xcode 12.4 12D4e

comment:28 Changed 3 years ago by hakusaro (Lucas Nicodemus)

I had the same problem, found after googling in resignation and landing at #62360. I'm on macOS 11.2.2 with Little Snitch 5.1.1. The interesting attributes I have are that I'm on an M1 system, so I figured it was a Rosetta issue or Apple Silicon issue. It didn't appear to be. While I couldn't get around the problem by disabling the network filter from the Little Snitch menu, completely removing the network filter interface in the Network prefpane worked.

I sent Apple Feedback a ticket with ID FB9031599. If you encounter the problem, it's worth running sudo sysdiagnose as the problem happens, then attach the CLI generated tar.gz that starts with sysdiagnose to a feedback report using applefeedback://. If you mention this thread and the feedback ID I used, it might help this get elevated up the chain at Apple so that someone can solve it.

My guess is that network filters exacerbate some kind of race condition or deadlock in the network stack, but the problem probably isn't directly related. Nevertheless, if you encounter the problem, it's worth reporting it, removing any network filter interfaces, and/or trying the max ping workaround.

Fundamentally, useful sysdiagnose dumps and feedback reports are going to be the way to get this to the right people at Apple.

comment:29 Changed 3 years ago by BishopBerkeley

For the record, I haven’t had this problem since the bug was fixed by apple in macos 11.1. I use a 2017 macbook, latest xcode and little snitch 5.1.1

comment:30 Changed 3 years ago by BishopBerkeley

I stand corrected. I got a kernel panic this morning at this stage of installing perl modules:

Attempting to fetch p5.28-date-manip-6.850.0_0.darwin_20.noarch.tbz2 from https://mse.uk.packages.macports.org/p5.28-date-manip
--->  Fetching distfiles for p5.28-date-manip

I have followed the above instructions and submitted the sysdiagnose file to the ticket ID above.

Last edited 3 years ago by ryandesign (Ryan Carsten Schmidt) (previous) (diff)

comment:31 Changed 3 years ago by jmroot (Joshua Root)

Summary: Kernel panic on Big Sur when trying to install perl modulesremoted hangs on Big Sur when installing p5* ports, losing network and sometimes panicking kernel

Retitling since the issue doesn't always result in a kernel panic, sometimes just loss of network connectivity.

comment:32 Changed 3 years ago by BishopBerkeley

I just managed to install everything without incident with BS 11.2.3.

comment:33 in reply to:  26 ; Changed 3 years ago by jakw0j

Replying to krissen:

  • Installed port from git master (port reports v. 2.6.99).
  • (Now have the line set max_hosts_to_ping 50 in /opt/local/libexec/macports/lib/port1.0/fetch_common.tcl.)
  • Set max_hosts_to_ping to 10.
  • Can now fetch perl ports without kernel panics. 🎉

how to install port from git master? i can't find a solution

Last edited 3 years ago by jakw0j (previous) (diff)

comment:34 in reply to:  33 ; Changed 3 years ago by krissen (Kristian Niemi)

Replying to jakw0j:

how to install port from git master? i can't find a solution

Follow the instructions for Git install here but stay on master instead of checking out tag 2.6.4 (i.e. skip the line where the install instructions say "# skip this if you want to use the development version"). Good luck!

comment:35 in reply to:  34 ; Changed 3 years ago by jakw0j

is it going to make all my ports to use 2.6.99? im confused now. so i am forced to use not latest version of macports?

Replying to krissen:

Replying to jakw0j:

how to install port from git master? i can't find a solution

Follow the instructions for Git install here but stay on master instead of checking out tag 2.6.4 (i.e. skip the line where the install instructions say "# skip this if you want to use the development version"). Good luck!

comment:36 in reply to:  35 Changed 3 years ago by krissen (Kristian Niemi)

Replying to jakw0j:

is it going to make all my ports to use 2.6.99? im confused now. so i am forced to use not latest version of macports?

We’re talking about installing the development branch of port. If you’re unsure what that means or entails, I would suggest it it perhaps better not to proceed.

comment:37 Changed 3 years ago by jakw0j

i see no other option as i cant install i.e. git because my machine crashes everytime on 2.6.4 :(

port: p5.28-mozilla-ca

comment:38 Changed 3 years ago by kencu (Ken)

grand total time invested = 120 seconds

cd /tmp
git clone https://github.com/macports/macports-base.git
cd macports-base
./configure && make && sudo make install
cd ..
rm -rf macports-base

comment:39 Changed 3 years ago by jakw0j

well, i've done that - no need to be rude

edit: anyway, thanks for help :)

Last edited 3 years ago by jakw0j (previous) (diff)

comment:40 Changed 3 years ago by kencu (Ken)

there are downsides to doing this. there are (AFAIK) no update prompts to tell you when MP has been updated, or to run a new "update" automatically. So you just "have to know". And some features are not fully debugged.

Good luck!

Version 5, edited 3 years ago by kencu (Ken) (previous) (next) (diff)

comment:41 in reply to:  38 Changed 3 years ago by minusf

Replying to kencu:

grand total time invested = 120 seconds

cd /tmp
git clone https://github.com/macports/macports-base.git
cd macports-base
./configure && make && sudo make install
cd ..
rm -rf macports-base

my grand total time invested in this issue is in days already.

this issue is not new. it has been widely reported by many people. although it causes no kernel panic at the moment ("only" full networking loss until reboot), the going advice is "use master" and/or with some undocumented option overrides.

yes, it's not directly macports' fault per se, but it triggers it, and it makes running macports akin to playing russian roulette. i know macports is a volunteer effort, but i find it strange not to release a minor patch version with a temporary fix while the dust settles around the other software involved: macos, little snitch, etc. the others would be hardly hugely affected, 200+ mirrors for any package makes little sense to me, just to speed test them would be slower than grabbing a random mirror.

running master is not officially recommended by the project, otherwise what would be the point of releases. can we have a fix please even if only temporary?

comment:42 in reply to:  32 Changed 3 years ago by krissen (Kristian Niemi)

Replying to BishopBerkeley:

I just managed to install everything without incident with BS 11.2.3.

The perl files weren't cached, were they? Anyway, I was curious, on BS 11.2.3. Although the release notes didn't seem to mention anything related, I reverted the changes to fetch_common.tcl and downgraded to the latest release of MacPorts (2.6.4). However, a port fetch -d p5.26-http-message brought the network down and eventually panicked the kernel as well.

Reinstalling MacPorts master (2.6.99), changing max_hosts_to_ping back to 10, and the port fetch command worked again, without kernel panics etc. *shrug*

comment:43 Changed 3 years ago by kencu (Ken)

I'm sorry but I'm not on the chain of people that can help with a new release. I haven't seen or heard any rumblings about a new release, but that doesn't mean one might not happen later today either for all I know.

My only input here, not particularly appreciated it seems, was to try to point out that to install base from master is a really simple process that (IMHO) should not be considered a major obstacle to happiness. You can install the next release right over top of master whenever it comes.

comment:44 Changed 3 years ago by jakw0j

i really appreciate your input. im sorry if i misunderstood your intentions and assumed that you were rude. im still learning, so that is quite an adventure for me to be honest

edit: okay, ive done some troubleshooting here and there.
since i was unable to install port p5.28-mozilla-ca on my main machine i've decided to create VM with Big Sur

my setup
VMware Fusion 12.1.0
host os: macOS Big Sur 11.2.3 (20D91)
guest os: macOS Big Sur beta 1.3 (20E5210c) with VMware tools v. 11.2.5.26209 (build-17337674)
Xcode 12.5 beta 3 (released 02.03.2021)
Command Line Tools for Xcode 12.5 beta 3 (released 02.03.2021)
with listed configuration i was able to install p5.28-mozilla-ca package with MacPorts Version: 2.6.4
seems that it'll be fixed with macOS 11.3

Last edited 3 years ago by jakw0j (previous) (diff)

comment:45 Changed 3 years ago by jmroot (Joshua Root)

If you would like a minimal workaround rather than installing from master, making this change to /opt/local/libexec/macports/lib/port1.0/fetch_common.tcl will prevent anything from being pinged (which means sites will be tried in the order they are listed):

  • /opt/local/libexec/macports/lib/port1.0/fetch_common.tcl

    old new  
    263263                # first check the persistent cache
    264264                set pingtimes($host) [get_pingtime $host]
    265265                if {$pingtimes($host) eq {}} {
    266                     if {[catch {set fds($host) [open "|ping -noq -c3 -t3 $host"]}]} {
     266                    if {[catch {set fds($host) [open "|false -noq -c3 -t3 $host"]}]} {
    267267                        ui_debug "Spawning ping for $host failed"
    268268                        # will end up after all hosts that were pinged OK but before those that didn't respond
    269269                        set pingtimes($host) 5000

comment:46 Changed 3 years ago by blair (Blair Zajac)

Cc: blair added

comment:47 in reply to:  45 ; Changed 3 years ago by minusf

Replying to jmroot:

thank you for the quick fix. i am in a back and forth with a little snitch dev but he cannot reproduce this no matter what. really sad state of the network stack in macos right now.

comment:48 Changed 3 years ago by conradwt (Conrad Taylor)

At this time, I have 161 CPAN packages installed which are 22.2% of the total installed packages. Does any know which CPAN ports are required for MacPorts?

comment:49 Changed 3 years ago by conradwt (Conrad Taylor)

Cc: conradwt added

comment:50 in reply to:  47 Changed 3 years ago by minusf

the latest info is that this particular bug is already fixed in 11.3 beta. let's see how it goes.

comment:51 Changed 3 years ago by ryandesign (Ryan Carsten Schmidt)

Has duplicate #62724.

comment:52 Changed 3 years ago by nhchan

Cc: nhchan added

comment:53 Changed 3 years ago by acheronfail

Cc: acheronfail added

comment:54 Changed 3 years ago by acheronfail

I've also found that has been happening to me quite consistently when installing p5.28-sgmlspm. The above patch (changing ping to false) has resolved the problem for me.

Let's hope that 11.3 fixes this issue.

My system: 11.2.3, latest macports, Xcode SDK installed and also have Little Snitch 5.1.2 installed. If you'd like any more info hit me up. :)

comment:55 Changed 3 years ago by dsavransky (Dmitry Savransky)

Cc: dsavransky added

comment:56 Changed 3 years ago by jkdufair (Jason Dufair)

I will add that I had this issue as well and the culprit was Cisco AnyConnect VPN. Tried to install the git port and experienced the same problems described in this ticket. Uninstalling Cisco AnyConnect seems to have been a suitable workaround

comment:57 Changed 3 years ago by blair (Blair Zajac)

Since upgrading to Big Sur 11.3 I haven't been able to reproduce the problem with multiple invocations of

$ rm /opt/local/var/macports/pingtimes
$ /opt/local/bin/port clean --all p5.30-libwww-perl
$ /opt/local/bin/port -v fetch p5.30-libwww-perl

comment:58 Changed 3 years ago by dsavransky (Dmitry Savransky)

Another confirmation that 11.3 appears to fix this. Previously (on 11.2.1) doing port install git +svn (which pulls down multiple p5* ports) was a guaranteed kernel panic. Just updated to 11.3 (and updated xcode to 12.5) and was able to the install with zero issues. I also have Cisco AnyConnect installed (which has its own share of issues on big sur) and did not need to uninstall.

Last edited 3 years ago by dsavransky (Dmitry Savransky) (previous) (diff)

comment:59 Changed 3 years ago by jmroot (Joshua Root)

Resolution: fixed
Status: newclosed

OK, given the above comments saying 11.3 works, and no reports to the contrary, I think it's safe to close this.

Note: See TracTickets for help on using tickets.