ccache @4.6.1: process sometimes blocks for significant periods of time, across multiple macOS releases

[mascguy (Christopher Nielsen)]

This issue was first observed via a macOS 10.8 VM. And while that is a very clean install - dedicated to MacPorts development and testing - initially I chalked it up to something I potentially did.

Since then, I've now encountered this issue on my physical macOS 10.13 installation. And that's made me a bit more suspicious. With the caveat that it may still be something specific to my installation(s).

It's important to note that I've been utilizing ccache everywhere for a few years now - across numerous releases - and have never seen an issue. (That includes the previous version, 4.6.0.) In short, this is new with version 4.6.1.

In any event, when the issue occurs, one or more ccache processes will simply block - with no apparent CPU or disk activity - for many seconds at a time. The pauses are very noticeable, and significantly increase port build times.

The temporary workaround is to simply disable use of ccache. But that's obviously not ideal.

Specifics for 10.8: There don't appear to be other processes with noticeable increases CPU time, or general activity. The ccache processes simply hang for extended periods.

As for 10.13: During the hangs, process opendirectoryd was very active, consuming 300%-ish of CPU time. Ditto for automountd, the latter consuming 80%-ish of CPU. Cancelling the MacPorts build via Ctrl-C resulted in those processes immediately setting down, to idle territory.

In terms of troubleshooting, more system detail needs to be captured - perhaps via the various dtrace-related tools - to better understand what's happening. As such, assigning the ticket to myself, until more investigation is done.

[ryandesign (Ryan Carsten Schmidt)]

We don't do anything special when installing ccache (we only have a couple patches used on ancient systems) so I'd assume it's not a MacPorts-specific issue and I'd recommend talking to the developers of ccache about it.

[mascguy (Christopher Nielsen)]

Replying to ryandesign:

We don't do anything special when installing ccache (we only have a couple patches used on ancient systems) so I'd assume it's not a MacPorts-specific issue and I'd recommend talking to the developers of ccache about it.

Yep, that will likely be the ultimate outcome. But also wanted to be sure this issue is formally tracked, in case anyone else encounters it.

The next step is to revert to 4.6.0, to see whether it resolves the issue. More to follow.

[mascguy (Christopher Nielsen)]

Update: 4.6.0 appears to be faster on macOS 10.8, though the ccache processes still routinely pause for a second or two.

On a related note, the following suggests that the issue might relate to synchronization at the file system level. (Perhaps in the macOS kernel.) Apparently slowdowns aren't limited to macOS though - others have mentioned that they see similar slowdowns in Linux - so it's unclear whether this can be fixed/improved.

​Issue 54 - ccache on OSX becomes slower when you have many CPU cores

[mascguy (Christopher Nielsen)]

While the behavior on macOS 10.8 is consistent, I'm not able to reproduce the issue on 10.13. So hopefully this was simply a one-time fluke.

And given that this is a known limitation on some platforms, we might as well close this ticket.