mame: 0.226 build fails for 10.7_x64 and 10.10_x64, due to potential python 3.8 process termination

[mascguy (Christopher Nielsen)]

Currently investigating two Mame 0.226 build jobs, both of which stopped within the first few minutes. In each case, the Python processes appear to have been killed.

Those jobs are:

10.7_x64:

​https://build.macports.org/builders/ports-10.7_x86_64-builder/builds/37091/steps/install-port/logs/stdio

10.10_x64:

​https://build.macports.org/builders/ports-10.10_x86_64-builder/builds/129148/steps/install-port/logs/stdio

Output at failure time for 10.7:

[...]
Compressing src/mame/layout/saiyukip.lay...
/opt/local/bin/python3.8 scripts/build/complay.py src/mame/layout/saiyukip.lay build/generated/mame/layout/saiyukip.lh layout_saiyukip
make: /opt/local/bin/python3.8: Permission denied
make: *** [build/generated/mame/layout/saitek_tstar432.lh] Killed: 9
make: *** Waiting for unfinished jobs....
make: *** [build/generated/mame/layout/saiyukip.lh] Error 1

Output at failure time for 10.10:

[...]
Compressing src/mame/layout/fidel_rsc.lay...
/opt/local/bin/python3.8 scripts/build/complay.py src/mame/layout/fidel_rsc.lay build/generated/mame/layout/fidel_rsc.lh layout_fidel_rsc
make: *** [build/generated/mame/layout/fidel_rsc.lh] Killed: 9
make: *** Waiting for unfinished jobs....
make: *** [build/generated/mame/layout/fidel_playmatic.lh] Killed: 9
make: *** [build/generated/mame/layout/fidel_phantom.lh] Killed: 9

[neverpanic (Clemens Lang)]

macOS kills processes that violate the sandbox. We're running builds inside a sandbox as configured in ​https://github.com/macports/macports-base/blob/master/src/port1.0/portsandbox.tcl. Violating this sandbox during a build may lead to this.

[mascguy (Christopher Nielsen)]

Replying to neverpanic:

macOS kills processes that violate the sandbox. We're running builds inside a sandbox as configured in ​https://github.com/macports/macports-base/blob/master/src/port1.0/portsandbox.tcl. Violating this sandbox during a build may lead to this.

Understood, and AFAIK, no sandbox violations are occurring.

Given that these process terminations are only occurring on the 10.7 and 10.10 builders -- and I assume sandboxing is being enforced across all builders? -- how can the port successfully build elsewhere?

[ryandesign (Ryan Carsten Schmidt)]

I don't know exactly what our sandboxing does, but I don't think it would prevent running python3.8, even if you hadn't declared a dependency on it, which you have. (Trace mode, on the other hand, would prevent using files belonging to ports that you haven't declared a dependency on, but trace mode is off by default and is not used on our build servers.) I don't think I've ever seen a port build where processes got killed unexpectedly like this, or if I have, then I haven't seen it often.

Anyway, both logs show python3.8 being invoked several times successfully before the problem occurs, which casts doubt on the accuracy of the "permission denied" error, which only appears in one of the two logs. I can only imagine that maybe a weird parallel build failure is occurring and just coincidentally affected those two builders but not the others. Usually I would suggest testing that theory by using use_parallel_build no but since mame takes so long to build, disabling parallel building for this port would be terrible...

Our buildbot workers are virtual machines on several different physical servers. The 10.7, 10.10 and 10.13 VMs are all on the same physical server, though I can't think of anything relevant that would be different about that server compared to the others.

[mascguy (Christopher Nielsen)]

Since this sounds rare, I won't worry too much about it. And some good news: I've been monitoring the Mame builds tonight, and none have failed prematurely. Definitely encouraging!

[mascguy (Christopher Nielsen)]

All of the x86_64 Mame builds succeeded last night, so we're good for now.

[ryandesign (Ryan Carsten Schmidt)]

Then let's close this for now but if you notice it again please reopen this ticket or file a new one.