Opened 9 years ago

Closed 8 years ago

#48173 closed defect (fixed)

Lion buildbot builder: No space left on device

Reported by: ryandesign (Ryan Carsten Schmidt) Owned by: admin@…
Priority: Normal Milestone:
Component: server/hosting Version:
Keywords: Cc: jeremyhu (Jeremy Huddleston Sequoia), mojca (Mojca Miklavec)
Port:

Description

The Lion buildbot builder (tenseven-slave.macports.org) ran out of disk space trying to build gcc6 during this build:

https://build.macports.org/builders/buildports-lion-x86_64/builds/29508

In file included from /opt/local/var/macports/build/_opt_mports_dports_lang_gcc6/libgcc-devel/work/gcc-6-20150621/libstdc++-v3/include/precompiled/stdc++.h:94:0:
/opt/local/var/macports/build/_opt_mports_dports_lang_gcc6/libgcc-devel/work/build/x86_64-apple-darwin11/libstdc++-v3/include/valarray:1233:1: error: while writing precompiled header: No space left on device
 } // namespace
 ^
/opt/local/var/macports/build/_opt_mports_dports_lang_gcc6/libgcc-devel/work/build/x86_64-apple-darwin11/libstdc++-v3/include/valarray:1233:1: fatal error: can't write PCH file: No space left on device
compilation terminated.

How much disk space is free? Can we get more? Or can we investigate if there's anything taking up space unnecessarily?

I imagine most of the disk space will be taken up by /opt/local/var/macports/software, because the way we have the buildbot builders set up is that all ports remain installed, but deactivated. This means that as more and more ports get added to MacPorts over time, more and more disk space is needed on the builders.

I'm not certain how the buildbot builder's disk volumes are set up... I thought a SAN or file share might be involved, but I'm not sure what parts are on that and what parts are on each builder's local disk volumes.

Is there a system in place that monitors our servers' disk space? It's important that we resolve low disk space issues before we run out completely, as happened here.

Change History (10)

comment:1 Changed 9 years ago by ryandesign (Ryan Carsten Schmidt)

Cc: jeremyhu@… added

This problem did not appear again for a few weeks and I thought maybe it was mysteriously resolved (as the disk space shortage on the packages server was mysteriously resolved in #48174), but Jeremy H reported that the issue resurfaced Sunday when building a large port (clang-3.7). Keith, can you please respond to this ticket in some way? Thanks.

comment:2 Changed 9 years ago by ryandesign (Ryan Carsten Schmidt)

We are still waiting on a response from the Mac OS Forge admin to this issue.

Lack of disk space may have now caused the registry to become corrupted; see #48486.

comment:3 in reply to:  2 ; Changed 9 years ago by ryandesign (Ryan Carsten Schmidt)

Replying to ryandesign@…:

We are still waiting on a response from the Mac OS Forge admin to this issue.

Correction: There was some further communication on this issue which took place in private email, I believe unintentionally. I'll copy it here:


From: Keith Dart
Subject: Re: [MacPorts] #48173: Lion buildbot builder: No space left on device
Date: July 17, 2015 at 3:56:22 PM CDT
To: MacPorts Dev
Cc: Ryan Schmidt, Mac OS Forge Admin, MacPorts Tickets, Jeremy Huddleston Sequoia

I did look at it and there seems to be 20GB free. That still might not be enough for some builds, I guess. I did enlarge the virtual disk by another 20 GB. But so far I’m been unable to resize the file system in the guest. I tried the disk utility and it seems to see the extra size and let me expand it by dragging a pull handle. Then I apply it, and the log entries claim it was successful. However, it is not actually resized.

So I guess another option is to boot into another image, such as net booting. Then try to resize the offline partition. But there seems to be no way to do that with these VMs. So right now I’m stuck at this point. Any suggestions?

BTW, I’m curious, does anybody still use these older OS X?


From: Jeremy Huddleston Sequoia
Subject: Re: [MacPorts] #48173: Lion buildbot builder: No space left on device
Date: July 17, 2015 at 8:12:28 PM CDT
To: Keith Dart
Cc: Ryan Schmidt, MacPorts Dev, Mac OS Forge Admin, MacPorts Tickets

Yeah, we still have a bunch of users staying on Snow Leopard because of various reasons. Lion was the last OS that supported EFI32, so there are some users who can't upgrade past Lion.

That being said, you can probably netboot to a newer OS image and use Disk Utility there to resize it. Another way of doing that if the netboot approach fails is to shutdown the VM and add its disk to another VM (selecting "shared" when adding it). Then resize it in the other VM and remove it (selecting "don't delete" when removing).


From: Joshua Root
Subject: Re: [MacPorts] #48173: Lion buildbot builder: No space left on device
Date: August 1, 2015 at 11:53:16 AM CDT
To: Keith Dart
Cc: Jeremy Huddleston Sequoia, Ryan Schmidt, MacPorts Dev, Mac OS Forge Admin

Any updates on this? It looks like the lack of space may now be causing the registry db to break.

Worst case, you should be able to create a new larger disk image and clone the old one onto it, or make a Time Machine backup and restore it onto a bigger volume.

comment:4 in reply to:  3 Changed 9 years ago by ryandesign (Ryan Carsten Schmidt)

Replying to keith_dart@…:

I did look at it and there seems to be 20GB free. That still might not be enough for some builds, I guess. I did enlarge the virtual disk by another 20 GB. But so far I’m been unable to resize the file system in the guest. I tried the disk utility and it seems to see the extra size and let me expand it by dragging a pull handle. Then I apply it, and the log entries claim it was successful. However, it is not actually resized.

What software is running the buildbot slave virtual machines? If it happens to be VMware ESXi, check if it's up to date: there was a bug in ESXi that prevents resizing disks of guests running OS X 10.6, 10.7, and 10.8 which was fixed in ESXi 5.1 update 3 and 5.5 update 2.

comment:5 Changed 8 years ago by mojca (Mojca Miklavec)

Cc: mojca@… added

Cc Me!

comment:6 Changed 8 years ago by mojca (Mojca Miklavec)

Any news about this?

comment:7 Changed 8 years ago by jeremyhu (Jeremy Huddleston Sequoia)

Do we care any more? Lion hasn't been a supported platform for a few years now. Maybe we should just leave the builder retired (and also retire 10.6 and 10.8 while we're at it).

comment:8 Changed 8 years ago by mojca (Mojca Miklavec)

I do care. Unless we are short of resources, especially the 10.6 buildbot is very valuable in early discovery of problems that would otherwise have to be reported by users. I use some virtual machines with 10.6 to create binaries to support i386 & ppc all the way back to 10.5 (for others). And I still have two computers running 10.7 with basically no serious issues (other than perhaps support for C++14 and suboptimal support for C++11). One of the two is still my main machine and all the software I need is running perfectly fine on it. It would be perfectly fine if we would switch to libc++ instead of libstdc++ though and start supporting that configuration with a higher priority.

Getting the latest buildbot up and running is certainly our highest priority and should be taken care of before worrying about 10.7, but honestly I wouldn't mind having another buildbot for 10.5/ppc.

Unless the resources are tight and we need to minimize the usage & burden on webmaster, I would fully support having the full stack of versions available for the buildbots.

comment:9 in reply to:  8 Changed 8 years ago by danielluke (Daniel J. Luke)

Replying to mojca@…:

And I still have two computers running 10.7 with basically no serious issues

not getting security updates from your OS vendor is a serious issue.

comment:10 Changed 8 years ago by mojca (Mojca Miklavec)

Resolution: fixed
Status: newclosed

After the transition to the new infrastructure on fresh build slaves this is no longer an issue.

Note: See TracTickets for help on using tickets.