Opened 12 years ago

Closed 12 years ago

#33256 closed defect (fixed)

pypy: gcc uses too much memory

Reported by: macports.org@… Owned by: jmroot (Joshua Root)
Priority: Normal Milestone:
Component: ports Version: 2.0.3
Keywords: Cc: ryandesign (Ryan Carsten Schmidt)
Port: pypy

Description

Yesterday and today I tried updating my pypy 1.7 to 1.8, on a 2010 MBP with 8GB RAM.

In both cases, the machine got swapped to death. The second time around I looked into what was happening, and I observed the driver Python process using ~2.5GB RAM (real memory) and 4 CC processes using 1~1.5GB RAM each. While using multiple jobs is good when the translation is CPU bound, forcing the machine to do everything on-disk because everything gets swapped out is not.

I looked into it more, and apparently I'm hitting a wontfixed GCC 4.2 bug which makes GCC eat insane amounts of memory when -O1 or more is enabled. This may not be an issue in Lion depending on its GCC version.

As a result, I think forcing --make-jobs to 1 (or removing SMP compilation altogether) in 10.6 would be a good idea.

Change History (3)

comment:1 Changed 12 years ago by ryandesign (Ryan Carsten Schmidt)

Cc: ryandesign@… added; jmr@… removed
Owner: changed from macports-tickets@… to jmr@…
Summary: Disable SMP compilation for Pypy translation (on 10.6)pypy: Disable parallel build because gcc uses too much memory

Apple will never ship a version of GCC newer than 4.2.1 because GCC changed its license to GPL 3 and Apple does not like that license. As of Xcode 4.2, there is no GCC in Xcode; with Xcode 4.0 and 4.1 we use llvm-gcc-4.2 instead of GCC; in Xcode 4.2 we use clang.

In other ports that use lots of memory like this at compile time, I've simply disabled the parallel build. See #29547. We could potentially revisit whether we should only do that when the compiler is GCC.

comment:2 in reply to:  1 Changed 12 years ago by macports.org@…

Replying to ryandesign@…:

Apple will never ship a version of GCC newer than 4.2.1 because GCC changed its license to GPL 3 and Apple does not like that license. As of Xcode 4.2, there is no GCC in Xcode; with Xcode 4.0 and 4.1 we use llvm-gcc-4.2 instead of GCC; in Xcode 4.2 we use clang.

In other ports that use lots of memory like this at compile time, I've simply disabled the parallel build. See #29547. We could potentially revisit whether we should only do that when the compiler is GCC.

An alternative could be to force gcc to 4.0 (in 10.6 cc is gcc 4.2, but 4.0 is also available, not sure about 10.7) for these problematic ports if it's compatible, that's what I ended up doing (manually) in order to compile pypy, via configure.cc=gcc-4.0.

Disabling parallel build work, although it's probably not as good since Pypy's translation already burns through a fuckton of RAM without taking gcc in account (hitting gcc's bug still results in the GCC process growing to the gigabyte range)

Side-note, I saw that translating pypy 1.8 via 1.7 was disabled in the portfile (while looking into knobs to get my translation working), I tried re-enabling it and it worked without any issue (using macport's 1.7, not a separate source checkout or anything).

comment:3 Changed 12 years ago by jmroot (Joshua Root)

Resolution: fixed
Status: newclosed
Summary: pypy: Disable parallel build because gcc uses too much memorypypy: gcc uses too much memory
Note: See TracTickets for help on using tickets.