buildbot: don't add ports that declare not to support ppc to the build queue of 10.5-ppc

[mojca (Mojca Miklavec)]

The 10.6-i386 buildbot avoids building many packages:

if [[ $exclude -eq 0 && "${os_version}" = "10.6" && "${is_64bit_capable}" = "0" ]]; then
    supported_archs=$("${option_prefix}/bin/port-tclsh" "${thisdir}/tools/supported-archs.tcl" "${port}")
    if [[ -n "${supported_archs}" && ! ("${supported_archs}" == *"x86_64"* && "${supported_archs}" == *"i386"*) ]]; then
        exclude=1
        exclude_reasons+=("the ${os_version}_x86_64 builder will build it")
    fi
fi

I believe that a similar strategy could be used to avoid building packages that are declared not to be supported on PPC.

See ​an example of the libvpx port which doesn't even start to build:

Error: libvpx cannot be installed for the configured build_arch 'ppc' because it only supports the arch(s) 'x86_64 i386'.
Error: Processing of port libvpx failed
Build of 'libvpx' failed.

[larryv (Lawrence Velázquez)]

Most of this fiddling should be done in the Buildbot configuration and not in mpbb.

[mojca (Mojca Miklavec)]

(Thanks for fixing my stupid mistake.)

No, I don't believe that the buildbot can possibly do that. Buildbot is just a stupid python script that doesn't know anything about ports unless we create an up-to-date database of all ports with all the required meta information on the build master ... or install MacPorts on the build master, but then it has to be able to switch between different OS versions for the portindex etc.

Until we implement that, the decision about whether or not a port is installable should be done by the build slave in my opinion.

[larryv (Lawrence Velázquez)]

The build step that invokes mpbb list-subports can specify the architecture(s) it can build, and mpbb can compare that to the ports’ supported_archs and filter appropriately.

mpbb’s knowledge of which buildslaves we are running (and any limitations thereof) is needlessly tight coupling that we should eliminate.

[mojca (Mojca Miklavec)]

I only partially agree. The above mentioned step only avoids building ports like wine on 10.6-i386 because 10.6-x86_64 already builds it and you cannot delegate the job of building wine to the i386 builder as you wouldn't get all the dependencies built universally. It's not really clear to me how you would simplify the code by telling the i386 builder to build for i386. Just about the only thing that could be simplified is the need for testing:

is_64bit_capable="$(sysctl -n hw.cpu64bit_capable)"
...
if [[ ... "${is_64bit_capable}" = "0"... ]]; then

For 10.5 we would currently run uname -m. With your suggested changes we would have to call

mpbb list-subports -arch ppc ...

or provide an environmental variable, but I don't see how that simplifies things in any way.

[jmroot (Joshua Root)]

One note: uname -m is not correct here (or in the buildbot README.md). It tells you the architecture of the kernel, and K64 is sufficient but not necessary for running 64-bit programs.

[mojca (Mojca Miklavec)]

That's already nitpicking. I'm talking about determining whether computer's architecture is PPC.

[mojca (Mojca Miklavec)]

The problem with tools/supported-archs.tcl is that it only returns something when the port specifies supported_archs explicitly.

So one can get i386 ppc or i386 x86_64 or i386 ppc x86_64, ... or noarch or an empty string as the answer. I'm not sure how to reliably tell whether a port builds on PPC. Just guessing:

• If the answer is not empty and not noarch, then ppc must be present, else skip building the port.

[mojca (Mojca Miklavec)]

Off-topic.

I just wanted to add one strange observation:

> time /opt/local/bin/port-tclsh ./tools/supported-archs.tcl wine
i386

real	0m26.122s

What's the deal with the super long waiting time? Sure, I don't have SSD yet and any subsequent run is much faster, but I'm still completely puzzled why it was so long in the first place (and we are experiencing super long time for trivial tasks on the buildbot all the time).

[ryandesign (Ryan Carsten Schmidt)]

Replying to mojca:

The 10.6-i386 buildbot avoids building many packages:

if [[ $exclude -eq 0 && "${os_version}" = "10.6" && "${is_64bit_capable}" = "0" ]]; then
    supported_archs=$("${option_prefix}/bin/port-tclsh" "${thisdir}/tools/supported-archs.tcl" "${port}")
    if [[ -n "${supported_archs}" && ! ("${supported_archs}" == *"x86_64"* && "${supported_archs}" == *"i386"*) ]]; then
        exclude=1
        exclude_reasons+=("the ${os_version}_x86_64 builder will build it")
    fi
fi

This code is there because it is buildbot-specific. The i386 builder could build i386-only ports, but we don't want it to, because want to build them on the x86_64 builder, so that their dependencies get built universal.

I believe that a similar strategy could be used to avoid building packages that are declared not to be supported on PPC.

We've discussed before that this is a generic MacPorts problem, not a buildbot-specific one. Instead of adding code for this to our buildbot system, we should write code in MacPorts base so that a client (e.g. a user or the buildbot) can determine in advance whether a port is buildable on a particular system. Then the buildbot can simply avoid attempting to build ports that are known not to be buildable on a particular system.

[mojca (Mojca Miklavec)]

I fully agree that the base should tell whether a port is "buildable". But until we have a working code to do that (which might take quite a while - both discussing and implementing), it might be a matter of just a few lines of code to avoid building some ports – similar to what we do with obsolete ports.

[jmroot (Joshua Root)]

Replying to mojca:

Off-topic.

I just wanted to add one strange observation:

> time /opt/local/bin/port-tclsh ./tools/supported-archs.tcl wine
i386

real	0m26.122s

What's the deal with the super long waiting time? Sure, I don't have SSD yet and any subsequent run is much faster, but I'm still completely puzzled why it was so long in the first place (and we are experiencing super long time for trivial tasks on the buildbot all the time).

I would again suggest becoming familiar with using the profiler package. This series of blog posts isn't a bad introduction, though the sample code may need some tweaking:

​https://wuhrr.wordpress.com/2011/01/14/simple-profile-for-your-tcl-script/ ​https://wuhrr.wordpress.com/2011/01/15/profiling-your-tcl-show-profilers-summary/ ​https://wuhrr.wordpress.com/2011/01/19/profiling-your-tcl-output-to-csv/

[mojca (Mojca Miklavec)]

(Sorry to everyone for the off-topic discussion.)

Thank you, I ended up with the following script (suggestions welcome):

#!/opt/local/bin/port-tclsh
 
# File: profile_me
package require profiler
profiler::init

proc srcx {} {
  set argv $::argv
  set argc $::argc

  set file   [lindex $::argv 0]
  set ::argv [lreplace $::argv 0 0]
  set ::argc [llength $::argv]
  set code [catch {uplevel [list source $file]} return]

  set ::argv $argv
  set ::argc $argc
  return -code $code $return
}

srcx

# Display profiler info
puts "\n[string repeat - 40]"
puts [::profiler::print]

and time ./profileme.tcl ./tools/supported-archs.tcl wine

but I need to wait. There seems to be some cache somewhere and only the first call is super slow, any subsequent calls are fast.

[mojca (Mojca Miklavec)]

I absolutely love the results of profiling (this was a fast run though):

Functions that are sorted by exclusiveRuntime:
-   15312828 - ::lshift
-   12601898 - ::puts
-    7224617 - ::ldindex
-    2379654 - ::lpush
-    1840049 - ::lpop
-    1336408 - ::_source_is_snapshot
-     816458 - ::ditem_key
...
-          0 - ::intersect
-    -138321 - ::macports::setxcodeinfo
-    -270916 - ::macports::set_developer_dir
-    -479661 - ::mportinit
-    -554829 - ::_mports_load_quickindex
-    -558463 - ::mportclose
-    -564781 - ::macports::ui_init
-    -666930 - ::macports::getindex
-    -688013 - ::mportlookup
-   -1302686 - ::msgcat::Init
-   -2878285 - ::mportopen
-  -26476148 - ::try

We have negative times, yay!

[mojca (Mojca Miklavec)]

This was fixed in 258d4167.

Long delays were fixed (worked around) as well.