PortGroup for ports requiring Python that can't use the Python build system

[RJVB (René Bertin)]

There are projects that require Python in some form, for building or because they build Python extensions (often as part of a more complex build) without using one of the regular Python build systems.

These usually need knowledge about the custom Python install layout in MacPorts. The logical approach would be to include the Python PG to obtain the necessary definitions, but that would redefine the build system in a way that is probably impossible to revert.

The other day I threw together a personal port for d-feet, which obliged me to copy a number of definitions from the Python PG.

It'd be much more efficient if those definitions and whatever other common code there is were available via a PortGroup, one that would probably also be included by the current Python PG.

In addition I would like to see some feature in that new PG for ports that just require a Python interpreter for their build. Now that Python 2 seems to be in the process of being dropped more and more for this purpose it would be ideal if there were a "system" python3 port for that purpose. Most users probably don't have a need for having multiple python3 installs and every interest to have only a single such install that gets updated as required.

In my port:audacity this build dependency is handled implicitly through the build (and fetch) dependency on port:git. Not ideal, IMHO.

In the meantime, it probably doesn't matter exactly which python3 version is used for building such ports (say, port:exiv2) so the proposed PG could include a bit of logic that makes them depend on the newest available interpreter (AFAIK that's more or less how clang dependencies are handled):

python.build_with_version_suggested 3.11
# ...
set python.build_with_version ${python.build_with_version_suggested}
for {set v [lindex [split ${python_version} .] 1]} {${v} >= 0} {incr v -1} {
    if {[file exists ${prefix}/bin/python3.${v}]} {
        ui_debug "Python 3.${v} is available; using it as a depends_build"
        set python.build_with_version "3.${v}"
        break
    }
}

depends_build-append port:python${python.build_with_version}

Or, using a do/while construct (e.g. ​https://wiki.tcl-lang.org/page/do..while) :

set v [lindex [split ${python.build_with_version} .] 1]
do {
    if {[file exists ${prefix}/bin/python3.${v}]} {
        ui_debug "Python 3.${v} is available; using it as a depends_build"
        set python.build_with_version "3.${v}"
        break
    }
} while {[incr v -1]}

[ryandesign (Ryan Carsten Schmidt)]

Replying to RJVB:

It'd be much more efficient if those definitions and whatever other common code there is were available via a PortGroup, one that would probably also be included by the current Python PG.

Each port still needs to communicate that information to the build system in a way that the build system understands. Some use the PYTHON environment variable at configure time. Some use a configure script that takes a flag like --with-python=/path/to/python. Some use cmake or meson which use arguments like -DPYTHON_EXECUTABLE=/path/to/python or -DPYTHON_EXEC=/path/to/python or -Dpython=/path/to/python. Some need the python path inserted into various source files (e.g. in #! lines) with reinplace. Some need to be told the library and include paths separately. So there is no generic one-size-fits-all solution that could be put into a portgroup and it's not that hard for ports to construct the necessary paths.

In addition I would like to see some feature in that new PG for ports that just require a Python interpreter for their build. Now that Python 2 seems to be in the process of being dropped more and more for this purpose it would be ideal if there were a "system" python3 port for that purpose. Most users probably don't have a need for having multiple python3 installs and every interest to have only a single such install that gets updated as required.

There has never been an intention to offer a "system" python3 port. It would directly conflict with the port select mechanism.

In my port:audacity this build dependency is handled implicitly through the build (and fetch) dependency on port:git. Not ideal, IMHO.

Right, if you use python (or anything) in your port, you have to declare the dependency in your port. Don't rely on it being there due to a dependency of another port because that other port could change in the future.

In the meantime, it probably doesn't matter exactly which python3 version is used for building such ports (say, port:exiv2) so the proposed PG could include a bit of logic that makes them depend on the newest available interpreter (AFAIK that's more or less how clang dependencies are handled):

That's contrary to the goal of ReproducibleBuilds. We do have a recommended python version. Currently it's 3.11. exiv2 uses that. You can read python-1.0.tcl to discover what the current recommended version is. It ​will be updated to the latest stable version each January 1st. Ports should be manually updated to that version as soon as possible after that. It should not be done automatically because there are any number of reasons why a port might be incompatible with a new version of python; for each port it needs to be tested individually.

Most of what you've written here is not actionable as a ticket because it conflicts with established best practices. You could open discussions for these items on the mailing list if you'd like more input on whether these proposals could be made to work somehow.

[RJVB (René Bertin)]

That's contrary to the goal of ReproducibleBuilds.

That's almost certainly not true for a good number of the concerned ports e.g. if python is just used to generate a file used in the build, or for a build system used by a "vendored" dependency. It's not in the project's interest that the python version used makes a difference.

I'm not proposing a mandatory feature, so ports where the python version does make a difference could and should refrain from using it.