py38-sip: sip.h installed in "wrong" directory?

[josephsacco]

py38-sip appears to install the include file sip.h in the wrong directory:

/opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8m/sip.h is provided by: py38-sip

On my system

/opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8m contains only sip.h.

All of the other python38 include files are located in

/opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8.

Why is that?

For python2.7, all the include files are located in a single directory:

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7

For python3.7, all the include files are located in a single directory:

/opt/local/Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m

Note the 'm' at the end of the python3.7 include path is absent from the python2.7 include path.

Why is that?

Thoughts?

-Joseph

[michaelld (Michael Dickens)]

The include directory is set by the Python Portgroup. It will vary depending on the version of Python. I do find it odd that for Py38 the 'm' is appended, since that's not the way it is on my install. I'll need to look at how SIP builds to see if maybe the option has changed.

[jmroot (Joshua Root)]

The m ABI flag was dropped in python 3.8.

[michaelld (Michael Dickens)]

sip.h should not be in the python3.8m directory; it should be in the python3.8 directory. Here's the flag we use to set the directory, in the py*-sip Portfile (as taken from the current py38-sip via python3.8 configure.py --help:

    -e DIR, --incdir=DIR
                        where the SIP header file will be installed [default:
                        /opt/local/Library/Frameworks/Python.framework/Version
                        s/3.8/include/python3.8]

So ... I don't know what's going on with your Python install there. Maybe try "uninstall py38-sip", "clean --all py38-sip", and then install it again?

[josephsacco]

Let's see:

• sudo port uninstall py38-sip

Select 'Y' when port complains that uninstalling py38-sip will break some stuff.

• sudo port clean --all py38-sip

Check to see that /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8m has vanished. It has...

• sudo port install py38-sip

• port contents py38-sip | grep include

/opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8m/sip.h

Hmmm...

Let me unpack and patch py38-sip. Yes, I see the '-e' flag when running

python3.8 configure.py -h

In the Portfile I see:

   configure.cmd       ${python.bin} configure.py
    configure.pre_args  -b ${python.prefix}/bin \
                        -d ${python.pkgd} \
                        -e ${python.include} \
                        -v ${python.prefix}/share/sip \
                        -p macx-g++

So where does the value for ${python.include} come from? I assume it comes from the port group, python-1.0.tcl. Poking around in that file I see

default python.include {[python_get_defaults include]

The procedure is defined in the TCL script:

proc python_get_defaults {var} {
    global python.version python.branch python.prefix
    switch -- $var {
        include {
            set inc_dir "${python.prefix}/include/python${python.branch}"
            if {[file exists ${inc_dir}]} {
                return ${inc_dir}
            } else {
                # look for "${inc_dir}*" and pick the first one found;
                # make assumptions if none are found
                if {[catch {set inc_dirs [glob ${inc_dir}*]}]} {
                    if {${python.version} < 30} {
                        return ${inc_dir}
                    } else {
                        return ${inc_dir}m
                    }
                } else {
                    return [lindex ${inc_dirs} 0]
                }
            }
        }

Where is ${python.branch} set?

-Joseph

[reneeotten (Renee Otten)]

Installing from the binary (sudo port install py38-sip) gives indeed the sip.h file in the wrong place

> port contents py38-sip | grep sip.h
  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8m/sip.h

installing from source (sudo port -s install py38-sip) solves this:

> port contents py38-sip | grep sip.h
  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8/sip.h

I think it's related to ticket:59518, which was solved in this commit. Likely py38-sip has never been rebuilt after that change, so probably a simple revision bump will solve this. Michael: does this make sense to you, and, if so, please make that change.

[josephsacco]

Verified on my iMac [still] running Mojave: Building py38-sip from source does the right thing.

Thank you, Renee.

-Joseph

[reneeotten (Renee Otten)]

actually there might be an additional issue in the PortGroup (lines 287-304), where the include directory is set. I don't think it would cause the problem we're seeing here as likely ${python.prefix}/include/python${python.branch} should already exist on the buildbot when building py-sip.

Nevertheless, I suspect line 295 (if {${python.version} < 30} ) should be changed to if {{${python.version} < 30 or ${python.version} > 37} . Correct?

[michaelld (Michael Dickens)]

I'm glad building from source worked.

@reneeotten : The commit you note is Dec 10 & Jan 5. I updated py-sip in early February ... so, a rev-bump won't help.

I do think you're onto the issue, which is that line in the PG, since the 'm' is added only for Py30 through Py37, not for Py38 or presumedly Py39+. Fixing that would indeed require a rev-bump ... like of any Python port that installs includes probably, to get that directory correct in the buildbot ... whew & sigh ...

[michaelld (Michael Dickens)]

​https://github.com/macports/macports-ports/pull/6428

[michaelld (Michael Dickens)]

In 493ffa29839dfd1dee572ae55499176536c821f9/macports-ports (master):

Python PG: fix appending of 'm' suffix to be just 3.0 <= PyVer <= 3.7

Noted in: #60104

Some binaries might need to be rebuilt for this change to take effect.

[reneeotten (Renee Otten)]

In 9708d9051063356d66ed07bc4c0f209670e7e67f/macports-ports (master):

py-sip: revbump for change in Python 3.8 include dir

Closes: #60104

[ryandesign (Ryan Carsten Schmidt)]

Replying to michaelld:

Some binaries might need to be rebuilt for this change to take effect.

Well we need to identify which ones those are and do that. See #60458 for a consequence of not doing that. How do we do it? I could run tar tjf on all the binaries on the server and look for any files installed in /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8m/ and then we could revbump those ports. Is that all that would be needed?

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

See #60458 for a consequence of not doing that.

My mistake, that was due to an unrelated issue.

How do we do it? I could run tar tjf on all the binaries on the server and look for any files installed in /opt/local/Library/Frameworks/Python.framework/Versions/3.8/include/python3.8m/ and then we could revbump those ports. Is that all that would be needed?

I've checked all the private archives; none of them have paths containing "python3.8m". It took longer than I thought, so before checking the much more numerous public archives I should run the script to delete outdated archives so we don't waste time checking them.