py-numpy 2.0.0

[Schamschula (Marius Schamschula)]

We are transitioning from py-numpy 1.26.4 to py-numpy 2.0.0, which was officially released on June 16, 2024.

From the numpy documentation: ​https://numpy.org/devdocs/dev/depending_on_numpy.html#numpy-2-0-specific-advice

So far we have run into the following ports that require >= 2.0.0rc2:

py-astropy 6.1.3: Only needed for PyPI wheels. See: ​https://github.com/macports/macports-ports/commit/9c190e872373cb073b7e3168787575944a054ed5

py-bottleneck 1.4.0: Only needed for PyPI wheels. See: ​https://github.com/macports/macports-ports/pull/25662

py-cartopy 0.23.0: Only needed for PyPI wheels. See: ​https://github.com/macports/macports-ports/commit/cdaae5629bec4ee4f5de51701695d28e8ebef86e

py-matplotlib 3.9.0

py-pyerfa 2.0.1.41: Only needed for PyPI wheels. See: ​https://github.com/macports/macports-ports/commit/1cb2757245ac645d5669299a765635cc9337ba35

py-sunpy 6.0.1: Only needed for PyPI wheels. See: ​https://github.com/macports/macports-ports/commit/6c882c5fe7c78d98ff47974931bc5b8a27dbf733

py-vispy 0.14.3: Only needed for PyPI wheels. See: ​https://github.com/macports/macports-ports/commit/dc64d200891ff5e80281097b7cc8c8e310d81457

py-wavelets 1.7.0

Updates have been held back until py-numpy >= 2.0.0 is available.

I will update this ticket as I run into additional ports.

[reneeotten (Renee Otten)]

latest version of py-matplotlib also requires it, and I have held back the update

[Marius Schamschula <mschamschula@…>]

In a55e7512c9aed21eccf0694a57c5d926031da80b/macports-ports (master):

py-tropycal: revert to version 1.2.1

See: #70251

[stromnov (Andrey Stromnov)]

py-bottleneck 1.4.0 (​https://github.com/pydata/bottleneck/blob/v1.4.0/pyproject.toml#L17)

py-wavelets 1.7.0 (​https://github.com/PyWavelets/pywt/blob/v1.7.0/pyproject.toml#L19)

[reneeotten (Renee Otten)]

I am working on this update and will put up a PR once I've finished up a few remaining things. NumPy v2.0.1 supports PY39, the latest upstream version v2.1.0 supports PY310 and above.

I would propose the following:

1. keep the current version as py-numpy-legacy, this has to conflict with the latest versions as Python packages cannot be installed side-by-side.
2. list all dependents and:

• remove all subports below 39 where possible, if needed going down the dependency chain
• if a port depends on Python < 3.9 and is deemed to be valuable and cannot be removed, then it should depend on pyXY-numpy-legacy
• update all ports if a newer upstream version is available, or if not, rebuild the port with NumPy v2.0 and enable/run the test-suite to verify that it still works correctly. If no issues are reported by using ruff check path/to/code/ --select NPY201 I think we assume that things are likely to work.

I *think* this is the only way it can work, but am open to comments/suggestions.

I will update here with a link to the PR once that is all fixed and we can start adding ports/updates there and once we are fairly convinced most of the things will work, then we can merge. Additionally, once that's the case people who are interested in older systems should verify that this all still works and if not provide patches to make that happen if possible. The build system has changed to meson and upstream is using a vendored version with NumPy specific changes. Our meson port has quite a few changes specifically intended to support older OSes, but integrating these modified meson versions sounds complicated and not something I am willing to invest time in.

[reneeotten (Renee Otten)]

Same considerations/issues apply to py-scipy as well. If we want to make any progress in keeping things up-to-date we will have to drop unsupported Python versions. Also here the build system was switched to meson and there is no point patching around that any longer; is someone wants older systems supported, then please do the work on getting meson to do TheRightThing there. I will try to put up an initial PR for NumPy and SciPy - and will try to clean-up Python subports below 3.9 to make things easier.

However, if there are expectations to keep older Python versions/OSes supported then I am not even going to bother; there is no sustainable way to do that given that upstream aggressively drops things.

[Schamschula (Marius Schamschula)]

It turns out that

[Schamschula (Marius Schamschula)]

Even though py-tifffile specifies numpy 2.1.0 in the release notes, it builds correctly with numpy < 2.0.0.

[Schamschula (Marius Schamschula)]

py-tropycal works with numpy < 2.0.0

[Marius Schamschula <mschamschula@…>]

In bbbcd2096b9409e39855703dd663408ddcef618f/macports-ports (master):

py-bottleneck: update to 1.4.0

Work around PyPI py-numpy >= 2.0.0 requirement
See: ​https://github.com/pydata/bottleneck/blob/master/pyproject.toml
See: #70251

[jjstickel (Jonathan Stickel)]

How is the progress going for updating numpy to 2.X?

[reneeotten (Renee Otten)]

the py-numpy and py-scipy ports were recently completely transitioned to the meson build system with proper support for BLAS/LAPACK libraries and such. So in that sense, we should be ready to look at the update to NumPy v2. It's on my to-do-list but it does involve making sure that all dependent ports build against that new version, and, if not, to update/fix those to resolve that.

My plan is to put up a PR with the NumPy update in the next few days and then port maintainers can test "their" ports to see if they build against it and/or fix them and then add them to the PR. If you're willing to help out in that process that would be great!

[jjstickel (Jonathan Stickel)]

Yes, I can test a PR. More specifically, I can test which numpy/scipy ecosystem ports that I use work with numpy-2 and which need attention.

[Schamschula (Marius Schamschula)]

I've just upgraded to numpy 2.1.3 on my main build machine. Remember numpy >= 2.0.0 is supposed to be backwardly compatible.

[reneeotten (Renee Otten)]

Replying to Schamschula:

I've just upgraded to numpy 2.1.3 on my main build machine. Remember numpy >= 2.0.0 is supposed to be backwardly compatible.

No, ​it is not:

This major release includes breaking changes that could not happen in a
regular minor (feature) release - including an ABI break, changes to
type promotion rules, and API changes which may not have been emitting
deprecation warnings in 1.26.x. Key documents related to how to adapt to
changes in NumPy 2.0, in addition to these release notes, include:
....

You are right that building with NumPy v2 will result in packages that still work with NumPy v1 - but that is irrelevant for MacPorts as we will never provide both versions. What needs to be checked is that -especially older- packages do not rely on changes in NumPy’s Python and C APIs. One way to do that is checking source code with: $ ruff check path/to/code/ --select NPY201.

A compatibility table is ​here, which would help to triage what needs to be done for MacPorts.

[reneeotten (Renee Otten)]

I started doing work in this and can put up a work-on-progress PR. There are several packages though that will not (yet) work but people who want to test things and help going through the remaining packages might find it useful.

[jjstickel (Jonathan Stickel)]

Replying to reneeotten:

I started doing work in this and can put up a work-on-progress PR. There are several packages though that will not (yet) work but people who want to test things and help going through the remaining packages might find it useful.

Can you please provide a link to the PR? I know I could search, but I want to make sure I got the right one.

[reneeotten (Renee Otten)]

Replying to jjstickel:

Replying to reneeotten:

I started doing work in this and can put up a work-on-progress PR. There are several packages though that will not (yet) work but people who want to test things and help going through the remaining packages might find it useful.

Can you please provide a link to the PR? I know I could search, but I want to make sure I got the right one.

there is ​a branch on my fork with the latest NumPy version and a collections of commits to rebuild packages that need to be, and some updates for others where upstream now supports the latest version 2.x Please note that it has been a while since I started this so some packages have seen updates again in the meantime; but the ones that are in the branch should work with NumPy v2 (and thus newer ones as well).

Some commits show "WIP" and those are not (yet) supported; I've attached here a list of packages that are known to not work, the ones that I have checked and need an update/revbump, the remainder of the list still needs to be checked.

[eborisch (Eric A. Borisch)]

For tracking: py-h5py 3.14.0 requires Numpy 2.0 to build — but I'm planning to patch around that and exclude the new/pseudo-optional (NumPy variable-width strings) functionality for now.

[jjstickel (Jonathan Stickel)]

Can we please have some movement on deploying numpy>2 in Macports?

FWIW, py312-numpy @2.2.5_0+gfortran+openblas (from @reneeotten's fork) is working for me. I only had to rebuild scipy. Matplotlib did not need a rebuild.

[Schamschula (Marius Schamschula)]

The only port on my build machine that doesn't work with numpy 2.x.x is napari. I keep activating different versions, as needed:

port installed py313-numpy
The following ports are currently installed:
  py313-numpy @1.26.4_3+accelerate+gfortran
  py313-numpy @2.3.1_0+gfortran
  py313-numpy @2.3.2_0+gfortran (active)

My current py-numpy Portfile is available @ ​https://github.com/Schamschula/macports/blob/master/python/py-numpy/Portfile

[reneeotten (Renee Otten)]

I would certainly be in favor of getting NumPy in MacPorts updated to version 2, it was after all released over a year ago by now. I have updated my ​the branch on my fork with the latest upstream version and updates/revbumps for other packages as needed. Feel free to test with your favorite ports and comment here whether or not things work.

Surely there will things that I have missed and/or packages that do not work (and will likely never work with NumPy 2), but at some point we'll need to just move forward. We can fix things as they come up where possible, or perhaps it's just time to remove old cruft that is not developed or supported anymore by upstream.

Perhaps a message to the -devel mailinglist about this topic would be a good idea?

[Schamschula (Marius Schamschula)]

On further review napari is fine with py313-numpy @2.3.2_0+gfortran+openblas. The problem was that py313-scipy, py313-pandas, and py313-scikit-image had been installed from the build-bot packages, and hence built against py313-numpy @1.26.4_3+accelerate+gfortran.

Once I rebuilt those packages against py313-numpy @2.3.2_0+gfortran+openblas, napari loaded correctly.

[jjstickel (Jonathan Stickel)]

Replying to reneeotten:

I would certainly be in favor of getting NumPy in MacPorts updated to version 2, ... I have updated my ​the branch on my fork with the latest upstream version and updates/revbumps for other packages as needed. Feel free to test with your favorite ports and comment here whether or not things work.

Thanks. The numpy/scipy ecosystem ports that I normally use are working fine. BTW, I could not build pandas without the patch that is in master (it was removed in your branch).

Perhaps a message to the -devel mailinglist about this topic would be a good idea?

Maybe? I feel like you or Michael (as numpy port maintainers) should write the message if you like. But I think the review process of a PR on github ought to be sufficient.

[jjstickel (Jonathan Stickel)]

To help move this along, I went ahead and created a PR: ​https://github.com/macports/macports-ports/pull/29291

[reneeotten (Renee Otten)]

Replying to jjstickel:

To help move this along, I went ahead and created a PR: ​https://github.com/macports/macports-ports/pull/29291

@jjstickel thanks, but as you noted this is based on my PR and I have updated that further earlier today as you might have noticed from my branch. I have close it, and will open my own PR once I have figured out a few more things. At the same time, I will reach out to the -dev mailinglist and post there to ask maintainers / interested parties to test their favorite packages and state my intention to merge an update by the end of this month.

[jjstickel (Jonathan Stickel)]

Replying to reneeotten:

Replying to jjstickel:

To help move this along, I went ahead and created a PR: ​https://github.com/macports/macports-ports/pull/29291

@jjstickel thanks, but as you noted this is based on my PR and I have updated that further earlier today as you might have noticed from my branch. I have close it, and will open my own PR once I have figured out a few more things. At the same time, I will reach out to the -dev mailinglist and post there to ask maintainers / interested parties to test their favorite packages and state my intention to merge an update by the end of this month.

Ok, that works too. I did look at your branch but did not see any updates since two weeks ago, so maybe your current work may not have been pushed (or I am looking in the wrong place). In any case, I will try to be patient, and I'll check in directly before I do anything more.

[reneeotten (Renee Otten)]

In 6c1c6ed8af664e2b7d1a777ae784677a6e683c88/macports-ports (master):

py-numpy: update to 2.3.3

• pin to version 2.0.2 for py39 subport
• pin to version 2.2.6 for py310 subport

Closes: #70251