Opened 7 years ago

Closed 6 years ago

Last modified 6 years ago

#54101 closed enhancement (fixed)

Ports that depend on hdf5-18 should switch to hdf5

Reported by: mf2k (Frank Schima) Owned by: tenomoto (Takeshi Enomoto)
Priority: Normal Milestone:
Component: ports Version:
Keywords: Cc: MarcusCalhoun-Lopez (Marcus Calhoun-Lopez), mamoll (Mark Moll), basmac, Dave-Allured (Dave Allured), petrrr
Port: h4h5tools hdfeos5 netcdf octave-devel py-tables ncarg

Description (last modified by mf2k (Frank Schima))

Ports that depend on hdf5-18 should be updated and tested to work with the newer hdf5 port.

See also #52289.

Port Maintainer status
h4h5tools - 8136a51881ec689cc5ff2a11dafa5bd4b240f0a5/macports-ports
hdfeos5 tenomoto changeset:259fb59a028dcbb823a616c554215e9aa0a325a4/macports-ports
netcdf tenomoto #54177
octave-devel MarcusCalhoun-Lopez Was fixed at some point
py-tables mamoll changeset:1fa97da43102e3502c9381c05999f5bd8dceee61/macports-ports
ncarg tenomoto changeset:8a1abdde9a46af2b26b4f301a72394764e06a4e3/macports-ports

Change History (23)

comment:1 Changed 7 years ago by mf2k (Frank Schima)

Type: defectenhancement

comment:2 Changed 7 years ago by mf2k (Frank Schima)

Description: modified (diff)

comment:3 Changed 7 years ago by basmac

Cc: basmac added

comment:4 Changed 7 years ago by mamoll (Mark Moll)

In 1fa97da43102e3502c9381c05999f5bd8dceee61/macports-ports:

py-tables: update to version 3.4.2, switch from hdf5-18 to hdf5 (see #54101)

comment:5 Changed 7 years ago by mf2k (Frank Schima)

Description: modified (diff)

comment:6 Changed 7 years ago by Dave-Allured (Dave Allured)

Please recall the original reasons for hdf5-18. It is a little hard to tell from the original tickets (ticket:51089 etc), but this alternative port solved two problems:

  1. All hdf5 versions starting with 1.10.0 (2016 March 30) could easily generate new hdf5 data files that were backward incompatible with all older, non-updated hdf5 applications. Hdf5-18 prevented this, and wrote only backward compatible files. The correct long term fix is to (a) insert a specific hdf5 compatibility function call into the calling application; then and only then, (b) upgrade to hdf5 1.10.x. For example, here is a discussion and compatibility fix that was applied in Netcdf version 4.4.1 (2016 June 28). The single function call to H5Pset_libver_bounds did the trick:

https://github.com/Unidata/netcdf-c/issues/250 (2016 April 7)

  1. Hdf5 versions 1.10.0 and 1.10.0-patch1 could not access network-mounted files from Macs. I am told this was specifically due to new usage of the system flock function, and shortcomings of NFS version 3 on Macs. This of course would not affect any Mac users that used only local disk files. Hdf5-18 avoided this problem because it never used flock. Hdf5 version 1.10.1 (2017 April 27) appears to fix this, using a special run-time environment variable. IMO the correct upgrade path is to link only with hdf5 1.10.1 or later, and then remind users to use the environment variable when needed. The 1.10.1 update is forthcoming in ticket:54139, which includes details about the environment variable setting.

I recommend holding off on switching hdf5-18 dependencies to hdf5 until either (a) the format compatibility fix is inserted in the calling packages; or (b) there is some kind of consensus that original package developers and users no longer care about generating backward-incompatible hdf5 data files.

Furthermore, because of (2), I recommend waiting for hdf5 1.10.1 (ticket:54139) before switching any more hdf5-18 dependencies back to hdf5.

comment:7 Changed 7 years ago by Dave-Allured (Dave Allured)

Cc: Dave-Allured added

comment:8 Changed 7 years ago by mf2k (Frank Schima)

Description: modified (diff)

comment:9 Changed 7 years ago by aaschwanden (Andy Aschwanden)

I just re-installed macports from scratch, and only the ports I really need (e.g. hdf5, hdf5-18, netcdf, py27-netcdf4, gdal, ...) and I'm already stuck with the usual hdf5 vs hdf5-18 library mismatch, this time when using netCDF4python. So far I have never been able to have a working macports installation that has hdf5 and hdf5-18 along side. So far my only working solution has been to overwrite all ports that use hdf5-18 and link them against hdf5. Not pretty and it only worked because I haven't use py-tables in a while. Maybe I have to compile things from scratch again.

Let's hope that hdf5 gets bumped up to 1.10.1 and we can get rid of hdf5-18 (I'd happily re-install macports again after that, well worth the trouble.)

comment:10 Changed 7 years ago by seanfarley (Sean Farley)

Now that hdf5 is 1.10.1 and that netcdf works with it, is there anything preventing netcdf from being updated? @tenomoto?

comment:11 in reply to:  10 ; Changed 7 years ago by Dave-Allured (Dave Allured)

Replying to seanfarley:

I concur that netcdf (only) is ready to update to netcdf version 4.4.1.1 (the latest), as well as its dependency to hdf5 1.10.1. Ticket #54177 is in place for this purpose.

Netcdf version 4.4.1.1 (and its predecessor 4.4.1) satisfied my complaint (1) above, backward compatible files. Hdf5 1.10.1 itself resolves my complaint (2), network mounted files.

comment:12 in reply to:  11 ; Changed 7 years ago by seanfarley (Sean Farley)

Replying to Dave-Allured:

But other ports that depend on hdf5-18 still need to put in the H5Pset_libver_bounds, correct?

comment:13 in reply to:  12 Changed 7 years ago by Dave-Allured (Dave Allured)

Replying to seanfarley:

Yes indeed. Thanks for asking.

Each package should be individually verified to be using H5Pset_libver_bounds so as to ensure writing backward compatible files, unless the relevant user community does not care. This is particularly important for packages like Netcdf that are commonly used for public distribution of large volumes of HDF5-based files.

This should be the responsibility of the original package developers, not MacPorts, IMO. See, for example, what the Netcdf developers did:

https://github.com/Unidata/netcdf-c/issues/250 (2016 April 7)

comment:14 Changed 7 years ago by mf2k (Frank Schima)

Cc: tenomoto removed
Owner: set to tenomoto
Status: newassigned

comment:15 Changed 7 years ago by tenomoto (Takeshi Enomoto)

comment:16 Changed 7 years ago by tenomoto (Takeshi Enomoto)

hdffeos5 in [259fb59a028dcbb823a616c554215e9aa0a325a4/macports-ports].

I had problem building ncarg (and h4h5tools) with hdf5.

comment:17 Changed 7 years ago by tenomoto (Takeshi Enomoto)

comment:18 Changed 7 years ago by tenomoto (Takeshi Enomoto)

h4h5tools builds fine with configure.cflags-append -DH5_USE_18_API. It appears that h4toh5 sets the version to the latest.

utils/h4toh5/h4toh5main.c: if (H5Pset_libver_bounds(fapl_id, H5F_LIBVER_LATEST, H5F_LIBVER_LATEST) < 0)

I'll wait for a few days and commit the changes if no objections.

comment:19 Changed 7 years ago by mf2k (Frank Schima)

Description: modified (diff)

comment:20 Changed 7 years ago by petrrr

Cc: petrrr added

comment:21 Changed 6 years ago by mf2k (Frank Schima)

Description: modified (diff)

comment:22 Changed 6 years ago by mf2k (Frank Schima)

Resolution: fixed
Status: assignedclosed

In 8136a51881ec689cc5ff2a11dafa5bd4b240f0a5/macports-ports:

h4h5tools: Update to version 2.2.3

  • Switch to hdf5

Fixes: #54101

comment:23 Changed 6 years ago by mf2k (Frank Schima)

Description: modified (diff)
Note: See TracTickets for help on using tickets.