hdf5 netcdf version problem

[basmac]

Can netcdf 4.4.0 be put back as an option to install as well as hdf5 1.8.16 as an option to select. When we use the latest updates we get for a fortran program

netid, reading from 0 /Projects/Reforecast2/netcdf/refcstv2_precip_ccpav3_012_to_024.nc netcdf error NetCDF: HDF error

For ncdump, we get the same errors for the latest versions.

When we go back to previous macports install of previous version the program and ncdump then works.

There is also this problem

​https://github.com/Unidata/netcdf4-python/issues/549

[basmac]

It looks like this is a known hdf problem where 1.10 generates incompatible data files. Can we get 1.8.16 put back as an option to install ASAP ? thanks barry

[tenomoto (Takeshi Enomoto)]

I am aware of the problem. The upstream developers are addressing the problem and I asked the maintainer of hdf5 to create something like hdf5-18. I'd like to see what happens in the next few days to decide whether to follow hdf5 or step back and stick with the old version of hdf5.

[basmac]

Thanks for the information. We would like the hdf-18 to be implemented ASAP.

[mamoll (Mark Moll)]

I am the maintainer of hdf5. I apologize for the mess that the version bump of hdf5 has caused. I am currently traveling and don't know when I'll be able to fix this. Takeshi, feel free to restore the previous version of hdf5 as a new port hdf5-18. In fact, feel free to add yourself as maintainer to hdf5 as well. Since you are maintaining several ports that critically depend on hdf5, that might not be a bad idea. We used to use hdf5 heavily internally for a couple projects, but lately I haven't used it much myself. From my limited testing of hdf5 1.10 I didn't anticipate all these difficulties.

[basmac]

I appreciate your time on this problem, not caused by macports, it would be great to get hdf5-18 as a port we can install.

[tenomoto (Takeshi Enomoto)]

Thank you Mark for maintaining such an important port and keep it up-to-date.

I'm trying to create hdf5-18, but I stumbled at destroot and I need to take time to find the reason why it is attempted to install the library directly to /opt/local/lib.

Rather than creating hdf5-18, I wonder if we could just roll back to 1.8.16 and create hdf5-devel for those who need the bleeding edge. If not, we don't create hdf5-devel and wait until most of the package is ready for 1.10.x.

[mamoll (Mark Moll)]

This is also fine with me. If you want to do that, please remember to add an epoch field, so that people who have already updated to hdf5 1.10 will upgrade to hdf5 1.8.

[seanfarley (Sean Farley)]

I'm also fine with rolling back, for what it's worth.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #51151.

[tenomoto (Takeshi Enomoto)]

I changed my mind and created hdf5-18 in r147799. I committed netcdf that depends upon hdf5-18 in r147800. I leave the ticket open for some time.

[gnurser (George Nurser)]

Hi, the two different versions of hdf5 are causing me problems with netcdf: I also have installed py27-h5py and py27-tables which depend on straight hdf5 (v 1.10): this means that I have the hdf5 1.10 headers installed in /opt/local/include, while the hdf5-18 headers are in /opt/local/include/hdf5-18. Unfortunately, the build of netcdf is taking the hdf5 1.10 headers from /opt/local/include but linking to hdf5-18, giving the mismatch warning & crashing.

I worked round the problem by moving the hdf5 1.10 headers into a directory /opt/local/include/hdf5-110, but a cleaner solution is required. netcdf 4.4.1 should be out very soon, so hopefully we won't need hdf-18 anymore ...

[basmac]

The two versions causes issues for us as well for programs like grads. We are using a separate netcdf rc1 to test out the release. Hopefully 4.4.1 is out soon and we can get out of version hell.

[tenomoto (Takeshi Enomoto)]

I corrected the specification of the include path in r147988. Now the path to hdf5-18 should be used during compilation of netcdf. The ports that depend on hdf5 could

• continue to depend on hdf5 if the source is compatible with the new 1.10.x interface.
• continue to depend on hdf5 with -DH5_USE_16_API or -DH5_USE_18_API.
• switch to hdf5-18 by specifying the include and library paths.

Of course a single hdf5 port is preferable. But some projects are active and adopt the new interface in short time but some are slow or frozen. We need hdf5-18 for a while. I will be able to switch to hdf5 when netcdf-4.4.1 is out. Note that it is a tentative fix.

[gnurser (George Nurser)]

Hi, Trying to upgrade to netcdf @4.4.0_3+dap+netcdf4+openmpi
But still having a problem here: the types of MPI_Comm & MPI_Info are redefined by ncdispatch.h from their values in mpi.h; this is giving errors
Also values of MPI_COMM_WORLD & MPI_INFO_NULL are being redefined by nc4dispatch.h from values set in mpi.h; this is giving warnings.

See attached main.log added to comment 15 above.

[basmac]

Can macports maintainers make it so we can install hdf5 1.10 or 1.8.16 by specifying the version. We have lots of separate issues with 1.10, and the side install of 1.8.16 won't work for us ?

[basmac]

This is getting urgent for us we cannot use hdf5 1.10, as its really an alpha release, and installing it as the default means all other macports ports that use hdf5 are useless to us. Please make the hdf5-18 install as the default instead of 1.10, so we can use the latest releases of other software - at present we are using a macports snapshot from December.

[basmac]

AN example of the problems with hdf5 1.10, here is a hdf5dump of generated netcdf files using

hdf5 1.8.16
mac56:~/hdf5 2> foreach f ( netcdf* )
foreach? /opt/local/bin/h5dump -BH $f | grep SUPERBLOCK
foreach? end
h5dump error: unable to open file "netcdf3.nc"
   SUPERBLOCK_VERSION 2
   SUPERBLOCK_VERSION 2
   SUPERBLOCK_VERSION 2
   SUPERBLOCK_VERSION 3
   SUPERBLOCK_VERSION 3
mac56:~/hdf5 5> ls -l
total 240
drwxr-xr-x  2 dallured  rain    102 Apr 26 18:22 logfiles
-rw-r--r--  1 dallured  rain    384 Apr 18 18:03 netcdf3.nc
-rw-r--r--  1 dallured  rain   6480 Apr 26 15:39 netcdf4-classic.nc
-rw-rw-r--  1 dallured  rain  20173 Apr  5 09:22 netcdf4-classic.rean.nc
-rw-r--r--  1 dallured  rain   6480 Apr 18 18:04 netcdf4.nc
-rw-r--r--  1 dallured  rain  36882 Apr  7 12:10 netcdf4.superblock-3.nc
-rw-r--r--  1 dallured  rain  36882 Apr  7 12:10 netcdf4.superblock-3.nc4
mac56:~/hdf5 6>

then using the default macports install of hdf5 1.10
mac27:~/hdf5 9> foreach f ( netcdf* )
foreach? /opt/local/bin/h5dump -BH $f | grep SUPERBLOCK
foreach? end
h5dump error: unable to open file "netcdf3.nc"
h5dump error: unable to open file "netcdf4-classic.nc"
h5dump error: unable to open file "netcdf4-classic.rean.nc"
h5dump error: unable to open file "netcdf4.nc"
h5dump error: unable to open file "netcdf4.superblock-3.nc"
h5dump error: unable to open file "netcdf4.superblock-3.nc4"
mac27:~/hdf5 12>

Which is why we cannot use the current macports default hdf5 install.

[massonseb]

Cc Me!

[tenomoto (Takeshi Enomoto)]

Replying to gnurser@…:

Trying to upgrade to netcdf @4.4.0_3+dap+netcdf4+openmpi
But still having a problem here: the types of MPI_Comm & MPI_Info are redefined by ncdispatch.h from their values in mpi.h; this is giving errors
Also values of MPI_COMM_WORLD & MPI_INFO_NULL are being redefined by nc4dispatch.h from values set in mpi.h; this is giving warnings.

Thanks for reporting, but this is separate from the hdf-1.10 issure. The redefinition of symbols can be avoided by supplying -DHDF5_PARALLEL in CFLAGS. Committed in r148314.

[tenomoto (Takeshi Enomoto)]

Replying to barry.j.mcinnes@…: I was able to dump air.mon.1981-2010.ltm.nc of NCEP/NCAR reanalysis from NOAA/ESRL/PSD, but you cannot with your data. This seems to be a problem of hdf5 port, apart from that of netcdf.

Some ports have been updated to hdf5-1.10 and more will, but hdf5-1.10 has a number of issues. That's why I created hdf5-18. It installs the binaries as well. You could use h5dump-18 in ${prefix}/bin while we wait for the problems with hdf5-1.10 to be fixed by the upstream developers. If you believe it is a problem of hdf5 port, please file a ticket.

I am closing this ticket because the hdf5 and netcdf version incompatibility is fixed by hdf5-18.