nco/ncks and netcdf

[basmac]

The current nco and netcdf on macports, currently take more than 2 hours (killed), whereas the versions on linux for the same files and command take 13 seconds. A similar nccopy command takes 13 seconds on the same mac for the same files.

The linux setup is

/usr/local/bin/ncks -r
NCO netCDF Operators version "4.4.5" last modified 2014/07/15 built Aug 29 2014 on linux64.psd.esrl.noaa.gov by root
ncks version 4.4.5
Linked to netCDF library version 4.3.3, compiled Feb 24 2015 07:21:57
Copyright (C) 1995--2014 Charlie Zender
This program is part of NCO, the netCDF Operators.
NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY
You can redistribute and/or modify NCO under the terms of the
GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file
GPL: http://www.gnu.org/copyleft/gpl.html
LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE
Homepage: http://nco.sf.net
User Guide: http://nco.sf.net/nco.html
Configuration Option:	Active?	Meaning or Reference:
Check _FillValue	Yes	http://nco.sf.net/nco.html#mss_val
Check missing_value	No	http://nco.sf.net/nco.html#mss_val
DAP clients		Yes	http://nco.sf.net/nco.html#dap
Debugging: Custom	No	Pedantic, bounds checking (slowest execution)
Debugging: Symbols	No	Produce symbols for debuggers (e.g., dbx, gdb)
GNU Scientific Library	Yes	http://nco.sf.net/nco.html#gsl
HDF4 support		Unknown	http://nco.sf.net/nco.html#hdf4
Internationalization	No	http://nco.sf.net/nco.html#i18n (pre-alpha)
MPI parallelization	No	http://nco.sf.net/nco.html#mpi (beta)
netCDF3 64-bit files	Yes	http://nco.sf.net/nco.html#lfs
netCDF4/HDF5 available	Yes	http://nco.sf.net/nco.html#nco4
netCDF4/HDF5 enabled	Yes	http://nco.sf.net/nco.html#nco4
OpenMP SMP threading	No	http://nco.sf.net/nco.html#omp
Optimization: run-time	No	Fastest execution possible (slowest compilation)
Parallel netCDF3	No	http://nco.sf.net/nco.html#pnetcdf (pre-alpha)
Regular Expressions	Yes	http://nco.sf.net/nco.html#rx
Shared libraries built	No	Small, dynamically linked executables
Static libraries built	Yes	Large executables with private namespaces
UDUnits conversions	Yes	http://nco.sf.net/nco.html#udunits
UDUnits2 conversions	Yes	http://nco.sf.net/nco.html#udunits

The macports is

ncks -r
NCO netCDF Operators version "4.4.7" last modified 2014/11/05 built Jan  6 2015 on mac73.psd.esrl.noaa.gov by root
ncks version 4.4.7
Linked to netCDF library version 4.3.2, compiled Feb  3 2015 08:18:38
Copyright (C) 1995--2014 Charlie Zender
This program is part of NCO, the netCDF Operators.
NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY
You can redistribute and/or modify NCO under the terms of the
GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file
GPL: http://www.gnu.org/copyleft/gpl.html
LICENSE: http://nco.cvs.sf.net/nco/nco/LICENSE
Homepage: http://nco.sf.net
User Guide: http://nco.sf.net/nco.html
Configuration Option:	Active?	Meaning or Reference:
Check _FillValue	Yes	http://nco.sf.net/nco.html#mss_val
Check missing_value	No	http://nco.sf.net/nco.html#mss_val
DAP clients		Yes	http://nco.sf.net/nco.html#dap
Debugging: Custom	No	Pedantic, bounds checking (slowest execution)
Debugging: Symbols	No	Produce symbols for debuggers (e.g., dbx, gdb)
GNU Scientific Library	Yes	http://nco.sf.net/nco.html#gsl
HDF4 support		Unknown	http://nco.sf.net/nco.html#hdf4
Internationalization	No	http://nco.sf.net/nco.html#i18n (pre-alpha)
MPI parallelization	No	http://nco.sf.net/nco.html#mpi (beta)
netCDF3 64-bit files	Yes	http://nco.sf.net/nco.html#lfs
netCDF4/HDF5 available	Yes	http://nco.sf.net/nco.html#nco4
netCDF4/HDF5 enabled	Yes	http://nco.sf.net/nco.html#nco4
OpenMP SMP threading	No	http://nco.sf.net/nco.html#omp
Optimization: run-time	No	Fastest execution possible (slowest compilation)
Parallel netCDF3	No	http://nco.sf.net/nco.html#pnetcdf (pre-alpha)
Regular Expressions	Yes	http://nco.sf.net/nco.html#rx
Shared libraries built	No	Small, dynamically linked executables
Static libraries built	Yes	Large executables with private namespaces
UDUnits conversions	Yes	http://nco.sf.net/nco.html#udunits
UDUnits2 conversions	Yes	http://nco.sf.net/nco.html#udunits

The command is

ncks -D 3 --fl_fmt=netcdf4_classic -L 5 --overwrite
huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc
/Projects/CMIP5A//Monthly/historical/ACCESS1-0//r1i1p1//huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc 

The file size is 200MB. Maybe an update to netcdf 4.3.3 will speed things up ? Maybe its the chunking size the macports cannot handle ? One thing I note is that the chunck size for the ncks for the huss variable on Mac ends up being: 1872, 145, 192 but using nccopy, it's 1, 145, 192 which is more in line with what I would expect.

[tenomoto (Takeshi Enomoto)]

netcdf has been updated to 4.3.3.1 in r133622 (See #47040). nco has been updated to 4.4.8 in r133623. Do these updates help in anyway?