nco/ncks and netcdf

[basmac]

The current nco and netcdf on macports, currently take more than 2 hours (killed), whereas the versions on linux for the same files and command take 13 seconds. A similar nccopy command takes 13 seconds on the same mac for the same files.

The linux setup is /usr/local/bin/ncks -r NCO netCDF Operators version "4.4.5" last modified 2014/07/15 built Aug 29 2014 on linux64.psd.esrl.noaa.gov by root ncks version 4.4.5 Linked to netCDF library version 4.3.3, compiled Feb 24 2015 07:21:57 Copyright (C) 1995--2014 Charlie Zender This program is part of NCO, the netCDF Operators. NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY You can redistribute and/or modify NCO under the terms of the GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file GPL: ​http://www.gnu.org/copyleft/gpl.html LICENSE: ​http://nco.cvs.sf.net/nco/nco/LICENSE Homepage: ​http://nco.sf.net User Guide: ​http://nco.sf.net/nco.html Configuration Option: Active? Meaning or Reference: Check _FillValue Yes ​http://nco.sf.net/nco.html#mss_val Check missing_value No ​http://nco.sf.net/nco.html#mss_val DAP clients Yes ​http://nco.sf.net/nco.html#dap Debugging: Custom No Pedantic, bounds checking (slowest execution) Debugging: Symbols No Produce symbols for debuggers (e.g., dbx, gdb) GNU Scientific Library Yes ​http://nco.sf.net/nco.html#gsl HDF4 support Unknown ​http://nco.sf.net/nco.html#hdf4 Internationalization No ​http://nco.sf.net/nco.html#i18n (pre-alpha) MPI parallelization No ​http://nco.sf.net/nco.html#mpi (beta) netCDF3 64-bit files Yes ​http://nco.sf.net/nco.html#lfs netCDF4/HDF5 available Yes ​http://nco.sf.net/nco.html#nco4 netCDF4/HDF5 enabled Yes ​http://nco.sf.net/nco.html#nco4 OpenMP SMP threading No ​http://nco.sf.net/nco.html#omp Optimization: run-time No Fastest execution possible (slowest compilation) Parallel netCDF3 No ​http://nco.sf.net/nco.html#pnetcdf (pre-alpha) Regular Expressions Yes ​http://nco.sf.net/nco.html#rx Shared libraries built No Small, dynamically linked executables Static libraries built Yes Large executables with private namespaces UDUnits conversions Yes ​http://nco.sf.net/nco.html#udunits UDUnits2 conversions Yes ​http://nco.sf.net/nco.html#udunits

The macports is ncks -r NCO netCDF Operators version "4.4.7" last modified 2014/11/05 built Jan 6 2015 on mac73.psd.esrl.noaa.gov by root ncks version 4.4.7 Linked to netCDF library version 4.3.2, compiled Feb 3 2015 08:18:38 Copyright (C) 1995--2014 Charlie Zender This program is part of NCO, the netCDF Operators. NCO is free software and comes with a BIG FAT KISS and ABOLUTELY NO WARRANTY You can redistribute and/or modify NCO under the terms of the GNU General Public License (GPL) Version 3 with exceptions described in the LICENSE file GPL: ​http://www.gnu.org/copyleft/gpl.html LICENSE: ​http://nco.cvs.sf.net/nco/nco/LICENSE Homepage: ​http://nco.sf.net User Guide: ​http://nco.sf.net/nco.html Configuration Option: Active? Meaning or Reference: Check _FillValue Yes ​http://nco.sf.net/nco.html#mss_val Check missing_value No ​http://nco.sf.net/nco.html#mss_val DAP clients Yes ​http://nco.sf.net/nco.html#dap Debugging: Custom No Pedantic, bounds checking (slowest execution) Debugging: Symbols No Produce symbols for debuggers (e.g., dbx, gdb) GNU Scientific Library Yes ​http://nco.sf.net/nco.html#gsl HDF4 support Unknown ​http://nco.sf.net/nco.html#hdf4 Internationalization No ​http://nco.sf.net/nco.html#i18n (pre-alpha) MPI parallelization No ​http://nco.sf.net/nco.html#mpi (beta) netCDF3 64-bit files Yes ​http://nco.sf.net/nco.html#lfs netCDF4/HDF5 available Yes ​http://nco.sf.net/nco.html#nco4 netCDF4/HDF5 enabled Yes ​http://nco.sf.net/nco.html#nco4 OpenMP SMP threading No ​http://nco.sf.net/nco.html#omp Optimization: run-time No Fastest execution possible (slowest compilation) Parallel netCDF3 No ​http://nco.sf.net/nco.html#pnetcdf (pre-alpha) Regular Expressions Yes ​http://nco.sf.net/nco.html#rx Shared libraries built No Small, dynamically linked executables Static libraries built Yes Large executables with private namespaces UDUnits conversions Yes ​http://nco.sf.net/nco.html#udunits UDUnits2 conversions Yes ​http://nco.sf.net/nco.html#udunits

The command is ncks -D 3 --fl_fmt=netcdf4_classic -L 5 --overwrite huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc /Projects/CMIP5AMonthly/historical/ACCESS1-0r1i1p1huss_Amon_ACCESS1-0_historical_r1i1p1_185001-200512.nc

The file size is 200MB. Maybe an update to netcdf 4.3.3 will speed things up ? Maybe its the chunking size the macports cannot handle ? One thing I note is that the chunck size for the ncks for the huss variable on Mac ends up being: 1872, 145, 192 but using nccopy, it's 1, 145, 192 which is more in line with what I would expect.