NCO Homepage
Author: Charles S. Zender
Keywords: NCO Homepage, netCDF, netCDF operator, GCM, HDF, scientific data

## Known Problems through 2012 (version 4.2.3)

Older Generic Run-time Problems:
- `MM3` slowdown: A longstanding “feature” of netCDF3 was identified in March, 2012, and is now known by the tag MM3. The MM3 issue can lead to unusually slow performance. The problem is triggered by an aggregate pattern of file access so the workaround must be implemented in the application software (e.g., NCO) rather than in the netCDF library itself. The name MM3 fits because the problem is normally encountered on Multi-record Multi-variable netCDF3 files. And we call our “solution” the MM3-workaround. If you encounter unusually slow NCO performance while using NCO to analyze MM3 files on a large blocksize filesystem, chances are you are encountering an MM3-induced slowdown. NCO release 4.1.0 implements the MM3-workaround for `ncks`. It speeds-up common ncks sub-setting on NCAR's GLADE by 10-50x. MM3-induced slowdowns are present in other NCO operators and we are prioritizing our MM3-patches to those encountered most often. Thanks to Gary Strand for reporting this problem, and to Russ Rew for creating the workaround algorithm, which is also now in `nccopy`.
- `NOFILL` bug: All netCDF versions prior to 4.1.3 may create corrupt netCDF3 files when linked to any version of NCO except 4.0.8. The solution is to install netCDF version 4.1.3 or later. The corruption occurs silently (without warning or error messages). The problem has been seen "in the wild" only on filesystems with large block sizes (e.g., Lustre), although it may be more widespread. It is caused by a netCDF bug that NCO triggers by invoking NOFILL mode for faster writes. Hence it is called the NOFILL bug. The bug is hard to trigger, it depends on a rare interaction of filesystem block-size, hyperslab size, and order-of-variable writing. The bug exists in all versions of netCDF through 4.1.2. If you have a large block filesystem and cannot upgrade your netCDF library, then use NCO version 4.0.8, which disables NOFILL mode (and thus writes files more slowly). NCO 4.0.8 and will workaround the NOFILL bug on all versions of netCDF (i.e., 4.1.2 and earlier). Hence NCO 4.0.8 will always correctly write netCDF3 files. Other temporary workarounds include creating only netCDF4 files (e.g., `ncks -4 ...`) instead of netCDF3 files. The NOFILL patch included in NCO 4.0.8 was subsequently removed in NCO 4.0.9, which assumes that netCDF 4.1.3 or later is installed.
- Degenerate hyperslabbing bug: Versions ???—4.0.6 could return incorrect hyperslabs when user-specified hyperslabs did not include at least one point. In such cases, instead of returning no data, hyperslabs could return all data. To determine whether your NCO is affected by this bug, run these commands: `ncks -O -v lat -d lat,20.,20.001 ~/nco/data/in.nc ~/foo.nc;ncks -H ~/foo.nc` If the returned hyperslab contains any data, then your NCO is buggy (because that hyperslab should be empty). This can lead to incorrect answers for hyperslabs that should be empty. Analogous problems would occur with empty auxiliary coordinate bounding boxes. Although most users do not specify empty hyperslabs, we urge all users to upgrade to NCO 4.0.7+ just to be safe.
- Threading problems with MSA: NCO version 3.9.5 has a nasty bug that causes threaded arithmetic operators, e.g., `nces` to produce incorrect results under some conditions. The problem may occur whenever OpenMP is enabled and the operators run on a multi-core CPU with more than one thread. These incorrect answers, if generated, are relatively easy to notice. The number of threads used to generate a file is, by default, recorded in the global attribute `nco_openmp_thread_number` which may be examined with `ncks -M foo.nc | grep nco_openmp_thread_number`. The only action that will correct a file that you think (or know) contains corrupted data because of this NCO bug is to re-process the file with a non-buggy NCO version. Version 3.9.5 is buggy and should be upgraded ASAP. Be careful with data processed using this NCO version on multi-core CPUs. The (one-line!) patch to fix this bug in 3.9.5 is here.
- Index-based hyperslab problems: NCO versions 2.7.3—2.8.3 have a nasty bug that causes index-based hyperslabs, e.g., `-d lat,1,` to behave like value-based hyperslabs, e.g., `-d lat,1.0` under some conditions. Unfortunately, the incorrect answers generated may be hard to notice! This problem was most often enountered by users trying to assemble monthly averages using the stride feature of `ncrcat`. One common symptom is that the time-offset of the output file is incorrect. Versions 2.7.3—2.8.3 are buggy and should be upgraded ASAP. Re-do any data-processing that used index-based hyperslabbing with these versions of NCO.

Older Operator-specific Run-time Problems:
- `ncks` bug with auxiliary coordinates: Versions 4.2.x–4.3.1 of `ncks` did not correctly support auxiliary coordinates (specified with `-X`). Auxiliary coordinates continued to work with the other hyperslabbing NCO operators. Auxiliary coordinates once again work in all hyperslabbing operators, including on netCDF4 group files in operators that support them. Fixed in version 4.3.2.
- `ncatted` bug on implicit attribute names: Versions 4.2.x–4.3.0 of `ncatted` could segfault when processing attributes specified implicitly (i.e., by leaving the attribute field blank in the `-a` specification. Fixed in version 4.3.1.
- `ncbo` bug handling certain special variables: Version 4.3.0 of `ncbo` inadvertently always turns off certain exceptions to variable list processing. This may cause some grid-related variables (e.g., `ntrm` and `nbdate`) and some non-grid variables (e.g., `ORO` and `gw`) to be arithmetically processed (e.g., subtracted) even when that makes no sense in most climate model datasets. Fixed in version 4.3.1.
- `ncks` bug copying metadata: Version 4.2.6 of `ncks` does not copy variable metadata by default. Thus output files appear stripped of metadata. One can work around this problem in 4.2.6 by specifying the `-m` option. Otherwise an upgrade is recommended. Fixed in version 4.3.0.
- `ncks` bug subsetting variables: Version 4.2.4 of `ncks` sometimes dumps core when subsetting variables with `-v var`. Fixed in version 4.2.5.
- `ncks` bug with altering record dimensions: Version 4.2.4 of `ncks` ignored both the `--mk_rec_dmn` and the `--fix_rec_dmn` switches. It exited successfully without altering the record variable. Fixed in version 4.2.5.
- `nces` bug with non-record files: Versions 4.2.1—4.2.3 of `nces` incorrectly referenced the record variable on files which do not contain it. This caused a segmentation violation and core dump.
- `ncra` bug when last file(s) is/are superfluous: Versions 4.2.1—4.2.3 of `ncra` incorrectly skipped writing the results of the final normalization when trailing files were superfluous (not used). In the most common case, all values are zeros in the output file. Upgrade if you call `ncra` with trailing superfluous files.
- `ncecat` bug when files generated with `-n`: Version 4.2.2 of `ncecat` could incorrectly skip the first input file in the default mode (RECORD_AGGREGATE) when the `-n NINTAP` switch is used to automate filename generation. Upgrade if you use `ncecat -n`.
- `ncra` bug handling CF `coordinates` attributes that contain the name of the record coordinate: Versions 4.0.3—4.0.4 of `ncra` incorrectly treat the record variable (usually `time`) as a fixed variable if it is specified in the `coordinates` attribute of any variable in a file processed with CCM/CCSM/CF metadata conventions. This bug caused core dumps, and even weirder behavior like creating imaginary time slices in the ouput. Upgrade recommended if you work with NCAR CCSM/CESM model output. One workaround that does not require NCO upgrades is to remove the record coordinate name (usually `time`) from the `coordinates` attribute of all variables in CF-compliant files before processing the file with `ncra`.
- `ncra` bug averaging YYYYMMDD-format `date` variables in CCSM/CF-compliant files: Versions ???—4.0.5 of `ncra` contain a bug which produces an incorrect average (usually zero) of the `date` variable which many CCSM/CF-compliant files use to track model dates in the human-readable YYYYMMDD-format. Averaging YYYYMMDD-format integers is intrinsically difficult, since such dates have calendar assumptions built-in. NCO attempts this in CCSM/CF-compliant files by using the `nbdate` (beginning date) and `time` (days since `nbdate`) variables to find the average date, converting that to YYYYMMDD, and writing that as the average value of `date`.
- `ncks` bug hyperslabbinging fixed netCDF4 dimensions: Versions 4.0.3—4.0.4 of `ncks` contain a bug which triggers a core-dump when hyperslabbing (along a non-record dimension) a netCDF4-format input file into a netCDF4-format output file, e.g., `ncks -d 0,1,lat in4.nc out4.nc`. Three workarounds that do not require NCO upgrades (or downgrades) are to explicitly specify chunking with, e.g., `ncks --cnk_plc=all -d 0,1,lat in4.nc out4.nc`, or, to use `nces` instead of `ncks` for hyperslabbing, e.g., `nces -d 0,1,lat in4.nc out4.nc` (`nces` does a no-op when there is only one input file), or to write to a netCDF3 file, `ncks -3 -d 0,1,lat in4.nc out3.nc`.
- Core dump with `ncks`: Printing variables to screen with `ncks` can trigger a segfault in NCO 3.9.9—4.0.3. Users may upgrade, downgrade, or apply this one-line patch to 3.9.9 sources: Remove this line “`*cnk_sz=(size_t)NULL;`” —near line 751 of `nco/src/nco/nco_netcdf.c`— should fix the problem. The problem in later NCO versions is due to a different bug and this patch will not work.
- `ncrename` erroneous error exit: Versions 4.0.1—4.0.3 of `ncrename` contain a bug where commands like `ncrename -a .old_nm,new_nm in.nc out.nc` would, if `old_nm` did not exist, write the correct file and then exit with an error message although no error had occurred. The files written were fine, and the error message can be safely ignored. This was due to not clearing an extraneous return code.
- `ncbo` segmentation fault: `ncbo` versions 4.0.0—4.0.2 incorrectly refreshed internal metadata, leading to segmentation faults and core dumps with some exacting compilers, notably xlC on AIX.
- `ncra` segmentation fault: `ncra` versions 4.0.0—4.0.1 mishandled some CF-compliant dates, leading to segmentation faults and core dumps.
- Arithmetic problems with `ncap` division, modulo, and exponentiation: `ncap` versions < 3.0.1 incorrectly exponentiate variables to variable powers (`V^V`). We recommend that all `ncap` users upgrade.
  `ncap` versions up to 2.9.1 incorrectly handle division, modulo, and exponentiation operations of the form `S/V`, `S%V`, and `S^V` where first operand (`S`) is scalar (i.e., either typed directly in the `ncap` script or converted from an attribute) and the second operand (`V`) is a full variable (i.e., stored in a file or computed by `ncap`). Instead of the requested quantity, `ncap` returned `V/S`, `V%S`, and `V^S`. In other words `ncap` treated some non-commutative operations as commutative. This is now fixed. The `V/V`, `V%V`, `V^V`, `V/S`, `V%S`, `V^S`, `S/S`, `S%S`, and `S^S` operations were never affected. We recommend that all `ncap` users upgrade.
- Incorrect `ncbo` output for packed input: `ncbo` versions ???—3.2.0 incorrectly write differences of packed input. This only affects packed variables.
- Problems with `ncflint` and `missing_value`s: The algorithm `ncflint` used to perform interpolation in versions up to 2.9.4 was not commutative. It returned the weighted valid datum when the other datum was `missing_value`, or it returned `missing_value`, depending on the order the input files were specified. As of version 2.9.5, `ncflint` always returns `missing_value` when either input datum is `missing_value`. Possible future implementations are discussed here.
- Problems with `ncra` and `nces` when `missing_value` = 0.0: The algorithm `ncra` and `nces` used to perform arithmetic in versions up to 2.9.2 breaks if `missing_value` is 0.0. Why, you ask? Running average (or total, etc.) algorithms must initialize the answer to 0.0. This is done since the sum accumulates in place as `ncra` and `nces` proceeds across records and files. (Normalizing this accumulation by the total number of records is the last step). The old algorithm compared both the current running average and the new record to the `missing_value`. If either comparison matched, then nothing accumulated for that record. This zero-initialization led to a state where it was impossible to ever recognize valid data. As a result nothing accumulated and the answer was always zero. The record and ensemble averages would also fail (in a non-obvious) way whenever an intermediate sum equalled `missing_value`. The chances of the latter event ever happening are exceedingly remote. The new algorithm compares only the new record to the `missing_value`. This fixes both problems and is faster, too.
- Packing problems with `ncwa`: NCO versions ???—2.9.0 have a bug that causes `ncwa` to fail (produce garbage answers) when processing packed `NC_FLOAT` data. Version 2.9.1 fixes this problem. This problem may have been noticed most by OPeNDAP users since many netCDF climate datasets served by OPeNDAP are packed `NC_FLOAT`s. Upgrade to 2.9.1 if you use `ncwa` on packed data.
- Packing problems with `ncap`: NCO versions 2.8.4—2.8.6 have a bug that causes the `ncap` intrinsic packing function `pack()` to fail. Version 2.8.7 fixes this problem.

Older Platform-specific Run-time Problems:
- Float-valued intrinsic arithmetic functions in `ncap` on AIX: `ncap` versions through 4.0.4 have a bug that causes all float-valued intrinsic math functions to fail under AIX. Float-valued math functions are the ISO C99 functions, e.g., `cosf()`, `fabsf()`, `logf()`. The user does not invoke these functions directly— the user always specifies the generic function name, e.g., `cos()`, `abs()`, `log()`. NCO automatically calls the native single precision (i.e., float-valued) math functions when the generic function argument is a native float (e.g., naked constants like `1.0f` or variables stored as `NC_FLOAT`). Double precision arguments cause NCO to invoke the standard (double-valued) form of the generic function, e.g., `cos()`, `fabs()`, `log()`.