cnmatrix solver crashes depending on the finidat file, as well as in an IHist simulation

[slevis-lmwg]

Summary:

• cnmatrix may crash when given a finidat file from a different simulation, even when using init_interp.
• cnmatrix may also crash in transient simulations.

Details:
I have encountered three examples this far (see #640):

1. I2000 test using the original ctsm5.2 f19 finidat file:

• I generated this finidat from the 5.1 file by running ctsm5.2 with init_interp for one step.
• I found that the patch with the error had a crop type in the ctsm5.2 fsurdat file that was not present in the same grid cell in the ctsm5.1 fsurdat file. This indicates that init_interp did not obtain satisfactory (or any?) data for that patch.
• I ran a full new spin-up (MatrixCN in ctsm5.2.001 (ctsm52001_[f09,f19,ne30]_GSWP3_bgccrop_[1850AD, 1850SASU, 1850pSASU, hist, 2000pSASU]) NCAR/LMWG_dev#58) and the I2000 test passed, so I have replaced the default finidat for cases such as this test.

2. NEON-NIWO single-point test with (ironically) the new ctsm5.2 f19 finidat file described in (1):

• I reran the test with the orig. ctsm5.2 f19 finidat file and it passed.
• The problem may be similar to (1)
• In this case we decided to replace the failing test with a NEON-HARV test, which is a better site to support than NIWO anyway according to @wwieder. This one works with the new ctsm5.2 f19 finidat file described in (1).

3. To spin up the model (see 1), I tried running all the phases (except AD) with cnmatrix; however, cnmatrix crashed in the historical simulation early in year 1901. The problem may be similar to (1).

Recommendations:

• It seems safest to perform a unique spin-up for each single-point site and for each model resolution.
• Also cnmatrix may be most useful in SASU mode because it results in much faster model spin-up.
• Transient simulations with cnmatrix may not work.

General bug information

CTSM version you are using:
I'm using the cnmatrix branch (#640).
The PR's notes show when the above issues emerged.

Configurations affected:
cnmatrix

[slevis-lmwg]

If these crashes occur consistently due to newly emerging pfts:

1. They may indicate a problem that has remained hidden until now.
2. We may wish to introduce a more graceful error message suggesting a cold start.