Should hpc-stack updates happen during off-hours?

[edwardhartnett]

During the recent hpc-stack-1.2.0 update there were problems caused by updating hpc-stack during work time:

@Hang-Lei-NOAA writes:

I try to update the hpc-stack with v1.2.0 release during work time, which made some users interrupted on dell.
So, I stopped and recovered all from the backup.

@Hang-Lei-NOAA also suggests:

Perhaps, future updates should be made during machine start up before or after maintenance.

This is a good suggestion. Also it would be good if we could find some way to announce to everyone before the update.

Hang also reports:
for hpc-stack, the libs under v1.1.0 and v1.2.0 for the same compiler version are installed in the same place, and shared. Updating some modules may interrupt the other.

[Hang-Lei-NOAA]

Yes, the hpc/1.1.0 are still available after adding v1.2.0. Both 1.1.0 and 1.2.0 code for the same compiler are in the same directory. Therefore, update some libs in v1.2.0 may interrupt the installation in v1.1.0 for a short time. After the installation, under the same compiler, the v1.1.0 and v1.2.0 are equal. All the modules under the same directory are accessible by both hpc/1.1.0 and hpc/1.2.0

…

On Mon, Aug 16, 2021 at 9:07 AM Edward Hartnett ***@***.***> wrote: Assigned #300 <#300> to @Hang-Lei-NOAA <https://github.com/Hang-Lei-NOAA>. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#300 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKWSMFA26VYREZNE4RJOAXTT5EESHANCNFSM5CHWMNNQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[aerorahul]

@Hang-Lei-NOAA
If the installation of v1.2.0 is in the same directory as v1.1.0, why is there an interruption expected?
When installing v1.2.0 of the hpc-stack in the same location as v1.1.0 and for the same compiler+mpi, were packages that did not need an update e.g. netcdf/4.7.4, hdf5/1.10.6, etc get overwritten?

[arunchawla-NOAA]

when we move to v1.2.0 are we updating libraries that are otherwise unchanged ?

[Hang-Lei-NOAA]

@Ruhual yes, most of libs are not necessary to update This time that ncio is needed for the bug. Boi also find issues for grib associated libs, these need to be updated.

…

On Mon, Aug 16, 2021 at 9:21 AM arun chawla ***@***.***> wrote: when we move to v1.2.0 are we updating libraries that are otherwise unchanged ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#300 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKWSMFGT52VO7BVYLYNIIMLT5EGDTANCNFSM5CHWMNNQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[aerorahul]

@arunchawla-NOAA
They should not.
The installation path of the stack is like a growing tree.
Every subsequent installation is an addition to the installation path, and not a replacement (except where a replacement of a package is warranted; e.g. corrupted installation, faulty installation, etc).
Otherwise, this path should keep on growing and the entire limbs should be pruned at the mpi or compiler level when the compiler is labeled as obsolete or unusable.

The versioning of hpc-stack pertains the the software of hpc-stack (modulefiles, build scripts etc), and not that of the software it installs (netcdf, hdf5, nceplibs, etc).

[aerorahul]

@Hang-Lei-NOAA
I understand that most libs are not necessary to be updated.
The question is, is that what was done or was the whole tree replaced?
And if @BoiVuong-NOAA found issues or bugs were found in ncio, then they need to make a release with the bug fix and that bugfix release should be installed before "silently" updating the installation.

[arunchawla-NOAA]

should we just have one stack location ? hpc-stack. The software itself can be versioned and grow as a function of time but the entire stack does not need to be reinstalled whenever the software is updated, only if the underlying library changes ?

[Hang-Lei-NOAA]

Boi reported the issue in grib_util. For the grib series, the problem in grib_util was associated with g2. The g2 was updated several times. This include changing the jasper lib v2.2.5, and adding new functions. Now, the new function was failed. We need to recompile to make it right.

[Hang-Lei-NOAA]

not a whole tree need to be updated

[aerorahul]

@Hang-Lei-NOAA
I apologize I don't follow:
Was the whole tree updated: Yes/No?

[Hang-Lei-NOAA]

No , just part of.

[Hang-Lei-NOAA]

I should not make it silent, and need to make announcement before updating, as Edward suggested

[aerorahul]

Then I am further confused as to how the packages of yafyaml disappeared from the path on wcoss_dell_p3 on Friday.

[Hang-Lei-NOAA]

There is never this package there for intel/18.0.1.163
They just mistakenly load. These two package for GEO updated were not compiled with intel/18.0.1.163.
That is why Arun ask me to test and build it with intel/18.0.5.274.
Kyle further tested and show that "If there's a failure in ufs_common it doesn't bubble up to the machine module, but if you try to load ufs_common directly it will give the error."
These libs are installed with intel/18.0.5.274. They should correct.

[edwardhartnett]

Hang also reports:

UFS users find that they cannot load the yafyaml and mapl with compiler intel/18.0.1.163.
This is the problem: Because the yafyaml and mapl can not compile with compiler intel/18.0.1.163. They were installed with intel/18.0.5.274.
Kyle tested it and showed that previously they used fine just because the error was not shown on the module, but the error is there.

[aerorahul]

Ok. So it is a different compiler version. Got it.

[Hang-Lei-NOAA]

Now, I have copied all from the backup. Before we give a clear map, I will not update the v1.2.0. In the same time, We also need to wait the FMS new release.

[arunchawla-NOAA]

ok, so the problem was that they were trying to load packages that were installed with intel/18.0.5 but were using intel/18.0.1 .. ?

[Hang-Lei-NOAA]

Yes

[BrianCurtis-NOAA]

ok, so the problem was that they were trying to load packages that were installed with intel/18.0.5 but were using intel/18.0.1 .. ?

If this from Friday's discussion. when you use module load on something that contains other module load statments, if one of them fails it does not propagate that message to the user and fails silently. So yes the yafyaml was trying to get loaded, but it kept failing silently so we did not know there was an issue.

Also, if you enable github discussions, this is a great candidate for that.

[climbfuji]

My suggestion is to install new major releases in a separate directory, but to make incremental updates such as a new wgrib2 or ESMF library for the same hpc-stack version in the existing directory (each time, take a backup beforehand). I will do this on Cheyenne at least, and

module use /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/modulefiles/stack
module load hpc/1.1.0

will become

module use /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.2.0/modulefiles/stack
module load hpc/1.2.0

with additional library version updates as needed. If ESMF needs to be updated within v1.2.0, I'll take a backup beforehand and roll out the update in /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.2.0.

This is safe and allows you to delete - without danger - the 1.1.0 installation when we know that nobody is developing on the old version anymore.

[aerorahul]

No. The UFS-weather-model was trying to load a compiler-version that does not support all the packages it needs.

[climbfuji]

If the attempted update to 1.2.0 had gone into a separate directory, it would have helped and we would not have wasted half a day of multiple people's time. Because the commit would not have even seen this update, but an unchanged stack instead.

In any case, it's up to the stack maintainers how to handle this on the platform(s) they are responsible for, and live with the pros and cons of the different suggestions.

[kgerheiser]

There are a couple of meta changes I can think of (wgrib2 build change,try-load(mkl)) that would break existing builds/tests, and installing it in a new location gives a chance to break cleanly, but as @aerorahul pointed out it leads to a lot of duplicate installations.

[aerorahul]

1.2.0 did not update the package versions. It did include a couple of bug fixes to a couple of existing packages, but that did not cause the issue on Friday with the UFSWM.
What @Hang-Lei-NOAA did was update the compiler(18.0.1.163 to 18.05.274), and the former compiler version was incompatible with one of the UFSWM dependencies.
Upon loading the modulefile for the UFSWM, that error was not transparent and the compilation continued until it tried to link.

See this:
#303 (comment)

The UFSWM is trying to load a dependency for a compiler that it does not work with, as far as I can tell from the discussions.
This has gone undetected in the UFSWM and crept up with the stack upgrade.
Perhaps, it should be fixed in the ufs_common where it is trying to blanket load all the modules even if some are not present (due to compiler incompatibilities)

I agree that there was miscommunication and that is perhaps the point of this discussion on how and when to make updates and how to protect existing installations while allowing newer capabilities to be tested alongside stable ones.

Installing various trees will solve one problem but is a nightmare from a system administration perspective.

[edwardhartnett]

Is there a way to detect such issues in the UFSWM? Are there any more of them?