GitLab CI: Direct build directory to $HOME

This patch directs the compilation of GitLab regression test models to
the $HOME directory.

This is motivated by incredibly slow file access times on GFDL's GPFS
file system (F5), with the Nvidia builds taking a prohibitively long
time (around 4 hours).  Compiling in $HOME reduces Nvidia times to 15
minutes, with similar reduction times in the other compilation.

This is achieved with only minimal changes to the build system.  Instead
of creating `build/`, we create a dedicated directory in
`$HOME/ci/$CI_PROJECT_ID/build` and a symbolic link to this directory in
the GPFS directory.  To support these builds, additional symlinks to
`src/` and `ocean_only/` (for static builds) are also created, from
$HOME and back into GPFS.

Once the job is completed, the temporary space in $HOME is deleted.

This patch should not be considered a permanent solution, it is only
meant to get us through a difficult period of testing.  Hopefully we can
revert this change in the near future.

.gitlab-ci.yml

@@ -32,6 +32,8 @@ p:clone:
   tags:
     - ncrc5
   script:
+    # NOTE: We could sweep any builds older than 3 days here if needed
+    #- find $HOME/ci/[0-9]* -mtime +3 -delete 2> /dev/null || true
     - .gitlab/pipeline-ci-tool.sh create-job-dir
 #.gitlab/pipeline-ci-tool.sh clean-job-dir
 
@@ -353,4 +355,5 @@ cleanup:
   before_script:
     - echo Skipping usual preamble
   script:
+    - rm -rf $HOME/ci/$CI_PIPELINE_ID
     - rm -rf $JOB_DIR

.gitlab/pipeline-ci-tool.sh

@@ -94,6 +94,13 @@ create-job-dir () {
     make -f tools/MRS/Makefile.clone clone_gfdl -j # Extras and link to datasets
     bash tools/MRS/generate_manifest.sh . tools/MRS/excluded-expts.txt > manifest.mk
     mkdir -p results
+    # Temporarily move build directory to $HOME to circumvent poor F5 performance
+    mkdir -p $HOME/ci/$CI_PIPELINE_ID/build
+    ln -s $HOME/ci/$CI_PIPELINE_ID/build build
+    # Builds need non-mangled access to src/.
+    ln -s "$(pwd)"/src $HOME/ci/$CI_PIPELINE_ID/src
+    # Static builds need access to ocean_only/
+    ln -s "$(pwd)"/ocean_only $HOME/ci/$CI_PIPELINE_ID/ocean_only
   fi
   section-end create-job-dir
 }