Add orthogonalization options to GMRES

[nbeams]

Add orthogonalization choices for GMRES (CGS and CGS2 in addition to the MGS available before).

This adds a new unified kernel for performing the multidot operation for CGS/CGS2. It also changes the data layout of the Hessenberg matrix so that all the data that needs to be communicated after the multidot is stored contiguously, and can be handled with one communication.

TODO:

• Benchmark check to make sure MGS performance is not hurt by the changes to the Hessenberg data layout.

[nbeams]

Some initial benchmarks comparing old/new Hessenberg use in MGS for the affected kernels (hessenberg_qr and solve_krylov). These were run on Guyot, so A100 GPUs and 2 AMD EPYC 7742 CPUs. The size parameter is the size given to generate the stencil matrix, using the 7 pt stencil, with weak scaling for increased MPI ranks. The points with the same size have various MPI ranks (1, 2, 4, or 8) and nrhs (1, 10, or 100).

CUDA - Hessenberg QR
CUDA - Solve Krylov
OpenMP - Hessenberg QR
OpenMP - Solve Krylov

To me, this seems to indicate that any difference between the old and new versions is basically just noise (especially as the absolute times for solve_krylov are so tiny). But I can run more tests if this isn't considered conclusive enough or looks odd to anybody.

[MarcelKoch]

LGTM. I've left mostly minor comments, except for the shape of hessenberg_iter.

[pratikvn]

LGTM! Some minor nits. Thanks for adding the performance comparisons.