WW3 Developers Meeting: First steps to porting WAVEWATCH III to GPU

[ukmo-ccbunney]

This thread is continue any discussion or Q&A on the following presentation from the recent WW3 Developers Meeting:

First steps to porting WAVEWATCH III to GPU

The original presentation can be downloaded here:
UK_MetOffice_GPU_Port_WW3_Dev_Meeting.pptx

[JessicaMeixner-NOAA]

@ukmo-ccbunney do you think porting to GPU for structured (or SMC) grids would also benefit from domain decomposition over the current card-deck approach?

[ukmo-ccbunney]

I don't think it would necessarily make much difference to the source term acceleration - that would still be working on a similar "block" of the full spectral array.

Domain decomposition might have some impact on the propagation acceleration if it meant each PE would propagate all spectral bins for its domain (potential for lots of large loops), rather than the current data transpose approach where we loop over all sea points for a specific spectral bin (which is not too bad if you number of seapoints is large).

[aliabdolali]

Some questions from the developer call:

Hendrik Tolman: Would be interesting to see if this will work as well with SNL3

Jessica Meixner: Two questions: It seems like CPU would also benefit from this by perhaps allowing compilers to use more for OMP threading or just for general optimization? Is it just conjecture that this will not speed things up for CPU or have you tested that it wouldn't? 2. Would having domain-decomposition instead of the card-deck help for GPU as well?

Chris Bunney: Hendrick - yes. Something I intend to look into. It depends if the loops are independent or can be collapsed in the SNL3 routine.

Chris Bunney: Jessica - yes. I had hoped that adding the seapoint loop in the source term routines might have allowed the compiler to do some clever stuff with optimizations, but so far I've not seen any evidence of that. However, you are right - it should allow for better OMP parallelisation for CPU cases.

Jun Wang: May I ask what is the reason to choose OpenAcc over other GPU programming languages such as CUDA/OpenMP?

Hendrik Tolman: Old approach in WAM on Cyber 205: Inner loop over space! Back to the future!

Chris Bunney: Jun Wang - CUDA is a lot more low level and would likely need a lot more work to implement. OpenACC is a lot cleaner to implement and should be cross vendor compatible (CUDA is only nvidia?)

Mike Bell: Chris: I believe OpenACC is Nvidia specific. We will want to support OpenMP directives as well as OpenACC ones

[ukmo-ccbunney]

Mike Bell: Chris: I believe OpenACC is Nvidia specific. We will want to support OpenMP directives as well as OpenACC ones

Although Nvidia have invested heavily in OpenACC, I believe the intention is for it to be a standard that will work across many vendors. As far as I am aware, it is supported by AMD GPUs too...

[breogangomez]

Hi all,

We would like to community to discuss a couple of aspects from this work :

• As it was introduced in the presentation, a refactoring of the source term will imply a significant restructuring of the code. We would like to know how the developer’s community and code owners feel about this. We wouldn’t want to start putting some significant effort into developing a final solution before there is some discussion of the changes, and some remote expectation that they will be accepted. We believe this has a significant potential both for GPU and OpenMP optimisation, so performance may be improved even for users not using GPUs. Is the community favourable on this change? In terms of CPU performance, would you like to see any evidence of performance improvement on OpenMP? Or a neutral change would suffice?

• A follow-up issue is how to implement the OpenACC pragmas in the code. This could be done manually (tacking each section one by one) or using code generation tools. We are exploring the usage of PSyclone (early stages) for this, as it is being used for other models at the Met Office. How open the community would be to introducing this into WAVEWATCH III?

Any comments welcome!

[JessicaMeixner-NOAA]

@breogangomez I personally think neutral CPU performance (with or without OpenMP) is sufficient as there's the potential benefit for everyone in the future if GPU's become more commonly used, etc. I'm curious to see if there is an improvement with OpenMP as I suspect there might be. Since the WW3 regression tests are more/less small tests, we might need to set up some larger cases that test the various types of grids to ensure that performance is not effected for SMC, structured, curvilinear, multi-grid, unstructured grids, etc for larger or higher resolution applications.

It looks like some groups at NOAA are using OpenACC (see here: https://ufscommunity.org/ufs-webinar-series/ufs-webinar-archive/#UFSgpu2 for example).

[breogangomez]

@JessicaMeixner-NOAA Thanks for this. I agree it makes sense to do some extensive testing and I understand results should be bit-comparable for non-GPU application (@ukmo-ccbunney please let me know if I am wrong!)

It is nice to see that other teams are exploring GPUs, this is also gaining significant momentum at the Met Office. The ocean area is also exploring GPU usage for NEMO and NEMO-VAR, and atmospheric developers are also considering it for LFric.

[ukmo-ccbunney]

@JessicaMeixner-NOAA Yes - I agree that we might need to set up some larger regtest to properly test GPU performance. Although, you could argue that we are testing conformance in the regression tests, not performance. However, one thing that we would want to be careful of with any GPU acceleration is that future code changes do not accidentally degrade the GPU performance (even if they did not change the results). Impact on runtimes it not something that we really check in the regtest at the moment, but for GPU code this would be quite an important check. A change might negatively impact a GPU kernel due to changes to looping structures or memory transfer requirements in a way that would not be detrimental when run on a CPU...

[aliabdolali]

Here are a couple of comments from my side:

• I wonder if the comparison you provided in the presenation is not large enough (if it is taken from WW3 regtests)? Would it be possbile to do the same analysis and scaling test on a bigger domain? on larger number of mpi threads?

• How much time is needed for tiling compared to the computations? Is it comparable, negiligible, more?

• I know you guys did it for card deck parallelization. How do you envision extending the work for the geographyical domain decomposotion which is implemented for unstrucured meshes in WW3?

[ukmo-ccbunney]

Ali - I can address points 2 and 3 from you query:

• The time spent on the tiling aught to be negligible, but I do see a small increase in timing between the original approach (looping through individual spectra in w3wave) and the refactored code with a tile size of 1. At the moment, I don't know if this is due to the tiling loop itself, or a negative affect of having the SEAPOINT loop in the actual source term routines themselves (unlikely, but I guess there could be cache misses?) I need to investigate more here...

• Regarding domain decomposition, see my answer to Jessica's question above.

[UKMO-lsampson]

Hi Ali, I will add a bit of a answer to your first comment;

The current "full configuration" port has been run on the Persian Gulf domain (shown in the presentation) and the European domain. The mini-apps or test cases we started with were based of the regression tests but we are now running with a realistic sized domain. Our main limiter in terms of MPI threads is that we are running all of this work on Isambard, which is running an interactive job on a single CPU and has a limit of 40 MPI threads (the same applies for OMP threads).

We are looking to extend this in a few ways, by moving to Microsoft Azure (currently being Sandboxed) and creating a hybrid MPI-OpenACC parallelism, we should be able to begin to explore multi-CPU and multi-GPU performance. I'm happy to keep you up to date on this as we make developments, but there is a few moving parts that are also changing so I'm not sure when this will be.