New Example CI + Fixes to some broken example!

[okBrian]

Description

Added New Example CI, - Average runtime - 30 minutes

Why MacOS runner?
From my initial testing its much faster than the ubuntu images. Some examples ran over two times faster on the MacOS runner

Why 15625, (125, 125), or (25, 25, 25) for cell boundaries?
Some 3D examples break if m < 25, and this seems like a good spot to put examples so they don't take too long

Fixes #474

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

Scope

• This PR comprises a set of related changes with a common goal

How Has This Been Tested?

• Github Actions Run all example with CI
• Run all examples locally

Test Configuration:
gcc 14, Ubuntu 22,04.4 LTS
Github Action Macos Runner

Checklist

• I have added comments for the new code
• I ran ./mfc.sh format before committing my code
• This PR does not introduce any repeated code (it follows the DRY principle)
• I cannot think of a way to condense this code and reduce any introduced additional line count

[okBrian]

I'm not exactly sure whats wrong with the test suite, but I think if I add the --generate flag they should be okay. Do we want the test suite to even test for examples because that would add another 30 minutes or more to them. Also I need to remove the -r flag from the example suites but it looks like it worked fine.

[sbryngelson]

@okBrian we can add this as a separate GitHub workflow (Examples-test.yml, Example Smoke Test).

[sbryngelson]

the -r --generate things I don't understand at the moment

[sbryngelson]

@okBrian It's important that these cases run and produce output that isn't nonsense. So you can run the "example suite" with ./mfc.sh test —a (+ your flags, I guess?), where -a already exists. ' This runs the files through post_process and then checks for NaNs/Infinity.

If parallel_IO is not true then I'm not sure if -a works or not. You may need to turn it on via your toolchain tricks.

[henryleberre]

I'm not exactly sure whats wrong with the test suite, but I think if I add the --generate flag they should be okay. Do we want the test suite to even test for examples because that would add another 30 minutes or more to them. Also I need to remove the -r flag from the example suites but it looks like it worked fine.

If you specify --generate it will compare the pre-process and simulation results against itself (not with the reference values from this PR). So you shouldn't pass --generate.

Whether we should or should not use the -r (--relentless) flag is debatable but we have currently opted not to use it so we should probably keep it that way. The main advantage is that we don't spend CI resources running more tests if one already failed and the workflow is marked as failed right away when the first test case fails.

[sbryngelson]

might want to put this back?

[okBrian]

It breaks the 1D bubble cases. Not sure why maybe its deleting the D folder I'm creating.

[sbryngelson]

perhaps associated with current bug in CI that if it has to 'retry' a simulation, it will try to create a directory that already exists

[okBrian]

I am expecting the code to not work with --no-mpi so thats another thing to debug... I think everything with mpi should pass hopefully.

[sbryngelson]

this won't work on Phoenix until @henryleberre merges #581 and you pull in the module changes for Phoenix (we moved to RHEL9 nodes).

[okBrian]

Oh my bad I thought they were done based on the most recent commit.

[henryleberre]

Well, they should be working but there's an issue with @sbryngelson's Python installation. I'm pretty sure it's because he deleted bits from his ~/.local folder. The CI is complaining that a "pip" executable cannot be found in ~/.local/bin/pip. @sbryngelson I put my .local in /storage/coda1/p-sbryngelson3/0/shared/local-for-spencer so you can copy what's missing.

[sbryngelson]

Well, they should be working but there's an issue with @sbryngelson's Python installation. I'm pretty sure it's because he deleted bits from his ~/.local folder. The CI is complaining that a "pip" executable cannot be found in ~/.local/bin/pip. @sbryngelson I put my .local in /storage/coda1/p-sbryngelson3/0/shared/local-for-spencer so you can copy what's missing.

I actually never deleted anything in my .local directory, just suggested you try it as a debug measure.

[sbryngelson]

You can see that this works by looking here: https://github.com/MFlowCode/MFC/actions/runs/10498547818/job/29083688602

@henryleberre also opened this empty PR to test things #586

[okBrian]

Hey is GT Phoenix fully migrated or should I keep waiting?

[sbryngelson]

It's not working yet, though the Docker should be?

[henryleberre]

@okBrian It should be working now ™. You should just need to rebase on upstream.

[sbryngelson]

@okBrian you can sync with master now, it should work

[sbryngelson]

Phoenix is down so none of the Phoenix CI will pass for now... will reboot those when they come back online

[sbryngelson]

@okBrian Phoenix CI is back online, but now you will need to rebase as @henryleberre made a change for Frontier CI as well as the packer golden files.

[henryleberre]

Also, @okBrian you need to regenerate the golden files for your new cases. Alternatively, I can run a script for you that strips the <x> <y> <z> entries for you. It's easiest if you regenerate.

[sbryngelson]

@okBrian phoenix runners are back up, though this branch needs to be synced with master