CI: Fix conditions for running mono, and coreclr jobs

[radical]

• this will fix the case where src/mono/ changes didn't trigger some
  mono jobs that depend on corresponding coreclr jobs.

1. Fix conditions for runtime tests

For all the non-wasm mono runtime tests:
- run if runtime tests themselves have changes
- or run if there are any mono changes

2. run coreclr and library build jobs when mono_except_wasm, coreclr, or installer have changes.
3. Disable AOT offsets, and crossaot jobs for wasm as they are not currently used.
4. runtime-dev-innerloop: condition jobs for coreclr, or mono accordingly
5. runtime-linker-tests: adjust conditions for wasm, and non wasm jobs
6. Run the installer jobs in runtime.yml only if the dependencies succeed

... otherwise they will fail when trying to get the results of
those dependencies. For example, `Installer Build and Test coreclr
OSX_arm64 Release` fails with:

[error]Artifact CoreCLRProduct___OSX_arm64_release not found for build 17835. Please ensure you have published artifacts in any previous phases of the current build.

.. when `CoreCLR Product Build OSX arm64 release` timed out.

[radical]

cc @fanyang-mono

[trylek]

LGTM, thanks for fixing this!

[radical]

@trylek One of the issues that I run into is that if you have a JobA depending on a JobB, and JobB is skipped for some reason then even JobA will get skipped, because the condition on the jobs has and(succeeded(), ... I'm wondering how we can catch such cases. I know we can get the result for a dependency and check it for =="Skipped", which maybe we can use in a "first step" to fail if any of the dependencies got skipped? Or is there a better way to do this?

[trylek]

Hmm, what is the situation where depending on a skipped job should result in a failure? Normally I would expect that if jobA gets skipped and jobB depends on it it's natural that jobB gets skipped too as it sits on a pruned branch of the build tree. I agree that if jobB requires some jobA to succeed and jobA gets skipped whereas jobB does not, that indeed seems conflicting and should trigger a failure. Can you please share a more detailed example where you're hitting this situation in the yml files?

[radical]

Hmm, what is the situation where depending on a skipped job should result in a failure? Normally I would expect that if jobA gets skipped and jobB depends on it it's natural that jobB gets skipped too as it sits on a pruned branch of the build tree. I agree that if jobB requires some jobA to succeed and jobA gets skipped whereas jobB does not, that indeed seems conflicting and should trigger a failure.

I'm thinking that the "leaf" jobs that will run, should have their full dependency also run. At least that makes sense to me in the cases where JobA wants to use results of JobB .

Can you please share a more detailed example where you're hitting this situation in the yml files?

Looking at https://dev.azure.com/dnceng-public/public/_build/results?buildId=7053&view=logs&j=8687d66b-8ae3-5bcb-b2ee-49e1823a16f0 , the job shows:

Expanded: and(False, or(eq(dependencies['evaluate_paths']['outputs']['SetPathVars_mono_excluding_wasm.containsChange'], True), eq(dependencies['evaluate_paths']['outputs']['SetPathVars_runtimetests.containsChange'], True), eq(variables['isR...

From the expanded yaml:

  - job: run_test_p0_monollvmfullaot__Linux_x64_release
    condition: >-
      and(succeeded(), or(
        eq(dependencies.evaluate_paths.outputs['SetPathVars_mono_excluding_wasm.containsChange'], true),
        eq(dependencies.evaluate_paths.outputs['SetPathVars_runtimetests.containsChange'], true),
        eq(variables['isRollingBuild'], true)))
    ...
    dependsOn:
    - evaluate_paths
    - mono_common_test_build_p0_AnyOS_AnyCPU_release
    - coreclr__product_build_Linux_x64_release
    - mono_llvmaot_product_build_Linux_x64_release
    - libraries_build_Linux_x64_Debug
    displayName: mono llvmfullaot Pri0 Runtime Tests Run  Linux x64 release
..

• IIUC, the job is getting skipped because succeeded() is False because coreclr__product_build_Linux_x64_release was skipped
• Because of this if we don't have the correct conditions on coreclr__product_build_Linux_x64_release, and it is skipped, then even though we have the correct conditions on the leaf job run_test_p0_monollvmfullaot__Linux_x64_release, it will also be skipped.
• And this will get noticed only if you are explicitly looking for it

And this became an issue because I have been changing the paths that trigger various builds. And I'm not sure how to catch cases where a build is unexpectedly getting skipped, because I messed up the condition on a dependency.

[radical]

In this PR for instances, I broadened the condition so all the mono, and coreclr jobs will get triggered on mono changes. But not all of the coreclr jobs are needed for mono runs - only the desktop ones.
And trying to map from the expanded yaml back to our various templates seems to be non-trivial.

[radical]

Updated to have better conditions. Still waiting for the evaluate jobs to run on all the pipelines, to test this.

[fanyang-mono]

The previous skipped runtime CI lanes are running now. Thanks for the prompt fix!

[fanyang-mono]

The failures on mono llvmaot Pri0 Runtime Tests Run Linux x64 release has been captured by #75606. Feel free to merge if the change is complete.