[tests] Run tests in parallel on non-Windows

[jonpryor]

What do we want? Faster unit test execution!

How do we do that? By running things in parallel!

Some of our existing tests are already run in parallel, such as the
src/Xamarin.Android.Build.Tasks/Tests tests which use NUnit's
[Parallelizable (ParallelScope.Children)], but there is currently no
way to run the Xamarin.Android.Build.Tasks NUnit tests
concurrently with the on-device .apk tests concurrently with the
Java.Interop unit tests concurrently with...

We think there might be a "win" here, as the
Xamarin.Android.Build.Tasks unit tests are heavily I/O bound, while
the .apk BCL tests are -- presumably -- CPU bound, so executing
these at the same time might net some nice time savings.

Spike the idea by updating the RunAllTests target to generate a
shell script which executes msbuild to run the appropriate test
target, in the background, then waiting for all jobs to finish.

The generated bin/Test$(Configuration)/parallel-targets.sh file
resembles:

echo Executing in background: msbuild …/build-tools/scripts/RunTests.targets /v:normal /binaryLogger:"…/msbuild-20181012T083249-Target-RunNUnitTests.binlog" /t:RunNUnitTests
msbuild …/build-tools/scripts/RunTests.targets /v:normal /binaryLogger:"…/msbuild-20181012T083249-Target-RunNUnitTests.binlog" /t:RunNUnitTests &
wait
exit 0

[jonathanpeppers]

This looks good.

The only thing I wonder. What will happen in monodroid? We have some MSBuild tests (InstallTests and InstantRunTests) that deploy to the device. Will those work in parallel with the APK tests?

[jonpryor]

@jonathanpeppers asked:

What will happen in monodroid?

Nothing; monodroid doesn't use make run-all-tests. (It probably should, but it doesn't now.)

[jonpryor]

Alas, the macOS PR Build is unstable, but wow is it not happy:

  Java.Interop/JreRuntime.cs(76,28): error CS0012: The type 'Object' is defined in an assembly that is not referenced. You must add a reference to assembly 'System.Runtime, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. […xamarin-android/external/Java.Interop/src/Java.Runtime.Environment/Java.Runtime.Environment.csproj] […/xamarin-android/build-tools/scripts/RunTests.targets] 
...
  …/xamarin-android/build-tools/scripts/RunTests.targets(65,5): error MSB3073: The command "make -C "/Users/builder/jenkins/workspace/xamarin-android-pr-builder/xa
marin-android/external/Java.Interop" CONFIGURATION=Debug all" exited with code 2.
  …/xamarin-android/build-tools/scripts/RunTests.targets(137,5): error MSB3073: The command "build-tools/scripts/parallel-targets.sh -t RunNUnitTests -t RunJavaInteropTests -t RunApkTests -t RunPerformanceTests /Users/builder/jenkins/workspace/xamarin-android-pr-builder/xamarin-android/build-tools/scripts/../../bin/Debug/bin/xabuild /v:diag /p:Configuration=Debug /p:XAIntegratedTests=False /Users/builder/jenkins/workspace/xamarin-android-pr-builder/xamarin-android/build-tools/scripts/RunTests.targets" exited with code -1

    953 Warning(s)
    315 Error(s)

Eep!

I suspect we have a build order dependency problem.

[jonpryor]

Merged PR #2001 (so that we'd get separate .binlog files for unit tests), then rebased this commit on top of now-current master (so that we'd get separate .binlog files for unit tests), which will hopefully let us know why things became unstable.

[jonpryor]

@jonpryor mentioned:

...wow is it not happy:
...
I suspect we have a build order dependency problem.

The actual problem appears to be because the initial effort was using xabuild to run the test targets, not msbuild (which is what make run-all-tests uses).

The latest effort uses msbuild, so I have high hopes it might actually work this time. (Maybe.)

Review is out of date because of most recent changes.

[jonpryor]

The macOS PR Build timed out:

  /Users/builder/jenkins/workspace/xamarin-android-pr-builder/xamarin-android/build-tools/scripts/../../bin/TestDebug/parallel-targets.sh: line 9: 12452 Terminated: 15          msbuild /Users/builder/jenkins/workspace/xamarin-android-pr-builder/xamarin-android/build-tools/scripts/RunTests.targets /v:normal /binaryLogger:"/Users/builder/jenkins/workspace/xamarin-android-pr-builder/xamarin-android/build-tools/scripts//../../bin/BuildDebug/msbuild-20181012T111859-Target-RunJavaInteropTests.binlog" /t:RunJavaInteropTests

Retrying, in the hope it'll actually work.

[jonpryor]

@jonpryor wrote

Retrying, in the hope it'll actually work.

Behold, it is green.

Now, is it worth it?

The PR build for this PR took 2h2min.

Meanwhile, perusing the green PR builds on xam-mac-mini-7 shows:

• PR Provide warnings around compileSdkVersion(TargetFrameworkVersion), targetSdkVersion, and minSdkVersion #2050 (build 4281): 2h23min
• PR [Xamarin.Android.Build.Tasks] Add CopyResource items to FileWrites. #2288 (build 4272): 2h12min
• PR [Xamarin.Android.Build.Tasks] Add CopyResource items to FileWrites. #2288 (build 4268): 2h11min
• PR [mono-sdks] Try to download Mono runtimes and BCL before building them. #2265 (build 4266): 2h20min
• PR D8 and R8 integration #2019 (build 4263): 4h36min

I'll immediately discount #2019; 4h36min is an outlier.

I've also restarted this PR, so that we have an additional data point, i.e. "how consistent" is 2h2min? https://jenkins.mono-project.com/job/xamarin-android-pr-builder/4285/

Assuming 2h2min is consistent, then we're seeing build time improvements between 7% (vs. 2h11min) and 15% (vs 2h23min). That seems pretty good.

@grendello, @jonathanpeppers: Thoughts? Is this worth merging?

[jonpryor]

I've also restarted this PR, so that we have an additional data point, i.e. "how consistent" is 2h2min? https://jenkins.mono-project.com/job/xamarin-android-pr-builder/4285/

The rebuild is 2h3min, which is 6.1% faster than the 2h11min time.

[jonathanpeppers]

@jonpryor my only concern is if it’s going to make our plots have slower times and make them fluctuate a lot? Could RunPerformanceTests still happen last, but not parallel? I think it only takes 5 min (I think that was my machine), so I would have to look on Jenkins.

We could also try this for a while and see what the plots do.

[jonpryor]

Updated the PR so that the RunPerformanceTests target isn't executed in parallel.

[jonpryor]

By not executing the RunPerformanceTests target in parallel, we get a build time of 1h54min.

Wait, what?

(Consistency, Conschmistency!)

[jonathanpeppers]

I would double check the log that they all ran since it was faster, but it looks right to me.

[grendello]

@jonpryor my only concern is if it’s going to make our plots have slower times and make them fluctuate a lot? Could RunPerformanceTests still happen last, but not parallel? I think it only takes 5 min (I think that was my machine), so I would have to look on Jenkins.

We could also try this for a while and see what the plots do.

That's why we need to apply some statistical treatment to the raw data, so that aberrations like PR 2019 don't blur the image

[grendello]

By not executing the RunPerformanceTests target in parallel, we get a build time of 1h54min.

Wait, what?

(Consistency, Conschmistency!)

Hm, maybe it kinda sorta makes sense? Those are performance tests, which means they are stressing the machine (I'm guessing)? So running them in parallel may slow things down by increasing process queue time.

[grendello]

All in all, I think it's worth saving the 6-7% of time we spend on build. It's not an insignificant amount of time.