Tests.proj doesn't filter to best TargetFramework when running Test target

[ericstj]

This results in the build only building for the best TargetFramework, but the Test Target running for all TargetFrameworks, which fails because none of the tests are built for those frameworks. For example:

Exec
    Assembly = Microsoft.Build.Tasks.Core, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
    CommandLineArguments = "D:\git\runtime_reflection2\artifacts\bin\Microsoft.Extensions.Logging.Console.Tests\Debug\net462\RunTests.cmd"
    Parameters
        Command = "D:\git\runtime_reflection2\artifacts\bin\Microsoft.Extensions.Logging.Console.Tests\Debug\net462\RunTests.cmd"
        IgnoreExitCode = True
        IgnoreStandardErrorWarningFormat = True
    ----- start Wed 03/02/2022 14:59:39.18 ===============  To repro directly: ===================================================== 
    pushd D:\git\runtime_reflection2\artifacts\bin\Microsoft.Extensions.Logging.Console.Tests\Debug\net462\
    xunit.console.exe Microsoft.Extensions.Logging.Console.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing 
    popd
    ===========================================================================================================
    'xunit.console.exe' is not recognized as an internal or external command,
    operable program or batch file.
    ----- end Wed 03/02/2022 14:59:39.22 ----- exit code 9009 ----------------------------------------------------------
    The command ""D:\git\runtime_reflection2\artifacts\bin\Microsoft.Extensions.Logging.Console.Tests\Debug\net462\RunTests.cmd"" exited with code 9009.
    OutputProperties
        TestRunExitCode = 9009

D:\git\runtime_reflection2\eng\testing\tests.targets(142,5): One or more tests failed while running tests from 'Microsoft.Extensions.Logging.Console.Tests'. [D:\git\runtime_reflection2\src\libraries\Microsoft.Extensions.Logging.Console\tests\Microsoft.Extensions.Logging.Console.Tests\Microsoft.Extensions.Logging.Console.Tests.csproj]

This regressed with 9ebe0ef#diff-5ef2fd16f58de91bff2f05ad601ab6ff000c0febabb2312d232f20cfa132a71b in #64000

This change made tests.proj switch from using the buildMultiTargeting to build targets in https://github.com/dotnet/arcade/tree/main/src/Microsoft.DotNet.Build.Tasks.TargetFramework/src

The buildMultiTargeting targets useAdditionalProperties to apply the best target framework, whereas the build targets use SetTargetFramework.

The MSBuild Traversal SDK's Build Target honors SetTargetFramework while the Test target does not. This is why the test target ends up running for all target frameworks.

As a workaround for folks that hit this try adding the following to tests.proj.

  <Target Name="WorkaroundIssue66122" BeforeTargets="Test">
    <ItemGroup>
      <ProjectReference Condition="'%(ProjectReference.SetTargetFramework)' != ''" 
                        AdditionalProperties="%(ProjectReference.AdditionalProperties);TargetFramework=%(ProjectReference.SetTargetFramework)" />
    </ItemGroup>
  </Target>

[ghost]

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This results in the build only building for the best TargetFramework, but the Test Target running for all TargetFrameworks, which fails because none of the tests are built for those frameworks. For example:

Exec
    Assembly = Microsoft.Build.Tasks.Core, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
    CommandLineArguments = "D:\git\runtime_reflection2\artifacts\bin\Microsoft.Extensions.Logging.Console.Tests\Debug\net462\RunTests.cmd"
    Parameters
        Command = "D:\git\runtime_reflection2\artifacts\bin\Microsoft.Extensions.Logging.Console.Tests\Debug\net462\RunTests.cmd"
        IgnoreExitCode = True
        IgnoreStandardErrorWarningFormat = True
    ----- start Wed 03/02/2022 14:59:39.18 ===============  To repro directly: ===================================================== 
    pushd D:\git\runtime_reflection2\artifacts\bin\Microsoft.Extensions.Logging.Console.Tests\Debug\net462\
    xunit.console.exe Microsoft.Extensions.Logging.Console.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing 
    popd
    ===========================================================================================================
    'xunit.console.exe' is not recognized as an internal or external command,
    operable program or batch file.
    ----- end Wed 03/02/2022 14:59:39.22 ----- exit code 9009 ----------------------------------------------------------
    The command ""D:\git\runtime_reflection2\artifacts\bin\Microsoft.Extensions.Logging.Console.Tests\Debug\net462\RunTests.cmd"" exited with code 9009.
    OutputProperties
        TestRunExitCode = 9009

D:\git\runtime_reflection2\eng\testing\tests.targets(142,5): One or more tests failed while running tests from 'Microsoft.Extensions.Logging.Console.Tests'. [D:\git\runtime_reflection2\src\libraries\Microsoft.Extensions.Logging.Console\tests\Microsoft.Extensions.Logging.Console.Tests\Microsoft.Extensions.Logging.Console.Tests.csproj]

This regressed with 9ebe0ef#diff-5ef2fd16f58de91bff2f05ad601ab6ff000c0febabb2312d232f20cfa132a71b in #64000

This change made tests.proj switch from using the buildMultiTargeting to build targets in https://github.com/dotnet/arcade/tree/main/src/Microsoft.DotNet.Build.Tasks.TargetFramework/src

The buildMultiTargeting targets useAdditionalProperties to apply the best target framework, whereas the build targets use SetTargetFramework.

The MSBuild Traversal SDK's Build Target honors SetTargetFramework while the Test target does not. This is why the test target ends up running for all target frameworks.

As a workaround for folks that hit this try adding the following to tests.proj.

  <Target Name="WorkaroundIssue" BeforeTargets="Test">
    <ItemGroup>
      <ProjectReference Condition="'%(ProjectReference.SetTargetFramework)' != ''" 
                        AdditionalProperties="%(ProjectReference.AdditionalProperties);TargetFramework=%(ProjectReference.SetTargetFramework)" />
    </ItemGroup>
  </Target>

Author:	ericstj
Assignees:	-
Labels:	area-Infrastructure-libraries, untriaged
Milestone:	-

[ericstj]

@ViktorHofer not asking you to fix this, but what do you think the right fix should be for this? Should we make a change to the traversal SDK, add a target to tests.proj, or something else?

[ViktorHofer]

Thanks for reporting. I didn't observe such behavior locally as I didn't run all tests via the traversal project. Unfortunately there is no CI protection for this path. Anyway, microsoft/MSBuildSdks#342 should fix this.

[steveharter]

UPDATE: I see Viktor just put a PR up, so you can ignore below:

@ericstj the workaround appears to have some of the tests attempt to execute the assembly directly:

  ----- start Thu 03/03/2022  9:22:43.78 ===============  To repro directly: =====================================================
  pushd C:\git\runtime\artifacts\bin\System.DirectoryServices.AccountManagement.Tests\Debug\
  System.DirectoryServices.AccountManagement.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing
  popd  ----- end Thu 03/03/2022  9:22:43.78 ----- exit code 9009 ----------------------------------------------------------

[ericstj]

I'll give things a try locally and see if I can provide a better workaround. Alternatively you could also try to apply @ViktorHofer's change manually to C:\Users\<userName>\.nuget\packages\microsoft.build.traversal\3.1.3\Sdk\Sdk.targets would have the added benefit of testing it.

[steveharter]

The linked PR fixes the issue (applied the changes locally to my Sdk.targets).