Consume Microsoft.DotNet.CoreFxTesting package (dotnet/corefx#33423)

* Consume Microsoft.DotNet.CoreFxTesting package

* Remove code coverage dedicated runtime dir config files and switch to Coverlet

* Add global tool support in corefx

* Update code coverage docs

Commit migrated from dotnet/corefx@ae38b7c

docs/libraries/building/code-coverage.md

@@ -1,7 +1,7 @@
 Code Coverage
 =============
 
-"Code coverage" is a measure that indicates how much of our library code is exercised by our test suites.  We measure code coverage using the [OpenCover](https://github.com/opencover/opencover), and a report of our latest code coverage results can be seen by clicking the coverage badge on the [CoreFX home page](https://github.com/dotnet/corefx), linking to the latest [Coverage Report](https://ci.dot.net/job/dotnet_corefx/job/master/job/code_coverage_windows/Code_Coverage_Report/).
+"Code coverage" is a measure that indicates how much of our library code is exercised by our test suites.  We measure code coverage using the [Coverlet](https://github.com/tonerdo/coverlet), and a report of our latest code coverage results can be seen by clicking the coverage badge on the [CoreFX home page](https://github.com/dotnet/corefx), linking to the latest [Coverage Report](https://ci.dot.net/job/dotnet_corefx/job/master/job/code_coverage_windows/Code_Coverage_Report/).
 
 This report shows each library currently being tested with code coverage and provides statistics around the quality of the code coverage for the library.  It also provides a line-by-line breakdown of what lines are being covered and what lines are not.
 
@@ -22,11 +22,11 @@ Our default, somewhat-arbitrary initial goal for a library is 90% code coverage.
 
 Issues are opened for a library when a cursory examination of its code coverage reveal that there are likely still some meaningful gaps that need to be addressed.  We welcome contributions to our test suites to help address these gaps and close these issues.  Many of these issues are marked as [up-for-grabs](https://github.com/dotnet/corefx/labels/up-for-grabs).
 
-An issue need not be addressed in its entirety.  We happily accept contributions that improve our tests and work towards improving code coverage numbers even if they only incrementally improve the situation.
+An issue need not be addressed in its entirety. We happily accept contributions that improve our tests and work towards improving code coverage numbers even if they only incrementally improve the situation.
 
 ## Automated Code Coverage Runs
 
-Code coverage runs are performed by Jenkins approximately twice a day.  The results of these runs are all available from the site linked to by the code coverage badge on the home page.
+Code coverage runs are performed approximately twice a day. The results of these runs are all available from the site linked to by the code coverage badge on the home page.
 
 ## PR Code Coverage Runs
 
@@ -40,23 +40,15 @@ You can navigate to this from your PR by clicking the "Details" link to the righ
 
 ## Local Code Coverage Runs
 
-You can perform code coverage runs locally on your own machine.  Normally to build your entire CoreFX repo, from the root of your repo you'd run:
-
-    build -includetests
-
-To include code coverage in this run, augment the `build -includetests` call with the `coverage` argument:
+You can perform code coverage runs for the entire repository locally by adding the `coverage` argument to the `build -includetests` command.
 
     build -includetests -coverage
 
-This will do the build and testing as with the normal ```build```, but it will run the tests using the OpenCover tool.  A resulting index.htm file providing the results of the run will be available at:
-
-    bin\tests\coverage\index.htm
-
-You can also build and test with code coverage for a particular test project rather than for the whole repo.  Normally to build and test a particular test suite, from the same directory as that test suite's .csproj, you'd run:
+This builds and tests the test assemblies and generates the full code coverage report. The resulting index.htm file providing the results of the run should be available at:
 
-    dotnet msbuild /t:BuildAndTest
+    artifacts\coverage\index.htm
 
-To do so with code coverage, append the ```/p:Coverage=true``` argument:
+You can also build and test with code coverage for a particular test project rather than for the whole repo with the ```/p:Coverage=true``` argument:
 
     dotnet msbuild /t:BuildAndTest /p:Coverage=true
 
@@ -67,44 +59,14 @@ The results for this one library will then show up in the aforementioned index.h
 
 And then once the run completes:
 
-    ..\..\..\bin\tests\coverage\index.htm
+    $(TestPath)\report\index.htm
 
 ## Code coverage with System.Private.CoreLib code
 
 Some of the libraries for which contracts and tests live in the corefx repo are actually fully or partially implemented in the core runtime library in another repo, e.g. the implementation that backs the System.Runtime contract is in System.Private.CoreLib.dll in either the coreclr or corert repo. Test projects for code that lives, fully or partially, in System.Private.CoreLib, should have the property `TestRuntime` set to `true` in order to obtain proper code coverage reports.
 
-If the test project does not set the property `TestRuntime` to `true` and you want to collect code coverage that includes types in System.Private.CoreLib.dll add `/p:CodeCoverageAssemblies="System.Private.CoreLib"` to the coverage build command listed above.
+If the test project does not set the property `TestRuntime` to `true` and you want to collect code coverage that includes types in System.Private.CoreLib.dll add `/p:TestRuntime=true` to the coverage build command listed above.
 
 If you want to get coverage report against a private build of System.Private.CoreLib follow the steps outlined at [Testing with Private CoreClr Bits](https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/developer-guide.md#testing-with-private-coreclr-bits).
 
-The build and test projects take care of copying assemblies and PDBs as needed for coverage runs. The resulting code coverage report should now also include details for System.Private.CoreLib.
-
-Note: as of 10/2017 OpenCover, the default coverage tool, requires PDBs to be Windows PDBs - the needed conversions are automatically performed by coverage runs. You can determine if it is a Windows PDB by doing 'more System.Private.CoreLib.pdb',  if it begins with 'Microsoft C/C++ MSF 7.00' it is a Windows PDB.  If you need a Windows PDB the Pdb2Pdb tool will convert (or you can do a dotnet msbuild /t:rebuild /p:DebugType=full).
-
-## Cross-platform Coverage
-As of 07/2018 CoreFx is only able to get coverage information on Windows. To correct this we are experimenting with [coverlet](https://github.com/tonerdo/coverlet).
-
-### Know Issues ###
-
-1. Instrumenting "System.Private.CoreLib" is causing test to crash (both on Windows and Unix).
-
-### Windows Instructions ###
-On Windows by default coverage runs will use OpenCover instead of coverlet, use the build property `UseCoverlet` to change this default. Currently the use of `dotnet msbuild` is required to avoid a problem with one of the coverlet dependencies. Here is the command:
-
-```
-dotnet msbuild /t:RebuildAndTest /p:Coverage=True /p:UseCoverlet=True
-```
-
-### Unix Instructions ###
-On Unix just specifying `/p:Coverage=True` triggers the usage of coverlet. However, in order to generate the html report a few setup steps are needed.
-
-1. Install the [dotnet/cli global tool reportgenerator](https://www.nuget.org/packages/dotnet-reportgenerator-globaltool), add it to the PATH if not automatically added by dotnet/cli (it can be only for the current session)
-2. On Linux install the Arial font required for report generation: `sudo apt-get install ttf-mscorefonts-installer`
-
-After that you request a coverage run from a test folder, e.g.:
-
-```
-dotnet msbuild /t:RebuildAndTest /p:Coverage=True
-```
-
-Open index.htm located at bin/tests/coverage to see the results of the coverage run.
+The build and test projects take care of copying assemblies and PDBs as needed for coverage runs. The resulting code coverage report should now also include details for System.Private.CoreLib.

docs/libraries/project-docs/developer-guide.md

@@ -425,11 +425,14 @@ Code coverage is built into the corefx build system.  It utilizes OpenCover for
 :: Run full coverage
 build -test -Coverage
 
+If coverage succeeds, the full report can be found at `artifacts\coverage\index.htm`.
+
 :: To run a single project with code coverage enabled pass the /p:Coverage=true property
 cd src\System.Collections.Immutable\tests
 dotnet msbuild /t:BuildAndTest /p:Coverage=true
 ```
-If coverage succeeds, the code coverage report will be generated automatically and placed in the bin\tests\coverage directory.  You can view the full report by opening index.htm
+
+If coverage succeeds, the individual report can be found at `$(TestPath)\report\index.htm`.
 
 Code coverage reports from the continuous integration system are available from the links on the front page of the corefx repo.
 
@@ -479,6 +482,4 @@ If you prefer, you can use a Debug build of System.Private.CoreLib, but if you d
 
 If the test project does not set the property `TestRuntime` to `true` and you want to collect code coverage that includes types in System.Private.CoreLib.dll, you'll need to follow the above steps, then
 
-`dotnet msbuild /t:rebuildandtest /p:Coverage=true /p:CodeCoverageAssemblies="System.Private.CoreLib"`
-
-In order to facilitate coverage tools that perform IL rewrite a dedicated shared framework directory is created by default for coverage runs. This shared runtime is copied from the default shared test runtime (the one with version 9.9.9, e.g.: `\corefx\bin\testhost\netcoreapp-Windows_NT-Debug-x64\shared\Microsoft.NETCore.App\9.9.9`). This behavior can be overridden by adding `/p:UseCoverageDedicatedRuntime=false` to the build command used to capture code coverage, which will cause the coverage run to use the same shared runtime as a test run using the native image of System.Private.CoreLib.dll, causing loss of source coverage information for it.
+`dotnet msbuild /t:rebuildandtest /p:Coverage=true /p:TestRuntime=true`

docs/libraries/project-docs/performance-tests.md

@@ -12,8 +12,8 @@ Performance test files (if present) are stored within a library's ```tests/Perfo
 **Step # 2:** Change directory to the performance tests directory: ```cd path/to/library/tests/Performance```
 
 **Step # 3:** Build and run the tests:
- - Windows ```dotnet msbuild /t:BuildAndTest /p:Performance=true /p:ConfigurationGroup=Release```
- - Linux: ```dotnet msbuild /t:BuildAndTest /p:Performance=true /p:ConfigurationGroup=Release```
+ - Windows ```dotnet msbuild /t:BuildAndTest /p:ConfigurationGroup=Release```
+ - Linux: ```dotnet msbuild /t:BuildAndTest /p:ConfigurationGroup=Release```
 
 **Note: Because test build runs tests concurrently, do not use it for executing performance tests. If you still want to run them concurrently you need to pass the flag `/p:Performance=true` to it: `build -test -release /p:Performance=true`.**
 
@@ -32,20 +32,12 @@ Performance tests should reside within their own "Performance" folder within the
 It's easiest to copy and modify an existing example like the one above. Notice that you'll need these lines in the tests csproj:
 ```
   <ItemGroup>
-    <!-- Performance Tests -->
-    <Compile Include="Performance\Perf.Dictionary.cs" />
-    <Compile Include="Performance\Perf.List.cs" />
-    <Compile Include="$(CommonTestPath)\System\PerfUtils.cs">
-      <Link>Common\System\PerfUtils.cs</Link>
-    </Compile>
+    <ProjectReference Include="$(CommonTestPath)\Performance\PerfRunner\PerfRunner.csproj">
+      <Project>{69e46a6f-9966-45a5-8945-2559fe337827}</Project>
+      <Name>PerfRunner</Name>
+    </ProjectReference>
   </ItemGroup>
-  <!-- Optimizations to configure Xunit for performance -->
-  <PropertyGroup>
-    <IncludePerformanceTests>true</IncludePerformanceTests>
-  </PropertyGroup>
 ```
-(Replace Dictionary/List with whatever class you’re testing.)
-
 Once that’s all done, you can actually add tests to the file.
 
 Writing Test Cases

eng/Resources.targets

@@ -4,7 +4,7 @@
     <ResourcesSourceFileExtension Condition="'$(MSBuildProjectExtension)' == '.csproj'">.cs</ResourcesSourceFileExtension>
     <ResourcesSourceFileExtension Condition="'$(MSBuildProjectExtension)' == '.vbproj'">.vb</ResourcesSourceFileExtension>
 
-    <StringResourcesPath Condition="'$(StringResourcesPath)'=='' And Exists('$(MSBuildProjectDirectory)/Resources/Strings.resx')">$(MSBuildProjectDirectory)/Resources/Strings.resx</StringResourcesPath>
+    <StringResourcesPath Condition="'$(StringResourcesPath)' == '' AND Exists('$(MSBuildProjectDirectory)/Resources/Strings.resx')">$(MSBuildProjectDirectory)/Resources/Strings.resx</StringResourcesPath>
     <StringResourcesNamespace Condition="'$(StringResourcesNamespace)' == ''">System</StringResourcesNamespace>
     <StringResourcesClassName Condition="'$(StringResourcesClassName)' == ''">SR</StringResourcesClassName>
     <StringResourcesName Condition="'$(StringResourcesName)' == ''">FxResources.$(AssemblyName).$(StringResourcesClassName)</StringResourcesName>
@@ -16,7 +16,7 @@
   </PropertyGroup>
 
   <!-- Include files under StringResourcesPath by convention unless OmitResources is set. -->
-  <ItemGroup Condition="'$(StringResourcesPath)'!='' AND '$(OmitResources)'!='true'">
+  <ItemGroup Condition="'$(StringResourcesPath)' != '' AND '$(OmitResources)' != 'true'">
     <EmbeddedResource Include="$(StringResourcesPath)">
       <Visible>true</Visible>
       <ManifestResourceName>$(StringResourcesName)</ManifestResourceName>
@@ -25,59 +25,11 @@
     </EmbeddedResource>
   </ItemGroup>
 
-  <Choose>
-    <!-- Include common SR helper when resources are included unless SkipCommonResourcesIncludes is set. -->
-    <When Condition="Exists('$(StringResourcesPath)') AND '$(OmitResources)'!='true' AND '$(SkipCommonResourcesIncludes)'==''">
-      <ItemGroup>
-        <Compile Include="$(CommonPath)/System/SR$(ResourcesSourceFileExtension)">
-          <Visible>true</Visible>
-          <Link>Resources/Common/SR$(ResourcesSourceFileExtension)</Link>
-        </Compile>
-      </ItemGroup>
-    </When>
-  </Choose>
-
-  <PropertyGroup Condition="'$(BuildingUAPVertical)' == 'true'">
-    <CompileDependsOn>
-      CopyResxFilesToReswFiles;
-      $(CompileDependsOn);
-    </CompileDependsOn>
-  </PropertyGroup>
-
-  <Target Name="CalculateResourcesFileNames" Condition="'$(BuildingUAPVertical)' == 'true'">
-    <ItemGroup>
-      <_AllResxFiles Include="@(EmbeddedResource->'%(FullPath)')" Condition="'%(Extension)' == '.resx'">
-        <ReswName Condition="'%(EmbeddedResource.ReswName)' == ''">%(EmbeddedResource.ManifestResourceName)</ReswName>
-      </_AllResxFiles>
-    </ItemGroup>
-
-    <PropertyGroup>
-      <ProjectHasResources Condition="'@(_AllResxFiles)' != ''">true</ProjectHasResources>
-      <TestProjectNeedsModifiedPriFile Condition="'$(IsTestProject)' == 'true' AND '$(ProjectHasResources)' == 'true'">true</TestProjectNeedsModifiedPriFile>
-      <TestResourcesFolderPath Condition="'$(TestProjectNeedsModifiedPriFile)' == 'true'">$(RuntimePath)$(AssemblyName).Resw</TestResourcesFolderPath>
-      <_ResWDestinationPath Condition="'$(TestResourcesFolderPath)' != ''">$(TestResourcesFolderPath)</_ResWDestinationPath>
-      <_ResWDestinationPath Condition="'$(TestResourcesFolderPath)' == ''">$(ResourcesFolderPath)</_ResWDestinationPath>
-    </PropertyGroup>
-
-    <ItemGroup>
-      <_ReswOutputFiles Include="@(_AllResxFiles->'$(_ResWDestinationPath)/%(ReswName).resw')" />
-    </ItemGroup>
-
-  </Target>
-
-  <Target Name="CopyResxFilesToReswFiles"
-          Condition="'$(BuildingUAPVertical)' == 'true'"
-          DependsOnTargets="CalculateResourcesFileNames"
-          Inputs="@(_AllResxFiles)"
-          Outputs="@(_ReswOutputFiles)" >
-
-    <Copy SourceFiles="@(_AllResxFiles)"
-          DestinationFiles="@(_ReswOutputFiles)"
-          SkipUnchangedFiles="true">
-          <Output TaskParameter="CopiedFiles" ItemName="FilesCreated" />
-    </Copy>
-
-    <!-- We need to touch the copied files so that the target that uses them can track the inputs and outputs with the copied timestamp -->
-    <Touch Files="@(FilesCreated)" />
-  </Target>
+  <!-- Include common SR helper when resources are included unless SkipCommonResourcesIncludes is set. -->
+  <ItemGroup Condition="Exists('$(StringResourcesPath)') AND '$(OmitResources)' != 'true' AND '$(SkipCommonResourcesIncludes)' == ''">
+    <Compile Include="$(CommonPath)/System/SR$(ResourcesSourceFileExtension)">
+      <Visible>true</Visible>
+      <Link>Resources/Common/SR$(ResourcesSourceFileExtension)</Link>
+    </Compile>
+  </ItemGroup>
 </Project>

eng/dependencies.props

@@ -47,12 +47,16 @@
     <RuntimeNativeSystemDataSqlClientSniPackageVersion>4.4.0</RuntimeNativeSystemDataSqlClientSniPackageVersion>
 
     <!-- xunit test dependency version  -->
-    <MicrosoftDotNetTestSdkVersion>15.8.0</MicrosoftDotNetTestSdkVersion>
+    <MicrosoftNetTestSdkPackageVersion>16.0.0-preview-20181204-03</MicrosoftNetTestSdkPackageVersion>
     <XUnitPackageVersion>2.4.1-pre.build.4059</XUnitPackageVersion>
     <XUnitPerformancePackageVersion>1.0.0-beta-build0020</XUnitPerformancePackageVersion>
     <TraceEventPackageVersion>2.0.5</TraceEventPackageVersion>
     <UAPToolsPackageVersion>1.0.23</UAPToolsPackageVersion>
 
+    <!-- Code coverage package version -->
+    <CoverletConsolePackageVersion>1.3.0</CoverletConsolePackageVersion>
+    <DotNetReportGeneratorGlobalToolPackageVersion>4.0.4</DotNetReportGeneratorGlobalToolPackageVersion>
+
     <!-- Roslyn optimization data package version -->
     <MicrosoftDotNetIBCMergePackageVersion>4.6.0-alpha-00001</MicrosoftDotNetIBCMergePackageVersion>
     <TestILCAmd64retPackageVersion>$(ProjectNTfsTestILCPackageVersion)</TestILCAmd64retPackageVersion>
@@ -241,7 +245,7 @@
     </StaticDependency>
 
     <StaticDependency Include="Microsoft.NET.Test.Sdk">
-      <Version>$(MicrosoftDotNetTestSdkVersion)</Version>
+      <Version>$(MicrosoftNetTestSdkPackageVersion)</Version>
     </StaticDependency>
 
     <PerformancePackDependency Include="xunit.performance.core" />

src/libraries/Common/tests/Common.Tests.csproj

@@ -4,7 +4,7 @@
     <TestCategories>InnerLoop;OuterLoop</TestCategories>
     <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
     <Configurations>netcoreapp-Unix-Debug;netcoreapp-Unix-Release;netcoreapp-Windows_NT-Debug;netcoreapp-Windows_NT-Release</Configurations>
-    <CodeCoverageEnabled>false</CodeCoverageEnabled>
+    <CoverageSupported>false</CoverageSupported>
   </PropertyGroup>
   <ItemGroup>
     <Service Include="{82A7F48D-3B50-4B1E-B82E-3ADA8210C358}" />

src/libraries/Common/tests/Performance/BenchmarkFilter.cs

@@ -8,7 +8,7 @@
 using Xunit.Abstractions;
 using Xunit.Sdk;
 
-// Usage: [assembly: TestCaseOrderer("Microsoft.DotNet.XUnitExtensions.BenchmarkFilter", "System.Drawing.Common.Performance.Tests")]
+// Usage: [assembly: TestCaseOrderer("Microsoft.DotNet.XUnitExtensions.BenchmarkFilter", "System.Drawing.Common.PerformanceTests")]
 
 namespace Microsoft.DotNet.XUnitExtensions
 {

src/libraries/Common/tests/Performance/Common.Performance.Tests.csproj → src/libraries/Common/tests/Performance/Common.PerformanceTests.csproj

@@ -14,7 +14,7 @@
     </Compile>
   </ItemGroup>
   <ItemGroup Condition="'$(DisableTests)' != 'true'">
-    <ProjectReference Include="$(CommonPath)\..\perf\PerfRunner\PerfRunner.csproj">
+    <ProjectReference Include="$(CommonTestPath)\Performance\PerfRunner\PerfRunner.csproj">
       <Project>{69e46a6f-9966-45a5-8945-2559fe337827}</Project>
       <Name>PerfRunner</Name>
     </ProjectReference>

src/libraries/Common/perf/PerfRunner/PerfRunner.cs → src/libraries/Common/tests/Performance/PerfRunner/PerfRunner.cs

@@ -41,6 +41,6 @@ private static string GetTestAssembly(string testName)
 
     private static IEnumerable<string> GetTestAssemblies()
     {
-        return Directory.EnumerateFiles(".", "*.Performance.Tests.dll");
+        return Directory.EnumerateFiles(".", "*.PerformanceTests.dll");
     }
 }