dotnet test/vstest on multiple assemblies failed with concurrency exception

[blushingpenguin]

Description

Running dotnet vstest on multiple test assemblies frequently fails with an exception concurrently modifying a system.dictionary object

Steps to reproduce

Not sure, we just run a bunch of tests that work ok together and often they fail

Expected behavior

The tests should pass or fail (the runner shouldn't break)

Actual behavior

Occasionally the runner breaks

Diagnostic logs

dotnet vstest --Parallel a/a.dll b/b.dll c/c.dll d/d.dll e/e.dll f/f.dll g/g.dll
Microsoft (R) Test Execution Command Line Tool Version 17.3.1 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 7 test files matched the specified pattern.
The active test run was aborted. Reason: Error setting value to 'StoreKeyValuePairs' on 'Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase'.
The active test run was aborted. Reason: Error setting value to 'StoreKeyValuePairs' on 'Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase'.
The active test run was aborted. Reason: Error setting value to 'StoreKeyValuePairs' on 'Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase'.

Test Run Aborted with error System.Exception: One or more errors occurred.
 ---> System.Exception: Error setting value to 'StoreKeyValuePairs' on 'Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase'.
 ---> System.Exception: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
   at System.Collections.Generic.Dictionary`2.FindValue(TKey key)
   at System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value)
   at Microsoft.VisualStudio.TestPlatform.ObjectModel.TestProperty.GetType(String typeName)
   at Microsoft.VisualStudio.TestPlatform.ObjectModel.TestProperty.GetValueType()
   at Microsoft.VisualStudio.TestPlatform.ObjectModel.TestObject.set_StoreKeyValuePairs(List`1 value)
   at Newtonsoft.Json.Serialization.ExpressionValueProvider.SetValue(Object target, Object value)
   --- End of inner exception stack trace ---
   at Newtonsoft.Json.Serialization.ExpressionValueProvider.SetValue(Object target, Object value)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.JsonDataSerializer.DeserializePayload[T](Message message)
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestSender.OnExecutionMessageReceived(MessageReceivedEventArgs messageReceived, IInternalTestRunEventsHandler testRunEventsHandler)
   --- End of inner exception stack trace ---.

Environment

ubuntu 22.04 in a container on microk8s on ubuntu 22.04
dotnet is 6.0.11
vstest is Microsoft (R) Test Execution Command Line Tool Version 17.3.1 (x64)

AB#1819418

[Evangelink]

Hi @blushingpenguin, thanks for reporting this issue.

[Evangelink]

Could you please try to use dotnet test instead of dotnet vstest, the latter is no longer recommended. Please note, you won't need --Parallel with dotnet test.

[blushingpenguin]

maybe fixed by:
#3867

[Evangelink]

I don't think so :) There is still a Dictionary in the base class of TestCase, see https://github.com/microsoft/vstest/blob/main/src/Microsoft.TestPlatform.ObjectModel/TestObject.cs#L29

[AndersNestech]

Could you please try to use dotnet test instead of dotnet vstest, the latter is no longer recommended. Please note, you won't need --Parallel with dotnet test.

For the record: We have the same issue running dotnet test:

"C:\Program Files\dotnet\dotnet.exe" test --settings Test.RunSettings -v normal --logger trx;logfilename=testResults.trx <a bunch of .dlls>
Microsoft (R) Test Execution Command Line Tool Version 17.3.1 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 7 test files matched the specified pattern.
The active test run was aborted. Reason: Error setting value to 'StoreKeyValuePairs' on 'Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase'.
The active test run was aborted. Reason: Error setting value to 'StoreKeyValuePairs' on 'Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase'.
The active test run was aborted. Reason: Error setting value to 'StoreKeyValuePairs' on 'Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase'.
The active test run was aborted. Reason: Error setting value to 'StoreKeyValuePairs' on 'Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase'.
The active test run was aborted. Reason: Error setting value to 'StoreKeyValuePairs' on 'Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase'.
The active test run was aborted. Reason: Error setting value to 'StoreKeyValuePairs' on 'Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase'.
The active test run was aborted. Reason: Error setting value to 'StoreKeyValuePairs' on 'Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase'.

Test Run Aborted with error System.Exception: One or more errors occurred.
 ---> System.Exception: Error setting value to 'StoreKeyValuePairs' on 'Microsoft.VisualStudio.TestPlatform.ObjectModel.TestCase'.
 ---> System.Exception: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
   at System.Collections.Generic.Dictionary`2.FindValue(TKey key)
   at System.Collections.Generic.Dictionary`2.TryGetValue(TKey key, TValue& value)

... etc ...

   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.JsonDataSerializer.DeserializePayload[T](Message message)
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestSender.OnExecutionMessageReceived(MessageReceivedEventArgs messageReceived, IInternalTestRunEventsHandler testRunEventsHandler)

[blushingpenguin]

I also tried this -- it did "fix" the problem, but the tests now run in series (the total time has gone up to the sum of the running time of the individual tests rather than around the running time of the longest test)

[blushingpenguin]

and I've tried that with --parallel, and get the same problem

[AndersNestech]

I don't think so :) There is still a Dictionary in the base class of TestCase, see https://github.com/microsoft/vstest/blob/main/src/Microsoft.TestPlatform.ObjectModel/TestObject.cs#L29

@Evangelink: Any chance of this being fixed in the near future?

[Evangelink]

@AndersNestech I will raise the point in our next standup to see if we could bump priority for fixing this issue.