Test failure in System.Transactions.Tests.OleTxTests.* tests

[BruceForstall]

These tests are failing in various JitStress pipelines on Windows arm64:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=45833&view=ms.vss-test-web.build-test-results-tab&runId=920554&paneView=debug&resultId=191806

e.g.,

    System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments(volatileCount: 0) [FAIL]
      System.AggregateException : One or more errors occurred. (The operation is not valid for the state of the transaction.) (The following constructor parameters did not have matching fixture data: OleTxFixture fixture)
      ---- System.Transactions.TransactionException : The operation is not valid for the state of the transaction.
      -------- System.Transactions.TransactionManagerCommunicationException : Communication with the underlying transaction manager has failed.
      ------------ System.Runtime.InteropServices.COMException : The Transaction Manager is not available. (0x8004D01B)
      ---- The following constructor parameters did not have matching fixture data: OleTxFixture fixture
      Stack Trace:
        
        ----- Inner Stack Trace #1 (System.Transactions.TransactionException) -----
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/TransactionState.cs(203,0): at System.Transactions.TransactionState.EnlistDurable(InternalTransaction tx, Guid resourceManagerIdentifier, IEnlistmentNotification enlistmentNotification, EnlistmentOptions enlistmentOptions, Transaction atomicTransaction)
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/Transaction.cs(513,0): at System.Transactions.Transaction.EnlistDurable(Guid resourceManagerIdentifier, IEnlistmentNotification enlistmentNotification, EnlistmentOptions enlistmentOptions)
        /_/src/libraries/System.Transactions.Local/tests/OleTxTests.cs(569,0): at System.Transactions.Tests.OleTxTests.OleTxFixture.<>c.<.ctor>b__0_0()
        /_/src/libraries/System.Transactions.Local/tests/OleTxTests.cs(512,0): at System.Transactions.Tests.OleTxTests.Test(Action action)
        /_/src/libraries/System.Transactions.Local/tests/OleTxTests.cs(558,0): at System.Transactions.Tests.OleTxTests.OleTxFixture..ctor()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/ConstructorInvoker.cs(86,0): at System.Reflection.ConstructorInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
        ----- Inner Stack Trace -----
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/Oletx/OletxTransactionManager.cs(610,0): at System.Transactions.Oletx.OletxTransactionManager.ProxyException(COMException comException)
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/Oletx/DtcTransactionManager.cs(67,0): at System.Transactions.Oletx.DtcTransactionManager.Initialize()
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/Oletx/DtcTransactionManager.cs(83,0): at System.Transactions.Oletx.DtcTransactionManager.get_ProxyShimFactory()
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/Oletx/OletxTransactionManager.cs(481,0): at System.Transactions.Oletx.OletxTransactionManager.CreateTransaction(TransactionOptions properties)
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/TransactionState.cs(2234,0): at System.Transactions.TransactionStatePromoted.EnterState(InternalTransaction tx)
        ----- Inner Stack Trace -----
           at System.Transactions.DtcProxyShim.DtcInterfaces.ITransactionImportWhereabouts.GetWhereaboutsSize(UInt32& pcbSize)
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/DtcProxyShim/OletxHelper.cs(54,0): at System.Transactions.DtcProxyShim.OletxHelper.Retry(Action action)
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/DtcProxyShim/DtcProxyShimFactory.cs(140,0): at System.Transactions.DtcProxyShim.DtcProxyShimFactory.ConnectToProxyCore(String nodeName, Guid resourceManagerIdentifier, Object managedIdentifier, Boolean& nodeNameMatches, Byte[]& whereabouts, ResourceManagerShim& resourceManagerShim)
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/Oletx/DtcTransactionManager.cs(67,0): at System.Transactions.Oletx.DtcTransactionManager.Initialize()
        ----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----
   

This looks like the same issue as #75822, theoretically fixed by #76310. However, it seems the fix isn't sufficient.

@roji Can you please take a look?

{
  "ErrorMessage": "System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments",
  "BuildRetry": false
}

Report

Build	Definition	Step Name	Console log	Pull Request
55940	dotnet/runtime	Send to Helix	Log	#77103
55707	dotnet/runtime	Send to Helix	Log
54855	dotnet/runtime	Send to Helix	Log	#77103
54385	dotnet/runtime	Send to Helix	Log
54100	dotnet/runtime	Send to Helix	Log
54038	dotnet/runtime	Send to Helix	Log
54017	dotnet/runtime	Send to Helix	Log
53798	dotnet/runtime	Send to Helix	Log
53768	dotnet/runtime	Send to Helix	Log
53591	dotnet/runtime	Send to Helix	Log
53566	dotnet/runtime	Send to Helix	Log
52661	dotnet/runtime	Send to Helix	Log	#75140
52628	dotnet/runtime	Send to Helix	Log	#75634
52442	dotnet/runtime	Send to Helix	Log
51029	dotnet/runtime	Send to Helix	Log
50934	dotnet/runtime	Send to Helix	Log
49280	dotnet/runtime	Send to Helix	Log
48224	dotnet/runtime	Send to Helix	Log	#76883
47531	dotnet/runtime	Send to Helix	Log
47404	dotnet/runtime	Send to Helix	Log
45833	dotnet/runtime	Send to Helix	Log

Build	Definition	Test	Pull Request
55940	dotnet/runtime	System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments	#77103
55707	dotnet/runtime	System.Transactions.Tests.OleTxTests.ImplicitDistributedTransactions_cannot_be_changed_after_being_read_as_true
54855	dotnet/runtime	System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments	#77103
54385	dotnet/runtime	System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments
54100	dotnet/runtime	System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments
54038	dotnet/runtime	System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments
54017	dotnet/runtime	System.Transactions.Tests.OleTxTests.GetDtcTransaction
53798	dotnet/runtime	System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments
53768	dotnet/runtime	System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments
53591	dotnet/runtime	System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments
53566	dotnet/runtime	System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments
52661	dotnet/runtime	System.Transactions.Tests.OleTxTests.GetDtcTransaction	#75140
52628	dotnet/runtime	System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments	#75634
52442	dotnet/runtime	System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments
51029	dotnet/runtime	System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments
50934	dotnet/runtime	System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments
49280	dotnet/runtime	System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments
48224	dotnet/runtime	System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments	#76883
47531	dotnet/runtime	System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments
47404	dotnet/runtime	System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments
45833	dotnet/runtime	System.Transactions.Tests.OleTxTests.TransmitterPropagationToken

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
4	34	42

[ghost]

Tagging subscribers to this area: @roji
See info in area-owners.md if you want to be subscribed.

Issue Details

---

These tests are failing in various JitStress pipelines on Windows arm64:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=45833&view=ms.vss-test-web.build-test-results-tab&runId=920554&paneView=debug&resultId=191806

e.g.,

    System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments(volatileCount: 0) [FAIL]
      System.AggregateException : One or more errors occurred. (The operation is not valid for the state of the transaction.) (The following constructor parameters did not have matching fixture data: OleTxFixture fixture)
      ---- System.Transactions.TransactionException : The operation is not valid for the state of the transaction.
      -------- System.Transactions.TransactionManagerCommunicationException : Communication with the underlying transaction manager has failed.
      ------------ System.Runtime.InteropServices.COMException : The Transaction Manager is not available. (0x8004D01B)
      ---- The following constructor parameters did not have matching fixture data: OleTxFixture fixture
      Stack Trace:
        
        ----- Inner Stack Trace #1 (System.Transactions.TransactionException) -----
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/TransactionState.cs(203,0): at System.Transactions.TransactionState.EnlistDurable(InternalTransaction tx, Guid resourceManagerIdentifier, IEnlistmentNotification enlistmentNotification, EnlistmentOptions enlistmentOptions, Transaction atomicTransaction)
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/Transaction.cs(513,0): at System.Transactions.Transaction.EnlistDurable(Guid resourceManagerIdentifier, IEnlistmentNotification enlistmentNotification, EnlistmentOptions enlistmentOptions)
        /_/src/libraries/System.Transactions.Local/tests/OleTxTests.cs(569,0): at System.Transactions.Tests.OleTxTests.OleTxFixture.<>c.<.ctor>b__0_0()
        /_/src/libraries/System.Transactions.Local/tests/OleTxTests.cs(512,0): at System.Transactions.Tests.OleTxTests.Test(Action action)
        /_/src/libraries/System.Transactions.Local/tests/OleTxTests.cs(558,0): at System.Transactions.Tests.OleTxTests.OleTxFixture..ctor()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/ConstructorInvoker.cs(86,0): at System.Reflection.ConstructorInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
        ----- Inner Stack Trace -----
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/Oletx/OletxTransactionManager.cs(610,0): at System.Transactions.Oletx.OletxTransactionManager.ProxyException(COMException comException)
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/Oletx/DtcTransactionManager.cs(67,0): at System.Transactions.Oletx.DtcTransactionManager.Initialize()
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/Oletx/DtcTransactionManager.cs(83,0): at System.Transactions.Oletx.DtcTransactionManager.get_ProxyShimFactory()
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/Oletx/OletxTransactionManager.cs(481,0): at System.Transactions.Oletx.OletxTransactionManager.CreateTransaction(TransactionOptions properties)
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/TransactionState.cs(2234,0): at System.Transactions.TransactionStatePromoted.EnterState(InternalTransaction tx)
        ----- Inner Stack Trace -----
           at System.Transactions.DtcProxyShim.DtcInterfaces.ITransactionImportWhereabouts.GetWhereaboutsSize(UInt32& pcbSize)
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/DtcProxyShim/OletxHelper.cs(54,0): at System.Transactions.DtcProxyShim.OletxHelper.Retry(Action action)
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/DtcProxyShim/DtcProxyShimFactory.cs(140,0): at System.Transactions.DtcProxyShim.DtcProxyShimFactory.ConnectToProxyCore(String nodeName, Guid resourceManagerIdentifier, Object managedIdentifier, Boolean& nodeNameMatches, Byte[]& whereabouts, ResourceManagerShim& resourceManagerShim)
        /_/src/libraries/System.Transactions.Local/src/System/Transactions/Oletx/DtcTransactionManager.cs(67,0): at System.Transactions.Oletx.DtcTransactionManager.Initialize()
        ----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----
   

This looks like the same issue as #75822, theoretically fixed by #76310. However, it seems the fix isn't sufficient.

@roji Can you please take a look?

Author:	BruceForstall
Assignees:	-
Labels:	arch-arm64, area-System.Transactions, os-windows
Milestone:	8.0.0

[BruceForstall]

More cases in another JitStress run:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=47404&view=ms.vss-test-web.build-test-results-tab

[roji]

Am trying to increase the timeout again in #76892, let's see if that makes it go away for good.

[BruceForstall]

There are still failures after #76892:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=49280&view=ms.vss-test-web.build-test-results-tab&runId=991986&resultId=197182&paneView=debug

[markples]

Failures are not limited to JITStress. For example:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=51029&view=logs&j=f9affed7-8d82-5c58-eb09-b33088a0b629&t=841cba2d-9b8a-52d6-e2e0-be8034011045

https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-02e36dcb1adc47a284/System.Transactions.Local.Tests/1/console.94f94a03.log?helixlogtype=result

D:\h\w\B22D09C8\w\A6F5097B\e>set COMPlus 
COMPlus_TieredCompilation=1

D:\h\w\B22D09C8\w\A6F5097B\e>call RunTests.cmd --runtime-path D:\h\w\B22D09C8\p 
----- start Thu 10/13/2022  3:57:27.62 ===============  To repro directly: ===================================================== 
pushd D:\h\w\B22D09C8\w\A6F5097B\e\
"D:\h\w\B22D09C8\p\dotnet.exe" exec --runtimeconfig System.Transactions.Local.Tests.runtimeconfig.json --depsfile System.Transactions.Local.Tests.deps.json xunit.console.dll System.Transactions.Local.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================

D:\h\w\B22D09C8\w\A6F5097B\e>"D:\h\w\B22D09C8\p\dotnet.exe" exec --runtimeconfig System.Transactions.Local.Tests.runtimeconfig.json --depsfile System.Transactions.Local.Tests.deps.json xunit.console.dll System.Transactions.Local.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Transactions.Local.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Transactions.Local.Tests (found 137 of 149 test cases)
  Starting:    System.Transactions.Local.Tests (parallel test collections = on, max threads = 8)
    System.Transactions.Tests.OleTxTests.Volatile_and_durable_enlistments(volatileCount: 0) [FAIL]
      System.AggregateException : One or more errors occurred. (The operation is not valid for the state of the transaction.) (The following constructor parameters did not have matching fixture data: OleTxFixture fixture)

[roji]

Thanks, I'll finish #77023 in the next few days.