ERROR: The process "corerun.exe" not found.

[karelz]

Quite a few failures of tests end up with the error above.

Frequency:

• System.Data.OleDb.Tests - quite a few PRs (mixed with other errors)
• I think I have seen it on another test suite, but don't remember - I hope you can query logs better than I

Example of Console log:

C:\h\w\A2A20979\w\B1FA098B\e>"C:\h\w\A2A20979\p\dotnet.exe" exec --runtimeconfig System.Data.OleDb.Tests.runtimeconfig.json --depsfile System.Data.OleDb.Tests.deps.json xunit.console.dll System.Data.OleDb.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
Failed to load the dll from [C:\h\w\A2A20979\p\shared\Microsoft.NETCore.App\8.0.0\coreclr.dll], HRESULT: 0x8007045A
----- end Sat 09/10/2022  0:04:55.39 ----- exit code -1073740791 ----------------------------------------------------------
2022-09-10T00:04:55.984Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2022-09-10T00:04:55.984Z	INFO   	run.py	__init__(42)	read_results	Searching 'C:\h\w\A2A20979\w\B1FA098B\e' for test results files
2022-09-10T00:04:55.984Z	INFO   	run.py	__init__(42)	read_results	Searching 'C:\h\w\A2A20979\w\B1FA098B\uploads' for test results files
2022-09-10T00:04:55.984Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2022-09-10T00:04:55.984Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to 'C:\h\w\A2A20979\w\B1FA098B\e\__test_report.json'
2022-09-10T00:04:55.984Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1434 bytes
ERROR: The process "corerun.exe" not found.
read file: C:\h\w\A2A20979\p\debug-dump-template.md
writing output file: C:\h\w\A2A20979\w\B1FA098B\uploads\how-to-debug-dump.md
done writing debug dump information
['System.Data.OleDb.Tests' END OF WORK ITEM LOG: Command exited with -1073740791]

[MattGal]

@karelz these are just helix precommands that the Runtime team inserts (example) into their workitems. If you taskkill a process that isn't provided and isn't running (corerun became dotnet.exe ages ago in most cases) this is the sort of error you will get.

The DLL Init failure that's actually causing trouble here is probably worth investigating by someone on the runtime team though.

[karelz]

Oh, I entirely missed the DLL load failure, sorry!

[karelz]

@MattGal is there a way for you to query if it happens across multiple test suites, or if it is specific to OleDb?

[MattGal]

@karelz a quick check of this query:

WorkItems
| where JobName == "051647e5-f164-47c1-9afd-72ac67eb91ca"

... shows me the -1073740791 error is on every single work item in this entire job so I'd wager this is a payload construction issue. I'll dig a little deeper and see if it's obvious why.

[MattGal]

Jobs | where Name == "051647e5-f164-47c1-9afd-72ac67eb91ca"

Shows me that this comes from dotnet/runtime#74623, which is some experimental run by @jkoritzinsky (also interesting to note, we have > 96 days of work logged for just this one PR)

Some takeaways:

• The problem is not specific to OleDB: This STATUS_STACK_BUFFER_OVERRUN is all over and likely caused by the changes in the linked PR.
• Kusto queries and other means of exploring should generally exclude anything from PRs, for this exact reason.
• Not a problem in dotnet/arcade

[jkoritzinsky]

Yes, this is an issue with getting AddressSanitizer online in dotnet/runtime. I'm working on resolving these issues with the MSVC team until we get to a point that things are green. In the meantime, I know some legs (in particular the libraries test legs) are on the floor.

[MattGal]

@karelz can this be closed?

[karelz]

@MattGal if you can check db and the main problem is gone, then yes. I am behind monitoring CI now :(

[MattGal]

@karelz yes, I will close this then. Checking out that exit code in recent times:

let jobs = 
WorkItems
| where Finished > ago (20d)
| where FriendlyName == "System.Data.OleDb.Tests"
| where ExitCode == "-1073740791"
| project JobId;
Jobs | where JobId  in (jobs)

we see 100% of this specific failure are Jeremy's PR still.