Assertion failed: (*card_word)==0 in DynamicGenerics tests

[jkotas]

Failed test: nativeaot_SmokeTests::_DynamicGenerics_DynamicGenerics_DynamicGenerics_cmd (nativeaot\\SmokeTests\\DynamicGenerics\\DynamicGenerics\\DynamicGenerics.cmd)
Assertion failed: (*card_word)==0, file D:\a\_work\1\s\src\coreclr\gc\gc.cpp, line 13092

Failed in #76799 (windows.x64.Checked). Full log: https://dev.azure.com/dnceng-public/public/_build/results?buildId=45701&view=logs&jobId=58dc7ccb-0414-5dd3-62a5-bf2e63258b7c&j=ea7d6d51-d18d-5306-47bd-e3b7db04467e&t=3c9d9518-9046-50a7-9208-2e9137da71b6

{
  "ErrorMessage": "(*card_word)==0",
  "BuildRetry": false
}

Report

Build	Definition	Step Name	Console log	Pull Request
58837	dotnet/runtime	Run tests in single file mode	Log	#76985
59002	dotnet/runtime	Run tests in single file mode	Log	#62863
58981	dotnet/runtime	Run tests in single file mode	Log	#77137
58748	dotnet/runtime	Run tests in single file mode	Log	#77289
58346	dotnet/runtime	Run tests in single file mode	Log	#77272
58445	dotnet/runtime	Run tests in single file mode	Log	#77280
58398	dotnet/runtime	Run tests in single file mode	Log	#76901
58362	dotnet/runtime	Run tests in single file mode	Log	#77277
58366	dotnet/runtime	Run tests in single file mode	Log	#77277
58323	dotnet/runtime	Run tests in single file mode	Log	#77078
58278	dotnet/runtime	Run tests in single file mode	Log	#77267
57679	dotnet/runtime	Run tests in single file mode	Log	#77237
58192	dotnet/runtime	Run tests in single file mode	Log	#73472
58181	dotnet/runtime	Run tests in single file mode	Log	#77213
57257	dotnet/runtime	Run tests in single file mode	Log	#77213
58029	dotnet/runtime	Run tests in single file mode	Log	#77245
57955	dotnet/runtime	Run tests in single file mode	Log	#77239
57923	dotnet/runtime	Run tests in single file mode	Log	#77243
57916	dotnet/runtime	Run tests in single file mode	Log	#77189
57869	dotnet/runtime	Run tests in single file mode	Log	#62689
57894	dotnet/runtime	Run tests in single file mode	Log	#76981
57856	dotnet/runtime	Run tests in single file mode	Log	#76985
57711	dotnet/runtime	Run tests in single file mode	Log	#77228
57651	dotnet/runtime	Run tests in single file mode	Log	#74963
57635	dotnet/runtime	Run tests in single file mode	Log	#74157
57589	dotnet/runtime	Run tests in single file mode	Log	#74820
57537	dotnet/runtime	Run tests in single file mode	Log	#77231
57502	dotnet/runtime	Run tests in single file mode	Log	#77011
57470	dotnet/runtime	Run tests in single file mode	Log	#76981
57458	dotnet/runtime	Run tests in single file mode	Log	#75140
56859	dotnet/runtime	Run tests in single file mode	Log	#77196
57079	dotnet/runtime	Run tests in single file mode	Log	#77176
57047	dotnet/runtime	Run tests in single file mode	Log	#77214
56766	dotnet/runtime	Run tests in single file mode	Log	#77197
56623	dotnet/runtime	Run tests in single file mode	Log	#77189
56287	dotnet/runtime	Run tests in single file mode	Log	#77052
56168	dotnet/runtime	Run tests in single file mode	Log	#77102
54722	dotnet/runtime	Run tests in single file mode	Log	#76924
55967	dotnet/runtime	Run tests in single file mode	Log	#70941
55952	dotnet/runtime	Run tests in single file mode	Log	#77166
55933	dotnet/runtime	Run tests in single file mode	Log	#77147
55914	dotnet/runtime	Run tests in single file mode	Log	#77163
55901	dotnet/runtime	Run tests in single file mode	Log	#74820
55803	dotnet/runtime	Run tests in single file mode	Log	#77151
55653	dotnet/runtime	Run tests in single file mode	Log	#77107
55265	dotnet/runtime	Run tests in single file mode	Log	#77106
55251	dotnet/runtime	Run tests in single file mode	Log	#77132
55243	dotnet/runtime	Run tests in single file mode	Log	#76981
55062	dotnet/runtime	Run tests in single file mode	Log	#75542
55028	dotnet/runtime	Run tests in single file mode	Log	#76942
54886	dotnet/runtime	Run tests in single file mode	Log	#73472
54750	dotnet/runtime	Run tests in single file mode	Log	#77043
54700	dotnet/runtime	Run tests in single file mode	Log	#77112
54551	dotnet/runtime	Run tests in single file mode	Log	#77113
54522	dotnet/runtime	Run tests in single file mode	Log	#77107
54457	dotnet/runtime	Run tests in single file mode	Log	#77103
54175	dotnet/runtime	Run tests in single file mode	Log	#77065
54303	dotnet/runtime	Run tests in single file mode	Log	#77103
54153	dotnet/runtime	Run tests in single file mode	Log	#77102
53820	dotnet/runtime	Run tests in single file mode	Log	#77065
53751	dotnet/runtime	Run tests in single file mode	Log	#77080
53655	dotnet/runtime	Run tests in single file mode	Log	#76779
53449	dotnet/runtime	Run tests in single file mode	Log	#77056
53287	dotnet/runtime	Run tests in single file mode	Log	#77067
53170	dotnet/runtime	Run tests in single file mode	Log	#77062
52849	dotnet/runtime	Run tests in single file mode	Log	#76851
52807	dotnet/runtime	Run tests in single file mode	Log	#77052
52590	dotnet/runtime	Run tests in single file mode	Log	#77042
52728	dotnet/runtime	Run tests in single file mode	Log	#70941
52655	dotnet/runtime	Run tests in single file mode	Log	#75140
52585	dotnet/runtime	Run tests in single file mode	Log	#77003
52413	dotnet/runtime	Run tests in single file mode	Log	#76904
52315	dotnet/runtime	Run tests in single file mode	Log	#77035
52242	dotnet/runtime	Run tests in single file mode	Log	#76997
52128	dotnet/runtime	Run tests in single file mode	Log	#77027
52118	dotnet/runtime	Run tests in single file mode	Log	#76985
51963	dotnet/runtime	Run tests in single file mode	Log	#76981
50357	dotnet/runtime	Run tests in single file mode	Log	#76971
50156	dotnet/runtime	Run tests in single file mode	Log	#76950
51782	dotnet/runtime	Run tests in single file mode	Log	#68994
51604	dotnet/runtime	Run tests in single file mode	Log	#76890
51454	dotnet/runtime	Run tests in single file mode	Log	#77003
51463	dotnet/runtime	Run tests in single file mode	Log	#76981
51434	dotnet/runtime	Run tests in single file mode	Log	#77011
51394	dotnet/runtime	Run tests in single file mode	Log	#73472
51277	dotnet/runtime	Run tests in single file mode	Log	#76811
51266	dotnet/runtime	Run tests in single file mode	Log	#76848
50960	dotnet/runtime	Run tests in single file mode	Log	#76985
50629	dotnet/runtime	Run tests in single file mode	Log	#76972
50557	dotnet/runtime	Run tests in single file mode	Log	#74963
50518	dotnet/runtime	Run tests in single file mode	Log	#76975
50008	dotnet/runtime	Run tests in single file mode	Log	#76941
49938	dotnet/runtime	Run tests in single file mode	Log	#74157
49860	dotnet/runtime	Run tests in single file mode	Log	#76883
49784	dotnet/runtime	Run tests in single file mode	Log	#76933
49747	dotnet/runtime	Run tests in single file mode	Log	#76848
49477	dotnet/runtime	Run tests in single file mode	Log	#76901
49169	dotnet/runtime	Run tests in single file mode	Log	#62863
48098	dotnet/runtime	Run tests in single file mode	Log	#76339
48968	dotnet/runtime	Run tests in single file mode	Log	#76705
Displaying 100 of 125 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
3	63	125

[ghost]

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Failed test: nativeaot_SmokeTests::_DynamicGenerics_DynamicGenerics_DynamicGenerics_cmd (nativeaot\\SmokeTests\\DynamicGenerics\\DynamicGenerics\\DynamicGenerics.cmd)
Assertion failed: (*card_word)==0, file D:\a\_work\1\s\src\coreclr\gc\gc.cpp, line 13092

Failed in #76799. Full log: https://dev.azure.com/dnceng-public/public/_build/results?buildId=45701&view=logs&jobId=58dc7ccb-0414-5dd3-62a5-bf2e63258b7c&j=ea7d6d51-d18d-5306-47bd-e3b7db04467e&t=3c9d9518-9046-50a7-9208-2e9137da71b6

{
  "ErrorMessage": "(*card_word)==0",
  "BuildRetry": false
}

Author:	jkotas
Assignees:	-
Labels:	area-GC-coreclr, Known Build Error
Milestone:	-

[jkotas]

@dotnet/gc I suspect that this may be related to regions and frozen objects. Could you please review verify_card_bundles logic for any obvious problems?

[EgorBo]

I assume it started to appear after #76737 ?

[mangod9]

Assigning to Maoni assuming its related to enabling regions on NativeAOT?

[Maoni0]

Andrew is looking at this.

[trylek]

I also think there's something wrong with reporting - I saw this in my recent PR testing and while detailed scrutiny of the log ultimately revealed the assertion failure, the summary at the end of the log says

cmdLine:D:\a\_work\1\s\artifacts\tests\coreclr\windows.x64.Checked\nativeaot\SmokeTests\DynamicGenerics\DynamicGenerics\DynamicGenerics.cmd Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 10/12/2022 8:44:52 PM, end: 10/12/2022 9:14:52 PM)
Test Harness Exitcode is : -100

A possible explanation is that in my testing the assertion failure happened very close to the 30 minute timeout for the test.

[cshung]

I also think there's something wrong with reporting - I saw this in my recent PR testing and while detailed scrutiny of the log ultimately revealed the assertion failure, the summary at the end of the log says

cmdLine:D:\a_work\1\s\artifacts\tests\coreclr\windows.x64.Checked\nativeaot\SmokeTests\DynamicGenerics\DynamicGenerics\DynamicGenerics.cmd Timed Out (timeout in milliseconds: 1800000 from variable __TestTimeout, start: 10/12/2022 8:44:52 PM, end: 10/12/2022 9:14:52 PM)
Test Harness Exitcode is : -100
A possible explanation is that in my testing the assertion failure happened very close to the 30-minute timeout for the test.

I noticed that when the assertion happened on my machine, it stuck at a dialog box, so that might explain the timeout.

[trylek]

Thanks Andrew, I guess that explains a lot, in CoreCLR testing we're going a long way towards suppressing all such dialogs by disabling WER UI and by other means, it might be the case that the relatively recent NativeAOT testing is still missing some of these measures.

[cshung]

Still stuck with this one, but I found something interesting.

It appears that when the assertion happens, the line being printed is always this:
Running Test: B282745.testMDArrayWith3Dimensions

[jkotas]

NativeAOT on Windows is building with hardware write watch currently. It does not build with FEATURE_USE_SOFTWARE_WRITE_WATCH_FOR_GC_HEAP and FEATURE_MANUALLY_MANAGED_CARD_BUNDLES defined. Can this config difference lead to this assertion failure?

[Maoni0]

I was surprised to see that these 2 defines are not defined for AoT. it does expose a bug with regions (when we make used larger for regions we are not using the OS writewatch to watch the extended portion of the card table so we are not getting updated from that part), however, I care more about making AoT have these 2 defined on windows. @cshung will open an issue for AoT folks. we'd like to get rid of the code that's still using the OS writewatch completely.

[cshung]

Before #77224, I am able to reproduce the bug locally in a few minutes. After the fix, I am unable to repro it locally for 4 hours. Closing this issue opportunistically to see if that repro again. Feel free to reopen if the bug repros.