[wasm][aot] System.Text.Json tests fail while linking due to OOM
#61524
Comments
dotnet-issue-labeler
bot
added
area-Infrastructure-mono
untriaged
|
Tagging subscribers to this area: @directhex Issue Details
The error log looks similar to this issue:
|
|
Tagging subscribers to 'arch-wasm': @lewing Issue Details
The error log looks similar to this issue: Same failure in #61491
|
This is the VM killing the test because it ran out of memory. cc @steveisok |
|
cc @ilyas1974 |
ericstj
added
the
blocking-clean-ci
|
Still happening in release/6.0. https://dev.azure.com/dnceng/public/_build/results?buildId=1541389&view=ms.vss-test-web.build-test-results-tab&runId=43596408&resultId=158013&paneView=debug |
|
In my case #63507 (comment) it was a 1 hour timeout. The one above would seem to be as well, although the step only took 45 mins not an hour. Is it possible to make wasm-opt more verbose (when in CI) so we might get an idea how far it gets ( |
|
Again #63527 |
|
@danmoseley I may be butchering the explanation somewhat, but STJ is making LLVM work a bit too hard (generated code size is pretty large) and sometimes we run out of memory. The only two options I know of are 1.) run outside of a docker container and see if the problem goes away or 2.) bump the memory. I had pushed for 2 outright, but we're going to have to make the case by trying 1. @lewing Correct me if I am wrong, but changing optimization levels I don't think will help all that much. |
We tried that before, and it was helping for some time but then it started to fail again like this. |
|
This test code may be large, but it's not interconnected. Why would a large but shallow assembly run into memory limits? To put it another way: I can imagine a fix here is to chunk the one test assembly into a bunch of smaller ones, then I'd ask: why can't LLVM realize it can do something like this when building? Don't we imagine customers running into similar limits? It doesn't feel like a build toolchain should ever OOM as a failure point. |
radical
changed the title
System.Text.Json tests fail while linking due to OOM
- dotnet#65356 - OOM while linking `System.Text.Json.SourceGeneration.Roslyn3.11.Tests` - dotnet#65411 - possible OOM when compiling `System.Text.Json.SourceGeneration.Roslyn4.0.Tests.dll.bc` -> `.o` `System.Text.Json.SourceGeneration.Roslyn4.0.Tests` - dotnet#61524 - OOM while linking `System.Text.Json.Tests`
radical
added
disabled-test
* [wasm] Disable aot tests failing due to OOM - #65356 - OOM while linking `System.Text.Json.SourceGeneration.Roslyn3.11.Tests` - #65411 - possible OOM when compiling `System.Text.Json.SourceGeneration.Roslyn4.0.Tests.dll.bc` -> `.o` `System.Text.Json.SourceGeneration.Roslyn4.0.Tests` - #61524 - OOM while linking `System.Text.Json.Tests` * Disable System.Collections.Immutable.Tests due to #65335
|
Disabled the tests in #65413, so removing |
radical
removed
the
blocking-clean-ci
|
@radical I'm still seeing |
…configuration Partial backport of dotnet#65413 dotnet#61524 - OOM while linking `System.Text.Json.Tests`
ghost
added
the
in-pr
…67017) Some library tests fail to AOT on helix by getting oomkill'ed. Since the build machine have more resources than helix, we can AOT these specific library tests there, and send the built output to helix to run. Specifically: - `System.Text.Json.SourceGeneration.Roslyn3.11.Tests` - `System.Text.Json.SourceGeneration.Roslyn4.0.Tests` - `System.Text.Json.Tests` - `Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests` This adds two new jobs to `runtime-wasm`, and for rolling builds - `LibraryTests_HighResource_AOT` for linux, and windows. Fixes #65356 Fixes #65411 Fixes #61524 Fixes #66647
ghost
removed
the
in-pr
…otnet#67017) Some library tests fail to AOT on helix by getting oomkill'ed. Since the build machine have more resources than helix, we can AOT these specific library tests there, and send the built output to helix to run. Specifically: - `System.Text.Json.SourceGeneration.Roslyn3.11.Tests` - `System.Text.Json.SourceGeneration.Roslyn4.0.Tests` - `System.Text.Json.Tests` - `Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests` This adds two new jobs to `runtime-wasm`, and for rolling builds - `LibraryTests_HighResource_AOT` for linux, and windows. Fixes dotnet#65356 Fixes dotnet#65411 Fixes dotnet#61524 Fixes dotnet#66647
ghost
locked as resolved and limited conversation to collaborators
System.Text.Json.Testsfailing inBuild Browser wasm Release AllSubsets_Mono_AOTfor [release/6.0] branch on this PRThe error log looks similar to this issue:
Same failure in #61491
Runfo Tracking Issue: System.Text.Json.Tests failing in Build Browser wasm Release AllSubsets_Mono_AOT leg
Build Result Summary
The text was updated successfully, but these errors were encountered: