Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System.IO.FileSystem.Tests fails on Libraries Test Run release coreclr OSX x64 Debug #70692

Open
buyaa-n opened this issue Jun 13, 2022 · 6 comments
Open

System.IO.FileSystem.Tests fails on Libraries Test Run release coreclr OSX x64 Debug #70692

buyaa-n opened this issue Jun 13, 2022 · 6 comments
Labels
area-System.IO
Milestone
Future

Comments

@buyaa-n
Copy link
Member

buyaa-n commented Jun 13, 2022

Failure found https://github.com/dotnet/runtime/pull/70534/checks?check_run_id=6867381404

LOG: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-70534-merge-9ef671fa37a148ccbf/System.IO.FileSystem.Tests/1/console.4e103995.log?helixlogtype=result

/tmp/helix/working/B98D0A0C/p/dotnet exec --runtimeconfig System.IO.FileSystem.Tests.runtimeconfig.json --depsfile System.IO.FileSystem.Tests.deps.json xunit.console.dll System.IO.FileSystem.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/private/tmp/helix/working/B98D0A0C/w/A43E08FB/e /private/tmp/helix/working/B98D0A0C/w/A43E08FB/e
  Discovering: System.IO.FileSystem.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.IO.FileSystem.Tests (found 3431 of 3946 test cases)
  Starting:    System.IO.FileSystem.Tests (parallel test collections = on, max threads = 6)
    System.IO.Tests.File_Create_str_i_fo.CaseSensitive [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.File_Exists.DoesCaseSensitiveComparisons [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.Directory_GetSetTimes.TimesIncludeMillisecondPart_LowTempRes [SKIP]
      Condition(s) not met: "LowTemporalResolution"
    System.IO.Tests.Directory_EnumFiles_str_str_so.SearchPatternCaseSensitive [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.Directory_EnumDir_str_str_so.SearchPatternCaseSensitive [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.Directory_EnumFiles_str_str.SearchPatternCaseSensitive [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.Directory_Exists.DoesCaseSensitiveComparisons [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.DirectoryInfo_GetDirectories_str_so.SearchPatternCaseSensitive [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.DirectoryInfo_EnumDir_str_str_so.SearchPatternCaseSensitive [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.DirectoryInfo_Exists.CaseSensitivity [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.Directory_EnumFSE_str_str.SearchPatternCaseSensitive [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.DirectoryInfo_EnumDir_str_str.SearchPatternCaseSensitive [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
./RunTests.sh: line 168: 57043 Segmentation fault: 11  "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.IO.FileSystem.Tests.runtimeconfig.json --depsfile System.IO.FileSystem.Tests.deps.json xunit.console.dll System.IO.FileSystem.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/private/tmp/helix/working/B98D0A0C/w/A43E08FB/e
----- end Mon Jun 13 11:25:08 PDT 2022 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.
ulimit -c value: 0
+ export _commandExitCode=139
+ _commandExitCode=139
+ /usr/local/bin/python3 /tmp/helix/working/B98D0A0C/p/reporter/run.py https://dev.azure.com/dnceng/ public 48320016 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJlNDI4NWM4Yy0zZmQ1LTQyNjctOWIxYy00MjE5NWM0N2E1NTMiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiJjYjVlYTE3NS1kZWJlLTQ0MmMtYmVhYS0zZGRjMjQ2MzIxOGQiLCJzaWQiOiJjYmY2NmZlMy1iN2EwLTRlNGItYWUyNS1hNjRhNDdkNjg2NGMiLCJCdWlsZElkIjoiOWVlNmQ0NzgtZDI4OC00N2Y3LWFhY2MtZjZlNmQwODJhZTZkOzE4MjA3MDUiLCJwcGlkIjoidnN0ZnM6Ly8vQnVpbGQvQnVpbGQvMTgyMDcwNSIsIm9yY2hpZCI6IjA3YmQwN2U5LTM1YzUtNDRiNi05OWI5LTUxMDUzMTExNGM4ZC5saWJyYXJpZXNfdGVzdF9ydW5fcmVsZWFzZV9jb3JlY2xyX29zeF94NjRfZGVidWcuX19kZWZhdWx0LjMiLCJpc3MiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tIiwiYXVkIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbXx2c286YjU1ZGU0ZWQtNGI1YS00MjE1LWE4ZTQtMGEwYTVmNzFlN2Q4IiwibmJmIjoxNjU1MTQzNTIxLCJleHAiOjE2NTUxNTM3MjF9.kn2lyvvqzHkTFYsn0v8jrbMlDdtDvK2LRKWM8qStFlh8Or3kQaHKurEM6yiDalwrEnKNRmeWKL1zDABKCearOQx62-eLSRnAcPeDCf9NX74yMmIVxBoGFw13hSK7p82bi2jEg07sK--Y0FnpNZVRgTu2VW3x9vH99eC_HLxVZ30W1kuedMqp_Q5qoU4Ft1oVVWmvPa0IZ6q9t3ulP56fo7cP5nezdJ212gZdGJarQwTcvj-f3dF23yk_B3sIW0ZA4WhtMHu0VjvWL8QUS1kD5WAO7z9aa3xIWocmvfOdbmOPoJ9MbcUCBaMUg1WhyCCV_gqUc67OB1sLzat5ApE5zA
2022-06-13T18:25:08.649Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2022-06-13T18:25:08.649Z	INFO   	run.py	__init__(42)	read_results	Searching '/private/tmp/helix/working/B98D0A0C/w/A43E08FB/e' for test results files
2022-06-13T18:25:08.650Z	INFO   	run.py	__init__(42)	read_results	Searching '/tmp/helix/working/B98D0A0C/w/A43E08FB/uploads' for test results files
2022-06-13T18:25:08.651Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2022-06-13T18:25:08.651Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/tmp/helix/working/B98D0A0C/w/A43E08FB/e/__test_report.json'
2022-06-13T18:25:08.651Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1411 bytes
+ /usr/local/bin/python3 /tmp/helix/working/B98D0A0C/p/gen-debug-dump-docs.py -buildid 1820705 -workitem System.IO.FileSystem.Tests -jobid 9ef671fa-37a1-48cc-bf4a-4cb08fd067b9 -outdir /tmp/helix/working/B98D0A0C/w/A43E08FB/uploads -templatedir /tmp/helix/working/B98D0A0C/p -dumpdir /cores -productver 7.0.0
Did not find dumps, skipping dump docs generation.
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jun 13, 2022
@ghost
Copy link

ghost commented Jun 13, 2022

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

Failure found https://github.com/dotnet/runtime/pull/70534/checks?check_run_id=6867381404

LOG: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-70534-merge-9ef671fa37a148ccbf/System.IO.FileSystem.Tests/1/console.4e103995.log?helixlogtype=result

/tmp/helix/working/B98D0A0C/p/dotnet exec --runtimeconfig System.IO.FileSystem.Tests.runtimeconfig.json --depsfile System.IO.FileSystem.Tests.deps.json xunit.console.dll System.IO.FileSystem.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/private/tmp/helix/working/B98D0A0C/w/A43E08FB/e /private/tmp/helix/working/B98D0A0C/w/A43E08FB/e
  Discovering: System.IO.FileSystem.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.IO.FileSystem.Tests (found 3431 of 3946 test cases)
  Starting:    System.IO.FileSystem.Tests (parallel test collections = on, max threads = 6)
    System.IO.Tests.File_Create_str_i_fo.CaseSensitive [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.File_Exists.DoesCaseSensitiveComparisons [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.Directory_GetSetTimes.TimesIncludeMillisecondPart_LowTempRes [SKIP]
      Condition(s) not met: "LowTemporalResolution"
    System.IO.Tests.Directory_EnumFiles_str_str_so.SearchPatternCaseSensitive [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.Directory_EnumDir_str_str_so.SearchPatternCaseSensitive [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.Directory_EnumFiles_str_str.SearchPatternCaseSensitive [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.Directory_Exists.DoesCaseSensitiveComparisons [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.DirectoryInfo_GetDirectories_str_so.SearchPatternCaseSensitive [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.DirectoryInfo_EnumDir_str_str_so.SearchPatternCaseSensitive [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.DirectoryInfo_Exists.CaseSensitivity [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.Directory_EnumFSE_str_str.SearchPatternCaseSensitive [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
    System.IO.Tests.DirectoryInfo_EnumDir_str_str.SearchPatternCaseSensitive [SKIP]
      Condition(s) not met: "IsCaseSensitiveOS"
./RunTests.sh: line 168: 57043 Segmentation fault: 11  "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.IO.FileSystem.Tests.runtimeconfig.json --depsfile System.IO.FileSystem.Tests.deps.json xunit.console.dll System.IO.FileSystem.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/private/tmp/helix/working/B98D0A0C/w/A43E08FB/e
----- end Mon Jun 13 11:25:08 PDT 2022 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.
ulimit -c value: 0
+ export _commandExitCode=139
+ _commandExitCode=139
+ /usr/local/bin/python3 /tmp/helix/working/B98D0A0C/p/reporter/run.py https://dev.azure.com/dnceng/ public 48320016 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJlNDI4NWM4Yy0zZmQ1LTQyNjctOWIxYy00MjE5NWM0N2E1NTMiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiJjYjVlYTE3NS1kZWJlLTQ0MmMtYmVhYS0zZGRjMjQ2MzIxOGQiLCJzaWQiOiJjYmY2NmZlMy1iN2EwLTRlNGItYWUyNS1hNjRhNDdkNjg2NGMiLCJCdWlsZElkIjoiOWVlNmQ0NzgtZDI4OC00N2Y3LWFhY2MtZjZlNmQwODJhZTZkOzE4MjA3MDUiLCJwcGlkIjoidnN0ZnM6Ly8vQnVpbGQvQnVpbGQvMTgyMDcwNSIsIm9yY2hpZCI6IjA3YmQwN2U5LTM1YzUtNDRiNi05OWI5LTUxMDUzMTExNGM4ZC5saWJyYXJpZXNfdGVzdF9ydW5fcmVsZWFzZV9jb3JlY2xyX29zeF94NjRfZGVidWcuX19kZWZhdWx0LjMiLCJpc3MiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tIiwiYXVkIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbXx2c286YjU1ZGU0ZWQtNGI1YS00MjE1LWE4ZTQtMGEwYTVmNzFlN2Q4IiwibmJmIjoxNjU1MTQzNTIxLCJleHAiOjE2NTUxNTM3MjF9.kn2lyvvqzHkTFYsn0v8jrbMlDdtDvK2LRKWM8qStFlh8Or3kQaHKurEM6yiDalwrEnKNRmeWKL1zDABKCearOQx62-eLSRnAcPeDCf9NX74yMmIVxBoGFw13hSK7p82bi2jEg07sK--Y0FnpNZVRgTu2VW3x9vH99eC_HLxVZ30W1kuedMqp_Q5qoU4Ft1oVVWmvPa0IZ6q9t3ulP56fo7cP5nezdJ212gZdGJarQwTcvj-f3dF23yk_B3sIW0ZA4WhtMHu0VjvWL8QUS1kD5WAO7z9aa3xIWocmvfOdbmOPoJ9MbcUCBaMUg1WhyCCV_gqUc67OB1sLzat5ApE5zA
2022-06-13T18:25:08.649Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2022-06-13T18:25:08.649Z	INFO   	run.py	__init__(42)	read_results	Searching '/private/tmp/helix/working/B98D0A0C/w/A43E08FB/e' for test results files
2022-06-13T18:25:08.650Z	INFO   	run.py	__init__(42)	read_results	Searching '/tmp/helix/working/B98D0A0C/w/A43E08FB/uploads' for test results files
2022-06-13T18:25:08.651Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2022-06-13T18:25:08.651Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/tmp/helix/working/B98D0A0C/w/A43E08FB/e/__test_report.json'
2022-06-13T18:25:08.651Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1411 bytes
+ /usr/local/bin/python3 /tmp/helix/working/B98D0A0C/p/gen-debug-dump-docs.py -buildid 1820705 -workitem System.IO.FileSystem.Tests -jobid 9ef671fa-37a1-48cc-bf4a-4cb08fd067b9 -outdir /tmp/helix/working/B98D0A0C/w/A43E08FB/uploads -templatedir /tmp/helix/working/B98D0A0C/p -dumpdir /cores -productver 7.0.0
Did not find dumps, skipping dump docs generation.
Author: buyaa-n
Assignees: -
Labels:

area-System.IO
Milestone: -

@danmoseley
Copy link
Member

No dump apparently. @MattGal I remember we were having issues because dumps on Mac could be huge. Did we switch them off entirely?

@MattGal
Copy link
Member

MattGal commented Jun 13, 2022

Did not find dumps, skipping dump docs generation.

No, all that checking occurs after the work item and the work item itself logged "Did not find dumps, skipping dump docs generation.". Sometimes cores just don't get generated, but core dump creation is definitely working in this queue:

let WorkIds=
WorkItems 
| where Finished > ago(10d)
| where QueueName == "osx.1200.amd64.open"
| project WorkItemId;
Files 
| where WorkItemId  in (WorkIds) 
| where FileName startswith "core."

@buyaa-n buyaa-n mentioned this issue Jun 14, 2022
Merged
@danmoseley
Copy link
Member

danmoseley commented Jun 15, 2022 •
edited
Loading

Interestingly though all the core dumps returned by that query are from runtime test configurations, mostly JIT. Eg., this returns nothing:

let WorkIds=
WorkItems 
| where Finished > ago(10d)
| where QueueName == "osx.1200.amd64.open"
| project WorkItemId;
Files 
| where WorkItemId  in (WorkIds) 
| where FileName startswith "core."
| where WorkItemFriendlyName startswith "System."

a while back I made the script dump ulimit, and i see it contains ulimit -c value: 0

and here is why

runtime/eng/testing/RunnerTemplate.sh

Lines 134 to 142 in 39078f5

if [[ ! -d "/cores" || ! "$(ls -A /cores)" ]]; then
# Disabling core dumps on macOS. System dumps are large (even for very small
# programs) and not configurable. As a result, if a single PR build causes a
# lot of tests to crash, we can take out the entire queue.
# See discussions in:
# https://github.com/dotnet/core-eng/issues/15333
# https://github.com/dotnet/core-eng/issues/15597
ulimit -c 0
fi

looking through the issues, it seems next action is here #65422

@hoyosjs @elinor-fung what do you see as the next action here and do either of you plan to take it on or do we need someone? (IMO we can't operate for long without the ability to diagnose test crashes on mac)

@hoyosjs
Copy link
Member

hoyosjs commented Jun 15, 2022

While we could, we still have a work item to investigate createdump dumps being large. capturing them is a matter of adding a couple env vars for all platforms, but I am not convinced that won't cause the same issue we saw before again (although helix has guards for it now)

@danmoseley
Copy link
Member

If Helix has throttling/limits now, is there any reason to not try re-enabling?

We could use the env vars recommended #65422 (comment) .. we'd at least have mini dumps, it seems.

@adamsitnik adamsitnik removed the untriaged New issue has not been triaged by the area owner label Jun 23, 2022
@adamsitnik adamsitnik added this to the Future milestone Jun 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-System.IO
Projects
None yet
Milestone
Future
Development

No branches or pull requests
5 participants
@adamsitnik @danmoseley @MattGal @hoyosjs @buyaa-n