Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate flaky test/async-hooks/test-callback-error #15985

Closed
efreiberg opened this issue Oct 6, 2017 · 19 comments
Closed

Investigate flaky test/async-hooks/test-callback-error #15985

efreiberg opened this issue Oct 6, 2017 · 19 comments
Labels
async_hooks Issues and PRs related to the async hooks subsystem. flaky-test Issues and PRs related to the tests with unstable failures on the CI. macos Issues and PRs related to the macOS platform / OSX. test Issues and PRs related to the tests.

Comments

@efreiberg
Copy link
Contributor

efreiberg commented Oct 6, 2017

Running test/async-hooks/test-callback-error.js test on the master branch(no local changes) reports failures on async hooks tests.

  • Version: v9.0.0-pre
  • Platform: Darwin Kernel Version 15.5.0: Tue Apr 19 18:36:36 PDT 2016; root:xnu-3248.50.21~8/RELEASE_X86_64 x86_64
  • Subsystem: async_hooks
$ ./node ./test/async-hooks/test-callback-error.js 
start case 1
end case 1: 85.844ms
start case 2
end case 2: 86.863ms
start case 3
end case 3: 4.478ms
Error: test_callback_abort
    at ActivityCollector.initHooks.oninit.common.mustCall (/Users/efreiberg/Documents/node_demo_2/node/test/async-hooks/test-callback-error.js:36:45)
    at ActivityCollector.oninit (/Users/efreiberg/Documents/node_demo_2/node/test/common/index.js:517:15)
    at ActivityCollector._init (/Users/efreiberg/Documents/node_demo_2/node/test/async-hooks/init-hooks.js:182:10)
    at emitInitNative (async_hooks.js:466:43)
    at Object.emitInitScript [as emitInit] (async_hooks.js:369:3)
    at Object.<anonymous> (/Users/efreiberg/Documents/node_demo_2/node/test/async-hooks/test-callback-error.js:38:17)
    at Module._compile (module.js:600:30)
    at Object.Module._extensions..js (module.js:611:10)
    at Module.load (module.js:521:32)
    at tryModuleLoad (module.js:484:12)
 1: node::Abort() [/Users/efreiberg/Documents/node_demo_2/node/out/Release/node]
 2: node::Chdir(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/efreiberg/Documents/node_demo_2/node/out/Release/node]
 3: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [/Users/efreiberg/Documents/node_demo_2/node/out/Release/node]
 4: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/Users/efreiberg/Documents/node_demo_2/node/out/Release/node]
 5: v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/Users/efreiberg/Documents/node_demo_2/node/out/Release/node]
 6: 0x35a344a0463d
$ sw_vers -productVersion 
10.11.5
@ghost
Copy link

ghost commented Oct 6, 2017

Getting the same error in OSX10.13 too. It is intermittent.

@mscdex mscdex added async_hooks Issues and PRs related to the async hooks subsystem. test Issues and PRs related to the tests. labels Oct 6, 2017
@cjihrig
Copy link
Contributor

cjihrig commented Oct 7, 2017

Related to #15208?

@Trott
Copy link
Member

Trott commented Oct 7, 2017

@Fishrock123 was investigating this at Code + Learn and might have some more information to add.

@MylesBorins
Copy link
Contributor

Just ran into this while running the 8.x test suite on my 10.10.5 machine

@gibfahn
Copy link
Member

gibfahn commented Dec 13, 2017

I think I'm seeing the same thing on v8.x, on both macOS and Linux.

=== release test-callback-error ===                                            
Path: async-hooks/test-callback-error
start case 1
end case 1: 380.437ms
start case 2
end case 2: 648.707ms
start case 3
end case 3: 12.625ms
Error: test_callback_abort
    at ActivityCollector.initHooks.oninit.common.mustCall (/Users/gib/wrk/com/DANGER/node/test/async-hooks/test-callback-error.js:36:45)
    at ActivityCollector.oninit (/Users/gib/wrk/com/DANGER/node/test/common/index.js:533:15)
    at ActivityCollector._init (/Users/gib/wrk/com/DANGER/node/test/async-hooks/init-hooks.js:185:10)
    at emitInitNative (async_hooks.js:472:43)
    at Object.emitInitScript [as emitInit] (async_hooks.js:388:3)
    at Object.<anonymous> (/Users/gib/wrk/com/DANGER/node/test/async-hooks/test-callback-error.js:38:17)
    at Module._compile (module.js:635:30)
    at Object.Module._extensions..js (module.js:646:10)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
 1: node::Abort() [/Users/gib/wrk/com/DANGER/node/out/Release/node]
 2: node::Chdir(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/gib/wrk/com/DANGER/node/out/Release/node]
 3: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [/Users/gib/wrk/com/DANGER/node/out/Release/node]
 4: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/Users/gib/wrk/com/DANGER/node/out/Release/node]
 5: v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/Users/gib/wrk/com/DANGER/node/out/Release/node]
 6: 0x2ae711d8463d

I'm also seeing cores dumped even when the test passes, which is odd. Testing with commit ab46b8e.

@Trott
Copy link
Member

Trott commented Dec 14, 2017

@nodejs/async_hooks

@maclover7 maclover7 added the macos Issues and PRs related to the macOS platform / OSX. label Dec 18, 2017
@kysnm
Copy link
Contributor

kysnm commented Dec 19, 2017

Getting the same error in
v10.0.0-pre
sw_vers -productVersion
10.11.6

@kysnm
Copy link
Contributor

kysnm commented Dec 19, 2017

Error does not occur in the following environment
v 10.0.0-pre
sw_vers -productVersion
10.12.6

@maclover7 maclover7 added the flaky-test Issues and PRs related to the tests with unstable failures on the CI. label Dec 25, 2017
@shisama
Copy link
Contributor

shisama commented May 25, 2018

Hasn't this problem been fixed yet?

Error occurs in
v 11.0.0-pre
sw_vers -productVersion
10.13.4

@Trott
Copy link
Member

Trott commented May 25, 2018

Hasn't this problem been fixed yet?

Unless someone knows the source of the problem, how can anyone know if it's fixed? The error happens unpredictably. I'm not convinced the macOS version is a significant factor. I'm running 10.11.6 and do not see this error occurring.

@shisama
Copy link
Contributor

shisama commented May 26, 2018

Unless someone knows the source of the problem, how can anyone know if it's fixed?

Sorry, I had a bad comment. I don't know the source of the problem.

I'm not convinced the macOS version is a significant factor.

The error occurs once yesterday. Today, the error doesn't occurs despite the same version.

@Trott
Copy link
Member

Trott commented May 27, 2018

Sorry, I had a bad comment. I don't know the source of the problem.

I was just trying to understand the question. I apologize if it seemed like I was judging the quality of your comment. I think I genuinely misunderstood you but I think I understand now.

@Trott
Copy link
Member

Trott commented Jun 8, 2018

Can folks who are experiencing this try this out to see if it solves the problem?

launchctl unload -w /System/Library/LaunchAgents/com.apple.ReportCrash.plist
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.ReportCrash.Root.plist

(I think that disables writing crash reports to disk. So, you know, if you care about that stuff, don't do it, or know how to reverse it, which is probably running launchctl load rather than launchctl unload but check the docs to make sure I'm right about that before doing anything, ok? ok.)

@shisama
Copy link
Contributor

shisama commented Jun 11, 2018

@Trott Althogh error occurred before running the launchctl unload commands, error doesn't occur after running them.

@Trott
Copy link
Member

Trott commented Jun 11, 2018

Hmmm...OK, so I'm of the opinion that while it's acceptable that the crash report writing on macOS might cause a test to timeout, it should not cause a test to crash.

@nodejs/async_hooks @nodejs/testing @nodejs/platform-macos Are we all on board that this is a genuine bug to be fixed? Or is there a subtlety I'm missing such that this is acceptable behavior for the test?

@addaleax
Copy link
Member

@Trott The crashes here are part of the tests, so, for one, we should probably move it to test/abort?

@joyeecheung
Copy link
Member

joyeecheung commented Aug 13, 2018

This happened quite a few times again recently (same stack)

Refs: nodejs/reliability#12

Reason     async-hooks/test-callback-error
Type       JS_TEST_FAILURE
Failed PR  6 (#22266, #22211, #22274, #22289, #22284, #22293)
Appeared   test-digitalocean-fedora27-x64-1
First CI   https://ci.nodejs.org/job/node-test-pull-request/16383/
Last CI    https://ci.nodejs.org/job/node-test-pull-request/16413/

@joyeecheung joyeecheung changed the title Async hook test 'test/async-hooks/test-callback-error' fails on OSX 10.11.5 Investigate flaky test/async-hooks/test-callback-error Aug 14, 2018
@joyeecheung
Copy link
Member

Note that this also happens on fedora

joyeecheung added a commit to joyeecheung/node that referenced this issue Aug 15, 2018
joyeecheung added a commit that referenced this issue Aug 15, 2018
Refs: #15985

PR-URL: #22330
Reviewed-By: Rod Vagg <rod@vagg.org>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
targos pushed a commit that referenced this issue Aug 19, 2018
Refs: #15985

PR-URL: #22330
Reviewed-By: Rod Vagg <rod@vagg.org>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
@addaleax
Copy link
Member

addaleax commented Sep 2, 2018

I have a fix in #22655.

addaleax added a commit to addaleax/node that referenced this issue Sep 2, 2018
Remove an unnecessary timeout.

Fixes: nodejs#15985
targos pushed a commit that referenced this issue Sep 3, 2018
Refs: #15985

PR-URL: #22330
Reviewed-By: Rod Vagg <rod@vagg.org>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
targos pushed a commit that referenced this issue Sep 5, 2018
Remove an unnecessary timeout.

Fixes: #15985

PR-URL: #22655
Reviewed-By: Denys Otrishko <shishugi@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: George Adams <george.adams@uk.ibm.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
targos pushed a commit that referenced this issue Sep 6, 2018
Remove an unnecessary timeout.

Fixes: #15985

PR-URL: #22655
Reviewed-By: Denys Otrishko <shishugi@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: George Adams <george.adams@uk.ibm.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
async_hooks Issues and PRs related to the async hooks subsystem. flaky-test Issues and PRs related to the tests with unstable failures on the CI. macos Issues and PRs related to the macOS platform / OSX. test Issues and PRs related to the tests.
Projects
None yet
Development

No branches or pull requests