Intermittent e2e test failures

[danielbachhuber]

The e2e test suite currently fails intermittently. Sometimes the test failure looks like this:

I have no idea on the cause of this, but it's getting in the way of other unrelated development.

[noisysocks]

I spotted a few mistakes to do with async functions in our E2E tests which could be contributing to this problem. #6971 attempts to fix some of them.

Update: #6995 should fix the specific error you encountered here.

[danielbachhuber]

Comment from @youknowriad to address (#6995 (comment)):

I'm pretty sure that's not the only place in our tests where we open the inserter, we should probably be consistent.
Do you think we should factorize that in a insertBlock( 'core' ) utility?
And also, can we add a comment explaining why this is necessary? Is there an API request or something?

[aduth]

There's a new suspect from the change-detection.test.js suite:

✕ Should prompt if changes made while save is in-flight (1747ms)

Have seen separately on both #7419 and #7220 .

As author of the offending test, I shall plan to investigate.

[gziolo]

Moving to 3.2. This is an issue with tests rather than with actual code.

[youknowriad]

I'm also starting to see this failure from time to time.

[aduth]

FYI #7493 will resolve at least two of the intermittent failing tests I've seen.

[danielbachhuber]

I just saw this:

On #7322, which was just rebased against master.

[aduth]

Based on the error message, I'm inclined to think it should be resolved with #7544 . Let's continue reopening if it persists, though.

[aduth]

A change was snuck into #6959 (689d79d) to run end-to-end tests in serial order (not parallel) which could help make the tests more deterministic and eliminate any chance of tests affecting one another (since, for example, some tests activate plugins which could have an impact site-wide if tests are run in parallel).

http://jestjs.io/docs/en/cli.html#runinband

[aduth]

I've been seeing this one a lot lately: #6956 (comment)

[aduth]

Starting to see a new one.

FAIL test/e2e/specs/preview.test.js
  ● Preview › Should open a preview window for a new post
    Protocol error (Runtime.callFunctionOn): Cannot find context with specified id undefined
      
      at Promise (node_modules/puppeteer/lib/Connection.js:198:56)
      at CDPSession.send (node_modules/puppeteer/lib/Connection.js:197:12)
      at ExecutionContext.evaluateHandle (node_modules/puppeteer/lib/ExecutionContext.js:71:75)
      at ExecutionContext.evaluate (node_modules/puppeteer/lib/ExecutionContext.js:46:31)
      at Frame.evaluate (node_modules/puppeteer/lib/FrameManager.js:299:20)

Example: #7841 https://travis-ci.org/WordPress/gutenberg/jobs/401939944

[talldan]

I'm also seeing block-icons.test.js fail locally consistently.

Seems to be due to the wrong block being focused during the test (the inserted blocks seem to have a nested paragraph block and the paragraph block end up being focused):
https://github.com/WordPress/gutenberg/blob/master/test/e2e/specs/block-icons.test.js#L119-L122

Had a quick look but couldn't figure out a workaround.

[gziolo]

@talldan, can you check PR from @jorgefilipecosta #7896 and verify if that resolves this issue? It has been opened for some time but we didn't come up to the consensus if it resolves the issue or hides it deeper :)

[aduth]

#8707 (comment)

[gziolo]

@aduth - there is an open issue for it logged for it. Actually, there are 2 open issues with e2e test failures which happen from time to time:

• test/e2e/specs/writing-flow.test.js (tests: e2e test for the writing flow started to fail #8712)
• test/e2e/specs/undo.js (Undo / Redo inconsistencies when using mouse or keyboard #5685 (comment)) - it's a part of a bigger issue which isn't easy to reproduce

[aduth]

Ok, #8712 appears to be the same as in #8707 (comment) so let's close in favor of that one.