Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(integration): emit console error when test fails #35507

Merged
merged 3 commits into from
Mar 28, 2022

Conversation

kwonoj
Copy link
Contributor

@kwonoj kwonoj commented Mar 21, 2022

Minor improvements to the integration test runner to emit process's stdout when its execution failed.

We have console.log for the output, but it doesn't emit actually since jest suppresses console output unlesss assertion fails. If we throws in before* hook, since there's no actual assertion fails jest won't release captured console logs.

PR adds one straightforward workaround (https://stackoverflow.com/questions/48695717/console-log-statements-output-nothing-at-all-in-jest) to emit if process exits with non-zero errorcode. To avoid any existing behavior, this is not enabled for the cases when process successfully exits.

Bug

  • Related issues linked using fixes #number
  • Integration tests added
  • Errors have helpful link attached, see contributing.md

Feature

  • Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
  • Related issues linked using fixes #number
  • Integration tests added
  • Documentation added
  • Telemetry added. In case of a feature if it's used or not.
  • Errors have helpful link attached, see contributing.md

Documentation / Examples

  • Make sure the linting passes by running yarn lint

@ijjk

This comment has been minimized.

@ijjk

This comment has been minimized.

@ijjk
Copy link
Member

ijjk commented Mar 22, 2022

Failing test suites

Commit: 2c452d6

yarn testheadless test/production/emit-decorator-metadata/index.test.ts

  • emitDecoratorMetadata SWC option > should compile with emitDecoratorMetadata enabled
  • emitDecoratorMetadata SWC option > should compile with emitDecoratorMetadata enabled for API
Expand output

● emitDecoratorMetadata SWC option › should compile with emitDecoratorMetadata enabled

thrown: "Exceeded timeout of 90000 ms for a hook.
Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

   9 |   let next: NextInstance
  10 |
> 11 |   beforeAll(async () => {
     |   ^
  12 |     next = await createNext({
  13 |       files: {
  14 |         'jsconfig.json': new FileRef(join(__dirname, 'app/jsconfig.json')),

  at production/emit-decorator-metadata/index.test.ts:11:3
  at Object.<anonymous> (production/emit-decorator-metadata/index.test.ts:8:1)

● emitDecoratorMetadata SWC option › should compile with emitDecoratorMetadata enabled for API

thrown: "Exceeded timeout of 90000 ms for a hook.
Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

   9 |   let next: NextInstance
  10 |
> 11 |   beforeAll(async () => {
     |   ^
  12 |     next = await createNext({
  13 |       files: {
  14 |         'jsconfig.json': new FileRef(join(__dirname, 'app/jsconfig.json')),

  at production/emit-decorator-metadata/index.test.ts:11:3
  at Object.<anonymous> (production/emit-decorator-metadata/index.test.ts:8:1)

● Test suite failed to run

TypeError: Cannot read properties of undefined (reading 'destroy')

  24 |   })
  25 |
> 26 |   afterAll(() => next.destroy())
     |                       ^
  27 |
  28 |   it('should compile with emitDecoratorMetadata enabled', async () => {
  29 |     let browser: BrowserInterface

  at production/emit-decorator-metadata/index.test.ts:26:23

● Test suite failed to run

next instance not destroyed before exiting, make sure to call .destroy() after the tests after finished

  91 |     if (nextInstance) {
  92 |       await nextInstance.destroy()
> 93 |       throw new Error(
     |             ^
  94 |         `next instance not destroyed before exiting, make sure to call .destroy() after the tests after finished`
  95 |       )
  96 |     }

  at lib/e2e-utils.ts:93:13

Read more about building and testing Next.js in contributing.md.

@ijjk

This comment has been minimized.

kodiakhq bot pushed a commit that referenced this pull request Mar 28, 2022
This updates our default test times as discussed to account for longer `yarn install` times for the isolated tests as GitHub actions network can be slow. 

x-ref: #35507 (comment)
x-ref: [slack thread](https://vercel.slack.com/archives/CGU8HUTUH/p1647919982840099)
@ijjk
Copy link
Member

ijjk commented Mar 28, 2022

Stats from current PR

Default Build (Increase detected ⚠️)
General
vercel/next.js canary kwonoj/next.js test-output-failed-std Change
buildDuration 14.8s 15.1s ⚠️ +283ms
buildDurationCached 6s 5.5s -462ms
nodeModulesSize 467 MB 467 MB
Page Load Tests Overall increase ✓
vercel/next.js canary kwonoj/next.js test-output-failed-std Change
/ failed reqs 0 0
/ total time (seconds) 2.969 3.017 ⚠️ +0.05
/ avg req/sec 842.14 828.64 ⚠️ -13.5
/error-in-render failed reqs 0 0
/error-in-render total time (seconds) 1.332 1.258 -0.07
/error-in-render avg req/sec 1877.13 1986.83 +109.7
Client Bundles (main, webpack)
vercel/next.js canary kwonoj/next.js test-output-failed-std Change
925.HASH.js gzip 179 B 179 B
framework-HASH.js gzip 42 kB 42 kB
main-HASH.js gzip 28.4 kB 28.4 kB
webpack-HASH.js gzip 1.44 kB 1.44 kB
Overall change 72.1 kB 72.1 kB
Legacy Client Bundles (polyfills)
vercel/next.js canary kwonoj/next.js test-output-failed-std Change
polyfills-HASH.js gzip 31 kB 31 kB
Overall change 31 kB 31 kB
Client Pages
vercel/next.js canary kwonoj/next.js test-output-failed-std Change
_app-HASH.js gzip 1.36 kB 1.36 kB
_error-HASH.js gzip 192 B 192 B
amp-HASH.js gzip 309 B 309 B
css-HASH.js gzip 327 B 327 B
dynamic-HASH.js gzip 2.57 kB 2.57 kB
head-HASH.js gzip 351 B 351 B
hooks-HASH.js gzip 920 B 920 B
image-HASH.js gzip 5.48 kB 5.48 kB
index-HASH.js gzip 263 B 263 B
link-HASH.js gzip 2.26 kB 2.26 kB
routerDirect..HASH.js gzip 320 B 320 B
script-HASH.js gzip 387 B 387 B
withRouter-HASH.js gzip 319 B 319 B
85e02e95b279..7e3.css gzip 107 B 107 B
Overall change 15.2 kB 15.2 kB
Client Build Manifests
vercel/next.js canary kwonoj/next.js test-output-failed-std Change
_buildManifest.js gzip 460 B 460 B
Overall change 460 B 460 B
Rendered Page Sizes
vercel/next.js canary kwonoj/next.js test-output-failed-std Change
index.html gzip 532 B 532 B
link.html gzip 546 B 546 B
withRouter.html gzip 526 B 526 B
Overall change 1.6 kB 1.6 kB

Default Build with SWC (Decrease detected ✓)
General
vercel/next.js canary kwonoj/next.js test-output-failed-std Change
buildDuration 17.2s 17s -200ms
buildDurationCached 5.5s 5.5s -29ms
nodeModulesSize 467 MB 467 MB
Page Load Tests Overall decrease ⚠️
vercel/next.js canary kwonoj/next.js test-output-failed-std Change
/ failed reqs 0 0
/ total time (seconds) 2.866 2.873 ⚠️ +0.01
/ avg req/sec 872.3 870.03 ⚠️ -2.27
/error-in-render failed reqs 0 0
/error-in-render total time (seconds) 1.249 1.265 ⚠️ +0.02
/error-in-render avg req/sec 2001.14 1977.03 ⚠️ -24.11
Client Bundles (main, webpack)
vercel/next.js canary kwonoj/next.js test-output-failed-std Change
925.HASH.js gzip 178 B 178 B
framework-HASH.js gzip 42.3 kB 42.3 kB
main-HASH.js gzip 28.8 kB 28.8 kB
webpack-HASH.js gzip 1.45 kB 1.45 kB
Overall change 72.7 kB 72.7 kB
Legacy Client Bundles (polyfills)
vercel/next.js canary kwonoj/next.js test-output-failed-std Change
polyfills-HASH.js gzip 31 kB 31 kB
Overall change 31 kB 31 kB
Client Pages
vercel/next.js canary kwonoj/next.js test-output-failed-std Change
_app-HASH.js gzip 1.35 kB 1.35 kB
_error-HASH.js gzip 179 B 179 B
amp-HASH.js gzip 313 B 313 B
css-HASH.js gzip 324 B 324 B
dynamic-HASH.js gzip 2.56 kB 2.56 kB
head-HASH.js gzip 351 B 351 B
hooks-HASH.js gzip 921 B 921 B
image-HASH.js gzip 5.59 kB 5.59 kB
index-HASH.js gzip 261 B 261 B
link-HASH.js gzip 2.33 kB 2.33 kB
routerDirect..HASH.js gzip 322 B 322 B
script-HASH.js gzip 388 B 388 B
withRouter-HASH.js gzip 317 B 317 B
85e02e95b279..7e3.css gzip 107 B 107 B
Overall change 15.3 kB 15.3 kB
Client Build Manifests
vercel/next.js canary kwonoj/next.js test-output-failed-std Change
_buildManifest.js gzip 458 B 458 B
Overall change 458 B 458 B
Rendered Page Sizes
vercel/next.js canary kwonoj/next.js test-output-failed-std Change
index.html gzip 530 B 530 B
link.html gzip 544 B 544 B
withRouter.html gzip 525 B 525 B
Overall change 1.6 kB 1.6 kB
Commit: abcff31

@kodiakhq kodiakhq bot merged commit abfbf41 into vercel:canary Mar 28, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 28, 2022
@kwonoj kwonoj deleted the test-output-failed-std branch May 9, 2022 16:53
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants