Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

performance improvement of static generation #25035

Merged
merged 7 commits into from
May 17, 2021

Conversation

sokra
Copy link
Member

@sokra sokra commented May 12, 2021

move all access to built pages into worker pool

to allow parallelizing and avoid loading the bundles in the main thread

This improves performance of the static check step a bit and helps reducing memory load in main thread

enable splitChunks for server build in webpack 5

This improves performance for static generation by loading less code due to reduced duplication

to allow parallelizing and avoid loading the bundles in the main thread
@sokra sokra force-pushed the perf/static-check-avoid-main branch from 346d41e to c62678b Compare May 12, 2021 09:54
@ijjk

This comment has been minimized.

sokra added 2 commits May 12, 2021 12:20
this allows to deduplicate code in the server logic
@sokra sokra changed the title move all access to built pages into worker pool performance improvement of static generation May 12, 2021
@ijjk

This comment has been minimized.

@ijjk

This comment has been minimized.

@sokra sokra marked this pull request as ready for review May 12, 2021 10:52
@ijjk

This comment has been minimized.

@ijjk

This comment has been minimized.

timneutkens
timneutkens previously approved these changes May 12, 2021
@ijjk

This comment has been minimized.

@ijjk

This comment has been minimized.

@ijjk

This comment has been minimized.

@ijjk
Copy link
Member

ijjk commented May 17, 2021

Failing test suites

Commit: 4b6b9cd

test/integration/app-document/test/index.test.js

  • Document and App > Client side > should keep state between page navigations
Expand output

● Document and App › Client side › should keep state between page navigations

thrown: "Exceeded timeout of 300000 ms for a test.
Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

  71 |     })
  72 |
> 73 |     it('should keep state between page navigations', async () => {
     |     ^
  74 |       const browser = await webdriver(context.appPort, '/')
  75 |
  76 |       const randomNumber = await browser.elementByCss('#random-number').text()

  at integration/app-document/test/client.js:73:5
  at _default (integration/app-document/test/client.js:8:3)
  at integration/app-document/test/index.test.js:52:3
  at Object.<anonymous> (integration/app-document/test/index.test.js:26:1)

test/integration/scroll-back-restoration/test/index.test.js

  • Scroll Restoration Support > dev mode > should restore the scroll position on navigating back
Expand output

● Scroll Restoration Support › dev mode › should restore the scroll position on navigating back

TIMED OUT: /hi from another/

<head><meta charset="utf-8"><meta name="viewport" content="width=device-width"><meta name="next-head-count" content="2"><noscript data-n-css=""></noscript><link rel="preload" href="/_next/static/chunks/webpack.js?ts=1621235467147" as="script"><link rel="preload" href="/_next/static/chunks/main.js?ts=1621235467147" as="script"><link rel="preload" href="/_next/static/chunks/pages/_app.js?ts=1621235467147" as="script"><link rel="preload" href="/_next/static/chunks/pages/index.js?ts=1621235467147" as="script"><noscript id="__next_css__DO_NOT_USE__"></noscript></head><body><div id="__next"><div style="width:10000px;height:10000px;background:orange"></div><a id="to-another" style="margin-left:5000px;width:95000px;display:block" href="/another">to another</a><div id="end-el">the end</div></div><script src="/_next/static/chunks/react-refresh.js?ts=1621235467147"></script><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"loaded":true},"__N_SSP":true},"page":"/","query":{},"buildId":"development","isFallback":false,"gssp":true}</script><script nomodule="" src="/_next/static/chunks/polyfills.js?ts=1621235467147"></script><script src="/_next/static/chunks/webpack.js?ts=1621235467147"></script><script src="/_next/static/chunks/main.js?ts=1621235467147"></script><script src="/_next/static/chunks/pages/_app.js?ts=1621235467147"></script><script src="/_next/static/chunks/pages/index.js?ts=1621235467147"></script><div id="__next-build-watcher" style="position: fixed; bottom: 10px; right: 20px; width: 0px; height: 0px; z-index: 99999;"></div><script src="/_next/static/development/_buildManifest.js?ts=1621235467147"></script><script src="/_next/static/development/_ssgManifest.js?ts=1621235467147"></script><next-route-announcer><p aria-live="assertive" id="__next-route-announcer__" role="alert" style="border: 0px; clip: rect(0px, 0px, 0px, 0px); height: 1px; margin: -1px; overflow: hidden; padding: 0px; position: absolute; width: 1px; white-space: nowrap; overflow-wrap: normal;"></p></next-route-announcer><script src="/_next/static/chunks/pages/another.js"></script></body>

  395 |
  396 |   if (hardError) {
> 397 |     throw new Error('TIMED OUT: ' + regex + '\n\n' + content)
      |           ^
  398 |   }
  399 |   return false
  400 | }

  at check (lib/next-test-utils.js:397:11)
  at Object.<anonymous> (integration/scroll-back-restoration/test/index.test.js:40:5)

test/integration/scroll-forward-restoration/test/index.test.js

  • Scroll Restoration Support > dev mode > should restore the scroll position on navigating forward
Expand output

● Scroll Restoration Support › dev mode › should restore the scroll position on navigating forward

TIMED OUT: /the end/

<head><meta charset="utf-8"><meta name="viewport" content="width=device-width"><meta name="next-head-count" content="2"><noscript data-n-css=""></noscript><link rel="preload" href="/_next/static/chunks/webpack.js?ts=1621235475193" as="script"><link rel="preload" href="/_next/static/chunks/main.js?ts=1621235475193" as="script"><link rel="preload" href="/_next/static/chunks/pages/_app.js?ts=1621235475193" as="script"><link rel="preload" href="/_next/static/chunks/pages/another.js?ts=1621235475193" as="script"><noscript id="__next_css__DO_NOT_USE__"></noscript></head><body><div id="__next"><p id="another">hi from another</p><a id="to-index" href="/">to index</a></div><script src="/_next/static/chunks/react-refresh.js?ts=1621235475193"></script><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{}},"page":"/another","query":{},"buildId":"development","nextExport":true,"autoExport":true,"isFallback":false}</script><script nomodule="" src="/_next/static/chunks/polyfills.js?ts=1621235475193"></script><script src="/_next/static/chunks/webpack.js?ts=1621235475193"></script><script src="/_next/static/chunks/main.js?ts=1621235475193"></script><script src="/_next/static/chunks/pages/_app.js?ts=1621235475193"></script><script src="/_next/static/chunks/pages/another.js?ts=1621235475193"></script><div id="__next-build-watcher" style="position: fixed; bottom: 10px; right: 20px; width: 0px; height: 0px; z-index: 99999;"></div><script src="/_next/static/development/_buildManifest.js?ts=1621235475193"></script><script src="/_next/static/development/_ssgManifest.js?ts=1621235475193"></script><next-route-announcer><p aria-live="assertive" id="__next-route-announcer__" role="alert" style="border: 0px; clip: rect(0px, 0px, 0px, 0px); height: 1px; margin: -1px; overflow: hidden; padding: 0px; position: absolute; width: 1px; white-space: nowrap; overflow-wrap: normal;"></p></next-route-announcer><script src="/_next/static/chunks/pages/index.js"></script></body>

  395 |
  396 |   if (hardError) {
> 397 |     throw new Error('TIMED OUT: ' + regex + '\n\n' + content)
      |           ^
  398 |   }
  399 |   return false
  400 | }

  at check (lib/next-test-utils.js:397:11)
  at Object.<anonymous> (integration/scroll-forward-restoration/test/index.test.js:25:5)

test/integration/trailing-slashes-href-resolving/test/index.test.js

  • href resolving trailing-slash > dev mode > should route to /blog/first-post/ correctly
  • href resolving trailing-slash > dev mode > should route to /catch-all/hello/world/ correctly
  • href resolving trailing-slash > dev mode > should route to /top-level-slug/ correctly
Expand output

● href resolving trailing-slash › dev mode › should route to /blog/first-post/ correctly

thrown: "Exceeded timeout of 120000 ms for a test.
Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

  27 |   })
  28 |
> 29 |   it('should route to /blog/first-post/ correctly', async () => {
     |   ^
  30 |     const browser = await webdriver(appPort, '/')
  31 |     await browser.elementByCss('#to-blog-post').click()
  32 |

  at runTests (integration/trailing-slashes-href-resolving/test/index.test.js:29:3)
  at integration/trailing-slashes-href-resolving/test/index.test.js:105:5
  at integration/trailing-slashes-href-resolving/test/index.test.js:98:3
  at Object.<anonymous> (integration/trailing-slashes-href-resolving/test/index.test.js:97:1)

● href resolving trailing-slash › dev mode › should route to /catch-all/hello/world/ correctly

thrown: "Exceeded timeout of 120000 ms for a test.
Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

  37 |   })
  38 |
> 39 |   it('should route to /catch-all/hello/world/ correctly', async () => {
     |   ^
  40 |     const browser = await webdriver(appPort, '/')
  41 |     await browser.elementByCss('#to-catch-all-item').click()
  42 |

  at runTests (integration/trailing-slashes-href-resolving/test/index.test.js:39:3)
  at integration/trailing-slashes-href-resolving/test/index.test.js:105:5
  at integration/trailing-slashes-href-resolving/test/index.test.js:98:3
  at Object.<anonymous> (integration/trailing-slashes-href-resolving/test/index.test.js:97:1)

● href resolving trailing-slash › dev mode › should route to /top-level-slug/ correctly

thrown: "Exceeded timeout of 120000 ms for a test.
Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

  65 |   })
  66 |
> 67 |   it('should route to /top-level-slug/ correctly', async () => {
     |   ^
  68 |     const browser = await webdriver(appPort, '/')
  69 |     await browser.elementByCss('#to-slug').click()
  70 |

  at runTests (integration/trailing-slashes-href-resolving/test/index.test.js:67:3)
  at integration/trailing-slashes-href-resolving/test/index.test.js:105:5
  at integration/trailing-slashes-href-resolving/test/index.test.js:98:3
  at Object.<anonymous> (integration/trailing-slashes-href-resolving/test/index.test.js:97:1)

@ijjk
Copy link
Member

ijjk commented May 17, 2021

Stats from current PR

Default Server Mode (Decrease detected ✓)
General Overall increase ⚠️
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
buildDuration 16.4s 17.3s ⚠️ +893ms
buildDurationCached 5.1s 4.8s -334ms
nodeModulesSize 46.6 MB 46.7 MB ⚠️ +1.39 kB
Page Load Tests Overall decrease ⚠️
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
/ failed reqs 0 0
/ total time (seconds) 2.827 2.887 ⚠️ +0.06
/ avg req/sec 884.39 866.01 ⚠️ -18.38
/error-in-render failed reqs 0 0
/error-in-render total time (seconds) 1.69 1.724 ⚠️ +0.03
/error-in-render avg req/sec 1479.28 1450.5 ⚠️ -28.78
Client Bundles (main, webpack, commons)
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
framework-HASH.js gzip 39.3 kB 39.3 kB
main-HASH.js gzip 19.4 kB 19.4 kB
webpack-HASH.js gzip 994 B 994 B
Overall change 59.7 kB 59.7 kB
Legacy Client Bundles (polyfills)
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
polyfills-HASH.js gzip 31.1 kB 31.1 kB
Overall change 31.1 kB 31.1 kB
Client Pages
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
_app-HASH.js gzip 1.02 kB 1.02 kB
_error-HASH.js gzip 3.06 kB 3.06 kB
amp-HASH.js gzip 526 B 526 B
css-HASH.js gzip 334 B 334 B
hooks-HASH.js gzip 890 B 890 B
index-HASH.js gzip 262 B 262 B
link-HASH.js gzip 1.65 kB 1.65 kB
routerDirect..HASH.js gzip 331 B 331 B
withRouter-HASH.js gzip 329 B 329 B
bb14e60e810b..30f.css gzip 125 B 125 B
Overall change 8.54 kB 8.54 kB
Client Build Manifests
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
_buildManifest.js gzip 390 B 390 B
Overall change 390 B 390 B
Rendered Page Sizes
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
index.html gzip 560 B 560 B
link.html gzip 569 B 569 B
withRouter.html gzip 557 B 557 B
Overall change 1.69 kB 1.69 kB

Serverless Mode (Decrease detected ✓)
General Overall increase ⚠️
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
buildDuration 20.8s 19s -1.8s
buildDurationCached 7.2s 5.9s -1.3s
nodeModulesSize 46.6 MB 46.7 MB ⚠️ +1.39 kB
Client Bundles (main, webpack, commons)
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
framework-HASH.js gzip 39.3 kB 39.3 kB
main-HASH.js gzip 19.4 kB 19.4 kB
webpack-HASH.js gzip 994 B 994 B
Overall change 59.7 kB 59.7 kB
Legacy Client Bundles (polyfills)
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
polyfills-HASH.js gzip 31.1 kB 31.1 kB
Overall change 31.1 kB 31.1 kB
Client Pages
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
_app-HASH.js gzip 1.02 kB 1.02 kB
_error-HASH.js gzip 3.06 kB 3.06 kB
amp-HASH.js gzip 526 B 526 B
css-HASH.js gzip 334 B 334 B
hooks-HASH.js gzip 890 B 890 B
index-HASH.js gzip 262 B 262 B
link-HASH.js gzip 1.65 kB 1.65 kB
routerDirect..HASH.js gzip 331 B 331 B
withRouter-HASH.js gzip 329 B 329 B
bb14e60e810b..30f.css gzip 125 B 125 B
Overall change 8.54 kB 8.54 kB
Client Build Manifests
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
_buildManifest.js gzip 390 B 390 B
Overall change 390 B 390 B
Serverless bundles Overall decrease ✓
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
_error.js 1.34 MB 16.9 kB -1.33 MB
404.html 2.42 kB 2.42 kB
500.html 2.41 kB 2.41 kB
amp.amp.html 10.8 kB 10.8 kB
amp.html 1.61 kB 1.61 kB
css.html 1.79 kB 1.79 kB
hooks.html 1.67 kB 1.67 kB
index.js 1.34 MB 17.2 kB -1.33 MB
link.js 1.4 MB 17.4 kB -1.39 MB
routerDirect.js 1.4 MB 17.4 kB -1.38 MB
withRouter.js 1.4 MB 17.4 kB -1.38 MB
Overall change 6.91 MB 107 kB -6.8 MB

Webpack 4 Mode (Increase detected ⚠️)
General Overall increase ⚠️
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
buildDuration 13.6s 14.3s ⚠️ +677ms
buildDurationCached 5.8s 5.8s ⚠️ +8ms
nodeModulesSize 46.6 MB 46.7 MB ⚠️ +1.39 kB
Page Load Tests Overall increase ✓
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
/ failed reqs 0 0
/ total time (seconds) 2.874 2.831 -0.04
/ avg req/sec 869.87 883.24 +13.37
/error-in-render failed reqs 0 0
/error-in-render total time (seconds) 1.688 1.584 -0.1
/error-in-render avg req/sec 1481.43 1577.8 +96.37
Client Bundles (main, webpack, commons)
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
677f882d2ed8..HASH.js gzip 13.3 kB 13.3 kB
framework.HASH.js gzip 39 kB 39 kB
main-HASH.js gzip 7.26 kB 7.26 kB
webpack-HASH.js gzip 751 B 751 B
Overall change 60.3 kB 60.3 kB
Legacy Client Bundles (polyfills)
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
polyfills-HASH.js gzip 31.3 kB 31.3 kB
Overall change 31.3 kB 31.3 kB
Client Pages
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
_app-HASH.js gzip 1.28 kB 1.28 kB
_error-HASH.js gzip 3.74 kB 3.74 kB
amp-HASH.js gzip 536 B 536 B
css-HASH.js gzip 339 B 339 B
hooks-HASH.js gzip 887 B 887 B
index-HASH.js gzip 227 B 227 B
link-HASH.js gzip 1.64 kB 1.64 kB
routerDirect..HASH.js gzip 303 B 303 B
withRouter-HASH.js gzip 302 B 302 B
e025d2764813..52f.css gzip 125 B 125 B
Overall change 9.38 kB 9.38 kB
Client Build Manifests
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
_buildManifest.js gzip 420 B 420 B
Overall change 420 B 420 B
Rendered Page Sizes
vercel/next.js canary vercel/next.js perf/static-check-avoid-main Change
index.html gzip 614 B 614 B
link.html gzip 620 B 620 B
withRouter.html gzip 606 B 606 B
Overall change 1.84 kB 1.84 kB
Commit: e706ae6

Copy link
Member

@timneutkens timneutkens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to land now 👍

@kodiakhq kodiakhq bot merged commit 3bf4ae3 into canary May 17, 2021
This was referenced May 19, 2021
flybayer pushed a commit to blitz-js/next.js that referenced this pull request Jun 1, 2021
### move all access to built pages into worker pool

to allow parallelizing and avoid loading the bundles in the main thread

This improves performance of the static check step a bit and helps reducing memory load in main thread

### enable splitChunks for server build in webpack 5

This improves performance for static generation by loading less code due to reduced duplication
@vercel vercel locked as resolved and limited conversation to collaborators Jan 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants