12.0.1 -> 12.0.4 collecting page data heap out of memory

[mercteil]

What version of Next.js are you using?

12.0.4

What version of Node.js are you using?

16.13.0

What browser are you using?

CMD

What operating system are you using?

macOs/Linus

How are you deploying your application?

next build

Describe the Bug

After upgrading from 12.0.1 to a higher version my Github actions as well as my Heroku started to fail when when building the application. Localhost does work and builds the application as expected.

But all CI/CD processes (Github actions, Heroku deployment) fail with the following issue:

info  - Checking validity of types...
info  - Creating an optimized production build...
info  - Disabled SWC as replacement for Babel because of custom Babel configuration ".babelrc.js" https://nextjs.org/docs/messages/swc-disabled
info  - Using external babel configuration from /home/runner/work/mercteil-shop-ui/mercteil-shop-ui/.babelrc.js
info  - Compiled successfully
info  - Collecting page data...

<--- Last few GCs --->

[1783:0x5300410]    47259 ms: Mark-sweep (reduce) 2047.0 (2083.8) -> 2046.6 (2084.5) MB, 3087.2 / 0.0 ms  (average mu = 0.213, current mu = 0.000) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb200e0 node::Abort() [/opt/hostedtoolcache/node/16.4.2/x64/bin/node]
 2: 0xa3c157 node::FatalError(char const*, char const*) [/opt/hostedtoolcache/node/16.4.2/x64/bin/node]
 3: 0xd083ae v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/opt/hostedtoolcache/node/16.4.2/x64/bin/node]
 4: 0xd08727 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/opt/hostedtoolcache/node/16.4.2/x64/bin/node]
 5: 0xee9785  [/opt/hostedtoolcache/node/16.4.2/x64/bin/node]
 6: 0xef84b5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/opt/hostedtoolcache/node/16.4.2/x64/bin/node]
 7: 0xefb5ac v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/hostedtoolcache/node/16.4.2/x64/bin/node]
 8: 0xec000c v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/opt/hostedtoolcache/node/16.4.2/x64/bin/node]
 9: 0x123695b v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/opt/hostedtoolcache/node/16.4.2/x64/bin/node]
10: 0x1640819  [/opt/hostedtoolcache/node/16.4.2/x64/bin/node]
warn  - Restarted collecting page data for /account/[[...params]] because it took more than 1500 seconds
warn  - See more info here https://nextjs.org/docs/messages/static-page-generation-timeout
warn  - Restarted collecting page data for /confirm-email/[token] because it took more than 1500 seconds

...

.babelrc

module.exports = {
  plugins: [
    ['lodash'],
    ['styled-components', { ssr: true, displayName: true }],
  ],
  env: {
    production: {
      presets: ['next/babel'],
      plugins: [['styled-components', { ssr: true, displayName: false }]],
    },
    
    local: { presets: ['next/babel', '@babel/preset-react'] },

    development: { presets: ['next/babel', '@babel/preset-react'] },

    test: {
      presets: ['@babel/preset-env', '@babel/preset-react'],
      plugins: [
        '@babel/plugin-transform-runtime',
        ['@babel/plugin-proposal-class-properties', { loose: true }],
        ['@babel/plugin-proposal-private-methods', { loose: true }],
        ['@babel/plugin-proposal-private-property-in-object', { loose: true }],
      ],
    },
  },
};

Downgrading to 12.0.1 resolves the issue.

Increasing the timeout https://nextjs.org/docs/messages/page-data-collection-timeout did not help.

Described here is a very similar issue, if not the same:
https://lzomedia.com/blog/getting-data-collection-timeout-when-trying-to-build-in-local-machine/

Expected Behavior

next build should just build the application without errors as it does locally or with 12.0.1.

Localhost works perfectly.

To Reproduce

I updated 12.0.1 to 12.0.4 and pushed the update to github & heroku. Both pipelines failed with this error.

[haggen]

Same upgrade path (12.0.1 -> 12.0.4), I believe same error, but on Node 14.18.0. See my logs: https://gist.github.com/haggen/37da26782a19d7875563c44921d35e08

[balazsorban44]

I also came across this in a fresh create-next-app (next@12.0.4) project, without a custom Babel configuration (so using SWC).

<--- Last few GCs --->

[453832:0x59c1090]   191666 ms: Scavenge (reduce) 2045.9 (2082.1) -> 2045.0 (2082.1) MB, 2.8 / 0.0 ms  (average mu = 0.182, current mu = 0.105) allocation failure 
[453832:0x59c1090]   193959 ms: Mark-sweep (reduce) 2046.4 (2082.6) -> 2045.3 (2082.6) MB, 2203.1 / 0.0 ms  (+ 94.5 ms in 20 steps since start of marking, biggest step 6.5 ms, walltime since start of marking 2310 ms) (average mu = 0.139, current mu = 0.09

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb02ec0 node::Abort() [/home/balazs/.nvm/versions/node/v16.13.0/bin/node]
 2: 0xa181fb node::FatalError(char const*, char const*) [/home/balazs/.nvm/versions/node/v16.13.0/bin/node]
 3: 0xced88e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/balazs/.nvm/versions/node/v16.13.0/bin/node]
 4: 0xcedc07 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/balazs/.nvm/versions/node/v16.13.0/bin/node]
 5: 0xea5ea5  [/home/balazs/.nvm/versions/node/v16.13.0/bin/node]
 6: 0xea6986  [/home/balazs/.nvm/versions/node/v16.13.0/bin/node]
 7: 0xeb48be  [/home/balazs/.nvm/versions/node/v16.13.0/bin/node]
 8: 0xeb5300 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/balazs/.nvm/versions/node/v16.13.0/bin/node]
 9: 0xeb827e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/balazs/.nvm/versions/node/v16.13.0/bin/node]
10: 0xe796aa v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/home/balazs/.nvm/versions/node/v16.13.0/bin/node]
11: 0x11f2e86 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/home/balazs/.nvm/versions/node/v16.13.0/bin/node]
12: 0x15e7879  [/home/balazs/.nvm/versions/node/v16.13.0/bin/node]
Aborted (core dumped)
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command

[balazsorban44]

@mercteil could you try upgrading to 12.0.7 or 12.0.8-canary.3 and see if this persists?

[anfinil]

@mercteil could you try upgrading to 12.0.7 or 12.0.8-canary.3 and see if this persists?

yes, it still persist

[sandcastle]

We are also seeing this issue with 12.0.7

[mercteil]

@mercteil could you try upgrading to 12.0.7 or 12.0.8-canary.3 and see if this persists?

Nothing changed with 12.0.8-canary3. Same error as before.

[jcolakk]

This started happening for us as well when we upgraded from 12.0.1 to 12.0.2.

[sandcastle]

@jcolakk

There are definitely issues caused by cyclical references in v12 (12.0.1 through 12.0.7 - I tried them all).

Builds were constantly failing and after a lot of screwing around, I was able to identify that it was being caused by references, especially where you are re-exporting things in index.ts files or referencing objects multiple different ways. After cleaning up the majority of our imports (using fewer index.ts files and importing things more explicitly), I was able to get our codebase to build on v12, but it still crashes when running in dev mode - I'm expecting that after more reference cleanups this will go away, but haven't had time to do any more refactoring yet - our codebase is massive.

Hope that helps

[jcolakk]

@sandcastle thanks for the writeup. But not sure I really understand what exactly the issue is and why everything works fine with 12.0.1 and suddenly breaks on 12.0.2 🤔

using fewer index.ts files and importing things more explicitly

can you elaborate a bit more on this?

[jcolakk]

Is there any news/progress on this?
I don't see anything wrong with our app, the build works but dev is not working. Only way I can make it work is if I remove everything from a single page and thats really not a solution.

[sandcastle]

We upgraded to 12.0.8 and changed our next.config.js to the following and now everything is working...

{
  experimental: {
      esmExternals: false
  }
}

[sandcastle]

We upgraded to 12.0.8 and changed our next.config.js to the following and now everything is working...

{
  experimental: {
      esmExternals: false
  }
}

We found the fix mentioned in this issue - #30330 (comment)

It looks like they may have fixed it in 12.0.9 - f7048b5

[jcolakk]

We upgraded to 12.0.8 and changed our next.config.js to the following and now everything is working...

{
  experimental: {
      esmExternals: false
  }
}

Tried this, still the same error 😞

Also tried upgrading to latest canary and same thing, still not working

[julioflima]

Suffering here with the same error when I try to use something of "mui/icons-material", 12.0.9-canary.4.

[balazsorban44]

We have tried to reproduce this for hours given the information in this issue thread, but we need a more reliable minimal reproduction in order to investigate further as we could not find an issue.

Preferably, try to cut out anything that is unnecessary, so the code is concentrated on the potential problem.

[mercteil]

The issue with the reproduction is that I can build locally, but not in GitHub nor in Heroku. I do not even know where to start to reproduce :)

[yuanzong]

The issue with the reproduction is that I can build locally, but not in GitHub nor in Heroku. I do not even know where to start to reproduce :)

I had an issue similar to yours, and I was seeing 50/50 chance of build failure on CI. I think the problem is newer version uses slightly more memory to build, and it's gone after I gave the build pipeline extra memory (from 4g to 6g)

[arunesh90]

We upgraded to 12.0.8 and changed our next.config.js to the following and now everything is working...

{
  experimental: {
      esmExternals: false
  }
}

Can confirm that trying this fixed the issue for my project

[jcolakk]

Can confirm that trying this fixed the issue for my project

Make sure you load test it first, I thought it fixed it on our end as well and we load tested it and it was still there..

[anfinil]

I can confirm that disabling esmExternals fixes the problem. But it would be great to find a way to fix without disabling it.

[arunesh90]

Make sure you load test it first, I thought it fixed it on our end as well and we load tested it and it was still there..

@jcolakk I have load tested it on my end by sending 5k successful page requests in a minute to a page that's server-side rendered, and so far 0 issues to report.

[balazsorban44]

This issue has been automatically closed because it received no activity for a month and had no reproduction to investigate. If you think this was closed by accident, please leave a comment. If you are running into a similar issue, please open a new issue with a reproduction. Thank you.

[mercteil]

Switched entirely to swc-project, it did not resolve the issue.

I can confirm that the following next config settings resolved the issue for me:

  staticPageGenerationTimeout: 240,

  experimental: {
    esmExternals: false,
  },

[github-actions]

This closed issue has been automatically locked because it had no new activity for a month. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.