Converting source in large projects results in JavaScript heap out of memory error

[jstvz]

Summary

Converting source in large projects results in JavaScript heap out of memory error

Steps To Reproduce

1. git clone --depth 1 https://github.com/SalesforceFoundation/NPSP.git npsp; cd npsp
2. sf project convert source

Expected result

$ sf project convert source
Source was successfully converted to Metadata API format and written to the location: /private/tmp/npsp/metadataPackage_1719260773403

Actual result

$ sf project convert source

<--- Last few GCs --->

[88567:0x7fc918008000]    99781 ms: Scavenge 3326.0 (4122.4) -> 3313.1 (4124.2) MB, 3.61 / 0.00 ms  (average mu = 0.281, current mu = 0.226) allocation failure;
[88567:0x7fc918008000]    99807 ms: Scavenge 3329.3 (4125.7) -> 3316.1 (4126.7) MB, 6.89 / 0.00 ms  (average mu = 0.281, current mu = 0.226) allocation failure;
[88567:0x7fc918008000]    99834 ms: Scavenge 3331.7 (4126.7) -> 3318.7 (4127.7) MB, 4.14 / 0.00 ms  (average mu = 0.281, current mu = 0.226) allocation failure;


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0x1042f9625 node::Abort() [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
 2: 0x1042f980d node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
 3: 0x1044a9d1c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
 4: 0x104699cf7 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
 5: 0x10469e2c7 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
 6: 0x10469a615 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
 7: 0x1046980b0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
 8: 0x10468cdc1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
 9: 0x10468d805 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
10: 0x10466f962 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
11: 0x104aed850 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
12: 0x104ebf8b6 Builtins_CEntry_Return1_ArgvOnStack_NoBuiltinExit [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]

System Information

{
  "architecture": "darwin-x64",
  "cliVersion": "@salesforce/cli/2.46.6",
  "nodeVersion": "node-v20.11.0",
  "osVersion": "Darwin 22.6.0",
  "rootPath": "/Users/jestevez/.volta/tools/image/packages/@salesforce/cli/lib/node_modules/@salesforce/cli",
  "shell": "zsh",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.1.2 (core)",
    "@oclif/plugin-commands 4.0.2 (core)",
    "@oclif/plugin-help 6.2.1 (core)",
    "@oclif/plugin-not-found 3.2.3 (core)",
    "@oclif/plugin-plugins 5.2.4 (core)",
    "@oclif/plugin-search 1.1.2 (core)",
    "@oclif/plugin-update 4.3.5 (core)",
    "@oclif/plugin-version 2.2.3 (core)",
    "@oclif/plugin-warn-if-update-available 3.1.5 (core)",
    "@oclif/plugin-which 3.2.2 (core)",
    "@salesforce/cli 2.46.6 (core)",
    "apex 3.1.18 (core)",
    "auth 3.6.21 (core)",
    "data 3.4.6 (core)",
    "deploy-retrieve 3.9.4 (core)",
    "info 3.3.6 (core)",
    "limits 3.3.12 (core)",
    "marketplace 1.2.12 (core)",
    "org 4.2.1 (core)",
    "packaging 2.4.6 (core)",
    "schema 3.3.11 (core)",
    "settings 2.3.1 (core)",
    "sobject 1.4.11 (core)",
    "source 3.4.2 (core)",
    "telemetry 3.4.0 (core)",
    "templates 56.2.11 (core)",
    "trust 3.7.5 (core)",
    "user 3.5.13 (core)"
  ]
}

Additional information

Attempted workaround from forcedotcom/salesforcedx-vscode#5589, but I still got heap errors on apex 3.1.3 (user) and apex 3.2.0 (user). However installing @salesforce/cli@2.36.8 worked:

{
  "architecture": "darwin-x64",
  "cliVersion": "@salesforce/cli/2.36.8",
  "nodeVersion": "node-v20.11.0",
  "osVersion": "Darwin 22.6.0",
  "rootPath": "/Users/jestevez/.volta/tools/image/packages/@salesforce/cli/lib/node_modules/@salesforce/cli",
  "shell": "zsh",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.0.13 (core)",
    "@oclif/plugin-commands 3.2.2 (core)",
    "@oclif/plugin-help 6.0.20 (core)",
    "@oclif/plugin-not-found 3.1.1 (core)",
    "@oclif/plugin-plugins 5.0.5 (core)",
    "@oclif/plugin-search 1.0.20 (core)",
    "@oclif/plugin-update 4.2.3 (core)",
    "@oclif/plugin-version 2.0.16 (core)",
    "@oclif/plugin-warn-if-update-available 3.0.15 (core)",
    "@oclif/plugin-which 3.1.7 (core)",
    "@salesforce/cli 2.36.8 (core)",
    "apex 3.2.0 (user)",
    "auth 3.5.1 (core)",
    "custom-metadata 2.2.13 (user)",
    "data 3.2.4 (core)",
    "deploy-retrieve 3.5.3 (core)",
    "dev 1.1.2 (user)",
    "env 2.1.18 (user)",
    "info 3.1.2 (core)",
    "limits 3.2.1 (core)",
    "marketplace 1.1.0 (core)",
    "org 3.6.3 (core)",
    "packaging 1.21.1 (user)",
    "schema 3.2.0 (core)",
    "settings 2.1.2 (core)",
    "sobject 1.2.1 (core)",
    "source 3.2.2 (core)",
    "telemetry 3.1.17 (core)",
    "templates 56.1.0 (core)",
    "trust 3.5.1 (core)",
    "user 3.4.3 (core)"
  ]
}

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[shetzel]

Thanks for the super simple repro! It's definitely a problem. We'll have to track down where this is coming from. As a temporary workaround you can export NODE_OPTIONS="--max-old-space-size=8192" and that was enough memory to complete the conversion. Alternatively, you can find a version of the @salesforce/plugin-deploy-retrieve plugin that accomplishes the conversion without bumping the heap size.

[git2gus]

This issue has been linked to a new work item: W-16096068

[shetzel]

@jstvz - I found that sf plugins install deploy-retrieve@3.6.9 or any version previous results in a successful conversion. Any version of the plugin after that results in OOM errors. Until the bug is fixed, you can either bump the heap size or install deploy-retrieve@3.6.9.

[jmccotter]

I imagine this is happening in the same plugin within your CLI, but also getting the same thing with sf project deploy start -c in a large repo:

% sf project deploy start -c

<--- Last few GCs --->

[10019:0x7fb771116000]   463061 ms: Scavenge 3402.1 (4124.6) -> 3389.1 (4125.1) MB, 5.9 / 0.0 ms  (average mu = 0.324, current mu = 0.305) allocation failure; 
[10019:0x7fb771116000]   463092 ms: Scavenge 3404.9 (4125.1) -> 3392.1 (4126.1) MB, 5.1 / 0.0 ms  (average mu = 0.324, current mu = 0.305) allocation failure; 
[10019:0x7fb771116000]   463122 ms: Scavenge 3406.9 (4126.1) -> 3394.7 (4127.8) MB, 6.3 / 0.0 ms  (average mu = 0.324, current mu = 0.305) allocation failure; 

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x109fd2ed5 node::Abort() [/Users/justinmccotter/.nvm/versions/node/v18.20.4/bin/node]
 2: 0x109fd30b5 node::OOMErrorHandler(char const*, bool) [/Users/justinmccotter/.nvm/versions/node/v18.20.4/bin/node]
 3: 0x10a15296c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/justinmccotter/.nvm/versions/node/v18.20.4/bin/node]
 4: 0x10a317125 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/justinmccotter/.nvm/versions/node/v18.20.4/bin/node]
 5: 0x10a31b402 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/Users/justinmccotter/.nvm/versions/node/v18.20.4/bin/node]
 6: 0x10a317e6c v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*, v8::GCCallbackFlags) [/Users/justinmccotter/.nvm/versions/node/v18.20.4/bin/node]
 7: 0x10a314c64 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/justinmccotter/.nvm/versions/node/v18.20.4/bin/node]
 8: 0x10a30733c v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/justinmccotter/.nvm/versions/node/v18.20.4/bin/node]
 9: 0x10a307d15 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/justinmccotter/.nvm/versions/node/v18.20.4/bin/node]
10: 0x10a2eaf3e v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/justinmccotter/.nvm/versions/node/v18.20.4/bin/node]
11: 0x10a719030 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/justinmccotter/.nvm/versions/node/v18.20.4/bin/node]
12: 0x10ab0b6b9 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/justinmccotter/.nvm/versions/node/v18.20.4/bin/node]
13: 0x10ab0c789 Builtins_StringAdd_CheckNone [/Users/justinmccotter/.nvm/versions/node/v18.20.4/bin/node]
14: 0x11004ab03 
zsh: abort      sf project deploy start -c

{
  "architecture": "darwin-x64",
  "cliVersion": "@salesforce/cli/2.48.6",
  "nodeVersion": "node-v18.20.4",
  "osVersion": "Darwin 23.5.0",
  "rootPath": "/Users/justinmccotter/.nvm/versions/node/v18.20.4/lib/node_modules/@salesforce/cli",
  "shell": "zsh",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.1.4 (core)",
    "@oclif/plugin-commands 4.0.4 (core)",
    "@oclif/plugin-help 6.2.3 (core)",
    "@oclif/plugin-not-found 3.2.6 (core)",
    "@oclif/plugin-plugins 5.3.2 (core)",
    "@oclif/plugin-search 1.2.0 (core)",
    "@oclif/plugin-update 4.4.3 (core)",
    "@oclif/plugin-version 2.2.4 (core)",
    "@oclif/plugin-warn-if-update-available 3.1.6 (core)",
    "@oclif/plugin-which 3.2.4 (core)",
    "@salesforce/cli 2.48.6 (core)",
    "apex 3.2.0 (core)",
    "auth 3.6.27 (core)",
    "data 3.5.2 (core)",
    "deploy-retrieve 3.9.9 (core)",
    "info 3.3.11 (core)",
    "limits 3.3.15 (core)",
    "marketplace 1.2.17 (core)",
    "org 4.3.0 (core)",
    "packaging 2.6.1 (core)",
    "schema 3.3.16 (core)",
    "settings 2.3.6 (core)",
    "sobject 1.4.14 (core)",
    "source 3.4.8 (core)",
    "telemetry 3.5.1 (core)",
    "templates 56.3.0 (core)",
    "trust 3.7.10 (core)",
    "user 3.5.17 (core)",
    "@salesforce/sfdx-plugin-lwc-test 1.1.0 (user) published 302 days ago (Mon Sep 11 2023) (latest is 1.2.0)"
  ]
}

I used the export NODE_OPTIONS to get it working and watched Node hit almost 7GB of memory which seems a bit excessive.

[jmccotter]

On a later run, it's hitting 7.7GB. It's also taking significantly longer to run just to return no changes to deploy or only deploy a file or two. Downgrading the CLI version to make it reasonable to work with.

[shetzel]

@jmccotter - I found the problem and have a fix ready. I'm working on tests to prevent regressions. The bug is around multiple custom labels files within a project, particularly with 1 small file (discovered first) and 1 or more large files. The fix should go into this week's CLI release candidate.