Mainnet crash on 1.7.2

[wemeetagain]

Reported on discord: https://discord.com/channels/593655374469660673/593655641445367808/1093575346474979328

mainnet crash on 1.7.2

Apr-06 11:05:41.000[]                 info: Synced - slot: 6164126 - head: 6164126 0x6ed3…f6aa - execution: valid(0xdad7…264d) - finalized: 0xaba9…89b1:192626 - peers: 50

<--- Last few GCs --->

[5892:0x5ecba50] 438607189 ms: Scavenge (reduce) 3691.2 (3935.5) -> 3691.1 (3936.5) MB, 8.0 / 0.0 ms  (average mu = 0.981, current mu = 0.868) allocation failure;
[5892:0x5ecba50] 438607818 ms: Mark-sweep (reduce) 4212.3 (4456.9) -> 3155.6 (4335.4) MB, 166.0 / 0.1 ms  (+ 17.5 ms in 341 steps since start of marking, biggest step 0.1 ms, walltime since start of marking 766 ms) (average mu = 0.975, current mu = 0.904)

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb7b3e0 node::Abort() [node]
 2: 0xa8c8aa  [node]
 3: 0xd69100 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xd694a7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xf46ba5  [node]
 6: 0xf5908d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 7: 0xf3378e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 8: 0xf34b57 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 9: 0xf150a0 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
10: 0xf0cb14 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [node]
11: 0xf0eea8 v8::internal::FactoryBase<v8::internal::Factory>::NewRawTwoByteString(int, v8::internal::AllocationType) [node]
12: 0x11ec145 v8::internal::String::SlowFlatten(v8::internal::Isolate*, v8::internal::Handle<v8::internal::ConsString>, v8::internal::AllocationType) [node]
13: 0xd74f0c v8::String::Utf8Length(v8::Isolate*) const [node]
14: 0xc70c81 node::StringBytes::Size(v8::Isolate*, v8::Local<v8::Value>, node::encoding) [node]
15: 0xc66cd7 node::StreamBase::Writev(v8::FunctionCallbackInfo<v8::Value> const&) [node]
16: 0xc68d25 void node::StreamBase::JSMethod<&node::StreamBase::Writev>(v8::FunctionCallbackInfo<v8::Value> const&) [node]
17: 0x168d3ef  [node]
./lodestar: line 7:  5892 Aborted                 node --trace-deprecation --max-old-space-size=4096 ./packages/cli/bin/lodestar.js "$@"

[nflaig]

There seems to be a memory leak, heap bytes and rss are slowly increasing (2 days timeframe)

Beacon node heap profile after running for ~5 minutes (1362403.clinic-heapprofiler.html.zip)

Beacon node heap profile after running for ~2 days (252794.clinic-heapprofiler.html.zip)

Beacon node heap profile after running for ~7 days (1401875.clinic-heapprofiler.html.zip)

[philknows]

I'm seeing something very similar on my node as well. But I can't tell if it had something to do with the Shapella fork as it has a noticeable increase around the same time... but it has been growing since.

Times are in UTC.

[nflaig]

Memory usage seems stable again (7 days timeframe), running from unstable branch, not really sure what fixed it but looks good, consistently below 4GB and I am running a VC with 2k keys (on goerli)

[philknows]

Is this unstable on mainnet for the last 7 days? Without much more information provided, it feels like this issue is a dead-end investigation of what potentially happened here...

[dapplion]

• Closing for v1.9.0-rc.0: Memory leak #5549, and noted there as preceding incident