Heapdump generation as part of Node core

[mhdawson]

Was discussed in Diagnostic summit today. Consensus was that ability to generate heapdumps should be in core versus needing modules like Heapdump.

We should work on

• Pulling sampling heap profiler into core
• Offering ability to generate heapdump as part of core APIs and on events like Unhandled Exception, etc. through command line options.

Lets use this issue to flesh out the APIs' functionality that should be exposed.

[mhdawson]

For references, older issue with some past discussions on the topic: nodejs/TSC#257

[sam-github]

Note that the core capability is already present:

const fs = require('fs');
const inspector = require('inspector');
const session = new inspector.Session();

session.connect();

fs.writeFileSync('_.heapsnapshot', '');

session.on('HeapProfiler.addHeapSnapshotChunk', function(m) {
  fs.appendFileSync('_.heapsnapshot', m.params.chunk);
});

session.post('HeapProfiler.takeHeapSnapshot', null, function(err, r) {
  console.log('Runtime.takeHeapSnapshot done:', err, r)
  session.disconnect();
});

It probably needs an example usage, like: https://nodejs.org/api/inspector.html#inspector_cpu_profiler

[jasnell]

I'll be opening a pr that essentially pulls in node-heapdump ... At least most of it. Expect that soonish

[misterdjules]

@jasnell My experience with node-heapdump has been that very often, it's not usable because it requires the process to be able to allocate as much memory as what is used for the JS heap, and that leads to the process actually crashing. Has there been further discussions during the summit about solving that problem?

I know for instance that there's been discussions in the past about being able to generate heapdumps from core dumps (e.g. in nodejs/post-mortem#13).

I'm not suggesting that this should block moving node-heapdump into node's core, but I'm wondering whether we should also think about a different approach that doesn't have node-heapdump's issues.

That being said, I'm also interested in hearing about the experience that others had with node-heapdump, there may be something that I'm missing that would solve the issues that I've experienced.

[jasnell]

Yes, this was discussed extensively today and there will be parallel efforts to see what is possible around using coredumps. The node-heapdump approach definitely can be imperfect so I think the approach, as I understood it from the conversation today, is to look at multiple solutions.

…

On Thu, Mar 7, 2019, 20:20 Julien Gilli ***@***.***> wrote: @jasnell <https://github.com/jasnell> My experience with node-heapdump has been that very often, it's not usable because it requires the process to be able to allocate as much memory as what is used for the JS heap, and that leads to the process actually crashing. Has there been further discussions during the summit about solving that problem? I know for instance that there's been discussions in the past about being able to generate heapdumps from core dumps (e.g. in nodejs/post-mortem#13 <nodejs/post-mortem#13>). I'm not suggesting that this should block moving node-heapdump into node's core, but I'm wondering whether we should also think about a different approach that doesn't have node-heapdump's issues. That being said, I'm also interested in hearing about the experience that others had with node-heapdump, there may be something that I'm missing that would solve the issues that I've experienced. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#279 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAa2eTcrH5uoSKKfzkU9VontIJ2PIxaSks5vUWaGgaJpZM4bjbUC> .

[misterdjules]

@jasnell

There will be parallel efforts to see what is possible around using coredump.

Also just wanted to make it clear that I'm not saying generating heap dumps from core dumps is necessarily what I'm looking for. I'm looking for a robust/reliable way to generate data that gives insights into the content of the JS heap, whether this is done via core dumps or another mechanism is an implementation detail for me.

[joyeecheung]

Previous discussions stalled on the format in which the heap dump should be returned to the user land: nodejs/node#23328

Note that we already have internalBinding('heap_utils').createHeapDump() that returns the heap snapshot as a JSON.parsed object. So if we are exposing this functionality to the users I'd expect we work on top of that instead of pulling in node-heapdump, which mostly just adds some convenient signal handlers and a file writer for that.

[naugtur]

The convenience of using signals by just adding a --require heapdump when running existing code without modification was nice though. Being able to do that with a flag (because someone might be using signals for something else I suppose) would be a nice equivalent.
If it doesn't warrant adding a flag, it makes sense to keep a simplistic heapdump module out there to enable the functionality.

[alexkozy]

I'd like to mention CLI tools that I built over last weekends. It utilizes inspector protocol to capture different profiles and tracing for given node process, it supports child processes, ondemand profiling and will support workers as well soon. It can capture sampling heap profiler.
Please take a look: https://github.com/ak239/thetool . It is npm installable module with only one dependency (update-notifier).

[vmarchaud]

I believe this can be closed since both the heap sampling profiler and the heapsnapshot can be launched with Node core.

[mhdawson]

@vmarchaud would you happen to have the link to the PR for adding the sampling profiler? It would be good to have that in this issue and then I think I can close.

[vmarchaud]

@mhdawson Here is it : nodejs/node#27596

[mhdawson]

@vmarchaud thanks !